Speed test matrix
Qwen3.8-Flash-Next on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+g50a061f79 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g50a061f79
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
50a061f792f3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_MMAP=1 "$HOME/env_rtxpro6k-qwen-flash-next-vllm-pr-54129/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --moe-backend triton
Prefill
prompt tok/s by context length8K context13.4K612 ms TTFT
16K context14.2K1,137 ms TTFT
32K context14.4K2,228 ms TTFT
64K context14.4K4,445 ms TTFT
128K context14.2K9,004 ms TTFT
251K context13.8K18,146 ms TTFT
256K context13.7K18,664 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 101.8629 ms TTFT | 165.01,885 ms TTFT | 274.1770 ms TTFT | 442.1366 ms TTFT | 681.51,059 ms TTFT | 1,029.41,092 ms TTFT | 1,574.42,251 ms TTFT | Over capacity |
| 16K | 101.11,192 ms TTFT | 164.51,217 ms TTFT | 272.3356 ms TTFT | 436.6524 ms TTFT | 673.3825 ms TTFT | 1,038.51,633 ms TTFT | Over capacity | Over capacity |
| 32K | 101.32,605 ms TTFT | 164.52,274 ms TTFT | 270.8601 ms TTFT | 427.5364 ms TTFT | 663.2507 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 101.34,483 ms TTFT | 162.84,536 ms TTFT | 272.3551 ms TTFT | 430.8852 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 100.99,045 ms TTFT | 163.09,104 ms TTFT | 271.0686 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 100.418,158 ms TTFT | 162.69,131 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.4K | — | 612 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.2K | — | 1,137 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.4K | — | 2,228 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.4K | — | 4,445 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.2K | — | 9,004 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.8K | — | 18,146 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.7K | — | 18,664 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 101.8 | 101.8 | 629 ms | 9.8 ms | 1.0 | 0 | 3,052 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 165.0 | 82.5 | 1,885 ms | 12.1 ms | 2.0 | 0 | 4,950 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 274.1 | 68.5 | 770 ms | 14.6 ms | 4.0 | 0 | 8,220 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 442.1 | 55.4 | 366 ms | 18.1 ms | 8.0 | 0 | 13,256 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 681.5 | 42.2 | 1,059 ms | 23.7 ms | 16.0 | 0 | 20,432 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,029.4 | 32.6 | 1,092 ms | 30.6 ms | 32.0 | 0 | 30,880 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,574.4 | 24.9 | 2,251 ms | 40.2 ms | 64.0 | 0 | 47,232 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 101.1 | 101.1 | 1,192 ms | 9.9 ms | 1.0 | 0 | 3,031 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 164.5 | 82.5 | 1,217 ms | 12.1 ms | 2.0 | 0 | 4,936 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 272.3 | 68.1 | 356 ms | 14.7 ms | 4.0 | 0 | 8,168 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 436.6 | 54.9 | 524 ms | 18.2 ms | 8.0 | 0 | 13,096 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 673.3 | 42.0 | 825 ms | 23.8 ms | 16.0 | 0 | 20,192 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,038.5 | 32.3 | 1,633 ms | 30.9 ms | 32.0 | 0 | 31,136 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 101.3 | 101.3 | 2,605 ms | 9.9 ms | 1.0 | 0 | 3,039 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 164.5 | 82.3 | 2,274 ms | 12.2 ms | 2.0 | 0 | 4,932 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 270.8 | 67.6 | 601 ms | 14.8 ms | 4.0 | 0 | 8,124 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 427.5 | 53.4 | 364 ms | 18.7 ms | 8.0 | 0 | 12,824 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 663.2 | 41.5 | 507 ms | 24.1 ms | 16.0 | 0 | 19,888 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 101.3 | 101.4 | 4,483 ms | 9.9 ms | 1.0 | 0 | 3,038 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 162.8 | 81.7 | 4,536 ms | 12.2 ms | 2.0 | 0 | 4,884 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 272.3 | 67.9 | 551 ms | 14.7 ms | 4.0 | 0 | 8,168 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 430.8 | 53.7 | 852 ms | 18.6 ms | 8.0 | 0 | 12,920 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 100.9 | 101.0 | 9,045 ms | 9.9 ms | 1.0 | 0 | 3,027 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 163.0 | 81.6 | 9,104 ms | 12.3 ms | 2.0 | 0 | 4,888 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 271.0 | 67.4 | 686 ms | 14.8 ms | 4.0 | 0 | 8,128 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 100.4 | 100.6 | 18,158 ms | 9.9 ms | 1.0 | 0 | 3,013 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 162.6 | 81.5 | 9,131 ms | 12.3 ms | 2.0 | 0 | 4,876 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.