Speed test matrix
Qwen3.8-Flash-Next on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.80 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":false}' --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context13.2K622 ms TTFT
16K context13.7K1,179 ms TTFT
32K context13.8K2,324 ms TTFT
64K context13.9K4,601 ms TTFT
128K context13.7K9,365 ms TTFT
251K context13.2K19,032 ms TTFT
256K context13.2K19,456 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 132.398 ms TTFT | 242.9178 ms TTFT | 426.6327 ms TTFT | 721.9524 ms TTFT | 1,164.4775 ms TTFT | 1,796.31,037 ms TTFT | 2,422.41,648 ms TTFT | 3,878.93,365 ms TTFT |
| 16K | 133.0109 ms TTFT | 243.2190 ms TTFT | 418.8401 ms TTFT | 721.1520 ms TTFT | 1,163.3626 ms TTFT | 1,793.61,386 ms TTFT | 2,399.32,003 ms TTFT | Over capacity |
| 32K | 133.1133 ms TTFT | 243.3218 ms TTFT | 431.5420 ms TTFT | 724.3449 ms TTFT | 1,145.2932 ms TTFT | 1,780.11,189 ms TTFT | Over capacity | Over capacity |
| 64K | 129.5190 ms TTFT | 242.9283 ms TTFT | 427.3470 ms TTFT | 723.1684 ms TTFT | 1,141.8951 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 130.2308 ms TTFT | 245.1352 ms TTFT | 433.5609 ms TTFT | 716.2998 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 132.0625 ms TTFT | 237.1918 ms TTFT | 431.01,392 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.2K | — | 622 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.7K | — | 1,179 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.8K | — | 2,324 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.9K | — | 4,601 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.7K | — | 9,365 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.2K | — | 19,032 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.2K | — | 19,456 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 132.3 | 135.4 | 98 ms | 7.4 ms | 1.0 | 1 | 3,968 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 242.9 | 121.9 | 178 ms | 8.2 ms | 2.0 | 2 | 7,286 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 426.6 | 101.7 | 327 ms | 9.8 ms | 4.0 | 0 | 12,795 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 721.9 | 82.2 | 524 ms | 12.2 ms | 8.0 | 0 | 21,658 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,164.4 | 68.9 | 775 ms | 14.5 ms | 16.0 | 0 | 34,926 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,796.3 | 53.7 | 1,037 ms | 18.6 ms | 32.0 | 0 | 53,869 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,422.4 | 33.4 | 1,648 ms | 29.9 ms | 64.0 | 0 | 72,669 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,878.9 | 28.8 | 3,365 ms | 34.7 ms | 128.0 | 0 | 116,373 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 133.0 | 133.0 | 109 ms | 7.5 ms | 1.0 | 1 | 3,990 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 243.2 | 122.2 | 190 ms | 8.2 ms | 2.0 | 2 | 7,295 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 418.8 | 105.0 | 401 ms | 9.5 ms | 4.0 | 0 | 12,559 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 721.1 | 90.3 | 520 ms | 11.1 ms | 8.0 | 0 | 21,627 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,163.3 | 72.5 | 626 ms | 13.8 ms | 16.0 | 0 | 34,897 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,793.6 | 55.5 | 1,386 ms | 18.0 ms | 32.0 | 0 | 53,791 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,399.3 | 36.9 | 2,003 ms | 27.1 ms | 64.0 | 0 | 71,980 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 133.1 | 136.7 | 133 ms | 7.3 ms | 1.0 | 1 | 3,992 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 243.3 | 122.3 | 218 ms | 8.2 ms | 2.0 | 2 | 7,299 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 431.5 | 107.7 | 420 ms | 9.3 ms | 4.0 | 0 | 12,942 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 724.3 | 90.2 | 449 ms | 11.1 ms | 8.0 | 0 | 21,723 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,145.2 | 71.4 | 932 ms | 14.0 ms | 16.0 | 0 | 34,353 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,780.1 | 55.2 | 1,189 ms | 18.1 ms | 32.0 | 0 | 53,376 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 129.5 | 132.5 | 190 ms | 7.5 ms | 1.0 | 1 | 3,885 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 242.9 | 125.4 | 283 ms | 8.0 ms | 2.0 | 2 | 7,276 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 427.3 | 105.4 | 470 ms | 9.5 ms | 4.0 | 0 | 12,819 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 723.1 | 90.1 | 684 ms | 11.1 ms | 8.0 | 0 | 21,681 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,141.8 | 70.4 | 951 ms | 14.2 ms | 16.0 | 0 | 34,231 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 130.2 | 134.6 | 308 ms | 7.4 ms | 1.0 | 1 | 3,906 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 245.1 | 124.2 | 352 ms | 8.1 ms | 2.0 | 1 | 7,351 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 433.5 | 108.3 | 609 ms | 9.2 ms | 4.0 | 0 | 13,002 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 716.2 | 89.6 | 998 ms | 11.2 ms | 8.0 | 0 | 21,476 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 132.0 | 135.6 | 625 ms | 7.4 ms | 1.0 | 1 | 3,960 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 237.1 | 119.9 | 918 ms | 8.3 ms | 2.0 | 1 | 7,112 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 431.0 | 107.1 | 1,392 ms | 9.3 ms | 4.0 | 0 | 12,925 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.