Speed test matrix
Qwen3.8-Flash-Next on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":false}' --moe-backend tritonPrefill
prompt tok/s by context length8K context13.8K593 ms TTFT
16K context14.2K1,136 ms TTFT
32K context14.5K2,216 ms TTFT
64K context14.6K4,395 ms TTFT
128K context14.4K8,889 ms TTFT
251K context13.9K18,106 ms TTFT
256K context13.8K18,573 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 111.971 ms TTFT | 172.3780 ms TTFT | 315.8223 ms TTFT | 546.7257 ms TTFT | 927.4322 ms TTFT | 1,486.3664 ms TTFT | 2,290.4830 ms TTFT | 3,084.61,590 ms TTFT |
| 16K | 111.686 ms TTFT | 172.1117 ms TTFT | 315.2254 ms TTFT | 544.8277 ms TTFT | 923.1553 ms TTFT | 1,483.9925 ms TTFT | 2,283.9923 ms TTFT | Over capacity |
| 32K | 111.7112 ms TTFT | 171.8169 ms TTFT | 315.0203 ms TTFT | 547.9314 ms TTFT | 922.2434 ms TTFT | 1,479.8730 ms TTFT | 2,257.41,244 ms TTFT | Over capacity |
| 64K | 111.0164 ms TTFT | 171.1230 ms TTFT | 313.3291 ms TTFT | 536.4438 ms TTFT | 905.7724 ms TTFT | 1,443.01,326 ms TTFT | Over capacity | Over capacity |
| 128K | 109.9303 ms TTFT | 169.1379 ms TTFT | 311.0534 ms TTFT | 539.3929 ms TTFT | 892.71,612 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 109.6635 ms TTFT | 169.1813 ms TTFT | 307.61,246 ms TTFT | 476.82,098 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.8K | — | 593 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.2K | — | 1,136 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.5K | — | 2,216 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.6K | — | 4,395 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.4K | — | 8,889 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.9K | — | 18,106 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.8K | — | 18,573 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 111.9 | 111.9 | 71 ms | 8.9 ms | 1.0 | 0 | 3,357 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 172.3 | 84.4 | 780 ms | 11.8 ms | 2.0 | 0 | 5,170 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 315.8 | 78.9 | 223 ms | 12.7 ms | 4.0 | 0 | 9,472 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 546.7 | 68.4 | 257 ms | 14.6 ms | 8.0 | 0 | 16,400 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 927.4 | 57.8 | 322 ms | 17.3 ms | 16.0 | 0 | 27,808 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,486.3 | 46.7 | 664 ms | 21.4 ms | 32.0 | 0 | 44,576 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,290.4 | 35.7 | 830 ms | 28.0 ms | 64.0 | 0 | 68,672 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,084.6 | 23.9 | 1,590 ms | 41.8 ms | 128.0 | 0 | 92,544 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 111.6 | 111.6 | 86 ms | 9.0 ms | 1.0 | 0 | 3,348 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 172.1 | 86.0 | 117 ms | 11.6 ms | 2.0 | 0 | 5,160 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 315.2 | 78.9 | 254 ms | 12.7 ms | 4.0 | 0 | 9,456 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 544.8 | 68.0 | 277 ms | 14.7 ms | 8.0 | 0 | 16,344 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 923.1 | 58.0 | 553 ms | 17.3 ms | 16.0 | 0 | 27,680 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,483.9 | 46.6 | 925 ms | 21.5 ms | 32.0 | 0 | 44,512 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,283.9 | 35.5 | 923 ms | 28.2 ms | 64.0 | 0 | 68,480 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 111.7 | 111.7 | 112 ms | 9.0 ms | 1.0 | 0 | 3,351 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 171.8 | 85.9 | 169 ms | 11.6 ms | 2.0 | 0 | 5,152 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 315.0 | 78.7 | 203 ms | 12.7 ms | 4.0 | 0 | 9,448 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 547.9 | 68.3 | 314 ms | 14.6 ms | 8.0 | 0 | 16,432 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 922.2 | 57.4 | 434 ms | 17.4 ms | 16.0 | 0 | 27,664 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,479.8 | 46.1 | 730 ms | 21.7 ms | 32.0 | 0 | 44,394 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,257.4 | 35.0 | 1,244 ms | 28.6 ms | 64.0 | 0 | 67,712 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 111.0 | 111.0 | 164 ms | 9.0 ms | 1.0 | 0 | 3,329 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 171.1 | 85.7 | 230 ms | 11.7 ms | 2.0 | 0 | 5,132 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 313.3 | 78.3 | 291 ms | 12.8 ms | 4.0 | 0 | 9,400 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 536.4 | 67.0 | 438 ms | 14.9 ms | 8.0 | 0 | 16,088 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 905.7 | 56.4 | 724 ms | 17.7 ms | 16.0 | 0 | 27,171 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,443.0 | 44.8 | 1,326 ms | 22.3 ms | 32.0 | 0 | 43,263 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 109.9 | 110.0 | 303 ms | 9.1 ms | 1.0 | 0 | 3,297 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 169.1 | 84.6 | 379 ms | 11.8 ms | 2.0 | 0 | 5,074 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 311.0 | 77.7 | 534 ms | 12.9 ms | 4.0 | 0 | 9,328 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 539.3 | 67.1 | 929 ms | 14.9 ms | 8.0 | 0 | 16,176 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 892.7 | 55.5 | 1,612 ms | 18.0 ms | 16.0 | 0 | 26,768 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 109.6 | 109.9 | 635 ms | 9.1 ms | 1.0 | 0 | 3,287 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 169.1 | 84.6 | 813 ms | 11.8 ms | 2.0 | 0 | 5,070 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 307.6 | 76.9 | 1,246 ms | 13.0 ms | 4.0 | 0 | 9,227 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 476.8 | 64.6 | 2,098 ms | 15.5 ms | 8.0 | 0 | 14,299 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.