Speed test matrix
Qwen3.8-Flash-Next on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":true}' --moe-backend tritonPrefill
prompt tok/s by context length8K context14.2K576 ms TTFT
16K context15K1,080 ms TTFT
32K context15.1K2,124 ms TTFT
64K context15.2K4,221 ms TTFT
128K context15.1K8,476 ms TTFT
251K context14.7K17,078 ms TTFT
256K context14.6K17,504 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 108.295 ms TTFT | 179.31,225 ms TTFT | 304.2219 ms TTFT | 517.0315 ms TTFT | 801.7476 ms TTFT | 1,216.7937 ms TTFT | 1,798.1992 ms TTFT | Over capacity |
| 16K | 107.6113 ms TTFT | 179.6167 ms TTFT | 307.1269 ms TTFT | 513.7345 ms TTFT | 817.4735 ms TTFT | 1,238.1867 ms TTFT | Over capacity | Over capacity |
| 32K | 107.295 ms TTFT | 178.9118 ms TTFT | 304.7152 ms TTFT | 504.8229 ms TTFT | 781.9405 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 107.0209 ms TTFT | 177.7277 ms TTFT | 300.4381 ms TTFT | 502.9605 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 106.3314 ms TTFT | 175.9400 ms TTFT | 298.6587 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 105.6700 ms TTFT | 174.1810 ms TTFT | 296.21,260 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.2K | — | 576 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15K | — | 1,080 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.1K | — | 2,124 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.2K | — | 4,221 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.1K | — | 8,476 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.7K | — | 17,078 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.6K | — | 17,504 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 108.2 | 108.3 | 95 ms | 9.2 ms | 1.0 | 0 | 3,246 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 179.3 | 86.9 | 1,225 ms | 11.5 ms | 2.0 | 0 | 5,380 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 304.2 | 76.1 | 219 ms | 13.1 ms | 4.0 | 0 | 9,124 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 517.0 | 64.7 | 315 ms | 15.5 ms | 8.0 | 0 | 15,504 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 801.7 | 50.2 | 476 ms | 19.9 ms | 16.0 | 0 | 24,048 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,216.7 | 38.5 | 937 ms | 26.0 ms | 32.0 | 0 | 36,480 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,798.1 | 28.0 | 992 ms | 35.7 ms | 64.0 | 0 | 53,888 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 107.6 | 107.5 | 113 ms | 9.3 ms | 1.0 | 0 | 3,226 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 179.6 | 89.8 | 167 ms | 11.1 ms | 2.0 | 0 | 5,386 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 307.1 | 76.8 | 269 ms | 13.0 ms | 4.0 | 0 | 9,212 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 513.7 | 64.2 | 345 ms | 15.6 ms | 8.0 | 0 | 15,408 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 817.4 | 51.3 | 735 ms | 19.5 ms | 16.0 | 0 | 24,512 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,238.1 | 38.7 | 867 ms | 25.8 ms | 32.0 | 0 | 37,120 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 107.2 | 107.2 | 95 ms | 9.3 ms | 1.0 | 0 | 3,217 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 178.9 | 89.4 | 118 ms | 11.2 ms | 2.0 | 0 | 5,366 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 304.7 | 76.1 | 152 ms | 13.1 ms | 4.0 | 0 | 9,140 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 504.8 | 63.0 | 229 ms | 15.9 ms | 8.0 | 0 | 15,144 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 781.9 | 48.8 | 405 ms | 20.5 ms | 16.0 | 0 | 23,456 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 107.0 | 107.0 | 209 ms | 9.3 ms | 1.0 | 0 | 3,208 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 177.7 | 88.8 | 277 ms | 11.3 ms | 2.0 | 0 | 5,332 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 300.4 | 74.9 | 381 ms | 13.4 ms | 4.0 | 0 | 9,012 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 502.9 | 62.5 | 605 ms | 16.0 ms | 8.0 | 0 | 15,080 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 106.3 | 106.5 | 314 ms | 9.4 ms | 1.0 | 0 | 3,189 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 175.9 | 87.9 | 400 ms | 11.4 ms | 2.0 | 0 | 5,278 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 298.6 | 74.4 | 587 ms | 13.4 ms | 4.0 | 0 | 8,956 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 105.6 | 105.9 | 700 ms | 9.4 ms | 1.0 | 0 | 3,169 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 174.1 | 87.1 | 810 ms | 11.5 ms | 2.0 | 0 | 5,220 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 296.2 | 74.0 | 1,260 ms | 13.5 ms | 4.0 | 0 | 8,884 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.