Speed test matrix
Qwen3.8-Flash-Next-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.1.dev5+ge77daef89 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":false}' --moe-backend tritonPrefill
prompt tok/s by context length8K context15.2K538 ms TTFT
16K context15.8K1,025 ms TTFT
32K context16K2,014 ms TTFT
64K context16K4,015 ms TTFT
128K context15.8K8,089 ms TTFT
251K context15.3K16,420 ms TTFT
256K context15.1K16,959 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 113.976 ms TTFT | 194.7465 ms TTFT | 355.3183 ms TTFT | 624.6285 ms TTFT | 1,021.9454 ms TTFT | 1,640.1850 ms TTFT | 2,342.5982 ms TTFT | 3,552.32,107 ms TTFT |
| 16K | 113.495 ms TTFT | 196.3152 ms TTFT | 354.7230 ms TTFT | 630.2272 ms TTFT | 1,019.9639 ms TTFT | 1,634.4959 ms TTFT | 2,337.41,382 ms TTFT | Over capacity |
| 32K | 113.582 ms TTFT | 194.3119 ms TTFT | 356.4140 ms TTFT | 623.9222 ms TTFT | 1,015.2384 ms TTFT | 1,626.6686 ms TTFT | 2,303.01,272 ms TTFT | Over capacity |
| 64K | 113.0184 ms TTFT | 194.5251 ms TTFT | 350.3332 ms TTFT | 613.7602 ms TTFT | 1,007.2910 ms TTFT | 1,612.21,527 ms TTFT | Over capacity | Over capacity |
| 128K | 112.1283 ms TTFT | 192.5409 ms TTFT | 348.3547 ms TTFT | 613.9877 ms TTFT | 990.21,544 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 111.7574 ms TTFT | 190.3795 ms TTFT | 344.71,225 ms TTFT | 598.62,064 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.2K | — | 538 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.8K | — | 1,025 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16K | — | 2,014 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16K | — | 4,015 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.8K | — | 8,089 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 15.3K | — | 16,420 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15.1K | — | 16,959 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 113.9 | 113.9 | 76 ms | 8.8 ms | 1.0 | 0 | 3,418 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 194.7 | 96.3 | 465 ms | 10.4 ms | 2.0 | 0 | 5,838 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 355.3 | 88.8 | 183 ms | 11.3 ms | 4.0 | 0 | 10,656 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 624.6 | 78.2 | 285 ms | 12.8 ms | 8.0 | 0 | 18,738 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,021.9 | 63.7 | 454 ms | 15.7 ms | 16.0 | 0 | 30,656 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,640.1 | 51.6 | 850 ms | 19.4 ms | 32.0 | 0 | 49,184 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,342.5 | 36.4 | 982 ms | 27.5 ms | 64.0 | 0 | 70,272 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,552.3 | 27.4 | 2,107 ms | 36.5 ms | 128.0 | 0 | 106,496 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 113.4 | 113.3 | 95 ms | 8.8 ms | 1.0 | 0 | 3,401 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 196.3 | 98.1 | 152 ms | 10.2 ms | 2.0 | 0 | 5,888 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 354.7 | 88.7 | 230 ms | 11.3 ms | 4.0 | 0 | 10,640 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 630.2 | 78.6 | 272 ms | 12.7 ms | 8.0 | 0 | 18,904 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,019.9 | 63.8 | 639 ms | 15.7 ms | 16.0 | 0 | 30,592 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,634.4 | 51.1 | 959 ms | 19.6 ms | 32.0 | 0 | 49,024 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,337.4 | 36.0 | 1,382 ms | 27.7 ms | 64.0 | 0 | 70,080 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 113.5 | 113.5 | 82 ms | 8.8 ms | 1.0 | 0 | 3,405 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 194.3 | 97.2 | 119 ms | 10.3 ms | 2.0 | 0 | 5,828 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 356.4 | 89.0 | 140 ms | 11.2 ms | 4.0 | 0 | 10,692 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 623.9 | 77.8 | 222 ms | 12.8 ms | 8.0 | 0 | 18,712 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,015.2 | 63.4 | 384 ms | 15.8 ms | 16.0 | 0 | 30,448 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,626.6 | 50.8 | 686 ms | 19.7 ms | 32.0 | 0 | 48,800 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,303.0 | 36.0 | 1,272 ms | 27.8 ms | 64.0 | 0 | 69,056 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 113.0 | 113.0 | 184 ms | 8.8 ms | 1.0 | 0 | 3,388 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 194.5 | 97.2 | 251 ms | 10.3 ms | 2.0 | 0 | 5,836 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 350.3 | 87.3 | 332 ms | 11.4 ms | 4.0 | 0 | 10,508 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 613.7 | 76.4 | 602 ms | 13.1 ms | 8.0 | 0 | 18,408 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,007.2 | 62.2 | 910 ms | 16.1 ms | 16.0 | 0 | 30,208 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,612.2 | 49.2 | 1,527 ms | 20.3 ms | 32.0 | 0 | 48,352 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 112.1 | 112.1 | 283 ms | 8.9 ms | 1.0 | 0 | 3,361 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 192.5 | 96.2 | 409 ms | 10.4 ms | 2.0 | 0 | 5,776 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 348.3 | 86.9 | 547 ms | 11.5 ms | 4.0 | 0 | 10,448 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 613.9 | 76.2 | 877 ms | 13.1 ms | 8.0 | 0 | 18,416 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 990.2 | 61.2 | 1,544 ms | 16.3 ms | 16.0 | 0 | 29,696 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 111.7 | 111.9 | 574 ms | 8.9 ms | 1.0 | 0 | 3,352 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 190.3 | 95.3 | 795 ms | 10.5 ms | 2.0 | 0 | 5,708 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 344.7 | 86.0 | 1,225 ms | 11.6 ms | 4.0 | 0 | 10,340 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 598.6 | 74.5 | 2,064 ms | 13.4 ms | 8.0 | 0 | 17,952 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.