Speed test matrix
Qwen3.8-Flash-Next-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.1.dev5+ge77daef89 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.77 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":false}' --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context14.7K559 ms TTFT
16K context15.1K1,068 ms TTFT
32K context15.3K2,104 ms TTFT
64K context15.3K4,203 ms TTFT
128K context15K8,538 ms TTFT
251K context14.4K17,399 ms TTFT
256K context14.3K17,874 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 150.7110 ms TTFT | 270.5127 ms TTFT | 484.1179 ms TTFT | 810.4612 ms TTFT | 1,292.6634 ms TTFT | 1,856.01,362 ms TTFT | 2,850.42,585 ms TTFT | 3,956.04,323 ms TTFT |
| 16K | 145.0121 ms TTFT | 268.6155 ms TTFT | 496.8183 ms TTFT | 796.4556 ms TTFT | 1,278.3732 ms TTFT | 1,838.61,506 ms TTFT | 2,834.82,269 ms TTFT | Over capacity |
| 32K | 148.3149 ms TTFT | 273.0167 ms TTFT | 486.8207 ms TTFT | 803.7561 ms TTFT | 1,281.5862 ms TTFT | 1,814.91,797 ms TTFT | Over capacity | Over capacity |
| 64K | 146.9217 ms TTFT | 269.8229 ms TTFT | 480.5324 ms TTFT | 803.8802 ms TTFT | 1,250.71,396 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 149.9343 ms TTFT | 281.3387 ms TTFT | 481.2613 ms TTFT | 801.11,135 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 146.6708 ms TTFT | 269.8851 ms TTFT | 482.4989 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.7K | — | 559 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.1K | — | 1,068 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.3K | — | 2,104 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.3K | — | 4,203 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15K | — | 8,538 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.4K | — | 17,399 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.3K | — | 17,874 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 150.7 | 151.6 | 110 ms | 6.6 ms | 1.0 | 1 | 4,520 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 270.5 | 135.7 | 127 ms | 7.4 ms | 2.0 | 2 | 8,113 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 484.1 | 121.1 | 179 ms | 8.3 ms | 4.0 | 4 | 14,520 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 810.4 | 100.0 | 612 ms | 10.0 ms | 8.0 | 0 | 24,309 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,292.6 | 80.1 | 634 ms | 12.5 ms | 16.0 | 0 | 38,763 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,856.0 | 57.0 | 1,362 ms | 17.5 ms | 32.0 | 0 | 55,673 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,850.4 | 42.6 | 2,585 ms | 23.5 ms | 64.0 | 0 | 85,413 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,956.0 | 28.3 | 4,323 ms | 35.4 ms | 128.0 | 0 | 118,670 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 145.0 | 146.5 | 121 ms | 6.8 ms | 1.0 | 1 | 4,350 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 268.6 | 135.5 | 155 ms | 7.4 ms | 2.0 | 2 | 8,058 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 496.8 | 124.2 | 183 ms | 8.1 ms | 4.0 | 4 | 14,890 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 796.4 | 99.6 | 556 ms | 10.0 ms | 8.0 | 0 | 23,891 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,278.3 | 80.1 | 732 ms | 12.5 ms | 16.0 | 0 | 38,340 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,838.6 | 56.7 | 1,506 ms | 17.6 ms | 32.0 | 0 | 55,130 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,834.8 | 42.7 | 2,269 ms | 23.4 ms | 64.0 | 0 | 84,952 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 148.3 | 150.7 | 149 ms | 6.6 ms | 1.0 | 1 | 4,448 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 273.0 | 135.6 | 167 ms | 7.4 ms | 2.0 | 2 | 8,190 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 486.8 | 121.7 | 207 ms | 8.2 ms | 4.0 | 4 | 14,604 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 803.7 | 99.6 | 561 ms | 10.0 ms | 8.0 | 0 | 24,107 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,281.5 | 79.3 | 862 ms | 12.6 ms | 16.0 | 0 | 38,428 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,814.9 | 55.9 | 1,797 ms | 17.9 ms | 32.0 | 0 | 54,410 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 146.9 | 147.9 | 217 ms | 6.8 ms | 1.0 | 1 | 4,407 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 269.8 | 136.0 | 229 ms | 7.4 ms | 2.0 | 2 | 8,093 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 480.5 | 121.2 | 324 ms | 8.3 ms | 4.0 | 3 | 14,413 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 803.8 | 98.8 | 802 ms | 10.1 ms | 8.0 | 0 | 24,102 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,250.7 | 77.0 | 1,396 ms | 13.0 ms | 16.0 | 0 | 37,502 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 149.9 | 152.7 | 343 ms | 6.6 ms | 1.0 | 1 | 4,495 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 281.3 | 142.8 | 387 ms | 7.0 ms | 2.0 | 2 | 8,438 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 481.2 | 121.4 | 613 ms | 8.2 ms | 4.0 | 2 | 14,432 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 801.1 | 98.8 | 1,135 ms | 10.1 ms | 8.0 | 0 | 24,028 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 146.6 | 152.7 | 708 ms | 6.6 ms | 1.0 | 1 | 4,398 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 269.8 | 140.0 | 851 ms | 7.1 ms | 1.9 | 2 | 8,094 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 482.4 | 124.8 | 989 ms | 8.0 ms | 3.9 | 4 | 14,368 | 29.8 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.