Speed test matrix
Qwen3.8-Flash-Next on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.1.1.dev5+ge77daef89 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.80 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":true}' --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context13.4K612 ms TTFT
16K context13.9K1,162 ms TTFT
32K context14.1K2,274 ms TTFT
64K context14.1K4,536 ms TTFT
128K context13.9K9,194 ms TTFT
251K context13.4K18,687 ms TTFT
256K context13.4K19,163 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 132.697 ms TTFT | 241.2121 ms TTFT | 423.9264 ms TTFT | 718.6431 ms TTFT | 1,166.4548 ms TTFT | 1,786.01,233 ms TTFT | 2,397.91,843 ms TTFT | 3,887.92,957 ms TTFT |
| 16K | 135.0107 ms TTFT | 247.9129 ms TTFT | 427.7270 ms TTFT | 725.6360 ms TTFT | 1,157.6883 ms TTFT | 1,799.51,086 ms TTFT | 2,375.91,880 ms TTFT | 3,894.03,047 ms TTFT |
| 32K | 130.8136 ms TTFT | 250.9150 ms TTFT | 424.6273 ms TTFT | 721.9497 ms TTFT | 1,144.5821 ms TTFT | 1,797.01,312 ms TTFT | 2,371.01,998 ms TTFT | Over capacity |
| 64K | 132.2193 ms TTFT | 239.4223 ms TTFT | 423.5345 ms TTFT | 725.1560 ms TTFT | 1,140.3935 ms TTFT | 1,768.91,637 ms TTFT | Over capacity | Over capacity |
| 128K | 134.8314 ms TTFT | 246.6341 ms TTFT | 433.0597 ms TTFT | 714.5944 ms TTFT | 1,119.21,641 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 128.3644 ms TTFT | 240.9665 ms TTFT | 429.71,334 ms TTFT | 700.22,097 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.4K | — | 612 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.9K | — | 1,162 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.1K | — | 2,274 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.1K | — | 4,536 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.9K | — | 9,194 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.4K | — | 18,687 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.4K | — | 19,163 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 132.6 | 136.5 | 97 ms | 7.3 ms | 1.0 | 1 | 3,977 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 241.2 | 122.0 | 121 ms | 8.2 ms | 2.0 | 2 | 7,217 | 29.9 s |
| Sustained decode | 8K | 4 | Valid | 423.9 | 106.0 | 264 ms | 9.4 ms | 4.0 | 0 | 12,714 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 718.6 | 89.8 | 431 ms | 11.1 ms | 8.0 | 0 | 21,547 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,166.4 | 72.6 | 548 ms | 13.8 ms | 16.0 | 0 | 34,991 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,786.0 | 55.6 | 1,233 ms | 18.0 ms | 32.0 | 0 | 53,539 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,397.9 | 36.7 | 1,843 ms | 27.2 ms | 64.0 | 0 | 71,855 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,887.9 | 28.8 | 2,957 ms | 34.7 ms | 128.0 | 0 | 116,649 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 135.0 | 137.7 | 107 ms | 7.3 ms | 1.0 | 1 | 4,048 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.9 | 123.9 | 129 ms | 8.1 ms | 2.0 | 2 | 7,438 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 427.7 | 106.7 | 270 ms | 9.4 ms | 4.0 | 0 | 12,831 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 725.6 | 90.4 | 360 ms | 11.1 ms | 8.0 | 0 | 21,760 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,157.6 | 72.3 | 883 ms | 13.8 ms | 16.0 | 0 | 34,723 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,799.5 | 55.9 | 1,086 ms | 17.9 ms | 32.0 | 0 | 53,960 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,375.9 | 36.6 | 1,880 ms | 27.3 ms | 64.0 | 0 | 71,227 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,894.0 | 29.1 | 3,047 ms | 34.4 ms | 128.0 | 0 | 116,706 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 130.8 | 134.0 | 136 ms | 7.5 ms | 1.0 | 1 | 3,925 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 250.9 | 126.3 | 150 ms | 7.9 ms | 2.0 | 2 | 7,527 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 424.6 | 105.8 | 273 ms | 9.5 ms | 4.0 | 0 | 12,732 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 721.9 | 89.6 | 497 ms | 11.2 ms | 8.0 | 0 | 21,647 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,144.5 | 71.5 | 821 ms | 14.0 ms | 16.0 | 0 | 34,312 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,797.0 | 55.8 | 1,312 ms | 17.9 ms | 32.0 | 0 | 53,872 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,371.0 | 36.5 | 1,998 ms | 27.4 ms | 64.0 | 0 | 71,068 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 132.2 | 135.8 | 193 ms | 7.4 ms | 1.0 | 1 | 3,967 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 239.4 | 120.6 | 223 ms | 8.3 ms | 2.0 | 2 | 7,163 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 423.5 | 106.2 | 345 ms | 9.4 ms | 4.0 | 0 | 12,701 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 725.1 | 90.0 | 560 ms | 11.1 ms | 8.0 | 0 | 21,750 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,140.3 | 70.2 | 935 ms | 14.2 ms | 16.0 | 0 | 34,196 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,768.9 | 54.1 | 1,637 ms | 18.5 ms | 32.0 | 0 | 53,037 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 134.8 | 135.7 | 314 ms | 7.4 ms | 1.0 | 1 | 4,042 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 246.6 | 128.2 | 341 ms | 7.8 ms | 2.0 | 2 | 7,395 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 433.0 | 109.3 | 597 ms | 9.2 ms | 4.0 | 0 | 12,984 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 714.5 | 89.4 | 944 ms | 11.2 ms | 8.0 | 0 | 21,430 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,119.2 | 69.7 | 1,641 ms | 14.3 ms | 16.0 | 0 | 33,566 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 128.3 | 134.2 | 644 ms | 7.5 ms | 1.0 | 1 | 3,848 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 240.9 | 127.4 | 665 ms | 7.9 ms | 2.0 | 1 | 7,224 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 429.7 | 106.7 | 1,334 ms | 9.4 ms | 4.0 | 0 | 12,888 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 700.2 | 87.3 | 2,097 ms | 11.5 ms | 8.0 | 0 | 21,000 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.