Speed test matrix
Qwen3.8-Flash-Next on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.1.dev1+g50a061f79 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g50a061f79
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
50a061f792f3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_MMAP=1 "$HOME/env_rtxpro6k-qwen-flash-next-vllm-pr-54129/bin/vllm" serve Qwen/Qwen3.8-Flash-Next --served-model-name qwen --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.78 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context11.9K686 ms TTFT
16K context12.3K1,311 ms TTFT
32K context12.5K2,569 ms TTFT
64K context12.5K5,125 ms TTFT
128K context12.4K10,363 ms TTFT
251K context12K20,907 ms TTFT
256K context11.9K21,521 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 112.3742 ms TTFT | 205.9634 ms TTFT | 351.9373 ms TTFT | 570.2567 ms TTFT | 923.81,421 ms TTFT | 1,479.41,610 ms TTFT | 2,062.82,585 ms TTFT | 3,349.54,871 ms TTFT |
| 16K | 112.21,355 ms TTFT | 197.7389 ms TTFT | 341.8554 ms TTFT | 559.1752 ms TTFT | 902.91,040 ms TTFT | 1,463.51,841 ms TTFT | 2,055.52,989 ms TTFT | 3,348.44,318 ms TTFT |
| 32K | 116.72,604 ms TTFT | Looped | 339.1619 ms TTFT | 556.4785 ms TTFT | 901.21,441 ms TTFT | 1,445.41,761 ms TTFT | 2,044.93,021 ms TTFT | Over capacity |
| 64K | 112.85,137 ms TTFT | 195.7431 ms TTFT | 344.2811 ms TTFT | 555.31,289 ms TTFT | 904.71,793 ms TTFT | 1,455.62,681 ms TTFT | Over capacity | Over capacity |
| 128K | 112.210,381 ms TTFT | 199.1982 ms TTFT | 350.3850 ms TTFT | 566.91,204 ms TTFT | 912.42,080 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 121.620,901 ms TTFT | 201.51,001 ms TTFT | 360.41,455 ms TTFT | 575.12,416 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.9K | — | 686 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.3K | — | 1,311 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.5K | — | 2,569 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.5K | — | 5,125 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.4K | — | 10,363 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12K | — | 20,907 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.9K | — | 21,521 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 112.3 | 112.5 | 742 ms | 8.9 ms | 1.0 | 0 | 3,369 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 205.9 | 101.5 | 634 ms | 9.9 ms | 2.0 | 0 | 6,176 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 351.9 | 88.1 | 373 ms | 11.4 ms | 4.0 | 0 | 10,555 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 570.2 | 67.3 | 567 ms | 14.9 ms | 8.0 | 0 | 17,099 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 923.8 | 56.8 | 1,421 ms | 17.6 ms | 16.0 | 0 | 27,694 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,479.4 | 46.2 | 1,610 ms | 21.6 ms | 32.0 | 0 | 44,365 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,062.8 | 31.4 | 2,585 ms | 31.8 ms | 64.0 | 0 | 61,844 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,349.5 | 24.3 | 4,871 ms | 41.1 ms | 128.0 | 0 | 100,378 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 112.2 | 112.0 | 1,355 ms | 8.9 ms | 1.0 | 0 | 3,366 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 197.7 | 98.3 | 389 ms | 10.2 ms | 2.0 | 0 | 5,931 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 341.8 | 85.0 | 554 ms | 11.8 ms | 4.0 | 0 | 10,254 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 559.1 | 70.3 | 752 ms | 14.2 ms | 8.0 | 0 | 16,774 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 902.9 | 56.3 | 1,040 ms | 17.7 ms | 16.0 | 0 | 27,074 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,463.5 | 45.7 | 1,841 ms | 21.9 ms | 32.0 | 0 | 43,880 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,055.5 | 32.0 | 2,989 ms | 31.2 ms | 64.0 | 0 | 61,631 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,348.4 | 25.4 | 4,318 ms | 39.3 ms | 128.0 | 0 | 100,352 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 116.7 | 116.4 | 2,604 ms | 8.6 ms | 1.0 | 0 | 3,499 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 339.1 | 84.7 | 619 ms | 11.8 ms | 4.0 | 0 | 10,173 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 556.4 | 69.9 | 785 ms | 14.3 ms | 8.0 | 0 | 16,690 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 901.2 | 56.0 | 1,441 ms | 17.8 ms | 16.0 | 0 | 27,012 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,445.4 | 45.0 | 1,761 ms | 22.2 ms | 32.0 | 0 | 43,325 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,044.9 | 31.5 | 3,021 ms | 31.7 ms | 64.0 | 0 | 61,266 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 112.8 | 113.4 | 5,137 ms | 8.8 ms | 1.0 | 0 | 3,384 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 195.7 | 97.7 | 431 ms | 10.2 ms | 2.0 | 0 | 5,872 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 344.2 | 85.7 | 811 ms | 11.7 ms | 4.0 | 0 | 10,322 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 555.3 | 69.3 | 1,289 ms | 14.4 ms | 8.0 | 0 | 16,654 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 904.7 | 56.3 | 1,793 ms | 17.8 ms | 16.0 | 0 | 27,137 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,455.6 | 44.3 | 2,681 ms | 22.6 ms | 32.0 | 0 | 43,669 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 112.2 | 112.3 | 10,381 ms | 8.9 ms | 1.0 | 0 | 3,364 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 199.1 | 98.2 | 982 ms | 10.2 ms | 2.0 | 0 | 5,973 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 350.3 | 86.2 | 850 ms | 11.6 ms | 4.0 | 0 | 10,507 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 566.9 | 70.0 | 1,204 ms | 14.3 ms | 8.0 | 0 | 16,997 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 912.4 | 56.0 | 2,080 ms | 17.9 ms | 16.0 | 0 | 27,362 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 121.6 | 121.5 | 20,901 ms | 8.2 ms | 1.0 | 1 | 3,630 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 201.5 | 101.1 | 1,001 ms | 9.9 ms | 2.0 | 0 | 6,041 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 360.4 | 89.5 | 1,455 ms | 11.2 ms | 4.0 | 0 | 10,810 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 575.1 | 70.8 | 2,416 ms | 14.1 ms | 8.0 | 0 | 17,241 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.