Speed test matrix
Qwen3.8-Flash-Next-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+g50a061f79 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g50a061f79
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
50a061f792f3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_MMAP=1 "$HOME/env_rtxpro6k-qwen-flash-next-vllm-pr-54129/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.79 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context13.1K627 ms TTFT
16K context13.8K1,169 ms TTFT
32K context14K2,291 ms TTFT
64K context14.4K4,461 ms TTFT
128K context14.2K9,043 ms TTFT
251K context13.8K18,246 ms TTFT
256K context12.9K19,906 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 119.6711 ms TTFT | 230.4364 ms TTFT | 400.0428 ms TTFT | 639.6707 ms TTFT | 1,026.61,661 ms TTFT | 1,516.31,848 ms TTFT | 2,462.12,999 ms TTFT | 3,501.25,223 ms TTFT |
| 16K | 118.11,253 ms TTFT | 225.7243 ms TTFT | 402.3559 ms TTFT | 635.8726 ms TTFT | 1,033.01,226 ms TTFT | 1,521.42,025 ms TTFT | 2,465.93,237 ms TTFT | 3,495.45,385 ms TTFT |
| 32K | 119.02,355 ms TTFT | Looped | 412.4530 ms TTFT | 645.6722 ms TTFT | 1,010.21,416 ms TTFT | 1,526.22,233 ms TTFT | 2,434.73,251 ms TTFT | 3,473.85,629 ms TTFT |
| 64K | 126.42,375 ms TTFT | 227.94,564 ms TTFT | 404.7653 ms TTFT | 633.3952 ms TTFT | 1,009.61,341 ms TTFT | 1,515.12,079 ms TTFT | 2,382.63,400 ms TTFT | 3,432.86,046 ms TTFT |
| 128K | 122.79,038 ms TTFT | Looped | 399.5805 ms TTFT | 640.81,179 ms TTFT | 1,015.81,990 ms TTFT | 1,498.93,686 ms TTFT | 2,308.86,887 ms TTFT | 3,339.513,877 ms TTFT |
| 251K | 124.59,528 ms TTFT | Looped | 390.11,563 ms TTFT | 611.52,387 ms TTFT | 978.34,322 ms TTFT | 1,470.18,024 ms TTFT | 2,278.015,291 ms TTFT | 3,244.530,765 ms TTFT |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.1K | — | 627 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.8K | — | 1,169 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14K | — | 2,291 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.4K | — | 4,461 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.2K | — | 9,043 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.8K | — | 18,246 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.9K | — | 19,906 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 119.6 | 121.0 | 711 ms | 8.3 ms | 1.0 | 0 | 3,588 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 230.4 | 114.7 | 364 ms | 8.7 ms | 2.0 | 0 | 6,909 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 400.0 | 99.7 | 428 ms | 10.0 ms | 4.0 | 0 | 11,995 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 639.6 | 75.8 | 707 ms | 13.2 ms | 8.0 | 0 | 19,184 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,026.6 | 63.5 | 1,661 ms | 15.7 ms | 16.0 | 0 | 30,798 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,516.3 | 47.4 | 1,848 ms | 21.1 ms | 32.0 | 0 | 45,471 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,462.1 | 36.6 | 2,999 ms | 27.3 ms | 64.0 | 0 | 73,833 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,501.2 | 25.1 | 5,223 ms | 39.9 ms | 128.0 | 0 | 104,994 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 118.1 | 118.5 | 1,253 ms | 8.4 ms | 1.0 | 0 | 3,543 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 225.7 | 113.3 | 243 ms | 8.8 ms | 2.0 | 0 | 6,772 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 402.3 | 100.3 | 559 ms | 10.0 ms | 4.0 | 0 | 12,067 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 635.8 | 80.0 | 726 ms | 12.5 ms | 8.0 | 0 | 19,063 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,033.0 | 64.3 | 1,226 ms | 15.6 ms | 16.0 | 0 | 30,988 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,521.4 | 47.2 | 2,025 ms | 21.2 ms | 32.0 | 0 | 45,622 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,465.9 | 37.4 | 3,237 ms | 26.7 ms | 64.0 | 0 | 73,962 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,495.4 | 25.4 | 5,385 ms | 39.3 ms | 128.0 | 0 | 104,683 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 119.0 | 120.0 | 2,355 ms | 8.3 ms | 1.0 | 0 | 3,571 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 412.4 | 102.7 | 530 ms | 9.7 ms | 4.0 | 0 | 12,370 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 645.6 | 79.5 | 722 ms | 12.6 ms | 8.0 | 0 | 19,358 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,010.2 | 63.6 | 1,416 ms | 15.7 ms | 16.0 | 0 | 30,282 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,526.2 | 47.4 | 2,233 ms | 21.1 ms | 32.0 | 0 | 45,778 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,434.7 | 37.0 | 3,251 ms | 27.1 ms | 64.0 | 0 | 73,025 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,473.8 | 25.3 | 5,629 ms | 39.5 ms | 128.0 | 0 | 104,124 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 126.4 | 140.0 | 2,375 ms | 7.2 ms | 1.0 | 1 | 3,792 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 227.9 | 113.8 | 4,564 ms | 8.8 ms | 2.0 | 0 | 6,834 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 404.7 | 100.2 | 653 ms | 10.0 ms | 4.0 | 0 | 12,139 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 633.3 | 79.0 | 952 ms | 12.7 ms | 8.0 | 0 | 18,991 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,009.6 | 62.5 | 1,341 ms | 16.0 ms | 16.0 | 0 | 30,283 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,515.1 | 46.6 | 2,079 ms | 21.5 ms | 32.0 | 0 | 45,446 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,382.6 | 35.6 | 3,400 ms | 28.1 ms | 64.0 | 0 | 71,398 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,432.8 | 24.5 | 6,046 ms | 40.7 ms | 128.0 | 0 | 102,864 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 122.7 | 123.4 | 9,038 ms | 8.1 ms | 1.0 | 1 | 3,634 | 29.6 s |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Valid | 399.5 | 99.9 | 805 ms | 10.0 ms | 4.0 | 0 | 11,980 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 640.8 | 79.3 | 1,179 ms | 12.6 ms | 8.0 | 0 | 19,220 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,015.8 | 62.2 | 1,990 ms | 16.1 ms | 16.0 | 0 | 30,471 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,498.9 | 44.5 | 3,686 ms | 22.5 ms | 32.0 | 0 | 44,934 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,308.8 | 33.0 | 6,887 ms | 30.3 ms | 64.0 | 0 | 69,161 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,339.5 | 20.8 | 13,877 ms | 48.1 ms | 128.0 | 0 | 100,076 | 30.0 s |
| Sustained decode | 251K | 1 | Valid | 124.5 | 140.7 | 9,528 ms | 7.2 ms | 1.0 | 1 | 3,733 | 30.0 s |
| Sustained decode | 251K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 4 | Valid | 390.1 | 95.7 | 1,563 ms | 10.5 ms | 4.0 | 0 | 11,702 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 611.5 | 74.1 | 2,387 ms | 13.5 ms | 8.0 | 0 | 18,335 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 978.3 | 56.8 | 4,322 ms | 17.6 ms | 16.0 | 0 | 29,349 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,470.1 | 43.5 | 8,024 ms | 23.0 ms | 32.0 | 0 | 44,084 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 2,278.0 | 32.8 | 15,291 ms | 30.4 ms | 64.0 | 0 | 68,299 | 30.0 s |
| Sustained decode | 251K | 128 | Valid | 3,244.5 | 23.2 | 30,765 ms | 43.1 ms | 128.0 | 0 | 97,233 | 30.0 s |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement.