Speed test matrix
Qwen3.8-Flash-Next-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+g50a061f79 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g50a061f79
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
50a061f792f3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLE_MMAP=1 "$HOME/env_rtxpro6k-qwen-flash-next-vllm-pr-54129/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.79 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --moe-backend triton
Prefill
prompt tok/s by context length8K context13.1K627 ms TTFT
16K context14.3K1,133 ms TTFT
32K context14.7K2,185 ms TTFT
64K context14.8K4,329 ms TTFT
128K context14.7K8,708 ms TTFT
251K context14.3K17,488 ms TTFT
256K context13.6K18,850 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 105.71,533 ms TTFT | 177.71,325 ms TTFT | 311.2366 ms TTFT | 526.8384 ms TTFT | 842.2571 ms TTFT | 1,332.61,172 ms TTFT | 1,988.22,573 ms TTFT | 3,120.63,170 ms TTFT |
| 16K | 105.51,152 ms TTFT | 176.8235 ms TTFT | 309.1426 ms TTFT | 536.4571 ms TTFT | 854.3794 ms TTFT | 1,352.51,316 ms TTFT | 1,968.02,436 ms TTFT | 3,093.03,467 ms TTFT |
| 32K | 105.52,714 ms TTFT | 177.7186 ms TTFT | 313.8587 ms TTFT | 532.2366 ms TTFT | 843.3566 ms TTFT | 1,340.8825 ms TTFT | 1,958.71,489 ms TTFT | 3,052.02,529 ms TTFT |
| 64K | 105.14,411 ms TTFT | 175.6332 ms TTFT | 312.4592 ms TTFT | 525.2808 ms TTFT | 824.91,199 ms TTFT | 1,305.91,830 ms TTFT | 1,927.93,051 ms TTFT | 3,003.25,621 ms TTFT |
| 128K | 104.28,742 ms TTFT | 176.9523 ms TTFT | 308.9637 ms TTFT | 511.91,261 ms TTFT | 807.31,987 ms TTFT | 1,251.83,569 ms TTFT | 1,842.66,299 ms TTFT | 2,867.412,731 ms TTFT |
| 251K | 104.517,585 ms TTFT | 173.6869 ms TTFT | 303.51,277 ms TTFT | 516.12,061 ms TTFT | 803.83,837 ms TTFT | 1,245.17,002 ms TTFT | 1,800.212,649 ms TTFT | 2,729.624,706 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.1K | — | 627 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.3K | — | 1,133 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.7K | — | 2,185 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.8K | — | 4,329 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.7K | — | 8,708 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.3K | — | 17,488 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.6K | — | 18,850 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 105.7 | 105.8 | 1,533 ms | 9.5 ms | 1.0 | 0 | 3,172 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 177.7 | 85.8 | 1,325 ms | 11.7 ms | 2.0 | 0 | 5,330 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 311.2 | 77.7 | 366 ms | 12.9 ms | 4.0 | 0 | 9,332 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 526.8 | 66.0 | 384 ms | 15.2 ms | 8.0 | 0 | 15,800 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 842.2 | 51.6 | 571 ms | 19.4 ms | 16.0 | 0 | 25,264 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,332.6 | 42.1 | 1,172 ms | 23.8 ms | 32.0 | 0 | 39,968 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,988.2 | 31.3 | 2,573 ms | 32.0 ms | 64.0 | 0 | 59,648 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,120.6 | 24.4 | 3,170 ms | 41.0 ms | 128.0 | 0 | 93,568 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 105.5 | 105.5 | 1,152 ms | 9.5 ms | 1.0 | 0 | 3,165 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 176.8 | 88.3 | 235 ms | 11.3 ms | 2.0 | 0 | 5,304 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 309.1 | 77.6 | 426 ms | 12.9 ms | 4.0 | 0 | 9,272 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 536.4 | 67.1 | 571 ms | 14.9 ms | 8.0 | 0 | 16,088 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 854.3 | 53.1 | 794 ms | 18.8 ms | 16.0 | 0 | 25,616 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,352.5 | 41.6 | 1,316 ms | 24.1 ms | 32.0 | 0 | 40,576 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,968.0 | 31.0 | 2,436 ms | 32.3 ms | 64.0 | 0 | 59,008 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,093.0 | 23.8 | 3,467 ms | 42.0 ms | 128.0 | 0 | 92,800 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 105.5 | 105.5 | 2,714 ms | 9.5 ms | 1.0 | 0 | 3,164 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 177.7 | 88.9 | 186 ms | 11.2 ms | 2.0 | 0 | 5,328 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 313.8 | 78.3 | 587 ms | 12.8 ms | 4.0 | 0 | 9,412 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 532.2 | 65.5 | 366 ms | 15.3 ms | 8.0 | 0 | 15,960 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 843.3 | 52.8 | 566 ms | 18.9 ms | 16.0 | 0 | 25,296 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,340.8 | 41.7 | 825 ms | 24.0 ms | 32.0 | 0 | 40,224 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,958.7 | 30.2 | 1,489 ms | 33.1 ms | 64.0 | 0 | 58,752 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,052.0 | 23.3 | 2,529 ms | 43.0 ms | 128.0 | 0 | 91,520 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 105.1 | 105.2 | 4,411 ms | 9.5 ms | 1.0 | 0 | 3,153 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 175.6 | 87.9 | 332 ms | 11.4 ms | 2.0 | 0 | 5,268 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 312.4 | 77.9 | 592 ms | 12.8 ms | 4.0 | 0 | 9,368 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 525.2 | 65.2 | 808 ms | 15.3 ms | 8.0 | 0 | 15,752 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 824.9 | 51.2 | 1,199 ms | 19.5 ms | 16.0 | 0 | 24,736 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,305.9 | 40.1 | 1,830 ms | 24.9 ms | 32.0 | 0 | 39,168 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,927.9 | 28.9 | 3,051 ms | 34.6 ms | 64.0 | 0 | 57,792 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,003.2 | 21.6 | 5,621 ms | 46.3 ms | 128.0 | 0 | 90,112 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 104.2 | 104.5 | 8,742 ms | 9.6 ms | 1.0 | 0 | 3,125 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 176.9 | 88.1 | 523 ms | 11.3 ms | 2.0 | 0 | 5,306 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 308.9 | 76.7 | 637 ms | 13.0 ms | 4.0 | 0 | 9,264 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 511.9 | 61.8 | 1,261 ms | 16.2 ms | 8.0 | 0 | 15,352 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 807.3 | 47.8 | 1,987 ms | 20.9 ms | 16.0 | 0 | 24,208 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,251.8 | 36.9 | 3,569 ms | 27.1 ms | 32.0 | 0 | 37,536 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,842.6 | 27.6 | 6,299 ms | 36.2 ms | 64.0 | 0 | 55,232 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 2,867.4 | 21.3 | 12,731 ms | 46.9 ms | 128.0 | 0 | 86,014 | 30.0 s |
| Sustained decode | 251K | 1 | Valid | 104.5 | 104.7 | 17,585 ms | 9.6 ms | 1.0 | 0 | 3,136 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 173.6 | 87.0 | 869 ms | 11.5 ms | 2.0 | 0 | 5,206 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 303.5 | 75.7 | 1,277 ms | 13.2 ms | 4.0 | 0 | 9,104 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 516.1 | 63.9 | 2,061 ms | 15.6 ms | 8.0 | 0 | 15,480 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 803.8 | 49.3 | 3,837 ms | 20.3 ms | 16.0 | 0 | 24,112 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,245.1 | 38.0 | 7,002 ms | 26.3 ms | 32.0 | 0 | 37,344 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 1,800.2 | 27.7 | 12,649 ms | 36.1 ms | 64.0 | 0 | 53,952 | 30.0 s |
| Sustained decode | 251K | 128 | Valid | 2,729.6 | 20.8 | 24,706 ms | 48.0 ms | 128.0 | 0 | 81,792 | 30.0 s |