Speed test matrix
Qwen3.8-Flash-Next-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.1.dev5+ge77daef89 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":true}' --moe-backend tritonPrefill
prompt tok/s by context length8K context15.2K540 ms TTFT
16K context15.7K1,029 ms TTFT
32K context15.8K2,029 ms TTFT
64K context15.9K4,039 ms TTFT
128K context15.7K8,139 ms TTFT
251K context15.1K16,617 ms TTFT
256K context15K17,036 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 113.472 ms TTFT | 193.6253 ms TTFT | 354.0190 ms TTFT | 622.4288 ms TTFT | 1,016.4345 ms TTFT | 1,635.1825 ms TTFT | 2,324.5990 ms TTFT | 3,535.51,970 ms TTFT |
| 16K | 113.098 ms TTFT | 195.1156 ms TTFT | 354.0238 ms TTFT | 622.5277 ms TTFT | 1,013.4706 ms TTFT | 1,638.1770 ms TTFT | 2,306.81,510 ms TTFT | 3,510.42,728 ms TTFT |
| 32K | 113.079 ms TTFT | 194.5144 ms TTFT | 352.7156 ms TTFT | 613.3242 ms TTFT | 992.6362 ms TTFT | 1,627.3716 ms TTFT | 2,290.11,295 ms TTFT | Over capacity |
| 64K | 112.6186 ms TTFT | 193.4283 ms TTFT | 352.1339 ms TTFT | 617.5529 ms TTFT | 992.0886 ms TTFT | 1,605.41,461 ms TTFT | Over capacity | Over capacity |
| 128K | 111.7285 ms TTFT | 190.8394 ms TTFT | 345.1545 ms TTFT | 607.0905 ms TTFT | 984.91,568 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 111.1654 ms TTFT | 190.4822 ms TTFT | 343.31,235 ms TTFT | 597.02,093 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.2K | — | 540 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.7K | — | 1,029 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.8K | — | 2,029 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.9K | — | 4,039 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.7K | — | 8,139 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 15.1K | — | 16,617 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15K | — | 17,036 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 113.4 | 113.4 | 72 ms | 8.8 ms | 1.0 | 0 | 3,402 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 193.6 | 96.4 | 253 ms | 10.4 ms | 2.0 | 0 | 5,808 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 354.0 | 88.5 | 190 ms | 11.3 ms | 4.0 | 0 | 10,620 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 622.4 | 78.0 | 288 ms | 12.8 ms | 8.0 | 0 | 18,672 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,016.4 | 63.4 | 345 ms | 15.8 ms | 16.0 | 0 | 30,480 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,635.1 | 51.3 | 825 ms | 19.5 ms | 32.0 | 0 | 49,056 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,324.5 | 36.0 | 990 ms | 27.7 ms | 64.0 | 0 | 69,696 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,535.5 | 27.1 | 1,970 ms | 36.9 ms | 128.0 | 0 | 105,984 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 113.0 | 113.0 | 98 ms | 8.9 ms | 1.0 | 0 | 3,390 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 195.1 | 97.6 | 156 ms | 10.2 ms | 2.0 | 0 | 5,854 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 354.0 | 88.7 | 238 ms | 11.3 ms | 4.0 | 0 | 10,620 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 622.5 | 77.6 | 277 ms | 12.9 ms | 8.0 | 0 | 18,672 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,013.4 | 63.4 | 706 ms | 15.8 ms | 16.0 | 0 | 30,400 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,638.1 | 50.9 | 770 ms | 19.7 ms | 32.0 | 0 | 49,120 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,306.8 | 35.6 | 1,510 ms | 28.1 ms | 64.0 | 0 | 69,184 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,510.4 | 26.6 | 2,728 ms | 37.6 ms | 128.0 | 0 | 105,216 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 113.0 | 112.9 | 79 ms | 8.9 ms | 1.0 | 0 | 3,389 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 194.5 | 97.2 | 144 ms | 10.3 ms | 2.0 | 0 | 5,834 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 352.7 | 88.2 | 156 ms | 11.3 ms | 4.0 | 0 | 10,580 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 613.3 | 76.5 | 242 ms | 13.1 ms | 8.0 | 0 | 18,392 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 992.6 | 61.9 | 362 ms | 16.2 ms | 16.0 | 0 | 29,776 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,627.3 | 50.5 | 716 ms | 19.8 ms | 32.0 | 0 | 48,800 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,290.1 | 35.6 | 1,295 ms | 28.1 ms | 64.0 | 0 | 68,672 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 112.6 | 112.7 | 186 ms | 8.9 ms | 1.0 | 0 | 3,379 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 193.4 | 96.6 | 283 ms | 10.3 ms | 2.0 | 0 | 5,800 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 352.1 | 87.7 | 339 ms | 11.4 ms | 4.0 | 0 | 10,564 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 617.5 | 76.6 | 529 ms | 13.1 ms | 8.0 | 0 | 18,520 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 992.0 | 61.4 | 886 ms | 16.3 ms | 16.0 | 0 | 29,760 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,605.4 | 49.1 | 1,461 ms | 20.4 ms | 32.0 | 0 | 48,160 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 111.7 | 111.8 | 285 ms | 8.9 ms | 1.0 | 0 | 3,350 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 190.8 | 95.4 | 394 ms | 10.5 ms | 2.0 | 0 | 5,724 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 345.1 | 86.1 | 545 ms | 11.6 ms | 4.0 | 0 | 10,352 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 607.0 | 75.4 | 905 ms | 13.3 ms | 8.0 | 0 | 18,208 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 984.9 | 61.0 | 1,568 ms | 16.4 ms | 16.0 | 0 | 29,536 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 111.1 | 111.4 | 654 ms | 9.0 ms | 1.0 | 0 | 3,334 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 190.4 | 95.2 | 822 ms | 10.5 ms | 2.0 | 0 | 5,710 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 343.3 | 85.8 | 1,235 ms | 11.7 ms | 4.0 | 0 | 10,296 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 597.0 | 74.5 | 2,093 ms | 13.4 ms | 8.0 | 0 | 17,904 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.