Speed test matrix
Qwen3.8-Flash-Next-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.1.dev5+ge77daef89 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.1.dev5+ge77daef89
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e77daef89e18
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-flash-next-vllm/bin/vllm" serve Qwen/Qwen3.8-Flash-Next-FP8 --served-model-name qwen --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.77 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --engram-config '{"cpu_offload":true}' --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":1}'Prefill
prompt tok/s by context length8K context14.6K563 ms TTFT
16K context15.1K1,072 ms TTFT
32K context15.2K2,111 ms TTFT
64K context15.2K4,214 ms TTFT
128K context14.9K8,567 ms TTFT
251K context14.4K17,466 ms TTFT
256K context14.3K17,894 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 147.6112 ms TTFT | 268.7129 ms TTFT | 484.1181 ms TTFT | 797.7649 ms TTFT | 1,277.0648 ms TTFT | 1,825.11,545 ms TTFT | 2,846.92,417 ms TTFT | 3,966.64,673 ms TTFT |
| 16K | 147.4125 ms TTFT | 267.7153 ms TTFT | 483.6178 ms TTFT | 786.7548 ms TTFT | 1,279.8886 ms TTFT | 1,827.31,268 ms TTFT | 2,853.42,485 ms TTFT | Over capacity |
| 32K | 145.3153 ms TTFT | 276.9199 ms TTFT | 489.4173 ms TTFT | 792.5495 ms TTFT | 1,262.0921 ms TTFT | 1,813.61,509 ms TTFT | 2,789.02,551 ms TTFT | Over capacity |
| 64K | 143.3207 ms TTFT | 269.5240 ms TTFT | 480.5295 ms TTFT | 791.5608 ms TTFT | 1,279.61,012 ms TTFT | 1,790.11,749 ms TTFT | Over capacity | Over capacity |
| 128K | 141.7355 ms TTFT | 260.3519 ms TTFT | 484.5649 ms TTFT | 801.21,112 ms TTFT | 1,256.21,958 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 147.9708 ms TTFT | 264.9978 ms TTFT | 482.21,181 ms TTFT | 780.62,311 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.6K | — | 563 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.1K | — | 1,072 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.2K | — | 2,111 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.2K | — | 4,214 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.9K | — | 8,567 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.4K | — | 17,466 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.3K | — | 17,894 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 147.6 | 147.9 | 112 ms | 6.8 ms | 1.0 | 1 | 4,428 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 268.7 | 134.6 | 129 ms | 7.4 ms | 2.0 | 2 | 8,058 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 484.1 | 120.6 | 181 ms | 8.3 ms | 4.0 | 4 | 14,523 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 797.7 | 99.8 | 649 ms | 10.0 ms | 8.0 | 0 | 23,928 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,277.0 | 79.5 | 648 ms | 12.6 ms | 16.0 | 0 | 38,294 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,825.1 | 56.4 | 1,545 ms | 17.7 ms | 32.0 | 0 | 54,745 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,846.9 | 42.7 | 2,417 ms | 23.4 ms | 64.0 | 0 | 85,378 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,966.6 | 28.4 | 4,673 ms | 35.3 ms | 128.0 | 0 | 118,949 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 147.4 | 147.7 | 125 ms | 6.8 ms | 1.0 | 1 | 4,420 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 267.7 | 135.2 | 153 ms | 7.4 ms | 2.0 | 2 | 8,029 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 483.6 | 121.1 | 178 ms | 8.3 ms | 4.0 | 3 | 14,505 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 786.7 | 97.9 | 548 ms | 10.2 ms | 8.0 | 0 | 23,591 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,279.8 | 79.7 | 886 ms | 12.6 ms | 16.0 | 0 | 38,378 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,827.3 | 56.4 | 1,268 ms | 17.7 ms | 32.0 | 0 | 54,812 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,853.4 | 43.0 | 2,485 ms | 23.2 ms | 64.0 | 0 | 85,523 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 145.3 | 149.0 | 153 ms | 6.7 ms | 1.0 | 1 | 4,357 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 276.9 | 139.3 | 199 ms | 7.2 ms | 2.0 | 2 | 8,304 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 489.4 | 123.0 | 173 ms | 8.1 ms | 4.0 | 4 | 14,668 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 792.5 | 98.8 | 495 ms | 10.1 ms | 8.0 | 0 | 23,767 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,262.0 | 78.5 | 921 ms | 12.7 ms | 16.0 | 0 | 37,860 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,813.6 | 55.9 | 1,509 ms | 17.9 ms | 32.0 | 0 | 54,406 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,789.0 | 42.0 | 2,551 ms | 23.8 ms | 64.0 | 0 | 83,660 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 143.3 | 146.3 | 207 ms | 6.8 ms | 1.0 | 1 | 4,298 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 269.5 | 135.7 | 240 ms | 7.4 ms | 2.0 | 2 | 8,084 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 480.5 | 119.5 | 295 ms | 8.4 ms | 4.0 | 2 | 14,410 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 791.5 | 97.8 | 608 ms | 10.2 ms | 8.0 | 0 | 23,741 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,279.6 | 79.1 | 1,012 ms | 12.6 ms | 16.0 | 0 | 38,373 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,790.1 | 54.5 | 1,749 ms | 18.3 ms | 32.0 | 0 | 53,689 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 141.7 | 148.2 | 355 ms | 6.8 ms | 1.0 | 1 | 4,252 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 260.3 | 133.0 | 519 ms | 7.5 ms | 2.0 | 2 | 7,809 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 484.5 | 122.0 | 649 ms | 8.2 ms | 4.0 | 2 | 14,483 | 29.9 s |
| Sustained decode | 128K | 8 | Valid | 801.2 | 99.0 | 1,112 ms | 10.1 ms | 8.0 | 0 | 24,025 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,256.2 | 77.2 | 1,958 ms | 13.0 ms | 16.0 | 0 | 37,673 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 147.9 | 151.0 | 708 ms | 6.6 ms | 1.0 | 1 | 4,436 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 264.9 | 138.0 | 978 ms | 7.2 ms | 2.0 | 2 | 7,946 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 482.2 | 122.3 | 1,181 ms | 8.2 ms | 4.0 | 3 | 14,464 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 780.6 | 95.1 | 2,311 ms | 10.5 ms | 8.0 | 0 | 23,417 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.