Speed test matrix
Qwen3.6-27B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/Qwen3.6-27B-NVFP4 --served-model-name qwen --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --quantization modelopt
Prefill
prompt tok/s by context length8K context7,0901,155 ms TTFT
16K context6,9122,347 ms TTFT
32K context6,3895,054 ms TTFT
64K context5,44011,843 ms TTFT
128K context4,21430,541 ms TTFT
251K context2,93885,857 ms TTFT
256K context2,90088,712 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 66.086 ms TTFT | 119.0126 ms TTFT | 228.7247 ms TTFT | 457.7409 ms TTFT | 796.6869 ms TTFT | 1,264.61,405 ms TTFT | 1,769.33,002 ms TTFT | 2,154.74,231 ms TTFT |
| 16K | 65.0143 ms TTFT | 117.5244 ms TTFT | 225.8412 ms TTFT | 444.9642 ms TTFT | 752.71,454 ms TTFT | 1,156.52,464 ms TTFT | 1,559.43,954 ms TTFT | Over capacity |
| 32K | 63.8235 ms TTFT | 115.1442 ms TTFT | 220.3683 ms TTFT | 422.01,478 ms TTFT | 675.82,491 ms TTFT | 986.74,226 ms TTFT | Over capacity | Over capacity |
| 64K | 61.2153 ms TTFT | 111.1208 ms TTFT | 210.7452 ms TTFT | 385.8429 ms TTFT | 570.5736 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 56.3291 ms TTFT | 103.2423 ms TTFT | 192.7634 ms TTFT | 326.41,091 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 48.51,316 ms TTFT | 90.41,805 ms TTFT | 165.13,152 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,090.0 | — | 1,155 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,912.0 | — | 2,347 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,389.0 | — | 5,054 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,440.0 | — | 11,843 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,214.0 | — | 30,541 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,938.0 | — | 85,857 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,900.0 | — | 88,712 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 66.0 | 66.0 | 86 ms | 15.1 ms | 1.0 | 0 | 1,981 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 119.0 | 59.5 | 126 ms | 16.8 ms | 2.0 | 0 | 3,570 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 228.7 | 57.2 | 247 ms | 17.5 ms | 4.0 | 0 | 6,860 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 457.7 | 57.2 | 409 ms | 17.5 ms | 8.0 | 0 | 13,728 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 796.6 | 49.9 | 869 ms | 20.0 ms | 16.0 | 0 | 23,888 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,264.6 | 39.3 | 1,405 ms | 25.4 ms | 32.0 | 0 | 37,920 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,769.3 | 27.4 | 3,002 ms | 36.4 ms | 64.0 | 0 | 53,056 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,154.7 | 15.8 | 4,231 ms | 63.2 ms | 128.0 | 0 | 64,640 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 65.0 | 65.0 | 143 ms | 15.4 ms | 1.0 | 0 | 1,950 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 117.5 | 58.7 | 244 ms | 17.0 ms | 2.0 | 0 | 3,524 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 225.8 | 56.5 | 412 ms | 17.7 ms | 4.0 | 0 | 6,772 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 444.9 | 55.5 | 642 ms | 18.0 ms | 8.0 | 0 | 13,344 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 752.7 | 47.1 | 1,454 ms | 21.2 ms | 16.0 | 0 | 22,576 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,156.5 | 35.9 | 2,464 ms | 27.9 ms | 32.0 | 0 | 34,688 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,559.4 | 23.3 | 3,954 ms | 42.8 ms | 64.0 | 0 | 46,784 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 63.8 | 63.8 | 235 ms | 15.7 ms | 1.0 | 0 | 1,913 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 115.1 | 57.6 | 442 ms | 17.4 ms | 2.0 | 0 | 3,452 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 220.3 | 55.0 | 683 ms | 18.2 ms | 4.0 | 0 | 6,608 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 422.0 | 52.8 | 1,478 ms | 18.9 ms | 8.0 | 0 | 12,656 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 675.8 | 41.9 | 2,491 ms | 23.9 ms | 16.0 | 0 | 20,272 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 986.7 | 29.7 | 4,226 ms | 33.6 ms | 32.0 | 0 | 29,600 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 61.2 | 61.2 | 153 ms | 16.3 ms | 1.0 | 0 | 1,836 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 111.1 | 55.6 | 208 ms | 18.0 ms | 2.0 | 0 | 3,332 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 210.7 | 52.7 | 452 ms | 19.0 ms | 4.0 | 0 | 6,321 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 385.8 | 48.1 | 429 ms | 20.8 ms | 8.0 | 0 | 11,568 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 570.5 | 35.5 | 736 ms | 28.2 ms | 16.0 | 0 | 17,104 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 56.3 | 56.3 | 291 ms | 17.7 ms | 1.0 | 0 | 1,689 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 103.2 | 51.6 | 423 ms | 19.4 ms | 2.0 | 0 | 3,096 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 192.7 | 48.1 | 634 ms | 20.8 ms | 4.0 | 0 | 5,780 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 326.4 | 40.6 | 1,091 ms | 24.6 ms | 8.0 | 0 | 9,784 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 48.5 | 48.5 | 1,316 ms | 20.6 ms | 1.0 | 0 | 1,454 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 90.4 | 44.7 | 1,805 ms | 22.4 ms | 2.0 | 0 | 2,710 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 165.1 | 40.4 | 3,152 ms | 24.8 ms | 4.0 | 0 | 4,952 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.