Speed test matrix
Qwen3.6-35B-A3B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-backend flashinfer_cutlass
Prefill
prompt tok/s by context length8K context40.4K203 ms TTFT
16K context39.1K415 ms TTFT
32K context35.8K901 ms TTFT
64K context28K2,302 ms TTFT
128K context19.2K6,689 ms TTFT
251K context11.9K21,222 ms TTFT
256K context11.7K21,950 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 176.182 ms TTFT | 292.8147 ms TTFT | 549.8155 ms TTFT | 915.2238 ms TTFT | 1,488.3325 ms TTFT | 2,231.5543 ms TTFT | 3,251.6952 ms TTFT | 4,311.71,594 ms TTFT |
| 16K | 174.060 ms TTFT | 289.783 ms TTFT | 537.7141 ms TTFT | 896.7249 ms TTFT | 1,442.0357 ms TTFT | 2,117.8556 ms TTFT | 2,920.8845 ms TTFT | 3,757.01,432 ms TTFT |
| 32K | 164.8116 ms TTFT | 274.1195 ms TTFT | 507.1274 ms TTFT | 836.5395 ms TTFT | 1,289.5515 ms TTFT | 1,801.7778 ms TTFT | 2,395.01,449 ms TTFT | Over capacity |
| 64K | 151.2136 ms TTFT | 252.3163 ms TTFT | 461.7278 ms TTFT | 741.8441 ms TTFT | 1,080.0779 ms TTFT | 1,441.11,433 ms TTFT | Over capacity | Over capacity |
| 128K | 131.1344 ms TTFT | 198.6477 ms TTFT | 373.4628 ms TTFT | 607.3961 ms TTFT | 823.71,618 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 107.2694 ms TTFT | 152.7914 ms TTFT | 285.71,282 ms TTFT | 452.22,008 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.4K | — | 203 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 39.1K | — | 415 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 35.8K | — | 901 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28K | — | 2,302 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.2K | — | 6,689 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.9K | — | 21,222 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.7K | — | 21,950 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 176.1 | 176.7 | 82 ms | 5.7 ms | 1.0 | 1 | 5,281 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 292.8 | 147.4 | 147 ms | 6.8 ms | 2.0 | 2 | 8,784 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 549.8 | 139.4 | 155 ms | 7.2 ms | 4.0 | 4 | 16,492 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 915.2 | 115.2 | 238 ms | 8.7 ms | 8.0 | 0 | 27,456 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,488.3 | 93.7 | 325 ms | 10.7 ms | 16.0 | 0 | 44,641 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,231.5 | 70.3 | 543 ms | 14.2 ms | 32.0 | 0 | 66,944 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,251.6 | 50.9 | 952 ms | 19.7 ms | 64.0 | 0 | 97,536 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,311.7 | 33.2 | 1,594 ms | 30.1 ms | 128.0 | 0 | 129,280 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 174.0 | 174.7 | 60 ms | 5.7 ms | 1.0 | 1 | 5,218 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 289.7 | 145.6 | 83 ms | 6.9 ms | 2.0 | 2 | 8,690 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 537.7 | 135.3 | 141 ms | 7.4 ms | 4.0 | 4 | 16,132 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 896.7 | 112.6 | 249 ms | 8.9 ms | 8.0 | 0 | 26,896 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,442.0 | 90.6 | 357 ms | 11.0 ms | 16.0 | 0 | 43,248 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,117.8 | 66.6 | 556 ms | 15.0 ms | 32.0 | 0 | 63,520 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,920.8 | 45.6 | 845 ms | 22.0 ms | 64.0 | 0 | 87,616 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,757.0 | 28.9 | 1,432 ms | 34.6 ms | 128.0 | 0 | 112,768 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 164.8 | 165.5 | 116 ms | 6.0 ms | 1.0 | 1 | 4,944 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 274.1 | 138.6 | 195 ms | 7.2 ms | 1.9 | 2 | 8,220 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 507.1 | 128.0 | 274 ms | 7.8 ms | 4.0 | 4 | 15,212 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 836.5 | 104.9 | 395 ms | 9.5 ms | 8.0 | 0 | 25,088 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,289.5 | 80.7 | 515 ms | 12.4 ms | 16.0 | 0 | 38,672 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,801.7 | 56.0 | 778 ms | 17.8 ms | 32.0 | 0 | 54,048 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,395.0 | 36.8 | 1,449 ms | 27.2 ms | 64.0 | 0 | 71,808 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 151.2 | 152.0 | 136 ms | 6.6 ms | 1.0 | 1 | 4,535 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 252.3 | 126.9 | 163 ms | 7.9 ms | 2.0 | 2 | 7,570 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 461.7 | 115.6 | 278 ms | 8.7 ms | 4.0 | 0 | 13,848 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 741.8 | 92.6 | 441 ms | 10.8 ms | 8.0 | 0 | 22,248 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,080.0 | 67.1 | 779 ms | 14.9 ms | 16.0 | 0 | 32,400 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,441.1 | 44.4 | 1,433 ms | 22.5 ms | 32.0 | 0 | 43,232 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 131.1 | 133.4 | 344 ms | 7.5 ms | 1.0 | 1 | 3,933 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 198.6 | 99.2 | 477 ms | 10.1 ms | 2.0 | 0 | 5,958 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 373.4 | 93.0 | 628 ms | 10.8 ms | 4.0 | 0 | 11,200 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 607.3 | 75.0 | 961 ms | 13.3 ms | 8.0 | 0 | 18,216 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 823.7 | 49.8 | 1,618 ms | 20.1 ms | 16.0 | 0 | 24,704 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 107.2 | 107.3 | 694 ms | 9.3 ms | 1.0 | 0 | 3,215 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 152.7 | 76.2 | 914 ms | 13.1 ms | 2.0 | 0 | 4,582 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 285.7 | 70.9 | 1,282 ms | 14.1 ms | 4.0 | 0 | 8,572 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 452.2 | 55.3 | 2,008 ms | 18.1 ms | 8.0 | 0 | 13,560 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.