Speed test matrix
Qwen3.5-9B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-9B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.95 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context21.4K383 ms TTFT
16K context21.3K762 ms TTFT
32K context19.8K1,631 ms TTFT
64K context17.5K3,678 ms TTFT
128K context14.1K9,149 ms TTFT
251K context10.2K24,742 ms TTFT
256K context10.1K25,577 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 83.145 ms TTFT | 155.265 ms TTFT | 309.281 ms TTFT | 619.1136 ms TTFT | 1,121.2187 ms TTFT | 1,889.5356 ms TTFT | 2,962.1654 ms TTFT | 4,422.01,165 ms TTFT |
| 16K | 81.861 ms TTFT | 153.3104 ms TTFT | 304.6104 ms TTFT | 605.2179 ms TTFT | 1,072.3304 ms TTFT | 1,760.9496 ms TTFT | 2,635.4909 ms TTFT | Over capacity |
| 32K | 79.672 ms TTFT | 149.8119 ms TTFT | 297.3147 ms TTFT | 582.4207 ms TTFT | 995.9351 ms TTFT | Looped | Over capacity | Over capacity |
| 64K | 75.1123 ms TTFT | 141.8190 ms TTFT | 281.1240 ms TTFT | 531.8389 ms TTFT | 861.4718 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 67.4287 ms TTFT | 127.0374 ms TTFT | 251.6538 ms TTFT | 453.0895 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 56.2565 ms TTFT | 97.6824 ms TTFT | 193.01,217 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 21.4K | — | 383 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.3K | — | 762 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.8K | — | 1,631 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.5K | — | 3,678 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.1K | — | 9,149 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.2K | — | 24,742 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.1K | — | 25,577 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 83.1 | 83.1 | 45 ms | 12.0 ms | 1.0 | 0 | 2,493 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 155.2 | 77.7 | 65 ms | 12.9 ms | 2.0 | 0 | 4,657 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 309.2 | 77.4 | 81 ms | 12.9 ms | 4.0 | 0 | 9,272 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 619.1 | 77.5 | 136 ms | 12.9 ms | 8.0 | 0 | 18,568 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,121.2 | 70.2 | 187 ms | 14.2 ms | 16.0 | 0 | 33,632 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,889.5 | 59.2 | 356 ms | 16.9 ms | 32.0 | 0 | 56,672 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,962.1 | 46.2 | 654 ms | 21.7 ms | 64.0 | 0 | 88,864 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,422.0 | 34.1 | 1,165 ms | 29.3 ms | 128.0 | 0 | 132,608 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 81.8 | 81.8 | 61 ms | 12.2 ms | 1.0 | 0 | 2,452 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 153.3 | 76.7 | 104 ms | 13.0 ms | 2.0 | 0 | 4,598 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 304.6 | 76.2 | 104 ms | 13.1 ms | 4.0 | 0 | 9,136 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 605.2 | 75.7 | 179 ms | 13.2 ms | 8.0 | 0 | 18,152 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,072.3 | 67.0 | 304 ms | 14.9 ms | 16.0 | 0 | 32,161 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,760.9 | 54.9 | 496 ms | 18.2 ms | 32.0 | 0 | 52,833 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,635.4 | 40.7 | 909 ms | 24.6 ms | 64.0 | 0 | 79,039 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 79.6 | 79.7 | 72 ms | 12.6 ms | 1.0 | 0 | 2,389 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 149.8 | 75.0 | 119 ms | 13.3 ms | 2.0 | 0 | 4,494 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 297.3 | 74.4 | 147 ms | 13.4 ms | 4.0 | 0 | 8,916 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 582.4 | 72.8 | 207 ms | 13.7 ms | 8.0 | 0 | 17,472 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 995.9 | 62.3 | 351 ms | 16.0 ms | 16.0 | 0 | 29,872 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 75.1 | 75.2 | 123 ms | 13.3 ms | 1.0 | 0 | 2,253 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 141.8 | 71.0 | 190 ms | 14.1 ms | 2.0 | 0 | 4,254 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 281.1 | 70.3 | 240 ms | 14.2 ms | 4.0 | 0 | 8,432 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 531.8 | 66.5 | 389 ms | 15.0 ms | 8.0 | 0 | 15,952 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 861.4 | 53.9 | 718 ms | 18.5 ms | 16.0 | 0 | 25,840 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 67.4 | 67.5 | 287 ms | 14.8 ms | 1.0 | 0 | 2,021 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 127.0 | 63.5 | 374 ms | 15.7 ms | 2.0 | 0 | 3,810 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 251.6 | 62.8 | 538 ms | 15.9 ms | 4.0 | 0 | 7,548 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 453.0 | 56.4 | 895 ms | 17.7 ms | 8.0 | 0 | 13,584 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 56.2 | 56.3 | 565 ms | 17.8 ms | 1.0 | 0 | 1,685 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 97.6 | 48.8 | 824 ms | 20.5 ms | 2.0 | 0 | 2,926 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 193.0 | 48.2 | 1,217 ms | 20.7 ms | 4.0 | 0 | 5,788 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.