Speed test matrix
Qwen3.5-2B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-2B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.93 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context70.8K116 ms TTFT
16K context74.5K218 ms TTFT
32K context71.1K454 ms TTFT
64K context60.6K1,064 ms TTFT
128K context45.6K2,821 ms TTFT
251K context30.8K8,201 ms TTFT
256K context30.2K8,520 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 286.532 ms TTFT | Looped | 1,012.569 ms TTFT | 1,894.2103 ms TTFT | 3,370.8154 ms TTFT | Looped | Looped | Looped |
| 16K | 281.345 ms TTFT | 508.968 ms TTFT | Looped | 1,856.1145 ms TTFT | Looped | Looped | Looped | Looped |
| 32K | 270.869 ms TTFT | 494.2107 ms TTFT | 959.1126 ms TTFT | 1,774.3209 ms TTFT | 3,000.4311 ms TTFT | 4,580.1656 ms TTFT | 6,473.61,419 ms TTFT | 7,616.72,545 ms TTFT |
| 64K | 251.9123 ms TTFT | 464.7167 ms TTFT | 903.0246 ms TTFT | 1,618.6399 ms TTFT | 2,575.4653 ms TTFT | Looped | 4,820.72,644 ms TTFT | Over capacity |
| 128K | 220.4255 ms TTFT | Looped | 797.7404 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity |
| 251K | 176.4536 ms TTFT | 330.3696 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 70.8K | — | 116 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 74.5K | — | 218 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 71.1K | — | 454 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 60.6K | — | 1,064 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 45.6K | — | 2,821 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 30.8K | — | 8,201 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 30.2K | — | 8,520 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 286.5 | 287.3 | 32 ms | 3.5 ms | 1.0 | 2 | 8,595 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 1,012.5 | 254.5 | 69 ms | 3.9 ms | 4.0 | 8 | 30,376 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,894.2 | 239.0 | 103 ms | 4.2 ms | 8.0 | 16 | 56,817 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,370.8 | 213.6 | 154 ms | 4.7 ms | 16.0 | 16 | 101,126 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 281.3 | 282.2 | 45 ms | 3.5 ms | 1.0 | 2 | 8,439 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 508.9 | 255.7 | 68 ms | 3.9 ms | 2.0 | 4 | 15,266 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 1,856.1 | 233.9 | 145 ms | 4.3 ms | 8.0 | 8 | 55,680 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 270.8 | 272.0 | 69 ms | 3.7 ms | 1.0 | 2 | 8,124 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 494.2 | 248.9 | 107 ms | 4.0 ms | 1.9 | 4 | 14,825 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 959.1 | 242.2 | 126 ms | 4.1 ms | 3.9 | 8 | 28,772 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,774.3 | 224.0 | 209 ms | 4.5 ms | 7.9 | 8 | 53,227 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,000.4 | 190.5 | 311 ms | 5.3 ms | 15.8 | 16 | 90,010 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,580.1 | 148.9 | 656 ms | 6.7 ms | 31.3 | 32 | 137,417 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,473.6 | 102.3 | 1,419 ms | 9.8 ms | 64.0 | 0 | 194,210 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 7,616.7 | 61.0 | 2,545 ms | 16.4 ms | 128.0 | 0 | 228,502 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 251.9 | 254.0 | 123 ms | 3.9 ms | 1.0 | 2 | 7,557 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 464.7 | 233.9 | 167 ms | 4.3 ms | 2.0 | 2 | 13,970 | 30.1 s |
| Sustained decode | 64K | 4 | Valid | 903.0 | 228.0 | 246 ms | 4.4 ms | 4.0 | 4 | 27,167 | 30.1 s |
| Sustained decode | 64K | 8 | Valid | 1,618.6 | 205.2 | 399 ms | 4.9 ms | 7.8 | 8 | 48,556 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,575.4 | 165.3 | 653 ms | 6.1 ms | 15.7 | 16 | 77,607 | 30.1 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Valid | 4,820.7 | 76.2 | 2,644 ms | 13.1 ms | 64.0 | 0 | 145,536 | 30.2 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 220.4 | 222.6 | 255 ms | 4.5 ms | 1.0 | 1 | 6,612 | 30.0 s |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Valid | 797.7 | 202.1 | 404 ms | 4.9 ms | 4.0 | 4 | 23,931 | 30.0 s |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 176.4 | 180.1 | 536 ms | 5.6 ms | 1.0 | 1 | 5,292 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 330.3 | 168.8 | 696 ms | 5.9 ms | 2.0 | 2 | 9,906 | 30.0 s |
| Sustained decode | 251K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.