Speed test matrix
Qwen3.5-0.8B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-0.8B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.90 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context94.2K87 ms TTFT
16K context108.6K149 ms TTFT
32K context98.3K328 ms TTFT
64K context76.6K841 ms TTFT
128K context51.9K2,479 ms TTFT
251K context31.6K7,984 ms TTFT
256K context30.9K8,334 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 381.337 ms TTFT | Looped | 1,460.588 ms TTFT | 2,567.2124 ms TTFT | 4,094.0193 ms TTFT | 5,841.8328 ms TTFT | 7,093.0537 ms TTFT | Looped |
| 16K | 389.448 ms TTFT | 702.975 ms TTFT | 1,358.499 ms TTFT | Looped | Looped | 5,139.1402 ms TTFT | 6,321.2724 ms TTFT | 7,983.81,353 ms TTFT |
| 32K | 334.371 ms TTFT | 674.296 ms TTFT | 1,299.2145 ms TTFT | 2,250.3197 ms TTFT | Looped | 4,445.3519 ms TTFT | 5,481.61,264 ms TTFT | 6,437.42,397 ms TTFT |
| 64K | 315.4124 ms TTFT | 599.4162 ms TTFT | 1,188.4199 ms TTFT | Looped | 2,934.4367 ms TTFT | Looped | Looped | Over capacity |
| 128K | Looped | 506.5304 ms TTFT | 1,064.5341 ms TTFT | 1,626.2451 ms TTFT | 2,183.3683 ms TTFT | Looped | Over capacity | Over capacity |
| 251K | 201.8540 ms TTFT | 405.8661 ms TTFT | 795.3706 ms TTFT | 1,189.5907 ms TTFT | 1,473.43,530 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 94.2K | — | 87 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 108.6K | — | 149 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 98.3K | — | 328 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 76.6K | — | 841 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 51.9K | — | 2,479 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 31.6K | — | 7,984 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 30.9K | — | 8,334 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 381.3 | 391.9 | 37 ms | 2.6 ms | 1.0 | 3 | 11,439 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 1,460.5 | 375.0 | 88 ms | 2.7 ms | 3.8 | 12 | 43,816 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,567.2 | 329.7 | 124 ms | 3.0 ms | 7.8 | 16 | 77,010 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,094.0 | 275.4 | 193 ms | 3.6 ms | 16.0 | 32 | 122,820 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,841.8 | 193.9 | 328 ms | 5.2 ms | 31.4 | 32 | 175,253 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 7,093.0 | 119.3 | 537 ms | 8.4 ms | 63.0 | 25 | 213,303 | 30.1 s |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 389.4 | 363.5 | 48 ms | 2.8 ms | 1.0 | 3 | 11,681 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 702.9 | 367.1 | 75 ms | 2.7 ms | 2.0 | 6 | 21,085 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,358.4 | 342.6 | 99 ms | 2.9 ms | 3.9 | 9 | 40,753 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 5,139.1 | 174.3 | 402 ms | 5.7 ms | 31.5 | 32 | 154,172 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 6,321.2 | 100.7 | 724 ms | 9.9 ms | 63.9 | 2 | 190,919 | 30.2 s |
| Sustained decode | 16K | 128 | Valid | 7,983.8 | 63.5 | 1,353 ms | 15.7 ms | 128.0 | 0 | 242,583 | 30.4 s |
| Sustained decode | 32K | 1 | Valid | 334.3 | 350.0 | 71 ms | 2.9 ms | 1.0 | 2 | 10,030 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 674.2 | 349.9 | 96 ms | 2.9 ms | 2.0 | 4 | 20,282 | 30.1 s |
| Sustained decode | 32K | 4 | Valid | 1,299.2 | 330.9 | 145 ms | 3.0 ms | 3.9 | 8 | 38,976 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,250.3 | 297.3 | 197 ms | 3.4 ms | 7.9 | 16 | 67,509 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 4,445.3 | 152.4 | 519 ms | 6.6 ms | 31.6 | 32 | 133,447 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,481.6 | 87.4 | 1,264 ms | 11.4 ms | 63.9 | 2 | 164,447 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,437.4 | 51.9 | 2,397 ms | 19.3 ms | 128.0 | 0 | 195,417 | 30.4 s |
| Sustained decode | 64K | 1 | Valid | 315.4 | 329.6 | 124 ms | 3.0 ms | 1.0 | 2 | 9,462 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 599.4 | 307.3 | 162 ms | 3.3 ms | 2.0 | 4 | 17,981 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,188.4 | 304.9 | 199 ms | 3.3 ms | 3.9 | 8 | 35,652 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 2,934.4 | 194.4 | 367 ms | 5.1 ms | 15.9 | 16 | 88,033 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Valid | 506.5 | 273.3 | 304 ms | 3.7 ms | 1.9 | 4 | 15,203 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,064.5 | 298.3 | 341 ms | 3.4 ms | 4.0 | 8 | 31,933 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,626.2 | 207.9 | 451 ms | 4.8 ms | 7.9 | 8 | 48,788 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,183.3 | 150.6 | 683 ms | 6.6 ms | 15.8 | 15 | 65,499 | 30.0 s |
| Sustained decode | 128K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 201.8 | 206.4 | 540 ms | 4.8 ms | 1.0 | 1 | 6,053 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 405.8 | 204.0 | 661 ms | 4.9 ms | 1.9 | 2 | 12,175 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 795.3 | 199.9 | 706 ms | 5.0 ms | 4.0 | 4 | 23,859 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,189.5 | 155.4 | 907 ms | 6.4 ms | 7.8 | 8 | 35,685 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,473.4 | 94.3 | 3,530 ms | 10.6 ms | 16.0 | 0 | 44,200 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.