Speed test matrix
Qwen3.5-35B-A3B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-z-lab-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-35B-A3B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context27.2K301 ms TTFT
16K context28.8K564 ms TTFT
32K context26.7K1,210 ms TTFT
64K context21.6K2,984 ms TTFT
128K context16K8,053 ms TTFT
251K context10.4K24,183 ms TTFT
256K context10.3K24,879 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 173.8266 ms TTFT | 279.5337 ms TTFT | 436.0592 ms TTFT | 568.8Requests queued993 ms TTFT | 554.0Requests queued993 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 143.1410 ms TTFT | 243.4690 ms TTFT | 410.3502 ms TTFT | 517.5Requests queued728 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 110.11,241 ms TTFT | 198.11,369 ms TTFT | 334.7705 ms TTFT | 350.9Requests queued2,332 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 74.9364 ms TTFT | 141.3545 ms TTFT | 238.6914 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 45.08,109 ms TTFT | 85.98,386 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 23.424,293 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.2K | — | 301 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.8K | — | 564 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 26.7K | — | 1,210 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.6K | — | 2,984 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16K | — | 8,053 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.4K | — | 24,183 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.3K | — | 24,879 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 173.8 | 177.3 | 266 ms | 5.6 ms | 1.0 | 1 | 5,212 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 279.5 | 143.9 | 337 ms | 7.0 ms | 2.0 | 2 | 8,384 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 436.0 | 111.7 | 592 ms | 9.0 ms | 4.0 | 0 | 13,073 | 30.0 s |
| Sustained decode | 8K | 8 | Requests queued | 568.8 | 94.2 | 993 ms | 10.6 ms | 6.0 | 10 | 17,042 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 554.0 | 88.7 | 993 ms | 11.3 ms | 6.0 | 7 | 16,605 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 143.1 | 153.3 | 410 ms | 6.5 ms | 1.0 | 1 | 4,291 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 243.4 | 129.2 | 690 ms | 7.7 ms | 2.0 | 1 | 7,302 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 410.3 | 103.5 | 502 ms | 9.7 ms | 4.0 | 0 | 12,306 | 30.0 s |
| Sustained decode | 16K | 8 | Requests queued | 517.5 | 85.4 | 728 ms | 11.7 ms | 6.0 | 6 | 15,526 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 110.1 | 111.6 | 1,241 ms | 9.0 ms | 1.0 | 0 | 3,301 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 198.1 | 100.5 | 1,369 ms | 10.0 ms | 2.0 | 0 | 5,938 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 334.7 | 86.0 | 705 ms | 11.6 ms | 4.0 | 0 | 10,034 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 350.9 | 73.0 | 2,332 ms | 13.7 ms | 5.3 | 6 | 10,527 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 74.9 | 76.8 | 364 ms | 13.0 ms | 1.0 | 0 | 2,246 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 141.3 | 71.9 | 545 ms | 13.9 ms | 2.0 | 0 | 4,235 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 238.6 | 60.7 | 914 ms | 16.5 ms | 4.0 | 0 | 7,156 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 45.0 | 46.7 | 8,109 ms | 21.4 ms | 1.0 | 0 | 1,346 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 85.9 | 43.9 | 8,386 ms | 22.8 ms | 2.0 | 0 | 2,575 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 23.4 | 24.5 | 24,293 ms | 40.8 ms | 1.0 | 0 | 700 | 29.9 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.