Speed test matrix
Qwen3.6-35B-A3B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-z-lab-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context27.3K300 ms TTFT
16K context28.3K573 ms TTFT
32K context26.8K1,207 ms TTFT
64K context21.6K2,984 ms TTFT
128K context16K8,044 ms TTFT
251K context10.4K24,180 ms TTFT
256K context10.3K24,882 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 141.7263 ms TTFT | 242.5454 ms TTFT | 416.1595 ms TTFT | 503.1Requests queued906 ms TTFT | 511.3Requests queued917 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 116.1607 ms TTFT | 211.3689 ms TTFT | 352.0501 ms TTFT | 461.9Requests queued836 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 91.61,239 ms TTFT | 173.31,370 ms TTFT | 292.5706 ms TTFT | 321.6Requests queued3,586 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 62.0379 ms TTFT | 122.1528 ms TTFT | 225.7923 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 43.48,098 ms TTFT | 79.18,383 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 26.324,338 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.3K | — | 300 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.3K | — | 573 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 26.8K | — | 1,207 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.6K | — | 2,984 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16K | — | 8,044 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.4K | — | 24,180 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.3K | — | 24,882 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 141.7 | 159.4 | 263 ms | 6.3 ms | 1.0 | 1 | 4,251 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 242.5 | 132.6 | 454 ms | 7.5 ms | 2.0 | 1 | 7,270 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 416.1 | 106.6 | 595 ms | 9.4 ms | 4.0 | 0 | 12,483 | 30.0 s |
| Sustained decode | 8K | 8 | Requests queued | 503.1 | 84.3 | 906 ms | 11.9 ms | 6.0 | 6 | 15,088 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 511.3 | 83.1 | 917 ms | 12.0 ms | 6.0 | 6 | 15,335 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 116.1 | 120.4 | 607 ms | 8.3 ms | 1.0 | 0 | 3,481 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 211.3 | 110.3 | 689 ms | 9.1 ms | 2.0 | 0 | 6,335 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 352.0 | 90.0 | 501 ms | 11.1 ms | 4.0 | 0 | 10,549 | 30.0 s |
| Sustained decode | 16K | 8 | Requests queued | 461.9 | 75.2 | 836 ms | 13.3 ms | 6.0 | 6 | 13,856 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 91.6 | 94.6 | 1,239 ms | 10.6 ms | 1.0 | 0 | 2,746 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 173.3 | 88.6 | 1,370 ms | 11.3 ms | 2.0 | 0 | 5,196 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 292.5 | 75.8 | 706 ms | 13.2 ms | 4.0 | 0 | 8,769 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 321.6 | 64.5 | 3,586 ms | 15.5 ms | 5.3 | 6 | 9,644 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 62.0 | 63.4 | 379 ms | 15.8 ms | 1.0 | 0 | 1,857 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 122.1 | 63.2 | 528 ms | 15.8 ms | 2.0 | 0 | 3,661 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 225.7 | 57.8 | 923 ms | 17.3 ms | 4.0 | 0 | 6,767 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 43.4 | 44.5 | 8,098 ms | 22.5 ms | 1.0 | 0 | 1,302 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 79.1 | 41.3 | 8,383 ms | 24.3 ms | 2.0 | 0 | 2,368 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 26.3 | 27.7 | 24,338 ms | 36.1 ms | 1.0 | 0 | 790 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.