Speed test matrix
Qwen3.6-35B-A3B on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881710 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context38.6K212 ms TTFT
16K context37.4K434 ms TTFT
32K context38.6K836 ms TTFT
64K context36.4K1,770 ms TTFT
128K context28K4,596 ms TTFT
251K context19.5K12,903 ms TTFT
256K context19.2K13,406 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 205.196 ms TTFT | 349.7161 ms TTFT | 584.4927 ms TTFT | 908.2241 ms TTFT | 1,575.6389 ms TTFT | 2,427.0651 ms TTFT | 3,667.3904 ms TTFT | 5,302.81,504 ms TTFT |
| 16K | 202.1111 ms TTFT | 346.9143 ms TTFT | 576.9223 ms TTFT | 886.0295 ms TTFT | 1,498.3369 ms TTFT | 2,278.2512 ms TTFT | 3,345.0836 ms TTFT | 4,536.61,462 ms TTFT |
| 32K | 197.0150 ms TTFT | 337.8235 ms TTFT | 554.1298 ms TTFT | 841.4378 ms TTFT | 1,390.0559 ms TTFT | 2,013.6829 ms TTFT | 2,740.91,330 ms TTFT | Over capacity |
| 64K | 188.5173 ms TTFT | 324.3216 ms TTFT | 518.3217 ms TTFT | 765.2573 ms TTFT | 1,196.0932 ms TTFT | 1,612.01,658 ms TTFT | Over capacity | Over capacity |
| 128K | 171.7327 ms TTFT | 296.2429 ms TTFT | 460.4623 ms TTFT | 647.71,026 ms TTFT | 921.81,813 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 147.5612 ms TTFT | 254.2715 ms TTFT | 377.41,219 ms TTFT | 494.81,922 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 38.6K | — | 212 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 37.4K | — | 434 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 38.6K | — | 836 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 36.4K | — | 1,770 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28K | — | 4,596 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.5K | — | 12,903 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.2K | — | 13,406 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 205.1 | 206.1 | 96 ms | 4.9 ms | 1.0 | 1 | 6,154 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 349.7 | 176.2 | 161 ms | 5.7 ms | 2.0 | 2 | 10,490 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 584.4 | 147.9 | 927 ms | 6.8 ms | 4.0 | 4 | 17,536 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 908.2 | 113.9 | 241 ms | 8.8 ms | 8.0 | 0 | 27,256 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,575.6 | 99.3 | 389 ms | 10.1 ms | 16.0 | 0 | 47,264 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,427.0 | 76.8 | 651 ms | 13.0 ms | 32.0 | 0 | 72,800 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,667.3 | 57.9 | 904 ms | 17.3 ms | 64.0 | 0 | 110,021 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,302.8 | 41.4 | 1,504 ms | 24.1 ms | 128.0 | 0 | 159,104 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 202.1 | 203.0 | 111 ms | 4.9 ms | 1.0 | 1 | 6,061 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 346.9 | 174.7 | 143 ms | 5.7 ms | 2.0 | 2 | 10,407 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 576.9 | 145.9 | 223 ms | 6.9 ms | 4.0 | 4 | 17,305 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 886.0 | 111.2 | 295 ms | 9.0 ms | 8.0 | 0 | 26,576 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,498.3 | 94.3 | 369 ms | 10.6 ms | 16.0 | 0 | 44,960 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,278.2 | 71.6 | 512 ms | 14.0 ms | 32.0 | 0 | 68,320 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,345.0 | 52.3 | 836 ms | 19.1 ms | 64.0 | 0 | 100,352 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,536.6 | 35.3 | 1,462 ms | 28.3 ms | 128.0 | 0 | 136,064 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 197.0 | 198.4 | 150 ms | 5.0 ms | 1.0 | 1 | 5,910 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 337.8 | 170.8 | 235 ms | 5.9 ms | 2.0 | 2 | 10,134 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 554.1 | 140.5 | 298 ms | 7.1 ms | 4.0 | 4 | 16,622 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 841.4 | 105.2 | 378 ms | 9.5 ms | 8.0 | 0 | 25,240 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,390.0 | 86.9 | 559 ms | 11.5 ms | 16.0 | 0 | 41,696 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,013.6 | 62.8 | 829 ms | 15.9 ms | 32.0 | 0 | 60,416 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,740.9 | 42.5 | 1,330 ms | 23.6 ms | 64.0 | 0 | 82,240 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 188.5 | 190.0 | 173 ms | 5.3 ms | 1.0 | 1 | 5,656 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 324.3 | 163.6 | 216 ms | 6.1 ms | 2.0 | 2 | 9,728 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 518.3 | 130.9 | 217 ms | 7.6 ms | 4.0 | 4 | 15,547 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 765.2 | 95.7 | 573 ms | 10.4 ms | 8.0 | 0 | 22,952 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,196.0 | 74.8 | 932 ms | 13.4 ms | 16.0 | 0 | 35,872 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,612.0 | 50.5 | 1,658 ms | 19.8 ms | 32.0 | 0 | 48,352 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 171.7 | 174.0 | 327 ms | 5.7 ms | 1.0 | 1 | 5,153 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 296.2 | 150.4 | 429 ms | 6.6 ms | 1.9 | 2 | 8,887 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 460.4 | 115.1 | 623 ms | 8.7 ms | 4.0 | 0 | 13,812 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 647.7 | 80.7 | 1,026 ms | 12.4 ms | 8.0 | 0 | 19,424 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 921.8 | 57.0 | 1,813 ms | 17.6 ms | 16.0 | 0 | 27,648 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 147.5 | 151.7 | 612 ms | 6.6 ms | 1.0 | 1 | 4,424 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 254.2 | 130.7 | 715 ms | 7.7 ms | 2.0 | 2 | 7,625 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 377.4 | 94.3 | 1,219 ms | 10.6 ms | 4.0 | 0 | 11,320 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 494.8 | 61.8 | 1,922 ms | 16.2 ms | 8.0 | 0 | 14,840 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.