Speed test matrix
Qwen3.6-35B-A3B-int4-mixed-AutoRound on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4+BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-intel-vllm/bin/vllm" serve Intel/Qwen3.6-35B-A3B-int4-mixed-AutoRound --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880718 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context35.1K233 ms TTFT
16K context34.2K474 ms TTFT
32K context34.3K941 ms TTFT
64K context32.3K1,996 ms TTFT
128K context25.4K5,061 ms TTFT
251K context18.1K13,927 ms TTFT
256K context17.7K14,511 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 180.797 ms TTFT | 341.9160 ms TTFT | 653.0983 ms TTFT | 1,180.2270 ms TTFT | 1,908.9466 ms TTFT | 3,024.6766 ms TTFT | 4,439.21,002 ms TTFT | 5,966.81,786 ms TTFT |
| 16K | 178.8111 ms TTFT | 339.3142 ms TTFT | 642.4217 ms TTFT | 1,133.4275 ms TTFT | 1,841.4360 ms TTFT | 2,793.6560 ms TTFT | 3,941.6820 ms TTFT | 5,080.01,397 ms TTFT |
| 32K | 174.5162 ms TTFT | 330.4261 ms TTFT | 613.5310 ms TTFT | 1,054.8352 ms TTFT | 1,659.7542 ms TTFT | 2,414.8874 ms TTFT | 3,191.61,437 ms TTFT | 3,767.32,437 ms TTFT |
| 64K | 167.4180 ms TTFT | 317.2196 ms TTFT | 569.9213 ms TTFT | 949.2529 ms TTFT | 1,398.9888 ms TTFT | 1,870.51,602 ms TTFT | 2,230.32,741 ms TTFT | Over capacity |
| 128K | 154.5332 ms TTFT | 288.9423 ms TTFT | 491.9554 ms TTFT | 775.21,018 ms TTFT | 1,039.61,660 ms TTFT | 1,240.73,167 ms TTFT | Over capacity | Over capacity |
| 251K | 134.2620 ms TTFT | 249.4725 ms TTFT | 403.51,172 ms TTFT | 562.71,902 ms TTFT | 665.73,415 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 35.1K | — | 233 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.2K | — | 474 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 34.3K | — | 941 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 32.3K | — | 1,996 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 25.4K | — | 5,061 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.1K | — | 13,927 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.7K | — | 14,511 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 180.7 | 181.6 | 97 ms | 5.5 ms | 1.0 | 1 | 5,422 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 341.9 | 172.3 | 160 ms | 5.8 ms | 2.0 | 2 | 10,257 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 653.0 | 165.2 | 983 ms | 6.1 ms | 4.0 | 4 | 19,588 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,180.2 | 149.0 | 270 ms | 6.7 ms | 8.0 | 8 | 35,403 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,908.9 | 121.8 | 466 ms | 8.2 ms | 15.5 | 16 | 57,047 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 3,024.6 | 95.4 | 766 ms | 10.5 ms | 32.0 | 0 | 90,749 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,439.2 | 69.3 | 1,002 ms | 14.4 ms | 64.0 | 0 | 133,184 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,966.8 | 46.0 | 1,786 ms | 21.7 ms | 128.0 | 0 | 178,944 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 178.8 | 179.7 | 111 ms | 5.6 ms | 1.0 | 1 | 5,364 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 339.3 | 170.7 | 142 ms | 5.9 ms | 2.0 | 2 | 10,180 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 642.4 | 162.3 | 217 ms | 6.2 ms | 4.0 | 4 | 19,270 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,133.4 | 143.8 | 275 ms | 7.0 ms | 7.7 | 8 | 34,002 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,841.4 | 115.7 | 360 ms | 8.6 ms | 16.0 | 0 | 55,232 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,793.6 | 87.5 | 560 ms | 11.4 ms | 32.0 | 0 | 83,808 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,941.6 | 61.4 | 820 ms | 16.3 ms | 64.0 | 0 | 118,272 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,080.0 | 39.2 | 1,397 ms | 25.5 ms | 128.0 | 0 | 152,320 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 174.5 | 175.9 | 162 ms | 5.7 ms | 1.0 | 1 | 5,235 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 330.4 | 167.1 | 261 ms | 6.0 ms | 1.9 | 2 | 9,910 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 613.5 | 155.5 | 310 ms | 6.4 ms | 3.9 | 4 | 18,401 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,054.8 | 133.8 | 352 ms | 7.5 ms | 7.7 | 8 | 31,644 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,659.7 | 103.9 | 542 ms | 9.6 ms | 16.0 | 0 | 49,793 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,414.8 | 75.1 | 874 ms | 13.3 ms | 32.0 | 0 | 72,449 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,191.6 | 49.1 | 1,437 ms | 20.4 ms | 64.0 | 0 | 95,744 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,767.3 | 28.6 | 2,437 ms | 35.0 ms | 128.0 | 0 | 113,025 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 167.4 | 168.9 | 180 ms | 5.9 ms | 1.0 | 1 | 5,022 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 317.2 | 160.2 | 196 ms | 6.2 ms | 1.9 | 2 | 9,514 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 569.9 | 144.0 | 213 ms | 6.9 ms | 3.9 | 4 | 17,095 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 949.2 | 118.8 | 529 ms | 8.4 ms | 8.0 | 0 | 28,472 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,398.9 | 87.5 | 888 ms | 11.4 ms | 16.0 | 0 | 41,969 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,870.5 | 58.6 | 1,602 ms | 17.1 ms | 32.0 | 0 | 56,096 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,230.3 | 35.0 | 2,741 ms | 28.6 ms | 64.0 | 0 | 66,880 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 154.5 | 157.1 | 332 ms | 6.4 ms | 1.0 | 1 | 4,635 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 288.9 | 147.0 | 423 ms | 6.8 ms | 2.0 | 2 | 8,666 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 491.9 | 125.0 | 554 ms | 8.0 ms | 4.0 | 4 | 14,753 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 775.2 | 96.4 | 1,018 ms | 10.4 ms | 8.0 | 0 | 23,256 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,039.6 | 64.2 | 1,660 ms | 15.6 ms | 16.0 | 0 | 31,184 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,240.7 | 38.0 | 3,167 ms | 26.3 ms | 32.0 | 0 | 37,216 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 134.2 | 138.7 | 620 ms | 7.2 ms | 1.0 | 1 | 4,028 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 249.4 | 127.7 | 725 ms | 7.8 ms | 1.9 | 2 | 7,480 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 403.5 | 100.8 | 1,172 ms | 9.9 ms | 4.0 | 0 | 12,105 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 562.7 | 70.3 | 1,902 ms | 14.2 ms | 8.0 | 0 | 16,881 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 665.7 | 41.4 | 3,415 ms | 24.2 ms | 16.0 | 0 | 19,968 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.