Speed test matrix
Qwen3.6-35B-A3B-int4-mixed-AutoRound on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4+BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-intel-vllm/bin/vllm" serve Intel/Qwen3.6-35B-A3B-int4-mixed-AutoRound --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880000 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context33.6K244 ms TTFT
16K context35.1K462 ms TTFT
32K context33.2K974 ms TTFT
64K context30.3K2,127 ms TTFT
128K context23.4K5,495 ms TTFT
251K context16.1K15,634 ms TTFT
256K context15.9K16,153 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 267.9100 ms TTFT | 503.6115 ms TTFT | 849.9165 ms TTFT | 1,637.9179 ms TTFT | 2,643.7261 ms TTFT | 3,815.41,055 ms TTFT | 5,528.71,783 ms TTFT | 7,320.83,174 ms TTFT |
| 16K | 265.8122 ms TTFT | 501.7171 ms TTFT | 848.8149 ms TTFT | 1,606.6201 ms TTFT | 2,539.0224 ms TTFT | 3,605.8940 ms TTFT | 5,091.21,480 ms TTFT | 6,584.52,580 ms TTFT |
| 32K | 258.1147 ms TTFT | 488.1171 ms TTFT | 827.7237 ms TTFT | 1,521.7322 ms TTFT | 2,351.7273 ms TTFT | 3,237.61,100 ms TTFT | 4,381.01,833 ms TTFT | Over capacity |
| 64K | 251.0208 ms TTFT | 478.1253 ms TTFT | 785.2248 ms TTFT | 1,385.9317 ms TTFT | 2,023.1330 ms TTFT | 2,695.41,587 ms TTFT | 3,370.12,942 ms TTFT | Over capacity |
| 128K | 246.1365 ms TTFT | 448.7427 ms TTFT | 692.5509 ms TTFT | 1,170.1531 ms TTFT | 1,652.12,055 ms TTFT | 1,969.53,738 ms TTFT | Over capacity | Over capacity |
| 251K | 214.1712 ms TTFT | 387.5766 ms TTFT | 577.21,276 ms TTFT | 912.12,026 ms TTFT | 1,127.63,602 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.6K | — | 244 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 35.1K | — | 462 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.2K | — | 974 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.3K | — | 2,127 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.4K | — | 5,495 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 16.1K | — | 15,634 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15.9K | — | 16,153 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 267.9 | 269.9 | 100 ms | 3.7 ms | 1.0 | 2 | 8,036 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 503.6 | 256.7 | 115 ms | 3.9 ms | 2.0 | 4 | 15,106 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 849.9 | 214.9 | 165 ms | 4.7 ms | 4.0 | 4 | 25,497 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,637.9 | 206.5 | 179 ms | 4.8 ms | 7.9 | 8 | 49,121 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,643.7 | 167.5 | 261 ms | 6.0 ms | 16.0 | 16 | 79,310 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,815.4 | 119.4 | 1,055 ms | 8.4 ms | 32.0 | 4 | 114,439 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,528.7 | 84.7 | 1,783 ms | 11.8 ms | 64.0 | 0 | 165,855 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,320.8 | 54.4 | 3,174 ms | 18.4 ms | 128.0 | 0 | 219,670 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 265.8 | 269.8 | 122 ms | 3.7 ms | 1.0 | 2 | 7,973 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 501.7 | 255.7 | 171 ms | 3.9 ms | 2.0 | 4 | 15,049 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 848.8 | 213.6 | 149 ms | 4.7 ms | 4.0 | 4 | 25,459 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,606.6 | 202.7 | 201 ms | 4.9 ms | 8.0 | 8 | 48,196 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,539.0 | 161.0 | 224 ms | 6.2 ms | 15.9 | 16 | 76,147 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,605.8 | 112.4 | 940 ms | 8.9 ms | 32.0 | 0 | 108,145 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,091.2 | 78.9 | 1,480 ms | 12.7 ms | 64.0 | 0 | 152,720 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,584.5 | 50.0 | 2,580 ms | 20.0 ms | 128.0 | 0 | 197,462 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 258.1 | 260.4 | 147 ms | 3.8 ms | 1.0 | 2 | 7,742 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 488.1 | 251.1 | 171 ms | 4.0 ms | 2.0 | 4 | 14,641 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 827.7 | 209.3 | 237 ms | 4.8 ms | 4.0 | 4 | 24,828 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,521.7 | 193.7 | 322 ms | 5.2 ms | 7.9 | 8 | 45,638 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,351.7 | 149.7 | 273 ms | 6.7 ms | 15.9 | 16 | 70,534 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,237.6 | 101.2 | 1,100 ms | 9.9 ms | 32.0 | 0 | 97,107 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,381.0 | 67.5 | 1,833 ms | 14.8 ms | 64.0 | 0 | 131,320 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 251.0 | 261.5 | 208 ms | 3.8 ms | 1.0 | 2 | 7,530 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 478.1 | 241.1 | 253 ms | 4.1 ms | 2.0 | 2 | 14,342 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 785.2 | 198.7 | 248 ms | 5.0 ms | 4.0 | 4 | 23,554 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,385.9 | 174.4 | 317 ms | 5.7 ms | 7.9 | 8 | 41,567 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,023.1 | 126.5 | 330 ms | 7.9 ms | 15.9 | 16 | 60,670 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,695.4 | 83.2 | 1,587 ms | 12.0 ms | 32.0 | 0 | 80,852 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,370.1 | 51.4 | 2,942 ms | 19.5 ms | 64.0 | 0 | 101,063 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 246.1 | 250.5 | 365 ms | 4.0 ms | 1.0 | 2 | 7,350 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 448.7 | 228.8 | 427 ms | 4.4 ms | 2.0 | 2 | 13,458 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 692.5 | 176.9 | 509 ms | 5.7 ms | 3.9 | 4 | 20,772 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,170.1 | 149.0 | 531 ms | 6.7 ms | 7.9 | 8 | 35,103 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,652.1 | 102.4 | 2,055 ms | 9.8 ms | 16.0 | 0 | 49,560 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,969.5 | 61.4 | 3,738 ms | 16.3 ms | 32.0 | 0 | 59,057 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 214.1 | 219.9 | 712 ms | 4.5 ms | 1.0 | 1 | 6,422 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 387.5 | 198.9 | 766 ms | 5.0 ms | 2.0 | 2 | 11,625 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 577.2 | 151.1 | 1,276 ms | 6.6 ms | 3.9 | 4 | 17,312 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 912.1 | 114.9 | 2,026 ms | 8.7 ms | 8.0 | 3 | 27,351 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,127.6 | 69.6 | 3,602 ms | 14.4 ms | 16.0 | 0 | 33,815 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.