Speed test matrix
Qwen3.6-35B-A3B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881006 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context37.8K217 ms TTFT
16K context37.4K434 ms TTFT
32K context39K828 ms TTFT
64K context36.5K1,764 ms TTFT
128K context28.2K4,564 ms TTFT
251K context19.6K12,848 ms TTFT
256K context19.2K13,388 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 200.997 ms TTFT | 355.2161 ms TTFT | 592.3931 ms TTFT | 917.6300 ms TTFT | 1,568.8425 ms TTFT | 2,466.2622 ms TTFT | 3,762.7960 ms TTFT | 5,351.11,539 ms TTFT |
| 16K | 197.9113 ms TTFT | 351.3143 ms TTFT | 580.6220 ms TTFT | 889.2278 ms TTFT | 1,521.3335 ms TTFT | 2,285.8444 ms TTFT | 3,402.9816 ms TTFT | 4,571.01,351 ms TTFT |
| 32K | 192.8152 ms TTFT | 342.6226 ms TTFT | 559.2279 ms TTFT | 848.5341 ms TTFT | 1,398.4521 ms TTFT | 2,044.3821 ms TTFT | 2,767.41,321 ms TTFT | Over capacity |
| 64K | 184.9165 ms TTFT | 328.4207 ms TTFT | 520.6250 ms TTFT | 773.6557 ms TTFT | 1,208.3918 ms TTFT | 1,637.91,621 ms TTFT | Over capacity | Over capacity |
| 128K | 169.1339 ms TTFT | 299.8402 ms TTFT | 464.7614 ms TTFT | 655.61,013 ms TTFT | 928.11,739 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 145.1601 ms TTFT | 257.3691 ms TTFT | 379.01,181 ms TTFT | 495.61,876 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 37.8K | — | 217 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 37.4K | — | 434 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 39K | — | 828 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 36.5K | — | 1,764 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28.2K | — | 4,564 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.6K | — | 12,848 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.2K | — | 13,388 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 200.9 | 201.9 | 97 ms | 5.0 ms | 1.0 | 1 | 6,026 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 355.2 | 178.9 | 161 ms | 5.6 ms | 2.0 | 2 | 10,656 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 592.3 | 149.7 | 931 ms | 6.7 ms | 4.0 | 4 | 17,768 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 917.6 | 115.1 | 300 ms | 8.7 ms | 8.0 | 0 | 27,528 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,568.8 | 99.2 | 425 ms | 10.1 ms | 16.0 | 0 | 47,056 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,466.2 | 78.0 | 622 ms | 12.8 ms | 32.0 | 0 | 73,984 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,762.7 | 59.3 | 960 ms | 16.9 ms | 64.0 | 0 | 112,896 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,351.1 | 41.9 | 1,539 ms | 23.9 ms | 128.0 | 0 | 160,512 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 197.9 | 199.0 | 113 ms | 5.0 ms | 1.0 | 1 | 5,936 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 351.3 | 176.9 | 143 ms | 5.7 ms | 2.0 | 2 | 10,537 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 580.6 | 146.9 | 220 ms | 6.8 ms | 4.0 | 4 | 17,418 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 889.2 | 111.6 | 278 ms | 9.0 ms | 8.0 | 0 | 26,688 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,521.3 | 95.6 | 335 ms | 10.5 ms | 16.0 | 0 | 45,632 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,285.8 | 71.9 | 444 ms | 13.9 ms | 32.0 | 0 | 68,576 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,402.9 | 53.2 | 816 ms | 18.8 ms | 64.0 | 0 | 102,080 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,571.0 | 35.5 | 1,351 ms | 28.1 ms | 128.0 | 0 | 137,088 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 192.8 | 194.2 | 152 ms | 5.2 ms | 1.0 | 1 | 5,783 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 342.6 | 173.2 | 226 ms | 5.8 ms | 1.9 | 2 | 10,276 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 559.2 | 141.5 | 279 ms | 7.1 ms | 4.0 | 4 | 16,774 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 848.5 | 106.2 | 341 ms | 9.4 ms | 8.0 | 0 | 25,448 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,398.4 | 87.6 | 521 ms | 11.4 ms | 16.0 | 0 | 41,952 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,044.3 | 63.7 | 821 ms | 15.7 ms | 32.0 | 0 | 61,312 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,767.4 | 42.9 | 1,321 ms | 23.3 ms | 64.0 | 0 | 83,008 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 184.9 | 186.4 | 165 ms | 5.4 ms | 1.0 | 1 | 5,548 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 328.4 | 165.8 | 207 ms | 6.0 ms | 2.0 | 2 | 9,852 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 520.6 | 131.7 | 250 ms | 7.6 ms | 4.0 | 4 | 15,617 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 773.6 | 96.8 | 557 ms | 10.3 ms | 8.0 | 0 | 23,208 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,208.3 | 75.5 | 918 ms | 13.2 ms | 16.0 | 0 | 36,240 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,637.9 | 51.3 | 1,621 ms | 19.5 ms | 32.0 | 0 | 49,120 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 169.1 | 171.6 | 339 ms | 5.8 ms | 1.0 | 1 | 5,072 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 299.8 | 152.1 | 402 ms | 6.6 ms | 2.0 | 2 | 8,995 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 464.7 | 116.1 | 614 ms | 8.6 ms | 4.0 | 0 | 13,940 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 655.6 | 81.6 | 1,013 ms | 12.2 ms | 8.0 | 0 | 19,681 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 928.1 | 57.4 | 1,739 ms | 17.4 ms | 16.0 | 0 | 27,856 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 145.1 | 149.4 | 601 ms | 6.7 ms | 1.0 | 1 | 4,353 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 257.3 | 131.8 | 691 ms | 7.6 ms | 1.9 | 2 | 7,718 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 379.0 | 94.8 | 1,181 ms | 10.6 ms | 4.0 | 0 | 11,370 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 495.6 | 61.9 | 1,876 ms | 16.1 ms | 8.0 | 0 | 14,864 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.