Speed test matrix
Qwen3.5-35B-A3B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881006 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context39.3K209 ms TTFT
16K context38.4K423 ms TTFT
32K context39.2K823 ms TTFT
64K context36.3K1,777 ms TTFT
128K context28K4,589 ms TTFT
251K context19.6K12,877 ms TTFT
256K context19.2K13,364 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 200.190 ms TTFT | 355.2152 ms TTFT | 589.3934 ms TTFT | 915.8277 ms TTFT | 1,564.4362 ms TTFT | 2,402.9675 ms TTFT | 3,659.9979 ms TTFT | 5,284.11,536 ms TTFT |
| 16K | 197.8111 ms TTFT | 351.1136 ms TTFT | 581.4188 ms TTFT | 886.2296 ms TTFT | 1,488.1345 ms TTFT | 2,276.3571 ms TTFT | 3,320.5817 ms TTFT | 4,498.11,380 ms TTFT |
| 32K | 192.5151 ms TTFT | 342.5215 ms TTFT | 556.5247 ms TTFT | 844.5370 ms TTFT | 1,376.3583 ms TTFT | 2,006.0850 ms TTFT | 2,741.11,346 ms TTFT | Over capacity |
| 64K | 184.1177 ms TTFT | 327.7207 ms TTFT | 520.4222 ms TTFT | 769.1529 ms TTFT | 1,188.2848 ms TTFT | 1,606.11,478 ms TTFT | Over capacity | Over capacity |
| 128K | 169.0314 ms TTFT | 299.0425 ms TTFT | 462.8636 ms TTFT | 651.31,021 ms TTFT | 911.41,692 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 145.1591 ms TTFT | 256.6684 ms TTFT | 377.21,161 ms TTFT | 494.31,974 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.3K | — | 209 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.4K | — | 423 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 39.2K | — | 823 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 36.3K | — | 1,777 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 28K | — | 4,589 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.6K | — | 12,877 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.2K | — | 13,364 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 200.1 | 200.9 | 90 ms | 5.0 ms | 1.0 | 1 | 6,005 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 355.2 | 178.9 | 152 ms | 5.6 ms | 2.0 | 2 | 10,656 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 589.3 | 149.1 | 934 ms | 6.7 ms | 4.0 | 4 | 17,676 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 915.8 | 115.0 | 277 ms | 8.7 ms | 8.0 | 0 | 27,488 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,564.4 | 98.8 | 362 ms | 10.1 ms | 16.0 | 0 | 46,928 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,402.9 | 76.1 | 675 ms | 13.1 ms | 32.0 | 0 | 72,096 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,659.9 | 57.8 | 979 ms | 17.3 ms | 64.0 | 0 | 109,824 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,284.1 | 41.4 | 1,536 ms | 24.2 ms | 128.0 | 0 | 158,464 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 197.8 | 198.8 | 111 ms | 5.0 ms | 1.0 | 1 | 5,934 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 351.1 | 176.7 | 136 ms | 5.7 ms | 2.0 | 2 | 10,533 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 581.4 | 146.6 | 188 ms | 6.8 ms | 4.0 | 4 | 17,441 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 886.2 | 111.3 | 296 ms | 9.0 ms | 8.0 | 0 | 26,584 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,488.1 | 93.6 | 345 ms | 10.7 ms | 16.0 | 0 | 44,640 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,276.3 | 71.6 | 571 ms | 14.0 ms | 32.0 | 0 | 68,289 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,320.5 | 52.0 | 817 ms | 19.2 ms | 64.0 | 0 | 99,584 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,498.1 | 35.0 | 1,380 ms | 28.5 ms | 128.0 | 0 | 134,912 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 192.5 | 194.0 | 151 ms | 5.2 ms | 1.0 | 1 | 5,776 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 342.5 | 173.0 | 215 ms | 5.8 ms | 2.0 | 2 | 10,274 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 556.5 | 141.1 | 247 ms | 7.1 ms | 4.0 | 4 | 16,693 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 844.5 | 105.8 | 370 ms | 9.5 ms | 8.0 | 0 | 25,336 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,376.3 | 86.2 | 583 ms | 11.6 ms | 16.0 | 0 | 41,280 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,006.0 | 62.5 | 850 ms | 16.0 ms | 32.0 | 0 | 60,160 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,741.1 | 42.4 | 1,346 ms | 23.6 ms | 64.0 | 0 | 82,240 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 184.1 | 185.6 | 177 ms | 5.4 ms | 1.0 | 1 | 5,521 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 327.7 | 165.4 | 207 ms | 6.0 ms | 2.0 | 2 | 9,831 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 520.4 | 131.6 | 222 ms | 7.6 ms | 4.0 | 4 | 15,612 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 769.1 | 96.3 | 529 ms | 10.4 ms | 8.0 | 0 | 23,073 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,188.2 | 74.2 | 848 ms | 13.5 ms | 16.0 | 0 | 35,632 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,606.1 | 50.3 | 1,478 ms | 19.9 ms | 32.0 | 0 | 48,160 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 169.0 | 171.5 | 314 ms | 5.8 ms | 1.0 | 1 | 5,070 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 299.0 | 151.9 | 425 ms | 6.6 ms | 2.0 | 2 | 8,969 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 462.8 | 115.7 | 636 ms | 8.6 ms | 4.0 | 0 | 13,885 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 651.3 | 81.1 | 1,021 ms | 12.3 ms | 8.0 | 0 | 19,536 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 911.4 | 56.3 | 1,692 ms | 17.7 ms | 16.0 | 0 | 27,344 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 145.1 | 149.1 | 591 ms | 6.7 ms | 1.0 | 1 | 4,353 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 256.6 | 131.3 | 684 ms | 7.6 ms | 1.9 | 2 | 7,703 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 377.2 | 94.3 | 1,161 ms | 10.6 ms | 4.0 | 0 | 11,312 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 494.3 | 61.7 | 1,974 ms | 16.2 ms | 8.0 | 0 | 14,824 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.