Speed test matrix
Qwen3.5-35B-A3B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context41.3K198 ms TTFT
16K context39.7K409 ms TTFT
32K context38.3K842 ms TTFT
64K context34.4K1,875 ms TTFT
128K context26K4,959 ms TTFT
251K context17.4K14,487 ms TTFT
256K context17.2K14,951 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 261.892 ms TTFT | 423.4251 ms TTFT | 641.8135 ms TTFT | 1,213.3167 ms TTFT | 1,879.7559 ms TTFT | 2,951.8928 ms TTFT | 4,410.61,502 ms TTFT | 5,905.2Requests queued2,688 ms TTFT |
| 16K | 203.9116 ms TTFT | 443.1145 ms TTFT | 646.6190 ms TTFT | 1,170.6190 ms TTFT | 1,826.1526 ms TTFT | 2,817.8786 ms TTFT | 4,118.41,306 ms TTFT | Over capacity |
| 32K | 256.2129 ms TTFT | 419.9177 ms TTFT | 626.2221 ms TTFT | 1,152.9205 ms TTFT | 1,735.1610 ms TTFT | 2,523.21,036 ms TTFT | Over capacity | Over capacity |
| 64K | 235.5205 ms TTFT | 412.8227 ms TTFT | 581.6237 ms TTFT | 1,034.4267 ms TTFT | 1,556.3939 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 235.3335 ms TTFT | 371.0381 ms TTFT | 525.0413 ms TTFT | 922.3951 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 204.3643 ms TTFT | 342.9703 ms TTFT | 419.91,225 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.3K | — | 198 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 39.7K | — | 409 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 38.3K | — | 842 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 34.4K | — | 1,875 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 26K | — | 4,959 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.4K | — | 14,487 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.2K | — | 14,951 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 261.8 | 262.3 | 92 ms | 3.8 ms | 1.0 | 2 | 7,854 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 423.4 | 213.5 | 251 ms | 4.7 ms | 2.0 | 2 | 12,702 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 641.8 | 162.0 | 135 ms | 6.2 ms | 4.0 | 4 | 19,251 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,213.3 | 152.3 | 167 ms | 6.6 ms | 8.0 | 8 | 36,395 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,879.7 | 117.3 | 559 ms | 8.5 ms | 16.0 | 6 | 56,377 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,951.8 | 92.2 | 928 ms | 10.8 ms | 32.0 | 0 | 88,560 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,410.6 | 68.0 | 1,502 ms | 14.7 ms | 64.0 | 0 | 132,291 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 5,905.2 | 44.6 | 2,688 ms | 22.4 ms | 127.6 | 0 | 177,028 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 203.9 | 219.0 | 116 ms | 5.0 ms | 1.0 | 1 | 6,116 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 443.1 | 225.9 | 145 ms | 4.4 ms | 2.0 | 2 | 13,292 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 646.6 | 161.7 | 190 ms | 6.2 ms | 4.0 | 4 | 19,394 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,170.6 | 145.1 | 190 ms | 6.9 ms | 8.0 | 8 | 35,115 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,826.1 | 115.6 | 526 ms | 8.6 ms | 16.0 | 0 | 54,775 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,817.8 | 88.8 | 786 ms | 11.3 ms | 32.0 | 0 | 84,507 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,118.4 | 63.5 | 1,306 ms | 15.8 ms | 64.0 | 0 | 123,452 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 256.2 | 247.9 | 129 ms | 4.0 ms | 1.0 | 2 | 7,685 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 419.9 | 212.3 | 177 ms | 4.7 ms | 2.0 | 2 | 12,596 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 626.2 | 156.8 | 221 ms | 6.4 ms | 4.0 | 4 | 18,783 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,152.9 | 145.4 | 205 ms | 6.9 ms | 8.0 | 8 | 34,580 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,735.1 | 109.7 | 610 ms | 9.1 ms | 16.0 | 0 | 52,055 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,523.2 | 78.8 | 1,036 ms | 12.7 ms | 32.0 | 0 | 75,685 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 235.5 | 238.4 | 205 ms | 4.2 ms | 1.0 | 1 | 7,065 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 412.8 | 209.4 | 227 ms | 4.8 ms | 2.0 | 2 | 12,384 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 581.6 | 145.7 | 237 ms | 6.9 ms | 4.0 | 4 | 17,448 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,034.4 | 128.9 | 267 ms | 7.8 ms | 7.9 | 8 | 31,024 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,556.3 | 95.8 | 939 ms | 10.4 ms | 16.0 | 0 | 46,675 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 235.3 | 240.4 | 335 ms | 4.2 ms | 1.0 | 1 | 7,058 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 371.0 | 194.1 | 381 ms | 5.2 ms | 2.0 | 2 | 11,129 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 525.0 | 138.3 | 413 ms | 7.2 ms | 4.0 | 4 | 15,746 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 922.3 | 114.2 | 951 ms | 8.8 ms | 8.0 | 1 | 27,668 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 204.3 | 208.4 | 643 ms | 4.8 ms | 1.0 | 1 | 6,131 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 342.9 | 175.5 | 703 ms | 5.7 ms | 2.0 | 2 | 10,282 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 419.9 | 107.1 | 1,225 ms | 9.3 ms | 4.0 | 0 | 12,596 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.