Speed test matrix
Qwen3.5-35B-A3B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880111 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context41.1K199 ms TTFT
16K context40K405 ms TTFT
32K context37.5K861 ms TTFT
64K context33.9K1,899 ms TTFT
128K context25.9K4,961 ms TTFT
251K context17.3K14,555 ms TTFT
256K context17.2K14,954 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 269.995 ms TTFT | 456.8246 ms TTFT | 716.1158 ms TTFT | 1,392.1188 ms TTFT | 2,172.3192 ms TTFT | 3,453.5930 ms TTFT | 5,233.61,489 ms TTFT | 7,082.02,690 ms TTFT |
| 16K | 269.2121 ms TTFT | 454.1149 ms TTFT | 704.6157 ms TTFT | 1,353.8195 ms TTFT | 2,097.6248 ms TTFT | 3,290.4796 ms TTFT | 4,824.11,295 ms TTFT | Over capacity |
| 32K | 263.1152 ms TTFT | 446.5173 ms TTFT | 692.5198 ms TTFT | 1,294.8240 ms TTFT | 1,953.4336 ms TTFT | 3,030.41,006 ms TTFT | Over capacity | Over capacity |
| 64K | 257.0213 ms TTFT | 433.5231 ms TTFT | 653.6280 ms TTFT | 1,189.2291 ms TTFT | 1,783.21,016 ms TTFT | 2,530.21,509 ms TTFT | Over capacity | Over capacity |
| 128K | 238.8354 ms TTFT | 407.1365 ms TTFT | 602.1580 ms TTFT | 1,015.4537 ms TTFT | 1,444.01,682 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 214.3665 ms TTFT | 360.8806 ms TTFT | 513.2837 ms TTFT | 827.01,961 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.1K | — | 199 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 40K | — | 405 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 37.5K | — | 861 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 33.9K | — | 1,899 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 25.9K | — | 4,961 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.3K | — | 14,555 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.2K | — | 14,954 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 269.9 | 273.0 | 95 ms | 3.7 ms | 1.0 | 2 | 8,097 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 456.8 | 229.9 | 246 ms | 4.4 ms | 2.0 | 2 | 13,702 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 716.1 | 181.9 | 158 ms | 5.5 ms | 4.0 | 4 | 21,478 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,392.1 | 176.7 | 188 ms | 5.7 ms | 8.0 | 8 | 41,759 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,172.3 | 139.2 | 192 ms | 7.2 ms | 16.0 | 16 | 65,142 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,453.5 | 108.7 | 930 ms | 9.2 ms | 32.0 | 0 | 103,600 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,233.6 | 81.4 | 1,489 ms | 12.3 ms | 64.0 | 0 | 156,995 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,082.0 | 54.0 | 2,690 ms | 18.5 ms | 128.0 | 0 | 212,550 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 269.2 | 272.6 | 121 ms | 3.7 ms | 1.0 | 2 | 8,075 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 454.1 | 228.7 | 149 ms | 4.4 ms | 2.0 | 2 | 13,623 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 704.6 | 179.3 | 157 ms | 5.6 ms | 4.0 | 4 | 21,137 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,353.8 | 171.5 | 195 ms | 5.8 ms | 8.0 | 8 | 40,601 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,097.6 | 133.6 | 248 ms | 7.5 ms | 15.9 | 16 | 62,899 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,290.4 | 103.8 | 796 ms | 9.6 ms | 32.0 | 0 | 98,722 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,824.1 | 75.5 | 1,295 ms | 13.3 ms | 64.0 | 0 | 144,623 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 263.1 | 266.6 | 152 ms | 3.8 ms | 1.0 | 2 | 7,892 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 446.5 | 225.3 | 173 ms | 4.4 ms | 2.0 | 2 | 13,395 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 692.5 | 176.5 | 198 ms | 5.7 ms | 4.0 | 4 | 20,775 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,294.8 | 164.7 | 240 ms | 6.1 ms | 7.9 | 8 | 38,834 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,953.4 | 124.5 | 336 ms | 8.0 ms | 16.0 | 15 | 58,587 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,030.4 | 95.0 | 1,006 ms | 10.5 ms | 32.0 | 0 | 90,922 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 257.0 | 260.6 | 213 ms | 3.8 ms | 1.0 | 2 | 7,708 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 433.5 | 218.3 | 231 ms | 4.6 ms | 2.0 | 2 | 13,004 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 653.6 | 166.3 | 280 ms | 6.0 ms | 4.0 | 4 | 19,602 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,189.2 | 151.3 | 291 ms | 6.6 ms | 8.0 | 8 | 35,668 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,783.2 | 111.4 | 1,016 ms | 9.0 ms | 16.0 | 0 | 53,486 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,530.2 | 78.6 | 1,509 ms | 12.7 ms | 32.0 | 0 | 75,904 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 238.8 | 242.7 | 354 ms | 4.1 ms | 1.0 | 1 | 7,163 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 407.1 | 206.4 | 365 ms | 4.8 ms | 2.0 | 2 | 12,211 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 602.1 | 153.9 | 580 ms | 6.5 ms | 3.9 | 4 | 18,057 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,015.4 | 128.6 | 537 ms | 7.8 ms | 7.9 | 8 | 30,452 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,444.0 | 89.5 | 1,682 ms | 11.2 ms | 16.0 | 0 | 43,307 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 214.3 | 220.8 | 665 ms | 4.5 ms | 1.0 | 1 | 6,428 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 360.8 | 185.6 | 806 ms | 5.4 ms | 2.0 | 2 | 10,823 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 513.2 | 135.3 | 837 ms | 7.4 ms | 3.9 | 4 | 15,394 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 827.0 | 103.7 | 1,961 ms | 9.6 ms | 8.0 | 0 | 24,808 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.