Speed test matrix
Qwen3.6-35B-A3B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880662 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context40.5K202 ms TTFT
16K context39.7K408 ms TTFT
32K context40K807 ms TTFT
64K context34.2K1,881 ms TTFT
128K context25.9K4,962 ms TTFT
251K context17.4K14,519 ms TTFT
256K context17.2K14,918 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 271.593 ms TTFT | 453.6297 ms TTFT | 879.6136 ms TTFT | 1,414.4165 ms TTFT | 2,231.2221 ms TTFT | 3,480.1941 ms TTFT | 5,159.61,452 ms TTFT | 6,835.32,703 ms TTFT |
| 16K | 263.3118 ms TTFT | 445.5143 ms TTFT | 879.3187 ms TTFT | 1,374.4181 ms TTFT | 2,178.9219 ms TTFT | 3,324.9781 ms TTFT | 4,787.61,300 ms TTFT | Over capacity |
| 32K | 265.2146 ms TTFT | 439.3175 ms TTFT | 849.8195 ms TTFT | 1,327.2259 ms TTFT | 2,043.1275 ms TTFT | 3,052.51,025 ms TTFT | Over capacity | Over capacity |
| 64K | 248.1219 ms TTFT | 422.0219 ms TTFT | 822.2263 ms TTFT | 1,224.2344 ms TTFT | 1,837.4924 ms TTFT | 2,568.51,566 ms TTFT | Over capacity | Over capacity |
| 128K | 237.7352 ms TTFT | 399.7392 ms TTFT | 728.6427 ms TTFT | 1,055.9437 ms TTFT | 1,495.22,016 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 210.1665 ms TTFT | 356.0862 ms TTFT | 596.1739 ms TTFT | 845.12,103 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.5K | — | 202 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 39.7K | — | 408 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 40K | — | 807 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 34.2K | — | 1,881 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 25.9K | — | 4,962 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.4K | — | 14,519 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.2K | — | 14,918 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 271.5 | 274.8 | 93 ms | 3.6 ms | 1.0 | 2 | 8,146 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 453.6 | 227.8 | 297 ms | 4.4 ms | 2.0 | 2 | 13,606 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 879.6 | 221.6 | 136 ms | 4.5 ms | 4.0 | 4 | 26,379 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,414.4 | 179.5 | 165 ms | 5.6 ms | 7.9 | 8 | 42,432 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,231.2 | 143.1 | 221 ms | 7.0 ms | 15.9 | 16 | 66,930 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,480.1 | 110.1 | 941 ms | 9.1 ms | 32.0 | 0 | 104,334 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,159.6 | 80.2 | 1,452 ms | 12.5 ms | 64.0 | 0 | 154,766 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,835.3 | 52.2 | 2,703 ms | 19.2 ms | 128.0 | 0 | 205,074 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 263.3 | 264.4 | 118 ms | 3.8 ms | 1.0 | 2 | 7,898 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 445.5 | 225.3 | 143 ms | 4.4 ms | 2.0 | 2 | 13,366 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 879.3 | 222.3 | 187 ms | 4.5 ms | 4.0 | 4 | 26,377 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,374.4 | 175.9 | 181 ms | 5.7 ms | 8.0 | 8 | 41,219 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,178.9 | 139.7 | 219 ms | 7.2 ms | 15.9 | 16 | 65,370 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,324.9 | 104.9 | 781 ms | 9.5 ms | 32.0 | 0 | 99,705 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,787.6 | 75.3 | 1,300 ms | 13.3 ms | 64.0 | 0 | 143,547 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 265.2 | 272.7 | 146 ms | 3.7 ms | 1.0 | 2 | 7,961 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 439.3 | 222.2 | 175 ms | 4.5 ms | 2.0 | 2 | 13,178 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 849.8 | 214.2 | 195 ms | 4.7 ms | 4.0 | 4 | 25,488 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,327.2 | 168.1 | 259 ms | 6.0 ms | 8.0 | 8 | 39,804 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,043.1 | 129.4 | 275 ms | 7.7 ms | 16.0 | 15 | 61,288 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,052.5 | 96.1 | 1,025 ms | 10.4 ms | 32.0 | 0 | 91,568 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 248.1 | 252.1 | 219 ms | 4.0 ms | 1.0 | 2 | 7,443 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 422.0 | 212.7 | 219 ms | 4.7 ms | 2.0 | 2 | 12,657 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 822.2 | 208.0 | 263 ms | 4.8 ms | 4.0 | 4 | 24,659 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,224.2 | 157.3 | 344 ms | 6.4 ms | 8.0 | 8 | 36,711 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,837.4 | 115.2 | 924 ms | 8.7 ms | 16.0 | 0 | 55,103 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,568.5 | 79.9 | 1,566 ms | 12.5 ms | 32.0 | 0 | 76,997 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 237.7 | 241.4 | 352 ms | 4.1 ms | 1.0 | 1 | 7,130 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 399.7 | 202.8 | 392 ms | 4.9 ms | 2.0 | 2 | 11,987 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 728.6 | 187.2 | 427 ms | 5.3 ms | 4.0 | 4 | 21,854 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,055.9 | 134.3 | 437 ms | 7.4 ms | 7.9 | 8 | 31,666 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,495.2 | 93.0 | 2,016 ms | 10.8 ms | 16.0 | 0 | 44,849 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 210.1 | 216.5 | 665 ms | 4.6 ms | 1.0 | 1 | 6,303 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 356.0 | 185.0 | 862 ms | 5.4 ms | 2.0 | 2 | 10,679 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 596.1 | 153.0 | 739 ms | 6.5 ms | 3.9 | 4 | 17,874 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 845.1 | 106.2 | 2,103 ms | 9.4 ms | 8.0 | 0 | 25,346 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.