Speed test matrix
Qwen3.5-9B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.885109 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context30.4K270 ms TTFT
16K context29.9K542 ms TTFT
32K context29.2K1,105 ms TTFT
64K context27K2,385 ms TTFT
128K context22.4K5,746 ms TTFT
251K context17K14,864 ms TTFT
256K context16.7K15,376 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 247.568 ms TTFT | 481.988 ms TTFT | 938.898 ms TTFT | 1,773.2117 ms TTFT | 3,048.6136 ms TTFT | 4,716.3194 ms TTFT | 6,320.11,074 ms TTFT | 7,520.11,477 ms TTFT |
| 16K | 250.5111 ms TTFT | 482.3115 ms TTFT | 933.0166 ms TTFT | 1,673.2161 ms TTFT | 2,793.9212 ms TTFT | 4,028.9238 ms TTFT | 5,301.21,348 ms TTFT | 6,063.22,487 ms TTFT |
| 32K | 243.6110 ms TTFT | 468.1124 ms TTFT | 870.2154 ms TTFT | 1,520.5150 ms TTFT | 2,368.2221 ms TTFT | 3,305.0794 ms TTFT | 3,923.81,482 ms TTFT | Over capacity |
| 64K | 231.4167 ms TTFT | 439.9211 ms TTFT | 799.1230 ms TTFT | 1,273.2322 ms TTFT | 1,813.3940 ms TTFT | 2,301.31,531 ms TTFT | Over capacity | Over capacity |
| 128K | 217.0319 ms TTFT | 394.4332 ms TTFT | 650.3410 ms TTFT | 957.1946 ms TTFT | 1,242.51,597 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 179.3579 ms TTFT | 314.6631 ms TTFT | 476.3631 ms TTFT | 635.61,920 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.4K | — | 270 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.9K | — | 542 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.2K | — | 1,105 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 27K | — | 2,385 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.4K | — | 5,746 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17K | — | 14,864 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.7K | — | 15,376 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 247.5 | 252.2 | 68 ms | 4.0 ms | 1.0 | 2 | 7,424 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 481.9 | 246.4 | 88 ms | 4.1 ms | 2.0 | 4 | 14,454 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 938.8 | 238.5 | 98 ms | 4.2 ms | 4.0 | 6 | 28,158 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,773.2 | 222.7 | 117 ms | 4.5 ms | 8.0 | 8 | 53,185 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,048.6 | 192.8 | 136 ms | 5.2 ms | 15.9 | 16 | 91,463 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,716.3 | 151.3 | 194 ms | 6.6 ms | 31.9 | 32 | 141,493 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,320.1 | 99.8 | 1,074 ms | 10.0 ms | 64.0 | 0 | 189,650 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,520.1 | 58.8 | 1,477 ms | 17.0 ms | 128.0 | 0 | 225,678 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 250.5 | 252.3 | 111 ms | 4.0 ms | 1.0 | 2 | 7,513 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 482.3 | 245.9 | 115 ms | 4.1 ms | 2.0 | 4 | 14,467 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 933.0 | 236.5 | 166 ms | 4.2 ms | 4.0 | 4 | 27,984 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,673.2 | 211.4 | 161 ms | 4.7 ms | 8.0 | 8 | 50,191 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,793.9 | 177.2 | 212 ms | 5.6 ms | 15.9 | 16 | 83,809 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,028.9 | 126.0 | 238 ms | 7.9 ms | 31.8 | 32 | 120,854 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,301.2 | 82.5 | 1,348 ms | 12.1 ms | 64.0 | 0 | 158,941 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,063.2 | 46.4 | 2,487 ms | 21.6 ms | 128.0 | 0 | 181,945 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 243.6 | 245.8 | 110 ms | 4.1 ms | 1.0 | 2 | 7,306 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 468.1 | 237.0 | 124 ms | 4.2 ms | 2.0 | 2 | 14,042 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 870.2 | 219.3 | 154 ms | 4.6 ms | 4.0 | 4 | 26,101 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,520.5 | 194.1 | 150 ms | 5.2 ms | 8.0 | 8 | 45,611 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,368.2 | 150.8 | 221 ms | 6.6 ms | 15.9 | 16 | 71,027 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,305.0 | 103.3 | 794 ms | 9.7 ms | 32.0 | 0 | 99,105 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,923.8 | 61.4 | 1,482 ms | 16.3 ms | 64.0 | 0 | 117,633 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 231.4 | 233.5 | 167 ms | 4.3 ms | 1.0 | 1 | 6,942 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 439.9 | 222.2 | 211 ms | 4.5 ms | 2.0 | 2 | 13,193 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 799.1 | 203.5 | 230 ms | 4.9 ms | 4.0 | 4 | 23,970 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,273.2 | 163.9 | 322 ms | 6.1 ms | 8.0 | 8 | 38,186 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,813.3 | 114.4 | 940 ms | 8.7 ms | 16.0 | 0 | 54,370 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,301.3 | 72.3 | 1,531 ms | 13.8 ms | 32.0 | 0 | 69,031 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 217.0 | 221.9 | 319 ms | 4.5 ms | 1.0 | 1 | 6,510 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 394.4 | 199.3 | 332 ms | 5.0 ms | 2.0 | 2 | 11,831 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 650.3 | 168.7 | 410 ms | 5.9 ms | 4.0 | 4 | 19,506 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 957.1 | 120.5 | 946 ms | 8.3 ms | 8.0 | 2 | 28,698 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,242.5 | 77.9 | 1,597 ms | 12.8 ms | 16.0 | 0 | 37,249 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 179.3 | 187.0 | 579 ms | 5.4 ms | 1.0 | 1 | 5,379 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 314.6 | 165.4 | 631 ms | 6.0 ms | 1.9 | 2 | 9,434 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 476.3 | 124.5 | 631 ms | 8.0 ms | 4.0 | 3 | 14,281 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 635.6 | 80.9 | 1,920 ms | 12.4 ms | 8.0 | 0 | 19,066 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.