Speed test matrix
Qwen3.5-9B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-9B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context32K256 ms TTFT
16K context31.2K520 ms TTFT
32K context30.2K1,071 ms TTFT
64K context28.4K2,269 ms TTFT
128K context24K5,362 ms TTFT
251K context18.2K13,831 ms TTFT
256K context17.9K14,337 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 160.370 ms TTFT | 315.9119 ms TTFT | 579.2Requests queued914 ms TTFT | 1,143.0180 ms TTFT | 2,077.7231 ms TTFT | 3,448.9352 ms TTFT | 5,090.3611 ms TTFT | 6,570.2973 ms TTFT |
| 16K | 158.296 ms TTFT | 311.2120 ms TTFT | 592.3158 ms TTFT | 1,079.3203 ms TTFT | 1,896.9292 ms TTFT | 2,949.5427 ms TTFT | 3,985.9769 ms TTFT | 4,810.71,359 ms TTFT |
| 32K | 155.1123 ms TTFT | 300.3141 ms TTFT | 557.1219 ms TTFT | 973.5235 ms TTFT | 1,600.0437 ms TTFT | 2,241.5689 ms TTFT | 2,763.81,197 ms TTFT | Over capacity |
| 64K | 148.8172 ms TTFT | 281.2207 ms TTFT | 500.8226 ms TTFT | 820.9512 ms TTFT | 1,187.0796 ms TTFT | 1,492.91,385 ms TTFT | Over capacity | Over capacity |
| 128K | 137.6289 ms TTFT | 249.1312 ms TTFT | 415.4540 ms TTFT | 602.9860 ms TTFT | 766.71,526 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 121.9568 ms TTFT | 202.9827 ms TTFT | 301.21,181 ms TTFT | 388.71,979 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32K | — | 256 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.2K | — | 520 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.2K | — | 1,071 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.4K | — | 2,269 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24K | — | 5,362 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.2K | — | 13,831 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.9K | — | 14,337 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 160.3 | 160.9 | 70 ms | 6.2 ms | 1.0 | 1 | 4,811 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 315.9 | 158.9 | 119 ms | 6.3 ms | 1.9 | 2 | 9,476 | 30.0 s |
| Sustained decode | 8K | 4 | Requests queued | 579.2 | 154.0 | 914 ms | 6.5 ms | 3.7 | 4 | 17,382 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,143.0 | 144.5 | 180 ms | 6.9 ms | 8.0 | 8 | 34,291 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,077.7 | 131.1 | 231 ms | 7.6 ms | 16.0 | 16 | 62,321 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,448.9 | 108.3 | 352 ms | 9.2 ms | 32.0 | 0 | 103,456 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,090.3 | 79.8 | 611 ms | 12.5 ms | 64.0 | 0 | 152,714 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,570.2 | 51.1 | 973 ms | 19.6 ms | 128.0 | 0 | 197,120 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 158.2 | 159.0 | 96 ms | 6.3 ms | 1.0 | 1 | 4,749 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 311.2 | 156.5 | 120 ms | 6.4 ms | 2.0 | 2 | 9,340 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 592.3 | 149.4 | 158 ms | 6.7 ms | 4.0 | 4 | 17,770 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,079.3 | 136.1 | 203 ms | 7.3 ms | 8.0 | 8 | 32,380 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,896.9 | 118.8 | 292 ms | 8.4 ms | 16.0 | 0 | 56,896 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,949.5 | 92.2 | 427 ms | 10.8 ms | 32.0 | 0 | 88,480 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,985.9 | 62.0 | 769 ms | 16.1 ms | 64.0 | 0 | 119,552 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,810.7 | 37.0 | 1,359 ms | 27.0 ms | 128.0 | 0 | 144,255 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 155.1 | 156.1 | 123 ms | 6.4 ms | 1.0 | 1 | 4,653 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 300.3 | 151.1 | 141 ms | 6.6 ms | 2.0 | 2 | 9,007 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 557.1 | 140.9 | 219 ms | 7.1 ms | 4.0 | 4 | 16,711 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 973.5 | 122.8 | 235 ms | 8.1 ms | 8.0 | 8 | 29,203 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,600.0 | 100.3 | 437 ms | 10.0 ms | 16.0 | 0 | 48,000 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,241.5 | 70.2 | 689 ms | 14.3 ms | 32.0 | 0 | 67,232 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,763.8 | 43.4 | 1,197 ms | 23.1 ms | 64.0 | 0 | 82,880 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 148.8 | 150.2 | 172 ms | 6.7 ms | 1.0 | 1 | 4,465 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 281.2 | 142.1 | 207 ms | 7.0 ms | 2.0 | 2 | 8,437 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 500.8 | 126.9 | 226 ms | 7.9 ms | 4.0 | 4 | 15,020 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 820.9 | 102.8 | 512 ms | 9.7 ms | 8.0 | 0 | 24,624 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,187.0 | 74.3 | 796 ms | 13.5 ms | 16.0 | 0 | 35,600 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,492.9 | 46.8 | 1,385 ms | 21.4 ms | 32.0 | 0 | 44,768 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 137.6 | 140.2 | 289 ms | 7.1 ms | 1.0 | 1 | 4,129 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 249.1 | 130.9 | 312 ms | 7.7 ms | 1.9 | 2 | 7,474 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 415.4 | 104.0 | 540 ms | 9.6 ms | 4.0 | 0 | 12,468 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 602.9 | 75.4 | 860 ms | 13.3 ms | 8.0 | 0 | 18,080 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 766.7 | 47.9 | 1,526 ms | 20.9 ms | 16.0 | 0 | 23,008 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 121.9 | 122.3 | 568 ms | 8.2 ms | 1.0 | 0 | 3,659 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 202.9 | 101.8 | 827 ms | 9.8 ms | 2.0 | 0 | 6,088 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 301.2 | 75.6 | 1,181 ms | 13.2 ms | 4.0 | 0 | 9,039 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 388.7 | 48.8 | 1,979 ms | 20.5 ms | 8.0 | 0 | 11,664 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.