Speed test matrix
Qwen3.5-9B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.884269 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context31.7K258 ms TTFT
16K context30.7K529 ms TTFT
32K context30.1K1,071 ms TTFT
64K context28.4K2,269 ms TTFT
128K context23.9K5,394 ms TTFT
251K context18.2K13,865 ms TTFT
256K context18K14,330 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 160.488 ms TTFT | 316.7122 ms TTFT | 612.2901 ms TTFT | 1,144.6177 ms TTFT | 2,079.9250 ms TTFT | 3,446.8423 ms TTFT | 5,091.6625 ms TTFT | 6,571.61,081 ms TTFT |
| 16K | 158.5104 ms TTFT | 311.4149 ms TTFT | 592.6203 ms TTFT | 1,080.8211 ms TTFT | 1,899.4334 ms TTFT | 2,945.4462 ms TTFT | 3,977.6776 ms TTFT | 4,805.41,430 ms TTFT |
| 32K | 155.1116 ms TTFT | 300.8177 ms TTFT | 557.2239 ms TTFT | 974.9239 ms TTFT | 1,600.8435 ms TTFT | 2,240.7728 ms TTFT | 2,762.31,214 ms TTFT | Over capacity |
| 64K | 148.8176 ms TTFT | 281.5237 ms TTFT | 500.8260 ms TTFT | 821.1535 ms TTFT | 1,187.3889 ms TTFT | 1,489.91,459 ms TTFT | Over capacity | Over capacity |
| 128K | 138.1292 ms TTFT | 248.4374 ms TTFT | 415.5556 ms TTFT | 603.1899 ms TTFT | 766.31,499 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 122.2530 ms TTFT | 203.0771 ms TTFT | 301.71,125 ms TTFT | 388.71,882 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.7K | — | 258 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.7K | — | 529 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.1K | — | 1,071 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.4K | — | 2,269 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.9K | — | 5,394 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.2K | — | 13,865 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18K | — | 14,330 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 160.4 | 161.1 | 88 ms | 6.2 ms | 1.0 | 1 | 4,811 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 316.7 | 159.3 | 122 ms | 6.3 ms | 1.9 | 2 | 9,500 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 612.2 | 154.2 | 901 ms | 6.5 ms | 4.0 | 4 | 18,364 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,144.6 | 144.7 | 177 ms | 6.9 ms | 8.0 | 8 | 34,334 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,079.9 | 131.5 | 250 ms | 7.6 ms | 16.0 | 16 | 62,393 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,446.8 | 108.4 | 423 ms | 9.2 ms | 32.0 | 0 | 103,392 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,091.6 | 79.9 | 625 ms | 12.5 ms | 64.0 | 0 | 152,768 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,571.6 | 51.2 | 1,081 ms | 19.5 ms | 128.0 | 0 | 197,120 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 158.5 | 159.4 | 104 ms | 6.3 ms | 1.0 | 1 | 4,758 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 311.4 | 156.9 | 149 ms | 6.4 ms | 2.0 | 2 | 9,346 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 592.6 | 149.7 | 203 ms | 6.7 ms | 4.0 | 4 | 17,776 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,080.8 | 136.5 | 211 ms | 7.3 ms | 8.0 | 8 | 32,423 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,899.4 | 119.1 | 334 ms | 8.4 ms | 16.0 | 0 | 56,976 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,945.4 | 92.1 | 462 ms | 10.9 ms | 32.0 | 0 | 88,352 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,977.6 | 61.9 | 776 ms | 16.2 ms | 64.0 | 0 | 119,296 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,805.4 | 37.0 | 1,430 ms | 27.1 ms | 128.0 | 0 | 144,128 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 155.1 | 156.2 | 116 ms | 6.4 ms | 1.0 | 1 | 4,652 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 300.8 | 151.8 | 177 ms | 6.6 ms | 2.0 | 2 | 9,022 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 557.2 | 141.1 | 239 ms | 7.1 ms | 4.0 | 4 | 16,714 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 974.9 | 123.1 | 239 ms | 8.1 ms | 8.0 | 8 | 29,258 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,600.8 | 100.3 | 435 ms | 10.0 ms | 16.0 | 0 | 48,016 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,240.7 | 70.2 | 728 ms | 14.2 ms | 32.0 | 0 | 67,200 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,762.3 | 43.3 | 1,214 ms | 23.1 ms | 64.0 | 0 | 82,816 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 148.8 | 150.3 | 176 ms | 6.7 ms | 1.0 | 1 | 4,464 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 281.5 | 142.4 | 237 ms | 7.0 ms | 2.0 | 2 | 8,449 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 500.8 | 126.8 | 260 ms | 7.9 ms | 4.0 | 4 | 15,022 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 821.1 | 102.8 | 535 ms | 9.7 ms | 8.0 | 0 | 24,632 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,187.3 | 74.4 | 889 ms | 13.4 ms | 16.0 | 0 | 35,616 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,489.9 | 46.7 | 1,459 ms | 21.4 ms | 32.0 | 0 | 44,672 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 138.1 | 140.8 | 292 ms | 7.1 ms | 1.0 | 1 | 4,143 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 248.4 | 136.4 | 374 ms | 7.4 ms | 1.9 | 2 | 7,452 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 415.5 | 104.0 | 556 ms | 9.6 ms | 4.0 | 0 | 12,465 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 603.1 | 75.4 | 899 ms | 13.3 ms | 8.0 | 0 | 18,088 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 766.3 | 47.9 | 1,499 ms | 20.9 ms | 16.0 | 0 | 22,976 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 122.2 | 122.6 | 530 ms | 8.2 ms | 1.0 | 0 | 3,668 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 203.0 | 101.9 | 771 ms | 9.8 ms | 2.0 | 0 | 6,090 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 301.7 | 75.7 | 1,125 ms | 13.2 ms | 4.0 | 0 | 9,047 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 388.7 | 48.8 | 1,882 ms | 20.5 ms | 8.0 | 0 | 11,656 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.