Speed test matrix
Qwen3.5-9B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-9B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context29.6K277 ms TTFT
16K context29.7K546 ms TTFT
32K context29K1,115 ms TTFT
64K context26.9K2,392 ms TTFT
128K context22.6K5,700 ms TTFT
251K context16.9K14,894 ms TTFT
256K context16.7K15,367 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 212.664 ms TTFT | 411.7264 ms TTFT | 813.895 ms TTFT | 1,537.8118 ms TTFT | 2,570.4135 ms TTFT | 4,008.0260 ms TTFT | 5,534.21,284 ms TTFT | 6,561.11,511 ms TTFT |
| 16K | 214.6107 ms TTFT | 424.1120 ms TTFT | 738.6139 ms TTFT | 1,456.6150 ms TTFT | 2,364.9166 ms TTFT | 3,564.4795 ms TTFT | 4,598.51,358 ms TTFT | Over capacity |
| 32K | 210.2124 ms TTFT | 392.7136 ms TTFT | 750.3180 ms TTFT | 1,274.5185 ms TTFT | 2,013.5243 ms TTFT | 2,878.7843 ms TTFT | 3,418.11,496 ms TTFT | Over capacity |
| 64K | 198.7194 ms TTFT | 374.9219 ms TTFT | 649.0251 ms TTFT | 1,052.3266 ms TTFT | 1,639.2798 ms TTFT | 2,031.61,406 ms TTFT | Over capacity | Over capacity |
| 128K | 189.9305 ms TTFT | 356.3352 ms TTFT | 559.2342 ms TTFT | 838.9898 ms TTFT | 1,093.21,539 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 172.7606 ms TTFT | 256.8648 ms TTFT | 420.71,132 ms TTFT | 561.41,888 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 29.6K | — | 277 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.7K | — | 546 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29K | — | 1,115 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.9K | — | 2,392 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.6K | — | 5,700 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 16.9K | — | 14,894 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.7K | — | 15,367 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 212.6 | 213.5 | 64 ms | 4.7 ms | 1.0 | 1 | 6,377 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 411.7 | 209.2 | 264 ms | 4.8 ms | 2.0 | 2 | 12,349 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 813.8 | 202.3 | 95 ms | 4.9 ms | 4.0 | 4 | 24,415 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,537.8 | 194.8 | 118 ms | 5.1 ms | 8.0 | 8 | 46,129 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,570.4 | 162.6 | 135 ms | 6.1 ms | 16.0 | 16 | 77,109 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,008.0 | 125.1 | 260 ms | 8.0 ms | 31.9 | 28 | 120,198 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,534.2 | 86.9 | 1,284 ms | 11.5 ms | 64.0 | 0 | 165,955 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,561.1 | 50.7 | 1,511 ms | 19.7 ms | 128.0 | 0 | 196,836 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 214.6 | 215.6 | 107 ms | 4.6 ms | 1.0 | 1 | 6,436 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 424.1 | 213.5 | 120 ms | 4.7 ms | 2.0 | 2 | 12,721 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 738.6 | 196.6 | 139 ms | 5.1 ms | 4.0 | 4 | 22,153 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,456.6 | 183.1 | 150 ms | 5.5 ms | 8.0 | 8 | 43,691 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,364.9 | 147.9 | 166 ms | 6.8 ms | 16.0 | 16 | 70,948 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,564.4 | 111.4 | 795 ms | 9.0 ms | 32.0 | 1 | 106,926 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,598.5 | 71.0 | 1,358 ms | 14.1 ms | 64.0 | 0 | 137,985 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 210.2 | 211.4 | 124 ms | 4.7 ms | 1.0 | 1 | 6,305 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 392.7 | 198.3 | 136 ms | 5.0 ms | 2.0 | 2 | 11,779 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 750.3 | 191.5 | 180 ms | 5.2 ms | 4.0 | 4 | 22,510 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,274.5 | 165.4 | 185 ms | 6.0 ms | 7.9 | 8 | 38,234 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,013.5 | 120.0 | 243 ms | 8.3 ms | 16.0 | 15 | 60,385 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,878.7 | 90.2 | 843 ms | 11.1 ms | 32.0 | 0 | 86,353 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,418.1 | 53.1 | 1,496 ms | 18.8 ms | 64.0 | 0 | 102,558 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 198.7 | 200.6 | 194 ms | 5.0 ms | 1.0 | 1 | 5,960 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 374.9 | 191.6 | 219 ms | 5.2 ms | 2.0 | 2 | 11,247 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 649.0 | 172.8 | 251 ms | 5.8 ms | 3.9 | 4 | 19,466 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,052.3 | 139.9 | 266 ms | 7.1 ms | 7.9 | 7 | 31,566 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,639.2 | 101.2 | 798 ms | 9.9 ms | 16.0 | 0 | 49,156 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,031.6 | 62.8 | 1,406 ms | 15.9 ms | 32.0 | 0 | 60,935 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 189.9 | 191.9 | 305 ms | 5.2 ms | 1.0 | 1 | 5,698 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 356.3 | 176.8 | 352 ms | 5.7 ms | 2.0 | 2 | 10,689 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 559.2 | 144.9 | 342 ms | 6.9 ms | 4.0 | 4 | 16,775 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 838.9 | 105.3 | 898 ms | 9.5 ms | 8.0 | 0 | 25,156 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,093.2 | 67.2 | 1,539 ms | 14.9 ms | 16.0 | 0 | 32,795 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 172.7 | 172.4 | 606 ms | 5.8 ms | 1.0 | 1 | 5,181 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 256.8 | 141.9 | 648 ms | 7.0 ms | 2.0 | 1 | 7,704 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 420.7 | 106.9 | 1,132 ms | 9.4 ms | 4.0 | 0 | 12,618 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 561.4 | 70.3 | 1,888 ms | 14.2 ms | 8.0 | 0 | 16,834 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.