Speed test matrix
Qwen3.5-2B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-2B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.973540 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context87.7K93 ms TTFT
16K context88.9K182 ms TTFT
32K context96K336 ms TTFT
64K context90.8K710 ms TTFT
128K context75.5K1,705 ms TTFT
251K context54.9K4,593 ms TTFT
256K context53.9K4,773 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 399.570 ms TTFT | 790.196 ms TTFT | 1,519.9110 ms TTFT | 2,723.6183 ms TTFT | 5,111.3174 ms TTFT | 8,259.9247 ms TTFT | 12.3K437 ms TTFT | 15.4K640 ms TTFT |
| 16K | 392.394 ms TTFT | 775.3131 ms TTFT | 1,474.4128 ms TTFT | 2,719.6191 ms TTFT | 4,643.2219 ms TTFT | 7,198.3312 ms TTFT | 9,896.9396 ms TTFT | 12K1,339 ms TTFT |
| 32K | 382.5112 ms TTFT | 752.2135 ms TTFT | 1,391.3153 ms TTFT | 2,470.0145 ms TTFT | 3,950.3293 ms TTFT | 5,680.4320 ms TTFT | 7,188.71,252 ms TTFT | 8,095.62,234 ms TTFT |
| 64K | 362.5146 ms TTFT | 708.1163 ms TTFT | 1,251.5211 ms TTFT | 2,083.3187 ms TTFT | 3,061.6247 ms TTFT | 3,872.6625 ms TTFT | 4,516.02,496 ms TTFT | 4,805.54,875 ms TTFT |
| 128K | 326.1268 ms TTFT | 628.3321 ms TTFT | 1,037.4327 ms TTFT | 1,567.2403 ms TTFT | 1,971.6375 ms TTFT | 2,372.82,862 ms TTFT | 2,586.75,494 ms TTFT | Over capacity |
| 251K | 272.8528 ms TTFT | 511.1621 ms TTFT | 769.4648 ms TTFT | 1,016.3674 ms TTFT | 1,227.43,579 ms TTFT | 1,336.96,622 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 87.7K | — | 93 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 88.9K | — | 182 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 96K | — | 336 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 90.8K | — | 710 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 75.5K | — | 1,705 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 54.9K | — | 4,593 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 53.9K | — | 4,773 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 399.5 | 402.6 | 70 ms | 2.5 ms | 1.0 | 3 | 11,984 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 790.1 | 399.1 | 96 ms | 2.5 ms | 2.0 | 6 | 23,704 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,519.9 | 384.9 | 110 ms | 2.6 ms | 4.0 | 12 | 45,597 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,723.6 | 364.8 | 183 ms | 2.7 ms | 7.6 | 16 | 81,705 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,111.3 | 323.8 | 174 ms | 3.1 ms | 16.0 | 32 | 153,337 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 8,259.9 | 263.3 | 247 ms | 3.8 ms | 32.0 | 64 | 247,799 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 12.3K | 196.4 | 437 ms | 5.1 ms | 63.1 | 64 | 370,421 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15.4K | 123.8 | 640 ms | 8.1 ms | 126.1 | 128 | 462,769 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 392.3 | 396.8 | 94 ms | 2.5 ms | 1.0 | 3 | 11,770 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 775.3 | 393.1 | 131 ms | 2.5 ms | 2.0 | 6 | 23,259 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,474.4 | 373.7 | 128 ms | 2.7 ms | 4.0 | 12 | 44,228 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,719.6 | 344.6 | 191 ms | 2.9 ms | 8.0 | 16 | 81,590 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 4,643.2 | 295.0 | 219 ms | 3.4 ms | 16.0 | 32 | 139,297 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,198.3 | 226.7 | 312 ms | 4.4 ms | 31.5 | 32 | 215,951 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 9,896.9 | 155.5 | 396 ms | 6.4 ms | 63.5 | 64 | 296,894 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12K | 93.3 | 1,339 ms | 10.7 ms | 128.0 | 0 | 360,959 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 382.5 | 387.1 | 112 ms | 2.6 ms | 1.0 | 3 | 11,473 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 752.2 | 381.7 | 135 ms | 2.6 ms | 1.9 | 6 | 22,567 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,391.3 | 351.4 | 153 ms | 2.8 ms | 4.0 | 8 | 41,737 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,470.0 | 311.5 | 145 ms | 3.2 ms | 8.0 | 16 | 74,094 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,950.3 | 251.7 | 293 ms | 4.0 ms | 15.5 | 32 | 118,505 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 5,680.4 | 178.9 | 320 ms | 5.6 ms | 31.8 | 32 | 170,404 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,188.7 | 112.5 | 1,252 ms | 8.9 ms | 64.0 | 0 | 215,679 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,095.6 | 63.7 | 2,234 ms | 15.7 ms | 128.0 | 0 | 242,815 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 362.5 | 366.9 | 146 ms | 2.7 ms | 1.0 | 2 | 10,876 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 708.1 | 358.2 | 163 ms | 2.8 ms | 2.0 | 4 | 21,241 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,251.5 | 317.4 | 211 ms | 3.2 ms | 4.0 | 8 | 37,543 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,083.3 | 263.7 | 187 ms | 3.8 ms | 8.0 | 16 | 62,499 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,061.6 | 192.9 | 247 ms | 5.2 ms | 15.9 | 16 | 91,847 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,872.6 | 121.9 | 625 ms | 8.2 ms | 31.9 | 23 | 116,155 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,516.0 | 70.9 | 2,496 ms | 14.1 ms | 64.0 | 0 | 135,489 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,805.5 | 38.0 | 4,875 ms | 26.3 ms | 128.0 | 0 | 144,126 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 326.1 | 332.8 | 268 ms | 3.0 ms | 1.0 | 2 | 9,782 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 628.3 | 321.7 | 321 ms | 3.1 ms | 2.0 | 4 | 18,848 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,037.4 | 265.4 | 327 ms | 3.8 ms | 3.9 | 8 | 31,121 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,567.2 | 198.5 | 403 ms | 5.0 ms | 7.9 | 8 | 47,014 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,971.6 | 125.2 | 375 ms | 8.0 ms | 15.9 | 16 | 59,148 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,372.8 | 74.2 | 2,862 ms | 13.5 ms | 32.0 | 0 | 71,167 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,586.7 | 40.6 | 5,494 ms | 24.6 ms | 64.0 | 0 | 77,568 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 272.8 | 283.1 | 528 ms | 3.5 ms | 1.0 | 2 | 8,185 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 511.1 | 266.9 | 621 ms | 3.7 ms | 1.9 | 4 | 15,332 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 769.4 | 197.6 | 648 ms | 5.1 ms | 3.9 | 4 | 23,083 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,016.3 | 130.5 | 674 ms | 7.7 ms | 7.8 | 8 | 30,490 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,227.4 | 77.4 | 3,579 ms | 12.9 ms | 16.0 | 0 | 36,816 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,336.9 | 41.9 | 6,622 ms | 23.9 ms | 32.0 | 0 | 40,096 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.