Speed test matrix
Qwen3.5-2B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-2B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.946376 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context75.7K108 ms TTFT
16K context83K195 ms TTFT
32K context87K371 ms TTFT
64K context83.6K771 ms TTFT
128K context68.8K1,871 ms TTFT
251K context49.5K5,099 ms TTFT
256K context48.3K5,327 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 314.460 ms TTFT | 597.671 ms TTFT | 1,225.579 ms TTFT | 2,273.080 ms TTFT | 3,769.098 ms TTFT | 6,263.3146 ms TTFT | 8,677.0279 ms TTFT | 11.5K821 ms TTFT |
| 16K | 301.887 ms TTFT | 586.495 ms TTFT | 1,172.499 ms TTFT | 2,026.0106 ms TTFT | 3,656.8115 ms TTFT | 5,605.1161 ms TTFT | 7,506.5527 ms TTFT | 9,533.71,241 ms TTFT |
| 32K | 270.9107 ms TTFT | 582.5121 ms TTFT | 1,111.1127 ms TTFT | 1,966.3123 ms TTFT | 3,216.3188 ms TTFT | 4,519.8215 ms TTFT | 6,051.31,102 ms TTFT | 7,097.22,209 ms TTFT |
| 64K | 287.7152 ms TTFT | 532.3186 ms TTFT | 1,017.2195 ms TTFT | 1,704.4199 ms TTFT | 2,570.9270 ms TTFT | 3,534.3972 ms TTFT | 4,056.42,646 ms TTFT | Over capacity |
| 128K | 270.7266 ms TTFT | 504.6304 ms TTFT | 887.4297 ms TTFT | 1,427.8344 ms TTFT | 1,723.61,031 ms TTFT | 2,225.52,687 ms TTFT | Over capacity | Over capacity |
| 251K | 289.7547 ms TTFT | 468.1577 ms TTFT | 717.8590 ms TTFT | 972.1972 ms TTFT | 1,205.03,106 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 75.7K | — | 108 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 83K | — | 195 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 87K | — | 371 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 83.6K | — | 771 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 68.8K | — | 1,871 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 49.5K | — | 5,099 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 48.3K | — | 5,327 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 314.4 | 336.6 | 60 ms | 3.0 ms | 1.0 | 2 | 9,432 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 597.6 | 307.9 | 71 ms | 3.3 ms | 2.0 | 4 | 17,925 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,225.5 | 301.9 | 79 ms | 3.3 ms | 4.0 | 8 | 36,763 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,273.0 | 296.5 | 80 ms | 3.4 ms | 7.9 | 16 | 68,185 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,769.0 | 245.4 | 98 ms | 4.1 ms | 15.9 | 25 | 113,058 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 6,263.3 | 204.8 | 146 ms | 4.9 ms | 31.9 | 32 | 187,876 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 8,677.0 | 146.1 | 279 ms | 6.8 ms | 63.9 | 58 | 260,312 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 11.5K | 90.8 | 821 ms | 11.0 ms | 128.0 | 1 | 344,190 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 301.8 | 308.3 | 87 ms | 3.2 ms | 1.0 | 2 | 9,058 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 586.4 | 297.5 | 95 ms | 3.4 ms | 2.0 | 4 | 17,591 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,172.4 | 294.7 | 99 ms | 3.4 ms | 4.0 | 8 | 35,169 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,026.0 | 269.7 | 106 ms | 3.7 ms | 8.0 | 15 | 60,781 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,656.8 | 232.9 | 115 ms | 4.3 ms | 16.0 | 22 | 109,704 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,605.1 | 181.0 | 161 ms | 5.5 ms | 31.9 | 32 | 168,153 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,506.5 | 122.9 | 527 ms | 8.1 ms | 63.9 | 30 | 225,173 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 9,533.7 | 75.6 | 1,241 ms | 13.2 ms | 128.0 | 0 | 285,939 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 270.9 | 277.1 | 107 ms | 3.6 ms | 1.0 | 2 | 8,128 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 582.5 | 307.0 | 121 ms | 3.3 ms | 1.9 | 4 | 17,475 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,111.1 | 287.0 | 127 ms | 3.5 ms | 3.9 | 8 | 33,332 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,966.3 | 266.6 | 123 ms | 3.8 ms | 7.8 | 14 | 58,986 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,216.3 | 204.3 | 188 ms | 4.9 ms | 16.0 | 16 | 96,485 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,519.8 | 150.6 | 215 ms | 6.6 ms | 31.8 | 32 | 135,613 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,051.3 | 94.3 | 1,102 ms | 10.6 ms | 64.0 | 5 | 181,555 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 7,097.2 | 56.8 | 2,209 ms | 17.6 ms | 128.0 | 0 | 212,811 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 287.7 | 301.1 | 152 ms | 3.3 ms | 1.0 | 2 | 8,634 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 532.3 | 281.8 | 186 ms | 3.6 ms | 2.0 | 4 | 15,966 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,017.2 | 276.7 | 195 ms | 3.6 ms | 3.9 | 7 | 30,514 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,704.4 | 214.6 | 199 ms | 4.7 ms | 8.0 | 9 | 51,132 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,570.9 | 163.5 | 270 ms | 6.1 ms | 15.9 | 16 | 77,133 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,534.3 | 116.0 | 972 ms | 8.6 ms | 31.8 | 11 | 106,031 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,056.4 | 65.3 | 2,646 ms | 15.3 ms | 64.0 | 0 | 121,714 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 270.7 | 280.4 | 266 ms | 3.6 ms | 1.0 | 2 | 8,122 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 504.6 | 289.4 | 304 ms | 3.5 ms | 2.0 | 4 | 15,137 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 887.4 | 235.0 | 297 ms | 4.3 ms | 4.0 | 5 | 26,618 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,427.8 | 189.6 | 344 ms | 5.3 ms | 7.9 | 8 | 42,829 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,723.6 | 117.5 | 1,031 ms | 8.5 ms | 16.0 | 6 | 51,705 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,225.5 | 71.2 | 2,687 ms | 14.0 ms | 32.0 | 0 | 66,745 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 289.7 | 272.7 | 547 ms | 3.7 ms | 0.9 | 2 | 8,690 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 468.1 | 263.3 | 577 ms | 3.8 ms | 1.9 | 3 | 14,041 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 717.8 | 187.1 | 590 ms | 5.3 ms | 3.9 | 4 | 21,531 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 972.1 | 136.6 | 972 ms | 7.3 ms | 7.8 | 7 | 29,161 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,205.0 | 78.0 | 3,106 ms | 12.8 ms | 16.0 | 1 | 36,135 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.