Speed test matrix
Qwen3.5-4B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883870 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context45.5K180 ms TTFT
16K context44.2K367 ms TTFT
32K context43.5K743 ms TTFT
64K context38.9K1,657 ms TTFT
128K context30.3K4,242 ms TTFT
251K context21.1K11,972 ms TTFT
256K context20.7K12,406 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 349.070 ms TTFT | 672.5132 ms TTFT | 1,259.289 ms TTFT | 2,261.387 ms TTFT | 3,748.7141 ms TTFT | 5,685.0188 ms TTFT | 7,309.8696 ms TTFT | 8,578.71,225 ms TTFT |
| 16K | 342.199 ms TTFT | 649.5102 ms TTFT | 1,214.3115 ms TTFT | 2,118.2140 ms TTFT | 3,362.3200 ms TTFT | 4,812.9217 ms TTFT | 5,973.4913 ms TTFT | 6,704.91,904 ms TTFT |
| 32K | 330.6113 ms TTFT | 625.3124 ms TTFT | 1,125.2132 ms TTFT | 1,880.5156 ms TTFT | 2,857.1206 ms TTFT | 3,698.0739 ms TTFT | 4,299.61,401 ms TTFT | Over capacity |
| 64K | 322.7183 ms TTFT | 583.7196 ms TTFT | 1,003.6200 ms TTFT | 1,547.4284 ms TTFT | 2,069.0376 ms TTFT | 2,477.91,561 ms TTFT | Over capacity | Over capacity |
| 128K | 288.8293 ms TTFT | 498.6324 ms TTFT | 795.2379 ms TTFT | 1,060.7513 ms TTFT | 1,318.31,818 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 245.2568 ms TTFT | 384.8829 ms TTFT | 537.2714 ms TTFT | 682.52,209 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 45.5K | — | 180 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 44.2K | — | 367 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 43.5K | — | 743 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 38.9K | — | 1,657 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 30.3K | — | 4,242 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 21.1K | — | 11,972 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 20.7K | — | 12,406 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 349.0 | 353.5 | 70 ms | 2.8 ms | 1.0 | 2 | 10,469 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 672.5 | 336.4 | 132 ms | 3.0 ms | 2.0 | 4 | 20,171 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,259.2 | 318.7 | 89 ms | 3.1 ms | 4.0 | 8 | 37,774 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,261.3 | 286.2 | 87 ms | 3.5 ms | 8.0 | 16 | 67,831 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,748.7 | 236.9 | 141 ms | 4.2 ms | 16.0 | 20 | 112,451 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,685.0 | 181.0 | 188 ms | 5.5 ms | 31.9 | 32 | 170,572 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 7,309.8 | 116.1 | 696 ms | 8.6 ms | 64.0 | 4 | 219,186 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 8,578.7 | 68.1 | 1,225 ms | 14.7 ms | 128.0 | 0 | 257,345 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 342.1 | 343.3 | 99 ms | 2.9 ms | 1.0 | 2 | 10,263 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 649.5 | 326.6 | 102 ms | 3.1 ms | 2.0 | 4 | 19,480 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,214.3 | 305.1 | 115 ms | 3.3 ms | 4.0 | 8 | 36,426 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,118.2 | 269.8 | 140 ms | 3.7 ms | 7.9 | 16 | 63,536 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,362.3 | 213.1 | 200 ms | 4.7 ms | 16.0 | 16 | 100,849 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,812.9 | 154.3 | 217 ms | 6.5 ms | 31.9 | 32 | 144,358 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,973.4 | 93.5 | 913 ms | 10.7 ms | 64.0 | 0 | 179,133 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,704.9 | 52.0 | 1,904 ms | 19.2 ms | 128.0 | 0 | 201,178 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 330.6 | 333.6 | 113 ms | 3.0 ms | 1.0 | 2 | 9,919 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 625.3 | 316.2 | 124 ms | 3.2 ms | 2.0 | 4 | 18,758 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,125.2 | 285.5 | 132 ms | 3.5 ms | 4.0 | 8 | 33,753 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,880.5 | 237.6 | 156 ms | 4.2 ms | 7.9 | 11 | 56,398 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,857.1 | 181.7 | 206 ms | 5.5 ms | 15.9 | 16 | 85,714 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,698.0 | 117.2 | 739 ms | 8.5 ms | 32.0 | 1 | 110,905 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,299.6 | 67.9 | 1,401 ms | 14.7 ms | 64.0 | 0 | 128,966 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 322.7 | 325.1 | 183 ms | 3.1 ms | 1.0 | 2 | 9,680 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 583.7 | 295.5 | 196 ms | 3.4 ms | 2.0 | 4 | 17,508 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,003.6 | 257.7 | 200 ms | 3.9 ms | 4.0 | 8 | 30,105 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,547.4 | 195.7 | 284 ms | 5.1 ms | 8.0 | 8 | 46,419 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,069.0 | 132.3 | 376 ms | 7.6 ms | 15.9 | 16 | 62,066 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,477.9 | 78.4 | 1,561 ms | 12.8 ms | 32.0 | 0 | 74,292 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 288.8 | 296.9 | 293 ms | 3.4 ms | 1.0 | 2 | 8,662 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 498.6 | 258.2 | 324 ms | 3.9 ms | 1.9 | 4 | 14,958 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 795.2 | 203.3 | 379 ms | 4.9 ms | 3.9 | 4 | 23,852 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,060.7 | 138.3 | 513 ms | 7.2 ms | 7.9 | 8 | 31,817 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,318.3 | 84.3 | 1,818 ms | 11.9 ms | 16.0 | 0 | 39,523 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 245.2 | 251.9 | 568 ms | 4.0 ms | 1.0 | 1 | 7,359 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 384.8 | 199.6 | 829 ms | 5.0 ms | 1.9 | 2 | 11,544 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 537.2 | 140.6 | 714 ms | 7.1 ms | 3.9 | 4 | 16,111 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 682.5 | 87.3 | 2,209 ms | 11.5 ms | 8.0 | 0 | 20,466 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.