Speed test matrix
Qwen3.5-4B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883396 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context47.1K174 ms TTFT
16K context48.3K336 ms TTFT
32K context45K718 ms TTFT
64K context41.1K1,567 ms TTFT
128K context32.2K4,003 ms TTFT
251K context22.5K11,215 ms TTFT
256K context22.2K11,601 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 237.773 ms TTFT | 464.3119 ms TTFT | 884.7894 ms TTFT | 1,619.2155 ms TTFT | 2,732.1209 ms TTFT | 4,256.2220 ms TTFT | 5,987.6498 ms TTFT | 7,436.0877 ms TTFT |
| 16K | 233.595 ms TTFT | 453.4103 ms TTFT | 846.3181 ms TTFT | 1,495.4201 ms TTFT | 2,406.6260 ms TTFT | 3,526.6436 ms TTFT | 4,551.1719 ms TTFT | 5,285.51,323 ms TTFT |
| 32K | 226.7117 ms TTFT | 431.8115 ms TTFT | 777.8229 ms TTFT | 1,306.0244 ms TTFT | 1,935.7272 ms TTFT | 2,577.3689 ms TTFT | 3,030.51,158 ms TTFT | Over capacity |
| 64K | 214.0173 ms TTFT | 393.4222 ms TTFT | 674.0214 ms TTFT | 1,025.7239 ms TTFT | 1,386.6858 ms TTFT | 1,636.71,263 ms TTFT | Over capacity | Over capacity |
| 128K | 192.4299 ms TTFT | 333.3379 ms TTFT | 515.5328 ms TTFT | 712.6894 ms TTFT | 847.01,505 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 159.9549 ms TTFT | 248.4602 ms TTFT | 352.71,155 ms TTFT | 433.11,887 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 47.1K | — | 174 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 48.3K | — | 336 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 45K | — | 718 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.1K | — | 1,567 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 32.2K | — | 4,003 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 22.5K | — | 11,215 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.2K | — | 11,601 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 237.7 | 238.4 | 73 ms | 4.2 ms | 1.0 | 1 | 7,130 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 464.3 | 233.3 | 119 ms | 4.3 ms | 1.9 | 2 | 13,930 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 884.7 | 222.6 | 894 ms | 4.5 ms | 4.0 | 4 | 26,539 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,619.2 | 203.8 | 155 ms | 4.9 ms | 8.0 | 8 | 48,571 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,732.1 | 172.9 | 209 ms | 5.8 ms | 16.0 | 16 | 81,955 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,256.2 | 134.8 | 220 ms | 7.4 ms | 31.8 | 32 | 127,686 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,987.6 | 94.3 | 498 ms | 10.6 ms | 64.0 | 0 | 179,648 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,436.0 | 58.2 | 877 ms | 17.2 ms | 128.0 | 0 | 223,104 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 233.5 | 234.5 | 95 ms | 4.3 ms | 1.0 | 1 | 7,006 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 453.4 | 228.0 | 103 ms | 4.4 ms | 2.0 | 2 | 13,602 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 846.3 | 213.2 | 181 ms | 4.7 ms | 4.0 | 4 | 25,388 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,495.4 | 188.1 | 201 ms | 5.3 ms | 8.0 | 8 | 44,856 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,406.6 | 152.0 | 260 ms | 6.6 ms | 16.0 | 16 | 72,199 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,526.6 | 110.6 | 436 ms | 9.0 ms | 32.0 | 0 | 105,792 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,551.1 | 71.0 | 719 ms | 14.1 ms | 64.0 | 0 | 136,512 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,285.5 | 41.0 | 1,323 ms | 24.4 ms | 128.0 | 0 | 158,592 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 226.7 | 227.8 | 117 ms | 4.4 ms | 1.0 | 1 | 6,799 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 431.8 | 217.0 | 115 ms | 4.6 ms | 2.0 | 2 | 12,954 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 777.8 | 196.3 | 229 ms | 5.1 ms | 4.0 | 4 | 23,331 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,306.0 | 165.1 | 244 ms | 6.1 ms | 8.0 | 8 | 39,180 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,935.7 | 122.1 | 272 ms | 8.2 ms | 15.7 | 16 | 58,061 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,577.3 | 80.8 | 689 ms | 12.4 ms | 32.0 | 0 | 77,312 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,030.5 | 47.6 | 1,158 ms | 21.0 ms | 64.0 | 0 | 90,880 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 214.0 | 215.6 | 173 ms | 4.6 ms | 1.0 | 1 | 6,420 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 393.4 | 198.4 | 222 ms | 5.0 ms | 2.0 | 2 | 11,806 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 674.0 | 170.0 | 214 ms | 5.9 ms | 4.0 | 4 | 20,218 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,025.7 | 129.3 | 239 ms | 7.7 ms | 8.0 | 8 | 30,768 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,386.6 | 86.9 | 858 ms | 11.5 ms | 16.0 | 0 | 41,601 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,636.7 | 51.3 | 1,263 ms | 19.5 ms | 32.0 | 0 | 49,088 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 192.4 | 195.0 | 299 ms | 5.1 ms | 1.0 | 1 | 5,776 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 333.3 | 169.3 | 379 ms | 5.9 ms | 1.9 | 2 | 9,998 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 515.5 | 130.6 | 328 ms | 7.7 ms | 4.0 | 4 | 15,462 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 712.6 | 89.1 | 894 ms | 11.2 ms | 8.0 | 0 | 21,376 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 847.0 | 53.0 | 1,505 ms | 18.9 ms | 16.0 | 0 | 25,408 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 159.9 | 163.9 | 549 ms | 6.1 ms | 1.0 | 1 | 4,797 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 248.4 | 133.0 | 602 ms | 7.5 ms | 2.0 | 2 | 7,450 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 352.7 | 88.5 | 1,155 ms | 11.3 ms | 4.0 | 0 | 10,584 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 433.1 | 54.4 | 1,887 ms | 18.4 ms | 8.0 | 0 | 12,992 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.