Speed test matrix
Qwen3.5-4B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-4B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context46.8K175 ms TTFT
16K context46.2K351 ms TTFT
32K context44.7K722 ms TTFT
64K context40.7K1,581 ms TTFT
128K context32.2K3,998 ms TTFT
251K context22.6K11,159 ms TTFT
256K context22.3K11,555 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 237.590 ms TTFT | 464.691 ms TTFT | 840.9Requests queued897 ms TTFT | 1,615.9192 ms TTFT | 2,730.3212 ms TTFT | 4,250.9280 ms TTFT | 5,983.7514 ms TTFT | 7,441.6851 ms TTFT |
| 16K | 233.6111 ms TTFT | 453.3121 ms TTFT | 846.0175 ms TTFT | 1,494.4241 ms TTFT | 2,409.1261 ms TTFT | 3,528.9493 ms TTFT | 4,548.5723 ms TTFT | 5,297.41,326 ms TTFT |
| 32K | 227.0107 ms TTFT | 431.4146 ms TTFT | 779.0236 ms TTFT | 1,307.2157 ms TTFT | 1,936.0262 ms TTFT | 2,580.3690 ms TTFT | 3,033.21,131 ms TTFT | Over capacity |
| 64K | 213.9173 ms TTFT | 393.0237 ms TTFT | 674.1223 ms TTFT | 1,026.3233 ms TTFT | 1,388.0869 ms TTFT | 1,635.31,301 ms TTFT | Over capacity | Over capacity |
| 128K | 192.6284 ms TTFT | 338.6392 ms TTFT | 527.6329 ms TTFT | 721.3883 ms TTFT | 856.71,583 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 160.2510 ms TTFT | 253.9629 ms TTFT | 357.31,148 ms TTFT | 435.61,899 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 46.8K | — | 175 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.2K | — | 351 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.7K | — | 722 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 40.7K | — | 1,581 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 32.2K | — | 3,998 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 22.6K | — | 11,159 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.3K | — | 11,555 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 237.5 | 238.4 | 90 ms | 4.2 ms | 1.0 | 1 | 7,123 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 464.6 | 233.3 | 91 ms | 4.3 ms | 2.0 | 2 | 13,938 | 30.0 s |
| Sustained decode | 8K | 4 | Requests queued | 840.9 | 222.8 | 897 ms | 4.5 ms | 3.7 | 4 | 25,226 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,615.9 | 203.9 | 192 ms | 4.9 ms | 8.0 | 8 | 48,474 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,730.3 | 172.7 | 212 ms | 5.8 ms | 16.0 | 16 | 81,907 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,250.9 | 135.8 | 280 ms | 7.4 ms | 32.0 | 32 | 127,527 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,983.7 | 94.3 | 514 ms | 10.6 ms | 64.0 | 0 | 179,520 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,441.6 | 58.2 | 851 ms | 17.2 ms | 128.0 | 0 | 223,232 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 233.6 | 234.7 | 111 ms | 4.3 ms | 1.0 | 1 | 7,006 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 453.3 | 227.9 | 121 ms | 4.4 ms | 2.0 | 2 | 13,601 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 846.0 | 213.1 | 175 ms | 4.7 ms | 4.0 | 4 | 25,379 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,494.4 | 188.8 | 241 ms | 5.3 ms | 8.0 | 8 | 44,831 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,409.1 | 152.3 | 261 ms | 6.6 ms | 16.0 | 16 | 72,264 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,528.9 | 110.9 | 493 ms | 9.0 ms | 32.0 | 0 | 105,856 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,548.5 | 71.0 | 723 ms | 14.1 ms | 64.0 | 0 | 136,448 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,297.4 | 41.1 | 1,326 ms | 24.3 ms | 128.0 | 0 | 158,848 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 227.0 | 228.1 | 107 ms | 4.4 ms | 1.0 | 1 | 6,809 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 431.4 | 217.1 | 146 ms | 4.6 ms | 2.0 | 2 | 12,942 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 779.0 | 196.4 | 236 ms | 5.1 ms | 4.0 | 4 | 23,369 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,307.2 | 164.5 | 157 ms | 6.1 ms | 8.0 | 8 | 39,215 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,936.0 | 122.2 | 262 ms | 8.2 ms | 16.0 | 16 | 58,077 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,580.3 | 80.9 | 690 ms | 12.4 ms | 32.0 | 0 | 77,408 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,033.2 | 47.6 | 1,131 ms | 21.0 ms | 64.0 | 0 | 91,010 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 213.9 | 215.6 | 173 ms | 4.6 ms | 1.0 | 1 | 6,421 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 393.0 | 198.4 | 237 ms | 5.0 ms | 2.0 | 2 | 11,789 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 674.1 | 170.2 | 223 ms | 5.9 ms | 4.0 | 4 | 20,219 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,026.3 | 129.3 | 233 ms | 7.7 ms | 8.0 | 8 | 30,784 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,388.0 | 87.0 | 869 ms | 11.5 ms | 16.0 | 0 | 41,632 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,635.3 | 51.3 | 1,301 ms | 19.5 ms | 32.0 | 0 | 49,056 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 192.6 | 194.9 | 284 ms | 5.1 ms | 1.0 | 1 | 5,779 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 338.6 | 173.2 | 392 ms | 5.8 ms | 2.0 | 2 | 10,159 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 527.6 | 134.4 | 329 ms | 7.4 ms | 4.0 | 4 | 15,826 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 721.3 | 90.3 | 883 ms | 11.1 ms | 8.0 | 0 | 21,640 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 856.7 | 53.6 | 1,583 ms | 18.7 ms | 16.0 | 0 | 25,698 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 160.2 | 163.5 | 510 ms | 6.1 ms | 1.0 | 1 | 4,805 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 253.9 | 133.5 | 629 ms | 7.5 ms | 2.0 | 2 | 7,615 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 357.3 | 89.8 | 1,148 ms | 11.1 ms | 4.0 | 0 | 10,716 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 435.6 | 54.7 | 1,899 ms | 18.3 ms | 8.0 | 0 | 13,064 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.