Speed test matrix
Qwen3.5-4B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-4B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.946901 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context46.6K176 ms TTFT
16K context46.5K349 ms TTFT
32K context44.9K720 ms TTFT
64K context39.6K1,628 ms TTFT
128K context30.3K4,244 ms TTFT
251K context21K12,015 ms TTFT
256K context20.7K12,438 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 263.066 ms TTFT | 516.986 ms TTFT | 946.483 ms TTFT | 1,753.4127 ms TTFT | 2,852.5132 ms TTFT | 4,419.4189 ms TTFT | 5,932.2712 ms TTFT | 7,196.31,126 ms TTFT |
| 16K | 287.5102 ms TTFT | 578.693 ms TTFT | 948.5105 ms TTFT | 1,674.8145 ms TTFT | 2,669.5174 ms TTFT | 3,754.0454 ms TTFT | 4,823.51,010 ms TTFT | 5,627.91,928 ms TTFT |
| 32K | 248.0113 ms TTFT | 494.9119 ms TTFT | 897.2145 ms TTFT | 1,498.4143 ms TTFT | 2,177.0181 ms TTFT | 2,969.9809 ms TTFT | 3,587.61,358 ms TTFT | Over capacity |
| 64K | 233.5179 ms TTFT | 450.8189 ms TTFT | 782.2248 ms TTFT | 1,209.1239 ms TTFT | 1,687.7815 ms TTFT | 2,055.81,464 ms TTFT | Over capacity | Over capacity |
| 128K | 227.6314 ms TTFT | 407.8332 ms TTFT | 548.6355 ms TTFT | 858.9860 ms TTFT | 1,072.81,559 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 184.8614 ms TTFT | 265.8614 ms TTFT | 384.51,177 ms TTFT | 562.31,882 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 46.6K | — | 176 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.5K | — | 349 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.9K | — | 720 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 39.6K | — | 1,628 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 30.3K | — | 4,244 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 21K | — | 12,015 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 20.7K | — | 12,438 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 263.0 | 283.8 | 66 ms | 3.5 ms | 1.0 | 2 | 7,890 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 516.9 | 275.4 | 86 ms | 3.6 ms | 2.0 | 4 | 15,508 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 946.4 | 255.5 | 83 ms | 3.9 ms | 4.0 | 7 | 28,389 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,753.4 | 223.8 | 127 ms | 4.5 ms | 8.0 | 8 | 52,597 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,852.5 | 188.0 | 132 ms | 5.3 ms | 15.9 | 16 | 85,582 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,419.4 | 142.9 | 189 ms | 7.0 ms | 31.9 | 31 | 132,580 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,932.2 | 95.7 | 712 ms | 10.4 ms | 64.0 | 0 | 177,994 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,196.3 | 56.4 | 1,126 ms | 17.7 ms | 128.0 | 0 | 215,956 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 287.5 | 291.8 | 102 ms | 3.4 ms | 1.0 | 2 | 8,625 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 578.6 | 280.2 | 93 ms | 3.6 ms | 2.0 | 4 | 17,355 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 948.5 | 246.4 | 105 ms | 4.1 ms | 4.0 | 6 | 28,454 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,674.8 | 215.6 | 145 ms | 4.6 ms | 8.0 | 8 | 50,234 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,669.5 | 173.5 | 174 ms | 5.8 ms | 15.9 | 16 | 80,069 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,754.0 | 116.9 | 454 ms | 8.6 ms | 31.9 | 14 | 112,581 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,823.5 | 76.2 | 1,010 ms | 13.1 ms | 64.0 | 0 | 144,676 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,627.9 | 43.3 | 1,928 ms | 23.1 ms | 128.0 | 0 | 168,709 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 248.0 | 255.1 | 113 ms | 3.9 ms | 1.0 | 2 | 7,439 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 494.9 | 258.8 | 119 ms | 3.9 ms | 2.0 | 4 | 14,846 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 897.2 | 233.4 | 145 ms | 4.3 ms | 4.0 | 4 | 26,915 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,498.4 | 200.9 | 143 ms | 5.0 ms | 8.0 | 8 | 44,952 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,177.0 | 137.2 | 181 ms | 7.3 ms | 15.8 | 16 | 65,296 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,969.9 | 94.3 | 809 ms | 10.6 ms | 32.0 | 0 | 89,068 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,587.6 | 56.7 | 1,358 ms | 17.6 ms | 64.0 | 0 | 107,577 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 233.5 | 237.6 | 179 ms | 4.2 ms | 1.0 | 1 | 7,003 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 450.8 | 232.5 | 189 ms | 4.3 ms | 2.0 | 2 | 13,525 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 782.2 | 208.4 | 248 ms | 4.8 ms | 4.0 | 4 | 23,464 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,209.1 | 156.1 | 239 ms | 6.4 ms | 8.0 | 8 | 36,265 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,687.7 | 106.1 | 815 ms | 9.4 ms | 16.0 | 1 | 50,610 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,055.8 | 65.4 | 1,464 ms | 15.3 ms | 32.0 | 0 | 61,637 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 227.6 | 233.0 | 314 ms | 4.3 ms | 1.0 | 1 | 6,827 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 407.8 | 209.0 | 332 ms | 4.8 ms | 2.0 | 2 | 12,233 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 548.6 | 147.8 | 355 ms | 6.8 ms | 4.0 | 3 | 16,455 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 858.9 | 108.9 | 860 ms | 9.2 ms | 8.0 | 0 | 25,764 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,072.8 | 69.8 | 1,559 ms | 14.3 ms | 16.0 | 0 | 32,167 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 184.8 | 194.3 | 614 ms | 5.1 ms | 1.0 | 1 | 5,544 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 265.8 | 153.4 | 614 ms | 6.6 ms | 2.0 | 2 | 7,974 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 384.5 | 98.8 | 1,177 ms | 10.1 ms | 4.0 | 0 | 11,533 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 562.3 | 71.7 | 1,882 ms | 13.9 ms | 8.0 | 0 | 16,858 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.