Speed test matrix
Qwen3.5-2B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-2B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883122 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context74.6K110 ms TTFT
16K context80.3K202 ms TTFT
32K context90.1K359 ms TTFT
64K context83.6K770 ms TTFT
128K context69K1,866 ms TTFT
251K context49.5K5,097 ms TTFT
256K context48.6K5,290 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 455.363 ms TTFT | 859.175 ms TTFT | 1,592.773 ms TTFT | 3,048.775 ms TTFT | 5,240.7112 ms TTFT | 8,418.3103 ms TTFT | 11.8K200 ms TTFT | 15.2K732 ms TTFT |
| 16K | 414.991 ms TTFT | 825.997 ms TTFT | 1,591.498 ms TTFT | 2,905.2103 ms TTFT | 4,941.5120 ms TTFT | 7,666.6123 ms TTFT | 10.3K214 ms TTFT | 12.3K1,288 ms TTFT |
| 32K | 439.778 ms TTFT | 812.9129 ms TTFT | 1,512.4135 ms TTFT | 2,715.4133 ms TTFT | 4,426.0142 ms TTFT | 6,411.8204 ms TTFT | 8,059.8739 ms TTFT | 8,915.62,274 ms TTFT |
| 64K | 409.6145 ms TTFT | 781.3202 ms TTFT | 1,471.9187 ms TTFT | 2,525.4189 ms TTFT | 3,705.8221 ms TTFT | 4,643.5370 ms TTFT | 5,629.12,696 ms TTFT | Over capacity |
| 128K | 389.7273 ms TTFT | 731.4327 ms TTFT | 1,291.2322 ms TTFT | 1,917.3339 ms TTFT | 2,577.4412 ms TTFT | 3,107.62,827 ms TTFT | Over capacity | Over capacity |
| 251K | 309.1527 ms TTFT | 676.0668 ms TTFT | 949.8670 ms TTFT | 1,439.4689 ms TTFT | 1,718.42,607 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 74.6K | — | 110 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 80.3K | — | 202 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 90.1K | — | 359 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 83.6K | — | 770 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 69K | — | 1,866 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 49.5K | — | 5,097 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 48.6K | — | 5,290 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 455.3 | 440.6 | 63 ms | 2.3 ms | 1.0 | 3 | 13,659 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 859.1 | 398.1 | 75 ms | 2.5 ms | 2.0 | 6 | 25,771 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,592.7 | 399.7 | 73 ms | 2.5 ms | 4.0 | 12 | 47,777 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,048.7 | 371.0 | 75 ms | 2.7 ms | 7.9 | 23 | 91,455 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,240.7 | 323.9 | 112 ms | 3.1 ms | 15.9 | 33 | 157,229 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 8,418.3 | 259.2 | 103 ms | 3.9 ms | 31.8 | 64 | 252,547 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 11.8K | 180.9 | 200 ms | 5.5 ms | 63.8 | 64 | 354,459 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15.2K | 115.5 | 732 ms | 8.7 ms | 127.8 | 39 | 455,580 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 414.9 | 414.8 | 91 ms | 2.4 ms | 1.0 | 3 | 12,447 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 825.9 | 424.5 | 97 ms | 2.4 ms | 2.0 | 6 | 24,776 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,591.4 | 377.2 | 98 ms | 2.7 ms | 4.0 | 12 | 47,742 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,905.2 | 363.0 | 103 ms | 2.8 ms | 7.8 | 22 | 87,153 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 4,941.5 | 303.5 | 120 ms | 3.3 ms | 15.9 | 32 | 148,239 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,666.6 | 230.7 | 123 ms | 4.3 ms | 32.0 | 44 | 229,963 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 10.3K | 155.9 | 214 ms | 6.4 ms | 63.9 | 64 | 308,102 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12.3K | 94.0 | 1,288 ms | 10.6 ms | 128.0 | 6 | 370,457 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 439.7 | 431.7 | 78 ms | 2.3 ms | 1.0 | 3 | 13,190 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 812.9 | 405.3 | 129 ms | 2.5 ms | 2.0 | 6 | 24,387 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,512.4 | 390.4 | 135 ms | 2.6 ms | 3.9 | 12 | 45,370 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,715.4 | 340.1 | 133 ms | 2.9 ms | 8.0 | 17 | 81,464 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,426.0 | 266.7 | 142 ms | 3.7 ms | 15.9 | 32 | 132,755 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 6,411.8 | 195.4 | 204 ms | 5.1 ms | 32.0 | 32 | 192,338 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 8,059.8 | 120.1 | 739 ms | 8.3 ms | 63.8 | 41 | 241,777 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,915.6 | 67.6 | 2,274 ms | 14.8 ms | 128.0 | 1 | 267,393 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 409.6 | 405.2 | 145 ms | 2.5 ms | 1.0 | 3 | 12,288 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 781.3 | 392.0 | 202 ms | 2.6 ms | 2.0 | 6 | 23,440 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,471.9 | 352.8 | 187 ms | 2.8 ms | 4.0 | 12 | 44,153 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,525.4 | 306.1 | 189 ms | 3.3 ms | 7.9 | 16 | 75,761 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,705.8 | 225.7 | 221 ms | 4.4 ms | 15.9 | 19 | 111,157 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,643.5 | 141.2 | 370 ms | 7.1 ms | 31.8 | 32 | 139,310 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,629.1 | 86.6 | 2,696 ms | 11.5 ms | 64.0 | 2 | 168,863 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 389.7 | 407.2 | 273 ms | 2.5 ms | 1.0 | 3 | 11,691 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 731.4 | 350.0 | 327 ms | 2.9 ms | 1.9 | 5 | 21,940 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,291.2 | 321.8 | 322 ms | 3.1 ms | 4.0 | 8 | 38,731 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,917.3 | 232.0 | 339 ms | 4.3 ms | 7.9 | 12 | 57,515 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,577.4 | 152.3 | 412 ms | 6.6 ms | 15.9 | 16 | 77,310 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 3,107.6 | 97.5 | 2,827 ms | 10.3 ms | 32.0 | 4 | 93,193 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 309.1 | 327.2 | 527 ms | 3.1 ms | 1.0 | 2 | 9,272 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 676.0 | 348.1 | 668 ms | 2.9 ms | 1.9 | 4 | 20,281 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 949.8 | 241.5 | 670 ms | 4.1 ms | 3.9 | 6 | 28,495 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,439.4 | 170.2 | 689 ms | 5.9 ms | 7.9 | 8 | 43,172 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,718.4 | 102.2 | 2,607 ms | 9.8 ms | 15.9 | 7 | 51,543 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.