Speed test matrix
Qwen3.5-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.884227 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context10.3K794 ms TTFT
16K context9,9821,625 ms TTFT
32K context9,5133,394 ms TTFT
64K context8,7067,399 ms TTFT
128K context7,33317,550 ms TTFT
251K context5,65044,637 ms TTFT
256K context5,62545,729 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 116.8128 ms TTFT | 230.7596 ms TTFT | 446.6474 ms TTFT | 848.1784 ms TTFT | 1,487.91,652 ms TTFT | 2,355.22,891 ms TTFT | 3,103.94,184 ms TTFT | Over capacity |
| 16K | 116.2173 ms TTFT | 228.8274 ms TTFT | 441.9485 ms TTFT | 812.2966 ms TTFT | 1,390.41,615 ms TTFT | 2,106.82,683 ms TTFT | Over capacity | Over capacity |
| 32K | 116.6214 ms TTFT | 225.8356 ms TTFT | 420.3613 ms TTFT | 758.91,201 ms TTFT | 1,233.21,936 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 110.9297 ms TTFT | 215.3461 ms TTFT | 390.9788 ms TTFT | 666.91,393 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 105.2534 ms TTFT | 195.1714 ms TTFT | 342.41,344 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 99.4823 ms TTFT | 166.81,042 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.3K | — | 794 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,982.0 | — | 1,625 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,513.0 | — | 3,394 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,706.0 | — | 7,399 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,333.0 | — | 17,550 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,650.0 | — | 44,637 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,625.0 | — | 45,729 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 116.8 | 117.7 | 128 ms | 8.5 ms | 1.0 | 0 | 3,504 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 230.7 | 115.7 | 596 ms | 8.6 ms | 2.0 | 0 | 6,921 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 446.6 | 112.6 | 474 ms | 8.9 ms | 4.0 | 0 | 13,397 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 848.1 | 105.9 | 784 ms | 9.4 ms | 8.0 | 0 | 25,441 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,487.9 | 93.4 | 1,652 ms | 10.7 ms | 16.0 | 0 | 44,602 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,355.2 | 73.4 | 2,891 ms | 13.6 ms | 32.0 | 0 | 70,630 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,103.9 | 46.3 | 4,184 ms | 21.6 ms | 64.0 | 0 | 93,111 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 116.2 | 117.2 | 173 ms | 8.5 ms | 1.0 | 0 | 3,486 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 228.8 | 115.1 | 274 ms | 8.7 ms | 2.0 | 0 | 6,862 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 441.9 | 110.7 | 485 ms | 9.0 ms | 4.0 | 0 | 13,255 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 812.2 | 102.0 | 966 ms | 9.8 ms | 8.0 | 0 | 24,352 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,390.4 | 87.2 | 1,615 ms | 11.5 ms | 16.0 | 0 | 41,700 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,106.8 | 65.0 | 2,683 ms | 15.4 ms | 32.0 | 0 | 63,144 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 116.6 | 117.1 | 214 ms | 8.5 ms | 1.0 | 0 | 3,498 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 225.8 | 113.5 | 356 ms | 8.8 ms | 2.0 | 0 | 6,773 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 420.3 | 105.4 | 613 ms | 9.5 ms | 4.0 | 0 | 12,600 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 758.9 | 95.4 | 1,201 ms | 10.5 ms | 8.0 | 0 | 22,763 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,233.2 | 77.3 | 1,936 ms | 12.9 ms | 16.0 | 0 | 36,980 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 110.9 | 111.5 | 297 ms | 9.0 ms | 1.0 | 0 | 3,328 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 215.3 | 107.9 | 461 ms | 9.3 ms | 2.0 | 0 | 6,458 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 390.9 | 97.8 | 788 ms | 10.2 ms | 4.0 | 0 | 11,730 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 666.9 | 83.3 | 1,393 ms | 12.0 ms | 8.0 | 0 | 20,000 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 105.2 | 106.4 | 534 ms | 9.4 ms | 1.0 | 0 | 3,156 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 195.1 | 97.6 | 714 ms | 10.2 ms | 2.0 | 0 | 5,854 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 342.4 | 86.1 | 1,344 ms | 11.6 ms | 4.0 | 0 | 10,266 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 99.4 | 100.2 | 823 ms | 10.0 ms | 1.0 | 0 | 2,983 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 166.8 | 83.8 | 1,042 ms | 11.9 ms | 2.0 | 0 | 5,003 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.