Speed test matrix
Qwen3.5-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve Qwen/Qwen3.5-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.915371 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-27B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context10.2K803 ms TTFT
16K context9,8931,640 ms TTFT
32K context9,4763,408 ms TTFT
64K context8,7407,371 ms TTFT
128K context7,40917,370 ms TTFT
251K context5,67444,445 ms TTFT
256K context5,62745,713 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 168.3506 ms TTFT | 313.41,400 ms TTFT | 592.2366 ms TTFT | 1,034.6686 ms TTFT | 1,708.02,221 ms TTFT | 2,214.84,614 ms TTFT | 1,659.4Requests queued6,013 ms TTFT | 1,651.5Requests queued6,124 ms TTFT |
| 16K | 168.6924 ms TTFT | 310.41,000 ms TTFT | 591.1334 ms TTFT | 1,024.0539 ms TTFT | 1,604.82,183 ms TTFT | 2,054.44,262 ms TTFT | 1,640.3Requests queued5,238 ms TTFT | 1,739.5Requests queued5,279 ms TTFT |
| 32K | 159.11,833 ms TTFT | 321.91,939 ms TTFT | 575.6467 ms TTFT | 971.81,274 ms TTFT | 1,449.02,962 ms TTFT | 1,817.74,677 ms TTFT | 1,640.0Requests queued7,466 ms TTFT | 1,644.9Requests queued7,592 ms TTFT |
| 64K | 155.73,854 ms TTFT | 280.53,944 ms TTFT | 522.2505 ms TTFT | 870.11,414 ms TTFT | 1,199.32,698 ms TTFT | 1,425.14,960 ms TTFT | 411.1Requests queued7,483 ms TTFT | 452.0Requests queued7,596 ms TTFT |
| 128K | 143.98,966 ms TTFT | 263.29,183 ms TTFT | 483.01,239 ms TTFT | 690.52,193 ms TTFT | 890.24,150 ms TTFT | 986.07,001 ms TTFT | 52.0Requests queued9,629 ms TTFT | Over capacity |
| 251K | 135.722,709 ms TTFT | 223.244,768 ms TTFT | 372.51,505 ms TTFT | 480.72,465 ms TTFT | 562.64,325 ms TTFT | 631.37,881 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.2K | — | 803 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,893.0 | — | 1,640 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,476.0 | — | 3,408 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,740.0 | — | 7,371 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,409.0 | — | 17,370 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,674.0 | — | 44,445 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,627.0 | — | 45,713 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 168.3 | 166.8 | 506 ms | 6.0 ms | 1.0 | 1 | 5,045 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 313.4 | 159.3 | 1,400 ms | 6.3 ms | 2.0 | 2 | 9,399 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 592.2 | 149.6 | 366 ms | 6.7 ms | 4.0 | 4 | 17,763 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,034.6 | 130.8 | 686 ms | 7.7 ms | 7.9 | 8 | 31,018 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,708.0 | 106.7 | 2,221 ms | 9.4 ms | 16.0 | 0 | 51,204 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,214.8 | 69.4 | 4,614 ms | 14.4 ms | 32.0 | 0 | 66,358 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,659.4 | 47.5 | 6,013 ms | 21.0 ms | 36.8 | 9 | 49,738 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,651.5 | 47.0 | 6,124 ms | 21.3 ms | 37.1 | 5 | 49,115 | 29.7 s |
| Sustained decode | 16K | 1 | Valid | 168.6 | 173.7 | 924 ms | 5.8 ms | 1.0 | 1 | 5,054 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 310.4 | 158.7 | 1,000 ms | 6.3 ms | 2.0 | 2 | 9,309 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 591.1 | 149.8 | 334 ms | 6.7 ms | 4.0 | 4 | 17,730 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,024.0 | 127.4 | 539 ms | 7.9 ms | 8.0 | 7 | 30,703 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,604.8 | 100.9 | 2,183 ms | 9.9 ms | 16.0 | 0 | 48,146 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,054.4 | 65.5 | 4,262 ms | 15.3 ms | 32.0 | 0 | 61,624 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,640.3 | 46.6 | 5,238 ms | 21.5 ms | 37.0 | 5 | 49,152 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,739.5 | 46.2 | 5,279 ms | 21.7 ms | 37.3 | 4 | 51,718 | 29.7 s |
| Sustained decode | 32K | 1 | Valid | 159.1 | 160.5 | 1,833 ms | 6.2 ms | 1.0 | 1 | 4,769 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 321.9 | 159.6 | 1,939 ms | 6.3 ms | 2.0 | 2 | 9,653 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 575.6 | 144.4 | 467 ms | 6.9 ms | 3.9 | 4 | 17,256 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 971.8 | 122.7 | 1,274 ms | 8.1 ms | 8.0 | 3 | 29,141 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,449.0 | 92.2 | 2,962 ms | 10.8 ms | 16.0 | 0 | 43,417 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,817.7 | 56.9 | 4,677 ms | 17.6 ms | 32.0 | 0 | 54,481 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,640.0 | 40.0 | 7,466 ms | 25.0 ms | 37.5 | 0 | 49,167 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,644.9 | 40.5 | 7,592 ms | 24.7 ms | 37.4 | 0 | 49,260 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 155.7 | 157.0 | 3,854 ms | 6.4 ms | 1.0 | 1 | 4,670 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 280.5 | 141.3 | 3,944 ms | 7.1 ms | 2.0 | 2 | 8,411 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 522.2 | 134.8 | 505 ms | 7.4 ms | 3.9 | 4 | 15,652 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 870.1 | 108.9 | 1,414 ms | 9.2 ms | 8.0 | 1 | 26,098 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,199.3 | 75.4 | 2,698 ms | 13.3 ms | 16.0 | 0 | 35,955 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,425.1 | 44.3 | 4,960 ms | 22.6 ms | 32.0 | 0 | 42,730 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 411.1 | 28.8 | 7,483 ms | 34.7 ms | 29.8 | 20 | 11,925 | 29.0 s |
| Sustained decode | 64K | 128 | Requests queued | 452.0 | 29.5 | 7,596 ms | 33.9 ms | 30.5 | 26 | 13,523 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 143.9 | 148.4 | 8,966 ms | 6.7 ms | 1.0 | 1 | 4,319 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 263.2 | 135.9 | 9,183 ms | 7.4 ms | 1.9 | 2 | 7,894 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 483.0 | 119.1 | 1,239 ms | 8.4 ms | 4.0 | 1 | 14,478 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 690.5 | 84.8 | 2,193 ms | 11.8 ms | 8.0 | 0 | 20,695 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 890.2 | 54.6 | 4,150 ms | 18.3 ms | 16.0 | 0 | 26,682 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 986.0 | 30.7 | 7,001 ms | 32.6 ms | 32.0 | 0 | 29,479 | 29.9 s |
| Sustained decode | 128K | 64 | Requests queued | 52.0 | 20.5 | 9,629 ms | 48.7 ms | 24.0 | 12 | 1,522 | 29.3 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 135.7 | 138.1 | 22,709 ms | 7.2 ms | 1.0 | 1 | 4,072 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 223.2 | 112.8 | 44,768 ms | 8.9 ms | 2.0 | 0 | 6,697 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 372.5 | 92.0 | 1,505 ms | 10.9 ms | 4.0 | 0 | 11,161 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 480.7 | 59.7 | 2,465 ms | 16.7 ms | 8.0 | 0 | 14,398 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 562.6 | 35.0 | 4,325 ms | 28.6 ms | 16.0 | 0 | 16,844 | 29.9 s |
| Sustained decode | 251K | 32 | Valid | 631.3 | 19.0 | 7,881 ms | 52.5 ms | 32.0 | 0 | 18,881 | 29.9 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.