Speed test matrix
Qwen3.6-27B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.934611 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-27B-DFlash","num_speculative_tokens":15}'Prefill
prompt tok/s by context length8K context10.1K815 ms TTFT
16K context10.2K1,592 ms TTFT
32K context9,6223,356 ms TTFT
64K context8,8737,260 ms TTFT
128K context7,52117,110 ms TTFT
251K context5,77343,689 ms TTFT
256K context5,73444,860 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 118.1805 ms TTFT | 227.4921 ms TTFT | 388.1602 ms TTFT | 652.51,077 ms TTFT | 887.02,175 ms TTFT | 766.8Requests queued3,010 ms TTFT | 773.9Requests queued3,046 ms TTFT | 860.5Requests queued2,550 ms TTFT |
| 16K | 125.8907 ms TTFT | 202.21,740 ms TTFT | 400.6743 ms TTFT | 639.61,421 ms TTFT | 838.72,676 ms TTFT | 825.2Requests queued3,761 ms TTFT | 817.0Requests queued3,843 ms TTFT | 858.6Requests queued3,857 ms TTFT |
| 32K | 115.33,355 ms TTFT | 224.93,514 ms TTFT | 404.7665 ms TTFT | 607.31,298 ms TTFT | 819.42,241 ms TTFT | 738.7Requests queued2,774 ms TTFT | 748.4Requests queued2,822 ms TTFT | 749.1Requests queued2,859 ms TTFT |
| 64K | 108.77,247 ms TTFT | 211.97,447 ms TTFT | 345.2975 ms TTFT | 559.31,662 ms TTFT | 715.43,228 ms TTFT | 291.0Requests queued4,486 ms TTFT | 301.2Requests queued4,531 ms TTFT | 438.5Requests queued4,618 ms TTFT |
| 128K | 102.217,112 ms TTFT | 177.617,438 ms TTFT | 310.71,498 ms TTFT | 460.42,956 ms TTFT | 584.84,583 ms TTFT | 201.9Requests queued6,343 ms TTFT | 35.1Requests queued6,464 ms TTFT | Over capacity |
| 251K | 95.143,632 ms TTFT | 167.944,131 ms TTFT | 258.22,217 ms TTFT | 348.23,631 ms TTFT | 435.86,117 ms TTFT | 3.6Requests queued6,391 ms TTFT | 364.3Requests queued6,536 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.1K | — | 815 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.2K | — | 1,592 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,622.0 | — | 3,356 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,873.0 | — | 7,260 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,521.0 | — | 17,110 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,773.0 | — | 43,689 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,734.0 | — | 44,860 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 118.1 | 121.5 | 805 ms | 8.2 ms | 1.0 | 1 | 3,538 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 227.4 | 116.6 | 921 ms | 8.6 ms | 2.0 | 0 | 6,816 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 388.1 | 101.9 | 602 ms | 9.8 ms | 4.0 | 0 | 11,638 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 652.5 | 82.9 | 1,077 ms | 12.1 ms | 8.0 | 0 | 19,563 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 887.0 | 56.9 | 2,175 ms | 17.6 ms | 16.0 | 0 | 26,595 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 766.8 | 40.3 | 3,010 ms | 24.8 ms | 21.4 | 0 | 22,985 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 773.9 | 38.7 | 3,046 ms | 25.8 ms | 21.2 | 0 | 23,181 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 860.5 | 41.5 | 2,550 ms | 24.1 ms | 21.3 | 1 | 25,804 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 125.8 | 175.0 | 907 ms | 6.2 ms | 1.0 | 1 | 3,774 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 202.2 | 105.3 | 1,740 ms | 9.5 ms | 2.0 | 0 | 6,063 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 400.6 | 104.6 | 743 ms | 9.6 ms | 4.0 | 0 | 12,019 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 639.6 | 81.7 | 1,421 ms | 12.2 ms | 8.0 | 0 | 19,191 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 838.7 | 54.0 | 2,676 ms | 18.5 ms | 16.0 | 0 | 25,131 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 825.2 | 40.6 | 3,761 ms | 24.6 ms | 21.0 | 0 | 24,718 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 817.0 | 40.7 | 3,843 ms | 24.5 ms | 21.0 | 1 | 24,498 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 858.6 | 41.3 | 3,857 ms | 24.2 ms | 21.0 | 0 | 25,708 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 115.3 | 120.1 | 3,355 ms | 8.3 ms | 1.0 | 0 | 3,460 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 224.9 | 114.7 | 3,514 ms | 8.7 ms | 2.0 | 0 | 6,744 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 404.7 | 105.0 | 665 ms | 9.5 ms | 4.0 | 0 | 12,141 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 607.3 | 78.9 | 1,298 ms | 12.7 ms | 8.0 | 0 | 18,219 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 819.4 | 53.0 | 2,241 ms | 18.9 ms | 16.0 | 0 | 24,561 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 738.7 | 37.6 | 2,774 ms | 26.6 ms | 21.0 | 0 | 22,160 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 748.4 | 37.4 | 2,822 ms | 26.7 ms | 21.0 | 0 | 22,444 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 749.1 | 36.2 | 2,859 ms | 27.7 ms | 21.0 | 0 | 22,430 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 108.7 | 111.3 | 7,247 ms | 9.0 ms | 1.0 | 0 | 3,259 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 211.9 | 110.2 | 7,447 ms | 9.1 ms | 2.0 | 0 | 6,361 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 345.2 | 88.7 | 975 ms | 11.3 ms | 4.0 | 0 | 10,351 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 559.3 | 71.7 | 1,662 ms | 14.0 ms | 8.0 | 0 | 16,765 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 715.4 | 45.9 | 3,228 ms | 21.8 ms | 16.0 | 0 | 21,436 | 30.0 s |
| Sustained decode | 64K | 32 | Requests queued | 291.0 | 28.9 | 4,486 ms | 34.6 ms | 17.6 | 14 | 8,696 | 29.9 s |
| Sustained decode | 64K | 64 | Requests queued | 301.2 | 29.2 | 4,531 ms | 34.2 ms | 17.4 | 12 | 9,014 | 29.9 s |
| Sustained decode | 64K | 128 | Requests queued | 438.5 | 30.9 | 4,618 ms | 32.4 ms | 18.9 | 16 | 13,126 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 102.2 | 105.4 | 17,112 ms | 9.5 ms | 1.0 | 0 | 3,063 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 177.6 | 92.7 | 17,438 ms | 10.9 ms | 2.0 | 0 | 5,330 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 310.7 | 80.2 | 1,498 ms | 12.5 ms | 4.0 | 0 | 9,313 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 460.4 | 59.2 | 2,956 ms | 16.9 ms | 8.0 | 0 | 13,798 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 584.8 | 36.7 | 4,583 ms | 27.2 ms | 16.0 | 0 | 17,525 | 30.0 s |
| Sustained decode | 128K | 32 | Requests queued | 201.9 | 26.3 | 6,343 ms | 38.1 ms | 11.7 | 19 | 6,046 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 35.1 | 21.8 | 6,464 ms | 45.9 ms | 13.9 | 9 | 999 | 28.5 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 95.1 | 98.2 | 43,632 ms | 10.2 ms | 1.0 | 0 | 2,853 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 167.9 | 86.5 | 44,131 ms | 11.6 ms | 2.0 | 0 | 5,033 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 258.2 | 65.7 | 2,217 ms | 15.2 ms | 4.0 | 0 | 7,749 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 348.2 | 44.7 | 3,631 ms | 22.4 ms | 8.0 | 0 | 10,438 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 435.8 | 26.7 | 6,117 ms | 37.5 ms | 16.0 | 0 | 13,063 | 30.0 s |
| Sustained decode | 251K | 32 | Requests queued | 3.6 | 22.6 | 6,391 ms | 44.4 ms | 8.6 | 9 | 87 | 24.3 s |
| Sustained decode | 251K | 64 | Requests queued | 364.3 | 25.3 | 6,536 ms | 39.6 ms | 15.4 | 2 | 10,920 | 30.0 s |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.