Speed test matrix
Qwen3.6-27B-int4-AutoRound on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4+BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-intel-vllm/bin/vllm" serve Intel/Qwen3.6-27B-int4-AutoRound --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883651 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context9,884828 ms TTFT
16K context9,9281,634 ms TTFT
32K context9,3573,451 ms TTFT
64K context8,6417,455 ms TTFT
128K context7,29217,647 ms TTFT
251K context5,56345,334 ms TTFT
256K context5,53746,451 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 86.083 ms TTFT | 168.2126 ms TTFT | 326.21,775 ms TTFT | 604.4366 ms TTFT | 1,083.9691 ms TTFT | 1,723.21,294 ms TTFT | 2,306.52,450 ms TTFT | 2,904.94,059 ms TTFT |
| 16K | 84.9136 ms TTFT | 165.0180 ms TTFT | 315.9338 ms TTFT | 573.8495 ms TTFT | 986.7879 ms TTFT | 1,490.71,806 ms TTFT | 1,840.62,824 ms TTFT | Over capacity |
| 32K | 83.0122 ms TTFT | 158.9166 ms TTFT | 297.0269 ms TTFT | 519.1321 ms TTFT | 838.2545 ms TTFT | 1,167.5856 ms TTFT | Over capacity | Over capacity |
| 64K | 79.3167 ms TTFT | 149.3249 ms TTFT | 266.9393 ms TTFT | 435.0517 ms TTFT | 637.0845 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 73.7277 ms TTFT | 132.7399 ms TTFT | 221.1601 ms TTFT | 324.4918 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 64.7665 ms TTFT | 108.1899 ms TTFT | 163.11,291 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,884.0 | — | 828 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,928.0 | — | 1,634 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,357.0 | — | 3,451 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,641.0 | — | 7,455 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,292.0 | — | 17,647 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,563.0 | — | 45,334 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,537.0 | — | 46,451 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 86.0 | 86.0 | 83 ms | 11.6 ms | 1.0 | 0 | 2,580 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 168.2 | 84.1 | 126 ms | 11.9 ms | 2.0 | 0 | 5,046 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 326.2 | 81.6 | 1,775 ms | 12.3 ms | 4.0 | 0 | 9,784 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 604.4 | 75.6 | 366 ms | 13.2 ms | 8.0 | 0 | 18,136 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,083.9 | 68.0 | 691 ms | 14.7 ms | 16.0 | 0 | 32,512 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,723.2 | 54.1 | 1,294 ms | 18.5 ms | 32.0 | 0 | 51,680 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,306.5 | 36.2 | 2,450 ms | 27.6 ms | 64.0 | 0 | 69,184 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,904.9 | 22.4 | 4,059 ms | 44.6 ms | 128.0 | 0 | 87,168 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 84.9 | 85.0 | 136 ms | 11.8 ms | 1.0 | 0 | 2,548 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 165.0 | 82.5 | 180 ms | 12.1 ms | 2.0 | 0 | 4,952 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 315.9 | 79.1 | 338 ms | 12.6 ms | 4.0 | 0 | 9,480 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 573.8 | 71.7 | 495 ms | 13.9 ms | 8.0 | 0 | 17,216 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 986.7 | 61.6 | 879 ms | 16.2 ms | 16.0 | 0 | 29,600 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,490.7 | 46.6 | 1,806 ms | 21.5 ms | 32.0 | 0 | 44,704 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,840.6 | 28.1 | 2,824 ms | 35.6 ms | 64.0 | 0 | 55,232 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 83.0 | 83.1 | 122 ms | 12.0 ms | 1.0 | 0 | 2,490 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.9 | 79.5 | 166 ms | 12.6 ms | 2.0 | 0 | 4,768 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 297.0 | 74.3 | 269 ms | 13.5 ms | 4.0 | 0 | 8,908 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 519.1 | 64.8 | 321 ms | 15.4 ms | 8.0 | 0 | 15,568 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 838.2 | 52.3 | 545 ms | 19.1 ms | 16.0 | 0 | 25,136 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,167.5 | 36.3 | 856 ms | 27.6 ms | 32.0 | 0 | 35,008 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 79.3 | 79.4 | 167 ms | 12.6 ms | 1.0 | 0 | 2,377 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 149.3 | 74.7 | 249 ms | 13.4 ms | 2.0 | 0 | 4,480 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 266.9 | 66.8 | 393 ms | 15.0 ms | 4.0 | 0 | 8,008 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 435.0 | 54.3 | 517 ms | 18.4 ms | 8.0 | 0 | 13,048 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 637.0 | 39.6 | 845 ms | 25.2 ms | 16.0 | 0 | 19,104 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 73.7 | 73.8 | 277 ms | 13.5 ms | 1.0 | 0 | 2,210 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 132.7 | 66.5 | 399 ms | 15.0 ms | 2.0 | 0 | 3,978 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 221.1 | 55.3 | 601 ms | 18.1 ms | 4.0 | 0 | 6,632 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 324.4 | 40.6 | 918 ms | 24.6 ms | 8.0 | 0 | 9,730 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 64.7 | 64.8 | 665 ms | 15.4 ms | 1.0 | 0 | 1,939 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 108.1 | 54.1 | 899 ms | 18.5 ms | 2.0 | 0 | 3,242 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 163.1 | 40.7 | 1,291 ms | 24.6 ms | 4.0 | 0 | 4,892 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.