Speed test matrix
Qwen3.5-27B-GPTQ-Int4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.884422 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context10K817 ms TTFT
16K context9,8971,639 ms TTFT
32K context9,4613,413 ms TTFT
64K context8,6997,406 ms TTFT
128K context7,36717,470 ms TTFT
251K context5,60944,963 ms TTFT
256K context5,58546,055 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 78.086 ms TTFT | 153.0133 ms TTFT | 296.91,832 ms TTFT | 553.3343 ms TTFT | 995.7592 ms TTFT | 1,631.71,252 ms TTFT | 2,283.82,469 ms TTFT | Over capacity |
| 16K | 77.2133 ms TTFT | 150.8188 ms TTFT | 288.2337 ms TTFT | 524.7495 ms TTFT | 909.5903 ms TTFT | 1,406.41,853 ms TTFT | 1,813.02,931 ms TTFT | Over capacity |
| 32K | 75.6127 ms TTFT | 145.5181 ms TTFT | 272.2242 ms TTFT | 476.6332 ms TTFT | 772.3504 ms TTFT | 1,081.4777 ms TTFT | Over capacity | Over capacity |
| 64K | 72.7170 ms TTFT | 136.9229 ms TTFT | 245.6332 ms TTFT | 399.7487 ms TTFT | 574.5791 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 68.0290 ms TTFT | 121.5391 ms TTFT | 202.7577 ms TTFT | 291.1909 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 60.0673 ms TTFT | 98.8910 ms TTFT | 146.91,244 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10K | — | 817 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,897.0 | — | 1,639 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,461.0 | — | 3,413 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,699.0 | — | 7,406 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,367.0 | — | 17,470 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,609.0 | — | 44,963 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,585.0 | — | 46,055 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 78.0 | 78.0 | 86 ms | 12.8 ms | 1.0 | 0 | 2,340 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 153.0 | 76.5 | 133 ms | 13.1 ms | 2.0 | 0 | 4,590 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 296.9 | 74.3 | 1,832 ms | 13.5 ms | 4.0 | 0 | 8,908 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 553.3 | 69.2 | 343 ms | 14.4 ms | 8.0 | 0 | 16,592 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 995.7 | 62.3 | 592 ms | 16.1 ms | 16.0 | 0 | 29,872 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,631.7 | 51.2 | 1,252 ms | 19.5 ms | 32.0 | 0 | 48,928 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,283.8 | 35.9 | 2,469 ms | 27.9 ms | 64.0 | 0 | 68,480 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 77.2 | 77.2 | 133 ms | 12.9 ms | 1.0 | 0 | 2,317 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 150.8 | 75.4 | 188 ms | 13.3 ms | 2.0 | 0 | 4,522 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 288.2 | 72.1 | 337 ms | 13.9 ms | 4.0 | 0 | 8,644 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 524.7 | 65.6 | 495 ms | 15.3 ms | 8.0 | 0 | 15,736 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 909.5 | 56.8 | 903 ms | 17.6 ms | 16.0 | 0 | 27,280 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,406.4 | 43.9 | 1,853 ms | 22.8 ms | 32.0 | 0 | 42,176 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,813.0 | 27.8 | 2,931 ms | 36.0 ms | 64.0 | 0 | 54,400 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 75.6 | 75.7 | 127 ms | 13.2 ms | 1.0 | 0 | 2,270 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 145.5 | 72.9 | 181 ms | 13.7 ms | 2.0 | 0 | 4,367 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 272.2 | 68.1 | 242 ms | 14.7 ms | 4.0 | 0 | 8,168 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 476.6 | 59.5 | 332 ms | 16.8 ms | 8.0 | 0 | 14,296 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 772.3 | 48.2 | 504 ms | 20.8 ms | 16.0 | 0 | 23,168 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,081.4 | 33.6 | 777 ms | 29.8 ms | 32.0 | 0 | 32,416 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 72.7 | 72.8 | 170 ms | 13.7 ms | 1.0 | 0 | 2,181 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 136.9 | 68.5 | 229 ms | 14.6 ms | 2.0 | 0 | 4,106 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 245.6 | 61.4 | 332 ms | 16.3 ms | 4.0 | 0 | 7,372 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 399.7 | 49.9 | 487 ms | 20.0 ms | 8.0 | 0 | 11,984 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 574.5 | 35.8 | 791 ms | 28.0 ms | 16.0 | 0 | 17,248 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 68.0 | 68.1 | 290 ms | 14.7 ms | 1.0 | 0 | 2,039 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 121.5 | 60.9 | 391 ms | 16.4 ms | 2.0 | 0 | 3,647 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 202.7 | 50.7 | 577 ms | 19.7 ms | 4.0 | 0 | 6,084 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 291.1 | 36.4 | 909 ms | 27.4 ms | 8.0 | 0 | 8,736 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 60.0 | 60.2 | 673 ms | 16.6 ms | 1.0 | 0 | 1,800 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 98.8 | 49.4 | 910 ms | 20.2 ms | 2.0 | 0 | 2,962 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 146.9 | 36.6 | 1,244 ms | 27.3 ms | 4.0 | 0 | 4,404 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.