Speed test matrix
Qwen3.5-122B-A10B-GPTQ-Int4 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.862690 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context22.5K363 ms TTFT
16K context22.8K713 ms TTFT
32K context22.1K1,462 ms TTFT
64K context20.2K3,189 ms TTFT
128K context16.6K7,752 ms TTFT
251K context12.2K20,633 ms TTFT
256K context12K21,515 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 237.1196 ms TTFT | 431.6279 ms TTFT | 742.7222 ms TTFT | 1,306.0380 ms TTFT | 2,023.2532 ms TTFT | 3,321.52,716 ms TTFT | 4,632.05,246 ms TTFT | 6,342.610,187 ms TTFT |
| 16K | 239.3222 ms TTFT | 428.2236 ms TTFT | 717.5335 ms TTFT | 1,290.8537 ms TTFT | 1,995.5595 ms TTFT | 3,217.32,674 ms TTFT | 4,467.35,046 ms TTFT | 6,004.610,004 ms TTFT |
| 32K | 234.4228 ms TTFT | 427.4250 ms TTFT | 765.3275 ms TTFT | 1,266.5456 ms TTFT | 1,966.8726 ms TTFT | 3,055.02,552 ms TTFT | 4,145.64,657 ms TTFT | 5,413.98,875 ms TTFT |
| 64K | 225.8353 ms TTFT | 413.0363 ms TTFT | 733.5479 ms TTFT | 1,202.3Requests queued811 ms TTFT | 1,923.61,824 ms TTFT | 2,766.43,556 ms TTFT | 3,592.56,727 ms TTFT | Over capacity |
| 128K | 218.5599 ms TTFT | 390.8598 ms TTFT | 647.1776 ms TTFT | 1,066.21,299 ms TTFT | 1,710.93,375 ms TTFT | 2,307.86,163 ms TTFT | Over capacity | Over capacity |
| 251K | 198.0976 ms TTFT | 355.4941 ms TTFT | 596.01,310 ms TTFT | 977.02,007 ms TTFT | 1,382.84,089 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 22.5K | — | 363 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 22.8K | — | 713 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.1K | — | 1,462 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.2K | — | 3,189 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.6K | — | 7,752 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.2K | — | 20,633 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12K | — | 21,515 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 237.1 | 239.8 | 196 ms | 4.2 ms | 1.0 | 1 | 7,113 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 431.6 | 215.7 | 279 ms | 4.6 ms | 2.0 | 2 | 12,946 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 742.7 | 187.9 | 222 ms | 5.3 ms | 4.0 | 4 | 22,275 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,306.0 | 163.1 | 380 ms | 6.1 ms | 8.0 | 8 | 39,181 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,023.2 | 125.1 | 532 ms | 8.0 ms | 15.9 | 16 | 60,696 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,321.5 | 99.0 | 2,716 ms | 10.1 ms | 32.0 | 0 | 99,653 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,632.0 | 64.7 | 5,246 ms | 15.4 ms | 64.0 | 0 | 138,963 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,342.6 | 40.2 | 10,187 ms | 24.9 ms | 128.0 | 0 | 190,216 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 239.3 | 240.7 | 222 ms | 4.2 ms | 1.0 | 1 | 7,178 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 428.2 | 215.8 | 236 ms | 4.6 ms | 2.0 | 2 | 12,845 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 717.5 | 181.9 | 335 ms | 5.5 ms | 3.9 | 4 | 21,526 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,290.8 | 162.8 | 537 ms | 6.1 ms | 8.0 | 8 | 38,712 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,995.5 | 123.1 | 595 ms | 8.1 ms | 16.0 | 16 | 59,825 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,217.3 | 95.7 | 2,674 ms | 10.5 ms | 32.0 | 0 | 96,451 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,467.3 | 63.0 | 5,046 ms | 15.9 ms | 64.0 | 0 | 133,905 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,004.6 | 38.3 | 10,004 ms | 26.1 ms | 128.0 | 0 | 180,186 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 234.4 | 236.3 | 228 ms | 4.2 ms | 1.0 | 1 | 7,029 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 427.4 | 214.9 | 250 ms | 4.7 ms | 2.0 | 2 | 12,821 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 765.3 | 192.6 | 275 ms | 5.2 ms | 4.0 | 4 | 22,956 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,266.5 | 159.2 | 456 ms | 6.3 ms | 7.9 | 8 | 37,985 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,966.8 | 121.8 | 726 ms | 8.2 ms | 15.8 | 14 | 58,985 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,055.0 | 92.1 | 2,552 ms | 10.9 ms | 32.0 | 0 | 91,627 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,145.6 | 59.0 | 4,657 ms | 17.0 ms | 64.0 | 0 | 124,329 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,413.9 | 35.3 | 8,875 ms | 28.4 ms | 128.0 | 0 | 162,357 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 225.8 | 229.3 | 353 ms | 4.4 ms | 1.0 | 1 | 6,773 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 413.0 | 208.2 | 363 ms | 4.8 ms | 2.0 | 2 | 12,386 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 733.5 | 184.2 | 479 ms | 5.4 ms | 4.0 | 4 | 21,999 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 1,202.3 | 152.5 | 811 ms | 6.6 ms | 7.9 | 8 | 36,066 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,923.6 | 117.4 | 1,824 ms | 8.5 ms | 15.9 | 5 | 57,457 | 29.9 s |
| Sustained decode | 64K | 32 | Valid | 2,766.4 | 80.4 | 3,556 ms | 12.4 ms | 32.0 | 0 | 82,988 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,592.5 | 48.8 | 6,727 ms | 20.5 ms | 64.0 | 0 | 107,755 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 218.5 | 222.4 | 599 ms | 4.5 ms | 1.0 | 1 | 6,558 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 390.8 | 199.0 | 598 ms | 5.0 ms | 2.0 | 2 | 11,720 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 647.1 | 164.5 | 776 ms | 6.1 ms | 4.0 | 4 | 19,407 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,066.2 | 132.7 | 1,299 ms | 7.5 ms | 7.9 | 8 | 31,979 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,710.9 | 101.1 | 3,375 ms | 9.9 ms | 16.0 | 0 | 51,293 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,307.8 | 63.2 | 6,163 ms | 15.8 ms | 32.0 | 0 | 69,246 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 198.0 | 202.9 | 976 ms | 4.9 ms | 1.0 | 1 | 5,939 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 355.4 | 182.3 | 941 ms | 5.5 ms | 1.9 | 2 | 10,661 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 596.0 | 153.2 | 1,310 ms | 6.5 ms | 3.9 | 4 | 17,875 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 977.0 | 119.6 | 2,007 ms | 8.4 ms | 7.9 | 7 | 29,046 | 29.7 s |
| Sustained decode | 251K | 16 | Valid | 1,382.8 | 81.5 | 4,089 ms | 12.3 ms | 16.0 | 0 | 41,457 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.