Speed test matrix
Qwen3.5-122B-A10B-FP8 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.861299 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context29.2K280 ms TTFT
16K context29.3K554 ms TTFT
32K context27.9K1,156 ms TTFT
64K context25.1K2,569 ms TTFT
128K context19.8K6,506 ms TTFT
251K context13.9K18,100 ms TTFT
256K context13.8K18,671 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 239.8159 ms TTFT | 426.5232 ms TTFT | 731.4183 ms TTFT | 1,211.7377 ms TTFT | 1,963.0452 ms TTFT | 3,270.32,083 ms TTFT | 4,973.84,008 ms TTFT | 6,502.87,757 ms TTFT |
| 16K | 242.9192 ms TTFT | 429.1195 ms TTFT | 724.0262 ms TTFT | 1,220.9442 ms TTFT | 1,948.2Requests queued494 ms TTFT | 3,232.02,112 ms TTFT | 4,764.23,993 ms TTFT | 6,159.87,851 ms TTFT |
| 32K | 239.4195 ms TTFT | 421.9223 ms TTFT | 726.3283 ms TTFT | 1,187.9362 ms TTFT | 1,928.0Requests queued932 ms TTFT | 3,042.72,032 ms TTFT | 4,443.73,691 ms TTFT | Over capacity |
| 64K | 232.1294 ms TTFT | 411.2341 ms TTFT | 703.4644 ms TTFT | 1,138.4591 ms TTFT | 1,864.21,734 ms TTFT | 2,785.63,059 ms TTFT | Over capacity | Over capacity |
| 128K | 217.9534 ms TTFT | 393.1595 ms TTFT | 657.3653 ms TTFT | 1,014.91,213 ms TTFT | 1,660.93,039 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 193.6835 ms TTFT | 351.71,025 ms TTFT | 591.5930 ms TTFT | 957.22,296 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 29.2K | — | 280 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.3K | — | 554 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 27.9K | — | 1,156 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.1K | — | 2,569 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.8K | — | 6,506 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.9K | — | 18,100 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.8K | — | 18,671 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 239.8 | 242.0 | 159 ms | 4.1 ms | 1.0 | 1 | 7,191 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 426.5 | 215.4 | 232 ms | 4.6 ms | 2.0 | 2 | 12,793 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 731.4 | 177.5 | 183 ms | 5.6 ms | 4.0 | 4 | 21,937 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,211.7 | 152.3 | 377 ms | 6.6 ms | 8.0 | 8 | 36,336 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,963.0 | 120.0 | 452 ms | 8.3 ms | 16.0 | 15 | 58,866 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,270.3 | 98.1 | 2,083 ms | 10.2 ms | 32.0 | 0 | 98,078 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,973.8 | 72.0 | 4,008 ms | 13.9 ms | 64.0 | 0 | 149,221 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,502.8 | 43.9 | 7,757 ms | 22.8 ms | 128.0 | 0 | 194,997 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 242.9 | 245.4 | 192 ms | 4.1 ms | 1.0 | 1 | 7,285 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 429.1 | 215.1 | 195 ms | 4.6 ms | 2.0 | 2 | 12,868 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 724.0 | 181.7 | 262 ms | 5.5 ms | 4.0 | 4 | 21,720 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,220.9 | 153.2 | 442 ms | 6.5 ms | 7.9 | 8 | 36,626 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 1,948.2 | 119.8 | 494 ms | 8.3 ms | 15.9 | 13 | 58,443 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,232.0 | 97.7 | 2,112 ms | 10.2 ms | 32.0 | 0 | 96,941 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,764.2 | 68.8 | 3,993 ms | 14.5 ms | 64.0 | 0 | 142,835 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,159.8 | 41.1 | 7,851 ms | 24.3 ms | 128.0 | 0 | 184,649 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 239.4 | 241.8 | 195 ms | 4.1 ms | 1.0 | 1 | 7,179 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 421.9 | 213.0 | 223 ms | 4.7 ms | 2.0 | 2 | 12,655 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 726.3 | 182.5 | 283 ms | 5.5 ms | 4.0 | 4 | 21,785 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,187.9 | 149.9 | 362 ms | 6.7 ms | 8.0 | 8 | 35,622 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 1,928.0 | 121.0 | 932 ms | 8.3 ms | 15.9 | 13 | 57,474 | 29.8 s |
| Sustained decode | 32K | 32 | Valid | 3,042.7 | 92.1 | 2,032 ms | 10.9 ms | 32.0 | 0 | 91,232 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,443.7 | 64.5 | 3,691 ms | 15.5 ms | 64.0 | 0 | 133,244 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 232.1 | 234.4 | 294 ms | 4.3 ms | 1.0 | 1 | 6,962 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 411.2 | 209.0 | 341 ms | 4.8 ms | 2.0 | 2 | 12,335 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 703.4 | 178.5 | 644 ms | 5.6 ms | 3.9 | 4 | 21,098 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,138.4 | 141.3 | 591 ms | 7.1 ms | 7.9 | 8 | 34,141 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,864.2 | 114.3 | 1,734 ms | 8.8 ms | 16.0 | 0 | 55,913 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,785.6 | 81.7 | 3,059 ms | 12.2 ms | 32.0 | 0 | 83,574 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 217.9 | 221.5 | 534 ms | 4.5 ms | 1.0 | 1 | 6,536 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 393.1 | 198.8 | 595 ms | 5.0 ms | 2.0 | 2 | 11,790 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 657.3 | 163.2 | 653 ms | 6.1 ms | 4.0 | 4 | 19,714 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,014.9 | 128.6 | 1,213 ms | 7.8 ms | 7.9 | 8 | 30,436 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,660.9 | 98.4 | 3,039 ms | 10.2 ms | 16.0 | 0 | 49,820 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 193.6 | 199.7 | 835 ms | 5.0 ms | 1.0 | 1 | 5,811 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 351.7 | 181.9 | 1,025 ms | 5.5 ms | 1.9 | 2 | 10,548 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 591.5 | 152.3 | 930 ms | 6.6 ms | 3.9 | 4 | 17,737 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 957.2 | 115.1 | 2,296 ms | 8.7 ms | 8.0 | 1 | 28,710 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.