Speed test matrix
Qwen3.5-122B-A10B-FP8 on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.861222 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context29.9K274 ms TTFT
16K context28.2K576 ms TTFT
32K context28.5K1,132 ms TTFT
64K context26.1K2,464 ms TTFT
128K context18.9K6,807 ms TTFT
251K context14.5K17,414 ms TTFT
256K context14.5K17,678 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 159.1105 ms TTFT | 299.7148 ms TTFT | 518.7262 ms TTFT | 858.0354 ms TTFT | 1,356.1634 ms TTFT | 2,130.61,036 ms TTFT | 3,354.51,905 ms TTFT | 5,019.03,639 ms TTFT |
| 16K | 157.3130 ms TTFT | 296.5180 ms TTFT | 512.9282 ms TTFT | 845.5380 ms TTFT | 1,327.4630 ms TTFT | 2,075.01,118 ms TTFT | 3,188.01,885 ms TTFT | 4,628.83,339 ms TTFT |
| 32K | 153.7140 ms TTFT | 289.9200 ms TTFT | 502.3264 ms TTFT | 818.7390 ms TTFT | 1,266.2561 ms TTFT | 1,933.0960 ms TTFT | 2,868.31,659 ms TTFT | 3,967.72,737 ms TTFT |
| 64K | 147.1233 ms TTFT | 277.4297 ms TTFT | 480.6402 ms TTFT | 771.8697 ms TTFT | 1,168.71,126 ms TTFT | 1,704.81,839 ms TTFT | 2,396.33,160 ms TTFT | Over capacity |
| 128K | 135.6330 ms TTFT | 256.3439 ms TTFT | 453.7668 ms TTFT | 695.21,022 ms TTFT | 1,004.81,784 ms TTFT | 1,382.83,307 ms TTFT | Over capacity | Over capacity |
| 251K | 119.5611 ms TTFT | 228.5922 ms TTFT | 401.91,259 ms TTFT | 584.72,011 ms TTFT | 797.23,450 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 29.9K | — | 274 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.2K | — | 576 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.5K | — | 1,132 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.1K | — | 2,464 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.9K | — | 6,807 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.5K | — | 17,414 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.5K | — | 17,678 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 159.1 | 159.9 | 105 ms | 6.3 ms | 1.0 | 1 | 4,772 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 299.7 | 150.5 | 148 ms | 6.6 ms | 2.0 | 2 | 8,990 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 518.7 | 130.9 | 262 ms | 7.6 ms | 4.0 | 4 | 15,561 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 858.0 | 107.2 | 354 ms | 9.3 ms | 8.0 | 0 | 25,736 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,356.1 | 84.7 | 634 ms | 11.8 ms | 16.0 | 0 | 40,672 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,130.6 | 65.8 | 1,036 ms | 15.2 ms | 32.0 | 0 | 63,904 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,354.5 | 50.7 | 1,905 ms | 19.7 ms | 64.0 | 0 | 100,608 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,019.0 | 36.4 | 3,639 ms | 27.5 ms | 128.0 | 0 | 150,528 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 157.3 | 158.2 | 130 ms | 6.3 ms | 1.0 | 1 | 4,722 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 296.5 | 149.0 | 180 ms | 6.7 ms | 2.0 | 2 | 8,896 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 512.9 | 129.5 | 282 ms | 7.7 ms | 3.9 | 4 | 15,384 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 845.5 | 105.8 | 380 ms | 9.5 ms | 8.0 | 0 | 25,360 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,327.4 | 83.2 | 630 ms | 12.0 ms | 16.0 | 0 | 39,824 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,075.0 | 64.7 | 1,118 ms | 15.5 ms | 32.0 | 0 | 62,240 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,188.0 | 48.6 | 1,885 ms | 20.6 ms | 64.0 | 0 | 95,616 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,628.8 | 33.9 | 3,339 ms | 29.5 ms | 128.0 | 0 | 138,880 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 153.7 | 154.8 | 140 ms | 6.5 ms | 1.0 | 1 | 4,613 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 289.9 | 145.9 | 200 ms | 6.9 ms | 2.0 | 2 | 8,696 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 502.3 | 126.7 | 264 ms | 7.9 ms | 4.0 | 4 | 15,068 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 818.7 | 102.4 | 390 ms | 9.8 ms | 8.0 | 0 | 24,560 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,266.2 | 79.2 | 561 ms | 12.6 ms | 16.0 | 0 | 37,984 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,933.0 | 60.2 | 960 ms | 16.6 ms | 32.0 | 0 | 57,984 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,868.3 | 44.3 | 1,659 ms | 22.6 ms | 64.0 | 0 | 86,016 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,967.7 | 29.9 | 2,737 ms | 33.4 ms | 128.0 | 0 | 119,040 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 147.1 | 148.8 | 233 ms | 6.7 ms | 1.0 | 1 | 4,414 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 277.4 | 140.3 | 297 ms | 7.1 ms | 2.0 | 2 | 8,322 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 480.6 | 121.6 | 402 ms | 8.2 ms | 3.9 | 4 | 14,416 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 771.8 | 96.3 | 697 ms | 10.4 ms | 8.0 | 0 | 23,152 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,168.7 | 72.2 | 1,126 ms | 13.9 ms | 16.0 | 0 | 35,056 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,704.8 | 51.9 | 1,839 ms | 19.3 ms | 32.0 | 0 | 51,136 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,396.3 | 35.7 | 3,160 ms | 28.0 ms | 64.0 | 0 | 71,872 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 135.6 | 138.4 | 330 ms | 7.2 ms | 1.0 | 1 | 4,070 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 256.3 | 130.5 | 439 ms | 7.7 ms | 1.9 | 2 | 7,688 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 453.7 | 113.4 | 668 ms | 8.8 ms | 4.0 | 0 | 13,612 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 695.2 | 86.6 | 1,022 ms | 11.5 ms | 8.0 | 0 | 20,856 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,004.8 | 62.3 | 1,784 ms | 16.1 ms | 16.0 | 0 | 30,145 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,382.8 | 42.5 | 3,307 ms | 23.5 ms | 32.0 | 0 | 41,472 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 119.5 | 119.8 | 611 ms | 8.3 ms | 1.0 | 0 | 3,584 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 228.5 | 114.4 | 922 ms | 8.7 ms | 2.0 | 0 | 6,854 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 401.9 | 100.4 | 1,259 ms | 10.0 ms | 4.0 | 0 | 12,056 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 584.7 | 72.9 | 2,011 ms | 13.7 ms | 8.0 | 0 | 17,536 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 797.2 | 49.6 | 3,450 ms | 20.2 ms | 16.0 | 0 | 23,904 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.