Speed test matrix
Qwen3.5-27B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881194 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context14.3K571 ms TTFT
16K context14K1,158 ms TTFT
32K context13.3K2,432 ms TTFT
64K context11.8K5,445 ms TTFT
128K context9,48913,562 ms TTFT
251K context6,85336,799 ms TTFT
256K context6,77937,945 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 156.5113 ms TTFT | 300.1363 ms TTFT | 587.4241 ms TTFT | 1,059.2283 ms TTFT | 1,850.01,246 ms TTFT | 2,793.52,085 ms TTFT | 3,576.03,346 ms TTFT | Over capacity |
| 16K | 155.4151 ms TTFT | 302.8195 ms TTFT | 563.6225 ms TTFT | 1,015.1290 ms TTFT | 1,724.81,366 ms TTFT | 2,475.22,290 ms TTFT | Over capacity | Over capacity |
| 32K | 153.7195 ms TTFT | 290.4261 ms TTFT | 540.3303 ms TTFT | 948.8969 ms TTFT | 1,513.41,536 ms TTFT | 2,010.02,542 ms TTFT | Over capacity | Over capacity |
| 64K | 144.6276 ms TTFT | 273.6357 ms TTFT | 491.5405 ms TTFT | 818.51,164 ms TTFT | 1,182.42,267 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 137.9495 ms TTFT | 250.6597 ms TTFT | 418.51,091 ms TTFT | 632.61,883 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 120.9750 ms TTFT | 209.21,010 ms TTFT | 319.21,456 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.3K | — | 571 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14K | — | 1,158 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.3K | — | 2,432 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.8K | — | 5,445 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,489.0 | — | 13,562 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,853.0 | — | 36,799 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,779.0 | — | 37,945 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 156.5 | 158.2 | 113 ms | 6.3 ms | 1.0 | 1 | 4,697 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 300.1 | 152.4 | 363 ms | 6.6 ms | 2.0 | 2 | 8,999 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 587.4 | 145.2 | 241 ms | 6.9 ms | 4.0 | 4 | 17,619 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,059.2 | 132.7 | 283 ms | 7.5 ms | 8.0 | 8 | 31,765 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,850.0 | 116.6 | 1,246 ms | 8.6 ms | 16.0 | 0 | 55,500 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,793.5 | 86.6 | 2,085 ms | 11.5 ms | 32.0 | 0 | 83,772 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,576.0 | 54.8 | 3,346 ms | 18.2 ms | 64.0 | 0 | 107,257 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 155.4 | 157.4 | 151 ms | 6.4 ms | 1.0 | 1 | 4,660 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 302.8 | 153.3 | 195 ms | 6.5 ms | 2.0 | 2 | 9,088 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 563.6 | 142.9 | 225 ms | 7.0 ms | 4.0 | 4 | 16,910 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,015.1 | 128.0 | 290 ms | 7.8 ms | 8.0 | 8 | 30,443 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,724.8 | 108.6 | 1,366 ms | 9.2 ms | 16.0 | 0 | 51,741 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,475.2 | 77.3 | 2,290 ms | 12.9 ms | 32.0 | 0 | 74,197 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 153.7 | 155.9 | 195 ms | 6.4 ms | 1.0 | 1 | 4,612 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 290.4 | 146.6 | 261 ms | 6.8 ms | 2.0 | 2 | 8,710 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 540.3 | 136.6 | 303 ms | 7.3 ms | 4.0 | 4 | 16,202 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 948.8 | 119.4 | 969 ms | 8.4 ms | 8.0 | 0 | 28,448 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,513.4 | 94.7 | 1,536 ms | 10.6 ms | 16.0 | 0 | 45,395 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,010.0 | 62.1 | 2,542 ms | 16.1 ms | 32.0 | 0 | 60,302 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 144.6 | 151.5 | 276 ms | 6.6 ms | 1.0 | 1 | 4,339 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 273.6 | 138.7 | 357 ms | 7.2 ms | 2.0 | 2 | 8,208 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 491.5 | 122.6 | 405 ms | 8.2 ms | 4.0 | 4 | 14,737 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 818.5 | 102.5 | 1,164 ms | 9.8 ms | 8.0 | 0 | 24,546 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,182.4 | 73.5 | 2,267 ms | 13.6 ms | 16.0 | 0 | 35,465 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 137.9 | 142.0 | 495 ms | 7.0 ms | 1.0 | 1 | 4,138 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 250.6 | 126.9 | 597 ms | 7.9 ms | 1.9 | 2 | 7,517 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 418.5 | 104.1 | 1,091 ms | 9.6 ms | 4.0 | 0 | 12,560 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 632.6 | 78.4 | 1,883 ms | 12.8 ms | 8.0 | 0 | 18,970 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 120.9 | 179.8 | 750 ms | 6.1 ms | 1.0 | 1 | 3,627 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 209.2 | 105.3 | 1,010 ms | 9.5 ms | 2.0 | 0 | 6,275 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 319.2 | 80.1 | 1,456 ms | 12.5 ms | 4.0 | 0 | 9,572 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.