Speed test matrix
Qwen3.5-27B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880982 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context14.6K560 ms TTFT
16K context14.1K1,149 ms TTFT
32K context13.3K2,433 ms TTFT
64K context11.8K5,448 ms TTFT
128K context9,46613,595 ms TTFT
251K context6,74137,414 ms TTFT
256K context6,70638,354 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 85.069 ms TTFT | 167.0119 ms TTFT | 320.81,733 ms TTFT | 596.7287 ms TTFT | 1,064.0528 ms TTFT | 1,706.6983 ms TTFT | 2,576.11,739 ms TTFT | Over capacity |
| 16K | 84.1104 ms TTFT | 164.0156 ms TTFT | 311.1267 ms TTFT | 564.9424 ms TTFT | 971.2690 ms TTFT | 1,481.91,457 ms TTFT | 2,048.52,227 ms TTFT | Over capacity |
| 32K | 82.3117 ms TTFT | 158.4160 ms TTFT | 293.2246 ms TTFT | 512.3304 ms TTFT | 827.8490 ms TTFT | 1,147.9742 ms TTFT | Over capacity | Over capacity |
| 64K | 79.0165 ms TTFT | 148.2230 ms TTFT | 264.3339 ms TTFT | 431.4511 ms TTFT | 632.1826 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 73.4270 ms TTFT | 131.2368 ms TTFT | 220.2579 ms TTFT | 323.8886 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 64.4648 ms TTFT | 107.1868 ms TTFT | 163.51,210 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.6K | — | 560 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.1K | — | 1,149 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.3K | — | 2,433 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.8K | — | 5,448 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,466.0 | — | 13,595 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,741.0 | — | 37,414 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,706.0 | — | 38,354 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 85.0 | 85.0 | 69 ms | 11.8 ms | 1.0 | 0 | 2,550 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 167.0 | 83.5 | 119 ms | 12.0 ms | 2.0 | 0 | 5,008 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 320.8 | 80.3 | 1,733 ms | 12.5 ms | 4.0 | 0 | 9,624 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 596.7 | 74.7 | 287 ms | 13.4 ms | 8.0 | 0 | 17,904 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,064.0 | 66.7 | 528 ms | 15.0 ms | 16.0 | 0 | 31,920 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,706.6 | 53.6 | 983 ms | 18.7 ms | 32.0 | 0 | 51,200 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,576.1 | 40.4 | 1,739 ms | 24.7 ms | 64.0 | 0 | 77,248 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 84.1 | 84.1 | 104 ms | 11.9 ms | 1.0 | 0 | 2,523 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 164.0 | 82.1 | 156 ms | 12.2 ms | 2.0 | 0 | 4,920 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 311.1 | 77.8 | 267 ms | 12.8 ms | 4.0 | 0 | 9,333 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 564.9 | 70.7 | 424 ms | 14.1 ms | 8.0 | 0 | 16,952 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 971.2 | 60.7 | 690 ms | 16.5 ms | 16.0 | 0 | 29,136 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,481.9 | 46.4 | 1,457 ms | 21.5 ms | 32.0 | 0 | 44,448 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,048.5 | 31.7 | 2,227 ms | 31.5 ms | 64.0 | 0 | 61,440 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 82.3 | 82.4 | 117 ms | 12.1 ms | 1.0 | 0 | 2,468 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.4 | 79.3 | 160 ms | 12.6 ms | 2.0 | 0 | 4,750 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 293.2 | 73.4 | 246 ms | 13.6 ms | 4.0 | 0 | 8,801 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 512.3 | 64.1 | 304 ms | 15.6 ms | 8.0 | 0 | 15,369 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 827.8 | 51.7 | 490 ms | 19.4 ms | 16.0 | 0 | 24,832 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,147.9 | 35.7 | 742 ms | 28.0 ms | 32.0 | 0 | 34,432 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 79.0 | 79.1 | 165 ms | 12.6 ms | 1.0 | 0 | 2,369 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 148.2 | 74.2 | 230 ms | 13.5 ms | 2.0 | 0 | 4,446 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 264.3 | 66.2 | 339 ms | 15.1 ms | 4.0 | 0 | 7,932 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 431.4 | 53.9 | 511 ms | 18.5 ms | 8.0 | 0 | 12,936 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 632.1 | 39.4 | 826 ms | 25.4 ms | 16.0 | 0 | 18,960 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 73.4 | 73.6 | 270 ms | 13.6 ms | 1.0 | 0 | 2,203 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 131.2 | 65.7 | 368 ms | 15.2 ms | 2.0 | 0 | 3,938 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 220.2 | 55.1 | 579 ms | 18.1 ms | 4.0 | 0 | 6,608 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 323.8 | 40.5 | 886 ms | 24.7 ms | 8.0 | 0 | 9,712 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 64.4 | 64.5 | 648 ms | 15.5 ms | 1.0 | 0 | 1,931 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 107.1 | 53.6 | 868 ms | 18.7 ms | 2.0 | 0 | 3,214 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 163.5 | 40.8 | 1,210 ms | 24.5 ms | 4.0 | 0 | 4,904 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.