Speed test matrix
Qwen3.8-27B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883918 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context14.1K584 ms TTFT
16K context14K1,166 ms TTFT
32K context13.2K2,445 ms TTFT
64K context11.8K5,461 ms TTFT
128K context9,46613,600 ms TTFT
251K context6,83836,887 ms TTFT
256K context6,78237,930 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 130.4124 ms TTFT | 255.5372 ms TTFT | 499.3289 ms TTFT | 896.0693 ms TTFT | 1,531.41,274 ms TTFT | 2,228.82,135 ms TTFT | 2,782.23,429 ms TTFT | Over capacity |
| 16K | 127.1168 ms TTFT | 245.5213 ms TTFT | 494.8239 ms TTFT | 859.0778 ms TTFT | 1,414.51,433 ms TTFT | 1,977.32,191 ms TTFT | Over capacity | Over capacity |
| 32K | 126.3213 ms TTFT | 246.4285 ms TTFT | 469.2506 ms TTFT | 780.21,000 ms TTFT | 1,244.31,711 ms TTFT | 1,620.02,333 ms TTFT | Over capacity | Over capacity |
| 64K | 123.2272 ms TTFT | 228.7453 ms TTFT | 431.0686 ms TTFT | 656.91,230 ms TTFT | 977.32,238 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 122.1494 ms TTFT | 204.6687 ms TTFT | 366.51,223 ms TTFT | 517.02,325 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 102.3781 ms TTFT | 192.21,054 ms TTFT | 275.81,479 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.1K | — | 584 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14K | — | 1,166 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.2K | — | 2,445 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.8K | — | 5,461 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,466.0 | — | 13,600 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,838.0 | — | 36,887 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,782.0 | — | 37,930 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 130.4 | 135.0 | 124 ms | 7.4 ms | 1.0 | 1 | 3,911 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 255.5 | 129.4 | 372 ms | 7.7 ms | 2.0 | 2 | 7,663 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 499.3 | 128.3 | 289 ms | 7.8 ms | 4.0 | 4 | 14,978 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 896.0 | 112.1 | 693 ms | 8.9 ms | 8.0 | 0 | 26,878 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,531.4 | 96.1 | 1,274 ms | 10.4 ms | 16.0 | 0 | 45,942 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,228.8 | 69.8 | 2,135 ms | 14.3 ms | 32.0 | 0 | 66,815 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,782.2 | 42.2 | 3,429 ms | 23.7 ms | 64.0 | 0 | 83,386 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 127.1 | 139.1 | 168 ms | 7.2 ms | 1.0 | 1 | 3,811 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 245.5 | 124.5 | 213 ms | 8.0 ms | 2.0 | 2 | 7,347 | 29.9 s |
| Sustained decode | 16K | 4 | Valid | 494.8 | 128.0 | 239 ms | 7.8 ms | 4.0 | 4 | 14,839 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 859.0 | 108.8 | 778 ms | 9.2 ms | 8.0 | 0 | 25,752 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,414.5 | 88.5 | 1,433 ms | 11.3 ms | 16.0 | 0 | 42,428 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,977.3 | 61.1 | 2,191 ms | 16.4 ms | 32.0 | 0 | 59,311 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 126.3 | 152.8 | 213 ms | 6.7 ms | 1.0 | 1 | 3,788 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 246.4 | 125.0 | 285 ms | 8.0 ms | 2.0 | 2 | 7,376 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 469.2 | 118.7 | 506 ms | 8.4 ms | 4.0 | 0 | 14,073 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 780.2 | 97.9 | 1,000 ms | 10.2 ms | 8.0 | 0 | 23,391 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,244.3 | 78.2 | 1,711 ms | 12.8 ms | 16.0 | 0 | 37,331 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,620.0 | 49.8 | 2,333 ms | 20.1 ms | 32.0 | 0 | 48,564 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 123.2 | 124.1 | 272 ms | 8.1 ms | 1.0 | 1 | 3,662 | 29.7 s |
| Sustained decode | 64K | 2 | Valid | 228.7 | 115.4 | 453 ms | 8.7 ms | 2.0 | 0 | 6,859 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 431.0 | 107.1 | 686 ms | 9.3 ms | 4.0 | 0 | 12,926 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 656.9 | 82.9 | 1,230 ms | 12.1 ms | 8.0 | 0 | 19,700 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 977.3 | 61.4 | 2,238 ms | 16.3 ms | 16.0 | 0 | 29,291 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 122.1 | 121.9 | 494 ms | 8.2 ms | 1.0 | 0 | 3,661 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 204.6 | 102.7 | 687 ms | 9.7 ms | 2.0 | 0 | 6,134 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 366.5 | 91.6 | 1,223 ms | 10.9 ms | 4.0 | 0 | 10,994 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 517.0 | 64.2 | 2,325 ms | 15.6 ms | 8.0 | 0 | 15,512 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 102.3 | 102.6 | 781 ms | 9.7 ms | 1.0 | 0 | 3,067 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 192.2 | 96.7 | 1,054 ms | 10.3 ms | 2.0 | 0 | 5,762 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 275.8 | 67.4 | 1,479 ms | 14.8 ms | 4.0 | 0 | 8,274 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.