Speed test matrix
Qwen3.8-27B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880982 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context14.5K567 ms TTFT
16K context14.2K1,142 ms TTFT
32K context13.4K2,422 ms TTFT
64K context11.8K5,448 ms TTFT
128K context9,46513,601 ms TTFT
251K context6,74237,412 ms TTFT
256K context6,68738,470 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 85.075 ms TTFT | 166.9137 ms TTFT | 320.71,738 ms TTFT | 596.6296 ms TTFT | 1,062.5497 ms TTFT | 1,703.51,075 ms TTFT | 2,570.21,743 ms TTFT | Over capacity |
| 16K | 84.0118 ms TTFT | 164.0166 ms TTFT | 311.0304 ms TTFT | 564.5480 ms TTFT | 970.4856 ms TTFT | 1,477.81,336 ms TTFT | 2,044.92,248 ms TTFT | Over capacity |
| 32K | 82.2107 ms TTFT | 158.4173 ms TTFT | 293.0248 ms TTFT | 511.4310 ms TTFT | 827.2470 ms TTFT | 1,146.5796 ms TTFT | Over capacity | Over capacity |
| 64K | 78.9164 ms TTFT | 148.2233 ms TTFT | 264.2334 ms TTFT | 430.9489 ms TTFT | 631.3764 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 73.3278 ms TTFT | 131.1405 ms TTFT | 220.1575 ms TTFT | 323.7926 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 64.4654 ms TTFT | 107.2856 ms TTFT | 163.51,309 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.5K | — | 567 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.2K | — | 1,142 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.4K | — | 2,422 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.8K | — | 5,448 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,465.0 | — | 13,601 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,742.0 | — | 37,412 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,687.0 | — | 38,470 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 85.0 | 85.1 | 75 ms | 11.8 ms | 1.0 | 0 | 2,550 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 166.9 | 83.5 | 137 ms | 12.0 ms | 2.0 | 0 | 5,006 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 320.7 | 80.3 | 1,738 ms | 12.5 ms | 4.0 | 0 | 9,620 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 596.6 | 74.6 | 296 ms | 13.4 ms | 8.0 | 0 | 17,896 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,062.5 | 66.5 | 497 ms | 15.0 ms | 16.0 | 0 | 31,872 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,703.5 | 53.5 | 1,075 ms | 18.7 ms | 32.0 | 0 | 51,104 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,570.2 | 40.1 | 1,743 ms | 24.9 ms | 64.0 | 0 | 77,120 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 84.0 | 84.0 | 118 ms | 11.9 ms | 1.0 | 0 | 2,520 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 164.0 | 82.1 | 166 ms | 12.2 ms | 2.0 | 0 | 4,920 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 311.0 | 77.9 | 304 ms | 12.8 ms | 4.0 | 0 | 9,332 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 564.5 | 70.7 | 480 ms | 14.2 ms | 8.0 | 0 | 16,936 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 970.4 | 60.8 | 856 ms | 16.5 ms | 16.0 | 0 | 29,104 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,477.8 | 46.0 | 1,336 ms | 21.7 ms | 32.0 | 0 | 44,320 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,044.9 | 31.5 | 2,248 ms | 31.8 ms | 64.0 | 0 | 61,312 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 82.2 | 82.3 | 107 ms | 12.2 ms | 1.0 | 0 | 2,467 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.4 | 79.3 | 173 ms | 12.6 ms | 2.0 | 0 | 4,752 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 293.0 | 73.4 | 248 ms | 13.6 ms | 4.0 | 0 | 8,792 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 511.4 | 63.9 | 310 ms | 15.6 ms | 8.0 | 0 | 15,336 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 827.2 | 51.6 | 470 ms | 19.4 ms | 16.0 | 0 | 24,816 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,146.5 | 35.6 | 796 ms | 28.1 ms | 32.0 | 0 | 34,400 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 78.9 | 79.0 | 164 ms | 12.7 ms | 1.0 | 0 | 2,367 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 148.2 | 74.1 | 233 ms | 13.5 ms | 2.0 | 0 | 4,446 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 264.2 | 66.1 | 334 ms | 15.1 ms | 4.0 | 0 | 7,924 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 430.9 | 53.8 | 489 ms | 18.6 ms | 8.0 | 0 | 12,920 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 631.3 | 39.3 | 764 ms | 25.4 ms | 16.0 | 0 | 18,928 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 73.3 | 73.5 | 278 ms | 13.6 ms | 1.0 | 0 | 2,201 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 131.1 | 65.7 | 405 ms | 15.2 ms | 2.0 | 0 | 3,934 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 220.1 | 55.1 | 575 ms | 18.2 ms | 4.0 | 0 | 6,604 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 323.7 | 40.5 | 926 ms | 24.7 ms | 8.0 | 0 | 9,712 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 64.4 | 64.5 | 654 ms | 15.5 ms | 1.0 | 0 | 1,932 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 107.2 | 53.6 | 856 ms | 18.7 ms | 2.0 | 0 | 3,214 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 163.5 | 40.8 | 1,309 ms | 24.5 ms | 4.0 | 0 | 4,904 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.