Speed test matrix
Qwen3.6-27B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880982 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context14.6K562 ms TTFT
16K context14K1,159 ms TTFT
32K context13.2K2,443 ms TTFT
64K context11.8K5,455 ms TTFT
128K context9,45213,615 ms TTFT
251K context6,74337,402 ms TTFT
256K context6,70038,392 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 85.171 ms TTFT | 167.0120 ms TTFT | 320.91,742 ms TTFT | 596.7293 ms TTFT | 1,064.2539 ms TTFT | 1,706.3992 ms TTFT | 2,574.41,825 ms TTFT | Over capacity |
| 16K | 84.0127 ms TTFT | 164.1165 ms TTFT | 311.2261 ms TTFT | 564.6417 ms TTFT | 971.1748 ms TTFT | 1,481.91,240 ms TTFT | 2,050.01,913 ms TTFT | Over capacity |
| 32K | 82.2117 ms TTFT | 158.4164 ms TTFT | 293.3256 ms TTFT | 512.0314 ms TTFT | 828.1501 ms TTFT | 1,148.8790 ms TTFT | Over capacity | Over capacity |
| 64K | 79.0184 ms TTFT | 148.3251 ms TTFT | 264.6389 ms TTFT | 431.4532 ms TTFT | 631.8792 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 73.5274 ms TTFT | 131.2403 ms TTFT | 220.3574 ms TTFT | 323.9911 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 64.4657 ms TTFT | 107.1878 ms TTFT | 163.71,220 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.6K | — | 562 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14K | — | 1,159 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.2K | — | 2,443 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.8K | — | 5,455 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,452.0 | — | 13,615 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,743.0 | — | 37,402 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,700.0 | — | 38,392 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 85.1 | 85.1 | 71 ms | 11.8 ms | 1.0 | 0 | 2,552 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 167.0 | 83.5 | 120 ms | 12.0 ms | 2.0 | 0 | 5,008 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 320.9 | 80.3 | 1,742 ms | 12.5 ms | 4.0 | 0 | 9,624 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 596.7 | 74.7 | 293 ms | 13.4 ms | 8.0 | 0 | 17,913 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,064.2 | 66.7 | 539 ms | 15.0 ms | 16.0 | 0 | 31,920 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,706.3 | 53.6 | 992 ms | 18.7 ms | 32.0 | 0 | 51,168 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,574.4 | 40.5 | 1,825 ms | 24.7 ms | 64.0 | 0 | 77,248 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 84.0 | 84.1 | 127 ms | 11.9 ms | 1.0 | 0 | 2,520 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 164.1 | 82.1 | 165 ms | 12.2 ms | 2.0 | 0 | 4,922 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 311.2 | 77.9 | 261 ms | 12.8 ms | 4.0 | 0 | 9,340 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 564.6 | 70.6 | 417 ms | 14.2 ms | 8.0 | 0 | 16,936 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 971.1 | 60.7 | 748 ms | 16.5 ms | 16.0 | 0 | 29,120 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,481.9 | 46.1 | 1,240 ms | 21.7 ms | 32.0 | 0 | 44,448 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,050.0 | 31.4 | 1,913 ms | 31.8 ms | 64.0 | 0 | 61,503 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 82.2 | 82.3 | 117 ms | 12.1 ms | 1.0 | 0 | 2,467 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.4 | 79.3 | 164 ms | 12.6 ms | 2.0 | 0 | 4,750 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 293.3 | 73.4 | 256 ms | 13.6 ms | 4.0 | 0 | 8,796 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 512.0 | 64.0 | 314 ms | 15.6 ms | 8.0 | 0 | 15,360 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 828.1 | 51.7 | 501 ms | 19.3 ms | 16.0 | 0 | 24,832 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,148.8 | 35.7 | 790 ms | 28.0 ms | 32.0 | 0 | 34,464 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 79.0 | 79.1 | 184 ms | 12.6 ms | 1.0 | 0 | 2,369 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 148.3 | 74.2 | 251 ms | 13.5 ms | 2.0 | 0 | 4,450 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 264.6 | 66.2 | 389 ms | 15.1 ms | 4.0 | 0 | 7,940 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 431.4 | 53.9 | 532 ms | 18.5 ms | 8.0 | 0 | 12,944 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 631.8 | 39.4 | 792 ms | 25.4 ms | 16.0 | 0 | 18,944 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 73.5 | 73.6 | 274 ms | 13.6 ms | 1.0 | 0 | 2,204 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 131.2 | 65.7 | 403 ms | 15.2 ms | 2.0 | 0 | 3,936 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 220.3 | 55.1 | 574 ms | 18.1 ms | 4.0 | 0 | 6,608 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 323.9 | 40.5 | 911 ms | 24.7 ms | 8.0 | 0 | 9,720 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 64.4 | 64.6 | 657 ms | 15.5 ms | 1.0 | 0 | 1,932 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 107.1 | 53.6 | 878 ms | 18.7 ms | 2.0 | 0 | 3,214 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 163.7 | 40.8 | 1,220 ms | 24.5 ms | 4.0 | 0 | 4,912 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.