Speed test matrix
Qwen3.6-27B-FP8 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881925 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context14.5K565 ms TTFT
16K context14K1,161 ms TTFT
32K context13.2K2,446 ms TTFT
64K context11.8K5,457 ms TTFT
128K context9,41213,674 ms TTFT
251K context6,71937,533 ms TTFT
256K context6,68438,484 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 86.669 ms TTFT | 165.5121 ms TTFT | 318.61,820 ms TTFT | 590.2278 ms TTFT | 1,049.8547 ms TTFT | 1,688.0951 ms TTFT | 2,554.01,704 ms TTFT | Over capacity |
| 16K | 85.6108 ms TTFT | 162.6170 ms TTFT | 309.1263 ms TTFT | 558.7425 ms TTFT | 959.6745 ms TTFT | 1,467.71,304 ms TTFT | 2,039.11,933 ms TTFT | Over capacity |
| 32K | 83.8118 ms TTFT | 157.1182 ms TTFT | 291.5267 ms TTFT | 507.3326 ms TTFT | 818.5503 ms TTFT | 1,142.4788 ms TTFT | Over capacity | Over capacity |
| 64K | 80.4183 ms TTFT | 147.1251 ms TTFT | 263.2371 ms TTFT | 427.8433 ms TTFT | 629.1704 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 74.6275 ms TTFT | 130.2391 ms TTFT | 219.1563 ms TTFT | 322.3902 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 65.3653 ms TTFT | 106.5861 ms TTFT | 163.51,263 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.5K | — | 565 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14K | — | 1,161 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.2K | — | 2,446 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.8K | — | 5,457 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,412.0 | — | 13,674 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,719.0 | — | 37,533 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,684.0 | — | 38,484 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 86.6 | 86.6 | 69 ms | 11.5 ms | 1.0 | 0 | 2,598 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 165.5 | 82.8 | 121 ms | 12.1 ms | 2.0 | 0 | 4,964 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 318.6 | 79.7 | 1,820 ms | 12.5 ms | 4.0 | 0 | 9,556 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 590.2 | 73.9 | 278 ms | 13.5 ms | 8.0 | 0 | 17,705 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,049.8 | 65.9 | 547 ms | 15.2 ms | 16.0 | 0 | 31,488 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,688.0 | 53.0 | 951 ms | 18.9 ms | 32.0 | 0 | 50,624 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,554.0 | 40.1 | 1,704 ms | 25.0 ms | 64.0 | 0 | 76,608 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 85.6 | 85.6 | 108 ms | 11.7 ms | 1.0 | 0 | 2,568 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 162.6 | 81.4 | 170 ms | 12.3 ms | 2.0 | 0 | 4,878 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 309.1 | 77.4 | 263 ms | 12.9 ms | 4.0 | 0 | 9,272 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 558.7 | 69.9 | 425 ms | 14.3 ms | 8.0 | 0 | 16,760 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 959.6 | 60.0 | 745 ms | 16.7 ms | 16.0 | 0 | 28,784 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,467.7 | 45.8 | 1,304 ms | 21.8 ms | 32.0 | 0 | 44,032 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,039.1 | 31.3 | 1,933 ms | 31.9 ms | 64.0 | 0 | 61,184 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 83.8 | 83.9 | 118 ms | 11.9 ms | 1.0 | 0 | 2,514 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 157.1 | 78.7 | 182 ms | 12.7 ms | 2.0 | 0 | 4,712 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 291.5 | 73.0 | 267 ms | 13.7 ms | 4.0 | 0 | 8,748 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 507.3 | 63.4 | 326 ms | 15.8 ms | 8.0 | 0 | 15,224 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 818.5 | 51.1 | 503 ms | 19.6 ms | 16.0 | 0 | 24,544 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,142.4 | 35.6 | 788 ms | 28.1 ms | 32.0 | 0 | 34,272 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 80.4 | 80.5 | 183 ms | 12.4 ms | 1.0 | 0 | 2,413 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 147.1 | 73.6 | 251 ms | 13.6 ms | 2.0 | 0 | 4,412 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 263.2 | 65.8 | 371 ms | 15.2 ms | 4.0 | 0 | 7,892 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 427.8 | 53.4 | 433 ms | 18.7 ms | 8.0 | 0 | 12,832 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 629.1 | 39.2 | 704 ms | 25.5 ms | 16.0 | 0 | 18,864 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 74.6 | 74.7 | 275 ms | 13.4 ms | 1.0 | 0 | 2,238 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 130.2 | 65.2 | 391 ms | 15.3 ms | 2.0 | 0 | 3,906 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 219.1 | 54.8 | 563 ms | 18.2 ms | 4.0 | 0 | 6,572 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 322.3 | 40.3 | 902 ms | 24.8 ms | 8.0 | 0 | 9,664 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 65.3 | 65.4 | 653 ms | 15.3 ms | 1.0 | 0 | 1,958 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 106.5 | 53.3 | 861 ms | 18.8 ms | 2.0 | 0 | 3,194 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 163.5 | 40.8 | 1,263 ms | 24.5 ms | 4.0 | 0 | 4,904 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.