Speed test matrix
Qwen3.8-27B-NVFP4 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_NVFP4_GEMM_BACKEND=marlin "$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/Qwen3.8-27B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883050 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context6,9381,187 ms TTFT
16K context6,8022,391 ms TTFT
32K context6,5544,934 ms TTFT
64K context5,93310,865 ms TTFT
128K context4,98725,814 ms TTFT
251K context3,77966,749 ms TTFT
256K context3,75068,608 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 89.5110 ms TTFT | 169.0357 ms TTFT | 322.31,866 ms TTFT | 586.3546 ms TTFT | 989.2917 ms TTFT | 1,449.11,906 ms TTFT | 1,970.13,523 ms TTFT | 2,348.04,898 ms TTFT |
| 16K | 87.9173 ms TTFT | 163.7246 ms TTFT | 306.5477 ms TTFT | 538.2703 ms TTFT | 858.81,597 ms TTFT | 1,184.72,629 ms TTFT | 1,511.44,816 ms TTFT | 1,723.17,047 ms TTFT |
| 32K | 84.9266 ms TTFT | 154.6483 ms TTFT | 279.0828 ms TTFT | 458.81,348 ms TTFT | 679.02,559 ms TTFT | 867.74,315 ms TTFT | 1,029.66,761 ms TTFT | Over capacity |
| 64K | 79.7184 ms TTFT | 140.0325 ms TTFT | 237.3342 ms TTFT | 357.7540 ms TTFT | 475.7895 ms TTFT | 562.31,534 ms TTFT | Over capacity | Over capacity |
| 128K | 72.2283 ms TTFT | 118.1459 ms TTFT | 183.6660 ms TTFT | 246.21,054 ms TTFT | 297.81,742 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 60.81,054 ms TTFT | 90.81,388 ms TTFT | 127.62,410 ms TTFT | 155.14,346 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,938.0 | — | 1,187 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,802.0 | — | 2,391 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,554.0 | — | 4,934 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,933.0 | — | 10,865 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,987.0 | — | 25,814 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,779.0 | — | 66,749 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,750.0 | — | 68,608 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 89.5 | 89.5 | 110 ms | 11.2 ms | 1.0 | 0 | 2,685 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 169.0 | 84.6 | 357 ms | 11.8 ms | 2.0 | 0 | 5,070 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 322.3 | 80.7 | 1,866 ms | 12.4 ms | 4.0 | 0 | 9,672 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 586.3 | 73.4 | 546 ms | 13.6 ms | 8.0 | 0 | 17,592 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 989.2 | 61.8 | 917 ms | 16.2 ms | 16.0 | 0 | 29,664 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,449.1 | 45.3 | 1,906 ms | 22.1 ms | 32.0 | 0 | 43,456 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,970.1 | 30.6 | 3,523 ms | 32.7 ms | 64.0 | 0 | 59,072 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,348.0 | 17.1 | 4,898 ms | 58.6 ms | 128.0 | 0 | 70,400 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 87.9 | 87.9 | 173 ms | 11.4 ms | 1.0 | 0 | 2,636 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 163.7 | 81.9 | 246 ms | 12.2 ms | 2.0 | 0 | 4,912 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 306.5 | 76.8 | 477 ms | 13.0 ms | 4.0 | 0 | 9,196 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 538.2 | 67.2 | 703 ms | 14.9 ms | 8.0 | 0 | 16,152 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 858.8 | 53.9 | 1,597 ms | 18.6 ms | 16.0 | 0 | 25,760 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,184.7 | 36.8 | 2,629 ms | 27.2 ms | 32.0 | 0 | 35,520 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,511.4 | 23.0 | 4,816 ms | 43.4 ms | 64.0 | 0 | 45,312 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,723.1 | 12.0 | 7,047 ms | 83.6 ms | 128.0 | 0 | 51,712 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 84.9 | 84.9 | 266 ms | 11.8 ms | 1.0 | 0 | 2,545 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 154.6 | 77.4 | 483 ms | 12.9 ms | 2.0 | 0 | 4,638 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 279.0 | 69.9 | 828 ms | 14.3 ms | 4.0 | 0 | 8,374 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 458.8 | 57.3 | 1,348 ms | 17.5 ms | 8.0 | 0 | 13,768 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 679.0 | 42.2 | 2,559 ms | 23.7 ms | 16.0 | 0 | 20,368 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 867.7 | 26.3 | 4,315 ms | 38.0 ms | 32.0 | 0 | 26,016 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,029.6 | 14.5 | 6,761 ms | 68.8 ms | 64.0 | 0 | 30,848 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 79.7 | 79.8 | 184 ms | 12.5 ms | 1.0 | 0 | 2,391 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 140.0 | 70.1 | 325 ms | 14.3 ms | 2.0 | 0 | 4,200 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 237.3 | 59.3 | 342 ms | 16.9 ms | 4.0 | 0 | 7,116 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 357.7 | 44.6 | 540 ms | 22.4 ms | 8.0 | 0 | 10,728 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 475.7 | 29.4 | 895 ms | 34.0 ms | 16.0 | 0 | 14,272 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 562.3 | 17.3 | 1,534 ms | 57.9 ms | 32.0 | 0 | 16,864 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 72.2 | 72.3 | 283 ms | 13.8 ms | 1.0 | 0 | 2,165 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 118.1 | 59.1 | 459 ms | 16.9 ms | 2.0 | 0 | 3,542 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 183.6 | 45.9 | 660 ms | 21.8 ms | 4.0 | 0 | 5,508 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 246.2 | 30.7 | 1,054 ms | 32.6 ms | 8.0 | 0 | 7,384 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 297.8 | 18.4 | 1,742 ms | 54.2 ms | 16.0 | 0 | 8,928 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 60.8 | 61.0 | 1,054 ms | 16.4 ms | 1.0 | 0 | 1,824 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 90.8 | 45.2 | 1,388 ms | 22.1 ms | 2.0 | 0 | 2,724 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 127.6 | 31.4 | 2,410 ms | 31.9 ms | 4.0 | 0 | 3,828 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 155.1 | 18.8 | 4,346 ms | 53.2 ms | 8.0 | 0 | 4,648 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.