Speed test matrix
Qwen3.6-27B-NVFP4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_NVFP4_GEMM_BACKEND=marlin "$HOME/env_h200-nvidia-vllm/bin/vllm" serve nvidia/Qwen3.6-27B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.884259 --host 0.0.0.0 --port 8000 --api-key <redacted> --quantization modelopt
Prefill
prompt tok/s by context length8K context6,6191,237 ms TTFT
16K context6,5442,479 ms TTFT
32K context6,2465,169 ms TTFT
64K context5,68011,342 ms TTFT
128K context4,80526,781 ms TTFT
251K context3,67768,587 ms TTFT
256K context3,64570,564 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 91.797 ms TTFT | 177.7153 ms TTFT | 338.01,818 ms TTFT | 614.6475 ms TTFT | 1,025.4991 ms TTFT | 1,503.81,749 ms TTFT | 1,968.83,395 ms TTFT | 2,301.05,126 ms TTFT |
| 16K | 90.0155 ms TTFT | 172.0238 ms TTFT | 320.5475 ms TTFT | 562.2707 ms TTFT | 887.61,556 ms TTFT | 1,221.42,618 ms TTFT | 1,513.14,092 ms TTFT | 1,693.86,800 ms TTFT |
| 32K | 86.8270 ms TTFT | 162.3416 ms TTFT | 292.4754 ms TTFT | 478.81,495 ms TTFT | 694.32,467 ms TTFT | 886.74,148 ms TTFT | 1,027.46,777 ms TTFT | Over capacity |
| 64K | 81.8188 ms TTFT | 145.8295 ms TTFT | 245.7417 ms TTFT | 366.1495 ms TTFT | 483.4872 ms TTFT | 570.41,486 ms TTFT | Over capacity | Over capacity |
| 128K | 73.2303 ms TTFT | 121.6458 ms TTFT | 187.7615 ms TTFT | 250.81,018 ms TTFT | 301.21,820 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 61.71,056 ms TTFT | 93.31,377 ms TTFT | 130.12,265 ms TTFT | 157.13,818 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,619.0 | — | 1,237 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,544.0 | — | 2,479 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,246.0 | — | 5,169 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,680.0 | — | 11,342 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,805.0 | — | 26,781 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,677.0 | — | 68,587 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,645.0 | — | 70,564 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 91.7 | 91.7 | 97 ms | 10.9 ms | 1.0 | 0 | 2,751 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 177.7 | 88.9 | 153 ms | 11.3 ms | 2.0 | 0 | 5,332 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 338.0 | 84.7 | 1,818 ms | 11.8 ms | 4.0 | 0 | 10,144 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 614.6 | 76.9 | 475 ms | 13.0 ms | 8.0 | 0 | 18,432 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,025.4 | 64.3 | 991 ms | 15.5 ms | 16.0 | 0 | 30,752 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,503.8 | 47.0 | 1,749 ms | 21.3 ms | 32.0 | 0 | 45,088 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,968.8 | 30.7 | 3,395 ms | 32.6 ms | 64.0 | 0 | 59,072 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,301.0 | 17.1 | 5,126 ms | 58.6 ms | 128.0 | 0 | 68,992 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 90.0 | 90.0 | 155 ms | 11.1 ms | 1.0 | 0 | 2,700 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 172.0 | 86.1 | 238 ms | 11.6 ms | 2.0 | 0 | 5,160 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 320.5 | 80.3 | 475 ms | 12.5 ms | 4.0 | 0 | 9,621 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 562.2 | 70.2 | 707 ms | 14.3 ms | 8.0 | 0 | 16,864 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 887.6 | 55.7 | 1,556 ms | 18.0 ms | 16.0 | 0 | 26,640 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,221.4 | 37.9 | 2,618 ms | 26.4 ms | 32.0 | 0 | 36,640 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,513.1 | 22.6 | 4,092 ms | 44.2 ms | 64.0 | 0 | 45,376 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,693.8 | 11.8 | 6,800 ms | 85.0 ms | 128.0 | 0 | 50,816 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 86.8 | 86.9 | 270 ms | 11.5 ms | 1.0 | 0 | 2,605 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 162.3 | 81.1 | 416 ms | 12.3 ms | 2.0 | 0 | 4,868 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 292.4 | 73.1 | 754 ms | 13.7 ms | 4.0 | 0 | 8,773 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 478.8 | 60.0 | 1,495 ms | 16.7 ms | 8.0 | 0 | 14,360 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 694.3 | 43.1 | 2,467 ms | 23.2 ms | 16.0 | 0 | 20,816 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 886.7 | 26.8 | 4,148 ms | 37.3 ms | 32.0 | 0 | 26,592 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,027.4 | 14.5 | 6,777 ms | 68.9 ms | 64.0 | 0 | 30,784 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 81.8 | 82.0 | 188 ms | 12.2 ms | 1.0 | 0 | 2,454 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 145.8 | 73.0 | 295 ms | 13.7 ms | 2.0 | 0 | 4,374 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 245.7 | 61.4 | 417 ms | 16.3 ms | 4.0 | 0 | 7,368 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 366.1 | 45.6 | 495 ms | 21.9 ms | 8.0 | 0 | 10,984 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 483.4 | 29.9 | 872 ms | 33.4 ms | 16.0 | 0 | 14,496 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 570.4 | 17.5 | 1,486 ms | 57.1 ms | 32.0 | 0 | 17,088 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 73.2 | 73.4 | 303 ms | 13.6 ms | 1.0 | 0 | 2,196 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 121.6 | 60.9 | 458 ms | 16.4 ms | 2.0 | 0 | 3,648 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 187.7 | 47.0 | 615 ms | 21.3 ms | 4.0 | 0 | 5,632 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 250.8 | 31.3 | 1,018 ms | 31.9 ms | 8.0 | 0 | 7,520 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 301.2 | 18.7 | 1,820 ms | 53.5 ms | 16.0 | 0 | 9,024 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 61.7 | 61.9 | 1,056 ms | 16.2 ms | 1.0 | 0 | 1,851 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 93.3 | 46.4 | 1,377 ms | 21.5 ms | 2.0 | 0 | 2,798 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 130.1 | 32.1 | 2,265 ms | 31.1 ms | 4.0 | 0 | 3,900 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 157.1 | 19.1 | 3,818 ms | 52.3 ms | 8.0 | 0 | 4,712 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.