Speed test matrix
Qwen3.6-35B-A3B-NVFP4 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_NVFP4_GEMM_BACKEND=marlin "$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881578 --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-backend marlin
Prefill
prompt tok/s by context length8K context30.8K266 ms TTFT
16K context30.1K540 ms TTFT
32K context31.1K1,037 ms TTFT
64K context29.3K2,196 ms TTFT
128K context23.8K5,411 ms TTFT
251K context17.3K14,617 ms TTFT
256K context17.1K15,064 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 202.5107 ms TTFT | 377.8157 ms TTFT | 697.0962 ms TTFT | 1,201.7309 ms TTFT | 1,906.5483 ms TTFT | 2,975.4861 ms TTFT | 4,253.31,120 ms TTFT | 5,833.61,843 ms TTFT |
| 16K | 200.1122 ms TTFT | 372.2151 ms TTFT | 680.6230 ms TTFT | 1,155.4308 ms TTFT | 1,822.1389 ms TTFT | 2,743.4592 ms TTFT | 3,816.3859 ms TTFT | 5,042.11,460 ms TTFT |
| 32K | 194.5163 ms TTFT | 363.0233 ms TTFT | 653.5311 ms TTFT | 1,073.7377 ms TTFT | 1,644.6602 ms TTFT | 2,396.8892 ms TTFT | 3,136.81,463 ms TTFT | 3,837.02,715 ms TTFT |
| 64K | 186.3194 ms TTFT | 345.1279 ms TTFT | 597.7263 ms TTFT | 959.1484 ms TTFT | 1,393.1928 ms TTFT | 1,889.51,625 ms TTFT | 2,268.32,769 ms TTFT | Over capacity |
| 128K | 169.9361 ms TTFT | 314.5430 ms TTFT | 517.4457 ms TTFT | 793.61,042 ms TTFT | 1,052.51,744 ms TTFT | 1,273.03,310 ms TTFT | Over capacity | Over capacity |
| 251K | 145.8678 ms TTFT | 267.6853 ms TTFT | 420.21,271 ms TTFT | 583.61,962 ms TTFT | 689.33,336 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.8K | — | 266 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.1K | — | 540 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.1K | — | 1,037 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.3K | — | 2,196 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.8K | — | 5,411 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.3K | — | 14,617 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.1K | — | 15,064 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 202.5 | 203.6 | 107 ms | 4.9 ms | 1.0 | 1 | 6,074 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 377.8 | 190.2 | 157 ms | 5.3 ms | 1.9 | 2 | 11,334 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 697.0 | 176.2 | 962 ms | 5.7 ms | 4.0 | 4 | 20,908 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,201.7 | 152.5 | 309 ms | 6.6 ms | 8.0 | 8 | 36,044 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,906.5 | 120.9 | 483 ms | 8.3 ms | 16.0 | 16 | 57,168 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,975.4 | 93.9 | 861 ms | 10.6 ms | 32.0 | 0 | 89,270 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,253.3 | 66.3 | 1,120 ms | 15.1 ms | 64.0 | 0 | 127,616 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,833.6 | 44.7 | 1,843 ms | 22.4 ms | 128.0 | 0 | 174,974 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 200.1 | 201.2 | 122 ms | 5.0 ms | 1.0 | 1 | 6,002 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 372.2 | 187.4 | 151 ms | 5.3 ms | 2.0 | 2 | 11,166 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 680.6 | 171.8 | 230 ms | 5.8 ms | 4.0 | 4 | 20,416 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,155.4 | 146.8 | 308 ms | 6.8 ms | 8.0 | 8 | 34,658 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,822.1 | 114.5 | 389 ms | 8.7 ms | 16.0 | 0 | 54,656 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,743.4 | 86.1 | 592 ms | 11.6 ms | 32.0 | 0 | 82,304 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,816.3 | 59.4 | 859 ms | 16.8 ms | 64.0 | 0 | 114,497 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,042.1 | 38.8 | 1,460 ms | 25.8 ms | 128.0 | 0 | 151,296 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 194.5 | 195.9 | 163 ms | 5.1 ms | 1.0 | 1 | 5,834 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 363.0 | 183.5 | 233 ms | 5.4 ms | 2.0 | 2 | 10,888 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 653.5 | 165.8 | 311 ms | 6.0 ms | 4.0 | 4 | 19,606 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,073.7 | 136.3 | 377 ms | 7.3 ms | 8.0 | 8 | 32,209 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,644.6 | 102.9 | 602 ms | 9.7 ms | 16.0 | 0 | 49,328 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,396.8 | 74.5 | 892 ms | 13.4 ms | 32.0 | 0 | 71,904 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,136.8 | 48.2 | 1,463 ms | 20.8 ms | 64.0 | 0 | 94,080 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,837.0 | 29.0 | 2,715 ms | 34.4 ms | 128.0 | 0 | 115,072 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 186.3 | 188.0 | 194 ms | 5.3 ms | 1.0 | 1 | 5,590 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 345.1 | 174.9 | 279 ms | 5.7 ms | 1.9 | 2 | 10,353 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 597.7 | 151.3 | 263 ms | 6.6 ms | 4.0 | 4 | 17,930 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 959.1 | 120.3 | 484 ms | 8.3 ms | 8.0 | 3 | 28,761 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,393.1 | 87.2 | 928 ms | 11.5 ms | 16.0 | 0 | 41,792 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,889.5 | 59.1 | 1,625 ms | 16.9 ms | 32.0 | 0 | 56,672 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,268.3 | 35.6 | 2,769 ms | 28.1 ms | 64.0 | 0 | 68,032 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 169.9 | 172.6 | 361 ms | 5.8 ms | 1.0 | 1 | 5,096 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 314.5 | 159.8 | 430 ms | 6.3 ms | 2.0 | 2 | 9,436 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 517.4 | 131.3 | 457 ms | 7.6 ms | 3.9 | 4 | 15,520 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 793.6 | 98.8 | 1,042 ms | 10.1 ms | 8.0 | 0 | 23,808 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,052.5 | 65.1 | 1,744 ms | 15.4 ms | 16.0 | 0 | 31,568 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,273.0 | 38.9 | 3,310 ms | 25.7 ms | 32.0 | 0 | 38,176 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 145.8 | 151.0 | 678 ms | 6.6 ms | 1.0 | 1 | 4,377 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 267.6 | 138.7 | 853 ms | 7.2 ms | 1.9 | 2 | 8,029 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 420.2 | 105.1 | 1,271 ms | 9.5 ms | 4.0 | 0 | 12,604 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 583.6 | 72.9 | 1,962 ms | 13.7 ms | 8.0 | 0 | 17,504 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 689.3 | 42.9 | 3,336 ms | 23.3 ms | 16.0 | 0 | 20,672 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.