Speed test matrix
Qwen3.6-35B-A3B-NVFP4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_NVFP4_GEMM_BACKEND=marlin "$HOME/env_h200-nvidia-vllm/bin/vllm" serve nvidia/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880000 --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-backend marlin --quantization modelopt
Prefill
prompt tok/s by context length8K context34.3K239 ms TTFT
16K context32.6K498 ms TTFT
32K context34.4K938 ms TTFT
64K context32.6K1,976 ms TTFT
128K context25.7K4,999 ms TTFT
251K context18.2K13,826 ms TTFT
256K context17.9K14,390 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 215.593 ms TTFT | 415.7154 ms TTFT | 763.9968 ms TTFT | 1,296.2307 ms TTFT | 2,055.9396 ms TTFT | 3,147.8592 ms TTFT | 4,436.6965 ms TTFT | 5,941.71,555 ms TTFT |
| 16K | 212.6115 ms TTFT | 409.9140 ms TTFT | 740.3181 ms TTFT | 1,254.9247 ms TTFT | 1,922.0330 ms TTFT | 2,893.1480 ms TTFT | 3,947.1810 ms TTFT | 5,132.11,420 ms TTFT |
| 32K | 206.5158 ms TTFT | 399.9206 ms TTFT | 712.1301 ms TTFT | 1,162.6341 ms TTFT | 1,745.1562 ms TTFT | 2,473.8869 ms TTFT | 3,237.71,406 ms TTFT | 3,930.82,493 ms TTFT |
| 64K | 196.1197 ms TTFT | 379.4227 ms TTFT | 652.1251 ms TTFT | 1,024.5235 ms TTFT | 1,468.1855 ms TTFT | 1,957.61,501 ms TTFT | 2,348.22,713 ms TTFT | Over capacity |
| 128K | 178.2350 ms TTFT | 341.9383 ms TTFT | 552.9448 ms TTFT | 841.0981 ms TTFT | 1,117.41,783 ms TTFT | 1,339.63,481 ms TTFT | Over capacity | Over capacity |
| 251K | 152.1647 ms TTFT | 288.5769 ms TTFT | 444.01,205 ms TTFT | 608.71,835 ms TTFT | 714.33,305 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.3K | — | 239 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.6K | — | 498 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 34.4K | — | 938 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 32.6K | — | 1,976 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 25.7K | — | 4,999 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.2K | — | 13,826 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.9K | — | 14,390 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 215.5 | 216.4 | 93 ms | 4.6 ms | 1.0 | 1 | 6,466 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 415.7 | 209.3 | 154 ms | 4.8 ms | 1.9 | 2 | 12,472 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 763.9 | 192.8 | 968 ms | 5.2 ms | 4.0 | 4 | 22,920 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,296.2 | 164.7 | 307 ms | 6.1 ms | 8.0 | 8 | 38,882 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,055.9 | 130.6 | 396 ms | 7.7 ms | 15.7 | 16 | 61,671 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,147.8 | 98.7 | 592 ms | 10.1 ms | 32.0 | 0 | 94,432 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,436.6 | 69.2 | 965 ms | 14.5 ms | 64.0 | 0 | 133,056 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,941.7 | 45.6 | 1,555 ms | 21.9 ms | 128.0 | 0 | 178,302 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 212.6 | 213.7 | 115 ms | 4.7 ms | 1.0 | 1 | 6,377 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 409.9 | 206.2 | 140 ms | 4.8 ms | 2.0 | 2 | 12,296 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 740.3 | 186.9 | 181 ms | 5.4 ms | 4.0 | 4 | 22,209 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,254.9 | 159.1 | 247 ms | 6.3 ms | 8.0 | 8 | 37,645 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,922.0 | 122.2 | 330 ms | 8.2 ms | 15.8 | 16 | 57,657 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,893.1 | 90.6 | 480 ms | 11.0 ms | 32.0 | 0 | 86,784 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,947.1 | 61.5 | 810 ms | 16.3 ms | 64.0 | 0 | 118,400 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,132.1 | 39.5 | 1,420 ms | 25.3 ms | 128.0 | 0 | 153,984 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 206.5 | 208.1 | 158 ms | 4.8 ms | 1.0 | 1 | 6,196 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 399.9 | 201.6 | 206 ms | 5.0 ms | 2.0 | 2 | 11,996 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 712.1 | 180.1 | 301 ms | 5.6 ms | 4.0 | 4 | 21,362 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,162.6 | 147.3 | 341 ms | 6.8 ms | 8.0 | 8 | 34,878 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,745.1 | 109.3 | 562 ms | 9.1 ms | 16.0 | 0 | 52,352 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,473.8 | 76.9 | 869 ms | 13.0 ms | 32.0 | 0 | 74,208 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,237.7 | 49.8 | 1,406 ms | 20.1 ms | 64.0 | 0 | 97,150 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,930.8 | 29.8 | 2,493 ms | 33.5 ms | 128.0 | 0 | 118,016 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 196.1 | 197.9 | 197 ms | 5.1 ms | 1.0 | 1 | 5,883 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 379.4 | 191.9 | 227 ms | 5.2 ms | 2.0 | 2 | 11,383 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 652.1 | 164.7 | 251 ms | 6.1 ms | 4.0 | 4 | 19,564 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,024.5 | 129.0 | 235 ms | 7.8 ms | 7.9 | 8 | 30,733 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,468.1 | 92.0 | 855 ms | 10.9 ms | 16.0 | 0 | 44,032 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,957.6 | 61.3 | 1,501 ms | 16.3 ms | 32.0 | 0 | 58,720 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,348.2 | 36.8 | 2,713 ms | 27.1 ms | 64.0 | 0 | 70,464 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 178.2 | 181.1 | 350 ms | 5.5 ms | 1.0 | 1 | 5,345 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 341.9 | 173.3 | 383 ms | 5.8 ms | 2.0 | 2 | 10,256 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 552.9 | 140.1 | 448 ms | 7.1 ms | 3.9 | 4 | 16,587 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 841.0 | 104.7 | 981 ms | 9.6 ms | 8.0 | 0 | 25,232 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,117.4 | 69.0 | 1,783 ms | 14.5 ms | 16.0 | 0 | 33,520 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,339.6 | 41.1 | 3,481 ms | 24.4 ms | 32.0 | 0 | 40,192 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 152.1 | 156.5 | 647 ms | 6.4 ms | 1.0 | 1 | 4,562 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 288.5 | 148.1 | 769 ms | 6.8 ms | 2.0 | 2 | 8,654 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 444.0 | 111.0 | 1,205 ms | 9.0 ms | 4.0 | 0 | 13,321 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 608.7 | 75.9 | 1,835 ms | 13.2 ms | 8.0 | 0 | 18,256 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 714.3 | 44.3 | 3,305 ms | 22.6 ms | 16.0 | 0 | 21,424 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.