Speed test matrix
Qwen3.5-397B-A17B-NVFP4 on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev562+g9c2d21046 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev562+g9c2d21046
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9c2d21046bb3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nemotron-ultra-vllm-9c2d21046/bin/vllm" serve nvidia/Qwen3.5-397B-A17B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.863133 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype fp8_e4m3 --attention-config '{"disable_flashinfer_q_quantization":true}' --mamba-cache-dtype auto --mamba-ssm-cache-dtype float32 --enable-prefix-caching --mamba-cache-mode align --attention-backend FLASHINFER --moe-backend emulation --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton","kv_cache_dtype":"bfloat16","attention_backend":"TRITON_ATTN"}' --quantization modelopt_fp4Prefill
prompt tok/s by context length8K context13.3K615 ms TTFT
16K context14.6K1,111 ms TTFT
32K context14K2,307 ms TTFT
64K context12.4K5,206 ms TTFT
128K context9,54713,480 ms TTFT
251K context6,55038,505 ms TTFT
256K context6,43739,956 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 57.4599 ms TTFT | 162.1875 ms TTFT | 307.52,263 ms TTFT | 505.02,444 ms TTFT | 802.04,302 ms TTFT | 1,073.48,057 ms TTFT | 1,608.715,588 ms TTFT | 1,863.2Requests queued29,970 ms TTFT |
| 16K | 59.21,212 ms TTFT | 165.31,511 ms TTFT | 302.91,695 ms TTFT | 496.52,668 ms TTFT | 782.04,528 ms TTFT | 1,062.58,268 ms TTFT | 1,570.615,751 ms TTFT | 2,249.8Requests queued31,007 ms TTFT |
| 32K | 57.1780 ms TTFT | 167.41,084 ms TTFT | 292.01,930 ms TTFT | 479.53,004 ms TTFT | 754.84,988 ms TTFT | 1,011.79,103 ms TTFT | 1,512.017,313 ms TTFT | 1,825.1Requests queued34,006 ms TTFT |
| 64K | 56.45,353 ms TTFT | 155.15,697 ms TTFT | 275.22,339 ms TTFT | 448.13,058 ms TTFT | 697.85,190 ms TTFT | 932.28,687 ms TTFT | 1,376.816,587 ms TTFT | 2,015.732,218 ms TTFT |
| 128K | 56.113,561 ms TTFT | 134.014,141 ms TTFT | 240.83,035 ms TTFT | 397.94,910 ms TTFT | 614.37,868 ms TTFT | 824.214,483 ms TTFT | 1,166.527,289 ms TTFT | 962.2Requests queued52,990 ms TTFT |
| 251K | 67.838,805 ms TTFT | 117.639,706 ms TTFT | 198.05,791 ms TTFT | 321.48,340 ms TTFT | 498.913,427 ms TTFT | 666.724,205 ms TTFT | 895.046,119 ms TTFT | 1,125.8Requests queued86,672 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.3K | — | 615 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.6K | — | 1,111 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14K | — | 2,307 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.4K | — | 5,206 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,547.0 | — | 13,480 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,550.0 | — | 38,505 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,437.0 | — | 39,956 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 57.4 | 58.1 | 599 ms | 17.2 ms | 1.0 | 0 | 1,719 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 162.1 | 81.1 | 875 ms | 12.3 ms | 2.0 | 0 | 4,864 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 307.5 | 68.7 | 2,263 ms | 14.6 ms | 4.0 | 0 | 9,219 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 505.0 | 59.8 | 2,444 ms | 16.7 ms | 8.0 | 0 | 15,147 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 802.0 | 44.7 | 4,302 ms | 22.4 ms | 16.0 | 0 | 24,044 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,073.4 | 29.4 | 8,057 ms | 34.0 ms | 32.0 | 0 | 32,222 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,608.7 | 19.7 | 15,588 ms | 50.6 ms | 64.0 | 0 | 48,120 | 29.9 s |
| Sustained decode | 8K | 128 | Requests queued | 1,863.2 | 11.9 | 29,970 ms | 84.2 ms | 120.9 | 0 | 55,809 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 59.2 | 59.4 | 1,212 ms | 16.8 ms | 1.0 | 0 | 1,773 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 165.3 | 82.8 | 1,511 ms | 12.1 ms | 2.0 | 0 | 4,959 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 302.9 | 74.8 | 1,695 ms | 13.4 ms | 4.0 | 0 | 9,084 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 496.5 | 60.9 | 2,668 ms | 16.4 ms | 8.0 | 0 | 14,885 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 782.0 | 45.9 | 4,528 ms | 21.8 ms | 16.0 | 0 | 23,436 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,062.5 | 29.2 | 8,268 ms | 34.3 ms | 32.0 | 0 | 31,828 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,570.6 | 19.4 | 15,751 ms | 51.5 ms | 64.0 | 0 | 46,976 | 29.9 s |
| Sustained decode | 16K | 128 | Requests queued | 2,249.8 | 12.4 | 31,007 ms | 80.7 ms | 127.2 | 0 | 67,211 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 57.1 | 57.5 | 780 ms | 17.4 ms | 1.0 | 0 | 1,713 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 167.4 | 83.3 | 1,084 ms | 12.0 ms | 2.0 | 0 | 5,017 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 292.0 | 73.7 | 1,930 ms | 13.6 ms | 4.0 | 0 | 8,754 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 479.5 | 58.8 | 3,004 ms | 17.0 ms | 8.0 | 0 | 14,371 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 754.8 | 44.6 | 4,988 ms | 22.4 ms | 16.0 | 0 | 22,616 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,011.7 | 27.9 | 9,103 ms | 35.9 ms | 32.0 | 0 | 30,280 | 29.9 s |
| Sustained decode | 32K | 64 | Valid | 1,512.0 | 18.4 | 17,313 ms | 54.4 ms | 64.0 | 0 | 45,313 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,825.1 | 10.6 | 34,006 ms | 94.7 ms | 125.8 | 0 | 54,690 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 56.4 | 57.4 | 5,353 ms | 17.4 ms | 1.0 | 0 | 1,693 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 155.1 | 78.5 | 5,697 ms | 12.7 ms | 2.0 | 0 | 4,652 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 275.2 | 68.9 | 2,339 ms | 14.5 ms | 4.0 | 0 | 8,247 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 448.1 | 55.1 | 3,058 ms | 18.1 ms | 8.0 | 0 | 13,442 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 697.8 | 41.4 | 5,190 ms | 24.2 ms | 16.0 | 0 | 20,893 | 29.9 s |
| Sustained decode | 64K | 32 | Valid | 932.2 | 25.6 | 8,687 ms | 39.0 ms | 32.0 | 0 | 27,906 | 29.9 s |
| Sustained decode | 64K | 64 | Valid | 1,376.8 | 16.3 | 16,587 ms | 61.3 ms | 64.0 | 0 | 41,317 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 2,015.7 | 10.3 | 32,218 ms | 96.9 ms | 128.0 | 0 | 60,325 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 56.1 | 57.1 | 13,561 ms | 17.5 ms | 1.0 | 0 | 1,682 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 134.0 | 68.8 | 14,141 ms | 14.5 ms | 2.0 | 0 | 4,022 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 240.8 | 57.8 | 3,035 ms | 17.3 ms | 4.0 | 0 | 7,220 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 397.9 | 47.6 | 4,910 ms | 21.0 ms | 8.0 | 0 | 11,924 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 614.3 | 34.7 | 7,868 ms | 28.8 ms | 16.0 | 0 | 18,387 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 824.2 | 20.4 | 14,483 ms | 49.0 ms | 32.0 | 0 | 24,719 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,166.5 | 11.9 | 27,289 ms | 83.7 ms | 64.0 | 0 | 34,936 | 29.9 s |
| Sustained decode | 128K | 128 | Requests queued | 962.2 | 6.2 | 52,990 ms | 161.0 ms | 126.6 | 0 | 28,635 | 29.8 s |
| Sustained decode | 251K | 1 | Valid | 67.8 | 68.7 | 38,805 ms | 14.6 ms | 1.0 | 0 | 2,030 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 117.6 | 59.9 | 39,706 ms | 16.7 ms | 2.0 | 0 | 3,526 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 198.0 | 48.6 | 5,791 ms | 20.6 ms | 4.0 | 0 | 5,933 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 321.4 | 36.8 | 8,340 ms | 27.2 ms | 8.0 | 0 | 9,639 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 498.9 | 25.3 | 13,427 ms | 39.6 ms | 16.0 | 0 | 14,951 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 666.7 | 14.2 | 24,205 ms | 70.7 ms | 32.0 | 0 | 19,914 | 29.9 s |
| Sustained decode | 251K | 64 | Valid | 895.0 | 7.4 | 46,119 ms | 135.4 ms | 64.0 | 0 | 26,853 | 30.0 s |
| Sustained decode | 251K | 128 | Requests queued | 1,125.8 | 4.2 | 86,672 ms | 235.9 ms | 124.0 | 0 | 33,752 | 30.0 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.