Speed test matrix
Qwen3.8-27B-NVFP4 on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev562+g9c2d21046 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev562+g9c2d21046
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9c2d21046bb3
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nemotron-ultra-vllm-9c2d21046/bin/vllm" serve nvidia/Qwen3.8-27B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.882494 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --mamba-cache-dtype auto --mamba-ssm-cache-dtype float32 --enable-prefix-caching --mamba-cache-mode align --linear-backend emulation --speculative-config '{"method":"mtp","num_speculative_tokens":3,"kv_cache_dtype":"bfloat16","attention_backend":"TRITON_ATTN"}' --quantization modelopt_mixedPrefill
prompt tok/s by context length8K context24.5K336 ms TTFT
16K context26.5K614 ms TTFT
32K context26.3K1,231 ms TTFT
64K context25.2K2,561 ms TTFT
128K context21.2K6,066 ms TTFT
251K context16.1K15,620 ms TTFT
256K context16K16,113 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 163.4131 ms TTFT | 313.2158 ms TTFT | 569.5162 ms TTFT | 1,131.7161 ms TTFT | 2,117.1252 ms TTFT | 3,718.5830 ms TTFT | 5,255.81,562 ms TTFT | 6,984.12,700 ms TTFT |
| 16K | 140.9164 ms TTFT | 279.3199 ms TTFT | 556.4215 ms TTFT | 1,070.9219 ms TTFT | 1,984.8343 ms TTFT | 3,443.71,143 ms TTFT | 4,857.71,675 ms TTFT | 6,337.22,904 ms TTFT |
| 32K | 132.3217 ms TTFT | 255.6211 ms TTFT | 525.3226 ms TTFT | 989.5358 ms TTFT | 1,864.4928 ms TTFT | 3,065.51,285 ms TTFT | 4,211.72,237 ms TTFT | 5,411.34,149 ms TTFT |
| 64K | 111.2300 ms TTFT | 233.0440 ms TTFT | 468.2439 ms TTFT | 848.2795 ms TTFT | 1,555.71,274 ms TTFT | 2,426.22,403 ms TTFT | 3,268.74,445 ms TTFT | Over capacity |
| 128K | 92.6482 ms TTFT | 187.0748 ms TTFT | 360.7917 ms TTFT | 640.71,466 ms TTFT | 1,170.92,559 ms TTFT | 1,758.64,759 ms TTFT | Over capacity | Over capacity |
| 251K | 61.6979 ms TTFT | 124.31,188 ms TTFT | 251.21,812 ms TTFT | 469.22,731 ms TTFT | 770.54,638 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24.5K | — | 336 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.5K | — | 614 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 26.3K | — | 1,231 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.2K | — | 2,561 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.2K | — | 6,066 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 16.1K | — | 15,620 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16K | — | 16,113 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 163.4 | 163.5 | 131 ms | 6.1 ms | 1.0 | 1 | 4,901 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 313.2 | 154.5 | 158 ms | 6.5 ms | 2.0 | 2 | 9,393 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 569.5 | 147.8 | 162 ms | 6.8 ms | 4.0 | 4 | 17,079 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,131.7 | 143.7 | 161 ms | 7.0 ms | 8.0 | 8 | 33,954 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,117.1 | 135.4 | 252 ms | 7.4 ms | 15.9 | 16 | 63,515 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,718.5 | 115.7 | 830 ms | 8.6 ms | 32.0 | 3 | 111,530 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,255.8 | 81.1 | 1,562 ms | 12.3 ms | 64.0 | 0 | 157,899 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,984.1 | 53.1 | 2,700 ms | 18.8 ms | 128.0 | 0 | 209,631 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 140.9 | 144.6 | 164 ms | 6.9 ms | 1.0 | 1 | 4,228 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 279.3 | 142.8 | 199 ms | 7.0 ms | 2.0 | 2 | 8,377 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 556.4 | 148.3 | 215 ms | 6.7 ms | 4.0 | 4 | 16,689 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,070.9 | 136.2 | 219 ms | 7.3 ms | 8.0 | 8 | 32,118 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,984.8 | 122.2 | 343 ms | 8.2 ms | 16.0 | 15 | 59,475 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,443.7 | 107.8 | 1,143 ms | 9.3 ms | 32.0 | 0 | 103,300 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,857.7 | 73.9 | 1,675 ms | 13.5 ms | 64.0 | 0 | 145,688 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,337.2 | 47.7 | 2,904 ms | 21.0 ms | 128.0 | 0 | 189,978 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 132.3 | 145.8 | 217 ms | 6.9 ms | 1.0 | 1 | 3,968 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 255.6 | 136.5 | 211 ms | 7.3 ms | 2.0 | 2 | 7,667 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 525.3 | 132.8 | 226 ms | 7.5 ms | 4.0 | 4 | 15,754 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 989.5 | 126.4 | 358 ms | 7.9 ms | 7.9 | 6 | 29,674 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,864.4 | 116.1 | 928 ms | 8.6 ms | 16.0 | 0 | 55,926 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,065.5 | 95.4 | 1,285 ms | 10.5 ms | 32.0 | 0 | 91,927 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,211.7 | 63.8 | 2,237 ms | 15.7 ms | 64.0 | 0 | 126,376 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,411.3 | 40.5 | 4,149 ms | 24.7 ms | 128.0 | 0 | 162,479 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 111.2 | 112.2 | 300 ms | 8.9 ms | 1.0 | 0 | 3,336 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 233.0 | 117.6 | 440 ms | 8.5 ms | 2.0 | 0 | 6,988 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 468.2 | 120.8 | 439 ms | 8.3 ms | 4.0 | 2 | 14,015 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 848.2 | 105.6 | 795 ms | 9.5 ms | 8.0 | 0 | 25,449 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,555.7 | 95.6 | 1,274 ms | 10.5 ms | 16.0 | 0 | 46,652 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,426.2 | 73.8 | 2,403 ms | 13.6 ms | 32.0 | 0 | 72,803 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,268.7 | 48.9 | 4,445 ms | 20.5 ms | 64.0 | 0 | 97,976 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 92.6 | 93.7 | 482 ms | 10.7 ms | 1.0 | 0 | 2,777 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 187.0 | 94.3 | 748 ms | 10.6 ms | 2.0 | 0 | 5,610 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 360.7 | 90.3 | 917 ms | 11.1 ms | 4.0 | 0 | 10,812 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 640.7 | 80.5 | 1,466 ms | 12.4 ms | 8.0 | 0 | 19,243 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,170.9 | 72.5 | 2,559 ms | 13.8 ms | 16.0 | 0 | 35,109 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,758.6 | 54.4 | 4,759 ms | 18.4 ms | 32.0 | 0 | 52,729 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 61.6 | 62.7 | 979 ms | 16.0 ms | 1.0 | 0 | 1,846 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 124.3 | 62.7 | 1,188 ms | 16.0 ms | 2.0 | 0 | 3,726 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 251.2 | 62.4 | 1,812 ms | 16.0 ms | 4.0 | 0 | 7,542 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 469.2 | 58.7 | 2,731 ms | 17.0 ms | 8.0 | 0 | 14,069 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 770.5 | 48.2 | 4,638 ms | 20.7 ms | 16.0 | 0 | 23,086 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.