Speed test matrix
Qwen3.6-35B-A3B-FP8 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881816 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context40.4K203 ms TTFT
16K context38.3K423 ms TTFT
32K context41.4K779 ms TTFT
64K context38.6K1,670 ms TTFT
128K context29.6K4,350 ms TTFT
251K context20K12,579 ms TTFT
256K context19.2K13,363 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 215.7108 ms TTFT | 405.2167 ms TTFT | 722.7924 ms TTFT | 1,214.5270 ms TTFT | 1,902.6898 ms TTFT | 2,997.7633 ms TTFT | 4,645.9880 ms TTFT | 6,506.01,343 ms TTFT |
| 16K | 213.3114 ms TTFT | 400.9161 ms TTFT | 704.0229 ms TTFT | 1,166.3296 ms TTFT | 1,814.9394 ms TTFT | 2,776.4547 ms TTFT | 4,137.8949 ms TTFT | 5,558.41,382 ms TTFT |
| 32K | 207.4166 ms TTFT | 389.0275 ms TTFT | 673.5294 ms TTFT | 1,090.0377 ms TTFT | 1,658.2587 ms TTFT | 2,426.1852 ms TTFT | 3,354.31,391 ms TTFT | Over capacity |
| 64K | 196.7185 ms TTFT | 370.1209 ms TTFT | 620.4217 ms TTFT | 970.6254 ms TTFT | 1,407.2918 ms TTFT | 1,924.21,514 ms TTFT | Over capacity | Over capacity |
| 128K | 179.8337 ms TTFT | 334.4428 ms TTFT | 534.5450 ms TTFT | 798.0969 ms TTFT | 1,068.61,665 ms TTFT | 1,304.53,105 ms TTFT | Over capacity | Over capacity |
| 251K | 153.1612 ms TTFT | 283.8713 ms TTFT | 431.61,199 ms TTFT | 595.91,950 ms TTFT | 716.53,372 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.4K | — | 203 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.3K | — | 423 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 41.4K | — | 779 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 38.6K | — | 1,670 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29.6K | — | 4,350 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 20K | — | 12,579 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.2K | — | 13,363 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 215.7 | 216.8 | 108 ms | 4.6 ms | 1.0 | 1 | 6,471 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 405.2 | 204.0 | 167 ms | 4.9 ms | 2.0 | 2 | 12,156 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 722.7 | 182.2 | 924 ms | 5.5 ms | 4.0 | 4 | 21,680 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,214.5 | 153.5 | 270 ms | 6.5 ms | 8.0 | 8 | 36,434 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,902.6 | 120.2 | 898 ms | 8.3 ms | 16.0 | 0 | 57,080 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,997.7 | 94.8 | 633 ms | 10.5 ms | 32.0 | 0 | 89,920 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,645.9 | 72.9 | 880 ms | 13.7 ms | 64.0 | 0 | 139,392 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,506.0 | 50.5 | 1,343 ms | 19.8 ms | 128.0 | 0 | 195,200 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 213.3 | 214.4 | 114 ms | 4.7 ms | 1.0 | 1 | 6,398 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 400.9 | 201.8 | 161 ms | 5.0 ms | 2.0 | 2 | 12,026 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 704.0 | 177.9 | 229 ms | 5.6 ms | 4.0 | 4 | 21,121 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,166.3 | 148.1 | 296 ms | 6.8 ms | 8.0 | 8 | 34,986 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,814.9 | 114.4 | 394 ms | 8.7 ms | 16.0 | 0 | 54,448 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,776.4 | 87.2 | 547 ms | 11.5 ms | 32.0 | 0 | 83,276 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,137.8 | 64.5 | 949 ms | 15.5 ms | 64.0 | 0 | 124,160 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,558.4 | 43.0 | 1,382 ms | 23.3 ms | 128.0 | 0 | 166,784 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 207.4 | 209.0 | 166 ms | 4.8 ms | 1.0 | 1 | 6,222 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 389.0 | 196.8 | 275 ms | 5.1 ms | 2.0 | 2 | 11,672 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 673.5 | 170.3 | 294 ms | 5.9 ms | 4.0 | 4 | 20,201 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,090.0 | 137.9 | 377 ms | 7.3 ms | 8.0 | 8 | 32,702 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,658.2 | 103.9 | 587 ms | 9.6 ms | 16.0 | 0 | 49,744 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,426.1 | 75.4 | 852 ms | 13.3 ms | 32.0 | 0 | 72,768 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,354.3 | 51.7 | 1,391 ms | 19.3 ms | 64.0 | 0 | 100,608 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 196.7 | 198.5 | 185 ms | 5.0 ms | 1.0 | 1 | 5,904 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 370.1 | 186.8 | 209 ms | 5.4 ms | 2.0 | 2 | 11,102 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 620.4 | 156.6 | 217 ms | 6.4 ms | 4.0 | 4 | 18,612 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 970.6 | 122.3 | 254 ms | 8.2 ms | 8.0 | 8 | 29,129 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,407.2 | 88.0 | 918 ms | 11.4 ms | 16.0 | 0 | 42,208 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,924.2 | 60.2 | 1,514 ms | 16.6 ms | 32.0 | 0 | 57,727 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 179.8 | 182.4 | 337 ms | 5.5 ms | 1.0 | 1 | 5,393 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 334.4 | 169.7 | 428 ms | 5.9 ms | 2.0 | 2 | 10,031 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 534.5 | 135.6 | 450 ms | 7.4 ms | 3.9 | 4 | 16,034 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 798.0 | 99.5 | 969 ms | 10.1 ms | 8.0 | 0 | 23,945 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,068.6 | 66.2 | 1,665 ms | 15.1 ms | 16.0 | 0 | 32,048 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,304.5 | 40.1 | 3,105 ms | 24.9 ms | 32.0 | 0 | 39,137 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 153.1 | 157.3 | 612 ms | 6.4 ms | 1.0 | 1 | 4,592 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 283.8 | 145.4 | 713 ms | 6.9 ms | 1.9 | 2 | 8,514 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 431.6 | 107.9 | 1,199 ms | 9.3 ms | 4.0 | 0 | 12,944 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 595.9 | 74.4 | 1,950 ms | 13.4 ms | 8.0 | 0 | 17,872 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 716.5 | 44.5 | 3,372 ms | 22.5 ms | 16.0 | 0 | 21,488 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.