Speed test matrix
Qwen3.6-35B-A3B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880874 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context43.1K190 ms TTFT
16K context42.8K379 ms TTFT
32K context40.5K797 ms TTFT
64K context36.2K1,780 ms TTFT
128K context27.1K4,756 ms TTFT
251K context17.9K14,068 ms TTFT
256K context17.6K14,589 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 316.497 ms TTFT | 562.9168 ms TTFT | 1,034.6119 ms TTFT | 1,737.6187 ms TTFT | 2,766.9217 ms TTFT | 4,155.3252 ms TTFT | 6,066.51,374 ms TTFT | 7,688.12,537 ms TTFT |
| 16K | 305.6125 ms TTFT | 559.4143 ms TTFT | 1,016.1139 ms TTFT | 1,677.5178 ms TTFT | 2,688.2288 ms TTFT | 3,918.4372 ms TTFT | 5,662.81,241 ms TTFT | 6,966.22,127 ms TTFT |
| 32K | 300.8129 ms TTFT | 548.1161 ms TTFT | 986.4162 ms TTFT | 1,605.4233 ms TTFT | 2,480.1301 ms TTFT | 3,661.9891 ms TTFT | 4,818.61,630 ms TTFT | Over capacity |
| 64K | 291.6195 ms TTFT | 516.3221 ms TTFT | 923.6283 ms TTFT | 1,425.8338 ms TTFT | 2,142.2337 ms TTFT | 3,021.51,418 ms TTFT | Over capacity | Over capacity |
| 128K | 267.2341 ms TTFT | 492.6390 ms TTFT | 812.7422 ms TTFT | 1,233.8611 ms TTFT | 1,739.41,957 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 236.4672 ms TTFT | 422.6711 ms TTFT | 665.1961 ms TTFT | 948.71,479 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 43.1K | — | 190 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 42.8K | — | 379 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 40.5K | — | 797 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 36.2K | — | 1,780 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 27.1K | — | 4,756 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.9K | — | 14,068 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.6K | — | 14,589 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 316.4 | 319.2 | 97 ms | 3.1 ms | 1.0 | 2 | 9,490 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 562.9 | 284.5 | 168 ms | 3.5 ms | 2.0 | 4 | 16,888 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,034.6 | 266.2 | 119 ms | 3.8 ms | 4.0 | 8 | 31,034 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,737.6 | 221.0 | 187 ms | 4.5 ms | 7.9 | 8 | 52,120 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,766.9 | 175.6 | 217 ms | 5.7 ms | 15.9 | 16 | 83,000 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,155.3 | 130.2 | 252 ms | 7.7 ms | 31.8 | 32 | 124,645 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,066.5 | 94.6 | 1,374 ms | 10.6 ms | 64.0 | 0 | 181,946 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,688.1 | 58.8 | 2,537 ms | 17.0 ms | 128.0 | 0 | 230,455 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 305.6 | 309.4 | 125 ms | 3.2 ms | 1.0 | 2 | 9,168 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 559.4 | 282.3 | 143 ms | 3.5 ms | 2.0 | 4 | 16,778 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,016.1 | 261.7 | 139 ms | 3.8 ms | 4.0 | 8 | 30,478 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,677.5 | 212.1 | 178 ms | 4.7 ms | 7.9 | 8 | 50,314 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,688.2 | 169.1 | 288 ms | 5.9 ms | 16.0 | 16 | 80,631 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,918.4 | 121.7 | 372 ms | 8.2 ms | 31.9 | 27 | 117,562 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,662.8 | 88.4 | 1,241 ms | 11.3 ms | 64.0 | 0 | 169,864 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,966.2 | 53.8 | 2,127 ms | 18.6 ms | 128.0 | 0 | 208,895 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 300.8 | 305.7 | 129 ms | 3.3 ms | 1.0 | 2 | 9,023 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 548.1 | 280.7 | 161 ms | 3.6 ms | 2.0 | 4 | 16,443 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 986.4 | 252.7 | 162 ms | 4.0 ms | 4.0 | 8 | 29,587 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,605.4 | 202.7 | 233 ms | 4.9 ms | 8.0 | 8 | 48,162 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,480.1 | 158.0 | 301 ms | 6.3 ms | 16.0 | 16 | 74,407 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,661.9 | 115.2 | 891 ms | 8.7 ms | 32.0 | 0 | 109,864 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,818.6 | 74.9 | 1,630 ms | 13.4 ms | 64.0 | 0 | 144,439 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 291.6 | 294.7 | 195 ms | 3.4 ms | 1.0 | 2 | 8,747 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 516.3 | 267.0 | 221 ms | 3.7 ms | 2.0 | 4 | 15,487 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 923.6 | 233.5 | 283 ms | 4.3 ms | 4.0 | 4 | 27,700 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,425.8 | 181.7 | 338 ms | 5.5 ms | 7.9 | 8 | 42,763 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,142.2 | 136.0 | 337 ms | 7.4 ms | 15.9 | 16 | 64,250 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,021.5 | 93.7 | 1,418 ms | 10.7 ms | 32.0 | 0 | 90,614 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 267.2 | 274.9 | 341 ms | 3.6 ms | 1.0 | 2 | 8,017 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 492.6 | 253.6 | 390 ms | 3.9 ms | 2.0 | 4 | 14,776 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 812.7 | 209.3 | 422 ms | 4.8 ms | 3.9 | 4 | 24,377 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,233.8 | 156.9 | 611 ms | 6.4 ms | 7.9 | 8 | 37,003 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,739.4 | 109.3 | 1,957 ms | 9.1 ms | 16.0 | 0 | 52,158 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 236.4 | 246.4 | 672 ms | 4.1 ms | 1.0 | 1 | 7,091 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 422.6 | 216.4 | 711 ms | 4.6 ms | 2.0 | 2 | 12,677 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 665.1 | 172.5 | 961 ms | 5.8 ms | 3.9 | 4 | 19,952 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 948.7 | 123.1 | 1,479 ms | 8.1 ms | 7.8 | 6 | 28,389 | 29.9 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.