Speed test matrix
Qwen3.5-35B-A3B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881503 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context43.8K187 ms TTFT
16K context44K368 ms TTFT
32K context40.7K793 ms TTFT
64K context36.3K1,772 ms TTFT
128K context27.2K4,736 ms TTFT
251K context17.9K14,128 ms TTFT
256K context17.7K14,498 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 300.297 ms TTFT | 544.2183 ms TTFT | 958.7114 ms TTFT | 1,662.7226 ms TTFT | 2,766.6250 ms TTFT | 4,137.2321 ms TTFT | 6,220.61,439 ms TTFT | 7,955.52,509 ms TTFT |
| 16K | 299.9127 ms TTFT | 537.4121 ms TTFT | 941.1199 ms TTFT | 1,622.0253 ms TTFT | 2,634.2227 ms TTFT | 3,904.8391 ms TTFT | 5,727.71,181 ms TTFT | 7,161.12,127 ms TTFT |
| 32K | 295.6139 ms TTFT | 529.7150 ms TTFT | 915.4234 ms TTFT | 1,554.8243 ms TTFT | 2,418.9262 ms TTFT | 3,610.2914 ms TTFT | 4,948.11,551 ms TTFT | Over capacity |
| 64K | 289.1209 ms TTFT | 511.8213 ms TTFT | 857.7321 ms TTFT | 1,405.9323 ms TTFT | 2,092.6338 ms TTFT | 2,980.71,484 ms TTFT | Over capacity | Over capacity |
| 128K | 262.5341 ms TTFT | 476.5475 ms TTFT | 765.2591 ms TTFT | 1,182.5599 ms TTFT | 1,710.01,995 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 230.2665 ms TTFT | 414.5952 ms TTFT | 632.4953 ms TTFT | 934.81,895 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 43.8K | — | 187 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 44K | — | 368 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 40.7K | — | 793 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 36.3K | — | 1,772 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 27.2K | — | 4,736 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.9K | — | 14,128 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.7K | — | 14,498 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 300.2 | 303.7 | 97 ms | 3.3 ms | 1.0 | 2 | 9,005 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 544.2 | 276.4 | 183 ms | 3.6 ms | 2.0 | 4 | 16,326 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 958.7 | 245.3 | 114 ms | 4.1 ms | 4.0 | 8 | 28,761 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,662.7 | 211.8 | 226 ms | 4.7 ms | 8.0 | 8 | 49,866 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,766.6 | 175.2 | 250 ms | 5.7 ms | 15.9 | 16 | 82,994 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,137.2 | 131.0 | 321 ms | 7.6 ms | 31.9 | 32 | 124,076 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,220.6 | 96.4 | 1,439 ms | 10.4 ms | 64.0 | 0 | 186,530 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,955.5 | 60.8 | 2,509 ms | 16.5 ms | 128.0 | 0 | 238,689 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 299.9 | 307.5 | 127 ms | 3.3 ms | 1.0 | 2 | 8,997 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 537.4 | 273.2 | 121 ms | 3.7 ms | 2.0 | 4 | 16,118 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 941.1 | 238.1 | 199 ms | 4.2 ms | 4.0 | 6 | 28,173 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 1,622.0 | 207.0 | 253 ms | 4.8 ms | 8.0 | 8 | 48,649 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,634.2 | 168.5 | 227 ms | 5.9 ms | 15.9 | 16 | 79,024 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,904.8 | 122.4 | 391 ms | 8.2 ms | 31.8 | 32 | 116,953 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,727.7 | 89.6 | 1,181 ms | 11.2 ms | 64.0 | 0 | 171,842 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 7,161.1 | 55.3 | 2,127 ms | 18.1 ms | 128.0 | 0 | 214,762 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 295.6 | 299.0 | 139 ms | 3.3 ms | 1.0 | 2 | 8,867 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 529.7 | 268.9 | 150 ms | 3.7 ms | 2.0 | 4 | 15,888 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 915.4 | 231.8 | 234 ms | 4.3 ms | 4.0 | 4 | 27,460 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,554.8 | 197.8 | 243 ms | 5.1 ms | 8.0 | 8 | 46,634 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,418.9 | 153.8 | 262 ms | 6.5 ms | 16.0 | 16 | 72,548 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,610.2 | 113.0 | 914 ms | 8.9 ms | 32.0 | 0 | 108,248 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,948.1 | 76.6 | 1,551 ms | 13.1 ms | 64.0 | 0 | 148,339 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 289.1 | 293.2 | 209 ms | 3.4 ms | 1.0 | 2 | 8,672 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 511.8 | 263.2 | 213 ms | 3.8 ms | 2.0 | 4 | 15,352 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 857.7 | 217.6 | 321 ms | 4.6 ms | 4.0 | 4 | 25,726 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,405.9 | 178.4 | 323 ms | 5.6 ms | 8.0 | 8 | 42,162 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,092.6 | 131.2 | 338 ms | 7.6 ms | 15.9 | 16 | 62,771 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,980.7 | 92.9 | 1,484 ms | 10.8 ms | 32.0 | 0 | 89,386 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 262.5 | 269.5 | 341 ms | 3.7 ms | 1.0 | 2 | 7,875 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 476.5 | 243.7 | 475 ms | 4.1 ms | 2.0 | 2 | 14,292 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 765.2 | 196.2 | 591 ms | 5.1 ms | 3.9 | 4 | 22,953 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,182.5 | 151.5 | 599 ms | 6.6 ms | 7.8 | 8 | 35,471 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,710.0 | 106.9 | 1,995 ms | 9.4 ms | 16.0 | 0 | 51,298 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 230.2 | 236.8 | 665 ms | 4.2 ms | 1.0 | 1 | 6,908 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 414.5 | 214.9 | 952 ms | 4.7 ms | 2.0 | 2 | 12,432 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 632.4 | 163.8 | 953 ms | 6.1 ms | 3.9 | 4 | 18,965 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 934.8 | 118.3 | 1,895 ms | 8.4 ms | 7.9 | 3 | 28,015 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.