Speed test matrix
Qwen3.5-35B-A3B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881063 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context41.9K195 ms TTFT
16K context42.2K385 ms TTFT
32K context43.5K743 ms TTFT
64K context39K1,650 ms TTFT
128K context30K4,287 ms TTFT
251K context20.2K12,490 ms TTFT
256K context19.6K13,100 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 216.099 ms TTFT | 412.1153 ms TTFT | 730.8930 ms TTFT | 1,224.1343 ms TTFT | 1,896.5410 ms TTFT | 3,018.3646 ms TTFT | 4,680.5887 ms TTFT | 6,535.71,341 ms TTFT |
| 16K | 213.482 ms TTFT | 405.9135 ms TTFT | 711.9227 ms TTFT | 1,183.9275 ms TTFT | 1,826.9368 ms TTFT | 2,810.0524 ms TTFT | 4,153.3811 ms TTFT | 5,556.11,418 ms TTFT |
| 32K | 207.6151 ms TTFT | 394.0250 ms TTFT | 679.1311 ms TTFT | 1,094.4375 ms TTFT | 1,658.0541 ms TTFT | 2,438.3814 ms TTFT | 3,386.51,318 ms TTFT | Over capacity |
| 64K | 197.9163 ms TTFT | 374.7213 ms TTFT | 627.4218 ms TTFT | 974.7239 ms TTFT | 1,407.7889 ms TTFT | 1,924.31,407 ms TTFT | Over capacity | Over capacity |
| 128K | 180.0303 ms TTFT | 338.2427 ms TTFT | 537.4509 ms TTFT | 803.9943 ms TTFT | 1,066.11,619 ms TTFT | 1,303.13,145 ms TTFT | Over capacity | Over capacity |
| 251K | 153.3594 ms TTFT | 286.2734 ms TTFT | 434.61,157 ms TTFT | 594.41,931 ms TTFT | 712.83,246 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.9K | — | 195 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 42.2K | — | 385 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 43.5K | — | 743 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 39K | — | 1,650 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 30K | — | 4,287 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 20.2K | — | 12,490 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.6K | — | 13,100 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 216.0 | 217.0 | 99 ms | 4.6 ms | 1.0 | 1 | 6,479 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 412.1 | 207.4 | 153 ms | 4.8 ms | 2.0 | 2 | 12,362 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 730.8 | 184.3 | 930 ms | 5.4 ms | 4.0 | 4 | 21,924 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,224.1 | 155.5 | 343 ms | 6.4 ms | 8.0 | 8 | 36,721 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,896.5 | 120.0 | 410 ms | 8.3 ms | 16.0 | 16 | 56,844 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,018.3 | 95.5 | 646 ms | 10.5 ms | 32.0 | 0 | 90,560 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,680.5 | 73.5 | 887 ms | 13.6 ms | 64.0 | 0 | 140,416 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,535.7 | 50.8 | 1,341 ms | 19.7 ms | 128.0 | 0 | 196,096 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 213.4 | 214.3 | 82 ms | 4.7 ms | 1.0 | 1 | 6,402 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 405.9 | 204.1 | 135 ms | 4.9 ms | 2.0 | 2 | 12,176 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 711.9 | 180.0 | 227 ms | 5.6 ms | 4.0 | 4 | 21,358 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,183.9 | 150.4 | 275 ms | 6.6 ms | 8.0 | 8 | 35,518 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,826.9 | 115.1 | 368 ms | 8.7 ms | 16.0 | 0 | 54,800 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,810.0 | 88.1 | 524 ms | 11.3 ms | 32.0 | 0 | 84,288 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,153.3 | 64.8 | 811 ms | 15.4 ms | 64.0 | 0 | 124,608 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,556.1 | 43.0 | 1,418 ms | 23.3 ms | 128.0 | 0 | 166,656 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 207.6 | 209.0 | 151 ms | 4.8 ms | 1.0 | 1 | 6,230 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 394.0 | 199.4 | 250 ms | 5.0 ms | 1.9 | 2 | 11,821 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 679.1 | 172.2 | 311 ms | 5.8 ms | 4.0 | 4 | 20,370 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,094.4 | 138.7 | 375 ms | 7.2 ms | 7.7 | 8 | 32,826 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,658.0 | 103.7 | 541 ms | 9.6 ms | 16.0 | 0 | 49,728 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,438.3 | 75.9 | 814 ms | 13.2 ms | 32.0 | 0 | 73,152 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,386.5 | 52.2 | 1,318 ms | 19.2 ms | 64.0 | 0 | 101,568 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 197.9 | 199.4 | 163 ms | 5.0 ms | 1.0 | 1 | 5,938 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 374.7 | 189.2 | 213 ms | 5.3 ms | 2.0 | 2 | 11,242 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 627.4 | 158.4 | 218 ms | 6.3 ms | 4.0 | 4 | 18,819 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 974.7 | 122.8 | 239 ms | 8.1 ms | 7.9 | 8 | 29,243 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,407.7 | 88.0 | 889 ms | 11.4 ms | 16.0 | 0 | 42,224 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,924.3 | 60.2 | 1,407 ms | 16.6 ms | 32.0 | 0 | 57,729 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 180.0 | 182.4 | 303 ms | 5.5 ms | 1.0 | 1 | 5,402 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 338.2 | 171.7 | 427 ms | 5.8 ms | 2.0 | 2 | 10,145 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 537.4 | 136.5 | 509 ms | 7.3 ms | 3.9 | 4 | 16,120 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 803.9 | 100.1 | 943 ms | 10.0 ms | 8.0 | 0 | 24,112 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,066.1 | 66.1 | 1,619 ms | 15.1 ms | 16.0 | 0 | 31,985 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,303.1 | 40.0 | 3,145 ms | 25.0 ms | 32.0 | 0 | 39,072 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 153.3 | 157.6 | 594 ms | 6.3 ms | 1.0 | 1 | 4,601 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 286.2 | 146.6 | 734 ms | 6.8 ms | 1.9 | 2 | 8,591 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 434.6 | 108.7 | 1,157 ms | 9.2 ms | 4.0 | 0 | 13,036 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 594.4 | 74.2 | 1,931 ms | 13.5 ms | 8.0 | 0 | 17,832 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 712.8 | 44.3 | 3,246 ms | 22.6 ms | 16.0 | 0 | 21,376 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.