Speed test matrix
Qwen3.5-35B-A3B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.896805 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-35B-A3B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context36.7K223 ms TTFT
16K context42.5K382 ms TTFT
32K context42.1K766 ms TTFT
64K context37.7K1,708 ms TTFT
128K context29.1K4,415 ms TTFT
251K context19.9K12,670 ms TTFT
256K context19.6K13,151 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 281.4157 ms TTFT | 519.1193 ms TTFT | 884.6270 ms TTFT | 1,441.9354 ms TTFT | 2,254.7501 ms TTFT | 3,465.91,763 ms TTFT | 3,439.0Requests queued2,895 ms TTFT | 3,448.2Requests queued2,965 ms TTFT |
| 16K | 284.6183 ms TTFT | 513.5221 ms TTFT | 850.4275 ms TTFT | 1,390.9346 ms TTFT | 2,161.4447 ms TTFT | 3,330.11,954 ms TTFT | 3,287.3Requests queued2,804 ms TTFT | 3,357.8Requests queued2,828 ms TTFT |
| 32K | 268.3256 ms TTFT | 505.6261 ms TTFT | 849.1340 ms TTFT | 1,333.4425 ms TTFT | 1,994.2814 ms TTFT | 3,077.42,722 ms TTFT | 2,662.3Requests queued3,948 ms TTFT | 2,426.7Requests queued3,941 ms TTFT |
| 64K | 251.7314 ms TTFT | 476.8358 ms TTFT | 798.7416 ms TTFT | 1,227.7519 ms TTFT | 1,876.71,303 ms TTFT | 2,606.72,565 ms TTFT | 1,894.5Requests queued10,102 ms TTFT | 1,937.1Requests queued71,771 ms TTFT |
| 128K | 235.72,458 ms TTFT | 439.62,578 ms TTFT | 704.3806 ms TTFT | 1,054.3732 ms TTFT | 1,534.12,281 ms TTFT | 1,999.83,689 ms TTFT | 989.7Requests queued90,745 ms TTFT | Over capacity |
| 251K | 212.46,824 ms TTFT | 380.47,070 ms TTFT | 571.21,194 ms TTFT | 901.52,228 ms TTFT | 1,127.94,096 ms TTFT | 1,296.57,406 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 36.7K | — | 223 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 42.5K | — | 382 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 42.1K | — | 766 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 37.7K | — | 1,708 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29.1K | — | 4,415 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.9K | — | 12,670 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.6K | — | 13,151 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 281.4 | 287.2 | 157 ms | 3.5 ms | 1.0 | 2 | 8,442 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 519.1 | 267.5 | 193 ms | 3.7 ms | 2.0 | 4 | 15,569 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 884.6 | 224.4 | 270 ms | 4.5 ms | 4.0 | 4 | 26,539 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,441.9 | 183.4 | 354 ms | 5.5 ms | 8.0 | 8 | 43,252 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,254.7 | 143.3 | 501 ms | 7.0 ms | 15.9 | 16 | 67,612 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,465.9 | 109.0 | 1,763 ms | 9.2 ms | 32.0 | 0 | 103,892 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 3,439.0 | 85.7 | 2,895 ms | 11.7 ms | 38.7 | 41 | 102,798 | 29.9 s |
| Sustained decode | 8K | 128 | Requests queued | 3,448.2 | 83.1 | 2,965 ms | 12.0 ms | 39.1 | 41 | 103,351 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 284.6 | 284.3 | 183 ms | 3.5 ms | 1.0 | 2 | 8,536 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 513.5 | 271.6 | 221 ms | 3.7 ms | 2.0 | 4 | 15,404 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 850.4 | 218.0 | 275 ms | 4.6 ms | 4.0 | 4 | 25,503 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,390.9 | 179.5 | 346 ms | 5.6 ms | 7.9 | 8 | 41,722 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,161.4 | 137.9 | 447 ms | 7.3 ms | 15.9 | 16 | 64,808 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,330.1 | 105.5 | 1,954 ms | 9.5 ms | 32.0 | 0 | 99,824 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 3,287.3 | 79.3 | 2,804 ms | 12.6 ms | 39.8 | 41 | 98,599 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 3,357.8 | 80.6 | 2,828 ms | 12.4 ms | 39.7 | 41 | 100,661 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 268.3 | 273.9 | 256 ms | 3.7 ms | 1.0 | 2 | 8,046 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 505.6 | 261.2 | 261 ms | 3.8 ms | 2.0 | 4 | 15,168 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 849.1 | 213.9 | 340 ms | 4.7 ms | 4.0 | 4 | 25,471 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,333.4 | 173.3 | 425 ms | 5.8 ms | 7.9 | 8 | 39,986 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,994.2 | 128.8 | 814 ms | 7.8 ms | 15.9 | 13 | 59,797 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,077.4 | 97.0 | 2,722 ms | 10.3 ms | 32.0 | 0 | 92,241 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 2,662.3 | 70.6 | 3,948 ms | 14.2 ms | 39.4 | 38 | 79,841 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 2,426.7 | 68.1 | 3,941 ms | 14.7 ms | 38.3 | 38 | 72,736 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 251.7 | 258.7 | 314 ms | 3.9 ms | 1.0 | 2 | 7,550 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 476.8 | 250.8 | 358 ms | 4.0 ms | 2.0 | 3 | 14,302 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 798.7 | 204.2 | 416 ms | 4.9 ms | 4.0 | 4 | 23,958 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,227.7 | 157.4 | 519 ms | 6.4 ms | 7.9 | 8 | 36,820 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,876.7 | 118.9 | 1,303 ms | 8.4 ms | 16.0 | 6 | 56,258 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,606.7 | 81.2 | 2,565 ms | 12.3 ms | 32.0 | 0 | 78,205 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 1,894.5 | 56.1 | 10,102 ms | 17.8 ms | 36.0 | 62 | 56,649 | 29.9 s |
| Sustained decode | 64K | 128 | Requests queued | 1,937.1 | 56.6 | 71,771 ms | 17.7 ms | 36.6 | 60 | 57,848 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 235.7 | 241.1 | 2,458 ms | 4.1 ms | 1.0 | 1 | 7,069 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 439.6 | 228.1 | 2,578 ms | 4.4 ms | 2.0 | 2 | 13,185 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 704.3 | 181.6 | 806 ms | 5.5 ms | 4.0 | 4 | 21,124 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,054.3 | 136.2 | 732 ms | 7.3 ms | 7.8 | 8 | 31,620 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,534.1 | 94.7 | 2,281 ms | 10.6 ms | 16.0 | 0 | 46,017 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,999.8 | 59.5 | 3,689 ms | 16.8 ms | 32.0 | 0 | 59,983 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 989.7 | 41.7 | 90,745 ms | 24.0 ms | 31.6 | 64 | 29,647 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 212.4 | 218.5 | 6,824 ms | 4.6 ms | 1.0 | 1 | 6,372 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 380.4 | 199.2 | 7,070 ms | 5.0 ms | 2.0 | 2 | 11,409 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 571.2 | 145.3 | 1,194 ms | 6.9 ms | 3.9 | 4 | 17,127 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 901.5 | 113.7 | 2,228 ms | 8.8 ms | 7.9 | 2 | 27,027 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,127.9 | 67.9 | 4,096 ms | 14.7 ms | 16.0 | 0 | 33,815 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,296.5 | 37.7 | 7,406 ms | 26.6 ms | 32.0 | 0 | 38,874 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.