Speed test matrix
Qwen3.5-35B-A3B-GPTQ-Int4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881019 --host 0.0.0.0 --port 8000 --api-key <redacted> --quantization moe_wna16
Prefill
prompt tok/s by context length8K context7,3721,111 ms TTFT
16K context7,3292,213 ms TTFT
32K context7,8094,135 ms TTFT
64K context8,0358,017 ms TTFT
128K context7,58016,978 ms TTFT
251K context6,81237,025 ms TTFT
256K context6,72438,254 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 97.0245 ms TTFT | 123.1390 ms TTFT | 160.62,137 ms TTFT | 192.4973 ms TTFT | 268.21,753 ms TTFT | 428.32,931 ms TTFT | 676.84,799 ms TTFT | 1,037.97,689 ms TTFT |
| 16K | 96.4252 ms TTFT | 122.4351 ms TTFT | 158.5547 ms TTFT | 193.3660 ms TTFT | 267.51,166 ms TTFT | 424.91,526 ms TTFT | 676.32,648 ms TTFT | 1,018.34,298 ms TTFT |
| 32K | 95.2275 ms TTFT | 121.4468 ms TTFT | 155.4677 ms TTFT | 190.81,075 ms TTFT | 264.21,459 ms TTFT | 414.72,504 ms TTFT | 643.73,656 ms TTFT | 970.26,603 ms TTFT |
| 64K | 93.0141 ms TTFT | 119.5301 ms TTFT | 154.7340 ms TTFT | 187.2505 ms TTFT | 257.8930 ms TTFT | 386.41,455 ms TTFT | 578.12,671 ms TTFT | Over capacity |
| 128K | 89.1490 ms TTFT | 115.8617 ms TTFT | 147.71,029 ms TTFT | 179.31,438 ms TTFT | 244.72,425 ms TTFT | 363.63,828 ms TTFT | Over capacity | Over capacity |
| 251K | 82.3779 ms TTFT | 109.21,013 ms TTFT | 136.41,393 ms TTFT | 162.32,405 ms TTFT | 216.63,566 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,372.0 | — | 1,111 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,329.0 | — | 2,213 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,809.0 | — | 4,135 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,035.0 | — | 8,017 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,580.0 | — | 16,978 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,812.0 | — | 37,025 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,724.0 | — | 38,254 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 97.0 | 97.0 | 245 ms | 10.3 ms | 1.0 | 0 | 2,910 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 123.1 | 61.5 | 390 ms | 16.3 ms | 2.0 | 0 | 3,692 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 160.6 | 40.6 | 2,137 ms | 24.6 ms | 4.0 | 0 | 4,816 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 192.4 | 24.8 | 973 ms | 40.3 ms | 8.0 | 0 | 5,768 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 268.2 | 18.2 | 1,753 ms | 55.0 ms | 16.0 | 0 | 8,048 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 428.3 | 14.5 | 2,931 ms | 69.2 ms | 32.0 | 0 | 12,832 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 676.8 | 10.9 | 4,799 ms | 91.7 ms | 64.0 | 0 | 20,288 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,037.9 | 7.7 | 7,689 ms | 129.8 ms | 128.0 | 0 | 31,104 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 96.4 | 96.4 | 252 ms | 10.4 ms | 1.0 | 0 | 2,891 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 122.4 | 61.1 | 351 ms | 16.4 ms | 2.0 | 0 | 3,670 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 158.5 | 40.2 | 547 ms | 24.8 ms | 4.0 | 0 | 4,756 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 193.3 | 24.8 | 660 ms | 40.3 ms | 8.0 | 0 | 5,800 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 267.5 | 17.8 | 1,166 ms | 56.2 ms | 16.0 | 0 | 8,016 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 424.9 | 14.1 | 1,526 ms | 70.8 ms | 32.0 | 0 | 12,736 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 676.3 | 11.0 | 2,648 ms | 91.2 ms | 64.0 | 0 | 20,288 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,018.3 | 8.0 | 4,298 ms | 124.5 ms | 128.0 | 0 | 30,464 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 95.2 | 95.3 | 275 ms | 10.5 ms | 1.0 | 0 | 2,856 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 121.4 | 60.8 | 468 ms | 16.5 ms | 2.0 | 0 | 3,642 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 155.4 | 39.2 | 677 ms | 25.5 ms | 4.0 | 0 | 4,660 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 190.8 | 24.7 | 1,075 ms | 40.5 ms | 8.0 | 0 | 5,720 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 264.2 | 17.5 | 1,459 ms | 57.1 ms | 16.0 | 0 | 7,920 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 414.7 | 13.7 | 2,504 ms | 73.2 ms | 32.0 | 0 | 12,448 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 643.7 | 10.1 | 3,656 ms | 98.6 ms | 64.0 | 0 | 19,264 | 29.9 s |
| Sustained decode | 32K | 128 | Valid | 970.2 | 7.3 | 6,603 ms | 137.5 ms | 128.0 | 0 | 29,056 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 93.0 | 93.1 | 141 ms | 10.7 ms | 1.0 | 0 | 2,791 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 119.5 | 59.9 | 301 ms | 16.7 ms | 2.0 | 0 | 3,588 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 154.7 | 38.8 | 340 ms | 25.8 ms | 4.0 | 0 | 4,640 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 187.2 | 23.5 | 505 ms | 42.6 ms | 8.0 | 0 | 5,616 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 257.8 | 16.2 | 930 ms | 61.8 ms | 16.0 | 0 | 7,728 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 386.4 | 12.3 | 1,455 ms | 81.6 ms | 32.0 | 0 | 11,584 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 578.1 | 9.1 | 2,671 ms | 109.3 ms | 64.0 | 0 | 17,344 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 89.1 | 89.2 | 490 ms | 11.2 ms | 1.0 | 0 | 2,672 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 115.8 | 57.8 | 617 ms | 17.3 ms | 2.0 | 0 | 3,476 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 147.7 | 36.8 | 1,029 ms | 27.1 ms | 4.0 | 0 | 4,428 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 179.3 | 22.3 | 1,438 ms | 44.9 ms | 8.0 | 0 | 5,376 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 244.7 | 15.2 | 2,425 ms | 65.8 ms | 16.0 | 0 | 7,344 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 363.6 | 10.9 | 3,828 ms | 91.4 ms | 32.0 | 0 | 10,880 | 29.9 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 82.3 | 82.6 | 779 ms | 12.1 ms | 1.0 | 0 | 2,470 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 109.2 | 54.5 | 1,013 ms | 18.3 ms | 2.0 | 0 | 3,274 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 136.4 | 34.0 | 1,393 ms | 29.4 ms | 4.0 | 0 | 4,092 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 162.3 | 20.0 | 2,405 ms | 50.0 ms | 8.0 | 0 | 4,864 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 216.6 | 13.0 | 3,566 ms | 77.1 ms | 16.0 | 0 | 6,496 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.