Speed test matrix
Qwen3.5-35B-A3B-GPTQ-Int4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.880124 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}' --quantization moe_wna16Prefill
prompt tok/s by context length8K context7,4851,094 ms TTFT
16K context7,7862,083 ms TTFT
32K context7,7454,169 ms TTFT
64K context7,8078,252 ms TTFT
128K context7,43517,310 ms TTFT
251K context6,50938,746 ms TTFT
256K context6,49139,627 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 104.7332 ms TTFT | 133.9971 ms TTFT | 134.51,118 ms TTFT | 274.41,866 ms TTFT | 434.43,173 ms TTFT | 653.65,058 ms TTFT | 976.78,037 ms TTFT | 1,829.514,671 ms TTFT |
| 16K | 106.1307 ms TTFT | 132.0507 ms TTFT | 135.7853 ms TTFT | 276.61,307 ms TTFT | 426.32,305 ms TTFT | 653.23,348 ms TTFT | 939.76,030 ms TTFT | 1,777.010,450 ms TTFT |
| 32K | 105.3350 ms TTFT | 135.0553 ms TTFT | 133.0927 ms TTFT | 274.01,409 ms TTFT | 429.22,321 ms TTFT | 646.83,636 ms TTFT | 938.65,937 ms TTFT | Over capacity |
| 64K | 103.8401 ms TTFT | 132.5636 ms TTFT | 132.81,055 ms TTFT | 267.11,748 ms TTFT | 423.12,878 ms TTFT | 632.04,195 ms TTFT | Over capacity | Over capacity |
| 128K | 102.0543 ms TTFT | 132.6742 ms TTFT | 128.81,233 ms TTFT | 259.21,902 ms TTFT | 391.82,741 ms TTFT | 579.44,573 ms TTFT | Over capacity | Over capacity |
| 251K | 96.1867 ms TTFT | 126.31,195 ms TTFT | 125.11,620 ms TTFT | 250.02,536 ms TTFT | 358.04,195 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,485.0 | — | 1,094 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,786.0 | — | 2,083 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,745.0 | — | 4,169 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,807.0 | — | 8,252 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,435.0 | — | 17,310 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,509.0 | — | 38,746 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,491.0 | — | 39,627 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 104.7 | 105.3 | 332 ms | 9.5 ms | 1.0 | 0 | 3,142 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 133.9 | 67.5 | 971 ms | 14.8 ms | 2.0 | 0 | 4,017 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 134.5 | 34.6 | 1,118 ms | 28.9 ms | 4.0 | 0 | 4,029 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 274.4 | 35.4 | 1,866 ms | 28.3 ms | 8.0 | 0 | 8,225 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 434.4 | 28.2 | 3,173 ms | 35.5 ms | 16.0 | 0 | 13,009 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 653.6 | 20.8 | 5,058 ms | 48.1 ms | 32.0 | 0 | 19,579 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 976.7 | 14.3 | 8,037 ms | 70.0 ms | 64.0 | 0 | 29,292 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,829.5 | 11.7 | 14,671 ms | 85.6 ms | 128.0 | 0 | 54,781 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 106.1 | 107.0 | 307 ms | 9.3 ms | 1.0 | 0 | 3,183 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 132.0 | 66.6 | 507 ms | 15.0 ms | 2.0 | 0 | 3,957 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 135.7 | 34.8 | 853 ms | 28.7 ms | 4.0 | 0 | 4,068 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 276.6 | 35.9 | 1,307 ms | 27.9 ms | 8.0 | 0 | 8,292 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 426.3 | 27.6 | 2,305 ms | 36.2 ms | 16.0 | 0 | 12,774 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 653.2 | 20.9 | 3,348 ms | 47.8 ms | 32.0 | 0 | 19,565 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 939.7 | 14.4 | 6,030 ms | 69.6 ms | 64.0 | 0 | 28,149 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,777.0 | 12.3 | 10,450 ms | 81.6 ms | 128.0 | 0 | 53,190 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 105.3 | 106.1 | 350 ms | 9.4 ms | 1.0 | 0 | 3,160 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 135.0 | 68.2 | 553 ms | 14.7 ms | 2.0 | 0 | 4,049 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 133.0 | 34.4 | 927 ms | 29.0 ms | 4.0 | 0 | 3,983 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 274.0 | 35.4 | 1,409 ms | 28.3 ms | 8.0 | 0 | 8,216 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 429.2 | 27.8 | 2,321 ms | 35.9 ms | 16.0 | 0 | 12,858 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 646.8 | 20.4 | 3,636 ms | 49.0 ms | 32.0 | 0 | 19,348 | 29.9 s |
| Sustained decode | 32K | 64 | Valid | 938.6 | 14.2 | 5,937 ms | 70.2 ms | 64.0 | 0 | 28,084 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 103.8 | 104.9 | 401 ms | 9.5 ms | 1.0 | 0 | 3,113 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 132.5 | 66.8 | 636 ms | 15.0 ms | 2.0 | 0 | 3,972 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 132.8 | 34.0 | 1,055 ms | 29.4 ms | 4.0 | 0 | 3,983 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 267.1 | 34.3 | 1,748 ms | 29.2 ms | 8.0 | 0 | 8,006 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 423.1 | 27.4 | 2,878 ms | 36.5 ms | 16.0 | 0 | 12,696 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 632.0 | 20.1 | 4,195 ms | 49.8 ms | 32.0 | 0 | 18,931 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 102.0 | 102.9 | 543 ms | 9.7 ms | 1.0 | 0 | 3,060 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 132.6 | 66.8 | 742 ms | 15.0 ms | 2.0 | 0 | 3,977 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 128.8 | 32.8 | 1,233 ms | 30.4 ms | 4.0 | 0 | 3,865 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 259.2 | 33.2 | 1,902 ms | 30.2 ms | 8.0 | 0 | 7,773 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 391.8 | 24.6 | 2,741 ms | 40.6 ms | 16.0 | 0 | 11,742 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 579.4 | 18.0 | 4,573 ms | 55.4 ms | 32.0 | 0 | 17,319 | 29.9 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 96.1 | 97.2 | 867 ms | 10.3 ms | 1.0 | 0 | 2,883 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 126.3 | 63.9 | 1,195 ms | 15.6 ms | 2.0 | 0 | 3,789 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 125.1 | 31.7 | 1,620 ms | 31.6 ms | 4.0 | 0 | 3,747 | 29.9 s |
| Sustained decode | 251K | 8 | Valid | 250.0 | 30.9 | 2,536 ms | 32.3 ms | 8.0 | 0 | 7,503 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 358.0 | 21.9 | 4,195 ms | 45.6 ms | 16.0 | 0 | 10,714 | 29.9 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.