Speed test matrix
Qwen3.6-35B-A3B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.896805 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.6-35B-A3B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context38.4K213 ms TTFT
16K context41.1K395 ms TTFT
32K context41.8K772 ms TTFT
64K context38K1,695 ms TTFT
128K context29.2K4,403 ms TTFT
251K context19.9K12,687 ms TTFT
256K context19.4K13,236 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 231.4191 ms TTFT | 452.1229 ms TTFT | 775.8274 ms TTFT | 1,224.4413 ms TTFT | 1,901.71,021 ms TTFT | 2,954.81,885 ms TTFT | 2,863.8Requests queued2,912 ms TTFT | 2,849.4Requests queued2,944 ms TTFT |
| 16K | 229.9298 ms TTFT | 453.7344 ms TTFT | 752.2270 ms TTFT | 1,220.8475 ms TTFT | 1,895.21,020 ms TTFT | 2,842.41,948 ms TTFT | 2,739.8Requests queued2,789 ms TTFT | 2,819.0Requests queued2,866 ms TTFT |
| 32K | 253.8221 ms TTFT | 453.6279 ms TTFT | 723.2312 ms TTFT | 1,164.5417 ms TTFT | 1,773.21,582 ms TTFT | 2,613.32,561 ms TTFT | 1,972.7Requests queued3,752 ms TTFT | 2,103.6Requests queued3,835 ms TTFT |
| 64K | 226.01,010 ms TTFT | 416.31,076 ms TTFT | 707.4396 ms TTFT | 1,072.8404 ms TTFT | 1,622.91,523 ms TTFT | 2,264.02,367 ms TTFT | 1,793.9Requests queued4,456 ms TTFT | 1,865.7Requests queued4,340 ms TTFT |
| 128K | 204.82,447 ms TTFT | 401.02,594 ms TTFT | 636.3715 ms TTFT | 939.91,217 ms TTFT | 1,327.22,282 ms TTFT | 1,726.94,282 ms TTFT | 1,371.4Requests queued8,162 ms TTFT | Over capacity |
| 251K | 194.86,818 ms TTFT | 331.67,087 ms TTFT | 521.41,205 ms TTFT | 766.22,551 ms TTFT | 986.84,126 ms TTFT | 1,234.07,638 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 38.4K | — | 213 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 41.1K | — | 395 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 41.8K | — | 772 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 38K | — | 1,695 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29.2K | — | 4,403 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 19.9K | — | 12,687 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.4K | — | 13,236 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 231.4 | 237.8 | 191 ms | 4.2 ms | 1.0 | 1 | 6,941 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 452.1 | 232.8 | 229 ms | 4.3 ms | 2.0 | 2 | 13,562 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 775.8 | 198.5 | 274 ms | 5.0 ms | 4.0 | 4 | 23,272 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,224.4 | 165.8 | 413 ms | 6.0 ms | 7.9 | 8 | 36,719 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,901.7 | 124.6 | 1,021 ms | 8.0 ms | 15.9 | 10 | 57,044 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,954.8 | 93.2 | 1,885 ms | 10.7 ms | 32.0 | 0 | 88,574 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 2,863.8 | 74.1 | 2,912 ms | 13.5 ms | 39.3 | 39 | 85,850 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 2,849.4 | 73.8 | 2,944 ms | 13.6 ms | 38.9 | 39 | 85,453 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 229.9 | 234.0 | 298 ms | 4.3 ms | 1.0 | 1 | 6,900 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 453.7 | 232.8 | 344 ms | 4.3 ms | 2.0 | 2 | 13,607 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 752.2 | 190.4 | 270 ms | 5.3 ms | 4.0 | 4 | 22,566 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,220.8 | 160.1 | 475 ms | 6.2 ms | 8.0 | 8 | 36,614 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,895.2 | 122.1 | 1,020 ms | 8.2 ms | 16.0 | 8 | 56,830 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,842.4 | 91.0 | 1,948 ms | 11.0 ms | 32.0 | 0 | 85,266 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 2,739.8 | 70.3 | 2,789 ms | 14.2 ms | 40.0 | 39 | 82,201 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 2,819.0 | 71.1 | 2,866 ms | 14.1 ms | 40.0 | 39 | 84,550 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 253.8 | 262.8 | 221 ms | 3.8 ms | 1.0 | 2 | 7,613 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 453.6 | 233.6 | 279 ms | 4.3 ms | 2.0 | 3 | 13,604 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 723.2 | 185.0 | 312 ms | 5.4 ms | 4.0 | 4 | 21,689 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,164.5 | 155.3 | 417 ms | 6.4 ms | 7.9 | 8 | 34,921 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,773.2 | 114.4 | 1,582 ms | 8.7 ms | 16.0 | 3 | 53,179 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,613.3 | 84.4 | 2,561 ms | 11.9 ms | 32.0 | 0 | 78,327 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,972.7 | 60.5 | 3,752 ms | 16.5 ms | 37.6 | 38 | 59,156 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 2,103.6 | 63.7 | 3,835 ms | 15.7 ms | 37.4 | 38 | 63,110 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 226.0 | 233.3 | 1,010 ms | 4.3 ms | 1.0 | 1 | 6,779 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 416.3 | 218.7 | 1,076 ms | 4.6 ms | 2.0 | 2 | 12,489 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 707.4 | 181.0 | 396 ms | 5.5 ms | 4.0 | 4 | 21,217 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,072.8 | 138.4 | 404 ms | 7.2 ms | 7.9 | 8 | 32,181 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,622.9 | 103.6 | 1,523 ms | 9.7 ms | 16.0 | 0 | 48,668 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,264.0 | 70.4 | 2,367 ms | 14.2 ms | 32.0 | 0 | 67,898 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 1,793.9 | 50.1 | 4,456 ms | 20.0 ms | 36.6 | 40 | 53,784 | 30.0 s |
| Sustained decode | 64K | 128 | Requests queued | 1,865.7 | 50.3 | 4,340 ms | 19.9 ms | 37.0 | 41 | 55,900 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 204.8 | 212.1 | 2,447 ms | 4.7 ms | 1.0 | 1 | 6,144 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 401.0 | 207.4 | 2,594 ms | 4.8 ms | 2.0 | 2 | 12,027 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 636.3 | 170.0 | 715 ms | 5.9 ms | 3.9 | 4 | 19,088 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 939.9 | 121.3 | 1,217 ms | 8.2 ms | 8.0 | 2 | 28,196 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,327.2 | 81.3 | 2,282 ms | 12.3 ms | 16.0 | 0 | 39,807 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,726.9 | 53.0 | 4,282 ms | 18.9 ms | 32.0 | 0 | 51,789 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 1,371.4 | 37.5 | 8,162 ms | 26.7 ms | 33.1 | 39 | 41,100 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 194.8 | 201.2 | 6,818 ms | 5.0 ms | 1.0 | 1 | 5,849 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 331.6 | 175.1 | 7,087 ms | 5.7 ms | 1.9 | 2 | 9,943 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 521.4 | 140.6 | 1,205 ms | 7.1 ms | 3.9 | 4 | 15,607 | 29.9 s |
| Sustained decode | 251K | 8 | Valid | 766.2 | 95.2 | 2,551 ms | 10.5 ms | 8.0 | 0 | 22,975 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 986.8 | 59.0 | 4,126 ms | 16.9 ms | 16.0 | 0 | 29,597 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,234.0 | 35.6 | 7,638 ms | 28.1 ms | 32.0 | 0 | 36,969 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.