Speed test matrix
Qwen3.6-35B-A3B-FP8 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.881063 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context40.8K201 ms TTFT
16K context41.2K394 ms TTFT
32K context41.3K782 ms TTFT
64K context39.2K1,644 ms TTFT
128K context29.6K4,352 ms TTFT
251K context20.2K12,496 ms TTFT
256K context19.9K12,917 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 216.8102 ms TTFT | 411.6158 ms TTFT | 732.3959 ms TTFT | 1,229.0234 ms TTFT | 1,948.1366 ms TTFT | 3,095.4617 ms TTFT | 4,752.1841 ms TTFT | 6,581.01,329 ms TTFT |
| 16K | 213.8107 ms TTFT | 405.6156 ms TTFT | 710.8225 ms TTFT | 1,181.4291 ms TTFT | 1,843.4383 ms TTFT | 2,853.2557 ms TTFT | 4,174.3806 ms TTFT | 5,584.11,371 ms TTFT |
| 32K | 207.8163 ms TTFT | 394.2231 ms TTFT | 680.2307 ms TTFT | 1,096.8420 ms TTFT | 1,671.6555 ms TTFT | 2,456.4844 ms TTFT | 3,390.11,348 ms TTFT | Over capacity |
| 64K | 197.5178 ms TTFT | 375.2217 ms TTFT | 631.2218 ms TTFT | 976.1229 ms TTFT | 1,413.9840 ms TTFT | 1,926.31,535 ms TTFT | Over capacity | Over capacity |
| 128K | 180.5327 ms TTFT | 339.2376 ms TTFT | 540.4419 ms TTFT | 809.7955 ms TTFT | 1,079.91,634 ms TTFT | 1,312.63,084 ms TTFT | Over capacity | Over capacity |
| 251K | 153.6610 ms TTFT | 286.5745 ms TTFT | 435.91,220 ms TTFT | 598.41,990 ms TTFT | 719.13,507 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.8K | — | 201 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 41.2K | — | 394 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 41.3K | — | 782 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 39.2K | — | 1,644 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29.6K | — | 4,352 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 20.2K | — | 12,496 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.9K | — | 12,917 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 216.8 | 217.7 | 102 ms | 4.6 ms | 1.0 | 1 | 6,503 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 411.6 | 207.2 | 158 ms | 4.8 ms | 2.0 | 2 | 12,346 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 732.3 | 184.8 | 959 ms | 5.4 ms | 4.0 | 4 | 21,968 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,229.0 | 155.5 | 234 ms | 6.4 ms | 8.0 | 8 | 36,871 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,948.1 | 123.7 | 366 ms | 8.1 ms | 16.0 | 16 | 58,442 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,095.4 | 97.5 | 617 ms | 10.3 ms | 32.0 | 0 | 92,865 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,752.1 | 74.5 | 841 ms | 13.4 ms | 64.0 | 0 | 142,591 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,581.0 | 51.0 | 1,329 ms | 19.6 ms | 128.0 | 0 | 197,376 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 213.8 | 214.9 | 107 ms | 4.7 ms | 1.0 | 1 | 6,413 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 405.6 | 204.1 | 156 ms | 4.9 ms | 2.0 | 2 | 12,168 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 710.8 | 179.6 | 225 ms | 5.6 ms | 4.0 | 4 | 21,322 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,181.4 | 150.0 | 291 ms | 6.7 ms | 7.7 | 8 | 35,437 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,843.4 | 116.1 | 383 ms | 8.6 ms | 16.0 | 0 | 55,303 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,853.2 | 89.3 | 557 ms | 11.2 ms | 32.0 | 0 | 85,600 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,174.3 | 65.1 | 806 ms | 15.4 ms | 64.0 | 0 | 125,248 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,584.1 | 43.2 | 1,371 ms | 23.2 ms | 128.0 | 0 | 167,549 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 207.8 | 209.4 | 163 ms | 4.8 ms | 1.0 | 1 | 6,234 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 394.2 | 199.2 | 231 ms | 5.0 ms | 1.9 | 2 | 11,826 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 680.2 | 172.6 | 307 ms | 5.8 ms | 4.0 | 4 | 20,407 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,096.8 | 139.3 | 420 ms | 7.2 ms | 7.7 | 8 | 32,903 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,671.6 | 104.7 | 555 ms | 9.6 ms | 16.0 | 0 | 50,144 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,456.4 | 76.5 | 844 ms | 13.1 ms | 32.0 | 0 | 73,696 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,390.1 | 52.3 | 1,348 ms | 19.1 ms | 64.0 | 0 | 101,708 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 197.5 | 199.3 | 178 ms | 5.0 ms | 1.0 | 1 | 5,925 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 375.2 | 189.4 | 217 ms | 5.3 ms | 2.0 | 2 | 11,256 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 631.2 | 159.1 | 218 ms | 6.3 ms | 4.0 | 4 | 18,935 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 976.1 | 123.0 | 229 ms | 8.1 ms | 7.9 | 8 | 29,283 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,413.9 | 88.5 | 840 ms | 11.3 ms | 16.0 | 0 | 42,416 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,926.3 | 60.3 | 1,535 ms | 16.6 ms | 32.0 | 0 | 57,792 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 180.5 | 183.0 | 327 ms | 5.5 ms | 1.0 | 1 | 5,414 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 339.2 | 171.8 | 376 ms | 5.8 ms | 2.0 | 2 | 10,177 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 540.4 | 136.9 | 419 ms | 7.3 ms | 4.0 | 4 | 16,210 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 809.7 | 100.9 | 955 ms | 9.9 ms | 8.0 | 0 | 24,286 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,079.9 | 66.8 | 1,634 ms | 15.0 ms | 16.0 | 0 | 32,398 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,312.6 | 40.3 | 3,084 ms | 24.8 ms | 32.0 | 0 | 39,360 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 153.6 | 158.2 | 610 ms | 6.3 ms | 1.0 | 1 | 4,612 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 286.5 | 146.8 | 745 ms | 6.8 ms | 1.9 | 2 | 8,593 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 435.9 | 109.0 | 1,220 ms | 9.2 ms | 4.0 | 0 | 13,076 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 598.4 | 74.7 | 1,990 ms | 13.4 ms | 8.0 | 0 | 17,944 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 719.1 | 44.6 | 3,507 ms | 22.4 ms | 16.0 | 0 | 21,568 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.