Speed test matrix
Qwen3.5-9B on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.882466 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-9B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context30.5K268 ms TTFT
16K context30.1K539 ms TTFT
32K context29.5K1,094 ms TTFT
64K context28.1K2,295 ms TTFT
128K context24.2K5,310 ms TTFT
251K context18.6K13,528 ms TTFT
256K context18.3K14,026 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 294.781 ms TTFT | 597.399 ms TTFT | 1,138.1109 ms TTFT | 2,021.696 ms TTFT | 3,303.3140 ms TTFT | 4,144.3239 ms TTFT | 4,928.41,646 ms TTFT | 5,252.12,738 ms TTFT |
| 16K | 287.4118 ms TTFT | 577.8131 ms TTFT | 1,134.0149 ms TTFT | 1,964.1153 ms TTFT | 3,069.8212 ms TTFT | 3,813.4813 ms TTFT | 4,482.41,911 ms TTFT | 4,675.34,263 ms TTFT |
| 32K | 290.2150 ms TTFT | 554.0163 ms TTFT | 1,041.6165 ms TTFT | 1,817.0213 ms TTFT | 2,636.2264 ms TTFT | 3,328.4954 ms TTFT | 3,697.21,691 ms TTFT | 3,886.53,170 ms TTFT |
| 64K | 271.9204 ms TTFT | 514.6219 ms TTFT | 974.5214 ms TTFT | 1,583.4248 ms TTFT | 2,093.4317 ms TTFT | 2,538.01,397 ms TTFT | 2,749.72,507 ms TTFT | 2,845.05,003 ms TTFT |
| 128K | 250.8324 ms TTFT | 475.22,899 ms TTFT | 815.5403 ms TTFT | 1,119.1549 ms TTFT | 1,483.21,720 ms TTFT | 1,701.92,979 ms TTFT | 1,779.55,590 ms TTFT | 1,816.511,358 ms TTFT |
| 251K | 192.97,103 ms TTFT | 366.07,368 ms TTFT | 569.8747 ms TTFT | 745.72,074 ms TTFT | 969.43,615 ms TTFT | 1,001.46,537 ms TTFT | 1,038.913,514 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.5K | — | 268 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.1K | — | 539 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.5K | — | 1,094 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.1K | — | 2,295 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24.2K | — | 5,310 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.6K | — | 13,528 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18.3K | — | 14,026 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 294.7 | 307.3 | 81 ms | 3.3 ms | 1.0 | 2 | 8,838 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 597.3 | 302.2 | 99 ms | 3.3 ms | 2.0 | 4 | 17,919 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,138.1 | 289.0 | 109 ms | 3.5 ms | 4.0 | 8 | 34,134 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,021.6 | 262.6 | 96 ms | 3.8 ms | 8.0 | 16 | 60,641 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,303.3 | 210.6 | 140 ms | 4.7 ms | 16.0 | 16 | 99,092 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,144.3 | 135.2 | 239 ms | 7.4 ms | 31.8 | 31 | 124,317 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,928.4 | 78.4 | 1,646 ms | 12.8 ms | 64.0 | 0 | 147,724 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,252.1 | 42.0 | 2,738 ms | 23.8 ms | 128.0 | 0 | 157,457 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 287.4 | 297.5 | 118 ms | 3.4 ms | 1.0 | 2 | 8,621 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 577.8 | 295.0 | 131 ms | 3.4 ms | 2.0 | 4 | 17,332 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,134.0 | 293.4 | 149 ms | 3.4 ms | 4.0 | 8 | 34,016 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,964.1 | 255.2 | 153 ms | 3.9 ms | 7.9 | 14 | 58,912 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,069.8 | 194.5 | 212 ms | 5.1 ms | 15.9 | 16 | 92,061 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,813.4 | 125.0 | 813 ms | 8.0 ms | 31.9 | 13 | 114,403 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,482.4 | 71.1 | 1,911 ms | 14.1 ms | 64.0 | 0 | 134,438 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,675.3 | 37.2 | 4,263 ms | 26.9 ms | 128.0 | 0 | 140,054 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 290.2 | 300.3 | 150 ms | 3.3 ms | 1.0 | 2 | 8,707 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 554.0 | 295.4 | 163 ms | 3.4 ms | 2.0 | 4 | 16,616 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,041.6 | 269.8 | 165 ms | 3.7 ms | 4.0 | 8 | 31,246 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,817.0 | 229.0 | 213 ms | 4.4 ms | 8.0 | 8 | 54,508 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,636.2 | 169.3 | 264 ms | 5.9 ms | 16.0 | 16 | 79,051 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,328.4 | 104.8 | 954 ms | 9.5 ms | 32.0 | 0 | 99,817 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,697.2 | 59.1 | 1,691 ms | 16.9 ms | 64.0 | 0 | 110,874 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,886.5 | 30.8 | 3,170 ms | 32.5 ms | 128.0 | 0 | 116,653 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 271.9 | 286.1 | 204 ms | 3.5 ms | 1.0 | 2 | 8,157 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 514.6 | 269.1 | 219 ms | 3.7 ms | 2.0 | 4 | 15,434 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 974.5 | 251.6 | 214 ms | 4.0 ms | 3.9 | 8 | 29,235 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,583.4 | 203.1 | 248 ms | 4.9 ms | 7.9 | 8 | 47,495 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,093.4 | 136.6 | 317 ms | 7.3 ms | 15.8 | 14 | 62,789 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,538.0 | 81.4 | 1,397 ms | 12.3 ms | 32.0 | 0 | 76,082 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,749.7 | 45.0 | 2,507 ms | 22.2 ms | 64.0 | 0 | 82,394 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 2,845.0 | 23.8 | 5,003 ms | 41.9 ms | 128.0 | 0 | 85,084 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 250.8 | 271.0 | 324 ms | 3.7 ms | 1.0 | 2 | 7,522 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 475.2 | 236.5 | 2,899 ms | 4.2 ms | 2.0 | 2 | 14,256 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 815.5 | 209.8 | 403 ms | 4.8 ms | 4.0 | 4 | 24,457 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,119.1 | 147.7 | 549 ms | 6.8 ms | 8.0 | 8 | 33,571 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,483.2 | 94.3 | 1,720 ms | 10.6 ms | 16.0 | 0 | 44,487 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,701.9 | 54.0 | 2,979 ms | 18.5 ms | 32.0 | 0 | 51,009 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,779.5 | 29.0 | 5,590 ms | 34.5 ms | 64.0 | 0 | 53,387 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 1,816.5 | 15.7 | 11,358 ms | 63.6 ms | 128.0 | 0 | 54,403 | 29.9 s |
| Sustained decode | 251K | 1 | Valid | 192.9 | 200.7 | 7,103 ms | 5.0 ms | 1.0 | 1 | 5,786 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 366.0 | 195.7 | 7,368 ms | 5.1 ms | 2.0 | 2 | 10,978 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 569.8 | 148.2 | 747 ms | 6.8 ms | 4.0 | 4 | 17,092 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 745.7 | 96.1 | 2,074 ms | 10.4 ms | 8.0 | 0 | 22,362 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 969.4 | 62.2 | 3,615 ms | 16.1 ms | 16.0 | 0 | 29,081 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,001.4 | 33.2 | 6,537 ms | 30.1 ms | 32.0 | 0 | 30,026 | 30.0 s |
| Sustained decode | 251K | 64 | Valid | 1,038.9 | 18.1 | 13,514 ms | 55.1 ms | 64.0 | 0 | 31,018 | 29.9 s |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.