Speed test matrix
Qwen3.5-0.8B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-0.8B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883600 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context91.8K89 ms TTFT
16K context102.9K158 ms TTFT
32K context113.3K285 ms TTFT
64K context111.2K579 ms TTFT
128K context87.5K1,470 ms TTFT
251K context57.8K4,364 ms TTFT
256K context57.4K4,478 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 441.666 ms TTFT | 854.375 ms TTFT | 1,681.395 ms TTFT | 3,146.578 ms TTFT | 5,380.3101 ms TTFT | 8,517.1103 ms TTFT | 12.1K201 ms TTFT | 15.6K620 ms TTFT |
| 16K | 419.994 ms TTFT | 854.0100 ms TTFT | 1,699.3100 ms TTFT | 3,049.4100 ms TTFT | 5,066.8106 ms TTFT | 7,777.5139 ms TTFT | 10.6K231 ms TTFT | 12.7K1,306 ms TTFT |
| 32K | 422.6120 ms TTFT | 827.6119 ms TTFT | 1,566.1128 ms TTFT | 2,849.0126 ms TTFT | 4,739.3146 ms TTFT | 6,475.4208 ms TTFT | 7,880.8531 ms TTFT | 8,975.82,455 ms TTFT |
| 64K | 404.2154 ms TTFT | 792.5168 ms TTFT | 1,440.4173 ms TTFT | 2,592.7198 ms TTFT | 3,627.0266 ms TTFT | 4,496.8349 ms TTFT | 5,206.42,686 ms TTFT | Over capacity |
| 128K | 367.8282 ms TTFT | 713.0343 ms TTFT | 1,277.9356 ms TTFT | 1,969.8405 ms TTFT | 2,458.1524 ms TTFT | 2,864.63,063 ms TTFT | Over capacity | Over capacity |
| 251K | 349.4534 ms TTFT | 626.4Requests queued610 ms TTFT | 975.3639 ms TTFT | 1,387.1687 ms TTFT | 1,523.33,292 ms TTFT | 1,592.36,697 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 91.8K | — | 89 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 102.9K | — | 158 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 113.3K | — | 285 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 111.2K | — | 579 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 87.5K | — | 1,470 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 57.8K | — | 4,364 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 57.4K | — | 4,478 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 441.6 | 446.4 | 66 ms | 2.2 ms | 1.0 | 3 | 13,248 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 854.3 | 422.2 | 75 ms | 2.4 ms | 2.0 | 6 | 25,628 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,681.3 | 415.4 | 95 ms | 2.4 ms | 4.0 | 12 | 50,435 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,146.5 | 393.1 | 78 ms | 2.5 ms | 7.9 | 24 | 94,397 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,380.3 | 338.4 | 101 ms | 3.0 ms | 15.9 | 33 | 161,399 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 8,517.1 | 264.5 | 103 ms | 3.8 ms | 31.8 | 64 | 255,504 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 12.1K | 186.4 | 201 ms | 5.4 ms | 63.8 | 65 | 363,563 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15.6K | 120.3 | 620 ms | 8.3 ms | 127.7 | 58 | 466,830 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 419.9 | 427.7 | 94 ms | 2.3 ms | 1.0 | 3 | 12,596 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 854.0 | 432.6 | 100 ms | 2.3 ms | 2.0 | 6 | 25,618 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,699.3 | 418.1 | 100 ms | 2.4 ms | 4.0 | 12 | 50,974 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,049.4 | 378.4 | 100 ms | 2.6 ms | 8.0 | 24 | 91,475 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 5,066.8 | 315.1 | 106 ms | 3.2 ms | 15.9 | 32 | 152,006 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 7,777.5 | 240.6 | 139 ms | 4.2 ms | 31.7 | 55 | 233,191 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 10.6K | 162.1 | 231 ms | 6.2 ms | 63.7 | 64 | 317,151 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12.7K | 97.4 | 1,306 ms | 10.3 ms | 128.0 | 4 | 381,572 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 422.6 | 422.1 | 120 ms | 2.4 ms | 1.0 | 3 | 12,678 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 827.6 | 408.5 | 119 ms | 2.4 ms | 2.0 | 6 | 24,841 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,566.1 | 390.5 | 128 ms | 2.6 ms | 3.9 | 12 | 46,978 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,849.0 | 354.7 | 126 ms | 2.8 ms | 7.9 | 19 | 85,471 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,739.3 | 284.0 | 146 ms | 3.5 ms | 15.8 | 32 | 142,181 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 6,475.4 | 199.0 | 208 ms | 5.0 ms | 31.9 | 32 | 194,272 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,880.8 | 118.4 | 531 ms | 8.4 ms | 63.7 | 49 | 236,356 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,975.8 | 68.2 | 2,455 ms | 14.7 ms | 128.0 | 0 | 269,202 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 404.2 | 408.0 | 154 ms | 2.5 ms | 1.0 | 3 | 12,132 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 792.5 | 405.1 | 168 ms | 2.5 ms | 2.0 | 6 | 23,774 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,440.4 | 368.6 | 173 ms | 2.7 ms | 3.9 | 12 | 43,178 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,592.7 | 321.7 | 198 ms | 3.1 ms | 7.9 | 16 | 77,783 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,627.0 | 222.5 | 266 ms | 4.5 ms | 16.0 | 18 | 108,795 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,496.8 | 140.5 | 349 ms | 7.1 ms | 31.8 | 32 | 134,893 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,206.4 | 80.5 | 2,686 ms | 12.4 ms | 64.0 | 0 | 156,138 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 367.8 | 372.1 | 282 ms | 2.7 ms | 1.0 | 2 | 11,034 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 713.0 | 364.6 | 343 ms | 2.7 ms | 2.0 | 4 | 21,388 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,277.9 | 324.3 | 356 ms | 3.1 ms | 4.0 | 8 | 38,337 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,969.8 | 248.5 | 405 ms | 4.0 ms | 7.9 | 16 | 59,090 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,458.1 | 151.6 | 524 ms | 6.6 ms | 15.9 | 16 | 73,741 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,864.6 | 88.1 | 3,063 ms | 11.3 ms | 32.0 | 0 | 85,903 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 349.4 | 345.7 | 534 ms | 2.9 ms | 1.0 | 2 | 10,482 | 30.0 s |
| Sustained decode | 251K | 2 | Requests queued | 626.4 | 323.3 | 610 ms | 3.1 ms | 1.9 | 4 | 18,793 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 975.3 | 257.6 | 639 ms | 3.9 ms | 3.8 | 8 | 29,259 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,387.1 | 158.7 | 687 ms | 6.3 ms | 7.8 | 8 | 41,608 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,523.3 | 93.1 | 3,292 ms | 10.7 ms | 15.9 | 4 | 45,686 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,592.3 | 50.8 | 6,697 ms | 19.7 ms | 32.0 | 0 | 47,751 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.