Speed test matrix
Qwen3.5-0.8B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-0.8B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context118.1K69 ms TTFT
16K context111.3K146 ms TTFT
32K context117.2K276 ms TTFT
64K context112.5K573 ms TTFT
128K context89.1K1,445 ms TTFT
251K context58.2K4,334 ms TTFT
256K context57.1K4,506 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 295.460 ms TTFT | 623.781 ms TTFT | 1,185.170 ms TTFT | 2,321.782 ms TTFT | 3,954.983 ms TTFT | 6,341.7152 ms TTFT | 8,980.1201 ms TTFT | 11.9K847 ms TTFT |
| 16K | 314.887 ms TTFT | 606.4103 ms TTFT | 1,209.8101 ms TTFT | 2,216.5105 ms TTFT | 3,688.7138 ms TTFT | 5,819.3163 ms TTFT | 7,914.9488 ms TTFT | 9,656.61,283 ms TTFT |
| 32K | 309.8106 ms TTFT | 594.8121 ms TTFT | 1,214.3144 ms TTFT | 2,123.6143 ms TTFT | 3,403.7143 ms TTFT | 4,837.4232 ms TTFT | 5,952.41,214 ms TTFT | 6,948.72,281 ms TTFT |
| 64K | 296.4169 ms TTFT | 605.7188 ms TTFT | 1,069.2177 ms TTFT | 1,920.7194 ms TTFT | 2,742.1216 ms TTFT | 3,393.01,237 ms TTFT | 3,901.32,636 ms TTFT | Over capacity |
| 128K | 269.6298 ms TTFT | 530.9292 ms TTFT | 948.9297 ms TTFT | 1,496.2400 ms TTFT | 1,840.01,014 ms TTFT | 2,231.52,778 ms TTFT | Over capacity | Over capacity |
| 251K | 240.3540 ms TTFT | 478.6577 ms TTFT | 714.1714 ms TTFT | 938.9955 ms TTFT | 1,121.13,230 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 118.1K | — | 69 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 111.3K | — | 146 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 117.2K | — | 276 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 112.5K | — | 573 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 89.1K | — | 1,445 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 58.2K | — | 4,334 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 57.1K | — | 4,506 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 295.4 | 298.7 | 60 ms | 3.3 ms | 1.0 | 2 | 8,863 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 623.7 | 324.4 | 81 ms | 3.1 ms | 2.0 | 4 | 18,710 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,185.1 | 300.0 | 70 ms | 3.3 ms | 4.0 | 8 | 35,552 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,321.7 | 292.9 | 82 ms | 3.4 ms | 8.0 | 16 | 69,649 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,954.9 | 252.9 | 83 ms | 4.0 ms | 16.0 | 29 | 118,655 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 6,341.7 | 201.5 | 152 ms | 5.0 ms | 31.9 | 32 | 190,263 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 8,980.1 | 147.0 | 201 ms | 6.8 ms | 63.8 | 64 | 269,416 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 11.9K | 93.8 | 847 ms | 10.7 ms | 128.0 | 1 | 356,550 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 314.8 | 326.6 | 87 ms | 3.1 ms | 1.0 | 2 | 9,445 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 606.4 | 308.7 | 103 ms | 3.2 ms | 2.0 | 4 | 18,192 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,209.8 | 313.4 | 101 ms | 3.2 ms | 4.0 | 8 | 36,292 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,216.5 | 283.6 | 105 ms | 3.5 ms | 8.0 | 16 | 66,498 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,688.7 | 235.6 | 138 ms | 4.2 ms | 15.9 | 20 | 110,664 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,819.3 | 185.0 | 163 ms | 5.4 ms | 32.0 | 32 | 174,556 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,914.9 | 122.5 | 488 ms | 8.2 ms | 63.9 | 35 | 237,419 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 9,656.6 | 75.5 | 1,283 ms | 13.2 ms | 128.0 | 1 | 289,714 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 309.8 | 306.4 | 106 ms | 3.3 ms | 1.0 | 2 | 9,293 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 594.8 | 308.6 | 121 ms | 3.2 ms | 2.0 | 4 | 17,844 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,214.3 | 298.9 | 144 ms | 3.3 ms | 4.0 | 8 | 36,431 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,123.6 | 276.0 | 143 ms | 3.6 ms | 8.0 | 16 | 63,701 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,403.7 | 211.8 | 143 ms | 4.7 ms | 15.9 | 17 | 102,107 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,837.4 | 156.2 | 232 ms | 6.4 ms | 31.9 | 32 | 145,109 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,952.4 | 94.9 | 1,214 ms | 10.5 ms | 64.0 | 2 | 178,603 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,948.7 | 54.8 | 2,281 ms | 18.2 ms | 128.0 | 0 | 208,385 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 296.4 | 303.9 | 169 ms | 3.3 ms | 1.0 | 2 | 8,896 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 605.7 | 303.3 | 188 ms | 3.3 ms | 2.0 | 4 | 18,169 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,069.2 | 278.3 | 177 ms | 3.6 ms | 4.0 | 8 | 32,077 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,920.7 | 254.8 | 194 ms | 3.9 ms | 7.9 | 14 | 57,619 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,742.1 | 176.5 | 216 ms | 5.7 ms | 15.9 | 16 | 82,259 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,393.0 | 108.5 | 1,237 ms | 9.2 ms | 32.0 | 3 | 101,780 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,901.3 | 64.3 | 2,636 ms | 15.6 ms | 64.0 | 0 | 117,061 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 269.6 | 287.2 | 298 ms | 3.5 ms | 1.0 | 2 | 8,088 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 530.9 | 274.4 | 292 ms | 3.6 ms | 1.9 | 4 | 15,926 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 948.9 | 245.5 | 297 ms | 4.1 ms | 3.9 | 6 | 28,425 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,496.2 | 192.3 | 400 ms | 5.2 ms | 8.0 | 8 | 44,886 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,840.0 | 117.3 | 1,014 ms | 8.5 ms | 16.0 | 8 | 55,116 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,231.5 | 70.0 | 2,778 ms | 14.3 ms | 32.0 | 0 | 66,945 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 240.3 | 254.5 | 540 ms | 4.0 ms | 1.0 | 1 | 7,210 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 478.6 | 264.5 | 577 ms | 3.8 ms | 2.0 | 4 | 14,354 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 714.1 | 185.2 | 714 ms | 5.4 ms | 3.9 | 4 | 21,421 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 938.9 | 132.3 | 955 ms | 7.6 ms | 7.9 | 6 | 28,077 | 29.9 s |
| Sustained decode | 251K | 16 | Valid | 1,121.1 | 71.0 | 3,230 ms | 14.1 ms | 16.0 | 1 | 33,623 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.