Speed test matrix
Qwen3.5-0.8B-Base on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-0.8B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context86.8K94 ms TTFT
16K context111.2K146 ms TTFT
32K context122.8K263 ms TTFT
64K context120K537 ms TTFT
128K context97.1K1,326 ms TTFT
251K context65.8K3,832 ms TTFT
256K context63.8K4,033 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 494.749 ms TTFT | 983.184 ms TTFT | 1,947.9113 ms TTFT | 3,720.3136 ms TTFT | 6,393.9195 ms TTFT | 10K247 ms TTFT | 14K448 ms TTFT | 17.1K509 ms TTFT |
| 16K | 484.591 ms TTFT | 959.690 ms TTFT | 1,878.8129 ms TTFT | 3,459.4189 ms TTFT | 5,707.9264 ms TTFT | 8,308.0263 ms TTFT | 10.9K472 ms TTFT | 12.9K1,337 ms TTFT |
| 32K | 467.2114 ms TTFT | 922.299 ms TTFT | 1,731.1156 ms TTFT | 3,048.1188 ms TTFT | 4,728.2248 ms TTFT | 6,390.3314 ms TTFT | 7,539.41,039 ms TTFT | 8,469.02,337 ms TTFT |
| 64K | 434.6171 ms TTFT | 855.8166 ms TTFT | 1,528.8166 ms TTFT | 2,504.8187 ms TTFT | 3,474.3250 ms TTFT | 4,121.8255 ms TTFT | 4,641.62,411 ms TTFT | 4,938.64,812 ms TTFT |
| 128K | 383.0276 ms TTFT | 739.8305 ms TTFT | 1,229.9307 ms TTFT | 1,788.6354 ms TTFT | 2,131.8378 ms TTFT | 2,466.32,720 ms TTFT | 2,600.35,183 ms TTFT | Over capacity |
| 251K | 312.7532 ms TTFT | 590.0611 ms TTFT | 875.7589 ms TTFT | 1,097.3634 ms TTFT | 1,272.33,274 ms TTFT | 1,361.06,501 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 86.8K | — | 94 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 111.2K | — | 146 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 122.8K | — | 263 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 120K | — | 537 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 97.1K | — | 1,326 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 65.8K | — | 3,832 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 63.8K | — | 4,033 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 494.7 | 498.8 | 49 ms | 2.0 ms | 1.0 | 4 | 14,841 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 983.1 | 497.3 | 84 ms | 2.0 ms | 1.9 | 8 | 29,494 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,947.9 | 495.6 | 113 ms | 2.0 ms | 3.9 | 16 | 58,437 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,720.3 | 472.8 | 136 ms | 2.1 ms | 8.0 | 24 | 111,611 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 6,393.9 | 408.6 | 195 ms | 2.4 ms | 16.0 | 48 | 191,933 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 10K | 318.8 | 247 ms | 3.1 ms | 32.0 | 64 | 300,475 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 14K | 222.4 | 448 ms | 4.5 ms | 64.0 | 64 | 418,856 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 17.1K | 136.6 | 509 ms | 7.3 ms | 125.7 | 128 | 514,556 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 484.5 | 489.6 | 91 ms | 2.0 ms | 1.0 | 3 | 14,535 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 959.6 | 486.5 | 90 ms | 2.1 ms | 1.9 | 8 | 28,788 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,878.8 | 476.3 | 129 ms | 2.1 ms | 4.0 | 12 | 56,363 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,459.4 | 441.0 | 189 ms | 2.3 ms | 7.8 | 24 | 103,785 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 5,707.9 | 363.5 | 264 ms | 2.8 ms | 15.7 | 43 | 171,260 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 8,308.0 | 264.5 | 263 ms | 3.8 ms | 31.7 | 64 | 249,343 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 10.9K | 170.8 | 472 ms | 5.9 ms | 63.1 | 64 | 325,511 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12.9K | 100.0 | 1,337 ms | 10.0 ms | 128.0 | 0 | 387,705 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 467.2 | 473.4 | 114 ms | 2.1 ms | 1.0 | 3 | 14,017 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 922.2 | 468.3 | 99 ms | 2.1 ms | 2.0 | 6 | 27,667 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,731.1 | 441.2 | 156 ms | 2.3 ms | 3.9 | 12 | 51,935 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 3,048.1 | 388.6 | 188 ms | 2.6 ms | 8.0 | 24 | 91,442 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,728.2 | 299.9 | 248 ms | 3.3 ms | 15.9 | 32 | 141,844 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 6,390.3 | 200.8 | 314 ms | 5.0 ms | 31.9 | 32 | 191,709 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,539.4 | 117.9 | 1,039 ms | 8.5 ms | 63.9 | 18 | 226,186 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,469.0 | 66.5 | 2,337 ms | 15.0 ms | 128.0 | 0 | 254,076 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 434.6 | 442.5 | 171 ms | 2.3 ms | 1.0 | 3 | 13,039 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 855.8 | 435.4 | 166 ms | 2.3 ms | 1.9 | 6 | 25,675 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,528.8 | 388.8 | 166 ms | 2.6 ms | 4.0 | 12 | 45,865 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,504.8 | 317.1 | 187 ms | 3.2 ms | 8.0 | 16 | 75,143 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,474.3 | 219.1 | 250 ms | 4.6 ms | 15.9 | 16 | 104,225 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,121.8 | 130.1 | 255 ms | 7.7 ms | 31.9 | 32 | 123,646 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,641.6 | 72.9 | 2,411 ms | 13.7 ms | 64.0 | 0 | 139,264 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,938.6 | 39.1 | 4,812 ms | 25.6 ms | 128.0 | 0 | 148,223 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 383.0 | 393.9 | 276 ms | 2.5 ms | 0.9 | 3 | 11,488 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 739.8 | 380.3 | 305 ms | 2.6 ms | 2.0 | 6 | 22,193 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,229.9 | 314.1 | 307 ms | 3.2 ms | 3.9 | 8 | 36,897 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,788.6 | 226.5 | 354 ms | 4.4 ms | 7.9 | 8 | 53,659 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,131.8 | 135.1 | 378 ms | 7.4 ms | 15.9 | 16 | 63,949 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,466.3 | 77.1 | 2,720 ms | 13.0 ms | 32.0 | 0 | 73,982 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,600.3 | 40.9 | 5,183 ms | 24.4 ms | 64.0 | 0 | 78,016 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 312.7 | 324.5 | 532 ms | 3.1 ms | 1.0 | 2 | 9,385 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 590.0 | 308.0 | 611 ms | 3.2 ms | 2.0 | 4 | 17,705 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 875.7 | 223.9 | 589 ms | 4.5 ms | 3.9 | 4 | 26,269 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,097.3 | 141.0 | 634 ms | 7.1 ms | 7.8 | 8 | 32,918 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,272.3 | 80.2 | 3,274 ms | 12.5 ms | 16.0 | 0 | 38,160 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,361.0 | 42.7 | 6,501 ms | 23.4 ms | 32.0 | 0 | 40,833 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.