Speed test matrix
Qwen3.5-0.8B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.5-0.8B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.882880 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context93.6K88 ms TTFT
16K context116.5K139 ms TTFT
32K context132.9K243 ms TTFT
64K context118.5K544 ms TTFT
128K context95.8K1,343 ms TTFT
251K context65.8K3,833 ms TTFT
256K context63.4K4,054 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 497.759 ms TTFT | 979.677 ms TTFT | 1,937.893 ms TTFT | 3,732.2135 ms TTFT | 6,411.7192 ms TTFT | 10K253 ms TTFT | 13.9K489 ms TTFT | 17.1K777 ms TTFT |
| 16K | 487.881 ms TTFT | 956.790 ms TTFT | 1,860.7140 ms TTFT | 3,463.5161 ms TTFT | 5,705.1270 ms TTFT | 8,323.5358 ms TTFT | 10.9K356 ms TTFT | 12.9K1,302 ms TTFT |
| 32K | 470.5100 ms TTFT | 922.199 ms TTFT | 1,719.5157 ms TTFT | 3,039.1219 ms TTFT | 4,727.6266 ms TTFT | 6,408.9266 ms TTFT | 7,541.81,006 ms TTFT | 8,482.12,418 ms TTFT |
| 64K | 436.6162 ms TTFT | 852.4151 ms TTFT | 1,512.7184 ms TTFT | 2,500.5212 ms TTFT | 3,480.8244 ms TTFT | 4,134.8281 ms TTFT | 4,639.72,386 ms TTFT | 4,940.44,714 ms TTFT |
| 128K | 382.6276 ms TTFT | 740.9308 ms TTFT | 1,223.1315 ms TTFT | 1,789.5326 ms TTFT | 2,127.0381 ms TTFT | 2,467.72,789 ms TTFT | 2,595.95,298 ms TTFT | Over capacity |
| 251K | 311.8551 ms TTFT | 589.6617 ms TTFT | 870.5614 ms TTFT | 1,096.7607 ms TTFT | 1,274.53,213 ms TTFT | 1,359.26,115 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 93.6K | — | 88 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 116.5K | — | 139 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 132.9K | — | 243 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 118.5K | — | 544 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 95.8K | — | 1,343 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 65.8K | — | 3,833 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 63.4K | — | 4,054 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 497.7 | 501.8 | 59 ms | 2.0 ms | 1.0 | 4 | 14,931 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 979.6 | 495.3 | 77 ms | 2.0 ms | 2.0 | 8 | 29,388 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,937.8 | 491.3 | 93 ms | 2.0 ms | 4.0 | 16 | 58,133 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,732.2 | 473.8 | 135 ms | 2.1 ms | 8.0 | 24 | 111,966 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 6,411.7 | 408.2 | 192 ms | 2.4 ms | 16.0 | 48 | 192,345 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 10K | 319.1 | 253 ms | 3.1 ms | 32.0 | 64 | 301,062 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 13.9K | 222.3 | 489 ms | 4.5 ms | 64.0 | 64 | 417,921 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 17.1K | 139.2 | 777 ms | 7.2 ms | 124.9 | 128 | 513,911 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 487.8 | 492.2 | 81 ms | 2.0 ms | 1.0 | 3 | 14,634 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 956.7 | 484.2 | 90 ms | 2.1 ms | 2.0 | 8 | 28,702 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,860.7 | 472.3 | 140 ms | 2.1 ms | 4.0 | 12 | 55,824 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 3,463.5 | 440.6 | 161 ms | 2.3 ms | 7.9 | 24 | 103,905 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 5,705.1 | 363.3 | 270 ms | 2.8 ms | 15.6 | 32 | 171,144 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 8,323.5 | 265.7 | 358 ms | 3.8 ms | 31.4 | 64 | 249,705 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 10.9K | 170.6 | 356 ms | 5.9 ms | 63.5 | 64 | 326,465 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12.9K | 100.0 | 1,302 ms | 10.0 ms | 128.0 | 0 | 387,457 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 470.5 | 475.6 | 100 ms | 2.1 ms | 1.0 | 3 | 14,116 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 922.1 | 465.9 | 99 ms | 2.1 ms | 2.0 | 6 | 27,661 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,719.5 | 437.2 | 157 ms | 2.3 ms | 3.9 | 12 | 51,584 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 3,039.1 | 388.6 | 219 ms | 2.6 ms | 7.9 | 24 | 91,169 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 4,727.6 | 300.1 | 266 ms | 3.3 ms | 15.8 | 32 | 141,828 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 6,408.9 | 201.0 | 266 ms | 5.0 ms | 31.9 | 32 | 192,261 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,541.8 | 117.9 | 1,006 ms | 8.5 ms | 63.9 | 18 | 226,240 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,482.1 | 66.6 | 2,418 ms | 15.0 ms | 128.0 | 0 | 254,464 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 436.6 | 444.2 | 162 ms | 2.3 ms | 1.0 | 3 | 13,097 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 852.4 | 433.2 | 151 ms | 2.3 ms | 1.9 | 6 | 25,570 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,512.7 | 385.1 | 184 ms | 2.6 ms | 3.9 | 12 | 45,382 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,500.5 | 317.6 | 212 ms | 3.1 ms | 8.0 | 16 | 75,014 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,480.8 | 219.2 | 244 ms | 4.6 ms | 15.9 | 16 | 104,418 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,134.8 | 130.5 | 281 ms | 7.7 ms | 31.8 | 32 | 124,042 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,639.7 | 73.0 | 2,386 ms | 13.7 ms | 64.0 | 0 | 139,200 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,940.4 | 39.2 | 4,714 ms | 25.5 ms | 128.0 | 0 | 148,224 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 382.6 | 393.9 | 276 ms | 2.5 ms | 0.9 | 3 | 11,477 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 740.9 | 379.1 | 308 ms | 2.6 ms | 2.0 | 6 | 22,136 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 1,223.1 | 312.7 | 315 ms | 3.2 ms | 3.9 | 8 | 36,693 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,789.5 | 226.4 | 326 ms | 4.4 ms | 8.0 | 8 | 53,680 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,127.0 | 135.0 | 381 ms | 7.4 ms | 15.8 | 16 | 63,801 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,467.7 | 77.2 | 2,789 ms | 13.0 ms | 32.0 | 0 | 74,017 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,595.9 | 40.9 | 5,298 ms | 24.5 ms | 64.0 | 0 | 77,888 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 311.8 | 324.4 | 551 ms | 3.1 ms | 1.0 | 2 | 9,355 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 589.6 | 307.4 | 617 ms | 3.3 ms | 2.0 | 4 | 17,687 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 870.5 | 222.6 | 614 ms | 4.5 ms | 4.0 | 4 | 26,112 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,096.7 | 140.6 | 607 ms | 7.1 ms | 7.9 | 8 | 32,896 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,274.5 | 80.4 | 3,213 ms | 12.4 ms | 16.0 | 0 | 38,224 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,359.2 | 42.7 | 6,115 ms | 23.4 ms | 32.0 | 0 | 40,768 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.