Speed test matrix
lift on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- lift
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-datalab-vllm/bin/vllm" serve datalab-to/lift --served-model-name lift --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.884400 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context30.8K266 ms TTFT
16K context30K540 ms TTFT
32K context29K1,113 ms TTFT
64K context26.9K2,393 ms TTFT
128K context22.5K5,717 ms TTFT
251K context17K14,877 ms TTFT
256K context16.7K15,433 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 249.267 ms TTFT | 441.6363 ms TTFT | 762.192 ms TTFT | 1,613.498 ms TTFT | 2,635.4112 ms TTFT | 4,102.7280 ms TTFT | 5,700.11,176 ms TTFT | 6,830.51,554 ms TTFT |
| 16K | 232.4101 ms TTFT | 413.1113 ms TTFT | 857.1130 ms TTFT | 1,455.7122 ms TTFT | 2,484.0165 ms TTFT | 3,594.3682 ms TTFT | 4,872.91,361 ms TTFT | 5,587.62,514 ms TTFT |
| 32K | 229.6121 ms TTFT | 447.3127 ms TTFT | 778.0149 ms TTFT | 1,389.1163 ms TTFT | 2,162.6194 ms TTFT | 3,013.2795 ms TTFT | 3,589.01,506 ms TTFT | Over capacity |
| 64K | 212.4193 ms TTFT | 431.8207 ms TTFT | 716.4211 ms TTFT | 1,175.8219 ms TTFT | 1,663.2884 ms TTFT | 2,119.91,506 ms TTFT | Over capacity | Over capacity |
| 128K | 201.7281 ms TTFT | 371.1342 ms TTFT | 565.0352 ms TTFT | 886.7927 ms TTFT | 1,115.31,712 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 203.7586 ms TTFT | 318.9649 ms TTFT | 449.01,328 ms TTFT | 576.72,259 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.8K | — | 266 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30K | — | 540 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29K | — | 1,113 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.9K | — | 2,393 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.5K | — | 5,717 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17K | — | 14,877 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.7K | — | 15,433 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 249.2 | 236.1 | 67 ms | 4.2 ms | 1.0 | 2 | 7,478 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 441.6 | 222.4 | 363 ms | 4.5 ms | 2.0 | 2 | 13,247 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 762.1 | 195.9 | 92 ms | 5.1 ms | 4.0 | 4 | 22,877 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,613.4 | 204.5 | 98 ms | 4.9 ms | 8.0 | 8 | 48,391 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,635.4 | 168.6 | 112 ms | 5.9 ms | 16.0 | 16 | 79,051 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,102.7 | 127.6 | 280 ms | 7.8 ms | 32.0 | 26 | 123,077 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,700.1 | 90.3 | 1,176 ms | 11.1 ms | 64.0 | 0 | 171,042 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,830.5 | 53.4 | 1,554 ms | 18.7 ms | 128.0 | 0 | 204,799 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 232.4 | 233.1 | 101 ms | 4.3 ms | 1.0 | 1 | 6,975 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 413.1 | 212.8 | 113 ms | 4.7 ms | 2.0 | 2 | 12,400 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 857.1 | 217.8 | 130 ms | 4.6 ms | 4.0 | 4 | 25,708 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,455.7 | 192.7 | 122 ms | 5.2 ms | 8.0 | 8 | 43,673 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,484.0 | 154.2 | 165 ms | 6.5 ms | 15.9 | 16 | 74,500 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,594.3 | 112.9 | 682 ms | 8.9 ms | 32.0 | 5 | 107,833 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,872.9 | 74.9 | 1,361 ms | 13.4 ms | 64.0 | 0 | 146,201 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,587.6 | 41.9 | 2,514 ms | 23.8 ms | 128.0 | 0 | 167,607 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 229.6 | 231.0 | 121 ms | 4.3 ms | 1.0 | 1 | 6,889 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 447.3 | 223.8 | 127 ms | 4.5 ms | 2.0 | 2 | 13,424 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 778.0 | 199.2 | 149 ms | 5.0 ms | 4.0 | 4 | 23,341 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,389.1 | 177.0 | 163 ms | 5.6 ms | 8.0 | 8 | 41,666 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,162.6 | 130.0 | 194 ms | 7.7 ms | 16.0 | 15 | 64,867 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,013.2 | 93.7 | 795 ms | 10.7 ms | 32.0 | 0 | 90,386 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,589.0 | 55.7 | 1,506 ms | 18.0 ms | 64.0 | 0 | 107,674 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 212.4 | 212.2 | 193 ms | 4.7 ms | 1.0 | 1 | 6,375 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 431.8 | 216.7 | 207 ms | 4.6 ms | 2.0 | 2 | 12,950 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 716.4 | 180.9 | 211 ms | 5.5 ms | 4.0 | 4 | 21,491 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,175.8 | 147.5 | 219 ms | 6.8 ms | 8.0 | 8 | 35,273 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,663.2 | 104.5 | 884 ms | 9.6 ms | 16.0 | 0 | 49,886 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,119.9 | 66.0 | 1,506 ms | 15.1 ms | 32.0 | 0 | 63,599 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 201.7 | 207.5 | 281 ms | 4.8 ms | 1.0 | 1 | 6,052 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 371.1 | 187.5 | 342 ms | 5.3 ms | 2.0 | 2 | 11,129 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 565.0 | 151.1 | 352 ms | 6.6 ms | 4.0 | 4 | 16,960 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 886.7 | 110.5 | 927 ms | 9.1 ms | 8.0 | 0 | 26,602 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,115.3 | 70.3 | 1,712 ms | 14.2 ms | 16.0 | 0 | 33,465 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 203.7 | 206.4 | 586 ms | 4.8 ms | 1.0 | 1 | 6,111 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 318.9 | 162.0 | 649 ms | 6.2 ms | 2.0 | 2 | 9,564 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 449.0 | 111.8 | 1,328 ms | 8.9 ms | 4.0 | 0 | 13,468 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 576.7 | 72.8 | 2,259 ms | 13.8 ms | 8.0 | 0 | 17,301 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.