Speed test matrix
lift on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- lift
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-datalab-vllm/bin/vllm" serve datalab-to/lift --served-model-name lift --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.883900 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context31.3K262 ms TTFT
16K context30.9K526 ms TTFT
32K context30.3K1,068 ms TTFT
64K context28.4K2,268 ms TTFT
128K context23.9K5,381 ms TTFT
251K context18.2K13,883 ms TTFT
256K context18K14,336 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 160.778 ms TTFT | 317.3119 ms TTFT | 613.9894 ms TTFT | 1,146.7179 ms TTFT | 2,081.3265 ms TTFT | 3,449.4446 ms TTFT | 5,074.6640 ms TTFT | 6,566.01,024 ms TTFT |
| 16K | 159.0107 ms TTFT | 312.1124 ms TTFT | 594.1180 ms TTFT | 1,082.3201 ms TTFT | 1,900.4280 ms TTFT | 2,938.2459 ms TTFT | 3,971.3747 ms TTFT | 4,804.91,363 ms TTFT |
| 32K | 155.5125 ms TTFT | 301.2158 ms TTFT | 558.7224 ms TTFT | 974.5298 ms TTFT | 1,599.5432 ms TTFT | 2,237.3668 ms TTFT | 2,753.41,203 ms TTFT | Over capacity |
| 64K | 149.1167 ms TTFT | 281.6228 ms TTFT | 501.7198 ms TTFT | 820.5506 ms TTFT | 1,185.5839 ms TTFT | 1,485.91,555 ms TTFT | Over capacity | Over capacity |
| 128K | 138.2292 ms TTFT | 248.5355 ms TTFT | 414.7552 ms TTFT | 600.3865 ms TTFT | 764.91,490 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 122.3570 ms TTFT | 203.5759 ms TTFT | 300.31,138 ms TTFT | 388.12,001 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.3K | — | 262 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.9K | — | 526 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.3K | — | 1,068 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.4K | — | 2,268 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.9K | — | 5,381 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 18.2K | — | 13,883 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18K | — | 14,336 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 160.7 | 161.4 | 78 ms | 6.2 ms | 1.0 | 1 | 4,823 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 317.3 | 159.6 | 119 ms | 6.3 ms | 1.9 | 2 | 9,526 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 613.9 | 154.5 | 894 ms | 6.5 ms | 4.0 | 4 | 18,416 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,146.7 | 145.0 | 179 ms | 6.9 ms | 8.0 | 8 | 34,400 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,081.3 | 131.5 | 265 ms | 7.6 ms | 16.0 | 16 | 62,434 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,449.4 | 108.6 | 446 ms | 9.2 ms | 32.0 | 0 | 103,488 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,074.6 | 79.7 | 640 ms | 12.6 ms | 64.0 | 0 | 152,256 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,566.0 | 51.1 | 1,024 ms | 19.6 ms | 128.0 | 0 | 196,992 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 159.0 | 159.9 | 107 ms | 6.3 ms | 1.0 | 1 | 4,770 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 312.1 | 157.1 | 124 ms | 6.4 ms | 2.0 | 2 | 9,364 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 594.1 | 149.9 | 180 ms | 6.7 ms | 4.0 | 4 | 17,820 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,082.3 | 136.3 | 201 ms | 7.3 ms | 8.0 | 8 | 32,465 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,900.4 | 119.0 | 280 ms | 8.4 ms | 16.0 | 0 | 57,008 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,938.2 | 91.9 | 459 ms | 10.9 ms | 32.0 | 0 | 88,128 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,971.3 | 61.8 | 747 ms | 16.2 ms | 64.0 | 0 | 119,104 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,804.9 | 37.0 | 1,363 ms | 27.0 ms | 128.0 | 0 | 144,128 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 155.5 | 156.6 | 125 ms | 6.4 ms | 1.0 | 1 | 4,665 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 301.2 | 151.9 | 158 ms | 6.6 ms | 2.0 | 2 | 9,035 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 558.7 | 141.2 | 224 ms | 7.1 ms | 4.0 | 4 | 16,761 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 974.5 | 123.1 | 298 ms | 8.1 ms | 8.0 | 8 | 29,235 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,599.5 | 100.2 | 432 ms | 10.0 ms | 16.0 | 0 | 47,984 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,237.3 | 70.1 | 668 ms | 14.3 ms | 32.0 | 0 | 67,104 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,753.4 | 43.1 | 1,203 ms | 23.2 ms | 64.0 | 0 | 82,560 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 149.1 | 150.7 | 167 ms | 6.6 ms | 1.0 | 1 | 4,474 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 281.6 | 142.4 | 228 ms | 7.0 ms | 2.0 | 2 | 8,446 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 501.7 | 126.4 | 198 ms | 7.9 ms | 4.0 | 4 | 15,049 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 820.5 | 102.7 | 506 ms | 9.7 ms | 8.0 | 0 | 24,616 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,185.5 | 74.2 | 839 ms | 13.5 ms | 16.0 | 0 | 35,552 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,485.9 | 46.6 | 1,555 ms | 21.5 ms | 32.0 | 0 | 44,576 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 138.2 | 140.9 | 292 ms | 7.1 ms | 1.0 | 1 | 4,147 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 248.5 | 133.3 | 355 ms | 7.5 ms | 1.9 | 2 | 7,454 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 414.7 | 103.8 | 552 ms | 9.6 ms | 4.0 | 0 | 12,440 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 600.3 | 75.1 | 865 ms | 13.3 ms | 8.0 | 0 | 18,016 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 764.9 | 47.8 | 1,490 ms | 20.9 ms | 16.0 | 0 | 22,944 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 122.3 | 122.6 | 570 ms | 8.2 ms | 1.0 | 0 | 3,669 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 203.5 | 102.0 | 759 ms | 9.8 ms | 2.0 | 0 | 6,104 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 300.3 | 75.3 | 1,138 ms | 13.3 ms | 4.0 | 0 | 9,008 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 388.1 | 48.6 | 2,001 ms | 20.6 ms | 8.0 | 0 | 11,640 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.