Speed test matrix
lift on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- lift
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_GDN_DECODE_KERNEL=cuda "$HOME/env_rtxpro6k-datalab-to-vllm-73a5831127a9/bin/vllm" serve datalab-to/lift --served-model-name lift --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --mm-encoder-attn-backend FLASH_ATTN --gdn-prefill-backend flashinfer --attention-config '{"backend":"FLASH_ATTN","flash_attn_version":2}' --linear-backend torchPrefill
prompt tok/s by context length8K context21.4K382 ms TTFT
16K context21.3K760 ms TTFT
32K context19.8K1,633 ms TTFT
64K context17.5K3,690 ms TTFT
128K context14.2K9,094 ms TTFT
251K context10.3K24,451 ms TTFT
256K context10.2K25,174 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 83.144 ms TTFT | 154.967 ms TTFT | 307.482 ms TTFT | 615.9129 ms TTFT | 1,123.4180 ms TTFT | Looped | 2,959.2667 ms TTFT | 4,003.91,162 ms TTFT |
| 16K | 82.059 ms TTFT | 152.283 ms TTFT | 300.8114 ms TTFT | 595.0184 ms TTFT | 1,053.1313 ms TTFT | Looped | 2,546.91,097 ms TTFT | Over capacity |
| 32K | 79.880 ms TTFT | 147.0104 ms TTFT | 287.7194 ms TTFT | 557.6283 ms TTFT | 940.1362 ms TTFT | 1,420.0668 ms TTFT | Over capacity | Over capacity |
| 64K | 75.6136 ms TTFT | 131.4186 ms TTFT | 258.1270 ms TTFT | 490.7421 ms TTFT | 767.8742 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 68.6281 ms TTFT | 113.3371 ms TTFT | 219.8546 ms TTFT | 396.8842 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 58.1580 ms TTFT | 89.0754 ms TTFT | 170.01,138 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 21.4K | — | 382 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.3K | — | 760 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.8K | — | 1,633 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.5K | — | 3,690 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.2K | — | 9,094 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.3K | — | 24,451 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.2K | — | 25,174 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 83.1 | 83.2 | 44 ms | 12.0 ms | 1.0 | 0 | 2,494 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 154.9 | 77.5 | 67 ms | 12.9 ms | 2.0 | 0 | 4,647 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 307.4 | 76.9 | 82 ms | 13.0 ms | 4.0 | 0 | 9,220 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 615.9 | 77.0 | 129 ms | 13.0 ms | 8.0 | 0 | 18,472 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,123.4 | 70.2 | 180 ms | 14.3 ms | 16.0 | 0 | 33,696 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 2,959.2 | 46.0 | 667 ms | 21.8 ms | 64.0 | 0 | 88,768 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,003.9 | 30.9 | 1,162 ms | 32.4 ms | 128.0 | 0 | 120,064 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 82.0 | 82.0 | 59 ms | 12.2 ms | 1.0 | 0 | 2,459 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 152.2 | 76.1 | 83 ms | 13.1 ms | 2.0 | 0 | 4,566 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 300.8 | 75.2 | 114 ms | 13.3 ms | 4.0 | 0 | 9,024 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 595.0 | 74.3 | 184 ms | 13.5 ms | 8.0 | 0 | 17,848 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,053.1 | 65.7 | 313 ms | 15.2 ms | 16.0 | 0 | 31,584 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Valid | 2,546.9 | 39.2 | 1,097 ms | 25.5 ms | 64.0 | 0 | 76,352 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 79.8 | 79.8 | 80 ms | 12.5 ms | 1.0 | 0 | 2,393 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 147.0 | 73.5 | 104 ms | 13.6 ms | 2.0 | 0 | 4,408 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 287.7 | 72.0 | 194 ms | 13.9 ms | 4.0 | 0 | 8,628 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 557.6 | 69.7 | 283 ms | 14.3 ms | 8.0 | 0 | 16,720 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 940.1 | 58.6 | 362 ms | 17.1 ms | 16.0 | 0 | 28,192 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,420.0 | 44.1 | 668 ms | 22.7 ms | 32.0 | 0 | 42,592 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 75.6 | 75.6 | 136 ms | 13.2 ms | 1.0 | 0 | 2,268 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 131.4 | 65.7 | 186 ms | 15.2 ms | 2.0 | 0 | 3,940 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 258.1 | 64.5 | 270 ms | 15.5 ms | 4.0 | 0 | 7,740 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 490.7 | 61.3 | 421 ms | 16.3 ms | 8.0 | 0 | 14,720 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 767.8 | 47.8 | 742 ms | 20.9 ms | 16.0 | 0 | 23,024 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 68.6 | 68.6 | 281 ms | 14.6 ms | 1.0 | 0 | 2,056 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 113.3 | 56.7 | 371 ms | 17.6 ms | 2.0 | 0 | 3,399 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 219.8 | 54.9 | 546 ms | 18.2 ms | 4.0 | 0 | 6,592 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 396.8 | 49.2 | 842 ms | 20.3 ms | 8.0 | 0 | 11,896 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 58.1 | 58.2 | 580 ms | 17.2 ms | 1.0 | 0 | 1,741 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 89.0 | 44.6 | 754 ms | 22.4 ms | 2.0 | 0 | 2,670 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 170.0 | 42.4 | 1,138 ms | 23.6 ms | 4.0 | 0 | 5,096 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.