Speed test matrix
lift on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- lift
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_GDN_DECODE_KERNEL=cuda "$HOME/env_rtxpro6k-datalab-to-vllm-73a5831127a9/bin/vllm" serve datalab-to/lift --served-model-name lift --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --mm-encoder-attn-backend FLASH_ATTN --gdn-prefill-backend flashinfer --attention-config '{"backend":"FLASH_ATTN","flash_attn_version":2}' --linear-backend torch --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context20.7K396 ms TTFT
16K context20.5K793 ms TTFT
32K context19K1,698 ms TTFT
64K context16.7K3,867 ms TTFT
128K context13.3K9,675 ms TTFT
251K context9,58026,329 ms TTFT
256K context9,49627,093 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 114.454 ms TTFT | 195.287 ms TTFT | 407.9145 ms TTFT | Looped | 1,259.91,042 ms TTFT | Looped | Looped | 3,207.12,260 ms TTFT |
| 16K | 71.489 ms TTFT | 179.9133 ms TTFT | 370.9222 ms TTFT | 691.2395 ms TTFT | Looped | 1,587.31,054 ms TTFT | 1,993.81,921 ms TTFT | Over capacity |
| 32K | 72.7111 ms TTFT | 150.6162 ms TTFT | 288.4235 ms TTFT | 574.9407 ms TTFT | Looped | 1,154.21,053 ms TTFT | Over capacity | Over capacity |
| 64K | 56.3188 ms TTFT | 105.4254 ms TTFT | 214.3386 ms TTFT | 399.2577 ms TTFT | 522.8916 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 32.3329 ms TTFT | 62.2464 ms TTFT | 127.4710 ms TTFT | 260.81,025 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 18.8673 ms TTFT | 38.3886 ms TTFT | 72.41,354 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.7K | — | 396 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.5K | — | 793 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19K | — | 1,698 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.7K | — | 3,867 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.3K | — | 9,675 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,580.0 | — | 26,329 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,496.0 | — | 27,093 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 114.4 | 115.9 | 54 ms | 8.6 ms | 1.0 | 0 | 3,430 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 195.2 | 98.4 | 87 ms | 10.2 ms | 2.0 | 0 | 5,854 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 407.9 | 104.0 | 145 ms | 9.6 ms | 4.0 | 0 | 12,230 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 1,259.9 | 80.0 | 1,042 ms | 12.5 ms | 16.0 | 0 | 37,766 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Valid | 3,207.1 | 24.4 | 2,260 ms | 41.0 ms | 128.0 | 0 | 96,037 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 71.4 | 72.7 | 89 ms | 13.8 ms | 1.0 | 0 | 2,141 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 179.9 | 90.8 | 133 ms | 11.0 ms | 2.0 | 0 | 5,394 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 370.9 | 93.0 | 222 ms | 10.8 ms | 4.0 | 0 | 11,126 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 691.2 | 87.7 | 395 ms | 11.4 ms | 8.0 | 0 | 20,720 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 1,587.3 | 49.3 | 1,054 ms | 20.3 ms | 32.0 | 0 | 47,599 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,993.8 | 30.4 | 1,921 ms | 32.9 ms | 64.0 | 0 | 59,706 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 72.7 | 72.9 | 111 ms | 13.7 ms | 1.0 | 0 | 2,179 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 150.6 | 75.0 | 162 ms | 13.3 ms | 2.0 | 0 | 4,515 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 288.4 | 71.7 | 235 ms | 13.9 ms | 4.0 | 0 | 8,648 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 574.9 | 71.0 | 407 ms | 14.1 ms | 8.0 | 0 | 17,245 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,154.2 | 35.8 | 1,053 ms | 27.9 ms | 32.0 | 0 | 34,572 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 56.3 | 56.6 | 188 ms | 17.7 ms | 1.0 | 0 | 1,690 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 105.4 | 52.9 | 254 ms | 19.0 ms | 2.0 | 0 | 3,159 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 214.3 | 53.4 | 386 ms | 18.7 ms | 4.0 | 0 | 6,429 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 399.2 | 50.1 | 577 ms | 20.0 ms | 8.0 | 0 | 11,971 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 522.8 | 32.6 | 916 ms | 30.7 ms | 16.0 | 0 | 15,677 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 32.3 | 32.2 | 329 ms | 31.1 ms | 1.0 | 0 | 967 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 62.2 | 31.0 | 464 ms | 32.3 ms | 2.0 | 0 | 1,866 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 127.4 | 32.7 | 710 ms | 30.6 ms | 4.0 | 0 | 3,818 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 260.8 | 32.1 | 1,025 ms | 31.1 ms | 8.0 | 0 | 7,817 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 18.8 | 18.6 | 673 ms | 53.7 ms | 1.0 | 0 | 562 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 38.3 | 19.3 | 886 ms | 51.8 ms | 2.0 | 0 | 1,147 | 29.9 s |
| Sustained decode | 251K | 4 | Valid | 72.4 | 17.9 | 1,354 ms | 55.9 ms | 4.0 | 0 | 2,168 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.