Speed test matrix
limite-1b-base-soup on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- limite-1b-base-soup
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLUGINS=limite MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-paradigma-inc-vllm-73a5831127a9/bin/vllm" serve paradigma-inc/limite-1b-base-soup --served-model-name limite-1b-base-soup --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context94.5K87 ms TTFT
16K context95.4K170 ms TTFT
32K context84.3K383 ms TTFT
64K context67K961 ms TTFT
123K context47K2,631 ms TTFT
128K context45.7K2,813 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 252.425 ms TTFT | 450.738 ms TTFT | 901.336 ms TTFT | 1,738.741 ms TTFT | 3,223.756 ms TTFT | 4,760.271 ms TTFT | 6,545.0410 ms TTFT | 8,125.7716 ms TTFT |
| 16K | 250.939 ms TTFT | 443.445 ms TTFT | 878.048 ms TTFT | 1,663.059 ms TTFT | 2,971.758 ms TTFT | Looped | Looped | Looped |
| 32K | 240.966 ms TTFT | 424.077 ms TTFT | 829.992 ms TTFT | 1,520.586 ms TTFT | Looped | 3,483.4651 ms TTFT | 4,376.41,189 ms TTFT | Over capacity |
| 64K | 220.8107 ms TTFT | 393.5119 ms TTFT | Looped | 1,304.9124 ms TTFT | Looped | 2,560.01,201 ms TTFT | 2,995.62,342 ms TTFT | Over capacity |
| 123K | 193.6257 ms TTFT | 340.4289 ms TTFT | 630.3313 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 94.5K | — | 87 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 95.4K | — | 170 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 84.3K | — | 383 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 67K | — | 961 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 47K | — | 2,631 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 45.7K | — | 2,813 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 252.4 | 253.9 | 25 ms | 3.9 ms | 1.0 | 2 | 7,571 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 450.7 | 225.6 | 38 ms | 4.4 ms | 2.0 | 2 | 13,528 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 901.3 | 225.9 | 36 ms | 4.4 ms | 4.0 | 4 | 27,036 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,738.7 | 217.9 | 41 ms | 4.6 ms | 8.0 | 8 | 52,158 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,223.7 | 202.5 | 56 ms | 4.9 ms | 16.0 | 16 | 96,698 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,760.2 | 150.6 | 71 ms | 6.6 ms | 32.0 | 32 | 142,791 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,545.0 | 102.8 | 410 ms | 9.7 ms | 64.0 | 0 | 196,352 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 8,125.7 | 64.0 | 716 ms | 15.6 ms | 128.0 | 0 | 243,712 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 250.9 | 251.7 | 39 ms | 4.0 ms | 1.0 | 2 | 7,526 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 443.4 | 222.1 | 45 ms | 4.5 ms | 2.0 | 2 | 13,300 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 878.0 | 220.1 | 48 ms | 4.5 ms | 4.0 | 4 | 26,337 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,663.0 | 208.5 | 59 ms | 4.8 ms | 8.0 | 8 | 49,883 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,971.7 | 186.5 | 58 ms | 5.4 ms | 16.0 | 16 | 89,141 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 240.9 | 241.3 | 66 ms | 4.1 ms | 1.0 | 1 | 7,225 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 424.0 | 212.8 | 77 ms | 4.7 ms | 2.0 | 2 | 12,720 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 829.9 | 208.1 | 92 ms | 4.8 ms | 4.0 | 4 | 24,896 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,520.5 | 190.7 | 86 ms | 5.2 ms | 8.0 | 8 | 45,614 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 3,483.4 | 108.9 | 651 ms | 9.2 ms | 32.0 | 0 | 104,480 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,376.4 | 67.9 | 1,189 ms | 14.7 ms | 64.0 | 0 | 131,264 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 220.8 | 221.7 | 107 ms | 4.5 ms | 1.0 | 1 | 6,624 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 393.5 | 197.8 | 119 ms | 5.1 ms | 2.0 | 2 | 11,804 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 1,304.9 | 164.0 | 124 ms | 6.1 ms | 8.0 | 8 | 39,148 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 2,560.0 | 80.5 | 1,201 ms | 12.4 ms | 32.0 | 0 | 76,800 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,995.6 | 47.6 | 2,342 ms | 21.0 ms | 64.0 | 0 | 89,856 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 193.6 | 195.4 | 257 ms | 5.1 ms | 1.0 | 1 | 5,807 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 340.4 | 171.9 | 289 ms | 5.8 ms | 2.0 | 2 | 10,211 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 630.3 | 159.1 | 313 ms | 6.3 ms | 4.0 | 4 | 18,909 | 30.0 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.