Speed test matrix
limite-1b-base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- limite-1b-base
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLUGINS=limite MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-paradigma-inc-vllm-73a5831127a9/bin/vllm" serve paradigma-inc/limite-1b-base --served-model-name limite-1b-base --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context94.2K87 ms TTFT
16K context95.7K170 ms TTFT
32K context84.5K382 ms TTFT
64K context66.7K965 ms TTFT
123K context46.9K2,638 ms TTFT
128K context45.7K2,815 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 254.125 ms TTFT | 452.930 ms TTFT | 901.737 ms TTFT | 1,741.755 ms TTFT | 3,218.664 ms TTFT | 4,772.298 ms TTFT | 6,548.6368 ms TTFT | Looped |
| 16K | 249.038 ms TTFT | 443.046 ms TTFT | 877.653 ms TTFT | 1,663.456 ms TTFT | 2,968.362 ms TTFT | Looped | Looped | 6,830.61,185 ms TTFT |
| 32K | 241.766 ms TTFT | 424.073 ms TTFT | 831.183 ms TTFT | Looped | 2,549.498 ms TTFT | Looped | 4,377.91,193 ms TTFT | Over capacity |
| 64K | 221.1125 ms TTFT | 392.8144 ms TTFT | 751.5144 ms TTFT | 1,298.4143 ms TTFT | 1,984.7154 ms TTFT | Looped | 3,001.82,484 ms TTFT | Over capacity |
| 123K | 194.1266 ms TTFT | Looped | 631.5335 ms TTFT | 1,009.3322 ms TTFT | 1,441.21,473 ms TTFT | 1,711.12,759 ms TTFT | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 94.2K | — | 87 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 95.7K | — | 170 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 84.5K | — | 382 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 66.7K | — | 965 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 46.9K | — | 2,638 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 45.7K | — | 2,815 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 254.1 | 255.3 | 25 ms | 3.9 ms | 1.0 | 2 | 7,622 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 452.9 | 226.8 | 30 ms | 4.4 ms | 2.0 | 2 | 13,588 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 901.7 | 226.0 | 37 ms | 4.4 ms | 4.0 | 4 | 27,052 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,741.7 | 218.4 | 55 ms | 4.6 ms | 8.0 | 8 | 52,248 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,218.6 | 202.2 | 64 ms | 4.9 ms | 16.0 | 16 | 96,548 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,772.2 | 151.1 | 98 ms | 6.6 ms | 32.0 | 32 | 143,154 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,548.6 | 102.9 | 368 ms | 9.7 ms | 64.0 | 0 | 196,416 | 30.0 s |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 249.0 | 250.8 | 38 ms | 4.0 ms | 1.0 | 2 | 7,469 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 443.0 | 222.0 | 46 ms | 4.5 ms | 2.0 | 2 | 13,291 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 877.6 | 219.8 | 53 ms | 4.6 ms | 4.0 | 4 | 26,326 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,663.4 | 208.6 | 56 ms | 4.8 ms | 8.0 | 8 | 49,899 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,968.3 | 186.3 | 62 ms | 5.4 ms | 16.0 | 16 | 89,047 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Valid | 6,830.6 | 53.3 | 1,185 ms | 18.7 ms | 128.0 | 0 | 204,928 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 241.7 | 242.2 | 66 ms | 4.1 ms | 1.0 | 1 | 7,251 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 424.0 | 212.7 | 73 ms | 4.7 ms | 2.0 | 2 | 12,721 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 831.1 | 208.4 | 83 ms | 4.8 ms | 4.0 | 4 | 24,935 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 2,549.4 | 159.9 | 98 ms | 6.3 ms | 15.9 | 16 | 76,481 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 4,377.9 | 67.9 | 1,193 ms | 14.7 ms | 64.0 | 0 | 131,328 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 221.1 | 222.1 | 125 ms | 4.5 ms | 1.0 | 1 | 6,632 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 392.8 | 197.5 | 144 ms | 5.1 ms | 2.0 | 2 | 11,785 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 751.5 | 188.8 | 144 ms | 5.3 ms | 4.0 | 4 | 22,543 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,298.4 | 163.0 | 143 ms | 6.1 ms | 8.0 | 8 | 38,965 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,984.7 | 124.9 | 154 ms | 8.0 ms | 15.9 | 16 | 59,541 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Valid | 3,001.8 | 46.2 | 2,484 ms | 21.6 ms | 64.0 | 0 | 90,048 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 194.1 | 196.1 | 266 ms | 5.1 ms | 1.0 | 1 | 5,823 | 30.0 s |
| Sustained decode | 123K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 4 | Valid | 631.5 | 159.4 | 335 ms | 6.3 ms | 4.0 | 4 | 18,944 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 1,009.3 | 127.6 | 322 ms | 7.8 ms | 7.9 | 8 | 30,280 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,441.2 | 89.9 | 1,473 ms | 11.1 ms | 16.0 | 0 | 43,232 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,711.1 | 53.1 | 2,759 ms | 18.8 ms | 32.0 | 0 | 51,328 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.