Speed test matrix
Ling-3.0-tiny-base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-inclusionai-vllm-73a5831127a9/bin/vllm" serve inclusionAI/Ling-3.0-tiny-base --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.70 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align --attention-backend TRITON_MLA
Prefill
prompt tok/s by context length8K context52.6K156 ms TTFT
16K context55.5K292 ms TTFT
32K context53.2K607 ms TTFT
64K context45.6K1,414 ms TTFT
128K context33.6K3,828 ms TTFT
251K context23K10,978 ms TTFT
256K context22.4K11,467 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 292.340 ms TTFT | Looped | 721.182 ms TTFT | 1,073.9115 ms TTFT | Looped | Looped | 3,232.8418 ms TTFT | 4,471.3753 ms TTFT |
| 16K | 286.578 ms TTFT | 444.6134 ms TTFT | 695.0158 ms TTFT | Looped | 1,410.7379 ms TTFT | 2,132.5612 ms TTFT | 2,906.7914 ms TTFT | 3,909.21,633 ms TTFT |
| 32K | 279.2106 ms TTFT | 436.2162 ms TTFT | 678.9228 ms TTFT | Looped | 1,340.5548 ms TTFT | Looped | 2,618.81,430 ms TTFT | 3,469.22,528 ms TTFT |
| 64K | 267.5170 ms TTFT | 413.3245 ms TTFT | Looped | 878.7505 ms TTFT | Looped | Looped | 2,081.72,657 ms TTFT | Over capacity |
| 128K | 247.4309 ms TTFT | 371.1376 ms TTFT | 538.8506 ms TTFT | 747.4966 ms TTFT | 989.61,728 ms TTFT | 1,310.83,309 ms TTFT | Over capacity | Over capacity |
| 251K | Looped | 311.1687 ms TTFT | 446.91,177 ms TTFT | 588.42,000 ms TTFT | 743.03,633 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 52.6K | — | 156 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 55.5K | — | 292 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 53.2K | — | 607 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 45.6K | — | 1,414 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 33.6K | — | 3,828 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 23K | — | 10,978 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.4K | — | 11,467 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 292.3 | 293.0 | 40 ms | 3.4 ms | 1.0 | 2 | 8,769 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 721.1 | 181.5 | 82 ms | 5.5 ms | 3.9 | 4 | 21,631 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,073.9 | 139.3 | 115 ms | 7.2 ms | 8.0 | 8 | 32,211 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 3,232.8 | 50.9 | 418 ms | 19.6 ms | 64.0 | 0 | 96,960 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,471.3 | 35.0 | 753 ms | 28.6 ms | 128.0 | 0 | 134,144 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 286.5 | 289.1 | 78 ms | 3.5 ms | 1.0 | 2 | 8,594 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 444.6 | 223.3 | 134 ms | 4.5 ms | 2.0 | 2 | 13,336 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 695.0 | 175.3 | 158 ms | 5.7 ms | 4.0 | 4 | 20,848 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 1,410.7 | 88.9 | 379 ms | 11.3 ms | 16.0 | 0 | 42,320 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,132.5 | 67.1 | 612 ms | 14.9 ms | 32.0 | 0 | 63,976 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,906.7 | 45.3 | 914 ms | 22.1 ms | 64.0 | 0 | 87,168 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,909.2 | 30.2 | 1,633 ms | 33.2 ms | 128.0 | 0 | 117,248 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 279.2 | 282.2 | 106 ms | 3.5 ms | 1.0 | 2 | 8,375 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 436.2 | 219.5 | 162 ms | 4.6 ms | 2.0 | 2 | 13,086 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 678.9 | 171.4 | 228 ms | 5.8 ms | 4.0 | 4 | 20,368 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 1,340.5 | 84.3 | 548 ms | 11.9 ms | 16.0 | 0 | 40,208 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 2,618.8 | 40.4 | 1,430 ms | 24.7 ms | 64.0 | 0 | 78,528 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,469.2 | 26.3 | 2,528 ms | 38.0 ms | 128.0 | 0 | 104,064 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 267.5 | 270.5 | 170 ms | 3.7 ms | 1.0 | 2 | 8,025 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 413.3 | 208.5 | 245 ms | 4.8 ms | 2.0 | 2 | 12,398 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 878.7 | 109.9 | 505 ms | 9.1 ms | 8.0 | 0 | 26,369 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Valid | 2,081.7 | 31.3 | 2,657 ms | 32.0 ms | 64.0 | 0 | 62,400 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 247.4 | 250.3 | 309 ms | 4.0 ms | 1.0 | 2 | 7,376 | 29.8 s |
| Sustained decode | 128K | 2 | Valid | 371.1 | 188.3 | 376 ms | 5.3 ms | 2.0 | 2 | 11,133 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 538.8 | 136.6 | 506 ms | 7.3 ms | 3.9 | 4 | 16,168 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 747.4 | 93.3 | 966 ms | 10.7 ms | 8.0 | 0 | 22,416 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 989.6 | 61.2 | 1,728 ms | 16.3 ms | 16.0 | 0 | 29,680 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,310.8 | 40.0 | 3,309 ms | 25.0 ms | 32.0 | 0 | 39,328 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 2 | Valid | 311.1 | 159.5 | 687 ms | 6.3 ms | 1.9 | 2 | 9,332 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 446.9 | 111.9 | 1,177 ms | 8.9 ms | 4.0 | 0 | 13,404 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 588.4 | 73.5 | 2,000 ms | 13.6 ms | 8.0 | 0 | 17,648 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 743.0 | 46.2 | 3,633 ms | 21.7 ms | 16.0 | 0 | 22,288 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.