Speed test matrix
Ling-3.0-tiny-base-midtrain on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-inclusionai-vllm-73a5831127a9/bin/vllm" serve inclusionAI/Ling-3.0-tiny-base-midtrain --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --max-model-len 262144 --gpu-memory-utilization 0.70 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align --attention-backend TRITON_MLA
Prefill
prompt tok/s by context length8K context52.6K156 ms TTFT
16K context55.9K290 ms TTFT
32K context53.2K607 ms TTFT
64K context45.4K1,420 ms TTFT
128K context34.1K3,775 ms TTFT
251K context22.9K11,016 ms TTFT
256K context22.4K11,469 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 292.239 ms TTFT | 454.9299 ms TTFT | 712.977 ms TTFT | Looped | Looped | 2,176.4256 ms TTFT | Looped | Looped |
| 16K | 285.473 ms TTFT | 443.3397 ms TTFT | 706.1153 ms TTFT | 989.0246 ms TTFT | 1,364.0384 ms TTFT | Looped | Looped | Looped |
| 32K | 279.4104 ms TTFT | 435.6503 ms TTFT | 670.3224 ms TTFT | 925.1332 ms TTFT | 1,324.3471 ms TTFT | Looped | 2,563.81,326 ms TTFT | Looped |
| 64K | 267.3171 ms TTFT | 413.2515 ms TTFT | 615.5323 ms TTFT | 869.9523 ms TTFT | Looped | 1,630.51,642 ms TTFT | 2,037.22,993 ms TTFT | Over capacity |
| 128K | 247.3313 ms TTFT | 371.9378 ms TTFT | 534.6481 ms TTFT | 756.4914 ms TTFT | Looped | 1,268.43,038 ms TTFT | Over capacity | Over capacity |
| 251K | 202.1605 ms TTFT | 311.6702 ms TTFT | 443.21,184 ms TTFT | 588.82,025 ms TTFT | 741.03,582 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 52.6K | — | 156 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 55.9K | — | 290 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 53.2K | — | 607 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 45.4K | — | 1,420 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.1K | — | 3,775 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 22.9K | — | 11,016 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.4K | — | 11,469 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 292.2 | 292.9 | 39 ms | 3.4 ms | 1.0 | 2 | 8,764 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 454.9 | 228.1 | 299 ms | 4.4 ms | 2.0 | 2 | 13,646 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 712.9 | 179.2 | 77 ms | 5.6 ms | 4.0 | 4 | 21,393 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 2,176.4 | 68.7 | 256 ms | 14.6 ms | 32.0 | 0 | 65,280 | 30.0 s |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 285.4 | 286.9 | 73 ms | 3.5 ms | 1.0 | 2 | 8,562 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 443.3 | 222.7 | 397 ms | 4.5 ms | 2.0 | 2 | 13,296 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 706.1 | 178.1 | 153 ms | 5.6 ms | 4.0 | 4 | 21,183 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 989.0 | 128.4 | 246 ms | 7.8 ms | 8.0 | 8 | 29,671 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,364.0 | 85.8 | 384 ms | 11.7 ms | 16.0 | 0 | 40,912 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 279.4 | 282.1 | 104 ms | 3.5 ms | 1.0 | 2 | 8,382 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 435.6 | 219.3 | 503 ms | 4.6 ms | 1.9 | 2 | 13,066 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 670.3 | 169.0 | 224 ms | 5.9 ms | 3.9 | 4 | 20,108 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 925.1 | 116.2 | 332 ms | 8.6 ms | 8.0 | 0 | 27,752 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,324.3 | 83.1 | 471 ms | 12.0 ms | 16.0 | 0 | 39,728 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 2,563.8 | 39.5 | 1,326 ms | 25.3 ms | 64.0 | 0 | 76,864 | 30.0 s |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 267.3 | 270.4 | 171 ms | 3.7 ms | 1.0 | 2 | 8,019 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 413.2 | 208.6 | 515 ms | 4.8 ms | 2.0 | 2 | 12,396 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 615.5 | 155.6 | 323 ms | 6.4 ms | 3.9 | 4 | 18,463 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 869.9 | 108.8 | 523 ms | 9.2 ms | 8.0 | 0 | 26,096 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 1,630.5 | 49.9 | 1,642 ms | 20.0 ms | 32.0 | 0 | 48,896 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,037.2 | 30.4 | 2,993 ms | 32.9 ms | 64.0 | 0 | 61,120 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 247.3 | 250.3 | 313 ms | 4.0 ms | 1.0 | 2 | 7,378 | 29.8 s |
| Sustained decode | 128K | 2 | Valid | 371.9 | 188.4 | 378 ms | 5.3 ms | 1.9 | 2 | 11,156 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 534.6 | 135.3 | 481 ms | 7.4 ms | 4.0 | 4 | 16,036 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 756.4 | 94.0 | 914 ms | 10.6 ms | 8.0 | 0 | 22,688 | 30.0 s |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Valid | 1,268.4 | 38.7 | 3,038 ms | 25.8 ms | 32.0 | 0 | 38,048 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 202.1 | 206.7 | 605 ms | 4.8 ms | 1.0 | 1 | 6,062 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 311.6 | 159.6 | 702 ms | 6.3 ms | 1.9 | 2 | 9,347 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 443.2 | 111.0 | 1,184 ms | 9.0 ms | 4.0 | 0 | 13,297 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 588.8 | 73.4 | 2,025 ms | 13.6 ms | 8.0 | 0 | 17,664 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 741.0 | 45.9 | 3,582 ms | 21.8 ms | 16.0 | 0 | 22,224 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.