Speed test matrix
Ling-3.0-tiny on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-inclusionai-vllm-73a5831127a9/bin/vllm" serve inclusionAI/Ling-3.0-tiny --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 131072 --gpu-memory-utilization 0.74 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align --attention-backend TRITON_MLA
Prefill
prompt tok/s by context length8K context53.3K154 ms TTFT
16K context55.2K294 ms TTFT
32K context53.8K601 ms TTFT
64K context45.6K1,413 ms TTFT
123K context35.2K3,514 ms TTFT
128K context34.1K3,770 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 297.242 ms TTFT | 465.069 ms TTFT | 744.073 ms TTFT | 1,133.8124 ms TTFT | Looped | 2,431.8280 ms TTFT | 3,500.9455 ms TTFT | 4,912.1782 ms TTFT |
| 16K | 294.176 ms TTFT | 457.3125 ms TTFT | 723.6154 ms TTFT | 1,079.7245 ms TTFT | 1,512.6385 ms TTFT | 2,287.2709 ms TTFT | 3,222.31,026 ms TTFT | 4,392.51,603 ms TTFT |
| 32K | 285.1108 ms TTFT | 446.1166 ms TTFT | Looped | Looped | 1,442.6483 ms TTFT | 2,092.9763 ms TTFT | 2,822.71,323 ms TTFT | 3,781.72,519 ms TTFT |
| 64K | 270.7169 ms TTFT | 423.2254 ms TTFT | 634.1351 ms TTFT | 915.8526 ms TTFT | 1,262.3830 ms TTFT | 1,743.61,496 ms TTFT | 2,294.92,756 ms TTFT | Over capacity |
| 123K | 247.6305 ms TTFT | 385.0375 ms TTFT | 560.1495 ms TTFT | 794.8957 ms TTFT | 1,055.11,696 ms TTFT | 1,388.13,224 ms TTFT | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 53.3K | — | 154 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 55.2K | — | 294 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 53.8K | — | 601 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 45.6K | — | 1,413 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 35.2K | — | 3,514 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.1K | — | 3,770 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 297.2 | 297.8 | 42 ms | 3.4 ms | 1.0 | 2 | 8,915 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 465.0 | 233.3 | 69 ms | 4.3 ms | 2.0 | 2 | 13,950 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 744.0 | 186.8 | 73 ms | 5.4 ms | 4.0 | 4 | 22,328 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,133.8 | 145.0 | 124 ms | 6.9 ms | 8.0 | 8 | 34,013 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 2,431.8 | 76.9 | 280 ms | 13.0 ms | 32.0 | 0 | 72,931 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,500.9 | 55.0 | 455 ms | 18.2 ms | 64.0 | 0 | 105,027 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,912.1 | 38.4 | 782 ms | 26.1 ms | 128.0 | 0 | 147,328 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 294.1 | 295.6 | 76 ms | 3.4 ms | 1.0 | 2 | 8,821 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 457.3 | 229.8 | 125 ms | 4.4 ms | 2.0 | 2 | 13,718 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 723.6 | 182.6 | 154 ms | 5.5 ms | 3.9 | 4 | 21,708 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,079.7 | 139.5 | 245 ms | 7.2 ms | 8.0 | 8 | 32,389 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,512.6 | 95.7 | 385 ms | 10.4 ms | 16.0 | 0 | 45,380 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,287.2 | 72.5 | 709 ms | 13.8 ms | 32.0 | 0 | 68,611 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,222.3 | 50.6 | 1,026 ms | 19.8 ms | 64.0 | 0 | 96,642 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,392.5 | 34.1 | 1,603 ms | 29.3 ms | 128.0 | 0 | 131,777 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 285.1 | 287.9 | 108 ms | 3.5 ms | 1.0 | 2 | 8,551 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 446.1 | 224.3 | 166 ms | 4.5 ms | 2.0 | 2 | 13,383 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 1,442.6 | 90.9 | 483 ms | 11.0 ms | 16.0 | 0 | 43,267 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,092.9 | 65.6 | 763 ms | 15.3 ms | 32.0 | 0 | 62,789 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,822.7 | 43.8 | 1,323 ms | 22.9 ms | 64.0 | 0 | 84,668 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,781.7 | 28.8 | 2,519 ms | 34.7 ms | 128.0 | 0 | 113,405 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 270.7 | 273.9 | 169 ms | 3.7 ms | 1.0 | 2 | 8,119 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 423.2 | 213.8 | 254 ms | 4.7 ms | 2.0 | 2 | 12,695 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 634.1 | 161.0 | 351 ms | 6.2 ms | 3.9 | 4 | 19,020 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 915.8 | 115.0 | 526 ms | 8.7 ms | 8.0 | 0 | 27,472 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,262.3 | 78.6 | 830 ms | 12.7 ms | 16.0 | 0 | 37,856 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,743.6 | 54.3 | 1,496 ms | 18.4 ms | 32.0 | 0 | 52,293 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,294.9 | 34.5 | 2,756 ms | 28.9 ms | 64.0 | 0 | 68,868 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 247.6 | 250.4 | 305 ms | 4.0 ms | 1.0 | 2 | 7,377 | 29.8 s |
| Sustained decode | 123K | 2 | Valid | 385.0 | 195.2 | 375 ms | 5.1 ms | 2.0 | 2 | 11,549 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 560.1 | 141.9 | 495 ms | 7.0 ms | 4.0 | 4 | 16,803 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 794.8 | 98.5 | 957 ms | 10.2 ms | 8.0 | 0 | 23,841 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,055.1 | 65.2 | 1,696 ms | 15.3 ms | 16.0 | 0 | 31,651 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,388.1 | 42.4 | 3,224 ms | 23.6 ms | 32.0 | 0 | 41,633 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.