Speed test matrix
Ling-3.0-tiny-fp8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 "$HOME/env_rtxpro6k-inclusionai-vllm-73a5831127a9/bin/vllm" serve inclusionAI/Ling-3.0-tiny-fp8 --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 131072 --gpu-memory-utilization 0.75 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align --hf-overrides '{"quantization_config":{"activation_scheme":"dynamic","fmt":"e4m3","modules_to_not_convert":["q_a_proj","q_b_proj","fused_qkv_a_proj_with_mqa","kv_a_proj_with_mqa","kv_b_proj","lm_head","word_embeddings","b_proj","mlp.gate","3.attention.g_proj","7.attention.g_proj","11.attention.g_proj","15.attention.g_proj","19.attention.g_proj","23.attention.g_proj","fused_qkv_a_proj"],"quant_method":"fp8","scale_fmt":"ue8m0","weight_block_size":[128,128]}}' --attention-backend TRITON_MLA --linear-backend triton --moe-backend tritonPrefill
prompt tok/s by context length8K context58.6K140 ms TTFT
16K context60.5K268 ms TTFT
32K context58.2K556 ms TTFT
64K context49.3K1,307 ms TTFT
123K context37K3,341 ms TTFT
128K context33K3,904 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 209.942 ms TTFT | 378.365 ms TTFT | 688.570 ms TTFT | 1,155.0110 ms TTFT | Looped | Looped | 3,976.9460 ms TTFT | 5,377.7780 ms TTFT |
| 16K | 207.481 ms TTFT | 379.7133 ms TTFT | 676.9153 ms TTFT | 1,117.9231 ms TTFT | 1,707.7377 ms TTFT | 2,612.5636 ms TTFT | 3,596.0899 ms TTFT | 4,766.71,478 ms TTFT |
| 32K | 203.0110 ms TTFT | 371.0168 ms TTFT | 655.6227 ms TTFT | 1,032.8338 ms TTFT | Looped | Looped | 3,120.11,338 ms TTFT | 4,057.82,487 ms TTFT |
| 64K | 195.3174 ms TTFT | 355.1252 ms TTFT | 608.2341 ms TTFT | 951.6519 ms TTFT | Looped | 1,937.91,569 ms TTFT | 2,493.02,991 ms TTFT | 3,086.25,954 ms TTFT |
| 123K | 185.7320 ms TTFT | 328.5368 ms TTFT | 533.2480 ms TTFT | Looped | 1,121.81,655 ms TTFT | 1,488.33,001 ms TTFT | 1,810.75,634 ms TTFT | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 58.6K | — | 140 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 60.5K | — | 268 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 58.2K | — | 556 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 49.3K | — | 1,307 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 37K | — | 3,341 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 33K | — | 3,904 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 209.9 | 210.2 | 42 ms | 4.8 ms | 1.0 | 1 | 6,297 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 378.3 | 189.9 | 65 ms | 5.3 ms | 2.0 | 2 | 11,350 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 688.5 | 172.8 | 70 ms | 5.8 ms | 4.0 | 4 | 20,652 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,155.0 | 147.2 | 110 ms | 6.8 ms | 8.0 | 8 | 34,661 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 3,976.9 | 62.3 | 460 ms | 16.1 ms | 64.0 | 0 | 119,295 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,377.7 | 41.9 | 780 ms | 23.9 ms | 128.0 | 0 | 161,283 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 207.4 | 208.0 | 81 ms | 4.8 ms | 1.0 | 1 | 6,221 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 379.7 | 190.8 | 133 ms | 5.2 ms | 1.9 | 2 | 11,392 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 676.9 | 170.5 | 153 ms | 5.9 ms | 4.0 | 4 | 20,306 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,117.9 | 143.0 | 231 ms | 7.0 ms | 8.0 | 8 | 33,534 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,707.7 | 107.3 | 377 ms | 9.3 ms | 16.0 | 0 | 51,219 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,612.5 | 82.2 | 636 ms | 12.2 ms | 32.0 | 0 | 78,367 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,596.0 | 56.3 | 899 ms | 17.8 ms | 64.0 | 0 | 107,837 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,766.7 | 36.9 | 1,478 ms | 27.1 ms | 128.0 | 0 | 142,972 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 203.0 | 203.9 | 110 ms | 4.9 ms | 1.0 | 1 | 6,089 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 371.0 | 186.8 | 168 ms | 5.4 ms | 2.0 | 2 | 11,128 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 655.6 | 165.4 | 227 ms | 6.0 ms | 4.0 | 4 | 19,668 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,032.8 | 131.4 | 338 ms | 7.6 ms | 8.0 | 8 | 30,985 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 3,120.1 | 48.1 | 1,338 ms | 20.8 ms | 64.0 | 0 | 93,573 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,057.8 | 30.8 | 2,487 ms | 32.5 ms | 128.0 | 0 | 121,733 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 195.3 | 196.7 | 174 ms | 5.1 ms | 1.0 | 1 | 5,858 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 355.1 | 179.4 | 252 ms | 5.6 ms | 2.0 | 2 | 10,653 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 608.2 | 153.9 | 341 ms | 6.5 ms | 4.0 | 4 | 18,244 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 951.6 | 119.1 | 519 ms | 8.4 ms | 8.0 | 0 | 28,544 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 1,937.9 | 59.5 | 1,569 ms | 16.8 ms | 32.0 | 0 | 58,137 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,493.0 | 37.2 | 2,991 ms | 26.9 ms | 64.0 | 0 | 74,760 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,086.2 | 22.1 | 5,954 ms | 45.3 ms | 128.0 | 0 | 92,542 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 185.7 | 187.9 | 320 ms | 5.3 ms | 1.0 | 1 | 5,571 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 328.5 | 166.3 | 368 ms | 6.0 ms | 2.0 | 2 | 9,855 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 533.2 | 134.9 | 480 ms | 7.4 ms | 3.9 | 4 | 15,996 | 30.0 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Valid | 1,121.8 | 69.1 | 1,655 ms | 14.5 ms | 16.0 | 0 | 33,649 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,488.3 | 45.1 | 3,001 ms | 22.2 ms | 32.0 | 0 | 44,640 | 30.0 s |
| Sustained decode | 123K | 64 | Valid | 1,810.7 | 26.9 | 5,634 ms | 37.2 ms | 64.0 | 0 | 54,276 | 30.0 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.