Speed test matrix
Ling-2.6-flash-int4 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev236+g1093c501d · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev236+g1093c501d
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1093c501dfef
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-2.6-flash-int4 --served-model-name ling --trust-remote-code --tp 1 --tool-call-parser qwen --context-length 131072 --mem-fraction-static 0.82 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton
Prefill
prompt tok/s by context length8K context13K632 ms TTFT
16K context12.9K1,256 ms TTFT
32K context9,3593,449 ms TTFT
64K context6,18810,404 ms TTFT
123K context3,39236,434 ms TTFT
128K context3,24539,626 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 114.067 ms TTFT | 191.0213 ms TTFT | 326.0764 ms TTFT | 512.1464 ms TTFT | 778.3214 ms TTFT | 986.8Requests queued409 ms TTFT | 988.4Requests queued816 ms TTFT | 978.0Requests queued1,525 ms TTFT |
| 16K | 113.069 ms TTFT | 189.0105 ms TTFT | 317.3158 ms TTFT | 485.5217 ms TTFT | 717.0422 ms TTFT | 909.9Requests queued860 ms TTFT | 929.4Requests queued1,621 ms TTFT | Over capacity |
| 32K | 111.5107 ms TTFT | 184.6137 ms TTFT | 303.9218 ms TTFT | 459.8427 ms TTFT | 649.3860 ms TTFT | Looped | Over capacity | Over capacity |
| 64K | 108.8186 ms TTFT | 178.6242 ms TTFT | 288.5403 ms TTFT | Looped | 594.02,335 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 104.6320 ms TTFT | 168.5438 ms TTFT | Looped | 378.31,467 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13K | — | 632 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.9K | — | 1,256 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,359.0 | — | 3,449 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,188.0 | — | 10,404 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 3,392.0 | — | 36,434 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,245.0 | — | 39,626 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 114.0 | 114.0 | 67 ms | 8.8 ms | 1.0 | 0 | 3,418 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 191.0 | 95.4 | 213 ms | 10.5 ms | 2.0 | 0 | 5,728 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 326.0 | 81.6 | 764 ms | 12.3 ms | 4.0 | 0 | 9,776 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 512.1 | 64.3 | 464 ms | 15.5 ms | 8.0 | 0 | 15,360 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 778.3 | 49.2 | 214 ms | 20.3 ms | 16.0 | 0 | 23,344 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 986.8 | 38.4 | 409 ms | 26.0 ms | 26.0 | 0 | 29,588 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 988.4 | 38.8 | 816 ms | 25.8 ms | 26.0 | 0 | 29,640 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 978.0 | 38.6 | 1,525 ms | 25.9 ms | 26.0 | 0 | 29,328 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 113.0 | 113.1 | 69 ms | 8.8 ms | 1.0 | 0 | 3,391 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 189.0 | 94.6 | 105 ms | 10.6 ms | 2.0 | 0 | 5,669 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 317.3 | 79.5 | 158 ms | 12.6 ms | 4.0 | 0 | 9,520 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 485.5 | 61.1 | 217 ms | 16.4 ms | 8.0 | 0 | 14,560 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 717.0 | 45.3 | 422 ms | 22.1 ms | 16.0 | 0 | 21,504 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 909.9 | 35.7 | 860 ms | 28.0 ms | 26.0 | 0 | 27,274 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 929.4 | 36.6 | 1,621 ms | 27.3 ms | 26.0 | 0 | 27,872 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 111.5 | 111.5 | 107 ms | 9.0 ms | 1.0 | 0 | 3,345 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 184.6 | 92.3 | 137 ms | 10.8 ms | 2.0 | 0 | 5,538 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 303.9 | 76.0 | 218 ms | 13.2 ms | 4.0 | 0 | 9,116 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 459.8 | 57.8 | 427 ms | 17.3 ms | 8.0 | 0 | 13,792 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 649.3 | 41.1 | 860 ms | 24.3 ms | 16.0 | 0 | 19,472 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 108.8 | 108.8 | 186 ms | 9.2 ms | 1.0 | 0 | 3,262 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 178.6 | 89.2 | 242 ms | 11.2 ms | 2.0 | 0 | 5,358 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 288.5 | 72.1 | 403 ms | 13.9 ms | 4.0 | 0 | 8,652 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 594.0 | 37.9 | 2,335 ms | 26.4 ms | 16.0 | 0 | 17,808 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 104.6 | 104.6 | 320 ms | 9.6 ms | 1.0 | 0 | 3,137 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 168.5 | 84.2 | 438 ms | 11.9 ms | 2.0 | 0 | 5,054 | 30.0 s |
| Sustained decode | 123K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 8 | Valid | 378.3 | 47.6 | 1,467 ms | 21.0 ms | 8.0 | 0 | 11,351 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.