Speed test matrix
Devstral-Small-2-24B-Instruct-2512 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev220+gfa7784d14 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev220+gfa7784d14
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
fa7784d140ee
Run
- Served model
- devstral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-mistralai-sglang-fa7784d140ee/bin/sglang" serve --model-path mistralai/Devstral-Small-2-24B-Instruct-2512 --served-model-name devstral --trust-remote-code --tp 1 --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton
Prefill
prompt tok/s by context length8K context11.8K694 ms TTFT
16K context9,4901,714 ms TTFT
32K context6,7184,821 ms TTFT
64K context3,38919,073 ms TTFT
128K context1,79871,803 ms TTFT
251K context934.0270,881 ms TTFT
256K context921.0280,195 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 46.747 ms TTFT | 93.767 ms TTFT | 186.7655 ms TTFT | 368.4108 ms TTFT | 629.0138 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 40.668 ms TTFT | 81.898 ms TTFT | 163.0128 ms TTFT | 321.4164 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 32.1111 ms TTFT | 65.2162 ms TTFT | 130.2214 ms TTFT | 256.5284 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 22.6196 ms TTFT | 46.2291 ms TTFT | 92.6381 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 14.2368 ms TTFT | 29.3548 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 8.3Requests queued699 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.8K | — | 694 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,490.0 | — | 1,714 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,718.0 | — | 4,821 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,389.0 | — | 19,073 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,798.0 | — | 71,803 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 934.0 | — | 270,881 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 921.0 | — | 280,195 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 46.7 | 46.8 | 47 ms | 21.4 ms | 1.0 | 0 | 1,401 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 93.7 | 46.9 | 67 ms | 21.3 ms | 2.0 | 0 | 2,810 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 186.7 | 46.8 | 655 ms | 21.4 ms | 4.0 | 0 | 5,600 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 368.4 | 46.2 | 108 ms | 21.7 ms | 8.0 | 0 | 11,048 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 629.0 | 39.4 | 138 ms | 25.4 ms | 16.0 | 0 | 18,864 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 40.6 | 40.6 | 68 ms | 24.6 ms | 1.0 | 0 | 1,217 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 81.8 | 40.9 | 98 ms | 24.5 ms | 2.0 | 0 | 2,452 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 163.0 | 40.8 | 128 ms | 24.5 ms | 4.0 | 0 | 4,888 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 321.4 | 40.3 | 164 ms | 24.8 ms | 8.0 | 0 | 9,640 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 32.1 | 32.2 | 111 ms | 31.1 ms | 1.0 | 0 | 964 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 65.2 | 32.6 | 162 ms | 30.7 ms | 2.0 | 0 | 1,954 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 130.2 | 32.6 | 214 ms | 30.7 ms | 4.0 | 0 | 3,904 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 256.5 | 32.1 | 284 ms | 31.2 ms | 8.0 | 0 | 7,688 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 22.6 | 22.6 | 196 ms | 44.2 ms | 1.0 | 0 | 679 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 46.2 | 23.0 | 291 ms | 43.4 ms | 2.0 | 0 | 1,384 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 92.6 | 23.2 | 381 ms | 43.2 ms | 4.0 | 0 | 2,776 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 14.2 | 14.2 | 368 ms | 70.3 ms | 1.0 | 0 | 427 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 29.3 | 14.5 | 548 ms | 68.8 ms | 2.0 | 0 | 876 | 29.9 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Requests queued | 8.3 | 8.3 | 699 ms | 120.8 ms | 0.9 | 0 | 247 | 29.9 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.