Speed test matrix
Shieldstral-1.0-3B on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev220+gfa7784d14 · BF163 contexts (8K–27K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 27K, 32K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev220+gfa7784d14
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
fa7784d140ee
Run
- Served model
- shieldstral
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-mistralai-sglang-fa7784d140ee/bin/sglang" serve --model-path mistralai/Shieldstral-1.0-3B --served-model-name shieldstral --tp 1 --context-length 32768 --mem-fraction-static 0.80 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend flashinfer
Prefill
prompt tok/s by context length8K context39.4K208 ms TTFT
16K context34.3K475 ms TTFT
27K context28.1K972 ms TTFT
32K context26.4K1,228 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 150.116 ms TTFT | 295.124 ms TTFT | 560.030 ms TTFT | 993.030 ms TTFT | 1,521.944 ms TTFT | 2,190.979 ms TTFT | Over capacity | Over capacity |
| 16K | 138.416 ms TTFT | 271.324 ms TTFT | 512.528 ms TTFT | 881.135 ms TTFT | 1,241.158 ms TTFT | 1,659.899 ms TTFT | Over capacity | Over capacity |
| 27K | 124.717 ms TTFT | 244.325 ms TTFT | 461.533 ms TTFT | 757.437 ms TTFT | 969.255 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.4K | — | 208 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.3K | — | 475 ms | — | — | 1 | — | — |
| Prefill | 27K | 1 | Valid | 28.1K | — | 972 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 26.4K | — | 1,228 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 150.1 | 151.3 | 16 ms | 6.6 ms | 1.0 | 1 | 4,503 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 295.1 | 149.8 | 24 ms | 6.7 ms | 2.0 | 2 | 8,852 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 560.0 | 144.6 | 30 ms | 6.9 ms | 4.0 | 4 | 16,800 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 993.0 | 131.2 | 30 ms | 7.6 ms | 7.8 | 8 | 29,784 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,521.9 | 96.9 | 44 ms | 10.3 ms | 16.0 | 0 | 45,648 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,190.9 | 70.2 | 79 ms | 14.2 ms | 32.0 | 0 | 65,728 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 138.4 | 139.7 | 16 ms | 7.2 ms | 1.0 | 1 | 4,151 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 271.3 | 137.9 | 24 ms | 7.3 ms | 2.0 | 2 | 8,141 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 512.5 | 133.2 | 28 ms | 7.5 ms | 3.9 | 4 | 15,375 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 881.1 | 111.1 | 35 ms | 9.0 ms | 8.0 | 0 | 26,432 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,241.1 | 78.3 | 58 ms | 12.8 ms | 16.0 | 0 | 37,233 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,659.8 | 52.4 | 99 ms | 19.1 ms | 32.0 | 0 | 49,793 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 27K | 1 | Valid | 124.7 | 126.0 | 17 ms | 7.9 ms | 1.0 | 1 | 3,741 | 30.0 s |
| Sustained decode | 27K | 2 | Valid | 244.3 | 125.0 | 25 ms | 8.0 ms | 2.0 | 2 | 7,328 | 30.0 s |
| Sustained decode | 27K | 4 | Valid | 461.5 | 116.5 | 33 ms | 8.6 ms | 4.0 | 0 | 13,844 | 30.0 s |
| Sustained decode | 27K | 8 | Valid | 757.4 | 95.2 | 37 ms | 10.5 ms | 8.0 | 0 | 22,720 | 30.0 s |
| Sustained decode | 27K | 16 | Valid | 969.2 | 60.9 | 55 ms | 16.4 ms | 16.0 | 0 | 29,072 | 30.0 s |
| Sustained decode | 27K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 27K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 27K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.