Speed test matrix
Step-3.7-Flash-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
sglang · 0.0.0.dev1+gf92199b56 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev1+gf92199b56
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
f92199b569d2
Run
- Served model
- step3p7
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-stepfun-ai-sglang-f92199b569d2/bin/sglang" serve --model-path stepfun-ai/Step-3.7-Flash-FP8 --served-model-name step3p7 --trust-remote-code --tp 4 --reasoning-parser step3p5 --tool-call-parser step3p5 --context-length 262144 --mem-fraction-static 0.77 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --ep 4 --enable-multimodal --attention-backend fa4 --fp8-gemm-backend cutlass --moe-runner-backend triton
Prefill
prompt tok/s by context length8K context15K547 ms TTFT
16K context15.5K1,047 ms TTFT
32K context15.3K2,110 ms TTFT
64K context14.1K4,574 ms TTFT
128K context12.7K10,118 ms TTFT
251K context10.3K24,375 ms TTFT
256K context10.1K25,453 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 118.4498 ms TTFT | 175.6160 ms TTFT | 310.1316 ms TTFT | 512.5189 ms TTFT | 798.9525 ms TTFT | 1,310.0379 ms TTFT | Over capacity | Over capacity |
| 16K | 115.290 ms TTFT | 172.5138 ms TTFT | 306.1193 ms TTFT | 505.7264 ms TTFT | 798.5424 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 109.4116 ms TTFT | 165.5218 ms TTFT | 295.0383 ms TTFT | 490.8501 ms TTFT | 772.8809 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 99.2261 ms TTFT | 154.5237 ms TTFT | 276.8401 ms TTFT | 466.4831 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 84.5258 ms TTFT | 135.6432 ms TTFT | 246.8844 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 65.4501 ms TTFT | 109.8970 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15K | — | 547 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.5K | — | 1,047 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.3K | — | 2,110 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.1K | — | 4,574 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.7K | — | 10,118 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.3K | — | 24,375 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.1K | — | 25,453 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 118.4 | 118.5 | 498 ms | 8.4 ms | 1.0 | 0 | 3,553 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 175.6 | 87.7 | 160 ms | 11.4 ms | 2.0 | 0 | 5,268 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 310.1 | 77.7 | 316 ms | 12.9 ms | 4.0 | 0 | 9,304 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 512.5 | 64.3 | 189 ms | 15.6 ms | 8.0 | 0 | 15,374 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 798.9 | 50.3 | 525 ms | 19.9 ms | 16.0 | 0 | 23,952 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,310.0 | 41.5 | 379 ms | 24.1 ms | 32.0 | 0 | 39,296 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 115.2 | 115.2 | 90 ms | 8.7 ms | 1.0 | 0 | 3,455 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 172.5 | 86.3 | 138 ms | 11.6 ms | 2.0 | 0 | 5,176 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 306.1 | 76.5 | 193 ms | 13.1 ms | 4.0 | 0 | 9,180 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 505.7 | 63.4 | 264 ms | 15.8 ms | 8.0 | 0 | 15,168 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 798.5 | 50.3 | 424 ms | 19.9 ms | 16.0 | 0 | 23,952 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 109.4 | 109.4 | 116 ms | 9.1 ms | 1.0 | 0 | 3,283 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 165.5 | 82.8 | 218 ms | 12.1 ms | 2.0 | 0 | 4,964 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 295.0 | 73.8 | 383 ms | 13.6 ms | 4.0 | 0 | 8,848 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 490.8 | 61.4 | 501 ms | 16.3 ms | 8.0 | 0 | 14,720 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 772.8 | 48.6 | 809 ms | 20.6 ms | 16.0 | 0 | 23,185 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 99.2 | 99.2 | 261 ms | 10.1 ms | 1.0 | 0 | 2,975 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 154.5 | 77.2 | 237 ms | 12.9 ms | 2.0 | 0 | 4,634 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 276.8 | 69.2 | 401 ms | 14.4 ms | 4.0 | 0 | 8,304 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 466.4 | 58.4 | 831 ms | 17.1 ms | 8.0 | 0 | 13,984 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 84.5 | 84.5 | 258 ms | 11.8 ms | 1.0 | 0 | 2,534 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 135.6 | 68.0 | 432 ms | 14.7 ms | 2.0 | 0 | 4,066 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 246.8 | 62.1 | 844 ms | 16.1 ms | 4.0 | 0 | 7,405 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 65.4 | 65.5 | 501 ms | 15.3 ms | 1.0 | 0 | 1,962 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 109.8 | 55.2 | 970 ms | 18.1 ms | 2.0 | 0 | 3,294 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.