Speed test matrix
Step-3.7-Flash-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×2
sglang · 0.0.0.dev1+gf92199b56 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev1+gf92199b56
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
f92199b569d2
Run
- Served model
- step3p7
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-stepfun-ai-sglang-f92199b569d2/bin/sglang" serve --model-path stepfun-ai/Step-3.7-Flash-NVFP4 --served-model-name step3p7 --trust-remote-code --tp 2 --reasoning-parser step3p5 --tool-call-parser step3p5 --context-length 262144 --mem-fraction-static 0.76 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --ep 2 --kv-cache-dtype fp8_e4m3 --enable-multimodal --attention-backend flashinfer --fp4-gemm-backend flashinfer_cutlass --moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context15.9K516 ms TTFT
16K context15.3K1,059 ms TTFT
32K context14.6K2,207 ms TTFT
64K context12.9K4,987 ms TTFT
128K context10.4K12,343 ms TTFT
251K context7,56533,342 ms TTFT
256K context7,39434,791 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | 260.1179 ms TTFT | 427.5202 ms TTFT | 722.6285 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 94.679 ms TTFT | 149.0145 ms TTFT | 261.5172 ms TTFT | 430.4260 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 93.3102 ms TTFT | 148.2140 ms TTFT | 255.5238 ms TTFT | 417.0400 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 91.0158 ms TTFT | Looped | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | Looped | 140.1416 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.9K | — | 516 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.3K | — | 1,059 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.6K | — | 2,207 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.9K | — | 4,987 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.4K | — | 12,343 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 7,565.0 | — | 33,342 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,394.0 | — | 34,791 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 260.1 | 65.2 | 179 ms | 15.3 ms | 4.0 | 0 | 7,800 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 427.5 | 53.6 | 202 ms | 18.7 ms | 8.0 | 0 | 12,824 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 722.6 | 45.6 | 285 ms | 21.9 ms | 16.0 | 0 | 21,664 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 94.6 | 94.7 | 79 ms | 10.6 ms | 1.0 | 0 | 2,838 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 149.0 | 74.5 | 145 ms | 13.4 ms | 2.0 | 0 | 4,470 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 261.5 | 65.5 | 172 ms | 15.3 ms | 4.0 | 0 | 7,844 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 430.4 | 53.9 | 260 ms | 18.6 ms | 8.0 | 0 | 12,904 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 93.3 | 93.3 | 102 ms | 10.7 ms | 1.0 | 0 | 2,799 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 148.2 | 74.1 | 140 ms | 13.5 ms | 2.0 | 0 | 4,446 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 255.5 | 63.9 | 238 ms | 15.6 ms | 4.0 | 0 | 7,664 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 417.0 | 52.3 | 400 ms | 19.1 ms | 8.0 | 0 | 12,504 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 91.0 | 91.1 | 158 ms | 11.0 ms | 1.0 | 0 | 2,730 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Valid | 140.1 | 70.2 | 416 ms | 14.2 ms | 2.0 | 0 | 4,202 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.