Speed test matrix
Step-3.7-Flash-FP8 on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9269+g834400705 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- step
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-stepfun-sglang/bin/sglang" serve --model-path stepfun-ai/Step-3.7-Flash-FP8 --served-model-name step --trust-remote-code --tp 2 --reasoning-parser step3p5 --tool-call-parser step3p5 --context-length 262144 --mem-fraction-static 0.847942 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --ep 2
Prefill
prompt tok/s by context length8K context23.1K354 ms TTFT
16K context24.8K653 ms TTFT
32K context23.4K1,377 ms TTFT
64K context20.7K3,115 ms TTFT
128K context16.4K7,825 ms TTFT
251K context11.8K21,322 ms TTFT
256K context11.7K22,030 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 124.3100 ms TTFT | 213.2142 ms TTFT | 367.9181 ms TTFT | 594.6237 ms TTFT | 889.0415 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 123.3119 ms TTFT | 100.4174 ms TTFT | 298.8236 ms TTFT | 543.5348 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 122.1163 ms TTFT | 146.2236 ms TTFT | 318.2372 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 119.2261 ms TTFT | 172.5430 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 113.8465 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 104.8991 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 23.1K | — | 354 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.8K | — | 653 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 23.4K | — | 1,377 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.7K | — | 3,115 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.4K | — | 7,825 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.8K | — | 21,322 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.7K | — | 22,030 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 124.3 | 125.0 | 100 ms | 8.0 ms | 1.0 | 1 | 3,729 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 213.2 | 103.7 | 142 ms | 9.6 ms | 2.0 | 0 | 6,396 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 367.9 | 90.7 | 181 ms | 11.0 ms | 4.0 | 0 | 11,036 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 594.6 | 73.3 | 237 ms | 13.7 ms | 8.9 | 0 | 17,824 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 889.0 | 55.4 | 415 ms | 18.0 ms | 21.6 | 0 | 26,672 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 123.3 | 124.3 | 119 ms | 8.0 ms | 1.0 | 1 | 3,700 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 100.4 | 50.2 | 174 ms | 19.9 ms | 18.0 | 0 | 3,010 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 298.8 | 72.3 | 236 ms | 13.8 ms | 5.8 | 0 | 8,964 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 543.5 | 67.3 | 348 ms | 14.9 ms | 10.4 | 0 | 16,304 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 122.1 | 122.2 | 163 ms | 8.2 ms | 1.0 | 0 | 3,665 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 146.2 | 72.0 | 236 ms | 13.9 ms | 7.9 | 0 | 4,387 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 318.2 | 78.6 | 372 ms | 12.7 ms | 5.1 | 0 | 9,544 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 119.2 | 119.2 | 261 ms | 8.4 ms | 1.0 | 0 | 3,578 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 172.5 | 84.6 | 430 ms | 11.8 ms | 3.9 | 0 | 5,174 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 113.8 | 113.8 | 465 ms | 8.8 ms | 1.0 | 0 | 3,413 | 30.0 s |
| Sustained decode | 128K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 104.8 | 104.8 | 991 ms | 9.5 ms | 1.0 | 0 | 3,143 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.