Speed test matrix

Step-3.7-Flash-NVFP4 on H200 NVL 141GB ×2

sglang · 0.5.6.post3.dev9269+g834400705 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 1, 2026, 19:05 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9269+g834400705
Quantization: NVFP4
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 834400705f2d

Run

Served model: step
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-stepfun-sglang/bin/sglang" serve --model-path stepfun-ai/Step-3.7-Flash-NVFP4 --served-model-name step --trust-remote-code --tp 2 --reasoning-parser step3p5 --tool-call-parser step3p5 --context-length 262144 --mem-fraction-static 0.848762 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context16.9K484 ms TTFT

16K context17.3K938 ms TTFT

32K context16.8K1,924 ms TTFT

64K context15.3K4,215 ms TTFT

128K context12.7K10,103 ms TTFT

251K context9,77625,797 ms TTFT

256K context9,74126,406 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	137.2100 ms TTFT	231.1125 ms TTFT	410.0158 ms TTFT	675.4219 ms TTFT	1,037.4346 ms TTFT	1,432.7646 ms TTFT	Over capacity	Over capacity
16K	136.1112 ms TTFT	91.8167 ms TTFT	272.4213 ms TTFT	594.8348 ms TTFT	943.4650 ms TTFT	1,328.31,251 ms TTFT	Over capacity	Over capacity
32K	133.9155 ms TTFT	83.8236 ms TTFT	211.5374 ms TTFT	511.8716 ms TTFT	807.21,396 ms TTFT	Over capacity	Over capacity	Over capacity
64K	130.1258 ms TTFT	97.2417 ms TTFT	282.3764 ms TTFT	481.11,528 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	122.6466 ms TTFT	111.3818 ms TTFT	283.41,553 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	113.8921 ms TTFT	148.01,815 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	16.9K	—	484 ms	—	—	1	—	—
Prefill	16K	1	Valid	17.3K	—	938 ms	—	—	1	—	—
Prefill	32K	1	Valid	16.8K	—	1,924 ms	—	—	1	—	—
Prefill	64K	1	Valid	15.3K	—	4,215 ms	—	—	1	—	—
Prefill	128K	1	Valid	12.7K	—	10,103 ms	—	—	1	—	—
Prefill	251K	1	Valid	9,776.0	—	25,797 ms	—	—	1	—	—
Prefill	256K	1	Valid	9,741.0	—	26,406 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	137.2	137.8	100 ms	7.3 ms	1.0	1	4,117	30.0 s
Sustained decode	8K	2	Valid	231.1	115.7	125 ms	8.6 ms	2.0	0	6,930	30.0 s
Sustained decode	8K	4	Valid	410.0	102.6	158 ms	9.7 ms	4.0	0	12,292	30.0 s
Sustained decode	8K	8	Valid	675.4	83.2	219 ms	12.0 ms	8.0	0	20,256	30.0 s
Sustained decode	8K	16	Valid	1,037.4	64.2	346 ms	15.6 ms	19.5	0	31,120	30.0 s
Sustained decode	8K	32	Valid	1,432.7	45.0	646 ms	22.2 ms	48.0	0	42,944	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	136.1	136.8	112 ms	7.3 ms	1.0	1	4,085	30.0 s
Sustained decode	16K	2	Valid	91.8	45.8	167 ms	21.8 ms	34.0	0	2,753	30.0 s
Sustained decode	16K	4	Valid	272.4	65.2	213 ms	15.3 ms	18.8	0	8,172	30.0 s
Sustained decode	16K	8	Valid	594.8	73.9	348 ms	13.5 ms	10.5	0	17,832	30.0 s
Sustained decode	16K	16	Valid	943.4	58.8	650 ms	17.0 ms	21.3	0	28,288	30.0 s
Sustained decode	16K	32	Valid	1,328.3	41.9	1,251 ms	23.9 ms	48.0	0	39,840	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	133.9	134.8	155 ms	7.4 ms	1.0	1	4,017	30.0 s
Sustained decode	32K	2	Valid	83.8	41.7	236 ms	24.0 ms	37.7	0	2,512	30.0 s
Sustained decode	32K	4	Valid	211.5	51.6	374 ms	19.4 ms	28.4	0	6,348	30.0 s
Sustained decode	32K	8	Valid	511.8	63.7	716 ms	15.7 ms	12.3	0	15,344	30.0 s
Sustained decode	32K	16	Valid	807.2	51.2	1,396 ms	19.5 ms	23.7	0	24,208	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	130.1	131.3	258 ms	7.6 ms	1.0	1	3,901	30.0 s
Sustained decode	64K	2	Valid	97.2	48.7	417 ms	20.5 ms	18.0	0	2,916	30.0 s
Sustained decode	64K	4	Valid	282.3	68.1	764 ms	14.7 ms	8.6	0	8,460	30.0 s
Sustained decode	64K	8	Valid	481.1	60.6	1,528 ms	16.5 ms	11.0	0	14,424	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	122.6	125.0	466 ms	8.0 ms	1.0	1	3,679	30.0 s
Sustained decode	128K	2	Valid	111.3	55.7	818 ms	17.9 ms	9.7	0	3,341	30.0 s
Sustained decode	128K	4	Valid	283.4	71.5	1,553 ms	14.0 ms	5.6	0	8,504	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	113.8	113.8	921 ms	8.8 ms	1.0	0	3,412	30.0 s
Sustained decode	251K	2	Valid	148.0	73.3	1,815 ms	13.6 ms	4.3	0	4,436	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.