Speed test matrix

Step-3.7-Flash on H200 SXM 141GB ×4

vllm · 0.29.0 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 16, 2026, 17:06 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: stepfun-ai/Step-3.7-Flash
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve stepfun-ai/Step-3.7-Flash --served-model-name stepfun-ai/Step-3.7-Flash --trust-remote-code --tensor-parallel-size 4 --reasoning-parser step3p5 --enable-auto-tool-choice --tool-call-parser step3p5 --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-expert-parallel --disable-cascade-attn

Prefill

prompt tok/s by context length

8K context34.5K237 ms TTFT

16K context35.5K457 ms TTFT

32K context34.6K933 ms TTFT

64K context31.5K2,043 ms TTFT

128K context26.7K4,822 ms TTFT

251K context20.7K12,207 ms TTFT

256K context20.2K12,745 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	156.544 ms TTFT	259.951 ms TTFT	450.478 ms TTFT	737.3113 ms TTFT	1,193.1180 ms TTFT	1,891.8322 ms TTFT	3,062.1610 ms TTFT	4,829.41,160 ms TTFT
16K	155.662 ms TTFT	258.072 ms TTFT	448.7108 ms TTFT	730.8171 ms TTFT	1,170.5282 ms TTFT	1,843.1518 ms TTFT	2,935.2992 ms TTFT	Over capacity
32K	154.0104 ms TTFT	255.6117 ms TTFT	443.0186 ms TTFT	717.0305 ms TTFT	1,132.8534 ms TTFT	1,765.31,029 ms TTFT	Over capacity	Over capacity
64K	150.8191 ms TTFT	250.0207 ms TTFT	434.1375 ms TTFT	687.5605 ms TTFT	1,079.01,078 ms TTFT	Over capacity	Over capacity	Over capacity
128K	144.7381 ms TTFT	240.5402 ms TTFT	407.0772 ms TTFT	640.71,293 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	133.8787 ms TTFT	227.91,141 ms TTFT	375.31,730 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	34.5K	—	237 ms	—	—	1	—	—
Prefill	16K	1	Valid	35.5K	—	457 ms	—	—	1	—	—
Prefill	32K	1	Valid	34.6K	—	933 ms	—	—	1	—	—
Prefill	64K	1	Valid	31.5K	—	2,043 ms	—	—	1	—	—
Prefill	128K	1	Valid	26.7K	—	4,822 ms	—	—	1	—	—
Prefill	251K	1	Valid	20.7K	—	12,207 ms	—	—	1	—	—
Prefill	256K	1	Valid	20.2K	—	12,745 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	156.5	156.8	44 ms	6.4 ms	1.0	1	4,696	30.0 s
Sustained decode	8K	2	Valid	259.9	130.2	51 ms	7.7 ms	2.0	2	7,794	30.0 s
Sustained decode	8K	4	Valid	450.4	112.8	78 ms	8.9 ms	4.0	0	13,512	30.0 s
Sustained decode	8K	8	Valid	737.3	92.3	113 ms	10.8 ms	8.0	0	22,120	30.0 s
Sustained decode	8K	16	Valid	1,193.1	75.0	180 ms	13.3 ms	16.0	0	35,792	30.0 s
Sustained decode	8K	32	Valid	1,891.8	59.6	322 ms	16.8 ms	32.0	0	56,736	30.0 s
Sustained decode	8K	64	Valid	3,062.1	48.3	610 ms	20.7 ms	64.0	0	91,840	30.0 s
Sustained decode	8K	128	Valid	4,829.4	38.1	1,160 ms	26.2 ms	128.0	0	144,896	30.0 s
Sustained decode	16K	1	Valid	155.6	156.1	62 ms	6.4 ms	1.0	1	4,669	30.0 s
Sustained decode	16K	2	Valid	258.0	129.4	72 ms	7.7 ms	2.0	2	7,739	30.0 s
Sustained decode	16K	4	Valid	448.7	112.2	108 ms	8.9 ms	4.0	0	13,460	30.0 s
Sustained decode	16K	8	Valid	730.8	91.4	171 ms	10.9 ms	8.0	0	21,920	30.0 s
Sustained decode	16K	16	Valid	1,170.5	73.4	282 ms	13.6 ms	16.0	0	35,104	30.0 s
Sustained decode	16K	32	Valid	1,843.1	58.0	518 ms	17.2 ms	32.0	0	55,296	30.0 s
Sustained decode	16K	64	Valid	2,935.2	46.3	992 ms	21.6 ms	64.0	0	88,064	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	154.0	154.7	104 ms	6.5 ms	1.0	1	4,619	30.0 s
Sustained decode	32K	2	Valid	255.6	128.2	117 ms	7.8 ms	2.0	2	7,668	30.0 s
Sustained decode	32K	4	Valid	443.0	110.7	186 ms	9.0 ms	4.0	0	13,288	30.0 s
Sustained decode	32K	8	Valid	717.0	89.6	305 ms	11.2 ms	8.0	0	21,504	30.0 s
Sustained decode	32K	16	Valid	1,132.8	71.0	534 ms	14.1 ms	16.0	0	33,984	30.0 s
Sustained decode	32K	32	Valid	1,765.3	55.5	1,029 ms	18.0 ms	32.0	0	52,960	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	150.8	152.1	191 ms	6.6 ms	1.0	1	4,522	30.0 s
Sustained decode	64K	2	Valid	250.0	127.1	207 ms	7.9 ms	1.9	2	7,500	30.0 s
Sustained decode	64K	4	Valid	434.1	108.5	375 ms	9.2 ms	4.0	0	13,020	30.0 s
Sustained decode	64K	8	Valid	687.5	86.1	605 ms	11.6 ms	8.0	0	20,626	30.0 s
Sustained decode	64K	16	Valid	1,079.0	67.6	1,078 ms	14.8 ms	16.0	0	32,368	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	144.7	147.3	381 ms	6.8 ms	1.0	1	4,341	30.0 s
Sustained decode	128K	2	Valid	240.5	121.6	402 ms	8.2 ms	2.0	2	7,215	30.0 s
Sustained decode	128K	4	Valid	407.0	101.9	772 ms	9.8 ms	4.0	0	12,208	30.0 s
Sustained decode	128K	8	Valid	640.7	80.3	1,293 ms	12.5 ms	8.0	0	19,216	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	133.8	139.3	787 ms	7.2 ms	1.0	1	4,013	30.0 s
Sustained decode	251K	2	Valid	227.9	114.3	1,141 ms	8.7 ms	2.0	0	6,836	30.0 s
Sustained decode	251K	4	Valid	375.3	94.1	1,730 ms	10.6 ms	4.0	0	11,256	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.