Speed test matrix

Qwen3.5-4B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 15:20 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.863403 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion

Prefill

prompt tok/s by context length

8K context41.1K198 ms TTFT

16K context44.5K362 ms TTFT

32K context42.3K759 ms TTFT

64K context36.8K1,741 ms TTFT

128K context29.7K4,312 ms TTFT

251K context21.1K11,878 ms TTFT

256K context21.1K12,121 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	260.181 ms TTFT	459.993 ms TTFT	948.2100 ms TTFT	1,553.7206 ms TTFT	2,820.1344 ms TTFT	3,874.9665 ms TTFT	5,986.01,270 ms TTFT	6,277.12,449 ms TTFT
16K	254.9120 ms TTFT	100.5132 ms TTFT	728.7212 ms TTFT	1,471.5291 ms TTFT	2,328.7574 ms TTFT	2,480.61,275 ms TTFT	3,644.12,518 ms TTFT	Over capacity
32K	245.8150 ms TTFT	113.7253 ms TTFT	813.9386 ms TTFT	1,340.8581 ms TTFT	1,582.61,672 ms TTFT	2,516.13,077 ms TTFT	Over capacity	Over capacity
64K	231.2295 ms TTFT	224.6563 ms TTFT	707.2772 ms TTFT	855.31,994 ms TTFT	1,385.03,846 ms TTFT	Over capacity	Over capacity	Over capacity
128K	205.0503 ms TTFT	273.0966 ms TTFT	417.41,979 ms TTFT	715.63,723 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	166.3995 ms TTFT	224.01,929 ms TTFT	356.23,720 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	41.1K	—	198 ms	—	—	1	—	—
Prefill	16K	1	Valid	44.5K	—	362 ms	—	—	1	—	—
Prefill	32K	1	Valid	42.3K	—	759 ms	—	—	1	—	—
Prefill	64K	1	Valid	36.8K	—	1,741 ms	—	—	1	—	—
Prefill	128K	1	Valid	29.7K	—	4,312 ms	—	—	1	—	—
Prefill	251K	1	Valid	21.1K	—	11,878 ms	—	—	1	—	—
Prefill	256K	1	Valid	21.1K	—	12,121 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	260.1	261.7	81 ms	3.8 ms	1.0	2	7,804	30.0 s
Sustained decode	8K	2	Valid	459.9	227.3	93 ms	4.5 ms	2.3	2	13,795	30.0 s
Sustained decode	8K	4	Valid	948.2	238.8	100 ms	4.2 ms	4.0	8	28,445	30.0 s
Sustained decode	8K	8	Valid	1,553.7	199.8	206 ms	5.0 ms	10.0	8	46,612	30.0 s
Sustained decode	8K	16	Valid	2,820.1	177.8	344 ms	5.6 ms	16.8	16	84,591	30.0 s
Sustained decode	8K	32	Valid	3,874.9	119.8	665 ms	8.3 ms	38.9	0	116,256	30.0 s
Sustained decode	8K	64	Valid	5,986.0	94.9	1,270 ms	10.5 ms	64.0	0	179,586	30.0 s
Sustained decode	8K	128	Valid	6,277.1	46.7	2,449 ms	21.4 ms	157.6	0	188,288	30.0 s
Sustained decode	16K	1	Valid	254.9	257.2	120 ms	3.9 ms	1.0	2	7,646	30.0 s
Sustained decode	16K	2	Valid	100.5	50.6	132 ms	19.8 ms	130.0	0	3,014	30.0 s
Sustained decode	16K	4	Valid	728.7	189.8	212 ms	5.5 ms	34.9	4	21,860	30.0 s
Sustained decode	16K	8	Valid	1,471.5	186.7	291 ms	5.4 ms	9.1	8	44,142	30.0 s
Sustained decode	16K	16	Valid	2,328.7	150.7	574 ms	6.7 ms	17.9	16	69,860	30.0 s
Sustained decode	16K	32	Valid	2,480.6	78.8	1,275 ms	12.7 ms	48.0	0	74,400	30.0 s
Sustained decode	16K	64	Valid	3,644.1	57.6	2,518 ms	17.4 ms	85.0	0	109,312	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	245.8	248.5	150 ms	4.0 ms	1.0	2	7,372	30.0 s
Sustained decode	32K	2	Valid	113.7	57.2	253 ms	17.5 ms	65.7	0	3,410	30.0 s
Sustained decode	32K	4	Valid	813.9	204.7	386 ms	4.9 ms	4.2	4	24,418	30.0 s
Sustained decode	32K	8	Valid	1,340.8	168.0	581 ms	6.0 ms	8.2	8	40,221	30.0 s
Sustained decode	32K	16	Valid	1,582.6	99.6	1,672 ms	10.0 ms	21.2	0	47,472	30.0 s
Sustained decode	32K	32	Valid	2,516.1	78.0	3,077 ms	12.8 ms	32.9	0	75,488	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	231.2	233.5	295 ms	4.3 ms	1.0	1	6,935	30.0 s
Sustained decode	64K	2	Valid	224.6	105.5	563 ms	9.5 ms	21.8	0	6,738	30.0 s
Sustained decode	64K	4	Valid	707.2	181.0	772 ms	5.5 ms	4.0	4	21,199	30.0 s
Sustained decode	64K	8	Valid	855.3	106.3	1,994 ms	9.4 ms	9.9	0	25,656	30.0 s
Sustained decode	64K	16	Valid	1,385.0	91.2	3,846 ms	11.0 ms	16.0	0	41,552	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	205.0	208.4	503 ms	4.8 ms	1.0	1	6,148	30.0 s
Sustained decode	128K	2	Valid	273.0	152.8	966 ms	6.7 ms	6.6	2	8,189	30.0 s
Sustained decode	128K	4	Valid	417.4	106.5	1,979 ms	9.4 ms	5.8	0	12,520	30.0 s
Sustained decode	128K	8	Valid	715.6	94.3	3,723 ms	10.6 ms	8.0	0	21,464	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	166.3	172.1	995 ms	5.8 ms	1.0	1	4,988	30.0 s
Sustained decode	251K	2	Valid	224.0	105.6	1,929 ms	9.5 ms	3.8	0	6,718	30.0 s
Sustained decode	251K	4	Valid	356.2	93.7	3,720 ms	10.7 ms	4.0	0	10,684	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.