Speed test matrix

Qwen3.5-4B-Base on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 14:44 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-4B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.810900 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context39.3K207 ms TTFT

16K context43.1K374 ms TTFT

32K context40.7K788 ms TTFT

64K context35.4K1,810 ms TTFT

128K context27.9K4,586 ms TTFT

251K context19.5K12,894 ms TTFT

256K context19.4K13,194 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	304.986 ms TTFT	534.784 ms TTFT	963.294 ms TTFT	1,630.0146 ms TTFT	2,755.7242 ms TTFT	3,252.4682 ms TTFT	Not run	Not run
16K	284.5108 ms TTFT	500.3114 ms TTFT	941.0114 ms TTFT	1,578.2252 ms TTFT	2,352.5664 ms TTFT	2,424.11,282 ms TTFT	Not run	Over capacity
32K	292.4168 ms TTFT	348.3209 ms TTFT	926.7239 ms TTFT	1,440.3615 ms TTFT	1,788.91,595 ms TTFT	2,673.43,034 ms TTFT	Over capacity	Over capacity
64K	233.5290 ms TTFT	234.0500 ms TTFT	740.3633 ms TTFT	935.11,829 ms TTFT	1,090.43,612 ms TTFT	Over capacity	Over capacity	Over capacity
128K	219.1534 ms TTFT	195.2960 ms TTFT	604.91,459 ms TTFT	463.83,687 ms TTFT	617.07,189 ms TTFT	Over capacity	Over capacity	Over capacity
251K	174.2986 ms TTFT	83.01,879 ms TTFT	147.53,656 ms TTFT	264.17,814 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	39.3K	—	207 ms	—	—	1	—	—
Prefill	16K	1	Valid	43.1K	—	374 ms	—	—	1	—	—
Prefill	32K	1	Valid	40.7K	—	788 ms	—	—	1	—	—
Prefill	64K	1	Valid	35.4K	—	1,810 ms	—	—	1	—	—
Prefill	128K	1	Valid	27.9K	—	4,586 ms	—	—	1	—	—
Prefill	251K	1	Valid	19.5K	—	12,894 ms	—	—	1	—	—
Prefill	256K	1	Valid	19.4K	—	13,194 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	304.9	306.2	86 ms	3.3 ms	1.0	2	9,146	30.0 s
Sustained decode	8K	2	Valid	534.7	278.1	84 ms	3.6 ms	2.2	4	16,040	30.0 s
Sustained decode	8K	4	Valid	963.2	258.3	94 ms	3.9 ms	4.4	7	28,895	30.0 s
Sustained decode	8K	8	Valid	1,630.0	203.0	146 ms	4.9 ms	9.1	8	48,891	30.0 s
Sustained decode	8K	16	Valid	2,755.7	177.0	242 ms	5.7 ms	16.2	16	82,658	30.0 s
Sustained decode	8K	32	Valid	3,252.4	104.8	682 ms	9.5 ms	39.9	0	97,581	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	284.5	291.9	108 ms	3.4 ms	1.0	2	8,534	30.0 s
Sustained decode	16K	2	Valid	500.3	254.7	114 ms	3.9 ms	2.5	3	15,007	30.0 s
Sustained decode	16K	4	Valid	941.0	242.9	114 ms	4.1 ms	4.3	6	28,229	30.0 s
Sustained decode	16K	8	Valid	1,578.2	207.6	252 ms	4.8 ms	8.9	8	47,344	30.0 s
Sustained decode	16K	16	Valid	2,352.5	156.6	664 ms	6.4 ms	16.8	15	70,579	30.0 s
Sustained decode	16K	32	Valid	2,424.1	78.6	1,282 ms	12.7 ms	46.3	0	72,706	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	292.4	296.2	168 ms	3.4 ms	1.0	2	8,771	30.0 s
Sustained decode	32K	2	Valid	348.3	183.4	209 ms	5.7 ms	15.7	2	10,450	30.0 s
Sustained decode	32K	4	Valid	926.7	222.2	239 ms	4.5 ms	4.6	5	27,806	30.0 s
Sustained decode	32K	8	Valid	1,440.3	186.9	615 ms	5.4 ms	8.2	8	43,201	30.0 s
Sustained decode	32K	16	Valid	1,788.9	112.4	1,595 ms	8.9 ms	19.7	0	53,662	30.0 s
Sustained decode	32K	32	Valid	2,673.4	86.7	3,034 ms	11.5 ms	32.0	0	80,167	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	233.5	242.6	290 ms	4.1 ms	1.0	1	7,005	30.0 s
Sustained decode	64K	2	Valid	234.0	121.9	500 ms	8.2 ms	14.9	1	7,020	30.0 s
Sustained decode	64K	4	Valid	740.3	194.7	633 ms	5.1 ms	4.4	4	22,210	30.0 s
Sustained decode	64K	8	Valid	935.1	127.2	1,829 ms	7.9 ms	9.6	4	28,052	30.0 s
Sustained decode	64K	16	Valid	1,090.4	75.6	3,612 ms	13.2 ms	20.1	0	32,707	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	219.1	228.1	534 ms	4.4 ms	1.0	1	6,571	30.0 s
Sustained decode	128K	2	Valid	195.2	94.4	960 ms	10.6 ms	13.9	0	5,857	30.0 s
Sustained decode	128K	4	Valid	604.9	167.5	1,459 ms	6.0 ms	3.9	4	18,145	30.0 s
Sustained decode	128K	8	Valid	463.8	61.5	3,687 ms	16.3 ms	12.0	0	13,908	30.0 s
Sustained decode	128K	16	Valid	617.0	42.9	7,189 ms	23.3 ms	24.9	0	18,502	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	174.2	183.6	986 ms	5.4 ms	1.0	1	5,224	30.0 s
Sustained decode	251K	2	Valid	83.0	42.1	1,879 ms	23.8 ms	19.9	0	2,488	30.0 s
Sustained decode	251K	4	Valid	147.5	38.3	3,656 ms	26.1 ms	19.0	0	4,423	30.0 s
Sustained decode	251K	8	Valid	264.1	37.6	7,814 ms	26.6 ms	14.4	0	7,922	30.0 s
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.