Speed test matrix

Qwen3.5-2B-Base on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 06:00 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-2B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.782404 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context61.2K133 ms TTFT

16K context73.6K219 ms TTFT

32K context77.5K414 ms TTFT

64K context68.7K933 ms TTFT

128K context58.1K2,203 ms TTFT

251K context43.2K5,802 ms TTFT

256K context43.3K5,912 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	362.579 ms TTFT	618.785 ms TTFT	1,196.989 ms TTFT	2,210.088 ms TTFT	3,602.493 ms TTFT	5,110.7370 ms TTFT	Not run	Not run
16K	330.7100 ms TTFT	569.5102 ms TTFT	1,326.599 ms TTFT	2,142.5102 ms TTFT	3,341.1126 ms TTFT	4,664.01,229 ms TTFT	Not run	Not run
32K	336.6154 ms TTFT	447.1160 ms TTFT	1,135.0165 ms TTFT	1,910.3161 ms TTFT	2,585.9914 ms TTFT	3,220.03,111 ms TTFT	Not run	Not run
64K	331.6292 ms TTFT	533.3518 ms TTFT	1,049.0306 ms TTFT	1,598.61,205 ms TTFT	2,213.03,632 ms TTFT	1,955.47,377 ms TTFT	Not run	Over capacity
128K	289.2536 ms TTFT	262.2971 ms TTFT	815.11,440 ms TTFT	1,240.22,120 ms TTFT	1,228.57,432 ms TTFT	1,766.615,005 ms TTFT	Over capacity	Over capacity
251K	308.9987 ms TTFT	125.72,074 ms TTFT	521.63,806 ms TTFT	962.17,333 ms TTFT	890.914,637 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	61.2K	—	133 ms	—	—	1	—	—
Prefill	16K	1	Valid	73.6K	—	219 ms	—	—	1	—	—
Prefill	32K	1	Valid	77.5K	—	414 ms	—	—	1	—	—
Prefill	64K	1	Valid	68.7K	—	933 ms	—	—	1	—	—
Prefill	128K	1	Valid	58.1K	—	2,203 ms	—	—	1	—	—
Prefill	251K	1	Valid	43.2K	—	5,802 ms	—	—	1	—	—
Prefill	256K	1	Valid	43.3K	—	5,912 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	362.5	400.1	79 ms	2.5 ms	1.0	3	10,875	30.0 s
Sustained decode	8K	2	Valid	618.7	331.4	85 ms	3.0 ms	2.0	4	18,561	30.0 s
Sustained decode	8K	4	Valid	1,196.9	312.6	89 ms	3.2 ms	4.0	8	35,909	30.0 s
Sustained decode	8K	8	Valid	2,210.0	277.6	88 ms	3.6 ms	8.0	16	66,301	30.0 s
Sustained decode	8K	16	Valid	3,602.4	235.6	93 ms	4.2 ms	17.6	19	107,934	30.0 s
Sustained decode	8K	32	Valid	5,110.7	173.9	370 ms	5.8 ms	33.8	32	153,314	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	330.7	347.1	100 ms	2.9 ms	1.0	2	9,921	30.0 s
Sustained decode	16K	2	Valid	569.5	330.5	102 ms	3.0 ms	13.0	4	17,084	30.0 s
Sustained decode	16K	4	Valid	1,326.5	326.9	99 ms	3.1 ms	4.2	9	39,795	30.0 s
Sustained decode	16K	8	Valid	2,142.5	281.3	102 ms	3.6 ms	8.3	15	64,277	30.0 s
Sustained decode	16K	16	Valid	3,341.1	212.9	126 ms	4.7 ms	17.5	17	100,240	30.0 s
Sustained decode	16K	32	Valid	4,664.0	164.6	1,229 ms	6.1 ms	33.6	31	139,924	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	336.6	359.3	154 ms	2.8 ms	1.0	2	10,097	30.0 s
Sustained decode	32K	2	Valid	447.1	346.7	160 ms	2.9 ms	18.8	3	13,413	30.0 s
Sustained decode	32K	4	Valid	1,135.0	297.3	165 ms	3.4 ms	4.1	8	34,047	30.0 s
Sustained decode	32K	8	Valid	1,910.3	249.7	161 ms	4.0 ms	8.3	12	57,311	30.0 s
Sustained decode	32K	16	Valid	2,585.9	180.7	914 ms	5.5 ms	17.3	16	77,577	30.0 s
Sustained decode	32K	32	Valid	3,220.0	105.6	3,111 ms	9.5 ms	41.7	6	96,597	30.0 s
Sustained decode	32K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	331.6	342.2	292 ms	2.9 ms	1.0	2	9,949	30.0 s
Sustained decode	64K	2	Valid	533.3	309.7	518 ms	3.2 ms	5.3	3	15,999	30.0 s
Sustained decode	64K	4	Valid	1,049.0	271.5	306 ms	3.7 ms	4.2	8	31,467	30.0 s
Sustained decode	64K	8	Valid	1,598.6	211.3	1,205 ms	4.7 ms	9.1	8	47,958	30.0 s
Sustained decode	64K	16	Valid	2,213.0	159.1	3,632 ms	6.3 ms	16.4	14	66,390	30.0 s
Sustained decode	64K	32	Valid	1,955.4	73.1	7,377 ms	13.7 ms	44.1	0	58,647	30.0 s
Sustained decode	64K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	289.2	299.1	536 ms	3.3 ms	0.9	2	8,678	30.0 s
Sustained decode	128K	2	Valid	262.2	131.4	971 ms	7.6 ms	25.2	1	7,864	30.0 s
Sustained decode	128K	4	Valid	815.1	216.6	1,440 ms	4.6 ms	4.2	5	24,308	29.8 s
Sustained decode	128K	8	Valid	1,240.2	169.5	2,120 ms	5.9 ms	7.9	8	37,206	30.0 s
Sustained decode	128K	16	Valid	1,228.5	85.1	7,432 ms	11.8 ms	21.4	0	36,846	30.0 s
Sustained decode	128K	32	Valid	1,766.6	65.2	15,005 ms	15.3 ms	34.6	1	52,980	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	308.9	289.8	987 ms	3.5 ms	0.9	2	9,268	30.0 s
Sustained decode	251K	2	Valid	125.7	61.9	2,074 ms	17.0 ms	30.7	0	3,770	30.0 s
Sustained decode	251K	4	Valid	521.6	115.7	3,806 ms	8.6 ms	7.1	1	15,645	30.0 s
Sustained decode	251K	8	Valid	962.1	130.9	7,333 ms	7.6 ms	8.1	6	27,976	29.1 s
Sustained decode	251K	16	Valid	890.9	62.6	14,637 ms	16.0 ms	21.1	2	26,720	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.