Speed test matrix

Qwen3.5-2B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 06:41 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-2B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861540 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion

Prefill

prompt tok/s by context length

8K context62.3K131 ms TTFT

16K context77.6K208 ms TTFT

32K context80.6K399 ms TTFT

64K context74.5K860 ms TTFT

128K context63.9K2,004 ms TTFT

251K context47.8K5,245 ms TTFT

256K context47.9K5,342 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	447.178 ms TTFT	875.469 ms TTFT	1,673.978 ms TTFT	3,033.7129 ms TTFT	5,212.3298 ms TTFT	8,260.8623 ms TTFT	10.1K1,284 ms TTFT	12.3K2,637 ms TTFT
16K	438.997 ms TTFT	845.5105 ms TTFT	1,609.7144 ms TTFT	2,863.2169 ms TTFT	4,890.3127 ms TTFT	7,138.81,076 ms TTFT	9,709.81,663 ms TTFT	8,259.75,173 ms TTFT
32K	427.6149 ms TTFT	518.8243 ms TTFT	1,501.1220 ms TTFT	2,534.8250 ms TTFT	4,054.6341 ms TTFT	4,188.52,503 ms TTFT	4,428.76,499 ms TTFT	6,680.312,727 ms TTFT
64K	404.6270 ms TTFT	128.9537 ms TTFT	1,330.9480 ms TTFT	2,140.2261 ms TTFT	2,476.32,260 ms TTFT	2,949.78,385 ms TTFT	4,400.914,365 ms TTFT	Over capacity
128K	364.8489 ms TTFT	405.7993 ms TTFT	1,075.9490 ms TTFT	1,472.02,306 ms TTFT	1,855.67,329 ms TTFT	1,454.914,654 ms TTFT	Over capacity	Over capacity
251K	313.11,001 ms TTFT	103.31,951 ms TTFT	727.02,837 ms TTFT	959.37,369 ms TTFT	587.614,595 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	62.3K	—	131 ms	—	—	1	—	—
Prefill	16K	1	Valid	77.6K	—	208 ms	—	—	1	—	—
Prefill	32K	1	Valid	80.6K	—	399 ms	—	—	1	—	—
Prefill	64K	1	Valid	74.5K	—	860 ms	—	—	1	—	—
Prefill	128K	1	Valid	63.9K	—	2,004 ms	—	—	1	—	—
Prefill	251K	1	Valid	47.8K	—	5,245 ms	—	—	1	—	—
Prefill	256K	1	Valid	47.9K	—	5,342 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	447.1	450.8	78 ms	2.2 ms	1.0	3	13,413	30.0 s
Sustained decode	8K	2	Valid	875.4	440.3	69 ms	2.3 ms	2.0	6	26,261	30.0 s
Sustained decode	8K	4	Valid	1,673.9	421.0	78 ms	2.4 ms	4.0	12	50,215	30.0 s
Sustained decode	8K	8	Valid	3,033.7	384.3	129 ms	2.6 ms	8.0	24	91,012	30.0 s
Sustained decode	8K	16	Valid	5,212.3	340.3	298 ms	2.9 ms	17.3	32	156,360	30.0 s
Sustained decode	8K	32	Valid	8,260.8	262.2	623 ms	3.8 ms	31.7	64	247,825	30.0 s
Sustained decode	8K	64	Valid	10.1K	174.5	1,284 ms	5.8 ms	79.3	64	304,104	30.0 s
Sustained decode	8K	128	Valid	12.3K	97.5	2,637 ms	10.3 ms	174.7	0	368,864	30.0 s
Sustained decode	16K	1	Valid	438.9	443.1	97 ms	2.3 ms	1.0	3	13,168	30.0 s
Sustained decode	16K	2	Valid	845.5	432.3	105 ms	2.3 ms	6.2	6	25,365	30.0 s
Sustained decode	16K	4	Valid	1,609.7	408.1	144 ms	2.5 ms	4.0	12	48,289	30.0 s
Sustained decode	16K	8	Valid	2,863.2	361.9	169 ms	2.8 ms	8.1	16	85,896	30.0 s
Sustained decode	16K	16	Valid	4,890.3	309.8	127 ms	3.2 ms	15.8	32	146,707	30.0 s
Sustained decode	16K	32	Valid	7,138.8	225.0	1,076 ms	4.4 ms	32.5	32	214,169	30.0 s
Sustained decode	16K	64	Valid	9,709.8	160.4	1,663 ms	6.2 ms	62.8	64	291,292	30.0 s
Sustained decode	16K	128	Valid	8,259.7	69.1	5,173 ms	14.5 ms	190.6	0	247,808	30.0 s
Sustained decode	32K	1	Valid	427.6	434.1	149 ms	2.3 ms	1.0	3	12,826	30.0 s
Sustained decode	32K	2	Valid	518.8	291.9	243 ms	4.2 ms	59.7	2	15,563	30.0 s
Sustained decode	32K	4	Valid	1,501.1	384.6	220 ms	2.6 ms	4.0	12	45,033	30.0 s
Sustained decode	32K	8	Valid	2,534.8	327.1	250 ms	3.1 ms	8.4	16	76,041	30.0 s
Sustained decode	32K	16	Valid	4,054.6	256.8	341 ms	3.9 ms	15.7	32	121,638	30.0 s
Sustained decode	32K	32	Valid	4,188.5	152.7	2,503 ms	6.6 ms	41.3	32	125,648	30.0 s
Sustained decode	32K	64	Valid	4,428.7	76.0	6,499 ms	13.2 ms	95.9	0	132,864	30.0 s
Sustained decode	32K	128	Valid	6,680.3	58.9	12,727 ms	17.0 ms	154.0	0	200,544	30.0 s
Sustained decode	64K	1	Valid	404.6	415.6	270 ms	2.4 ms	1.0	3	12,137	30.0 s
Sustained decode	64K	2	Valid	128.9	63.9	537 ms	15.6 ms	118.4	0	3,868	30.0 s
Sustained decode	64K	4	Valid	1,330.9	340.6	480 ms	2.9 ms	3.9	8	39,925	30.0 s
Sustained decode	64K	8	Valid	2,140.2	273.5	261 ms	3.7 ms	7.9	16	64,202	30.0 s
Sustained decode	64K	16	Valid	2,476.3	172.6	2,260 ms	5.8 ms	19.1	16	74,287	30.0 s
Sustained decode	64K	32	Valid	2,949.7	99.7	8,385 ms	10.0 ms	40.8	0	88,496	30.0 s
Sustained decode	64K	64	Valid	4,400.9	84.8	14,365 ms	11.8 ms	64.0	0	132,032	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	364.8	383.7	489 ms	2.6 ms	1.0	3	10,944	30.0 s
Sustained decode	128K	2	Valid	405.7	246.9	993 ms	4.8 ms	24.3	2	12,157	30.0 s
Sustained decode	128K	4	Valid	1,075.9	280.0	490 ms	3.6 ms	3.9	8	32,276	30.0 s
Sustained decode	128K	8	Valid	1,472.0	201.7	2,306 ms	5.0 ms	8.9	8	44,160	30.0 s
Sustained decode	128K	16	Valid	1,855.6	118.3	7,329 ms	8.4 ms	16.9	6	55,156	29.7 s
Sustained decode	128K	32	Valid	1,454.9	58.2	14,654 ms	17.2 ms	38.0	0	43,648	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	313.1	335.6	1,001 ms	3.0 ms	0.9	2	9,392	30.0 s
Sustained decode	251K	2	Valid	103.3	51.5	1,951 ms	19.4 ms	32.5	0	3,100	30.0 s
Sustained decode	251K	4	Valid	727.0	198.3	2,837 ms	5.0 ms	4.3	4	21,809	30.0 s
Sustained decode	251K	8	Valid	959.3	126.9	7,369 ms	7.9 ms	7.9	8	29,197	30.4 s
Sustained decode	251K	16	Valid	587.6	47.0	14,595 ms	21.3 ms	24.0	0	17,616	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.