Speed test matrix

Qwen3.6-27B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 22:09 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.874606 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context9,618847 ms TTFT

16K context9,5311,692 ms TTFT

32K context9,1903,493 ms TTFT

64K context8,2687,748 ms TTFT

128K context7,00818,260 ms TTFT

251K context5,35746,821 ms TTFT

256K context5,32448,043 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	131.798 ms TTFT	252.0147 ms TTFT	434.41,639 ms TTFT	838.9203 ms TTFT	1,409.0351 ms TTFT	1,692.5Requests queued11,917 ms TTFT	Over capacity	Over capacity
16K	131.7116 ms TTFT	188.5215 ms TTFT	452.6243 ms TTFT	815.0348 ms TTFT	1,333.8663 ms TTFT	Over capacity	Over capacity	Over capacity
32K	135.7172 ms TTFT	211.2284 ms TTFT	441.5417 ms TTFT	755.0809 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
64K	119.4301 ms TTFT	220.6550 ms TTFT	408.31,021 ms TTFT	661.51,844 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	116.8538 ms TTFT	161.31,059 ms TTFT	331.61,909 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	103.81,021 ms TTFT	161.51,972 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	9,618.0	—	847 ms	—	—	1	—	—
Prefill	16K	1	Valid	9,531.0	—	1,692 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,190.0	—	3,493 ms	—	—	1	—	—
Prefill	64K	1	Valid	8,268.0	—	7,748 ms	—	—	1	—	—
Prefill	128K	1	Valid	7,008.0	—	18,260 ms	—	—	1	—	—
Prefill	251K	1	Valid	5,357.0	—	46,821 ms	—	—	1	—	—
Prefill	256K	1	Valid	5,324.0	—	48,043 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	131.7	143.4	98 ms	7.0 ms	1.0	1	3,950	30.0 s
Sustained decode	8K	2	Valid	252.0	136.3	147 ms	7.4 ms	2.0	2	7,559	30.0 s
Sustained decode	8K	4	Valid	434.4	108.8	1,639 ms	9.2 ms	6.0	0	13,032	30.0 s
Sustained decode	8K	8	Valid	838.9	106.2	203 ms	9.4 ms	8.0	0	25,157	30.0 s
Sustained decode	8K	16	Valid	1,409.0	87.4	351 ms	11.4 ms	16.1	0	42,265	30.0 s
Sustained decode	8K	32	Requests queued	1,692.5	73.9	11,917 ms	13.5 ms	22.8	23	50,757	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	131.7	138.8	116 ms	7.2 ms	1.0	1	3,948	30.0 s
Sustained decode	16K	2	Valid	188.5	94.8	215 ms	10.6 ms	10.1	0	5,656	30.0 s
Sustained decode	16K	4	Valid	452.6	111.9	243 ms	8.9 ms	4.4	0	13,576	30.0 s
Sustained decode	16K	8	Valid	815.0	102.7	348 ms	9.7 ms	8.0	0	24,433	30.0 s
Sustained decode	16K	16	Valid	1,333.8	82.5	663 ms	12.1 ms	16.0	0	39,976	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	135.7	132.8	172 ms	7.5 ms	1.0	1	4,071	30.0 s
Sustained decode	32K	2	Valid	211.2	103.1	284 ms	9.7 ms	6.8	0	6,334	30.0 s
Sustained decode	32K	4	Valid	441.5	111.6	417 ms	9.0 ms	4.0	0	13,237	30.0 s
Sustained decode	32K	8	Valid	755.0	96.3	809 ms	10.4 ms	8.0	0	22,643	30.0 s
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	119.4	150.3	301 ms	6.9 ms	1.0	1	3,581	30.0 s
Sustained decode	64K	2	Valid	220.6	106.5	550 ms	9.4 ms	3.3	0	6,615	30.0 s
Sustained decode	64K	4	Valid	408.3	104.1	1,021 ms	9.6 ms	4.0	0	12,246	30.0 s
Sustained decode	64K	8	Valid	661.5	83.4	1,844 ms	12.0 ms	8.1	0	19,838	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	116.8	118.5	538 ms	8.4 ms	1.0	0	3,505	30.0 s
Sustained decode	128K	2	Valid	161.3	79.7	1,059 ms	12.6 ms	6.1	0	4,836	30.0 s
Sustained decode	128K	4	Valid	331.6	84.0	1,909 ms	11.9 ms	4.6	0	9,946	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	103.8	104.3	1,021 ms	9.6 ms	1.0	0	3,112	30.0 s
Sustained decode	251K	2	Valid	161.5	80.4	1,972 ms	12.4 ms	3.3	0	4,842	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.