Speed test matrix

Qwen3.6-35B-A3B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 2, 2026, 04:14 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9269+g834400705
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 834400705f2d

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-z-lab-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.857035 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --linear-attn-prefill-backend flashinfer --linear-attn-decode-backend flashinfer --mamba-scheduler-strategy extra_buffer --max-running-requests 32 --cuda-graph-max-bs-decode 32 --cuda-graph-backend-prefill tc_piecewise --enable-flashinfer-allreduce-fusion --attention-backend flashinfer --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/Qwen3.6-35B-A3B-DFlash --speculative-dflash-block-size 8 --speculative-draft-attention-backend fa4

Prefill

prompt tok/s by context length

8K context31K262 ms TTFT

16K context32.4K498 ms TTFT

32K context29.1K1,104 ms TTFT

64K context22.9K2,793 ms TTFT

128K context16K8,012 ms TTFT

251K context9,96325,173 ms TTFT

256K context9,85625,952 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	338.193 ms TTFT	501.096 ms TTFT	756.1112 ms TTFT	1,258.1158 ms TTFT	1,605.9350 ms TTFT	2,412.6Requests queued1,806 ms TTFT	Over capacity	Over capacity
16K	307.7113 ms TTFT	384.9141 ms TTFT	815.2162 ms TTFT	1,252.0238 ms TTFT	1,552.2643 ms TTFT	2,389.0Requests queued1,489 ms TTFT	Over capacity	Over capacity
32K	302.7176 ms TTFT	434.4238 ms TTFT	779.8305 ms TTFT	1,146.5528 ms TTFT	1,382.61,532 ms TTFT	Over capacity	Over capacity	Over capacity
64K	295.7318 ms TTFT	429.9424 ms TTFT	712.3629 ms TTFT	1,009.41,857 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	265.1550 ms TTFT	213.71,014 ms TTFT	653.31,425 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	222.9995 ms TTFT	328.41,866 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	31K	—	262 ms	—	—	1	—	—
Prefill	16K	1	Valid	32.4K	—	498 ms	—	—	1	—	—
Prefill	32K	1	Valid	29.1K	—	1,104 ms	—	—	1	—	—
Prefill	64K	1	Valid	22.9K	—	2,793 ms	—	—	1	—	—
Prefill	128K	1	Valid	16K	—	8,012 ms	—	—	1	—	—
Prefill	251K	1	Valid	9,963.0	—	25,173 ms	—	—	1	—	—
Prefill	256K	1	Valid	9,856.0	—	25,952 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	338.1	350.9	93 ms	2.8 ms	1.0	2	10,142	30.0 s
Sustained decode	8K	2	Valid	501.0	260.2	96 ms	3.8 ms	2.0	4	15,029	30.0 s
Sustained decode	8K	4	Valid	756.1	193.3	112 ms	5.2 ms	4.8	4	22,679	30.0 s
Sustained decode	8K	8	Valid	1,258.1	163.6	158 ms	6.1 ms	8.5	8	37,737	30.0 s
Sustained decode	8K	16	Valid	1,605.9	105.2	350 ms	9.5 ms	21.3	0	48,148	30.0 s
Sustained decode	8K	32	Requests queued	2,412.6	88.9	1,806 ms	11.3 ms	28.0	32	72,342	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	307.7	310.3	113 ms	3.2 ms	1.0	2	9,232	30.0 s
Sustained decode	16K	2	Valid	384.9	210.1	141 ms	5.0 ms	5.7	2	11,544	30.0 s
Sustained decode	16K	4	Valid	815.2	207.0	162 ms	4.8 ms	4.1	4	24,456	30.0 s
Sustained decode	16K	8	Valid	1,252.0	163.6	238 ms	6.1 ms	8.1	8	37,547	30.0 s
Sustained decode	16K	16	Valid	1,552.2	100.8	643 ms	9.9 ms	21.9	0	46,560	30.0 s
Sustained decode	16K	32	Requests queued	2,389.0	82.5	1,489 ms	12.1 ms	31.0	28	71,663	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	302.7	312.5	176 ms	3.2 ms	1.0	2	9,078	30.0 s
Sustained decode	32K	2	Valid	434.4	214.6	238 ms	4.7 ms	3.7	2	13,031	30.0 s
Sustained decode	32K	4	Valid	779.8	204.6	305 ms	4.9 ms	4.0	4	23,389	30.0 s
Sustained decode	32K	8	Valid	1,146.5	152.6	528 ms	6.6 ms	8.3	8	34,392	30.0 s
Sustained decode	32K	16	Valid	1,382.6	91.2	1,532 ms	11.0 ms	23.1	0	41,478	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	295.7	303.5	318 ms	3.3 ms	1.0	2	8,869	30.0 s
Sustained decode	64K	2	Valid	429.9	209.4	424 ms	4.8 ms	4.3	2	12,897	30.0 s
Sustained decode	64K	4	Valid	712.3	181.8	629 ms	5.5 ms	3.9	4	21,368	30.0 s
Sustained decode	64K	8	Valid	1,009.4	138.4	1,857 ms	7.2 ms	9.0	8	30,259	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	265.1	275.4	550 ms	3.6 ms	0.9	2	7,952	30.0 s
Sustained decode	128K	2	Valid	213.7	111.1	1,014 ms	9.0 ms	8.6	0	6,411	30.0 s
Sustained decode	128K	4	Valid	653.3	173.6	1,425 ms	5.8 ms	3.9	4	19,599	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	222.9	235.5	995 ms	4.2 ms	1.0	1	6,688	30.0 s
Sustained decode	251K	2	Valid	328.4	183.6	1,866 ms	5.5 ms	3.3	2	9,852	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.