Speed test matrix

Qwen3.6-35B-A3B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 31, 2026, 01:03 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.852316 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context34.1K239 ms TTFT

16K context35.5K455 ms TTFT

32K context34.9K919 ms TTFT

64K context29.6K2,163 ms TTFT

128K context23.1K5,530 ms TTFT

251K context16.1K15,576 ms TTFT

256K context16K15,951 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	327.680 ms TTFT	565.2101 ms TTFT	902.3122 ms TTFT	1,479.5165 ms TTFT	2,153.8244 ms TTFT	3,236.4647 ms TTFT	Not run	Over capacity
16K	326.0113 ms TTFT	457.6142 ms TTFT	948.9123 ms TTFT	1,387.8243 ms TTFT	1,961.4744 ms TTFT	3,146.71,243 ms TTFT	Not run	Over capacity
32K	325.9174 ms TTFT	425.8212 ms TTFT	918.1370 ms TTFT	1,385.9573 ms TTFT	1,831.81,571 ms TTFT	2,944.93,043 ms TTFT	Over capacity	Over capacity
64K	314.1308 ms TTFT	505.1419 ms TTFT	835.9719 ms TTFT	1,272.61,468 ms TTFT	1,397.43,709 ms TTFT	Over capacity	Over capacity	Over capacity
128K	291.6531 ms TTFT	470.7960 ms TTFT	760.91,492 ms TTFT	1,053.12,762 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	260.41,032 ms TTFT	197.01,915 ms TTFT	600.62,788 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	34.1K	—	239 ms	—	—	1	—	—
Prefill	16K	1	Valid	35.5K	—	455 ms	—	—	1	—	—
Prefill	32K	1	Valid	34.9K	—	919 ms	—	—	1	—	—
Prefill	64K	1	Valid	29.6K	—	2,163 ms	—	—	1	—	—
Prefill	128K	1	Valid	23.1K	—	5,530 ms	—	—	1	—	—
Prefill	251K	1	Valid	16.1K	—	15,576 ms	—	—	1	—	—
Prefill	256K	1	Valid	16K	—	15,951 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	327.6	331.7	80 ms	3.0 ms	1.0	2	9,825	30.0 s
Sustained decode	8K	2	Valid	565.2	295.5	101 ms	3.4 ms	2.3	4	16,957	30.0 s
Sustained decode	8K	4	Valid	902.3	224.9	122 ms	4.4 ms	4.3	4	27,066	30.0 s
Sustained decode	8K	8	Valid	1,479.5	187.0	165 ms	5.3 ms	8.0	8	44,383	30.0 s
Sustained decode	8K	16	Valid	2,153.8	137.2	244 ms	7.3 ms	17.8	16	64,607	30.0 s
Sustained decode	8K	32	Valid	3,236.4	100.0	647 ms	10.0 ms	32.0	26	97,059	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	326.0	328.4	113 ms	3.0 ms	0.9	2	9,779	30.0 s
Sustained decode	16K	2	Valid	457.6	235.2	142 ms	4.5 ms	10.8	2	13,729	30.0 s
Sustained decode	16K	4	Valid	948.9	241.8	123 ms	4.1 ms	4.0	6	28,466	30.0 s
Sustained decode	16K	8	Valid	1,387.8	174.6	243 ms	5.7 ms	8.9	8	41,625	30.0 s
Sustained decode	16K	16	Valid	1,961.4	125.7	744 ms	8.0 ms	19.1	12	58,781	30.0 s
Sustained decode	16K	32	Valid	3,146.7	94.0	1,243 ms	10.6 ms	32.0	30	94,344	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	325.9	329.0	174 ms	3.0 ms	1.0	2	9,778	30.0 s
Sustained decode	32K	2	Valid	425.8	228.6	212 ms	4.8 ms	12.9	2	12,773	30.0 s
Sustained decode	32K	4	Valid	918.1	227.1	370 ms	4.4 ms	4.0	4	27,542	30.0 s
Sustained decode	32K	8	Valid	1,385.9	176.8	573 ms	5.7 ms	7.9	8	41,575	30.0 s
Sustained decode	32K	16	Valid	1,831.8	115.0	1,571 ms	8.7 ms	19.5	0	54,930	30.0 s
Sustained decode	32K	32	Valid	2,944.9	97.1	3,043 ms	10.3 ms	32.0	0	88,334	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	314.1	324.3	308 ms	3.1 ms	1.0	2	9,423	30.0 s
Sustained decode	64K	2	Valid	505.1	238.5	419 ms	4.3 ms	5.9	2	15,149	30.0 s
Sustained decode	64K	4	Valid	835.9	216.6	719 ms	4.6 ms	4.0	4	25,075	30.0 s
Sustained decode	64K	8	Valid	1,272.6	166.6	1,468 ms	6.0 ms	7.9	8	38,165	30.0 s
Sustained decode	64K	16	Valid	1,397.4	91.8	3,709 ms	10.9 ms	21.7	0	41,895	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	291.6	307.0	531 ms	3.3 ms	1.0	2	8,747	30.0 s
Sustained decode	128K	2	Valid	470.7	228.8	960 ms	4.4 ms	3.4	2	14,118	30.0 s
Sustained decode	128K	4	Valid	760.9	203.0	1,492 ms	4.9 ms	3.9	4	22,824	30.0 s
Sustained decode	128K	8	Valid	1,053.1	150.8	2,762 ms	6.6 ms	8.1	8	31,583	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	260.4	279.6	1,032 ms	3.6 ms	0.9	2	7,812	30.0 s
Sustained decode	251K	2	Valid	197.0	99.7	1,915 ms	10.0 ms	9.5	0	5,909	30.0 s
Sustained decode	251K	4	Valid	600.6	169.1	2,788 ms	5.9 ms	3.8	4	18,019	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.