Speed test matrix

Qwen3.5-35B-A3B-Base on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 08:46 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.852316 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context34.2K238 ms TTFT

16K context35.9K449 ms TTFT

32K context34.4K932 ms TTFT

64K context29.4K2,183 ms TTFT

128K context23.1K5,537 ms TTFT

251K context16.1K15,578 ms TTFT

256K context16K15,946 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	279.988 ms TTFT	497.095 ms TTFT	630.2126 ms TTFT	1,097.9157 ms TTFT	1,557.8347 ms TTFT	2,797.8680 ms TTFT	Not run	Over capacity
16K	260.3112 ms TTFT	423.2164 ms TTFT	745.9158 ms TTFT	1,120.2254 ms TTFT	1,438.0653 ms TTFT	2,486.51,266 ms TTFT	Not run	Over capacity
32K	242.8171 ms TTFT	384.8211 ms TTFT	722.3383 ms TTFT	1,060.9807 ms TTFT	1,275.91,572 ms TTFT	2,299.63,098 ms TTFT	Over capacity	Over capacity
64K	261.6290 ms TTFT	320.9544 ms TTFT	626.0757 ms TTFT	897.21,919 ms TTFT	1,427.83,631 ms TTFT	Over capacity	Over capacity	Over capacity
128K	251.7529 ms TTFT	355.7974 ms TTFT	547.61,875 ms TTFT	709.63,661 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	211.0988 ms TTFT	334.11,888 ms TTFT	449.43,696 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	34.2K	—	238 ms	—	—	1	—	—
Prefill	16K	1	Valid	35.9K	—	449 ms	—	—	1	—	—
Prefill	32K	1	Valid	34.4K	—	932 ms	—	—	1	—	—
Prefill	64K	1	Valid	29.4K	—	2,183 ms	—	—	1	—	—
Prefill	128K	1	Valid	23.1K	—	5,537 ms	—	—	1	—	—
Prefill	251K	1	Valid	16.1K	—	15,578 ms	—	—	1	—	—
Prefill	256K	1	Valid	16K	—	15,946 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	279.9	276.4	88 ms	3.6 ms	1.0	2	8,397	30.0 s
Sustained decode	8K	2	Valid	497.0	248.2	95 ms	4.0 ms	2.0	4	14,909	30.0 s
Sustained decode	8K	4	Valid	630.2	177.9	126 ms	5.6 ms	5.0	4	18,901	30.0 s
Sustained decode	8K	8	Valid	1,097.9	141.3	157 ms	7.1 ms	9.7	8	32,926	30.0 s
Sustained decode	8K	16	Valid	1,557.8	97.0	347 ms	10.3 ms	23.5	0	46,710	30.0 s
Sustained decode	8K	32	Valid	2,797.8	84.9	680 ms	11.8 ms	32.1	0	83,894	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	260.3	264.7	112 ms	3.8 ms	1.0	2	7,807	30.0 s
Sustained decode	16K	2	Valid	423.2	207.5	164 ms	5.0 ms	5.8	2	12,695	30.0 s
Sustained decode	16K	4	Valid	745.9	186.1	158 ms	5.4 ms	4.0	4	22,377	30.0 s
Sustained decode	16K	8	Valid	1,120.2	139.4	254 ms	7.2 ms	9.0	8	33,597	30.0 s
Sustained decode	16K	16	Valid	1,438.0	91.1	653 ms	11.0 ms	23.8	0	43,118	30.0 s
Sustained decode	16K	32	Valid	2,486.5	76.3	1,266 ms	13.1 ms	32.5	0	74,548	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	242.8	288.2	171 ms	3.5 ms	0.9	2	7,284	30.0 s
Sustained decode	32K	2	Valid	384.8	202.7	211 ms	5.2 ms	9.5	2	11,542	30.0 s
Sustained decode	32K	4	Valid	722.3	180.2	383 ms	5.6 ms	4.3	4	21,668	30.0 s
Sustained decode	32K	8	Valid	1,060.9	134.5	807 ms	7.4 ms	9.3	7	31,817	30.0 s
Sustained decode	32K	16	Valid	1,275.9	81.2	1,572 ms	12.3 ms	23.0	0	38,268	30.0 s
Sustained decode	32K	32	Valid	2,299.6	68.3	3,098 ms	14.7 ms	32.2	0	68,958	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	261.6	263.4	290 ms	3.8 ms	1.0	2	7,846	30.0 s
Sustained decode	64K	2	Valid	320.9	185.9	544 ms	5.7 ms	9.8	2	9,625	30.0 s
Sustained decode	64K	4	Valid	626.0	163.2	757 ms	6.1 ms	4.8	4	18,777	30.0 s
Sustained decode	64K	8	Valid	897.2	116.8	1,919 ms	8.6 ms	10.5	0	26,905	30.0 s
Sustained decode	64K	16	Valid	1,427.8	95.3	3,631 ms	10.5 ms	16.0	0	42,821	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	251.7	258.8	529 ms	3.9 ms	0.9	2	7,551	30.0 s
Sustained decode	128K	2	Valid	355.7	180.2	974 ms	5.6 ms	4.4	2	10,670	30.0 s
Sustained decode	128K	4	Valid	547.6	147.9	1,875 ms	6.8 ms	4.8	3	16,426	30.0 s
Sustained decode	128K	8	Valid	709.6	96.4	3,661 ms	10.4 ms	10.4	0	21,283	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	211.0	218.0	988 ms	4.6 ms	1.0	1	6,328	30.0 s
Sustained decode	251K	2	Valid	334.1	170.4	1,888 ms	5.9 ms	2.8	2	10,022	30.0 s
Sustained decode	251K	4	Valid	449.4	119.1	3,696 ms	8.4 ms	4.8	1	13,480	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.