Speed test matrix

Qwen3.5-35B-A3B-GPTQ-Int4 on H200 NVL 141GB

sglang · 0.5.6.post2 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 12:07 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post2
Quantization: INT4
Backend: cuda
Repository: github.com/Chevron7Locked/sglang
Build: cuda-13.2
Commit: ce79bc7e3964

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_ENABLE_SPEC_V2=1 "$HOME/env_h200-qwen-sglang-pr-22121/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.831698 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-scheduler-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context27.2K300 ms TTFT

16K context27.1K596 ms TTFT

32K context27.6K1,162 ms TTFT

64K context26.1K2,458 ms TTFT

128K context21.2K6,032 ms TTFT

251K context15.1K16,642 ms TTFT

256K context15K17,088 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	135.6126 ms TTFT	257.3135 ms TTFT	458.4171 ms TTFT	762.0213 ms TTFT	1,149.5359 ms TTFT	1,629.2675 ms TTFT	Not run	Not run
16K	134.4159 ms TTFT	253.9154 ms TTFT	448.6246 ms TTFT	737.4373 ms TTFT	1,100.9800 ms TTFT	1,530.31,458 ms TTFT	Not run	Over capacity
32K	132.0187 ms TTFT	247.2250 ms TTFT	432.2421 ms TTFT	694.0819 ms TTFT	1,014.01,534 ms TTFT	1,362.93,065 ms TTFT	Over capacity	Over capacity
64K	127.5363 ms TTFT	238.1487 ms TTFT	400.9896 ms TTFT	624.61,773 ms TTFT	872.03,421 ms TTFT	1,109.46,672 ms TTFT	Over capacity	Over capacity
128K	121.7594 ms TTFT	218.5894 ms TTFT	353.21,640 ms TTFT	517.83,243 ms TTFT	672.16,405 ms TTFT	Over capacity	Over capacity	Over capacity
251K	111.21,059 ms TTFT	188.91,816 ms TTFT	283.93,472 ms TTFT	383.36,863 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	27.2K	—	300 ms	—	—	1	—	—
Prefill	16K	1	Valid	27.1K	—	596 ms	—	—	1	—	—
Prefill	32K	1	Valid	27.6K	—	1,162 ms	—	—	1	—	—
Prefill	64K	1	Valid	26.1K	—	2,458 ms	—	—	1	—	—
Prefill	128K	1	Valid	21.2K	—	6,032 ms	—	—	1	—	—
Prefill	251K	1	Valid	15.1K	—	16,642 ms	—	—	1	—	—
Prefill	256K	1	Valid	15K	—	17,088 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	135.6	136.3	126 ms	7.3 ms	1.0	1	4,066	30.0 s
Sustained decode	8K	2	Valid	257.3	129.7	135 ms	7.7 ms	2.0	2	7,718	30.0 s
Sustained decode	8K	4	Valid	458.4	114.8	171 ms	8.7 ms	4.0	0	13,748	30.0 s
Sustained decode	8K	8	Valid	762.0	95.4	213 ms	10.5 ms	8.0	0	22,856	30.0 s
Sustained decode	8K	16	Valid	1,149.5	72.0	359 ms	13.9 ms	16.0	0	34,480	30.0 s
Sustained decode	8K	32	Valid	1,629.2	51.1	675 ms	19.6 ms	32.0	0	48,864	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	134.4	135.1	159 ms	7.4 ms	1.0	1	4,032	30.0 s
Sustained decode	16K	2	Valid	253.9	127.8	154 ms	7.8 ms	2.0	2	7,615	30.0 s
Sustained decode	16K	4	Valid	448.6	112.2	246 ms	8.9 ms	4.0	0	13,456	30.0 s
Sustained decode	16K	8	Valid	737.4	92.4	373 ms	10.8 ms	8.0	0	22,119	30.0 s
Sustained decode	16K	16	Valid	1,100.9	69.1	800 ms	14.5 ms	16.0	0	33,024	30.0 s
Sustained decode	16K	32	Valid	1,530.3	48.1	1,458 ms	20.8 ms	32.0	0	45,888	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	132.0	132.9	187 ms	7.5 ms	1.0	1	3,959	30.0 s
Sustained decode	32K	2	Valid	247.2	124.9	250 ms	8.0 ms	2.0	2	7,416	30.0 s
Sustained decode	32K	4	Valid	432.2	108.3	421 ms	9.2 ms	4.0	0	12,964	30.0 s
Sustained decode	32K	8	Valid	694.0	87.3	819 ms	11.4 ms	8.0	0	20,816	30.0 s
Sustained decode	32K	16	Valid	1,014.0	64.1	1,534 ms	15.6 ms	16.0	0	30,416	30.0 s
Sustained decode	32K	32	Valid	1,362.9	43.9	3,065 ms	22.8 ms	32.0	0	40,864	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	127.5	128.9	363 ms	7.8 ms	1.0	1	3,824	30.0 s
Sustained decode	64K	2	Valid	238.1	119.3	487 ms	8.4 ms	2.0	0	7,142	30.0 s
Sustained decode	64K	4	Valid	400.9	101.1	896 ms	9.9 ms	4.0	0	12,028	30.0 s
Sustained decode	64K	8	Valid	624.6	79.7	1,773 ms	12.6 ms	8.0	0	18,736	30.0 s
Sustained decode	64K	16	Valid	872.0	56.9	3,421 ms	17.6 ms	16.0	0	26,160	30.0 s
Sustained decode	64K	32	Valid	1,109.4	37.6	6,672 ms	26.6 ms	32.0	0	33,280	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	121.7	121.8	594 ms	8.2 ms	1.0	1	3,606	29.6 s
Sustained decode	128K	2	Valid	218.5	109.9	894 ms	9.1 ms	2.0	0	6,556	30.0 s
Sustained decode	128K	4	Valid	353.2	89.9	1,640 ms	11.1 ms	4.0	0	10,596	30.0 s
Sustained decode	128K	8	Valid	517.8	67.5	3,243 ms	14.8 ms	8.0	0	15,528	30.0 s
Sustained decode	128K	16	Valid	672.1	45.9	6,405 ms	21.8 ms	16.0	0	20,160	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	111.2	111.2	1,059 ms	9.0 ms	1.0	0	3,336	30.0 s
Sustained decode	251K	2	Valid	188.9	95.7	1,816 ms	10.5 ms	2.0	0	5,666	30.0 s
Sustained decode	251K	4	Valid	283.9	74.0	3,472 ms	13.5 ms	4.0	0	8,516	30.0 s
Sustained decode	251K	8	Valid	383.3	52.9	6,863 ms	18.9 ms	8.0	0	11,496	30.0 s
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.