Speed test matrix

Qwen3.5-35B-A3B-Base on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 08:06 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861674 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context34K240 ms TTFT

16K context36.5K442 ms TTFT

32K context35.6K902 ms TTFT

64K context31K2,064 ms TTFT

128K context24.5K5,227 ms TTFT

251K context17.2K14,611 ms TTFT

256K context17.2K14,900 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	214.384 ms TTFT	320.5105 ms TTFT	605.6138 ms TTFT	910.5201 ms TTFT	1,549.9350 ms TTFT	2,293.6662 ms TTFT	3,351.21,255 ms TTFT	Over capacity
16K	211.5104 ms TTFT	169.0161 ms TTFT	621.5209 ms TTFT	863.3346 ms TTFT	1,452.6691 ms TTFT	1,995.11,306 ms TTFT	2,962.42,553 ms TTFT	Over capacity
32K	207.0159 ms TTFT	132.1267 ms TTFT	549.8427 ms TTFT	753.0830 ms TTFT	1,287.21,728 ms TTFT	1,528.63,153 ms TTFT	Over capacity	Over capacity
64K	199.8286 ms TTFT	90.5514 ms TTFT	370.31,013 ms TTFT	828.81,812 ms TTFT	1,147.53,711 ms TTFT	Over capacity	Over capacity	Over capacity
128K	185.4524 ms TTFT	136.61,001 ms TTFT	453.01,901 ms TTFT	712.63,693 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	162.51,018 ms TTFT	229.41,949 ms TTFT	389.13,761 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	34K	—	240 ms	—	—	1	—	—
Prefill	16K	1	Valid	36.5K	—	442 ms	—	—	1	—	—
Prefill	32K	1	Valid	35.6K	—	902 ms	—	—	1	—	—
Prefill	64K	1	Valid	31K	—	2,064 ms	—	—	1	—	—
Prefill	128K	1	Valid	24.5K	—	5,227 ms	—	—	1	—	—
Prefill	251K	1	Valid	17.2K	—	14,611 ms	—	—	1	—	—
Prefill	256K	1	Valid	17.2K	—	14,900 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	214.3	214.9	84 ms	4.7 ms	1.0	1	6,428	30.0 s
Sustained decode	8K	2	Valid	320.5	165.3	105 ms	6.1 ms	2.3	2	9,613	30.0 s
Sustained decode	8K	4	Valid	605.6	154.7	138 ms	6.5 ms	4.8	4	18,166	30.0 s
Sustained decode	8K	8	Valid	910.5	112.6	201 ms	8.9 ms	10.7	0	27,312	30.0 s
Sustained decode	8K	16	Valid	1,549.9	97.0	350 ms	10.3 ms	16.0	0	46,496	30.0 s
Sustained decode	8K	32	Valid	2,293.6	70.3	662 ms	14.2 ms	34.7	0	68,800	30.0 s
Sustained decode	8K	64	Valid	3,351.2	47.2	1,255 ms	21.2 ms	64.0	0	100,544	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	211.5	212.4	104 ms	4.7 ms	1.0	1	6,345	30.0 s
Sustained decode	16K	2	Valid	169.0	79.5	161 ms	12.6 ms	31.3	0	5,068	30.0 s
Sustained decode	16K	4	Valid	621.5	155.6	209 ms	6.4 ms	4.1	4	18,644	30.0 s
Sustained decode	16K	8	Valid	863.3	106.9	346 ms	9.4 ms	10.8	0	25,896	30.0 s
Sustained decode	16K	16	Valid	1,452.6	90.4	691 ms	11.1 ms	16.0	0	43,568	30.0 s
Sustained decode	16K	32	Valid	1,995.1	61.8	1,306 ms	16.2 ms	37.9	0	59,856	30.0 s
Sustained decode	16K	64	Valid	2,962.4	41.3	2,553 ms	24.2 ms	64.0	0	88,833	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	207.0	208.1	159 ms	4.8 ms	1.0	1	6,209	30.0 s
Sustained decode	32K	2	Valid	132.1	62.7	267 ms	16.0 ms	34.9	0	3,964	30.0 s
Sustained decode	32K	4	Valid	549.8	142.8	427 ms	7.0 ms	4.6	4	16,492	30.0 s
Sustained decode	32K	8	Valid	753.0	94.7	830 ms	10.6 ms	11.9	0	22,584	30.0 s
Sustained decode	32K	16	Valid	1,287.2	79.3	1,728 ms	12.6 ms	17.3	0	38,608	30.0 s
Sustained decode	32K	32	Valid	1,528.6	48.9	3,153 ms	20.4 ms	43.3	0	45,856	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	199.8	202.0	286 ms	5.0 ms	1.0	1	5,994	30.0 s
Sustained decode	64K	2	Valid	90.5	45.4	514 ms	22.0 ms	34.0	0	2,716	30.0 s
Sustained decode	64K	4	Valid	370.3	87.7	1,013 ms	11.4 ms	16.2	0	11,108	30.0 s
Sustained decode	64K	8	Valid	828.8	102.9	1,812 ms	9.7 ms	8.6	0	24,864	30.0 s
Sustained decode	64K	16	Valid	1,147.5	71.8	3,711 ms	13.9 ms	16.1	0	34,416	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	185.4	188.8	524 ms	5.3 ms	1.0	1	5,563	30.0 s
Sustained decode	128K	2	Valid	136.6	66.3	1,001 ms	15.1 ms	14.9	0	4,098	30.0 s
Sustained decode	128K	4	Valid	453.0	112.9	1,901 ms	8.9 ms	4.6	0	13,588	30.0 s
Sustained decode	128K	8	Valid	712.6	93.4	3,693 ms	10.7 ms	8.0	0	21,376	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	162.5	168.4	1,018 ms	5.9 ms	1.0	1	4,875	30.0 s
Sustained decode	251K	2	Valid	229.4	108.4	1,949 ms	9.2 ms	3.5	0	6,880	30.0 s
Sustained decode	251K	4	Valid	389.1	101.9	3,761 ms	9.8 ms	4.0	0	11,672	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.