Speed test matrix

Qwen3.5-35B-A3B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 12:46 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861674 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion

Prefill

prompt tok/s by context length

8K context33.9K240 ms TTFT

16K context37K435 ms TTFT

32K context35.4K908 ms TTFT

64K context30.8K2,078 ms TTFT

128K context24.5K5,217 ms TTFT

251K context17.2K14,624 ms TTFT

256K context17.1K14,939 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	216.080 ms TTFT	323.4100 ms TTFT	609.3136 ms TTFT	921.9201 ms TTFT	1,571.2349 ms TTFT	2,457.3662 ms TTFT	3,598.71,251 ms TTFT	Over capacity
16K	212.8101 ms TTFT	180.2185 ms TTFT	623.0191 ms TTFT	884.8346 ms TTFT	1,488.9666 ms TTFT	2,183.31,303 ms TTFT	3,148.52,763 ms TTFT	Over capacity
32K	207.9158 ms TTFT	129.5273 ms TTFT	560.5388 ms TTFT	774.8851 ms TTFT	1,358.21,618 ms TTFT	1,688.53,153 ms TTFT	Over capacity	Over capacity
64K	200.6297 ms TTFT	92.0530 ms TTFT	429.91,047 ms TTFT	861.51,925 ms TTFT	1,183.83,672 ms TTFT	Over capacity	Over capacity	Over capacity
128K	186.3522 ms TTFT	160.4983 ms TTFT	467.61,879 ms TTFT	715.73,779 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	163.51,011 ms TTFT	232.51,910 ms TTFT	390.03,791 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	33.9K	—	240 ms	—	—	1	—	—
Prefill	16K	1	Valid	37K	—	435 ms	—	—	1	—	—
Prefill	32K	1	Valid	35.4K	—	908 ms	—	—	1	—	—
Prefill	64K	1	Valid	30.8K	—	2,078 ms	—	—	1	—	—
Prefill	128K	1	Valid	24.5K	—	5,217 ms	—	—	1	—	—
Prefill	251K	1	Valid	17.2K	—	14,624 ms	—	—	1	—	—
Prefill	256K	1	Valid	17.1K	—	14,939 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	216.0	216.6	80 ms	4.6 ms	1.0	1	6,479	30.0 s
Sustained decode	8K	2	Valid	323.4	166.4	100 ms	6.1 ms	2.4	2	9,701	30.0 s
Sustained decode	8K	4	Valid	609.3	156.7	136 ms	6.4 ms	4.8	4	18,280	30.0 s
Sustained decode	8K	8	Valid	921.9	115.2	201 ms	8.7 ms	10.5	0	27,656	30.0 s
Sustained decode	8K	16	Valid	1,571.2	99.5	349 ms	10.0 ms	16.0	0	47,136	30.0 s
Sustained decode	8K	32	Valid	2,457.3	75.4	662 ms	13.3 ms	33.6	0	73,728	30.0 s
Sustained decode	8K	64	Valid	3,598.7	51.0	1,251 ms	19.6 ms	64.0	0	107,968	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	212.8	213.6	101 ms	4.7 ms	1.0	1	6,383	30.0 s
Sustained decode	16K	2	Valid	180.2	84.6	185 ms	11.8 ms	30.6	0	5,406	30.0 s
Sustained decode	16K	4	Valid	623.0	155.1	191 ms	6.5 ms	4.1	4	18,687	30.0 s
Sustained decode	16K	8	Valid	884.8	110.1	346 ms	9.1 ms	10.7	0	26,544	30.0 s
Sustained decode	16K	16	Valid	1,488.9	93.9	666 ms	10.6 ms	16.0	0	44,656	30.0 s
Sustained decode	16K	32	Valid	2,183.3	67.4	1,303 ms	14.8 ms	36.4	0	65,504	30.0 s
Sustained decode	16K	64	Valid	3,148.5	44.4	2,763 ms	22.5 ms	64.0	0	94,464	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	207.9	209.0	158 ms	4.8 ms	1.0	1	6,237	30.0 s
Sustained decode	32K	2	Valid	129.5	61.7	273 ms	16.2 ms	37.7	0	3,884	30.0 s
Sustained decode	32K	4	Valid	560.5	145.4	388 ms	6.9 ms	4.7	4	16,813	30.0 s
Sustained decode	32K	8	Valid	774.8	97.7	851 ms	10.2 ms	11.7	0	23,240	30.0 s
Sustained decode	32K	16	Valid	1,358.2	83.5	1,618 ms	12.0 ms	16.8	0	40,736	30.0 s
Sustained decode	32K	32	Valid	1,688.5	53.7	3,153 ms	18.6 ms	41.4	0	50,656	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	200.6	202.7	297 ms	4.9 ms	1.0	1	6,018	30.0 s
Sustained decode	64K	2	Valid	92.0	46.2	530 ms	21.6 ms	34.0	0	2,758	30.0 s
Sustained decode	64K	4	Valid	429.9	101.1	1,047 ms	9.9 ms	10.8	0	12,896	30.0 s
Sustained decode	64K	8	Valid	861.5	108.1	1,925 ms	9.3 ms	8.0	0	25,840	30.0 s
Sustained decode	64K	16	Valid	1,183.8	77.2	3,672 ms	13.0 ms	16.0	0	35,504	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	186.3	189.6	522 ms	5.3 ms	1.0	1	5,588	30.0 s
Sustained decode	128K	2	Valid	160.4	76.8	983 ms	13.0 ms	12.9	0	4,812	30.0 s
Sustained decode	128K	4	Valid	467.6	116.2	1,879 ms	8.6 ms	4.4	0	14,028	30.0 s
Sustained decode	128K	8	Valid	715.7	94.0	3,779 ms	10.6 ms	8.0	0	21,464	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	163.5	169.2	1,011 ms	5.9 ms	1.0	1	4,904	30.0 s
Sustained decode	251K	2	Valid	232.5	109.8	1,910 ms	9.1 ms	3.5	0	6,974	30.0 s
Sustained decode	251K	4	Valid	390.0	102.2	3,791 ms	9.8 ms	4.0	0	11,701	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.