Speed test matrix

Qwen3.5-122B-A10B-GPTQ-Int4 on H200 NVL 141GB

sglang · 0.5.6.post2 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 29, 2026, 21:30 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post2
Quantization: INT4
Backend: cuda
Repository: github.com/Chevron7Locked/sglang
Build: cuda-13.2
Commit: ce79bc7e3964

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 SGLANG_ENABLE_SPEC_V2=1 "$HOME/env_h200-qwen-sglang-pr-22121/bin/sglang" serve --model-path Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.886040 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-scheduler-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context12.2K671 ms TTFT

16K context12.8K1,258 ms TTFT

32K context12.3K2,609 ms TTFT

64K context11.1K5,794 ms TTFT

128K context8,79814,546 ms TTFT

251K context6,27739,958 ms TTFT

256K context6,22641,083 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	85.5167 ms TTFT	150.7309 ms TTFT	254.9299 ms TTFT	397.5316 ms TTFT	571.4367 ms TTFT	553.9Requests queued696 ms TTFT	Over capacity	Over capacity
16K	85.1205 ms TTFT	149.7194 ms TTFT	250.5243 ms TTFT	391.2363 ms TTFT	561.9687 ms TTFT	Over capacity	Over capacity	Over capacity
32K	83.8259 ms TTFT	146.9305 ms TTFT	244.5421 ms TTFT	376.2840 ms TTFT	530.91,532 ms TTFT	Over capacity	Over capacity	Over capacity
64K	82.5376 ms TTFT	142.3517 ms TTFT	233.01,047 ms TTFT	349.71,714 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	79.1577 ms TTFT	134.3909 ms TTFT	214.21,677 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	73.71,081 ms TTFT	121.21,745 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	12.2K	—	671 ms	—	—	1	—	—
Prefill	16K	1	Valid	12.8K	—	1,258 ms	—	—	1	—	—
Prefill	32K	1	Valid	12.3K	—	2,609 ms	—	—	1	—	—
Prefill	64K	1	Valid	11.1K	—	5,794 ms	—	—	1	—	—
Prefill	128K	1	Valid	8,798.0	—	14,546 ms	—	—	1	—	—
Prefill	251K	1	Valid	6,277.0	—	39,958 ms	—	—	1	—	—
Prefill	256K	1	Valid	6,226.0	—	41,083 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	85.5	85.5	167 ms	11.7 ms	1.0	0	2,565	30.0 s
Sustained decode	8K	2	Valid	150.7	75.1	309 ms	13.3 ms	2.0	0	4,518	30.0 s
Sustained decode	8K	4	Valid	254.9	64.0	299 ms	15.6 ms	4.0	0	7,644	30.0 s
Sustained decode	8K	8	Valid	397.5	50.1	316 ms	20.0 ms	8.0	0	11,920	30.0 s
Sustained decode	8K	16	Valid	571.4	36.0	367 ms	27.8 ms	16.0	0	17,136	30.0 s
Sustained decode	8K	32	Requests queued	553.9	33.4	696 ms	30.0 ms	17.0	0	16,609	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	85.1	85.0	205 ms	11.8 ms	1.0	0	2,552	30.0 s
Sustained decode	16K	2	Valid	149.7	74.8	194 ms	13.4 ms	2.0	0	4,490	30.0 s
Sustained decode	16K	4	Valid	250.5	62.9	243 ms	15.9 ms	4.0	0	7,512	30.0 s
Sustained decode	16K	8	Valid	391.2	49.1	363 ms	20.3 ms	8.0	0	11,736	30.0 s
Sustained decode	16K	16	Valid	561.9	35.5	687 ms	28.2 ms	16.0	0	16,848	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	83.8	83.8	259 ms	11.9 ms	1.0	0	2,513	30.0 s
Sustained decode	32K	2	Valid	146.9	73.4	305 ms	13.6 ms	2.0	0	4,406	30.0 s
Sustained decode	32K	4	Valid	244.5	61.2	421 ms	16.3 ms	4.0	0	7,332	30.0 s
Sustained decode	32K	8	Valid	376.2	47.2	840 ms	21.2 ms	8.0	0	11,280	30.0 s
Sustained decode	32K	16	Valid	530.9	33.5	1,532 ms	29.8 ms	16.0	0	15,920	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	82.5	82.4	376 ms	12.1 ms	1.0	0	2,473	30.0 s
Sustained decode	64K	2	Valid	142.3	71.3	517 ms	14.0 ms	2.0	0	4,268	30.0 s
Sustained decode	64K	4	Valid	233.0	58.8	1,047 ms	17.0 ms	4.0	0	6,988	30.0 s
Sustained decode	64K	8	Valid	349.7	44.5	1,714 ms	22.5 ms	8.0	0	10,488	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	79.1	79.1	577 ms	12.6 ms	1.0	0	2,373	30.0 s
Sustained decode	128K	2	Valid	134.3	67.5	909 ms	14.8 ms	2.0	0	4,028	30.0 s
Sustained decode	128K	4	Valid	214.2	54.4	1,677 ms	18.4 ms	4.0	0	6,424	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	73.7	73.7	1,081 ms	13.6 ms	1.0	0	2,211	30.0 s
Sustained decode	251K	2	Valid	121.2	61.4	1,745 ms	16.3 ms	2.0	0	3,634	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.