Speed test matrix

Qwen3.5-0.8B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 29, 2026, 17:31 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-0.8B --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.861366 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-flashinfer-allreduce-fusion

Prefill

prompt tok/s by context length

8K context70.1K116 ms TTFT

16K context98.7K163 ms TTFT

32K context94.1K341 ms TTFT

64K context90K712 ms TTFT

128K context76.6K1,671 ms TTFT

251K context54.9K4,568 ms TTFT

256K context56K4,564 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	573.677 ms TTFT	1,105.650 ms TTFT	2,133.492 ms TTFT	4,037.2111 ms TTFT	6,861.4138 ms TTFT	10.2K313 ms TTFT	13.8K1,109 ms TTFT	16K2,495 ms TTFT
16K	560.598 ms TTFT	443.2119 ms TTFT	2,036.1130 ms TTFT	3,744.9126 ms TTFT	6,077.4131 ms TTFT	7,809.1827 ms TTFT	8,494.61,784 ms TTFT	8,846.45,079 ms TTFT
32K	540.6147 ms TTFT	686.2247 ms TTFT	1,872.3231 ms TTFT	3,259.4130 ms TTFT	4,862.8242 ms TTFT	5,897.32,283 ms TTFT	6,446.56,188 ms TTFT	8,140.812,809 ms TTFT
64K	502.0290 ms TTFT	492.9515 ms TTFT	1,605.4250 ms TTFT	2,606.6257 ms TTFT	3,486.21,993 ms TTFT	3,972.74,220 ms TTFT	2,748.315,234 ms TTFT	Over capacity
128K	445.2522 ms TTFT	88.61,008 ms TTFT	1,094.11,425 ms TTFT	1,677.22,383 ms TTFT	2,080.83,911 ms TTFT	1,445.814,771 ms TTFT	Over capacity	Over capacity
251K	361.61,007 ms TTFT	96.21,914 ms TTFT	777.63,333 ms TTFT	1,097.64,614 ms TTFT	671.416,221 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	70.1K	—	116 ms	—	—	1	—	—
Prefill	16K	1	Valid	98.7K	—	163 ms	—	—	1	—	—
Prefill	32K	1	Valid	94.1K	—	341 ms	—	—	1	—	—
Prefill	64K	1	Valid	90K	—	712 ms	—	—	1	—	—
Prefill	128K	1	Valid	76.6K	—	1,671 ms	—	—	1	—	—
Prefill	251K	1	Valid	54.9K	—	4,568 ms	—	—	1	—	—
Prefill	256K	1	Valid	56K	—	4,564 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	573.6	579.1	77 ms	1.7 ms	1.0	4	17,208	30.0 s
Sustained decode	8K	2	Valid	1,105.6	574.3	50 ms	1.7 ms	2.0	8	33,167	30.0 s
Sustained decode	8K	4	Valid	2,133.4	539.5	92 ms	1.9 ms	4.0	16	64,001	30.0 s
Sustained decode	8K	8	Valid	4,037.2	511.9	111 ms	2.0 ms	8.0	32	121,116	30.0 s
Sustained decode	8K	16	Valid	6,861.4	437.6	138 ms	2.3 ms	16.3	48	205,846	30.0 s
Sustained decode	8K	32	Valid	10.2K	322.8	313 ms	3.1 ms	31.8	64	305,459	30.0 s
Sustained decode	8K	64	Valid	13.8K	217.4	1,109 ms	4.6 ms	64.2	64	415,017	30.0 s
Sustained decode	8K	128	Valid	16K	127.2	2,495 ms	7.9 ms	127.1	128	479,581	30.0 s
Sustained decode	16K	1	Valid	560.5	568.0	98 ms	1.8 ms	1.0	4	16,814	30.0 s
Sustained decode	16K	2	Valid	443.2	351.1	119 ms	4.4 ms	98.5	2	13,296	30.0 s
Sustained decode	16K	4	Valid	2,036.1	518.1	130 ms	1.9 ms	3.9	16	61,080	30.0 s
Sustained decode	16K	8	Valid	3,744.9	473.5	126 ms	2.1 ms	8.0	24	112,348	30.0 s
Sustained decode	16K	16	Valid	6,077.4	387.1	131 ms	2.6 ms	15.8	48	182,318	30.0 s
Sustained decode	16K	32	Valid	7,809.1	231.3	827 ms	4.3 ms	35.5	42	234,243	30.0 s
Sustained decode	16K	64	Valid	8,494.6	149.1	1,784 ms	6.7 ms	69.2	64	254,833	30.0 s
Sustained decode	16K	128	Valid	8,846.4	74.2	5,079 ms	13.5 ms	192.0	0	265,376	30.0 s
Sustained decode	32K	1	Valid	540.6	551.5	147 ms	1.8 ms	1.0	4	16,217	30.0 s
Sustained decode	32K	2	Valid	686.2	534.9	247 ms	1.9 ms	55.8	4	20,585	30.0 s
Sustained decode	32K	4	Valid	1,872.3	477.0	231 ms	2.1 ms	3.9	12	56,169	30.0 s
Sustained decode	32K	8	Valid	3,259.4	416.2	130 ms	2.4 ms	7.9	24	97,782	30.0 s
Sustained decode	32K	16	Valid	4,862.8	311.4	242 ms	3.2 ms	16.1	32	145,886	30.0 s
Sustained decode	32K	32	Valid	5,897.3	196.6	2,283 ms	5.1 ms	33.8	32	176,903	30.0 s
Sustained decode	32K	64	Valid	6,446.5	104.4	6,188 ms	9.6 ms	72.9	0	193,408	30.0 s
Sustained decode	32K	128	Valid	8,140.8	76.8	12,809 ms	13.0 ms	128.0	0	244,224	30.0 s
Sustained decode	64K	1	Valid	502.0	522.7	290 ms	1.9 ms	1.0	4	15,060	30.0 s
Sustained decode	64K	2	Valid	492.9	321.9	515 ms	4.2 ms	58.7	2	14,783	30.0 s
Sustained decode	64K	4	Valid	1,605.4	413.4	250 ms	2.4 ms	3.9	12	48,160	30.0 s
Sustained decode	64K	8	Valid	2,606.6	333.1	257 ms	3.0 ms	7.8	16	78,196	30.0 s
Sustained decode	64K	16	Valid	3,486.2	224.7	1,993 ms	4.5 ms	15.8	17	104,600	30.0 s
Sustained decode	64K	32	Valid	3,972.7	135.5	4,220 ms	7.4 ms	31.9	32	119,166	30.0 s
Sustained decode	64K	64	Valid	2,748.3	52.9	15,234 ms	18.9 ms	95.2	0	82,432	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	445.2	472.4	522 ms	2.1 ms	0.9	3	13,355	30.0 s
Sustained decode	128K	2	Valid	88.6	44.4	1,008 ms	22.5 ms	74.4	0	2,658	30.0 s
Sustained decode	128K	4	Valid	1,094.1	339.5	1,425 ms	2.9 ms	7.4	8	32,825	30.0 s
Sustained decode	128K	8	Valid	1,677.2	229.8	2,383 ms	4.4 ms	8.9	8	50,312	30.0 s
Sustained decode	128K	16	Valid	2,080.8	137.4	3,911 ms	7.3 ms	15.7	16	62,424	30.0 s
Sustained decode	128K	32	Valid	1,445.8	55.2	14,771 ms	18.1 ms	46.7	0	43,376	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	361.6	402.2	1,007 ms	2.5 ms	0.9	3	10,848	30.0 s
Sustained decode	251K	2	Valid	96.2	48.4	1,914 ms	20.7 ms	35.2	0	2,884	30.0 s
Sustained decode	251K	4	Valid	777.6	209.3	3,333 ms	4.8 ms	4.3	4	23,325	30.0 s
Sustained decode	251K	8	Valid	1,097.6	149.6	4,614 ms	6.7 ms	7.7	8	32,927	30.0 s
Sustained decode	251K	16	Valid	671.4	53.3	16,221 ms	18.8 ms	23.4	0	20,144	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.