Speed test matrix

Qwen3.5-9B-Base on H200 NVL 141GB

sglang · 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 30, 2026, 17:12 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9160+g1cf2b8c54
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1cf2b8c54d81

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-qwen-sglang/bin/sglang" serve --model-path Qwen/Qwen3.5-9B-Base --served-model-name qwen --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.833999 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-radix-cache-strategy extra_buffer --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context27.1K301 ms TTFT

16K context28.4K567 ms TTFT

32K context27.7K1,160 ms TTFT

64K context24.8K2,579 ms TTFT

128K context20.9K6,121 ms TTFT

251K context15.8K15,892 ms TTFT

256K context15.8K16,157 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	218.482 ms TTFT	408.398 ms TTFT	836.1110 ms TTFT	1,464.2156 ms TTFT	2,309.0241 ms TTFT	2,710.2673 ms TTFT	Not run	Not run
16K	228.2102 ms TTFT	381.5136 ms TTFT	794.7166 ms TTFT	1,297.9242 ms TTFT	1,924.8674 ms TTFT	2,402.81,405 ms TTFT	Not run	Over capacity
32K	214.0163 ms TTFT	284.8212 ms TTFT	661.8333 ms TTFT	1,087.5619 ms TTFT	1,466.21,613 ms TTFT	2,298.23,142 ms TTFT	Over capacity	Over capacity
64K	203.6308 ms TTFT	195.3574 ms TTFT	707.5680 ms TTFT	895.21,949 ms TTFT	1,089.03,628 ms TTFT	Over capacity	Over capacity	Over capacity
128K	193.7534 ms TTFT	203.9971 ms TTFT	567.41,639 ms TTFT	469.13,922 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	139.01,006 ms TTFT	140.61,927 ms TTFT	393.03,844 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	27.1K	—	301 ms	—	—	1	—	—
Prefill	16K	1	Valid	28.4K	—	567 ms	—	—	1	—	—
Prefill	32K	1	Valid	27.7K	—	1,160 ms	—	—	1	—	—
Prefill	64K	1	Valid	24.8K	—	2,579 ms	—	—	1	—	—
Prefill	128K	1	Valid	20.9K	—	6,121 ms	—	—	1	—	—
Prefill	251K	1	Valid	15.8K	—	15,892 ms	—	—	1	—	—
Prefill	256K	1	Valid	15.8K	—	16,157 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	218.4	220.3	82 ms	4.6 ms	1.0	1	6,551	30.0 s
Sustained decode	8K	2	Valid	408.3	206.9	98 ms	4.9 ms	2.3	2	12,250	30.0 s
Sustained decode	8K	4	Valid	836.1	209.4	110 ms	4.8 ms	4.0	4	25,079	30.0 s
Sustained decode	8K	8	Valid	1,464.2	187.6	156 ms	5.3 ms	8.0	8	43,927	30.0 s
Sustained decode	8K	16	Valid	2,309.0	149.9	241 ms	6.7 ms	17.8	16	69,254	30.0 s
Sustained decode	8K	32	Valid	2,710.2	85.1	673 ms	11.7 ms	47.5	0	81,275	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	228.2	228.8	102 ms	4.4 ms	1.0	1	6,846	30.0 s
Sustained decode	16K	2	Valid	381.5	182.7	136 ms	5.5 ms	8.3	2	11,445	30.0 s
Sustained decode	16K	4	Valid	794.7	205.3	166 ms	4.9 ms	4.4	4	23,837	30.0 s
Sustained decode	16K	8	Valid	1,297.9	169.1	242 ms	5.9 ms	8.3	8	38,938	30.0 s
Sustained decode	16K	16	Valid	1,924.8	126.1	674 ms	7.9 ms	20.1	11	57,725	30.0 s
Sustained decode	16K	32	Valid	2,402.8	76.5	1,405 ms	13.1 ms	43.5	0	72,040	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	214.0	217.5	163 ms	4.6 ms	1.0	1	6,419	30.0 s
Sustained decode	32K	2	Valid	284.8	173.8	212 ms	6.1 ms	20.0	2	8,542	30.0 s
Sustained decode	32K	4	Valid	661.8	182.5	333 ms	5.5 ms	4.9	4	19,851	30.0 s
Sustained decode	32K	8	Valid	1,087.5	146.3	619 ms	6.8 ms	9.4	8	32,623	30.0 s
Sustained decode	32K	16	Valid	1,466.2	92.8	1,613 ms	10.8 ms	23.8	0	43,990	30.0 s
Sustained decode	32K	32	Valid	2,298.2	73.0	3,142 ms	13.7 ms	35.7	0	68,904	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	203.6	207.2	308 ms	4.8 ms	1.0	1	6,108	30.0 s
Sustained decode	64K	2	Valid	195.3	93.3	574 ms	10.7 ms	23.4	0	5,858	30.0 s
Sustained decode	64K	4	Valid	707.5	179.0	680 ms	5.6 ms	4.0	4	21,220	30.0 s
Sustained decode	64K	8	Valid	895.2	109.8	1,949 ms	9.1 ms	10.0	1	26,851	30.0 s
Sustained decode	64K	16	Valid	1,089.0	71.7	3,628 ms	14.0 ms	17.3	0	32,664	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	193.7	195.8	534 ms	5.1 ms	1.0	1	5,810	30.0 s
Sustained decode	128K	2	Valid	203.9	98.2	971 ms	10.3 ms	12.0	0	6,116	30.0 s
Sustained decode	128K	4	Valid	567.4	148.0	1,639 ms	6.8 ms	3.9	4	17,019	30.0 s
Sustained decode	128K	8	Valid	469.1	64.0	3,922 ms	15.6 ms	12.0	0	14,068	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	139.0	168.3	1,006 ms	6.1 ms	1.0	1	4,168	30.0 s
Sustained decode	251K	2	Valid	140.6	69.7	1,927 ms	14.4 ms	9.4	0	4,217	30.0 s
Sustained decode	251K	4	Valid	393.0	101.0	3,844 ms	9.9 ms	4.6	0	11,785	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.