Speed test matrix

Laguna-XS-2.1-INT4 on H200 NVL 141GB

sglang · 0.5.6.post3.dev9269+g834400705 · INT4+INT86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Aug 28, 2026, 22:02 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9269+g834400705
Quantization: INT4+INT8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 834400705f2d

Run

Served model: poolside
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-poolside-sglang/bin/sglang" serve --model-path poolside/Laguna-XS-2.1-INT4 --served-model-name poolside --trust-remote-code --tp 1 --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --context-length 262144 --mem-fraction-static 0.858987 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --page-size 1 --attention-backend fa3 --speculative-algorithm DFLASH --speculative-draft-model-path poolside/Laguna-XS-2.1-DFlash-INT4

Prefill

prompt tok/s by context length

8K context33.5K245 ms TTFT

16K context32.3K502 ms TTFT

32K context30.6K1,053 ms TTFT

64K context25.3K2,537 ms TTFT

128K context18.4K6,971 ms TTFT

251K context12K20,873 ms TTFT

256K context12K21,426 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	271.368 ms TTFT	512.774 ms TTFT	935.278 ms TTFT	1,771.381 ms TTFT	2,004.3203 ms TTFT	3,250.7351 ms TTFT	Over capacity	Over capacity
16K	408.880 ms TTFT	187.9135 ms TTFT	1,026.886 ms TTFT	1,161.1187 ms TTFT	2,413.6323 ms TTFT	Over capacity	Over capacity	Over capacity
32K	263.6121 ms TTFT	266.7160 ms TTFT	744.9153 ms TTFT	943.7366 ms TTFT	1,623.1699 ms TTFT	Over capacity	Over capacity	Over capacity
64K	326.7161 ms TTFT	214.0262 ms TTFT	797.8177 ms TTFT	714.1748 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	232.5272 ms TTFT	243.8430 ms TTFT	623.3743 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	194.5473 ms TTFT	228.1869 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	33.5K	—	245 ms	—	—	1	—	—
Prefill	16K	1	Valid	32.3K	—	502 ms	—	—	1	—	—
Prefill	32K	1	Valid	30.6K	—	1,053 ms	—	—	1	—	—
Prefill	64K	1	Valid	25.3K	—	2,537 ms	—	—	1	—	—
Prefill	128K	1	Valid	18.4K	—	6,971 ms	—	—	1	—	—
Prefill	251K	1	Valid	12K	—	20,873 ms	—	—	1	—	—
Prefill	256K	1	Valid	12K	—	21,426 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	271.3	292.4	68 ms	3.4 ms	1.0	2	8,137	30.0 s
Sustained decode	8K	2	Valid	512.7	225.3	74 ms	4.4 ms	2.0	3	15,380	30.0 s
Sustained decode	8K	4	Valid	935.2	189.4	78 ms	5.3 ms	3.9	6	28,053	30.0 s
Sustained decode	8K	8	Valid	1,771.3	182.0	81 ms	5.5 ms	8.7	11	53,119	30.0 s
Sustained decode	8K	16	Valid	2,004.3	104.6	203 ms	9.6 ms	19.4	7	60,131	30.0 s
Sustained decode	8K	32	Valid	3,250.7	73.0	351 ms	13.7 ms	39.7	13	97,431	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	408.8	394.6	80 ms	2.9 ms	1.0	3	12,265	30.0 s
Sustained decode	16K	2	Valid	187.9	90.8	135 ms	11.2 ms	22.8	0	5,638	30.0 s
Sustained decode	16K	4	Valid	1,026.8	233.3	86 ms	4.5 ms	4.0	6	30,799	30.0 s
Sustained decode	16K	8	Valid	1,161.1	137.8	187 ms	7.3 ms	9.1	6	34,834	30.0 s
Sustained decode	16K	16	Valid	2,413.6	115.2	323 ms	8.7 ms	17.5	12	72,406	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	263.6	241.3	121 ms	4.1 ms	1.0	2	7,905	30.0 s
Sustained decode	32K	2	Valid	266.7	147.4	160 ms	6.8 ms	9.6	1	7,999	30.0 s
Sustained decode	32K	4	Valid	744.9	169.8	153 ms	5.9 ms	4.4	4	22,341	30.0 s
Sustained decode	32K	8	Valid	943.7	113.1	366 ms	8.8 ms	8.6	2	28,312	30.0 s
Sustained decode	32K	16	Valid	1,623.1	81.0	699 ms	12.4 ms	17.2	6	48,676	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	326.7	225.2	161 ms	4.4 ms	1.0	2	9,799	30.0 s
Sustained decode	64K	2	Valid	214.0	103.7	262 ms	9.8 ms	11.0	0	6,416	30.0 s
Sustained decode	64K	4	Valid	797.8	177.0	177 ms	5.7 ms	4.0	5	23,925	30.0 s
Sustained decode	64K	8	Valid	714.1	80.5	748 ms	12.4 ms	9.5	1	21,416	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	232.5	233.8	272 ms	4.3 ms	1.0	1	6,973	30.0 s
Sustained decode	128K	2	Valid	243.8	118.4	430 ms	8.4 ms	5.1	0	7,312	30.0 s
Sustained decode	128K	4	Valid	623.3	141.7	743 ms	7.1 ms	4.0	3	18,691	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	194.5	207.8	473 ms	4.9 ms	1.0	1	5,833	30.0 s
Sustained decode	251K	2	Valid	228.1	123.5	869 ms	8.1 ms	3.3	1	6,842	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.