Speed test matrix

Laguna-S-2.1-INT4 on H200 NVL 141GB ×2

vllm · 0.27.1 · INT48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 5, 2026, 11:21 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.27.1
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: poolside
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-poolside-vllm/bin/vllm" serve poolside/Laguna-S-2.1-INT4 --served-model-name poolside --trust-remote-code --tensor-parallel-size 2 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 1048576 --gpu-memory-utilization 0.860342 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context24K342 ms TTFT

16K context24.2K670 ms TTFT

32K context23.6K1,364 ms TTFT

64K context22K2,915 ms TTFT

128K context18.5K6,943 ms TTFT

256K context13.8K18,534 ms TTFT

512K context9,20555,686 ms TTFT

1019K context5,567183,216 ms TTFT

1024K context5,529185,372 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	155.341 ms TTFT	283.951 ms TTFT	485.568 ms TTFT	792.8140 ms TTFT	1,243.6178 ms TTFT	1,904.7291 ms TTFT	2,878.6405 ms TTFT	4,021.1694 ms TTFT
16K	153.786 ms TTFT	279.672 ms TTFT	482.989 ms TTFT	764.5201 ms TTFT	1,198.0291 ms TTFT	1,831.9422 ms TTFT	2,638.7666 ms TTFT	3,605.81,168 ms TTFT
32K	151.0112 ms TTFT	273.6112 ms TTFT	465.4178 ms TTFT	734.7258 ms TTFT	1,124.5401 ms TTFT	1,666.0672 ms TTFT	2,321.81,130 ms TTFT	Over capacity
64K	146.8158 ms TTFT	260.7192 ms TTFT	440.8338 ms TTFT	678.9518 ms TTFT	996.8854 ms TTFT	1,392.81,407 ms TTFT	Over capacity	Over capacity
128K	138.4273 ms TTFT	242.3344 ms TTFT	400.7586 ms TTFT	583.8889 ms TTFT	817.61,489 ms TTFT	Over capacity	Over capacity	Over capacity
256K	124.0526 ms TTFT	209.1734 ms TTFT	334.41,102 ms TTFT	460.31,835 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
512K	105.31,216 ms TTFT	166.01,615 ms TTFT	252.72,510 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	75.6Requests queued2,456 ms TTFT	107.6Requests queued3,548 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	24K	—	342 ms	—	—	1	—	—
Prefill	16K	1	Valid	24.2K	—	670 ms	—	—	1	—	—
Prefill	32K	1	Valid	23.6K	—	1,364 ms	—	—	1	—	—
Prefill	64K	1	Valid	22K	—	2,915 ms	—	—	1	—	—
Prefill	128K	1	Valid	18.5K	—	6,943 ms	—	—	1	—	—
Prefill	256K	1	Valid	13.8K	—	18,534 ms	—	—	1	—	—
Prefill	512K	1	Valid	9,205.0	—	55,686 ms	—	—	1	—	—
Prefill	1019K	1	Valid	5,567.0	—	183,216 ms	—	—	1	—	—
Prefill	1024K	1	Valid	5,529.0	—	185,372 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	155.3	155.7	41 ms	6.4 ms	1.0	1	4,657	30.0 s
Sustained decode	8K	2	Valid	283.9	142.5	51 ms	7.0 ms	2.0	2	8,518	30.0 s
Sustained decode	8K	4	Valid	485.5	123.2	68 ms	8.1 ms	4.0	4	14,562	30.0 s
Sustained decode	8K	8	Valid	792.8	99.5	140 ms	10.1 ms	8.0	0	23,784	30.0 s
Sustained decode	8K	16	Valid	1,243.6	78.2	178 ms	12.8 ms	16.0	0	37,296	30.0 s
Sustained decode	8K	32	Valid	1,904.7	60.0	291 ms	16.7 ms	32.0	0	57,120	30.0 s
Sustained decode	8K	64	Valid	2,878.6	45.3	405 ms	22.1 ms	64.0	0	86,337	30.0 s
Sustained decode	8K	128	Valid	4,021.1	31.6	694 ms	31.6 ms	128.0	0	120,576	30.0 s
Sustained decode	16K	1	Valid	153.7	154.4	86 ms	6.5 ms	1.0	1	4,612	30.0 s
Sustained decode	16K	2	Valid	279.6	140.7	72 ms	7.1 ms	2.0	2	8,388	30.0 s
Sustained decode	16K	4	Valid	482.9	122.9	89 ms	8.1 ms	4.0	4	14,485	30.0 s
Sustained decode	16K	8	Valid	764.5	97.0	201 ms	10.3 ms	8.0	0	22,931	30.0 s
Sustained decode	16K	16	Valid	1,198.0	75.3	291 ms	13.3 ms	16.0	0	35,935	30.0 s
Sustained decode	16K	32	Valid	1,831.9	57.6	422 ms	17.4 ms	32.0	0	54,944	30.0 s
Sustained decode	16K	64	Valid	2,638.7	41.5	666 ms	24.1 ms	64.0	0	79,164	30.0 s
Sustained decode	16K	128	Valid	3,605.8	28.4	1,168 ms	35.2 ms	128.0	0	108,160	30.0 s
Sustained decode	32K	1	Valid	151.0	152.1	112 ms	6.6 ms	1.0	1	4,531	30.0 s
Sustained decode	32K	2	Valid	273.6	138.0	112 ms	7.2 ms	1.9	2	8,206	30.0 s
Sustained decode	32K	4	Valid	465.4	116.6	178 ms	8.6 ms	4.0	0	13,961	30.0 s
Sustained decode	32K	8	Valid	734.7	92.1	258 ms	10.9 ms	8.0	0	22,040	30.0 s
Sustained decode	32K	16	Valid	1,124.5	70.6	401 ms	14.2 ms	16.0	0	33,728	30.0 s
Sustained decode	32K	32	Valid	1,666.0	52.4	672 ms	19.1 ms	32.0	0	49,984	30.0 s
Sustained decode	32K	64	Valid	2,321.8	36.5	1,130 ms	27.4 ms	64.0	0	69,632	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	146.8	148.3	158 ms	6.7 ms	1.0	1	4,404	30.0 s
Sustained decode	64K	2	Valid	260.7	132.7	192 ms	7.5 ms	2.0	2	7,822	30.0 s
Sustained decode	64K	4	Valid	440.8	110.4	338 ms	9.1 ms	4.0	0	13,220	30.0 s
Sustained decode	64K	8	Valid	678.9	85.1	518 ms	11.7 ms	8.0	0	20,368	30.0 s
Sustained decode	64K	16	Valid	996.8	62.6	854 ms	16.0 ms	16.0	0	29,905	30.0 s
Sustained decode	64K	32	Valid	1,392.8	43.8	1,407 ms	22.8 ms	32.0	0	41,760	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	138.4	140.4	273 ms	7.1 ms	1.0	1	4,150	30.0 s
Sustained decode	128K	2	Valid	242.3	121.4	344 ms	8.2 ms	2.0	0	7,268	30.0 s
Sustained decode	128K	4	Valid	400.7	100.5	586 ms	10.0 ms	4.0	0	12,021	30.0 s
Sustained decode	128K	8	Valid	583.8	73.2	889 ms	13.7 ms	8.0	0	17,512	30.0 s
Sustained decode	128K	16	Valid	817.6	51.4	1,489 ms	19.5 ms	16.0	0	24,529	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	124.0	181.9	526 ms	6.1 ms	1.0	1	3,719	30.0 s
Sustained decode	256K	2	Valid	209.1	105.0	734 ms	9.5 ms	2.0	0	6,272	30.0 s
Sustained decode	256K	4	Valid	334.4	83.9	1,102 ms	11.9 ms	4.0	0	10,036	30.0 s
Sustained decode	256K	8	Valid	460.3	57.8	1,835 ms	17.3 ms	8.0	0	13,816	30.0 s
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	105.3	105.9	1,216 ms	9.4 ms	1.0	0	3,159	30.0 s
Sustained decode	512K	2	Valid	166.0	83.6	1,615 ms	12.0 ms	2.0	0	4,980	30.0 s
Sustained decode	512K	4	Valid	252.7	63.7	2,510 ms	15.7 ms	4.0	0	7,580	30.0 s
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Requests queued	75.6	80.8	2,456 ms	12.4 ms	0.9	0	2,267	30.0 s
Sustained decode	1019K	2	Requests queued	107.6	59.8	3,548 ms	16.7 ms	1.8	0	3,227	30.0 s
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.