Speed test matrix

Laguna-M.1-FP8 on H200 SXM 141GB ×4

vllm · 0.29.0 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 13, 2026, 10:47 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: poolside
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nex-n2-vllm-v0290/bin/vllm" serve poolside/Laguna-M.1-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 4 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.859321 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype fp8_e4m3 --default-chat-template-kwargs '{"enable_thinking":true}' --linear-backend triton

Prefill

prompt tok/s by context length

8K context18.2K451 ms TTFT

16K context17.8K907 ms TTFT

32K context16.3K1,981 ms TTFT

64K context13.4K4,796 ms TTFT

128K context9,71613,188 ms TTFT

251K context6,32139,723 ms TTFT

256K context6,22341,148 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	59.676 ms TTFT	113.1102 ms TTFT	209.6141 ms TTFT	Looped	705.9248 ms TTFT	1,145.9409 ms TTFT	2,035.0650 ms TTFT	3,200.81,156 ms TTFT
16K	Looped	112.0127 ms TTFT	205.2170 ms TTFT	367.7248 ms TTFT	660.9377 ms TTFT	Looped	1,729.11,013 ms TTFT	Over capacity
32K	57.5158 ms TTFT	109.5179 ms TTFT	197.5250 ms TTFT	345.4372 ms TTFT	593.1576 ms TTFT	879.31,044 ms TTFT	Over capacity	Over capacity
64K	55.1272 ms TTFT	104.8270 ms TTFT	183.4439 ms TTFT	307.3701 ms TTFT	491.81,160 ms TTFT	Over capacity	Over capacity	Over capacity
128K	50.9444 ms TTFT	96.8525 ms TTFT	160.6828 ms TTFT	252.11,313 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	44.5858 ms TTFT	84.41,133 ms TTFT	130.21,695 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	18.2K	—	451 ms	—	—	1	—	—
Prefill	16K	1	Valid	17.8K	—	907 ms	—	—	1	—	—
Prefill	32K	1	Valid	16.3K	—	1,981 ms	—	—	1	—	—
Prefill	64K	1	Valid	13.4K	—	4,796 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,716.0	—	13,188 ms	—	—	1	—	—
Prefill	251K	1	Valid	6,321.0	—	39,723 ms	—	—	1	—	—
Prefill	256K	1	Valid	6,223.0	—	41,148 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	59.6	59.6	76 ms	16.8 ms	1.0	0	1,787	30.0 s
Sustained decode	8K	2	Valid	113.1	56.6	102 ms	17.7 ms	2.0	0	3,392	30.0 s
Sustained decode	8K	4	Valid	209.6	52.5	141 ms	19.1 ms	4.0	0	6,288	30.0 s
Sustained decode	8K	8	Looped	—	—	—	—	—	0	0	—
Sustained decode	8K	16	Valid	705.9	44.3	248 ms	22.6 ms	16.0	0	21,168	30.0 s
Sustained decode	8K	32	Valid	1,145.9	36.0	409 ms	27.8 ms	32.0	0	34,368	30.0 s
Sustained decode	8K	64	Valid	2,035.0	32.0	650 ms	31.3 ms	64.0	0	61,056	30.0 s
Sustained decode	8K	128	Valid	3,200.8	25.1	1,156 ms	39.8 ms	128.0	0	95,999	30.0 s
Sustained decode	16K	1	Looped	—	—	—	—	—	0	0	—
Sustained decode	16K	2	Valid	112.0	56.0	127 ms	17.9 ms	2.0	0	3,358	30.0 s
Sustained decode	16K	4	Valid	205.2	51.4	170 ms	19.5 ms	4.0	0	6,156	30.0 s
Sustained decode	16K	8	Valid	367.7	46.1	248 ms	21.7 ms	8.0	0	11,024	30.0 s
Sustained decode	16K	16	Valid	660.9	41.5	377 ms	24.1 ms	16.0	0	19,827	30.0 s
Sustained decode	16K	32	Looped	—	—	—	—	—	0	0	—
Sustained decode	16K	64	Valid	1,729.1	27.1	1,013 ms	36.9 ms	64.0	0	51,840	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	57.5	57.6	158 ms	17.4 ms	1.0	0	1,726	30.0 s
Sustained decode	32K	2	Valid	109.5	54.7	179 ms	18.3 ms	2.0	0	3,284	30.0 s
Sustained decode	32K	4	Valid	197.5	49.4	250 ms	20.2 ms	4.0	0	5,924	30.0 s
Sustained decode	32K	8	Valid	345.4	43.3	372 ms	23.1 ms	8.0	0	10,360	30.0 s
Sustained decode	32K	16	Valid	593.1	37.2	576 ms	26.9 ms	16.0	0	17,792	30.0 s
Sustained decode	32K	32	Valid	879.3	27.6	1,044 ms	36.3 ms	32.0	0	26,368	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	55.1	55.2	272 ms	18.1 ms	1.0	0	1,653	30.0 s
Sustained decode	64K	2	Valid	104.8	52.4	270 ms	19.1 ms	2.0	0	3,142	30.0 s
Sustained decode	64K	4	Valid	183.4	45.9	439 ms	21.8 ms	4.0	0	5,500	30.0 s
Sustained decode	64K	8	Valid	307.3	38.5	701 ms	26.0 ms	8.0	0	9,216	30.0 s
Sustained decode	64K	16	Valid	491.8	30.9	1,160 ms	32.4 ms	16.0	0	14,752	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	50.9	51.0	444 ms	19.6 ms	1.0	0	1,527	30.0 s
Sustained decode	128K	2	Valid	96.8	48.5	525 ms	20.6 ms	2.0	0	2,902	30.0 s
Sustained decode	128K	4	Valid	160.6	40.3	828 ms	24.8 ms	4.0	0	4,818	30.0 s
Sustained decode	128K	8	Valid	252.1	31.7	1,313 ms	31.6 ms	8.0	0	7,560	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	44.5	44.7	858 ms	22.4 ms	1.0	0	1,336	30.0 s
Sustained decode	251K	2	Valid	84.4	42.4	1,133 ms	23.6 ms	2.0	0	2,532	30.0 s
Sustained decode	251K	4	Valid	130.2	32.7	1,695 ms	30.6 ms	4.0	0	3,904	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.