Speed test matrix

Laguna-S-2.1-FP8 on H200 NVL 141GB ×2

vllm · 0.27.1 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 5, 2026, 10:34 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.27.1
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: poolside
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 "$HOME/env_h200-poolside-vllm/bin/vllm" serve poolside/Laguna-S-2.1-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 2 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 1048576 --gpu-memory-utilization 0.859591 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager

Prefill

prompt tok/s by context length

8K context12.9K634 ms TTFT

16K context29.9K542 ms TTFT

32K context29.6K1,088 ms TTFT

64K context26.6K2,415 ms TTFT

128K context21.6K5,948 ms TTFT

256K context15.5K16,558 ms TTFT

512K context9,91951,677 ms TTFT

1019K context5,813175,465 ms TTFT

1024K context5,772177,589 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	30.766 ms TTFT	61.1283 ms TTFT	122.3147 ms TTFT	238.6565 ms TTFT	478.1630 ms TTFT	952.2332 ms TTFT	1,886.5528 ms TTFT	3,595.4848 ms TTFT
16K	30.6107 ms TTFT	61.3124 ms TTFT	122.4196 ms TTFT	238.8239 ms TTFT	477.8307 ms TTFT	944.4457 ms TTFT	1,872.2716 ms TTFT	Over capacity
32K	30.5133 ms TTFT	60.5192 ms TTFT	122.0251 ms TTFT	238.7314 ms TTFT	479.3483 ms TTFT	954.7759 ms TTFT	1,872.71,293 ms TTFT	Over capacity
64K	30.5183 ms TTFT	60.8262 ms TTFT	121.1351 ms TTFT	239.8525 ms TTFT	477.5800 ms TTFT	945.21,425 ms TTFT	Over capacity	Over capacity
128K	30.6337 ms TTFT	61.3433 ms TTFT	122.7652 ms TTFT	239.21,075 ms TTFT	476.81,924 ms TTFT	Over capacity	Over capacity	Over capacity
256K	30.6619 ms TTFT	61.1810 ms TTFT	121.91,211 ms TTFT	238.11,876 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
512K	30.71,193 ms TTFT	61.01,795 ms TTFT	121.72,604 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	30.4Requests queued2,684 ms TTFT	61.0Requests queued3,924 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	12.9K	—	634 ms	—	—	1	—	—
Prefill	16K	1	Valid	29.9K	—	542 ms	—	—	1	—	—
Prefill	32K	1	Valid	29.6K	—	1,088 ms	—	—	1	—	—
Prefill	64K	1	Valid	26.6K	—	2,415 ms	—	—	1	—	—
Prefill	128K	1	Valid	21.6K	—	5,948 ms	—	—	1	—	—
Prefill	256K	1	Valid	15.5K	—	16,558 ms	—	—	1	—	—
Prefill	512K	1	Valid	9,919.0	—	51,677 ms	—	—	1	—	—
Prefill	1019K	1	Valid	5,813.0	—	175,465 ms	—	—	1	—	—
Prefill	1024K	1	Valid	5,772.0	—	177,589 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	30.7	30.7	66 ms	32.6 ms	1.0	0	922	30.0 s
Sustained decode	8K	2	Valid	61.1	30.1	283 ms	33.2 ms	2.0	0	1,830	30.0 s
Sustained decode	8K	4	Valid	122.3	30.3	147 ms	33.0 ms	4.0	0	3,668	30.0 s
Sustained decode	8K	8	Valid	238.6	29.3	565 ms	34.2 ms	8.0	0	7,152	30.0 s
Sustained decode	8K	16	Valid	478.1	29.9	630 ms	33.4 ms	16.0	0	14,336	30.0 s
Sustained decode	8K	32	Valid	952.2	29.7	332 ms	33.6 ms	32.0	0	28,544	30.0 s
Sustained decode	8K	64	Valid	1,886.5	28.8	528 ms	34.7 ms	64.0	0	56,576	30.0 s
Sustained decode	8K	128	Valid	3,595.4	27.4	848 ms	36.5 ms	128.0	0	107,904	30.0 s
Sustained decode	16K	1	Valid	30.6	30.6	107 ms	32.7 ms	1.0	0	919	30.0 s
Sustained decode	16K	2	Valid	61.3	30.6	124 ms	32.6 ms	2.0	0	1,838	30.0 s
Sustained decode	16K	4	Valid	122.4	30.6	196 ms	32.7 ms	4.0	0	3,672	30.0 s
Sustained decode	16K	8	Valid	238.8	29.8	239 ms	33.5 ms	8.0	0	7,160	30.0 s
Sustained decode	16K	16	Valid	477.8	29.9	307 ms	33.5 ms	16.0	0	14,336	30.0 s
Sustained decode	16K	32	Valid	944.4	29.5	457 ms	33.9 ms	32.0	0	28,319	30.0 s
Sustained decode	16K	64	Valid	1,872.2	29.2	716 ms	34.2 ms	64.0	0	56,128	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	30.5	30.5	133 ms	32.8 ms	1.0	0	914	30.0 s
Sustained decode	32K	2	Valid	60.5	30.3	192 ms	33.0 ms	2.0	0	1,814	30.0 s
Sustained decode	32K	4	Valid	122.0	30.4	251 ms	32.9 ms	4.0	0	3,660	30.0 s
Sustained decode	32K	8	Valid	238.7	29.8	314 ms	33.5 ms	8.0	0	7,160	30.0 s
Sustained decode	32K	16	Valid	479.3	30.0	483 ms	33.4 ms	16.0	0	14,368	30.0 s
Sustained decode	32K	32	Valid	954.7	29.8	759 ms	33.6 ms	32.0	0	28,640	30.0 s
Sustained decode	32K	64	Valid	1,872.7	29.2	1,293 ms	34.2 ms	64.0	0	56,192	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	30.5	30.5	183 ms	32.8 ms	1.0	0	913	30.0 s
Sustained decode	64K	2	Valid	60.8	30.4	262 ms	32.9 ms	2.0	0	1,822	30.0 s
Sustained decode	64K	4	Valid	121.1	30.3	351 ms	33.0 ms	4.0	0	3,632	30.0 s
Sustained decode	64K	8	Valid	239.8	30.0	525 ms	33.4 ms	8.0	0	7,192	30.0 s
Sustained decode	64K	16	Valid	477.5	29.9	800 ms	33.5 ms	16.0	0	14,319	30.0 s
Sustained decode	64K	32	Valid	945.2	29.5	1,425 ms	33.9 ms	32.0	0	28,357	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	30.6	30.7	337 ms	32.6 ms	1.0	0	918	30.0 s
Sustained decode	128K	2	Valid	61.3	30.7	433 ms	32.6 ms	2.0	0	1,838	30.0 s
Sustained decode	128K	4	Valid	122.7	30.7	652 ms	32.6 ms	4.0	0	3,680	30.0 s
Sustained decode	128K	8	Valid	239.2	30.0	1,075 ms	33.4 ms	8.0	0	7,168	30.0 s
Sustained decode	128K	16	Valid	476.8	29.8	1,924 ms	33.6 ms	16.0	0	14,304	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	30.6	30.7	619 ms	32.6 ms	1.0	0	918	30.0 s
Sustained decode	256K	2	Valid	61.1	30.6	810 ms	32.7 ms	2.0	0	1,832	30.0 s
Sustained decode	256K	4	Valid	121.9	30.6	1,211 ms	32.7 ms	4.0	0	3,656	30.0 s
Sustained decode	256K	8	Valid	238.1	29.8	1,876 ms	33.6 ms	8.0	0	7,144	30.0 s
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	30.7	30.8	1,193 ms	32.5 ms	1.0	0	920	30.0 s
Sustained decode	512K	2	Valid	61.0	30.6	1,795 ms	32.6 ms	2.0	0	1,828	30.0 s
Sustained decode	512K	4	Valid	121.7	30.5	2,604 ms	32.8 ms	4.0	0	3,648	30.0 s
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Requests queued	30.4	30.8	2,684 ms	32.4 ms	1.0	0	911	30.0 s
Sustained decode	1019K	2	Requests queued	61.0	30.7	3,924 ms	32.6 ms	2.0	0	1,830	30.0 s
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.