Speed test matrix

ZAYA1-8B on H200 NVL 141GB

vllm · 0.1.dev18387+gb1b99a08b · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 8, 2026, 15:22 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.1.dev18387+gb1b99a08b
Quantization: BF16
Backend: cuda
Repository: github.com/Zyphra/vllm
Build: cuda-13.2
Commit: b1b99a08b20f

Run

Served model: zaya
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-zyphra-vllm/bin/vllm" serve Zyphra/ZAYA1-8B --served-model-name zaya --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser zaya_xml --max-model-len 131072 --gpu-memory-utilization 0.875161 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-dtype float32 --dtype bfloat16

Prefill

prompt tok/s by context length

8K context60.9K135 ms TTFT

16K context57.2K284 ms TTFT

32K context53.1K609 ms TTFT

64K context45.6K1,412 ms TTFT

123K context33K3,753 ms TTFT

128K context32.4K3,979 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	92.8127 ms TTFT	173.5227 ms TTFT	323.1421 ms TTFT	600.6710 ms TTFT	1,055.01,186 ms TTFT	1,839.52,074 ms TTFT	3,042.73,937 ms TTFT	4,637.77,580 ms TTFT
16K	91.7262 ms TTFT	170.3437 ms TTFT	313.7748 ms TTFT	571.51,220 ms TTFT	978.82,140 ms TTFT	1,616.94,037 ms TTFT	2,468.27,792 ms TTFT	3,454.915,514 ms TTFT
32K	89.6569 ms TTFT	164.9897 ms TTFT	297.61,509 ms TTFT	527.12,545 ms TTFT	851.24,671 ms TTFT	1,292.98,925 ms TTFT	1,802.117,598 ms TTFT	Over capacity
64K	86.31,368 ms TTFT	156.22,109 ms TTFT	271.43,492 ms TTFT	453.36,148 ms TTFT	674.811,512 ms TTFT	929.622,715 ms TTFT	Over capacity	Over capacity
123K	80.83,731 ms TTFT	142.85,672 ms TTFT	233.69,374 ms TTFT	359.316,714 ms TTFT	487.231,973 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	60.9K	—	135 ms	—	—	1	—	—
Prefill	16K	1	Valid	57.2K	—	284 ms	—	—	1	—	—
Prefill	32K	1	Valid	53.1K	—	609 ms	—	—	1	—	—
Prefill	64K	1	Valid	45.6K	—	1,412 ms	—	—	1	—	—
Prefill	123K	1	Valid	33K	—	3,753 ms	—	—	1	—	—
Prefill	128K	1	Valid	32.4K	—	3,979 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	92.8	92.8	127 ms	10.8 ms	1.0	0	2,783	30.0 s
Sustained decode	8K	2	Valid	173.5	86.9	227 ms	11.5 ms	2.0	0	5,206	30.0 s
Sustained decode	8K	4	Valid	323.1	80.8	421 ms	12.4 ms	4.0	0	9,692	30.0 s
Sustained decode	8K	8	Valid	600.6	74.9	710 ms	13.4 ms	8.0	0	18,016	30.0 s
Sustained decode	8K	16	Valid	1,055.0	65.1	1,186 ms	15.4 ms	16.0	0	31,648	30.0 s
Sustained decode	8K	32	Valid	1,839.5	55.7	2,074 ms	18.0 ms	32.0	0	55,168	30.0 s
Sustained decode	8K	64	Valid	3,042.7	44.1	3,937 ms	22.7 ms	64.0	0	91,264	30.0 s
Sustained decode	8K	128	Valid	4,637.7	31.3	7,580 ms	31.9 ms	128.0	0	139,137	30.0 s
Sustained decode	16K	1	Valid	91.7	91.8	262 ms	10.9 ms	1.0	0	2,752	30.0 s
Sustained decode	16K	2	Valid	170.3	85.2	437 ms	11.7 ms	2.0	0	5,107	30.0 s
Sustained decode	16K	4	Valid	313.7	78.1	748 ms	12.8 ms	4.0	0	9,416	30.0 s
Sustained decode	16K	8	Valid	571.5	70.4	1,220 ms	14.2 ms	8.0	0	17,144	30.0 s
Sustained decode	16K	16	Valid	978.8	59.0	2,140 ms	17.0 ms	16.0	0	29,360	30.0 s
Sustained decode	16K	32	Valid	1,616.9	46.7	4,037 ms	21.4 ms	32.0	0	48,512	30.0 s
Sustained decode	16K	64	Valid	2,468.2	33.2	7,792 ms	30.1 ms	64.0	0	74,049	30.0 s
Sustained decode	16K	128	Valid	3,454.9	20.8	15,514 ms	48.1 ms	128.0	0	103,680	30.0 s
Sustained decode	32K	1	Valid	89.6	89.8	569 ms	11.1 ms	1.0	0	2,690	30.0 s
Sustained decode	32K	2	Valid	164.9	82.2	897 ms	12.2 ms	2.0	0	4,950	30.0 s
Sustained decode	32K	4	Valid	297.6	73.3	1,509 ms	13.6 ms	4.0	0	8,928	30.0 s
Sustained decode	32K	8	Valid	527.1	63.3	2,545 ms	15.8 ms	8.0	0	15,816	30.0 s
Sustained decode	32K	16	Valid	851.2	48.7	4,671 ms	20.5 ms	16.0	0	25,537	30.0 s
Sustained decode	32K	32	Valid	1,292.9	34.1	8,925 ms	29.3 ms	32.0	0	38,784	30.0 s
Sustained decode	32K	64	Valid	1,802.1	20.9	17,598 ms	48.0 ms	64.0	0	54,016	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	86.3	86.5	1,368 ms	11.6 ms	1.0	0	2,588	30.0 s
Sustained decode	64K	2	Valid	156.2	77.1	2,109 ms	13.0 ms	2.0	0	4,686	30.0 s
Sustained decode	64K	4	Valid	271.4	64.9	3,492 ms	15.4 ms	4.0	0	8,138	30.0 s
Sustained decode	64K	8	Valid	453.3	50.8	6,148 ms	19.7 ms	8.0	0	13,598	30.0 s
Sustained decode	64K	16	Valid	674.8	33.6	11,512 ms	29.8 ms	16.0	0	20,240	30.0 s
Sustained decode	64K	32	Valid	929.6	19.7	22,715 ms	50.9 ms	32.0	0	27,872	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	80.8	81.0	3,731 ms	12.3 ms	1.0	0	2,424	30.0 s
Sustained decode	123K	2	Valid	142.8	68.6	5,672 ms	14.6 ms	2.0	0	4,286	30.0 s
Sustained decode	123K	4	Valid	233.6	51.2	9,374 ms	19.6 ms	4.0	0	7,008	30.0 s
Sustained decode	123K	8	Valid	359.3	33.5	16,714 ms	29.9 ms	8.0	0	10,784	30.0 s
Sustained decode	123K	16	Valid	487.2	18.1	31,973 ms	55.3 ms	16.0	0	14,608	30.0 s
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.