Speed test matrix

ZAYA1-8B-legacy on H200 NVL 141GB

vllm · 0.1.dev17064+g8df704c52 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 8, 2026, 14:30 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.1.dev17064+g8df704c52
Quantization: BF16
Backend: cuda
Repository: github.com/Zyphra/vllm
Build: cuda-13.2
Commit: 8df704c52588

Run

Served model: zaya
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-zyphra-legacy-vllm/bin/vllm" serve Zyphra/ZAYA1-8B-legacy --served-model-name zaya --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser zaya_xml --max-model-len 131072 --gpu-memory-utilization 0.874984 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-dtype float32 --dtype bfloat16

Prefill

prompt tok/s by context length

8K context55K149 ms TTFT

16K context50.7K320 ms TTFT

32K context51.8K624 ms TTFT

64K context44K1,466 ms TTFT

123K context32K3,870 ms TTFT

128K context30.9K4,164 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	86.7140 ms TTFT	166.5269 ms TTFT	316.9561 ms TTFT	589.5891 ms TTFT	1,055.41,429 ms TTFT	1,825.62,541 ms TTFT	3,057.54,484 ms TTFT	Over capacity
16K	85.8282 ms TTFT	164.1493 ms TTFT	307.4854 ms TTFT	562.81,430 ms TTFT	979.42,316 ms TTFT	1,608.84,922 ms TTFT	2,490.28,325 ms TTFT	Over capacity
32K	84.1591 ms TTFT	158.9988 ms TTFT	291.91,692 ms TTFT	520.12,967 ms TTFT	855.84,901 ms TTFT	1,295.410,737 ms TTFT	Over capacity	Over capacity
64K	81.21,428 ms TTFT	150.42,294 ms TTFT	267.83,928 ms TTFT	450.46,981 ms TTFT	679.912,901 ms TTFT	Over capacity	Over capacity	Over capacity
123K	76.33,858 ms TTFT	138.76,037 ms TTFT	231.610,270 ms TTFT	358.718,694 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	55K	—	149 ms	—	—	1	—	—
Prefill	16K	1	Valid	50.7K	—	320 ms	—	—	1	—	—
Prefill	32K	1	Valid	51.8K	—	624 ms	—	—	1	—	—
Prefill	64K	1	Valid	44K	—	1,466 ms	—	—	1	—	—
Prefill	123K	1	Valid	32K	—	3,870 ms	—	—	1	—	—
Prefill	128K	1	Valid	30.9K	—	4,164 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	86.7	86.7	140 ms	11.5 ms	1.0	0	2,600	30.0 s
Sustained decode	8K	2	Valid	166.5	83.3	269 ms	12.0 ms	2.0	0	4,996	30.0 s
Sustained decode	8K	4	Valid	316.9	79.3	561 ms	12.6 ms	4.0	0	9,508	30.0 s
Sustained decode	8K	8	Valid	589.5	73.4	891 ms	13.6 ms	8.0	0	17,680	30.0 s
Sustained decode	8K	16	Valid	1,055.4	65.1	1,429 ms	15.4 ms	16.0	0	31,681	30.0 s
Sustained decode	8K	32	Valid	1,825.6	55.0	2,541 ms	18.2 ms	32.0	0	54,752	30.0 s
Sustained decode	8K	64	Valid	3,057.5	44.1	4,484 ms	22.7 ms	64.0	0	91,712	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	85.8	85.8	282 ms	11.6 ms	1.0	0	2,573	30.0 s
Sustained decode	16K	2	Valid	164.1	82.0	493 ms	12.2 ms	2.0	0	4,922	30.0 s
Sustained decode	16K	4	Valid	307.4	76.5	854 ms	13.1 ms	4.0	0	9,224	30.0 s
Sustained decode	16K	8	Valid	562.8	69.2	1,430 ms	14.4 ms	8.0	0	16,880	30.0 s
Sustained decode	16K	16	Valid	979.4	58.9	2,316 ms	17.0 ms	16.0	0	29,376	30.0 s
Sustained decode	16K	32	Valid	1,608.8	46.2	4,922 ms	21.6 ms	32.0	0	48,256	30.0 s
Sustained decode	16K	64	Valid	2,490.2	33.1	8,325 ms	30.2 ms	64.0	0	74,688	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	84.1	84.2	591 ms	11.9 ms	1.0	0	2,523	30.0 s
Sustained decode	32K	2	Valid	158.9	79.1	988 ms	12.6 ms	2.0	0	4,766	30.0 s
Sustained decode	32K	4	Valid	291.9	71.9	1,692 ms	13.9 ms	4.0	0	8,761	30.0 s
Sustained decode	32K	8	Valid	520.1	62.4	2,967 ms	16.0 ms	8.0	0	15,600	30.0 s
Sustained decode	32K	16	Valid	855.8	48.8	4,901 ms	20.5 ms	16.0	0	25,664	30.0 s
Sustained decode	32K	32	Valid	1,295.4	33.8	10,737 ms	29.6 ms	32.0	0	38,848	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	81.2	81.3	1,428 ms	12.3 ms	1.0	0	2,434	30.0 s
Sustained decode	64K	2	Valid	150.4	74.2	2,294 ms	13.5 ms	2.0	0	4,512	30.0 s
Sustained decode	64K	4	Valid	267.8	64.0	3,928 ms	15.6 ms	4.0	0	8,032	30.0 s
Sustained decode	64K	8	Valid	450.4	50.3	6,981 ms	19.9 ms	8.0	0	13,512	30.0 s
Sustained decode	64K	16	Valid	679.9	33.5	12,901 ms	29.8 ms	16.0	0	20,384	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	76.3	76.5	3,858 ms	13.1 ms	1.0	0	2,289	30.0 s
Sustained decode	123K	2	Valid	138.7	66.6	6,037 ms	15.0 ms	2.0	0	4,163	30.0 s
Sustained decode	123K	4	Valid	231.6	50.7	10,270 ms	19.8 ms	4.0	0	6,948	30.0 s
Sustained decode	123K	8	Valid	358.7	33.2	18,694 ms	30.2 ms	8.0	0	10,760	30.0 s
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.