Speed test matrix

ZAYA1-8B-legacy on RTX PRO 6000 Blackwell Server Edition 96GB

vllm · 0.21.1rc1.dev409+g2ce2f26c2 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 11, 2026, 23:05 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: vllm
Version: 0.21.1rc1.dev409+g2ce2f26c2
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 2ce2f26c282d

Run

Served model: zaya
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-zyphra-vllm-pr-2ce2f26c282d/bin/vllm" serve Zyphra/ZAYA1-8B-legacy --served-model-name zaya --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser zaya_xml --max-model-len 131072 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-dtype float32 --dtype bfloat16

Prefill

prompt tok/s by context length

8K context42.5K193 ms TTFT

16K context49.6K328 ms TTFT

32K context40.8K792 ms TTFT

64K context30.2K2,138 ms TTFT

123K context19.6K6,327 ms TTFT

128K context19.3K6,692 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	105.3186 ms TTFT	166.2351 ms TTFT	312.9638 ms TTFT	549.31,023 ms TTFT	867.11,719 ms TTFT	1,307.73,119 ms TTFT	Over capacity	Over capacity
16K	102.3325 ms TTFT	157.1588 ms TTFT	286.41,045 ms TTFT	491.81,795 ms TTFT	731.03,313 ms TTFT	1,027.76,372 ms TTFT	Over capacity	Over capacity
32K	97.6789 ms TTFT	148.01,314 ms TTFT	258.72,305 ms TTFT	405.24,116 ms TTFT	556.67,778 ms TTFT	Over capacity	Over capacity	Over capacity
64K	89.12,143 ms TTFT	130.23,458 ms TTFT	211.46,014 ms TTFT	299.510,963 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
123K	76.36,332 ms TTFT	107.59,984 ms TTFT	156.117,120 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	42.5K	—	193 ms	—	—	1	—	—
Prefill	16K	1	Valid	49.6K	—	328 ms	—	—	1	—	—
Prefill	32K	1	Valid	40.8K	—	792 ms	—	—	1	—	—
Prefill	64K	1	Valid	30.2K	—	2,138 ms	—	—	1	—	—
Prefill	123K	1	Valid	19.6K	—	6,327 ms	—	—	1	—	—
Prefill	128K	1	Valid	19.3K	—	6,692 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	105.3	105.7	186 ms	9.5 ms	1.0	0	3,159	30.0 s
Sustained decode	8K	2	Valid	166.2	83.2	351 ms	12.0 ms	2.0	0	4,986	30.0 s
Sustained decode	8K	4	Valid	312.9	78.4	638 ms	12.8 ms	4.0	0	9,384	30.0 s
Sustained decode	8K	8	Valid	549.3	68.4	1,023 ms	14.6 ms	8.0	0	16,479	30.0 s
Sustained decode	8K	16	Valid	867.1	53.3	1,719 ms	18.8 ms	16.0	0	26,000	30.0 s
Sustained decode	8K	32	Valid	1,307.7	39.0	3,119 ms	25.6 ms	32.0	0	39,232	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	102.3	102.4	325 ms	9.8 ms	1.0	0	3,068	30.0 s
Sustained decode	16K	2	Valid	157.1	78.6	588 ms	12.7 ms	2.0	0	4,714	30.0 s
Sustained decode	16K	4	Valid	286.4	71.1	1,045 ms	14.1 ms	4.0	0	8,592	30.0 s
Sustained decode	16K	8	Valid	491.8	60.1	1,795 ms	16.6 ms	8.0	0	14,752	30.0 s
Sustained decode	16K	16	Valid	731.0	43.3	3,313 ms	23.1 ms	16.0	0	21,920	30.0 s
Sustained decode	16K	32	Valid	1,027.7	28.6	6,372 ms	35.0 ms	32.0	0	30,816	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	97.6	97.7	789 ms	10.2 ms	1.0	0	2,928	30.0 s
Sustained decode	32K	2	Valid	148.0	73.5	1,314 ms	13.6 ms	2.0	0	4,440	30.0 s
Sustained decode	32K	4	Valid	258.7	63.0	2,305 ms	15.9 ms	4.0	0	7,760	30.0 s
Sustained decode	32K	8	Valid	405.2	47.5	4,116 ms	21.1 ms	8.0	0	12,152	30.0 s
Sustained decode	32K	16	Valid	556.6	30.3	7,778 ms	33.0 ms	16.0	0	16,688	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	89.1	89.4	2,143 ms	11.2 ms	1.0	0	2,674	30.0 s
Sustained decode	64K	2	Valid	130.2	63.4	3,458 ms	15.8 ms	2.0	0	3,906	30.0 s
Sustained decode	64K	4	Valid	211.4	48.8	6,014 ms	20.5 ms	4.0	0	6,340	30.0 s
Sustained decode	64K	8	Valid	299.5	31.0	10,963 ms	32.3 ms	8.0	0	8,984	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	76.3	76.5	6,332 ms	13.1 ms	1.0	0	2,290	30.0 s
Sustained decode	123K	2	Valid	107.5	49.8	9,984 ms	20.2 ms	2.0	0	3,224	30.0 s
Sustained decode	123K	4	Valid	156.1	31.1	17,120 ms	32.3 ms	4.0	0	4,680	30.0 s
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.