Speed test matrix

Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150 on Tenstorrent P150 32GB

vllm · 0.1.dev14187+g03fa3af2e · BFP46 contexts (0–128K) × 5 concurrency levels (1–16) · prefill at 4K, 8K, 16K, 32K, 64K, 128K

by @Lottolabslmx v0.1.52 runMeasured Oct 14, 2026, 21:21 UTC

Hardware

Type: AI accelerator
GPU: Tenstorrent P150
VRAM: 32 GB
CPU: AMD Ryzen 9 9950X 16-Core Processor
OS: linux

Engine

Engine: vllm
Version: 0.1.dev14187+g03fa3af2e
Quantization: BFP4
Backend: tt-metal

Run

Served model: Qwen/Qwen3.6-35B-A3B
Revision: main
Benchmark tool: llm-inference-bench v0.7.7
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

python3 launch.py --mode batched --max-num-seqs 16 --device-ownership-confirmed  # Lottolabs/Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150 x6, image lottolabs/qwen36-35b-tt-p150:batched-x6

Prefill

prompt tok/s by context length

4K context1,8832,213 ms TTFT

8K context2,0144,067 ms TTFT

16K context1,9758,215 ms TTFT

32K context1,90516,957 ms TTFT

64K context1,78936,011 ms TTFT

128K context1,56982,032 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16
0	84.5249 ms TTFT	153.7372 ms TTFT	261.4746 ms TTFT	435.51,212 ms TTFT	628.52,059 ms TTFT
4K	85.2217 ms TTFT	151.9352 ms TTFT	253.4609 ms TTFT	412.81,603 ms TTFT	586.73,158 ms TTFT
16K	82.51,111 ms TTFT	144.21,684 ms TTFT	234.82,832 ms TTFT	361.55,168 ms TTFT	Over capacity
32K	79.81,179 ms TTFT	135.31,803 ms TTFT	210.83,037 ms TTFT	Over capacity	Over capacity
64K	74.4831 ms TTFT	121.91,665 ms TTFT	Over capacity	Over capacity	Over capacity
128K	65.91,860 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	4K	1	Valid	1,883.0	—	2,213 ms	—	—	1	—	—
Prefill	8K	1	Valid	2,014.0	—	4,067 ms	—	—	1	—	—
Prefill	16K	1	Valid	1,975.0	—	8,215 ms	—	—	1	—	—
Prefill	32K	1	Valid	1,905.0	—	16,957 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,789.0	—	36,011 ms	—	—	1	—	—
Prefill	128K	1	Valid	1,569.0	—	82,032 ms	—	—	1	—	—
Sustained decode	0	1	Valid	84.5	84.4	249 ms	11.8 ms	1.0	0	2,533	30.0 s
Sustained decode	0	2	Valid	153.7	76.6	372 ms	13.1 ms	2.0	0	4,610	30.0 s
Sustained decode	0	4	Valid	261.4	65.5	746 ms	15.3 ms	4.0	0	7,840	30.0 s
Sustained decode	0	8	Valid	435.5	54.8	1,212 ms	18.3 ms	8.0	0	13,064	30.0 s
Sustained decode	0	16	Valid	628.5	39.8	2,059 ms	25.1 ms	16.0	0	18,848	30.0 s
Sustained decode	4K	1	Valid	85.2	85.2	217 ms	11.7 ms	1.0	0	2,557	30.0 s
Sustained decode	4K	2	Valid	151.9	75.6	352 ms	13.2 ms	2.0	0	4,556	30.0 s
Sustained decode	4K	4	Valid	253.4	63.3	609 ms	15.8 ms	4.0	0	7,600	30.0 s
Sustained decode	4K	8	Valid	412.8	52.2	1,603 ms	19.2 ms	8.0	0	12,376	30.0 s
Sustained decode	4K	16	Valid	586.7	37.3	3,158 ms	26.8 ms	16.0	0	17,600	30.0 s
Sustained decode	16K	1	Valid	82.5	82.5	1,111 ms	12.1 ms	1.0	0	2,474	30.0 s
Sustained decode	16K	2	Valid	144.2	71.0	1,684 ms	14.1 ms	2.0	0	4,324	30.0 s
Sustained decode	16K	4	Valid	234.8	56.0	2,832 ms	17.9 ms	4.0	0	7,040	30.0 s
Sustained decode	16K	8	Valid	361.5	40.6	5,168 ms	24.6 ms	8.0	0	10,840	30.0 s
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	79.8	79.9	1,179 ms	12.5 ms	1.0	0	2,393	30.0 s
Sustained decode	32K	2	Valid	135.3	66.6	1,803 ms	15.0 ms	2.0	0	4,058	30.0 s
Sustained decode	32K	4	Valid	210.8	50.2	3,037 ms	19.9 ms	4.0	0	6,324	30.0 s
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	74.4	74.5	831 ms	13.4 ms	1.0	0	2,232	30.0 s
Sustained decode	64K	2	Valid	121.9	61.1	1,665 ms	16.4 ms	2.0	0	3,656	30.0 s
Sustained decode	64K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	65.9	66.0	1,860 ms	15.2 ms	1.0	0	1,976	30.0 s
Sustained decode	128K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.