Speed test matrix

Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150 on Tenstorrent P150 32GB

vllm · 0.1.dev14187+g03fa3af2e · BFP44 contexts (0–32K) × 5 concurrency levels (1–16) · prefill at 4K, 8K, 16K, 32K, 64K, 128K

by @Lottolabslmx v0.1.52 runMeasured Oct 14, 2026, 19:58 UTC

Hardware

Type: AI accelerator
GPU: Tenstorrent P150
VRAM: 32 GB
CPU: AMD Ryzen 9 9950X 16-Core Processor
OS: linux

Engine

Engine: vllm
Version: 0.1.dev14187+g03fa3af2e
Quantization: BFP4
Backend: tt-metal

Run

Served model: Qwen/Qwen3.6-35B-A3B
Revision: main
Benchmark tool: llm-inference-bench v0.7.7
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

python3 launch.py --mode batched --max-num-seqs 16 --device-ownership-confirmed  # Lottolabs/Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150 x6, image lottolabs/qwen36-35b-tt-p150:batched-x6

Prefill

prompt tok/s by context length

4K context1,8872,209 ms TTFT

8K context2,0164,063 ms TTFT

16K context1,9758,216 ms TTFT

32K context1,90016,994 ms TTFT

64K context1,79235,948 ms TTFT

128K context1,57081,976 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16
0	84.9258 ms TTFT	153.5325 ms TTFT	261.0736 ms TTFT	432.21,216 ms TTFT	626.22,091 ms TTFT
4K	85.4214 ms TTFT	152.4418 ms TTFT	255.4833 ms TTFT	413.11,427 ms TTFT	583.53,275 ms TTFT
16K	83.21,112 ms TTFT	144.11,682 ms TTFT	233.92,824 ms TTFT	364.95,173 ms TTFT	Over capacity
32K	79.61,176 ms TTFT	135.41,799 ms TTFT	212.13,025 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	4K	1	Valid	1,887.0	—	2,209 ms	—	—	1	—	—
Prefill	8K	1	Valid	2,016.0	—	4,063 ms	—	—	1	—	—
Prefill	16K	1	Valid	1,975.0	—	8,216 ms	—	—	1	—	—
Prefill	32K	1	Valid	1,900.0	—	16,994 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,792.0	—	35,948 ms	—	—	1	—	—
Prefill	128K	1	Valid	1,570.0	—	81,976 ms	—	—	1	—	—
Sustained decode	0	1	Valid	84.9	84.9	258 ms	11.8 ms	1.0	0	2,547	30.0 s
Sustained decode	0	2	Valid	153.5	76.8	325 ms	13.0 ms	2.0	0	4,606	30.0 s
Sustained decode	0	4	Valid	261.0	65.5	736 ms	15.3 ms	4.0	0	7,828	30.0 s
Sustained decode	0	8	Valid	432.2	54.3	1,216 ms	18.4 ms	8.0	0	12,960	30.0 s
Sustained decode	0	16	Valid	626.2	39.7	2,091 ms	25.2 ms	16.0	0	18,784	30.0 s
Sustained decode	4K	1	Valid	85.4	85.3	214 ms	11.7 ms	1.0	0	2,561	30.0 s
Sustained decode	4K	2	Valid	152.4	76.6	418 ms	13.0 ms	2.0	0	4,572	30.0 s
Sustained decode	4K	4	Valid	255.4	64.2	833 ms	15.6 ms	4.0	0	7,660	30.0 s
Sustained decode	4K	8	Valid	413.1	51.9	1,427 ms	19.3 ms	8.0	0	12,392	30.0 s
Sustained decode	4K	16	Valid	583.5	37.2	3,275 ms	26.9 ms	16.0	0	17,504	30.0 s
Sustained decode	16K	1	Valid	83.2	83.3	1,112 ms	12.0 ms	1.0	0	2,496	30.0 s
Sustained decode	16K	2	Valid	144.1	71.0	1,682 ms	14.1 ms	2.0	0	4,323	30.0 s
Sustained decode	16K	4	Valid	233.9	55.7	2,824 ms	17.9 ms	4.0	0	7,016	30.0 s
Sustained decode	16K	8	Valid	364.9	41.0	5,173 ms	24.4 ms	8.0	0	10,944	30.0 s
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	79.6	79.6	1,176 ms	12.6 ms	1.0	0	2,386	30.0 s
Sustained decode	32K	2	Valid	135.4	66.6	1,799 ms	15.0 ms	2.0	0	4,060	30.0 s
Sustained decode	32K	4	Valid	212.1	50.5	3,025 ms	19.8 ms	4.0	0	6,360	30.0 s
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.