Speed test matrix
Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150 on Tenstorrent P150 32GB
vllm · 0.1.dev14187+g03fa3af2e · BFP46 contexts (0–128K) × 1 concurrency level (1) · prefill at 4K, 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- AI accelerator
- GPU
- Tenstorrent P150
- VRAM
- 32 GB
- CPU
- AMD Ryzen 9 9950X 16-Core Processor
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev14187+g03fa3af2e
- Quantization
- BFP4
- Backend
- tt-metal
Run
- Served model
- Qwen/Qwen3.6-35B-A3B
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
python3 launch.py --device-ownership-confirmed # Lottolabs/Qwen3.6-35B-A3B-TT-Mixed-BFP4-BFP8-P150 x6, image lottolabs/qwen36-35b-tt-p150:single-x6
Prefill
prompt tok/s by context length4K context1,9372,150 ms TTFT
8K context2,0553,985 ms TTFT
16K context2,0297,993 ms TTFT
32K context1,95416,529 ms TTFT
64K context1,83535,102 ms TTFT
128K context1,60380,278 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 |
|---|---|
| 0 | 130.4239 ms TTFT |
| 4K | 126.0206 ms TTFT |
| 16K | 123.91,105 ms TTFT |
| 32K | 113.11,170 ms TTFT |
| 64K | 100.9816 ms TTFT |
| 128K | 78.31,848 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 4K | 1 | Valid | 1,937.0 | — | 2,150 ms | — | — | 1 | — | — |
| Prefill | 8K | 1 | Valid | 2,055.0 | — | 3,985 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,029.0 | — | 7,993 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,954.0 | — | 16,529 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,835.0 | — | 35,102 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,603.0 | — | 80,278 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 130.4 | 131.0 | 239 ms | 7.6 ms | 1.0 | 0 | 3,909 | 30.0 s |
| Sustained decode | 4K | 1 | Valid | 126.0 | 126.6 | 206 ms | 7.9 ms | 1.0 | 0 | 3,780 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 123.9 | 123.9 | 1,105 ms | 8.1 ms | 1.0 | 0 | 3,716 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 113.1 | 113.5 | 1,170 ms | 8.8 ms | 1.0 | 0 | 3,394 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 100.9 | 101.2 | 816 ms | 9.9 ms | 1.0 | 0 | 3,026 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 78.3 | 78.8 | 1,848 ms | 12.7 ms | 1.0 | 0 | 2,350 | 30.0 s |