Speed test matrix
Qwen3.6-35B-A3B on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · BF166 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- qwen36-35b
- Revision
- 995ad96eacd98c81ed38be0c5b274b04031597b0
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36b-u-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36b-u-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/Qwen3.6-35B-A3B --host 0.0.0.0 --served-model-name qwen36-35b --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context11.9K687 ms TTFT
32K context11.4K2,836 ms TTFT
64K context9,7246,626 ms TTFT
128K context7,51617,125 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 88.783 ms TTFT | 161.6124 ms TTFT | 294.5152 ms TTFT | 512.5147 ms TTFT | 855.6Requests queued239 ms TTFT |
| 8K | 86.0104 ms TTFT | 156.7144 ms TTFT | 269.5175 ms TTFT | 429.9171 ms TTFT | 676.9172 ms TTFT |
| 32K | 80.6187 ms TTFT | 142.5236 ms TTFT | 238.7Requests queued357 ms TTFT | 361.7Requests queued378 ms TTFT | 532.7Requests queued393 ms TTFT |
| 64K | 74.7284 ms TTFT | 129.4346 ms TTFT | 209.2Requests queued450 ms TTFT | 306.0Requests queued642 ms TTFT | 448.0Requests queued653 ms TTFT |
| 128K | 64.0507 ms TTFT | 103.6Requests queued558 ms TTFT | 164.4Requests queued927 ms TTFT | 231.5Requests queued1,336 ms TTFT | Over capacity |
| 200K | 54.2834 ms TTFT | 87.2Requests queued1,059 ms TTFT | 131.5Requests queued1,485 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.9K | — | 687 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.4K | — | 2,836 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,724.0 | — | 6,626 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,516.0 | — | 17,125 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 88.7 | 91.1 | 83 ms | 11.0 ms | 1.0 | 12 | 2,660 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 161.6 | 84.0 | 124 ms | 11.9 ms | 2.0 | 22 | 4,846 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 294.5 | 76.6 | 152 ms | 13.0 ms | 4.0 | 40 | 8,832 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 512.5 | 66.4 | 147 ms | 15.1 ms | 8.0 | 64 | 15,368 | 30.0 s |
| Sustained decode | 0 | 16 | Requests queued | 855.6 | 55.9 | 239 ms | 17.9 ms | 15.0 | 112 | 25,664 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 86.0 | 88.7 | 104 ms | 11.3 ms | 1.0 | 11 | 2,578 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 156.7 | 81.5 | 144 ms | 12.3 ms | 1.9 | 20 | 4,700 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 269.5 | 70.4 | 175 ms | 14.2 ms | 3.9 | 36 | 8,084 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 429.9 | 56.1 | 171 ms | 17.8 ms | 7.8 | 56 | 12,896 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 676.9 | 43.4 | 172 ms | 23.0 ms | 15.8 | 80 | 20,307 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 80.6 | 85.1 | 187 ms | 11.7 ms | 0.9 | 10 | 2,417 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 142.5 | 75.7 | 236 ms | 13.2 ms | 1.9 | 18 | 4,274 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 238.7 | 64.6 | 357 ms | 15.5 ms | 3.8 | 32 | 7,160 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 361.7 | 49.0 | 378 ms | 20.4 ms | 7.7 | 48 | 10,808 | 29.9 s |
| Sustained decode | 32K | 16 | Requests queued | 532.7 | 35.1 | 393 ms | 28.5 ms | 15.5 | 64 | 15,968 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 74.7 | 81.0 | 284 ms | 12.3 ms | 0.9 | 9 | 2,239 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 129.4 | 70.0 | 346 ms | 14.3 ms | 2.0 | 16 | 3,882 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 209.2 | 58.3 | 450 ms | 17.1 ms | 3.8 | 28 | 6,254 | 29.9 s |
| Sustained decode | 64K | 8 | Requests queued | 306.0 | 42.9 | 642 ms | 23.3 ms | 7.6 | 40 | 9,176 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 448.0 | 29.5 | 653 ms | 33.9 ms | 15.3 | 48 | 13,440 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 64.0 | 73.7 | 507 ms | 13.6 ms | 0.9 | 8 | 1,920 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 103.6 | 62.6 | 558 ms | 16.0 ms | 1.7 | 14 | 3,104 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 164.4 | 47.4 | 927 ms | 21.1 ms | 3.3 | 20 | 4,928 | 30.0 s |
| Sustained decode | 128K | 8 | Requests queued | 231.5 | 33.9 | 1,336 ms | 29.5 ms | 7.0 | 32 | 6,918 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 54.2 | 66.9 | 834 ms | 14.9 ms | 0.8 | 7 | 1,627 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 87.2 | 52.8 | 1,059 ms | 19.0 ms | 1.7 | 10 | 2,616 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 131.5 | 39.1 | 1,485 ms | 25.6 ms | 3.5 | 16 | 3,944 | 30.0 s |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.