Speed test matrix
Qwen3.6-35B-A3B on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · FP86 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- FP8
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- q36x
- Revision
- 995ad96eacd98c81ed38be0c5b274b04031597b0
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36x-if8-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36x-if8-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-35B-A3B --host 0.0.0.0 --served-model-name q36x --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context12.8K642 ms TTFT
32K context12K2,686 ms TTFT
64K context10.2K6,347 ms TTFT
128K context7,78816,525 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 99.977 ms TTFT | 177.6115 ms TTFT | 338.0140 ms TTFT | 613.4131 ms TTFT | 1,030.5160 ms TTFT |
| 8K | 97.0107 ms TTFT | 172.0Requests queued157 ms TTFT | 324.6181 ms TTFT | 559.7207 ms TTFT | 911.2276 ms TTFT |
| 32K | 90.5171 ms TTFT | 155.8236 ms TTFT | 279.2303 ms TTFT | 451.8Requests queued492 ms TTFT | 678.6Requests queued613 ms TTFT |
| 64K | 83.7266 ms TTFT | 139.0Requests queued360 ms TTFT | 236.9Requests queued489 ms TTFT | 360.7Requests queued696 ms TTFT | 510.4Requests queued731 ms TTFT |
| 128K | 69.7513 ms TTFT | 111.6Requests queued666 ms TTFT | 182.0Requests queued947 ms TTFT | 248.4Requests queued1,435 ms TTFT | Over capacity |
| 200K | 59.9841 ms TTFT | 90.7Requests queued863 ms TTFT | 143.1Requests queued1,478 ms TTFT | 182.8Requests queued2,071 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.8K | — | 642 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12K | — | 2,686 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.2K | — | 6,347 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,788.0 | — | 16,525 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 99.9 | 102.4 | 77 ms | 9.8 ms | 1.0 | 13 | 2,998 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 177.6 | 92.2 | 115 ms | 10.8 ms | 2.0 | 24 | 5,326 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 338.0 | 88.2 | 140 ms | 11.3 ms | 4.0 | 44 | 10,136 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 613.4 | 79.5 | 131 ms | 12.6 ms | 7.8 | 80 | 18,400 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,030.5 | 66.8 | 160 ms | 15.0 ms | 15.5 | 128 | 30,912 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 97.0 | 100.7 | 107 ms | 9.9 ms | 0.9 | 12 | 2,910 | 30.0 s |
| Sustained decode | 8K | 2 | Requests queued | 172.0 | 90.5 | 157 ms | 11.1 ms | 1.9 | 22 | 5,160 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 324.6 | 85.6 | 181 ms | 11.7 ms | 3.9 | 40 | 9,736 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 559.7 | 73.6 | 207 ms | 13.6 ms | 7.9 | 72 | 16,785 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 911.2 | 60.5 | 276 ms | 16.5 ms | 15.5 | 128 | 27,325 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 90.5 | 95.9 | 171 ms | 10.4 ms | 1.0 | 11 | 2,714 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 155.8 | 83.2 | 236 ms | 12.0 ms | 2.0 | 20 | 4,674 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 279.2 | 75.6 | 303 ms | 13.2 ms | 3.9 | 36 | 8,376 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 451.8 | 63.5 | 492 ms | 15.7 ms | 7.4 | 64 | 13,552 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 678.6 | 47.0 | 613 ms | 21.3 ms | 15.0 | 80 | 20,350 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 83.7 | 90.7 | 266 ms | 11.0 ms | 0.8 | 10 | 2,510 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 139.0 | 76.5 | 360 ms | 13.1 ms | 1.9 | 18 | 4,170 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 236.9 | 66.0 | 489 ms | 15.2 ms | 3.7 | 28 | 7,108 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 360.7 | 51.5 | 696 ms | 19.4 ms | 7.2 | 48 | 10,819 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 510.4 | 35.4 | 731 ms | 28.2 ms | 14.9 | 64 | 15,306 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 69.7 | 81.9 | 513 ms | 12.2 ms | 0.9 | 9 | 2,091 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 111.6 | 65.6 | 666 ms | 15.3 ms | 1.7 | 14 | 3,346 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 182.0 | 53.5 | 947 ms | 18.7 ms | 3.6 | 24 | 5,392 | 29.6 s |
| Sustained decode | 128K | 8 | Requests queued | 248.4 | 37.3 | 1,435 ms | 26.8 ms | 6.8 | 32 | 7,448 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 59.9 | 73.7 | 841 ms | 13.6 ms | 0.8 | 8 | 1,792 | 29.9 s |
| Sustained decode | 200K | 2 | Requests queued | 90.7 | 58.9 | 863 ms | 17.0 ms | 1.8 | 12 | 2,697 | 29.7 s |
| Sustained decode | 200K | 4 | Requests queued | 143.1 | 43.5 | 1,478 ms | 23.0 ms | 3.5 | 16 | 4,292 | 30.0 s |
| Sustained decode | 200K | 8 | Requests queued | 182.8 | 28.5 | 2,071 ms | 35.1 ms | 6.9 | 24 | 5,479 | 30.0 s |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.