Speed test matrix
Qwen3.6-35B-A3B on Intel Arc Pro B65 32GB
vllm · 0.31.0+xpu.llmscaler · INT46 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- VRAM
- 32 GB
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- INT4
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- qwen36-35b
- Revision
- 995ad96eacd98c81ed38be0c5b274b04031597b0
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36s2-i-tp1-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36s2-i-tp1-ctx/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-35B-A3B --host 0.0.0.0 --served-model-name qwen36-35b --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context7,1601,144 ms TTFT
32K context5,6615,706 ms TTFT
64K context4,30514,966 ms TTFT
128K context2,86544,931 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 80.381 ms TTFT | 150.6149 ms TTFT | 262.4Requests queued144 ms TTFT | 437.5Requests queued172 ms TTFT | 469.4214 ms TTFT |
| 8K | 77.6122 ms TTFT | 143.6181 ms TTFT | 246.4Requests queued235 ms TTFT | 398.6265 ms TTFT | 428.2401 ms TTFT |
| 32K | 69.0212 ms TTFT | 126.8280 ms TTFT | 207.9399 ms TTFT | 308.9519 ms TTFT | Over capacity |
| 64K | 59.8446 ms TTFT | 104.3Requests queued459 ms TTFT | 158.5Requests queued895 ms TTFT | Over capacity | Over capacity |
| 128K | 47.6766 ms TTFT | 82.0Requests queued967 ms TTFT | Over capacity | Over capacity | Over capacity |
| 200K | 38.21,388 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,160.0 | — | 1,144 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,661.0 | — | 5,706 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,305.0 | — | 14,966 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,865.0 | — | 44,931 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 80.3 | 82.2 | 81 ms | 12.2 ms | 0.9 | 11 | 2,409 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 150.6 | 78.4 | 149 ms | 12.8 ms | 1.9 | 20 | 4,516 | 30.0 s |
| Sustained decode | 0 | 4 | Requests queued | 262.4 | 68.2 | 144 ms | 14.7 ms | 3.8 | 36 | 7,872 | 30.0 s |
| Sustained decode | 0 | 8 | Requests queued | 437.5 | 56.5 | 172 ms | 17.7 ms | 7.5 | 56 | 13,120 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 469.4 | 30.3 | 214 ms | 33.0 ms | 16.0 | 64 | 14,072 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 77.6 | 80.3 | 122 ms | 12.5 ms | 0.9 | 10 | 2,328 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 143.6 | 75.6 | 181 ms | 13.2 ms | 2.0 | 18 | 4,306 | 30.0 s |
| Sustained decode | 8K | 4 | Requests queued | 246.4 | 64.8 | 235 ms | 15.4 ms | 3.7 | 32 | 7,392 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 398.6 | 52.7 | 265 ms | 19.0 ms | 7.7 | 48 | 11,958 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 428.2 | 27.9 | 401 ms | 35.8 ms | 15.2 | 48 | 12,833 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 69.0 | 73.5 | 212 ms | 13.6 ms | 1.0 | 9 | 2,070 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 126.8 | 68.1 | 280 ms | 14.7 ms | 2.0 | 16 | 3,802 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 207.9 | 55.9 | 399 ms | 17.9 ms | 3.8 | 26 | 6,235 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 308.9 | 41.8 | 519 ms | 23.9 ms | 7.6 | 40 | 9,264 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 59.8 | 66.5 | 446 ms | 15.0 ms | 0.9 | 7 | 1,793 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 104.3 | 58.1 | 459 ms | 17.2 ms | 1.8 | 14 | 3,114 | 29.8 s |
| Sustained decode | 64K | 4 | Requests queued | 158.5 | 46.6 | 895 ms | 21.5 ms | 3.7 | 20 | 4,756 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 47.6 | 55.9 | 766 ms | 17.9 ms | 0.8 | 6 | 1,427 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 82.0 | 48.7 | 967 ms | 20.6 ms | 1.8 | 10 | 2,458 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 38.2 | 46.9 | 1,388 ms | 21.3 ms | 0.8 | 5 | 1,131 | 29.6 s |
| Sustained decode | 200K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.