Speed test matrix
Qwen3.6-27B on Intel Arc Pro B65 32GB
vllm · 0.31.0+xpu.llmscaler · INT45 contexts (0–128K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- VRAM
- 32 GB
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- INT4
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- q36d
- Revision
- 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36d-iq4-tp1-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36d-iq4-tp1-ctx/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-27B --host 0.0.0.0 --served-model-name q36d --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 132096 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context1,3715,973 ms TTFT
32K context1,20926,715 ms TTFT
64K context1,04461,708 ms TTFT
128K context808.0159,370 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 20.3172 ms TTFT | 60.2210 ms TTFT | 116.3349 ms TTFT | 209.8550 ms TTFT | 162.2992 ms TTFT |
| 8K | 19.7258 ms TTFT | 57.1351 ms TTFT | 105.9523 ms TTFT | 183.5Requests queued817 ms TTFT | Over capacity |
| 32K | 18.3450 ms TTFT | 49.0640 ms TTFT | 87.71,018 ms TTFT | Over capacity | Over capacity |
| 64K | 16.6785 ms TTFT | 41.71,096 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 14.8934 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,371.0 | — | 5,973 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,209.0 | — | 26,715 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,044.0 | — | 61,708 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 808.0 | — | 159,370 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 20.3 | 20.4 | 172 ms | 49.0 ms | 1.0 | 2 | 607 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 60.2 | 31.3 | 210 ms | 32.0 ms | 1.9 | 8 | 1,803 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 116.3 | 30.0 | 349 ms | 33.3 ms | 4.0 | 13 | 3,488 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 209.8 | 27.6 | 550 ms | 36.2 ms | 8.0 | 24 | 6,288 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 162.2 | 10.4 | 992 ms | 96.0 ms | 15.7 | 16 | 4,856 | 29.9 s |
| Sustained decode | 8K | 1 | Valid | 19.7 | 20.0 | 258 ms | 50.0 ms | 1.0 | 2 | 591 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 57.1 | 29.4 | 351 ms | 34.0 ms | 2.0 | 6 | 1,712 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 105.9 | 27.8 | 523 ms | 36.0 ms | 4.0 | 12 | 3,175 | 30.0 s |
| Sustained decode | 8K | 8 | Requests queued | 183.5 | 24.3 | 817 ms | 41.2 ms | 7.4 | 24 | 5,453 | 29.7 s |
| Sustained decode | 8K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 18.3 | 18.8 | 450 ms | 53.2 ms | 1.0 | 2 | 549 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 49.0 | 26.0 | 640 ms | 38.4 ms | 1.9 | 6 | 1,468 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 87.7 | 23.3 | 1,018 ms | 42.9 ms | 3.9 | 8 | 2,628 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 16.6 | 17.4 | 785 ms | 57.3 ms | 0.9 | 2 | 498 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 41.7 | 22.3 | 1,096 ms | 44.9 ms | 1.9 | 4 | 1,250 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 14.8 | 15.2 | 934 ms | 65.7 ms | 1.0 | 1 | 443 | 30.0 s |
| Sustained decode | 128K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.