Speed test matrix
Qwen3.6-27B on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · INT46 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- INT4
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- q36d
- Revision
- 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36d-iq4-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36d-iq4-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-27B --host 0.0.0.0 --served-model-name q36d --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context3,7522,183 ms TTFT
32K context3,4479,369 ms TTFT
64K context3,09020,851 ms TTFT
128K context2,50551,379 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 50.586 ms TTFT | 129.9123 ms TTFT | 255.6167 ms TTFT | 471.7238 ms TTFT | 466.0366 ms TTFT |
| 8K | 49.5111 ms TTFT | 126.0155 ms TTFT | 240.1253 ms TTFT | 416.4321 ms TTFT | 424.5629 ms TTFT |
| 32K | 46.6218 ms TTFT | 110.8297 ms TTFT | 209.2Requests queued413 ms TTFT | 347.3Requests queued504 ms TTFT | 366.0573 ms TTFT |
| 64K | 43.4396 ms TTFT | 95.6Requests queued527 ms TTFT | 163.1Requests queued883 ms TTFT | 245.2Requests queued1,843 ms TTFT | 254.6Requests queued2,203 ms TTFT |
| 128K | 38.2634 ms TTFT | 78.1Requests queued812 ms TTFT | 131.1Requests queued1,277 ms TTFT | 185.2Requests queued1,773 ms TTFT | Over capacity |
| 200K | 33.21,093 ms TTFT | 61.4Requests queued1,419 ms TTFT | 93.7Requests queued2,100 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,752.0 | — | 2,183 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,447.0 | — | 9,369 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,090.0 | — | 20,851 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,505.0 | — | 51,379 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 50.5 | 51.1 | 86 ms | 19.6 ms | 1.0 | 6 | 1,516 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 129.9 | 66.6 | 123 ms | 15.0 ms | 1.9 | 16 | 3,898 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 255.6 | 66.2 | 167 ms | 15.1 ms | 4.0 | 32 | 7,664 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 471.7 | 62.2 | 238 ms | 16.1 ms | 8.0 | 64 | 14,144 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 466.0 | 30.1 | 366 ms | 33.2 ms | 15.5 | 49 | 13,967 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 49.5 | 50.4 | 111 ms | 19.8 ms | 1.0 | 6 | 1,485 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 126.0 | 65.2 | 155 ms | 15.3 ms | 2.0 | 16 | 3,780 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 240.1 | 63.6 | 253 ms | 15.7 ms | 3.9 | 32 | 7,204 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 416.4 | 56.2 | 321 ms | 17.8 ms | 7.7 | 56 | 12,486 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 424.5 | 28.2 | 629 ms | 35.4 ms | 15.6 | 48 | 12,729 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 46.6 | 48.5 | 218 ms | 20.6 ms | 0.9 | 6 | 1,397 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 110.8 | 59.2 | 297 ms | 16.9 ms | 1.9 | 14 | 3,322 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 209.2 | 56.9 | 413 ms | 17.6 ms | 3.8 | 28 | 6,238 | 29.8 s |
| Sustained decode | 32K | 8 | Requests queued | 347.3 | 47.1 | 504 ms | 21.3 ms | 7.5 | 40 | 10,416 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 366.0 | 23.5 | 573 ms | 42.6 ms | 15.6 | 36 | 10,929 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 43.4 | 46.3 | 396 ms | 21.6 ms | 0.9 | 5 | 1,301 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 95.6 | 53.3 | 527 ms | 18.8 ms | 1.9 | 12 | 2,868 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 163.1 | 47.7 | 883 ms | 21.0 ms | 3.7 | 20 | 4,892 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 245.2 | 40.5 | 1,843 ms | 24.7 ms | 6.9 | 32 | 7,352 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 254.6 | 18.6 | 2,203 ms | 53.6 ms | 14.8 | 32 | 7,638 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 38.2 | 42.5 | 634 ms | 23.5 ms | 0.9 | 5 | 1,147 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 78.1 | 45.1 | 812 ms | 22.2 ms | 1.8 | 10 | 2,344 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 131.1 | 39.3 | 1,277 ms | 25.5 ms | 3.6 | 16 | 3,932 | 30.0 s |
| Sustained decode | 128K | 8 | Requests queued | 185.2 | 27.3 | 1,773 ms | 36.6 ms | 7.2 | 24 | 5,552 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 33.2 | 38.8 | 1,093 ms | 25.8 ms | 0.9 | 4 | 997 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 61.4 | 37.9 | 1,419 ms | 26.4 ms | 1.7 | 8 | 1,840 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 93.7 | 28.6 | 2,100 ms | 34.9 ms | 3.4 | 12 | 2,812 | 30.0 s |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.