Speed test matrix
Qwen3.5-122B-A10B-GPTQ-Int4 on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · GPTQ-4bit6 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- GPTQ-4bit
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- q35g
- Revision
- 30cd92cba9707a9aba09d1e490ed4b66b78e9606
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q35g-i-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q35g-i-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.5-122B-A10B-GPTQ-Int4 --host 0.0.0.0 --served-model-name q35g --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context6,0101,362 ms TTFT
32K context5,4175,961 ms TTFT
64K context4,57714,073 ms TTFT
128K context3,43537,464 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 52.796 ms TTFT | 100.3120 ms TTFT | 181.3175 ms TTFT | 316.5Requests queued271 ms TTFT | 539.6274 ms TTFT |
| 8K | 52.1118 ms TTFT | 98.1169 ms TTFT | 177.4239 ms TTFT | 295.2265 ms TTFT | 480.0512 ms TTFT |
| 32K | 49.9187 ms TTFT | 91.8238 ms TTFT | 159.1292 ms TTFT | 259.2362 ms TTFT | 401.9443 ms TTFT |
| 64K | 46.1383 ms TTFT | 82.3Requests queued507 ms TTFT | 136.2782 ms TTFT | 213.0Requests queued1,504 ms TTFT | Over capacity |
| 128K | 42.0587 ms TTFT | 72.5Requests queued760 ms TTFT | 119.8Requests queued1,181 ms TTFT | Over capacity | Over capacity |
| 200K | 35.71,308 ms TTFT | 58.3Requests queued1,704 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,010.0 | — | 1,362 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,417.0 | — | 5,961 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,577.0 | — | 14,073 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,435.0 | — | 37,464 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 52.7 | 53.6 | 96 ms | 18.7 ms | 1.0 | 7 | 1,582 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 100.3 | 51.4 | 120 ms | 19.4 ms | 2.0 | 12 | 3,010 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 181.3 | 46.4 | 175 ms | 21.5 ms | 3.9 | 24 | 5,436 | 30.0 s |
| Sustained decode | 0 | 8 | Requests queued | 316.5 | 41.1 | 271 ms | 24.3 ms | 7.6 | 40 | 9,488 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 539.6 | 35.0 | 274 ms | 28.6 ms | 16.0 | 64 | 16,176 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 52.1 | 53.1 | 118 ms | 18.8 ms | 1.0 | 6 | 1,562 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 98.1 | 50.5 | 169 ms | 19.8 ms | 2.0 | 12 | 2,942 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 177.4 | 45.8 | 239 ms | 21.8 ms | 4.0 | 20 | 5,320 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 295.2 | 38.7 | 265 ms | 25.9 ms | 7.8 | 40 | 8,834 | 29.9 s |
| Sustained decode | 8K | 16 | Valid | 480.0 | 32.0 | 512 ms | 31.3 ms | 15.4 | 64 | 14,395 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 49.9 | 51.6 | 187 ms | 19.4 ms | 1.0 | 6 | 1,497 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 91.8 | 47.9 | 238 ms | 20.9 ms | 2.0 | 12 | 2,754 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 159.1 | 41.7 | 292 ms | 24.0 ms | 3.8 | 20 | 4,772 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 259.2 | 34.1 | 362 ms | 29.3 ms | 7.9 | 32 | 7,774 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 401.9 | 26.3 | 443 ms | 38.1 ms | 15.5 | 48 | 12,052 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 46.1 | 49.7 | 383 ms | 20.1 ms | 0.9 | 6 | 1,383 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 82.3 | 44.7 | 507 ms | 22.4 ms | 1.8 | 10 | 2,470 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 136.2 | 37.7 | 782 ms | 26.5 ms | 3.9 | 16 | 4,084 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 213.0 | 30.9 | 1,504 ms | 32.4 ms | 7.4 | 24 | 6,386 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 42.0 | 46.4 | 587 ms | 21.6 ms | 0.8 | 5 | 1,260 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 72.5 | 40.1 | 760 ms | 25.0 ms | 1.7 | 8 | 2,174 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 119.8 | 33.6 | 1,181 ms | 29.8 ms | 3.7 | 12 | 3,592 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 35.7 | 43.0 | 1,308 ms | 23.2 ms | 0.8 | 4 | 1,070 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 58.3 | 35.0 | 1,704 ms | 28.7 ms | 1.8 | 6 | 1,748 | 30.0 s |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.