Speed test matrix
gemma-4-26B-A4B-it on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · FP84 contexts (0–64K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- FP8
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- gemma-4
- Revision
- 4d7ae4984b7db7de8f8457170b3f1a419ee76d52
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-gem82-u-tp4-ctx131k --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/gem82-u-tp4-ctx131k/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/gemma-4-26B-A4B-it --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --quantization fp8 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context2,2193,695 ms TTFT
32K context680.047,593 ms TTFT
64K context341.0189,312 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 85.825 ms TTFT | 161.343 ms TTFT | 289.067 ms TTFT | 509.867 ms TTFT | 856.091 ms TTFT |
| 8K | 53.5110 ms TTFT | 99.0153 ms TTFT | 179.2197 ms TTFT | 306.3205 ms TTFT | 442.0295 ms TTFT |
| 32K | 33.8387 ms TTFT | 58.4487 ms TTFT | 102.6719 ms TTFT | 175.8819 ms TTFT | 232.81,310 ms TTFT |
| 64K | 26.6Requests queued676 ms TTFT | 42.5Requests queued888 ms TTFT | 71.81,359 ms TTFT | 119.11,873 ms TTFT | 142.12,010 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,219.0 | — | 3,695 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 680.0 | — | 47,593 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 341.0 | — | 189,312 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 85.8 | 86.2 | 25 ms | 11.6 ms | 1.0 | 11 | 2,574 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 161.3 | 81.4 | 43 ms | 12.3 ms | 2.0 | 20 | 4,838 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 289.0 | 73.4 | 67 ms | 13.6 ms | 4.0 | 40 | 8,668 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 509.8 | 64.6 | 67 ms | 15.5 ms | 8.0 | 64 | 15,288 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 856.0 | 54.4 | 91 ms | 18.4 ms | 15.5 | 112 | 25,672 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 53.5 | 54.5 | 110 ms | 18.3 ms | 1.0 | 7 | 1,606 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 99.0 | 50.8 | 153 ms | 19.7 ms | 2.0 | 12 | 2,968 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 179.2 | 46.3 | 197 ms | 21.6 ms | 4.0 | 24 | 5,376 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 306.3 | 39.5 | 205 ms | 25.3 ms | 7.8 | 40 | 9,187 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 442.0 | 28.4 | 295 ms | 35.2 ms | 15.4 | 48 | 13,261 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 33.8 | 35.5 | 387 ms | 28.2 ms | 1.0 | 4 | 1,013 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 58.4 | 30.8 | 487 ms | 32.4 ms | 1.9 | 6 | 1,750 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 102.6 | 27.6 | 719 ms | 36.3 ms | 4.0 | 12 | 3,076 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 175.8 | 23.2 | 819 ms | 43.0 ms | 7.8 | 16 | 5,272 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 232.8 | 15.0 | 1,310 ms | 66.7 ms | 15.7 | 16 | 6,974 | 30.0 s |
| Sustained decode | 64K | 1 | Requests queued | 26.6 | 28.6 | 676 ms | 34.9 ms | 0.9 | 3 | 797 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 42.5 | 22.8 | 888 ms | 43.8 ms | 1.9 | 4 | 1,276 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 71.8 | 19.7 | 1,359 ms | 50.8 ms | 3.8 | 8 | 2,152 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 119.1 | 15.8 | 1,873 ms | 63.5 ms | 7.9 | 8 | 3,568 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 142.1 | 9.5 | 2,010 ms | 105.8 ms | 15.4 | 16 | 4,254 | 29.9 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.