Speed test matrix
gemma-4-12B-it on Intel Arc Pro B65 32GB
vllm · 0.32.0 · BF161 context (0) × 5 concurrency levels (1–16)
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- VRAM
- 32 GB
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- g12
- Revision
- 707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-g12-u-tp1-ctx131k --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/g12-u-tp1-ctx131k/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/gemma-4-12B-it --host 0.0.0.0 --served-model-name g12 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 8192 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 21.872 ms TTFT | 42.6123 ms TTFT | 81.7158 ms TTFT | 151.7262 ms TTFT | 262.4513 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Sustained decode | 0 | 1 | Valid | 21.8 | 21.8 | 72 ms | 45.8 ms | 1.0 | 3 | 652 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 42.6 | 21.4 | 123 ms | 46.7 ms | 2.0 | 4 | 1,276 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 81.7 | 20.6 | 158 ms | 48.6 ms | 3.9 | 8 | 2,448 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 151.7 | 19.2 | 262 ms | 52.0 ms | 7.8 | 16 | 4,548 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 262.4 | 17.0 | 513 ms | 58.9 ms | 16.0 | 32 | 7,861 | 30.0 s |