Speed test matrix
gemma-4-31B-it on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · BF163 contexts (0–32K) × 5 concurrency levels (1–16) · prefill at 8K, 32K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- g31
- Revision
- 842da3794eaa0b77d5f08bae87a17459d91ff475
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-g31-u-tp4-ctx64k --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/g31-u-tp4-ctx64k/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 65536 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context592.013,859 ms TTFT
32K context170.0190,178 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 22.579 ms TTFT | 43.9171 ms TTFT | 85.2208 ms TTFT | 160.1300 ms TTFT | 284.5481 ms TTFT |
| 8K | 17.2201 ms TTFT | 33.4341 ms TTFT | 59.7418 ms TTFT | 107.2495 ms TTFT | 178.2667 ms TTFT |
| 32K | 12.3Requests queued1,013 ms TTFT | 23.01,469 ms TTFT | 37.43,576 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 592.0 | — | 13,859 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 170.0 | — | 190,178 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 22.5 | 22.6 | 79 ms | 44.3 ms | 1.0 | 3 | 674 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 43.9 | 22.2 | 171 ms | 45.1 ms | 2.0 | 4 | 1,316 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 85.2 | 21.5 | 208 ms | 46.5 ms | 4.0 | 8 | 2,552 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 160.1 | 20.4 | 300 ms | 49.0 ms | 8.0 | 16 | 4,800 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 284.5 | 18.3 | 481 ms | 54.7 ms | 16.0 | 32 | 8,528 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 17.2 | 17.4 | 201 ms | 57.5 ms | 1.0 | 2 | 516 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 33.4 | 17.0 | 341 ms | 58.9 ms | 2.0 | 4 | 1,000 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 59.7 | 15.1 | 418 ms | 66.3 ms | 4.0 | 4 | 1,788 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 107.2 | 13.6 | 495 ms | 73.6 ms | 8.0 | 8 | 3,216 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 178.2 | 11.3 | 667 ms | 88.2 ms | 15.9 | 16 | 5,338 | 30.0 s |
| Sustained decode | 32K | 1 | Requests queued | 12.3 | 13.0 | 1,013 ms | 76.8 ms | 0.9 | 1 | 369 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 23.0 | 12.0 | 1,469 ms | 83.6 ms | 2.0 | 2 | 688 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 37.4 | 10.6 | 3,576 ms | 94.5 ms | 4.0 | 4 | 1,120 | 29.9 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.