Speed test matrix
gemma-4-31B-it on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · FP84 contexts (0–64K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- FP8
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- g31
- Revision
- 842da3794eaa0b77d5f08bae87a17459d91ff475
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-g31-if8-tp4-ctx131k --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/g31-if8-tp4-ctx131k/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context1,1237,301 ms TTFT
32K context384.084,342 ms TTFT
64K context185.0349,647 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 36.479 ms TTFT | 69.1118 ms TTFT | 131.1163 ms TTFT | 231.7259 ms TTFT | 396.8432 ms TTFT |
| 8K | 24.0140 ms TTFT | 45.3221 ms TTFT | 80.4269 ms TTFT | 133.3291 ms TTFT | 224.6411 ms TTFT |
| 32K | 15.7Requests queued588 ms TTFT | 28.0817 ms TTFT | 47.51,673 ms TTFT | 75.52,948 ms TTFT | Over capacity |
| 64K | 12.7Requests queued698 ms TTFT | 21.2889 ms TTFT | 35.51,296 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,123.0 | — | 7,301 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 384.0 | — | 84,342 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 185.0 | — | 349,647 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 36.4 | 36.7 | 79 ms | 27.3 ms | 1.0 | 5 | 1,090 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 69.1 | 35.0 | 118 ms | 28.6 ms | 2.0 | 8 | 2,070 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 131.1 | 33.3 | 163 ms | 30.0 ms | 4.0 | 16 | 3,932 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 231.7 | 29.8 | 259 ms | 33.6 ms | 7.8 | 32 | 6,942 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 396.8 | 25.8 | 432 ms | 38.8 ms | 16.0 | 48 | 11,891 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 24.0 | 24.3 | 140 ms | 41.2 ms | 1.0 | 3 | 721 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 45.3 | 23.1 | 221 ms | 43.3 ms | 1.9 | 6 | 1,360 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 80.4 | 20.4 | 269 ms | 49.1 ms | 3.9 | 8 | 2,408 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 133.3 | 17.0 | 291 ms | 58.8 ms | 7.8 | 16 | 3,997 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 224.6 | 14.2 | 411 ms | 70.4 ms | 15.7 | 16 | 6,733 | 30.0 s |
| Sustained decode | 32K | 1 | Requests queued | 15.7 | 16.0 | 588 ms | 62.4 ms | 0.9 | 1 | 472 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 28.0 | 14.3 | 817 ms | 69.9 ms | 2.0 | 2 | 840 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 47.5 | 12.6 | 1,673 ms | 79.5 ms | 4.0 | 4 | 1,424 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 75.5 | 10.4 | 2,948 ms | 96.6 ms | 7.7 | 8 | 2,263 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Requests queued | 12.7 | 13.0 | 698 ms | 76.9 ms | 0.9 | 1 | 380 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 21.2 | 10.9 | 889 ms | 92.1 ms | 2.0 | 2 | 636 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 35.5 | 9.2 | 1,296 ms | 108.2 ms | 4.0 | 4 | 1,064 | 29.9 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.