Speed test matrix
Qwen3.8-27B on Intel Arc Pro B65 32GB
vllm · 0.31.0+xpu.llmscaler · INT45 contexts (0–128K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- VRAM
- 32 GB
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- INT4
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- qwen38-27b
- Revision
- 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q38s-i-tp1-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q38s-i-tp1-ctx/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.8-27B --host 0.0.0.0 --served-model-name qwen38-27b --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 135168 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context1,3795,943 ms TTFT
32K context1,22226,324 ms TTFT
64K context1,04161,582 ms TTFT
128K context809.0158,220 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 20.2183 ms TTFT | 59.9345 ms TTFT | 115.1433 ms TTFT | 204.3766 ms TTFT | 159.31,426 ms TTFT |
| 8K | 19.8229 ms TTFT | 57.4306 ms TTFT | 106.6550 ms TTFT | 184.5951 ms TTFT | 144.81,933 ms TTFT |
| 32K | 18.2538 ms TTFT | 48.2Requests queued743 ms TTFT | 84.01,315 ms TTFT | Over capacity | Over capacity |
| 64K | 16.7716 ms TTFT | 42.1Requests queued967 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 14.61,280 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,379.0 | — | 5,943 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,222.0 | — | 26,324 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,041.0 | — | 61,582 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 809.0 | — | 158,220 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 20.2 | 20.4 | 183 ms | 49.0 ms | 1.0 | 2 | 606 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 59.9 | 31.2 | 345 ms | 32.0 ms | 2.0 | 8 | 1,797 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 115.1 | 30.0 | 433 ms | 33.3 ms | 3.9 | 12 | 3,452 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 204.3 | 27.5 | 766 ms | 36.3 ms | 8.0 | 24 | 6,128 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 159.3 | 10.4 | 1,426 ms | 96.2 ms | 16.0 | 16 | 4,775 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 19.8 | 20.0 | 229 ms | 50.0 ms | 1.0 | 2 | 592 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 57.4 | 29.5 | 306 ms | 33.9 ms | 2.0 | 6 | 1,722 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 106.6 | 28.1 | 550 ms | 35.6 ms | 4.0 | 12 | 3,196 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 184.5 | 24.5 | 951 ms | 40.8 ms | 7.6 | 24 | 5,457 | 29.6 s |
| Sustained decode | 8K | 16 | Valid | 144.8 | 9.6 | 1,933 ms | 104.0 ms | 15.6 | 16 | 4,333 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 18.2 | 18.8 | 538 ms | 53.2 ms | 1.0 | 2 | 546 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 48.2 | 25.9 | 743 ms | 38.6 ms | 1.9 | 6 | 1,446 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 84.0 | 22.7 | 1,315 ms | 44.0 ms | 3.8 | 8 | 2,521 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 16.7 | 17.4 | 716 ms | 57.4 ms | 0.9 | 2 | 500 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 42.1 | 22.4 | 967 ms | 44.8 ms | 1.9 | 4 | 1,262 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 14.6 | 15.2 | 1,280 ms | 65.7 ms | 1.0 | 1 | 437 | 29.9 s |
| Sustained decode | 128K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.