Speed test matrix
Qwen3.8-27B on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · BF166 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- qwen38-27b
- Revision
- 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q38b-u-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q38b-u-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/Qwen3.8-27B --host 0.0.0.0 --served-model-name qwen38-27b --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context3,6992,216 ms TTFT
32K context3,4749,256 ms TTFT
64K context3,07920,824 ms TTFT
128K context2,50051,227 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 28.678 ms TTFT | 56.2152 ms TTFT | 110.0220 ms TTFT | 211.0329 ms TTFT | 389.5535 ms TTFT |
| 8K | 28.1139 ms TTFT | 55.2206 ms TTFT | 106.2234 ms TTFT | 197.9243 ms TTFT | 357.6286 ms TTFT |
| 32K | 27.1282 ms TTFT | 52.1372 ms TTFT | 96.5Requests queued481 ms TTFT | 176.9482 ms TTFT | 287.0502 ms TTFT |
| 64K | 26.0402 ms TTFT | 49.5Requests queued471 ms TTFT | 91.3629 ms TTFT | 159.1Requests queued808 ms TTFT | Over capacity |
| 128K | 24.0679 ms TTFT | 45.1892 ms TTFT | 79.3Requests queued1,326 ms TTFT | Over capacity | Over capacity |
| 200K | 22.71,007 ms TTFT | 40.31,281 ms TTFT | 67.7Requests queued1,874 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,699.0 | — | 2,216 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,474.0 | — | 9,256 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,079.0 | — | 20,824 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,500.0 | — | 51,227 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 28.6 | 28.7 | 78 ms | 34.8 ms | 1.0 | 3 | 858 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 56.2 | 28.4 | 152 ms | 35.2 ms | 2.0 | 6 | 1,686 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 110.0 | 28.0 | 220 ms | 35.7 ms | 3.9 | 12 | 3,300 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 211.0 | 27.2 | 329 ms | 36.8 ms | 7.8 | 24 | 6,328 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 389.5 | 25.6 | 535 ms | 39.0 ms | 15.5 | 48 | 11,680 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 28.1 | 28.4 | 139 ms | 35.3 ms | 1.0 | 3 | 842 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 55.2 | 28.0 | 206 ms | 35.7 ms | 2.0 | 6 | 1,654 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 106.2 | 27.1 | 234 ms | 36.9 ms | 4.0 | 12 | 3,184 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 197.9 | 25.3 | 243 ms | 39.5 ms | 7.9 | 24 | 5,936 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 357.6 | 22.5 | 286 ms | 44.4 ms | 15.8 | 39 | 10,701 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 27.1 | 27.7 | 282 ms | 36.1 ms | 1.0 | 3 | 812 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 52.1 | 26.9 | 372 ms | 37.1 ms | 2.0 | 6 | 1,562 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 96.5 | 25.3 | 481 ms | 39.5 ms | 3.9 | 12 | 2,896 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 176.9 | 22.4 | 482 ms | 44.7 ms | 7.9 | 16 | 5,304 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 287.0 | 18.3 | 502 ms | 54.5 ms | 15.7 | 32 | 8,608 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 26.0 | 27.0 | 402 ms | 37.0 ms | 1.0 | 3 | 780 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 49.5 | 25.9 | 471 ms | 38.7 ms | 1.8 | 6 | 1,484 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 91.3 | 24.1 | 629 ms | 41.5 ms | 3.8 | 12 | 2,730 | 29.9 s |
| Sustained decode | 64K | 8 | Requests queued | 159.1 | 20.7 | 808 ms | 48.2 ms | 7.7 | 16 | 4,768 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 24.0 | 25.7 | 679 ms | 39.0 ms | 1.0 | 3 | 721 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 45.1 | 23.8 | 892 ms | 42.1 ms | 1.9 | 4 | 1,352 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 79.3 | 21.3 | 1,326 ms | 47.0 ms | 3.7 | 8 | 2,376 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 22.7 | 24.3 | 1,007 ms | 41.1 ms | 0.9 | 3 | 681 | 29.9 s |
| Sustained decode | 200K | 2 | Valid | 40.3 | 21.9 | 1,281 ms | 45.7 ms | 1.9 | 4 | 1,208 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 67.7 | 18.8 | 1,874 ms | 53.1 ms | 3.7 | 8 | 2,032 | 30.0 s |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.