Speed test matrix
Qwen3.6-27B on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · BF166 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- q36d
- Revision
- 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36d-u-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36d-u-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/Qwen3.6-27B --host 0.0.0.0 --served-model-name q36d --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context3,7092,208 ms TTFT
32K context3,4449,377 ms TTFT
64K context3,08520,887 ms TTFT
128K context2,49351,627 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 28.676 ms TTFT | 56.3132 ms TTFT | 110.6182 ms TTFT | 212.8258 ms TTFT | 396.1397 ms TTFT |
| 8K | 28.1123 ms TTFT | 55.3183 ms TTFT | 106.7215 ms TTFT | 198.4237 ms TTFT | 358.1246 ms TTFT |
| 32K | 27.2213 ms TTFT | 52.7287 ms TTFT | 99.0363 ms TTFT | 177.8374 ms TTFT | 309.1383 ms TTFT |
| 64K | 26.0401 ms TTFT | 49.0533 ms TTFT | 91.5Requests queued844 ms TTFT | 154.7Requests queued944 ms TTFT | Over capacity |
| 128K | 24.1640 ms TTFT | 45.3824 ms TTFT | 80.31,231 ms TTFT | Over capacity | Over capacity |
| 200K | 22.61,043 ms TTFT | 39.9Requests queued1,371 ms TTFT | 66.3Requests queued2,052 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,709.0 | — | 2,208 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,444.0 | — | 9,377 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,085.0 | — | 20,887 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,493.0 | — | 51,627 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 28.6 | 28.7 | 76 ms | 34.8 ms | 1.0 | 3 | 858 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 56.3 | 28.5 | 132 ms | 35.1 ms | 2.0 | 6 | 1,688 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 110.6 | 28.0 | 182 ms | 35.7 ms | 3.9 | 12 | 3,316 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 212.8 | 27.2 | 258 ms | 36.8 ms | 8.0 | 24 | 6,384 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 396.1 | 25.7 | 397 ms | 39.0 ms | 16.0 | 48 | 11,872 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 28.1 | 28.3 | 123 ms | 35.3 ms | 1.0 | 3 | 843 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 55.3 | 28.0 | 183 ms | 35.7 ms | 2.0 | 6 | 1,658 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 106.7 | 27.1 | 215 ms | 36.8 ms | 4.0 | 12 | 3,196 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 198.4 | 25.3 | 237 ms | 39.5 ms | 7.9 | 24 | 5,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 358.1 | 22.6 | 246 ms | 44.3 ms | 15.8 | 40 | 10,706 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 27.2 | 27.7 | 213 ms | 36.1 ms | 1.0 | 3 | 817 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 52.7 | 27.0 | 287 ms | 37.1 ms | 2.0 | 6 | 1,580 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 99.0 | 25.6 | 363 ms | 39.1 ms | 3.9 | 12 | 2,968 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 177.8 | 23.0 | 374 ms | 43.5 ms | 7.8 | 24 | 5,333 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 309.1 | 19.5 | 383 ms | 51.3 ms | 15.7 | 32 | 9,264 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 26.0 | 27.0 | 401 ms | 37.1 ms | 1.0 | 3 | 780 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 49.0 | 25.8 | 533 ms | 38.8 ms | 1.9 | 6 | 1,470 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 91.5 | 24.0 | 844 ms | 41.7 ms | 3.9 | 11 | 2,742 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 154.7 | 20.3 | 944 ms | 49.2 ms | 7.7 | 16 | 4,640 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 24.1 | 25.7 | 640 ms | 39.0 ms | 0.9 | 3 | 724 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 45.3 | 23.8 | 824 ms | 42.0 ms | 2.0 | 4 | 1,360 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 80.3 | 21.4 | 1,231 ms | 46.7 ms | 3.9 | 8 | 2,408 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 22.6 | 24.3 | 1,043 ms | 41.2 ms | 0.9 | 2 | 678 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 39.9 | 21.8 | 1,371 ms | 45.9 ms | 1.9 | 4 | 1,196 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 66.3 | 18.6 | 2,052 ms | 53.6 ms | 3.6 | 8 | 1,988 | 30.0 s |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.