Speed test matrix
Qwen3.6-27B on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · FP86 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- FP8
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- q36d
- Revision
- 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q36d-if8-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q36d-if8-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-27B --host 0.0.0.0 --served-model-name q36d --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context3,8582,123 ms TTFT
32K context3,5599,074 ms TTFT
64K context3,16820,340 ms TTFT
128K context2,54850,522 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 48.180 ms TTFT | 90.8113 ms TTFT | 177.9154 ms TTFT | 347.7234 ms TTFT | 626.7362 ms TTFT |
| 8K | 47.2103 ms TTFT | 89.1113 ms TTFT | 170.9247 ms TTFT | 314.9339 ms TTFT | 556.0651 ms TTFT |
| 32K | 45.0187 ms TTFT | 82.3243 ms TTFT | 152.7316 ms TTFT | 274.9525 ms TTFT | 455.9426 ms TTFT |
| 64K | 41.5405 ms TTFT | 73.6543 ms TTFT | 127.4907 ms TTFT | 212.8Requests queued1,865 ms TTFT | 309.9Requests queued2,233 ms TTFT |
| 128K | 37.5629 ms TTFT | 61.9Requests queued856 ms TTFT | 108.4Requests queued1,339 ms TTFT | 169.5Requests queued1,833 ms TTFT | Over capacity |
| 200K | 32.11,082 ms TTFT | 52.1Requests queued1,443 ms TTFT | 87.3Requests queued2,120 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,858.0 | — | 2,123 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,559.0 | — | 9,074 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,168.0 | — | 20,340 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,548.0 | — | 50,522 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 48.1 | 48.5 | 80 ms | 20.6 ms | 1.0 | 6 | 1,442 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 90.8 | 46.2 | 113 ms | 21.6 ms | 2.0 | 12 | 2,721 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 177.9 | 45.7 | 154 ms | 21.9 ms | 4.0 | 24 | 5,332 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 347.7 | 45.2 | 234 ms | 22.1 ms | 7.8 | 48 | 10,431 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 626.7 | 41.5 | 362 ms | 24.1 ms | 15.5 | 80 | 18,800 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 47.2 | 48.0 | 103 ms | 20.8 ms | 0.9 | 6 | 1,416 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 89.1 | 45.8 | 113 ms | 21.8 ms | 2.0 | 12 | 2,674 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 170.9 | 44.4 | 247 ms | 22.5 ms | 4.0 | 20 | 5,124 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 314.9 | 41.7 | 339 ms | 24.0 ms | 7.8 | 40 | 9,443 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 556.0 | 37.9 | 651 ms | 26.4 ms | 15.4 | 64 | 16,669 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 45.0 | 46.3 | 187 ms | 21.6 ms | 1.0 | 5 | 1,350 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 82.3 | 42.7 | 243 ms | 23.4 ms | 1.9 | 10 | 2,468 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 152.7 | 40.2 | 316 ms | 24.9 ms | 3.9 | 20 | 4,580 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 274.9 | 36.5 | 525 ms | 27.4 ms | 7.7 | 32 | 8,248 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 455.9 | 29.5 | 426 ms | 33.9 ms | 15.5 | 48 | 13,673 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 41.5 | 44.3 | 405 ms | 22.6 ms | 0.9 | 5 | 1,244 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 73.6 | 39.4 | 543 ms | 25.4 ms | 2.0 | 8 | 2,208 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 127.4 | 36.1 | 907 ms | 27.7 ms | 3.8 | 16 | 3,820 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 212.8 | 32.6 | 1,865 ms | 30.7 ms | 7.4 | 24 | 6,384 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 309.9 | 22.5 | 2,233 ms | 44.5 ms | 14.8 | 32 | 9,286 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 37.5 | 40.8 | 629 ms | 24.5 ms | 0.9 | 5 | 1,119 | 29.9 s |
| Sustained decode | 128K | 2 | Requests queued | 61.9 | 34.8 | 856 ms | 28.7 ms | 1.7 | 8 | 1,856 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 108.4 | 31.0 | 1,339 ms | 32.3 ms | 3.7 | 12 | 3,248 | 30.0 s |
| Sustained decode | 128K | 8 | Requests queued | 169.5 | 23.4 | 1,833 ms | 42.7 ms | 7.4 | 16 | 5,079 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 32.1 | 37.3 | 1,082 ms | 26.8 ms | 0.9 | 4 | 963 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 52.1 | 30.4 | 1,443 ms | 33.0 ms | 1.8 | 6 | 1,562 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 87.3 | 24.7 | 2,120 ms | 40.5 ms | 3.6 | 8 | 2,617 | 30.0 s |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.