Speed test matrix
Qwen3.5-9B on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · FP86 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- FP8
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- q359
- Revision
- c202236235762e1c871ad0ccb60c8ee5ba337b9a
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q359-if8-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q359-if8-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context10.6K770 ms TTFT
32K context10.2K3,170 ms TTFT
64K context9,0657,107 ms TTFT
128K context7,45417,265 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 126.145 ms TTFT | 238.767 ms TTFT | 462.876 ms TTFT | 858.594 ms TTFT | 1,568.3131 ms TTFT |
| 8K | 122.671 ms TTFT | 232.388 ms TTFT | 436.4134 ms TTFT | 766.7162 ms TTFT | 1,319.3Requests queued200 ms TTFT |
| 32K | 112.9159 ms TTFT | 202.8217 ms TTFT | 360.0Requests queued278 ms TTFT | 605.7Requests queued384 ms TTFT | 938.6Requests queued558 ms TTFT |
| 64K | 103.4273 ms TTFT | 179.5Requests queued305 ms TTFT | 310.6Requests queued428 ms TTFT | 479.5Requests queued665 ms TTFT | 682.9Requests queued681 ms TTFT |
| 128K | 87.1481 ms TTFT | 143.8Requests queued601 ms TTFT | 231.8Requests queued848 ms TTFT | 319.6Requests queued1,142 ms TTFT | 440.0Requests queued1,356 ms TTFT |
| 200K | 73.3803 ms TTFT | 113.8Requests queued974 ms TTFT | 175.7Requests queued1,244 ms TTFT | 230.6Requests queued1,377 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.6K | — | 770 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.2K | — | 3,170 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,065.0 | — | 7,107 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,454.0 | — | 17,265 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 126.1 | 128.5 | 45 ms | 7.8 ms | 1.0 | 17 | 3,782 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 238.7 | 122.5 | 67 ms | 8.2 ms | 1.9 | 32 | 7,158 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 462.8 | 119.7 | 76 ms | 8.4 ms | 4.0 | 64 | 13,884 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 858.5 | 111.2 | 94 ms | 9.0 ms | 8.0 | 112 | 25,752 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,568.3 | 102.9 | 131 ms | 9.7 ms | 15.5 | 208 | 47,050 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 122.6 | 126.6 | 71 ms | 7.9 ms | 1.0 | 16 | 3,678 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 232.3 | 120.3 | 88 ms | 8.3 ms | 1.9 | 30 | 6,968 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 436.4 | 115.6 | 134 ms | 8.7 ms | 4.0 | 56 | 13,089 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 766.7 | 101.4 | 162 ms | 9.9 ms | 7.9 | 96 | 22,999 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 1,319.3 | 88.1 | 200 ms | 11.4 ms | 14.8 | 176 | 39,576 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 112.9 | 121.1 | 159 ms | 8.3 ms | 1.0 | 14 | 3,386 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 202.8 | 110.0 | 217 ms | 9.1 ms | 2.0 | 26 | 6,084 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 360.0 | 101.1 | 278 ms | 9.9 ms | 3.7 | 48 | 10,799 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 605.7 | 85.4 | 384 ms | 11.7 ms | 7.5 | 80 | 18,167 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 938.6 | 67.3 | 558 ms | 14.9 ms | 14.8 | 128 | 28,155 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 103.4 | 115.5 | 273 ms | 8.7 ms | 0.9 | 13 | 3,102 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 179.5 | 100.6 | 305 ms | 9.9 ms | 1.8 | 24 | 5,384 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 310.6 | 86.5 | 428 ms | 11.6 ms | 3.7 | 40 | 9,316 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 479.5 | 70.1 | 665 ms | 14.3 ms | 7.2 | 64 | 14,338 | 29.9 s |
| Sustained decode | 64K | 16 | Requests queued | 682.9 | 48.1 | 681 ms | 20.8 ms | 14.8 | 80 | 20,484 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 87.1 | 103.5 | 481 ms | 9.7 ms | 0.8 | 11 | 2,614 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 143.8 | 86.3 | 601 ms | 11.6 ms | 1.8 | 20 | 4,314 | 30.0 s |
| Sustained decode | 128K | 4 | Requests queued | 231.8 | 70.0 | 848 ms | 14.3 ms | 3.4 | 28 | 6,951 | 30.0 s |
| Sustained decode | 128K | 8 | Requests queued | 319.6 | 49.2 | 1,142 ms | 20.3 ms | 6.9 | 40 | 9,585 | 30.0 s |
| Sustained decode | 128K | 16 | Requests queued | 440.0 | 30.7 | 1,356 ms | 32.5 ms | 14.5 | 48 | 13,191 | 30.0 s |
| Sustained decode | 200K | 1 | Valid | 73.3 | 93.2 | 803 ms | 10.7 ms | 0.7 | 9 | 2,199 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 113.8 | 73.8 | 974 ms | 13.6 ms | 1.6 | 16 | 3,413 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 175.7 | 57.3 | 1,244 ms | 17.5 ms | 3.3 | 24 | 5,257 | 29.9 s |
| Sustained decode | 200K | 8 | Requests queued | 230.6 | 35.4 | 1,377 ms | 28.3 ms | 6.9 | 32 | 6,909 | 30.0 s |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.