Speed test matrix
Qwen3.5-9B on Intel Arc Pro B65 32GB
vllm · 0.32.0 · BF166 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- VRAM
- 32 GB
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- q359
- Revision
- c202236235762e1c871ad0ccb60c8ee5ba337b9a
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q359-u-tp1-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q359-u-tp1-ctx/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context4,8881,675 ms TTFT
32K context4,2607,580 ms TTFT
64K context3,56518,072 ms TTFT
128K context2,69447,763 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 34.261 ms TTFT | 67.484 ms TTFT | 132.4116 ms TTFT | 253.4199 ms TTFT | 469.7328 ms TTFT |
| 8K | 33.3113 ms TTFT | 65.0152 ms TTFT | 125.0193 ms TTFT | 235.0215 ms TTFT | 410.9Requests queued228 ms TTFT |
| 32K | 31.1254 ms TTFT | 60.2310 ms TTFT | 111.6Requests queued373 ms TTFT | Over capacity | Over capacity |
| 64K | 29.3293 ms TTFT | 55.4Requests queued359 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 25.7529 ms TTFT | 46.3610 ms TTFT | Over capacity | Over capacity | Over capacity |
| 200K | 22.7850 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,888.0 | — | 1,675 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,260.0 | — | 7,580 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,565.0 | — | 18,072 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,694.0 | — | 47,763 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 34.2 | 34.3 | 61 ms | 29.1 ms | 1.0 | 4 | 1,026 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 67.4 | 34.0 | 84 ms | 29.4 ms | 1.9 | 8 | 2,021 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 132.4 | 33.5 | 116 ms | 29.9 ms | 4.0 | 16 | 3,968 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 253.4 | 32.4 | 199 ms | 30.9 ms | 8.0 | 32 | 7,601 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 469.7 | 30.6 | 328 ms | 32.7 ms | 16.0 | 64 | 14,088 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 33.3 | 33.6 | 113 ms | 29.7 ms | 1.0 | 4 | 998 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 65.0 | 33.0 | 152 ms | 30.3 ms | 2.0 | 8 | 1,948 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 125.0 | 31.9 | 193 ms | 31.3 ms | 4.0 | 16 | 3,748 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 235.0 | 29.8 | 215 ms | 33.6 ms | 7.9 | 24 | 7,048 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 410.9 | 26.1 | 228 ms | 38.3 ms | 15.8 | 48 | 12,320 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 31.1 | 32.0 | 254 ms | 31.3 ms | 0.9 | 4 | 931 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 60.2 | 30.9 | 310 ms | 32.4 ms | 1.9 | 6 | 1,806 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 111.6 | 28.8 | 373 ms | 34.7 ms | 3.9 | 12 | 3,348 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 29.3 | 30.1 | 293 ms | 33.2 ms | 1.0 | 3 | 879 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 55.4 | 28.6 | 359 ms | 35.0 ms | 1.9 | 6 | 1,662 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 25.7 | 27.0 | 529 ms | 37.1 ms | 1.0 | 3 | 769 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 46.3 | 24.5 | 610 ms | 40.7 ms | 2.0 | 6 | 1,389 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 22.7 | 24.1 | 850 ms | 41.6 ms | 1.0 | 3 | 676 | 29.8 s |
| Sustained decode | 200K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.