Speed test matrix
Qwen3.8-27B-FP8 on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · FP86 contexts (0–200K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- FP8
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- qwen38-uvc
- Revision
- 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-q38f-u-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/q38f-u-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/Qwen3.8-27B-FP8 --host 0.0.0.0 --served-model-name qwen38-uvc --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context2,0993,905 ms TTFT
32K context2,01715,946 ms TTFT
64K context1,88334,049 ms TTFT
128K context1,65377,451 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 30.595 ms TTFT | 59.7169 ms TTFT | 116.2303 ms TTFT | 222.8511 ms TTFT | 404.2916 ms TTFT |
| 8K | 30.0173 ms TTFT | 58.6235 ms TTFT | 111.6288 ms TTFT | 205.8293 ms TTFT | 354.5302 ms TTFT |
| 32K | 28.7371 ms TTFT | 54.4466 ms TTFT | 98.5610 ms TTFT | 177.4621 ms TTFT | 269.7640 ms TTFT |
| 64K | 27.9350 ms TTFT | 52.3432 ms TTFT | 94.0Requests queued674 ms TTFT | 164.3Requests queued753 ms TTFT | 242.3Requests queued842 ms TTFT |
| 128K | 25.3745 ms TTFT | 46.31,000 ms TTFT | 78.9Requests queued1,548 ms TTFT | 117.2Requests queued2,220 ms TTFT | Over capacity |
| 200K | 23.21,030 ms TTFT | 41.31,373 ms TTFT | 66.9Requests queued1,968 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,099.0 | — | 3,905 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,017.0 | — | 15,946 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,883.0 | — | 34,049 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,653.0 | — | 77,451 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 30.5 | 30.8 | 95 ms | 32.5 ms | 1.0 | 4 | 915 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 59.7 | 30.4 | 169 ms | 32.9 ms | 1.9 | 8 | 1,792 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 116.2 | 30.0 | 303 ms | 33.4 ms | 3.9 | 16 | 3,468 | 29.9 s |
| Sustained decode | 0 | 8 | Valid | 222.8 | 29.2 | 511 ms | 34.2 ms | 7.8 | 24 | 6,680 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 404.2 | 27.7 | 916 ms | 36.1 ms | 16.0 | 48 | 12,112 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 30.0 | 30.4 | 173 ms | 32.9 ms | 1.0 | 3 | 899 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 58.6 | 29.9 | 235 ms | 33.5 ms | 2.0 | 6 | 1,756 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 111.6 | 28.6 | 288 ms | 35.0 ms | 3.9 | 12 | 3,348 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 205.8 | 26.4 | 293 ms | 37.9 ms | 7.9 | 24 | 6,168 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 354.5 | 23.1 | 302 ms | 43.3 ms | 15.7 | 48 | 10,625 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 28.7 | 29.7 | 371 ms | 33.7 ms | 1.0 | 3 | 860 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 54.4 | 28.4 | 466 ms | 35.3 ms | 2.0 | 6 | 1,630 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 98.5 | 26.1 | 610 ms | 38.2 ms | 3.9 | 12 | 2,952 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 177.4 | 22.4 | 621 ms | 44.6 ms | 7.9 | 16 | 5,320 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 269.7 | 17.6 | 640 ms | 56.8 ms | 15.7 | 32 | 8,080 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 27.9 | 28.9 | 350 ms | 34.7 ms | 1.0 | 3 | 838 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 52.3 | 27.2 | 432 ms | 36.8 ms | 1.9 | 6 | 1,568 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 94.0 | 24.9 | 674 ms | 40.2 ms | 3.8 | 12 | 2,820 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 164.3 | 21.3 | 753 ms | 47.0 ms | 7.8 | 16 | 4,928 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 242.3 | 16.2 | 842 ms | 61.8 ms | 15.4 | 32 | 7,248 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 25.3 | 27.3 | 745 ms | 36.7 ms | 0.9 | 3 | 760 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 46.3 | 24.6 | 1,000 ms | 40.6 ms | 1.9 | 6 | 1,368 | 29.6 s |
| Sustained decode | 128K | 4 | Requests queued | 78.9 | 21.4 | 1,548 ms | 46.7 ms | 3.7 | 8 | 2,368 | 30.0 s |
| Sustained decode | 128K | 8 | Requests queued | 117.2 | 16.7 | 2,220 ms | 60.0 ms | 7.3 | 16 | 3,474 | 29.6 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 1 | Valid | 23.2 | 25.7 | 1,030 ms | 38.9 ms | 0.9 | 3 | 695 | 30.0 s |
| Sustained decode | 200K | 2 | Valid | 41.3 | 22.6 | 1,373 ms | 44.3 ms | 1.9 | 4 | 1,236 | 30.0 s |
| Sustained decode | 200K | 4 | Requests queued | 66.9 | 18.7 | 1,968 ms | 53.3 ms | 3.6 | 8 | 2,004 | 30.0 s |
| Sustained decode | 200K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 200K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.