Speed test matrix
gpt-oss-20b on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · MXFP44 contexts (0–64K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- MXFP4
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- gpt-oss
- Revision
- 6cee5e81ee83917806bbde320786a8fb61efebee
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-oss20-u-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/oss20-u-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/gpt-oss-20b --host 0.0.0.0 --served-model-name gpt-oss --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000
Prefill
prompt tok/s by context length8K context14.3K572 ms TTFT
32K context12.6K2,543 ms TTFT
64K context10.7K6,004 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 179.932 ms TTFT | 317.243 ms TTFT | 555.046 ms TTFT | 970.758 ms TTFT | 1,670.089 ms TTFT |
| 8K | 173.754 ms TTFT | 301.277 ms TTFT | 523.781 ms TTFT | 911.1117 ms TTFT | 1,493.4144 ms TTFT |
| 32K | 162.969 ms TTFT | 288.3Requests queued73 ms TTFT | 496.385 ms TTFT | 857.3107 ms TTFT | 1,400.5119 ms TTFT |
| 64K | 146.7150 ms TTFT | 256.3184 ms TTFT | 420.9Requests queued164 ms TTFT | 665.2Requests queued251 ms TTFT | 978.8Requests queued509 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.3K | — | 572 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.6K | — | 2,543 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.7K | — | 6,004 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 179.9 | 183.4 | 32 ms | 5.5 ms | 1.0 | 24 | 5,396 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 317.2 | 162.4 | 43 ms | 6.2 ms | 2.0 | 42 | 9,516 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 555.0 | 142.6 | 46 ms | 7.0 ms | 3.9 | 76 | 16,651 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 970.7 | 125.3 | 58 ms | 8.0 ms | 8.0 | 128 | 29,119 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,670.0 | 108.0 | 89 ms | 9.3 ms | 15.6 | 224 | 50,105 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 173.7 | 179.6 | 54 ms | 5.6 ms | 1.0 | 22 | 5,210 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 301.2 | 158.2 | 77 ms | 6.3 ms | 2.0 | 40 | 9,036 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 523.7 | 136.5 | 81 ms | 7.3 ms | 3.9 | 68 | 15,711 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 911.1 | 119.8 | 117 ms | 8.3 ms | 8.0 | 120 | 27,330 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,493.4 | 101.4 | 144 ms | 9.9 ms | 15.6 | 192 | 44,798 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 162.9 | 169.7 | 69 ms | 5.9 ms | 0.9 | 21 | 4,886 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 288.3 | 151.6 | 73 ms | 6.6 ms | 1.9 | 38 | 8,649 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 496.3 | 129.5 | 85 ms | 7.7 ms | 3.8 | 64 | 14,888 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 857.3 | 111.7 | 107 ms | 8.9 ms | 7.8 | 112 | 25,716 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,400.5 | 91.5 | 119 ms | 10.9 ms | 15.7 | 176 | 42,009 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 146.7 | 160.4 | 150 ms | 6.2 ms | 0.9 | 19 | 4,400 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 256.3 | 142.5 | 184 ms | 7.0 ms | 2.0 | 34 | 7,687 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 420.9 | 114.0 | 164 ms | 8.8 ms | 3.8 | 56 | 12,620 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 665.2 | 91.4 | 251 ms | 10.9 ms | 7.4 | 88 | 19,953 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 978.8 | 78.5 | 509 ms | 12.7 ms | 14.2 | 123 | 29,346 | 30.0 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.