Speed test matrix
Qwen3.8-Flash-Next-W4A16 on Intel Arc Pro B65 32GB ×4
vllm · 0.30.0 · INT4_W4A166 contexts (0–200K) × 3 concurrency levels (1–4) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.30.0
- Quantization
- INT4_W4A16
Run
- Served model
- qwen38-flash-next
- Revision
- 40b8f18df4d4a32cb6e687a51c78207e5e438522
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-pf-mbt8192-u92 --net=host --shm-size=32g --ipc=host --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/models:ro -v /models/bench/fnpf-20261020/session-p3/pf-mbt8192-u92-matrix/session/cache:/cache --env-file /models/bench/fnpf-20261020/session-p3/pf-mbt8192-u92-matrix/session/server-configs/pf-mbt8192-u92.env kasa-lab-vllm-xpu:lumnus-b70.2-b103e0f vllm serve /models/Qwen3.8-Flash-Next-W4A16 --served-model-name qwen38-flash-next --host 0.0.0.0 --port 8000 --dtype bfloat16 --tensor-parallel-size 4 --enable-expert-parallel --trust-remote-code --gpu-memory-utilization 0.92 --max-model-len 262144 --max-num-batched-tokens 8192 --max-num-seqs 4 --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cudagraph-metrics --reasoning-parser qwen3 --override-generation-config \{\"temperature\":\ 0.7\} --enable-prompt-tokens-details --compilation-config \{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"\,\"cudagraph_capture_sizes\":\[1\,2\,4\,8\,256\,512\,1024\]\}Prefill
prompt tok/s by context length8K context5,3061,545 ms TTFT
16K context5,1503,140 ms TTFT
32K context4,9506,496 ms TTFT
64K context4,46714,356 ms TTFT
128K context3,60335,542 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 |
|---|---|---|---|
| 0 | 42.7117 ms TTFT | 85.2174 ms TTFT | 158.6246 ms TTFT |
| 16K | 40.8237 ms TTFT | 79.6329 ms TTFT | 145.0540 ms TTFT |
| 32K | 39.3380 ms TTFT | 75.3366 ms TTFT | 134.1844 ms TTFT |
| 64K | 38.6357 ms TTFT | 73.8466 ms TTFT | 128.0594 ms TTFT |
| 128K | 34.5782 ms TTFT | 61.3Requests queued1,004 ms TTFT | 102.81,562 ms TTFT |
| 200K | 31.21,046 ms TTFT | 56.4Requests queued1,316 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,306.0 | — | 1,545 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,150.0 | — | 3,140 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,950.0 | — | 6,496 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,467.0 | — | 14,356 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,603.0 | — | 35,542 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 42.7 | 43.4 | 117 ms | 23.0 ms | 1.0 | 5 | 1,281 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 85.2 | 43.8 | 174 ms | 22.9 ms | 2.0 | 10 | 2,557 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 158.6 | 41.1 | 246 ms | 24.3 ms | 3.9 | 20 | 4,756 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 40.8 | 42.4 | 237 ms | 23.6 ms | 1.0 | 5 | 1,225 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 79.6 | 42.0 | 329 ms | 23.8 ms | 1.9 | 10 | 2,386 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 145.0 | 38.8 | 540 ms | 25.7 ms | 4.0 | 16 | 4,348 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 39.3 | 41.7 | 380 ms | 24.0 ms | 0.9 | 5 | 1,177 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 75.3 | 41.4 | 366 ms | 24.2 ms | 1.9 | 10 | 2,238 | 29.7 s |
| Sustained decode | 32K | 4 | Valid | 134.1 | 37.6 | 844 ms | 26.6 ms | 3.8 | 16 | 4,020 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 38.6 | 40.4 | 357 ms | 24.7 ms | 1.0 | 5 | 1,150 | 29.8 s |
| Sustained decode | 64K | 2 | Valid | 73.8 | 39.1 | 466 ms | 25.6 ms | 2.0 | 8 | 2,212 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 128.0 | 34.6 | 594 ms | 28.9 ms | 4.0 | 16 | 3,840 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 34.5 | 38.4 | 782 ms | 26.1 ms | 0.9 | 4 | 1,035 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 61.3 | 35.3 | 1,004 ms | 28.3 ms | 1.8 | 8 | 1,838 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 102.8 | 30.4 | 1,562 ms | 32.9 ms | 3.8 | 12 | 3,084 | 30.0 s |
| Sustained decode | 200K | 1 | Valid | 31.2 | 36.0 | 1,046 ms | 27.8 ms | 0.9 | 4 | 935 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 56.4 | 32.3 | 1,316 ms | 31.0 ms | 1.8 | 6 | 1,692 | 30.0 s |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.