Speed test matrix
Qwen3.8-Flash-Next-W4A16 on Intel Arc Pro B65 32GB ×4
vllm · 0.30.0 · INT4_W4A166 contexts (0–200K) × 3 concurrency levels (1–4) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.30.0
- Quantization
- INT4_W4A16
Run
- Served model
- qwen38-flash-next
- Revision
- 40b8f18df4d4a32cb6e687a51c78207e5e438522
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-lb-g2s16-262k --net=host --shm-size=32g --ipc=host --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/models:ro -v /models/bench/lcc-20261021/session-matrix/lb-g2s16-262k-matrix/session/cache:/cache --env-file /models/bench/lcc-20261021/session-matrix/lb-g2s16-262k-matrix/session/server-configs/lb-g2s16-262k.env kasa-lab-vllm-xpu:lumnus-b70.2-b103e0f vllm serve /models/Qwen3.8-Flash-Next-W4A16 --served-model-name qwen38-flash-next --host 0.0.0.0 --port 8000 --dtype bfloat16 --tensor-parallel-size 4 --enable-expert-parallel --trust-remote-code --max-model-len 262144 --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cudagraph-metrics --reasoning-parser qwen3 --override-generation-config \{\"temperature\":\ 0.7\} --enable-prompt-tokens-details --compilation-config \{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"\,\"cudagraph_capture_sizes\":\[1\,2\,4\,8\,256\,512\,1024\]\} --max-num-batched-tokens 2048 --gpu-memory-utilization 0.92 --max-num-seqs 16Prefill
prompt tok/s by context length8K context3,8562,126 ms TTFT
16K context3,8034,253 ms TTFT
32K context3,6288,865 ms TTFT
64K context3,19120,097 ms TTFT
128K context2,58649,521 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 |
|---|---|---|---|
| 0 | 42.6118 ms TTFT | 83.7140 ms TTFT | 158.3245 ms TTFT |
| 16K | 40.7241 ms TTFT | 79.4346 ms TTFT | 144.6Requests queued542 ms TTFT |
| 32K | 39.2368 ms TTFT | 75.2Requests queued373 ms TTFT | 133.6846 ms TTFT |
| 64K | 38.5353 ms TTFT | 73.6459 ms TTFT | 127.8596 ms TTFT |
| 128K | 34.4751 ms TTFT | 61.2Requests queued997 ms TTFT | 102.81,561 ms TTFT |
| 200K | 31.21,025 ms TTFT | 56.4Requests queued1,326 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,856.0 | — | 2,126 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,803.0 | — | 4,253 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,628.0 | — | 8,865 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,191.0 | — | 20,097 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,586.0 | — | 49,521 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 42.6 | 43.3 | 118 ms | 23.1 ms | 1.0 | 5 | 1,277 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 83.7 | 43.9 | 140 ms | 22.8 ms | 2.0 | 10 | 2,512 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 158.3 | 41.0 | 245 ms | 24.4 ms | 3.9 | 20 | 4,748 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 40.7 | 42.2 | 241 ms | 23.7 ms | 1.0 | 5 | 1,221 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 79.4 | 41.9 | 346 ms | 23.9 ms | 1.9 | 10 | 2,382 | 30.0 s |
| Sustained decode | 16K | 4 | Requests queued | 144.6 | 38.8 | 542 ms | 25.8 ms | 3.6 | 16 | 4,336 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 39.2 | 41.6 | 368 ms | 24.1 ms | 0.9 | 5 | 1,175 | 30.0 s |
| Sustained decode | 32K | 2 | Requests queued | 75.2 | 41.3 | 373 ms | 24.2 ms | 1.9 | 10 | 2,238 | 29.8 s |
| Sustained decode | 32K | 4 | Valid | 133.6 | 37.4 | 846 ms | 26.7 ms | 3.9 | 16 | 4,004 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 38.5 | 40.3 | 353 ms | 24.8 ms | 0.9 | 5 | 1,151 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 73.6 | 39.0 | 459 ms | 25.6 ms | 2.0 | 8 | 2,206 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 127.8 | 34.5 | 596 ms | 28.9 ms | 4.0 | 16 | 3,832 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 34.4 | 38.1 | 751 ms | 26.2 ms | 0.9 | 4 | 1,033 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 61.2 | 35.3 | 997 ms | 28.4 ms | 1.8 | 8 | 1,836 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 102.8 | 30.3 | 1,561 ms | 33.0 ms | 3.8 | 12 | 3,084 | 30.0 s |
| Sustained decode | 200K | 1 | Valid | 31.2 | 36.0 | 1,025 ms | 27.8 ms | 0.9 | 4 | 935 | 30.0 s |
| Sustained decode | 200K | 2 | Requests queued | 56.4 | 32.3 | 1,326 ms | 31.0 ms | 1.8 | 6 | 1,690 | 30.0 s |
| Sustained decode | 200K | 4 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.