Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
Matching runs 693
Submitted by @keennay. Newest first. Page 8 of 28.- granite-4.2-8bRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev220+gfa7784d14 · BF165 contexts × 8 concurrency levels
- Peak decode
- 866.9 tok/s
- Peak prefill
- 17.7K tok/s
- granite-4.2-3bRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev220+gfa7784d14 · BF165 contexts × 8 concurrency levels
- Peak decode
- 3,350.4 tok/s
- Peak prefill
- 37K tok/s
- gemma-4-E4B-itRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 2,697.5 tok/s
- Peak prefill
- 27.1K tok/s
- gemma-4-E2B-itRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 9,035.1 tok/s
- Peak prefill
- 39.8K tok/s
- gemma-4-26B-A4B-itRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,702.1 tok/s
- Peak prefill
- 21.9K tok/s
- gemma-4-12B-itRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 719.3 tok/s
- Peak prefill
- 11K tok/s
- diffusiongemma-26B-A4B-itRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,469.4 tok/s
- Peak prefill
- 10.7K tok/s
- gemma-4-E4B-itRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · BF165 contexts × 8 concurrency levels
- Peak decode
- 152.4 tok/s
- Peak prefill
- 22.9K tok/s
- gemma-4-E2B-itRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · BF165 contexts × 8 concurrency levels
- Peak decode
- 7,851.5 tok/s
- Peak prefill
- 33.5K tok/s
- gemma-4-12B-itRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · BF166 contexts × 8 concurrency levels
- Peak decode
- 186.7 tok/s
- Peak prefill
- 9,728 tok/s
- gemma-4-26B-A4B-itRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · BF166 contexts × 8 concurrency levels
- Peak decode
- 119.4 tok/s
- Peak prefill
- 16.7K tok/s
- liftRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,207.1 tok/s
- Peak prefill
- 20.7K tok/s
- liftRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,003.9 tok/s
- Peak prefill
- 21.4K tok/s
- North-Mini-Code-1.0RTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF167 contexts × 8 concurrency levels
- Peak decode
- 1,217.3 tok/s
- Peak prefill
- 30.2K tok/s
- Trinity-Nano-BaseRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 3,299.0 tok/s
- Peak prefill
- 74K tok/s
- Trinity-MiniRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 1,560.9 tok/s
- Peak prefill
- 34.9K tok/s
- Trinity-Mini-W4A16RTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · INT45 contexts × 8 concurrency levels
- Peak decode
- 3,105.1 tok/s
- Peak prefill
- 34.8K tok/s
- Trinity-Mini-NVFP4RTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.21.1rc1.dev305+gc8414a827 · NVFP45 contexts × 8 concurrency levels
- Peak decode
- 2,969.9 tok/s
- Peak prefill
- 41K tok/s
- Trinity-Mini-FP8-BlockRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · FP85 contexts × 8 concurrency levels
- Peak decode
- 2,706.9 tok/s
- Peak prefill
- 39.5K tok/s
- Trinity-Mini-BaseRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 1,414.2 tok/s
- Peak prefill
- 34.3K tok/s
- Olmo-3.1-7B-RL-Zero-MathRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF164 contexts × 8 concurrency levels
- Peak decode
- 687.1 tok/s
- Peak prefill
- 23.2K tok/s
- Olmo-3.1-7B-RL-Zero-CodeRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF164 contexts × 8 concurrency levels
- Peak decode
- 686.4 tok/s
- Peak prefill
- 23.1K tok/s
- Olmo-3.1-32B-ThinkRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF164 contexts × 8 concurrency levels
- Peak decode
- 146.8 tok/s
- Peak prefill
- 5,602 tok/s
- Olmo-3.1-32B-InstructRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev151+g73a78e6f1 · BF164 contexts × 8 concurrency levels
- Peak decode
- 146.6 tok/s
- Peak prefill
- 5,618 tok/s
- Olmo-3.1-32B-Instruct-SFTRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev151+g73a78e6f1 · BF164 contexts × 8 concurrency levels
- Peak decode
- 146.7 tok/s
- Peak prefill
- 5,599 tok/s