Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
Matching runs 693
Submitted by @keennay. Newest first. Page 3 of 28.- MiniMax-M2.1RTX PRO 6000 Blackwell Server Edition 96GB ×4 · vllm 0.1.dev1+ge0317b28c · FP86 contexts × 8 concurrency levels
- Peak decode
- 612.7 tok/s
- Peak prefill
- 15.2K tok/s
- MiniMax-M2.5RTX PRO 6000 Blackwell Server Edition 96GB ×4 · vllm 0.1.dev1+ge0317b28c · FP86 contexts × 8 concurrency levels
- Peak decode
- 922.3 tok/s
- Peak prefill
- 15.1K tok/s
- MiniMax-M2.7RTX PRO 6000 Blackwell Server Edition 96GB ×4 · vllm 0.1.dev1+ge0317b28c · FP86 contexts × 8 concurrency levels
- Peak decode
- 629.4 tok/s
- Peak prefill
- 15.3K tok/s
- Nex-N2-miniRTX PRO 6000 Blackwell Server Edition 96GB ×2 · vllm 0.1.dev1+ge0317b28c · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,420.1 tok/s
- Peak prefill
- 30.9K tok/s
- Step-3.7-Flash-FP8RTX PRO 6000 Blackwell Server Edition 96GB ×4 · sglang 0.0.0.dev1+gf92199b56 · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,310.0 tok/s
- Peak prefill
- 15.5K tok/s
- Step-3.7-Flash-NVFP4RTX PRO 6000 Blackwell Server Edition 96GB ×2 · sglang 0.0.0.dev1+gf92199b56 · NVFP46 contexts × 8 concurrency levels
- Peak decode
- 722.6 tok/s
- Peak prefill
- 15.9K tok/s
- Ling-3.0-tinyRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 4,912.1 tok/s
- Peak prefill
- 55.2K tok/s
- Ling-3.0-tiny-int4RTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · INT45 contexts × 8 concurrency levels
- Peak decode
- 6,091.1 tok/s
- Peak prefill
- 57.5K tok/s
- Ling-3.0-tiny-fp8RTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · FP85 contexts × 8 concurrency levels
- Peak decode
- 5,377.7 tok/s
- Peak prefill
- 60.5K tok/s
- Ling-3.0-tiny-baseRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,471.3 tok/s
- Peak prefill
- 55.5K tok/s
- Ling-3.0-tiny-base-midtrainRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 2,563.8 tok/s
- Peak prefill
- 55.9K tok/s
- Ling-3.0-tinyRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · BF165 contexts × 8 concurrency levels
- Peak decode
- 6,109.2 tok/s
- Peak prefill
- 23.5K tok/s
- Ling-3.0-tiny-int4RTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · INT45 contexts × 8 concurrency levels
- Peak decode
- 5,636.2 tok/s
- Peak prefill
- 23.6K tok/s
- Ling-3.0-tiny-fp8RTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · FP85 contexts × 8 concurrency levels
- Peak decode
- 6,226.5 tok/s
- Peak prefill
- 23.1K tok/s
- Ling-3.0-tiny-baseRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · BF166 contexts × 8 concurrency levels
- Peak decode
- 7,051.4 tok/s
- Peak prefill
- 26.3K tok/s
- Ling-3.0-tiny-base-midtrainRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · BF166 contexts × 8 concurrency levels
- Peak decode
- 7,054.6 tok/s
- Peak prefill
- 26.1K tok/s
- Ling-3.0-flash-int4RTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · INT46 contexts × 8 concurrency levels
- Peak decode
- 572.1 tok/s
- Peak prefill
- 15.1K tok/s
- Ling-3.0-flash-fp4RTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · MXFP4+FP86 contexts × 8 concurrency levels
- Peak decode
- 888.4 tok/s
- Peak prefill
- 19.9K tok/s
- Ling-2.6-flash-int4RTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev236+g1093c501d · INT45 contexts × 8 concurrency levels
- Peak decode
- 988.4 tok/s
- Peak prefill
- 13K tok/s
- Muse-Glimmer-30BRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 1,022.7 tok/s
- Peak prefill
- 6,383 tok/s
- Muse-Glimmer-30BRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF165 contexts × 8 concurrency levels
- Peak decode
- 759.7 tok/s
- Peak prefill
- 6,382 tok/s
- Muse-Glimmer-30BRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev220+gfa7784d14 · BF165 contexts × 8 concurrency levels
- Peak decode
- 766.2 tok/s
- Peak prefill
- 6,729 tok/s
- Muse-Glimmer-30BRTX PRO 6000 Blackwell Server Edition 96GB · sglang 0.5.22.dev220+gfa7784d14 · BF165 contexts × 8 concurrency levels
- Peak decode
- 102.8 tok/s
- Peak prefill
- 6,028 tok/s
- Fara1.5-9BRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,887.0 tok/s
- Peak prefill
- 21.5K tok/s
- Fara1.5-4BRTX PRO 6000 Blackwell Server Edition 96GB · vllm 0.30.1rc1.dev452+g73a583112 · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,786.2 tok/s
- Peak prefill
- 34.6K tok/s