Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
Matching runs 693
Submitted by @keennay. Newest first. Page 22 of 28.- Mistral-Medium-3.5-128BH200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 647.5 tok/s
- Peak prefill
- 4,837 tok/s
- Ministral-3-8B-Reasoning-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,125.9 tok/s
- Peak prefill
- 18.8K tok/s
- Ministral-3-8B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 3,440.2 tok/s
- Peak prefill
- 26.2K tok/s
- Ministral-3-8B-Base-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,161.1 tok/s
- Peak prefill
- 30.6K tok/s
- Ministral-3-3B-Reasoning-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,953.3 tok/s
- Peak prefill
- 28.4K tok/s
- Ministral-3-3B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 4,312.9 tok/s
- Peak prefill
- 38.5K tok/s
- Ministral-3-3B-Base-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,346.0 tok/s
- Peak prefill
- 58.9K tok/s
- Ministral-3-14B-Reasoning-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,755.7 tok/s
- Peak prefill
- 13.9K tok/s
- Ministral-3-14B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,079.1 tok/s
- Peak prefill
- 20.2K tok/s
- Ministral-3-14B-Base-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,905.1 tok/s
- Peak prefill
- 20.3K tok/s
- Devstral-Small-2-24B-Instruct-2512H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,750.2 tok/s
- Peak prefill
- 14K tok/s
- Devstral-2-123B-Instruct-2512H200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 648.2 tok/s
- Peak prefill
- 4,984 tok/s
- Peak decode
- 6,566.4 tok/s
- Peak prefill
- 32.4K tok/s
- Peak decode
- 7,437.9 tok/s
- Peak prefill
- 48.5K tok/s
- Fara1.5-27BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 2,070.3 tok/s
- Peak prefill
- 10.4K tok/s
- LFM2.5-VL-3BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF163 contexts × 8 concurrency levels
- Peak decode
- 14.9K tok/s
- Peak prefill
- 69.4K tok/s
- LFM2.5-350MH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 24.3K tok/s
- Peak prefill
- 171.8K tok/s
- LFM2.5-230MH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 13.3K tok/s
- Peak prefill
- 192.2K tok/s
- LFM2.5-2.6BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 12.7K tok/s
- Peak prefill
- 66.2K tok/s
- LFM2.5-1.2B-InstructH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF165 contexts × 8 concurrency levels
- Peak decode
- 22.2K tok/s
- Peak prefill
- 108.8K tok/s
- Qwen3.6-35B-A3B-int4-mixed-AutoRoundH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts × 8 concurrency levels
- Peak decode
- 7,320.8 tok/s
- Peak prefill
- 35.1K tok/s
- Qwen3.6-35B-A3B-int4-mixed-AutoRoundH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts × 8 concurrency levels
- Peak decode
- 5,966.8 tok/s
- Peak prefill
- 35.1K tok/s
- Qwen3.6-27B-int4-AutoRoundH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts × 8 concurrency levels
- Peak decode
- 2,879.6 tok/s
- Peak prefill
- 9,975 tok/s
- Qwen3.6-27B-int4-AutoRoundH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts × 8 concurrency levels
- Peak decode
- 2,904.9 tok/s
- Peak prefill
- 9,928 tok/s
- gemma-4-31B-it-int4-AutoRound-V2H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT4+BF166 contexts × 8 concurrency levels
- Peak decode
- 1,219.6 tok/s
- Peak prefill
- 7,966 tok/s