Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
All runs 740
Newest first. Page 29 of 30.- Qwen3.5-35B-A3B-BaseH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,351.2 tok/s
- Peak prefill
- 36.5K tok/s
- Qwen3.5-2BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 12.1K tok/s
- Peak prefill
- 77.3K tok/s
- Qwen3.5-2BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 12.3K tok/s
- Peak prefill
- 80.6K tok/s
- Qwen3.5-2B-BaseH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 5,110.7 tok/s
- Peak prefill
- 77.5K tok/s
- Qwen3.5-2B-BaseH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 12.4K tok/s
- Peak prefill
- 79.8K tok/s
- Qwen3.5-27BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,843.2 tok/s
- Peak prefill
- 9,631 tok/s
- Qwen3.5-27BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,183.1 tok/s
- Peak prefill
- 9,731 tok/s
- Qwen3.5-27B-GPTQ-Int4H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · INT46 contexts × 8 concurrency levels
- Peak decode
- 1,910.6 tok/s
- Peak prefill
- 6,160 tok/s
- Qwen3.5-27B-GPTQ-Int4H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · INT46 contexts × 8 concurrency levels
- Peak decode
- 1,223.8 tok/s
- Peak prefill
- 6,236 tok/s
- Qwen3.5-27B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,263.4 tok/s
- Peak prefill
- 13.3K tok/s
- Qwen3.5-27B-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,582.8 tok/s
- Peak prefill
- 13.7K tok/s
- Qwen3.5-122B-A10B-GPTQ-Int4H200 NVL 141GB · sglang 0.5.6.post2 · INT46 contexts × 8 concurrency levels
- Peak decode
- 571.4 tok/s
- Peak prefill
- 12.8K tok/s
- Qwen3.5-122B-A10B-GPTQ-Int4H200 NVL 141GB · sglang 0.5.6.post2 · INT46 contexts × 8 concurrency levels
- Peak decode
- 2,006.6 tok/s
- Peak prefill
- 13.2K tok/s
- Qwen3.5-122B-A10B-FP8H200 NVL 141GB ×2 · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · FP86 contexts × 8 concurrency levels
- Peak decode
- 4,794.2 tok/s
- Peak prefill
- 28.4K tok/s
- Qwen3.5-0.8BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 12.3K tok/s
- Peak prefill
- 99K tok/s
- Qwen3.5-0.8BH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 16K tok/s
- Peak prefill
- 98.7K tok/s
- Qwen3.5-0.8B-BaseH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 5,199.7 tok/s
- Peak prefill
- 98.2K tok/s
- Qwen3.5-0.8B-BaseH200 NVL 141GB · sglang 0.5.6.post3.dev9160+g1cf2b8c54 · BF166 contexts × 8 concurrency levels
- Peak decode
- 15.7K tok/s
- Peak prefill
- 104.3K tok/s
- INTELLECT-3H200 NVL 141GB ×2 · sglang 0.5.6.post3.dev9335+gdd15fb57b · BF165 contexts × 8 concurrency levels
- Peak decode
- 607.4 tok/s
- Peak prefill
- 16.3K tok/s
- INTELLECT-3.1H200 NVL 141GB ×2 · sglang 0.5.6.post3.dev9335+gdd15fb57b · BF165 contexts × 8 concurrency levels
- Peak decode
- 604.2 tok/s
- Peak prefill
- 16.4K tok/s
- INTELLECT-2H200 NVL 141GB · sglang 0.5.6.post3.dev9335+gdd15fb57b · FP164 contexts × 8 concurrency levels
- Peak decode
- 481.2 tok/s
- Peak prefill
- 7,964 tok/s
- Laguna-XS.2H200 NVL 141GB · sglang 0.5.6.post3.dev9269+g834400705 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,861.9 tok/s
- Peak prefill
- 34.8K tok/s
- Laguna-XS.2-INT4H200 NVL 141GB · sglang 0.5.6.post3.dev9269+g834400705 · INT4+INT86 contexts × 8 concurrency levels
- Peak decode
- 2,619.0 tok/s
- Peak prefill
- 32.9K tok/s
- Laguna-XS.2-FP8H200 NVL 141GB · sglang 0.5.6.post3.dev9269+g834400705 · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,419.2 tok/s
- Peak prefill
- 41.6K tok/s
- Laguna-XS-2.1H200 NVL 141GB · sglang 0.5.6.post3.dev9269+g834400705 · BF166 contexts × 8 concurrency levels
- Peak decode
- 1,832.7 tok/s
- Peak prefill
- 34.9K tok/s