Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
Matching runs 693
Submitted by @keennay. Newest first. Page 21 of 28.- Peak decode
- 4,021.1 tok/s
- Peak prefill
- 24.2K tok/s
- Peak decode
- 3,595.4 tok/s
- Peak prefill
- 29.9K tok/s
- Peak decode
- 872.0 tok/s
- Peak prefill
- 8,929 tok/s
- gpt-oss-safeguard-20bH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts × 8 concurrency levels
- Peak decode
- 7,201.8 tok/s
- Peak prefill
- 45.2K tok/s
- gpt-oss-safeguard-120bH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts × 8 concurrency levels
- Peak decode
- 3,931.6 tok/s
- Peak prefill
- 29.7K tok/s
- gpt-oss-20bH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts × 8 concurrency levels
- Peak decode
- 6,952.6 tok/s
- Peak prefill
- 31.7K tok/s
- gpt-oss-120bH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts × 8 concurrency levels
- Peak decode
- 3,648.8 tok/s
- Peak prefill
- 18K tok/s
- Qwen3.6-35B-A3B-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP86 contexts × 8 concurrency levels
- Peak decode
- 5,941.7 tok/s
- Peak prefill
- 34.4K tok/s
- Qwen3.6-27B-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP86 contexts × 8 concurrency levels
- Peak decode
- 2,301.0 tok/s
- Peak prefill
- 6,619 tok/s
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP88 contexts × 8 concurrency levels
- Peak decode
- 5,827.8 tok/s
- Peak prefill
- 46.3K tok/s
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP88 contexts × 8 concurrency levels
- Peak decode
- 7,888.9 tok/s
- Peak prefill
- 43.3K tok/s
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP88 contexts × 8 concurrency levels
- Peak decode
- 5,854.1 tok/s
- Peak prefill
- 43.6K tok/s
- NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF168 contexts × 8 concurrency levels
- Peak decode
- 5,929.6 tok/s
- Peak prefill
- 44.7K tok/s
- NVIDIA-Nemotron-3-Super-120B-A12B-FP8H200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP88 contexts × 8 concurrency levels
- Peak decode
- 4,357.4 tok/s
- Peak prefill
- 31.5K tok/s
- NVIDIA-Nemotron-3-Super-120B-A12B-BF16H200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · BF168 contexts × 8 concurrency levels
- Peak decode
- 1,611.9 tok/s
- Peak prefill
- 24.1K tok/s
- Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP86 contexts × 8 concurrency levels
- Peak decode
- 4,286.1 tok/s
- Peak prefill
- 35.5K tok/s
- Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 4,789.5 tok/s
- Peak prefill
- 59.2K tok/s
- Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,462.0 tok/s
- Peak prefill
- 45.3K tok/s
- Gemma-4-26B-A4B-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP46 contexts × 8 concurrency levels
- Peak decode
- 3,935.3 tok/s
- Peak prefill
- 20.1K tok/s
- Peak decode
- 2,476.2 tok/s
- Peak prefill
- 29.5K tok/s
- Nanbeige4.2-3B-BaseH200 NVL 141GB · vllm 0.1.dev20429+g62f6de733 · BF165 contexts × 8 concurrency levels
- Peak decode
- 2,511.7 tok/s
- Peak prefill
- 30.8K tok/s
- Shieldstral-1.0-3BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF163 contexts × 8 concurrency levels
- Peak decode
- 4,360.4 tok/s
- Peak prefill
- 61.6K tok/s
- Mistral-Small-4-119B-2603H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,058.0 tok/s
- Peak prefill
- 21.6K tok/s
- Mistral-Small-4-119B-2603-NVFP4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · NVFP46 contexts × 8 concurrency levels
- Peak decode
- 1,290.9 tok/s
- Peak prefill
- 14.2K tok/s
- Mistral-Medium-3.5-128BH200 NVL 141GB ×2 · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 1,215.4 tok/s
- Peak prefill
- 4,755 tok/s