शुरुआत करेंलीडरबोर्डDecode calculatorमॉडलOrganizationsReportsMatrix runsहार्डवेयरबेंचमार्कमार्केटप्लेसरेंटलProAPI दस्तावेज़
भाषा
Actual Computer — Every computer, one endpoint

Matrix runs

A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.

Clear

Matching runs 351

Submitted by @Sandberg. Newest first. Page 2 of 15.
  • Qwen3.8-27B-INT4RTX 3090 24GB · vllm 0.32.0 · INT4 W4A16 g1284 contexts × 8 concurrency levels
    Peak decode
    328.0 tok/s
    Peak prefill
    1,859 tok/s
  • Mistral-Medium-3.5-128B-NVFP4CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · NVFP4 mixed2 contexts × 8 concurrency levels
    Peak decode
    6.1 tok/s
    Peak prefill
    368.0 tok/s
  • Qwen3.5-27B-AWQ-4bitRTX 3090 24GB · vllm 0.32.0 · AWQ 4-bit3 contexts × 8 concurrency levels
    Peak decode
    160.2 tok/s
    Peak prefill
    1,842 tok/s
  • GLM-4.7-185B-W4A16CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · GPTQ W4A164 contexts × 8 concurrency levels
    Peak decode
    31.3 tok/s
    Peak prefill
    594.0 tok/s
  • Nemotron-Cascade-2-30B-A3B-AWQ-4bitRTX 3090 24GB · vllm 0.32.0 · AWQ 4-bit6 contexts × 8 concurrency levels
    Peak decode
    1,020.5 tok/s
    Peak prefill
    9,394 tok/s
  • Fara1.5-27B-AWQ-W4A16RTX 3090 24GB · vllm 0.32.0 · AWQ W4A164 contexts × 8 concurrency levels
    Peak decode
    326.6 tok/s
    Peak prefill
    1,849 tok/s
  • Tess-4-9BRTX 3090 24GB · vllm 0.32.0 · BF166 contexts × 8 concurrency levels
    Peak decode
    439.4 tok/s
    Peak prefill
    5,181 tok/s
  • NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · NVFP4 mixed5 contexts × 8 concurrency levels
    Peak decode
    778.0 tok/s
    Peak prefill
    1,038 tok/s
  • Tess-4-9BRTX 3090 24GB · vllm 0.32.0 · BF166 contexts × 8 concurrency levels
    Peak decode
    1,165.5 tok/s
    Peak prefill
    5,802 tok/s
  • internlm_internlm3-8b-instruct-autoround-int8-gs128-symRTX 3090 24GB · vllm 0.32.0 · AutoRound INT8 W8A16 gs1283 contexts × 8 concurrency levels
    Peak decode
    1,107.2 tok/s
    Peak prefill
    4,541 tok/s
  • Qwen3-235B-A22B-Instruct-2507-REAP-nvfp4CMP 170HX 64GB ×2 · vllm 0.32.0rc1.dev249+g18305948a · NVFP46 contexts × 8 concurrency levels
    Peak decode
    366.1 tok/s
    Peak prefill
    1,048 tok/s
  • Qwen3-0.6B-FP8RTX 3090 24GB · vllm 0.32.0 · FP83 contexts × 8 concurrency levels
    Peak decode
    1,763.9 tok/s
    Peak prefill
    36.4K tok/s
  • Qwen3-1.7B-FP8RTX 3090 24GB · vllm 0.32.0 · FP83 contexts × 8 concurrency levels
    Peak decode
    1,630.6 tok/s
    Peak prefill
    20.1K tok/s
  • granite-4.2-3b-fp8RTX 3090 24GB · vllm 0.32.0 · FP85 contexts × 8 concurrency levels
    Peak decode
    1,567.1 tok/s
    Peak prefill
    11.3K tok/s
  • Llama-3.2-3B-Instruct-FP8-dynamicRTX 3090 24GB · vllm 0.32.0 · FP8 dynamic5 contexts × 8 concurrency levels
    Peak decode
    1,494.4 tok/s
    Peak prefill
    13.1K tok/s
  • NVIDIA-Nemotron-3-Nano-4B-FP8RTX 3090 24GB · vllm 0.32.0 · FP86 contexts × 8 concurrency levels
    Peak decode
    3,175.4 tok/s
    Peak prefill
    10.8K tok/s
  • Seed-OSS-36B-Instruct-FP8-StaticCMP 170HX 64GB · vllm 0.32.0rc1.dev249+g18305948a · FP8 static7 contexts × 8 concurrency levels
    Peak decode
    559.6 tok/s
    Peak prefill
    573.0 tok/s
  • Fara1.5-4B-FP8RTX 3090 24GB · vllm 0.32.0 · FP86 contexts × 8 concurrency levels
    Peak decode
    2,155.1 tok/s
    Peak prefill
    10.8K tok/s
  • Qwen3.5-4B-AWQ-4bitRTX 3060 12GB · vllm 0.32.0 · AWQ 4-bit6 contexts × 8 concurrency levels
    Peak decode
    1,152.0 tok/s
    Peak prefill
    2,707 tok/s
  • Agents-A1-4B-FP8RTX 3090 24GB · vllm 0.32.0 · FP86 contexts × 8 concurrency levels
    Peak decode
    2,124.5 tok/s
    Peak prefill
    10.8K tok/s
  • Qwen3.5-4B-AWQ-4bitRTX 3060 12GB · vllm 0.32.0 · AWQ 4-bit6 contexts × 8 concurrency levels
    Peak decode
    737.3 tok/s
    Peak prefill
    2,557 tok/s
  • Qwen3.5-4B-FP8-dynamicRTX 3090 24GB · vllm 0.32.0 · FP8 dynamic6 contexts × 8 concurrency levels
    Peak decode
    2,099.1 tok/s
    Peak prefill
    10.3K tok/s
  • Hy-MT2-7B-FP8CMP 170HX 64GB · vllm 0.32.0rc1.dev249+g18305948a · FP86 contexts × 8 concurrency levels
    Peak decode
    573.6 tok/s
    Peak prefill
    6,528 tok/s
  • Qwen3-8B-AWQRTX 3060 12GB · vllm 0.32.0 · AWQ3 contexts × 8 concurrency levels
    Peak decode
    218.2 tok/s
    Peak prefill
    1,481 tok/s
  • Ornith-1.5-9B-NVFP4RTX 3060 12GB · vllm 0.32.0 · NVFP44 contexts × 8 concurrency levels
    Peak decode
    367.0 tok/s
    Peak prefill
    1,594 tok/s