Matrix runs
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Run one with lmx speed-test matrix run.
Matching runs 693
Submitted by @keennay. Newest first. Page 19 of 28.- Peak decode
- 6,571.6 tok/s
- Peak prefill
- 31.7K tok/s
- Qwen3.5-9B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 6,561.1 tok/s
- Peak prefill
- 29.7K tok/s
- Qwen3.5-9B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 6,570.2 tok/s
- Peak prefill
- 32K tok/s
- Peak decode
- 8,578.7 tok/s
- Peak prefill
- 45.5K tok/s
- Peak decode
- 7,436.0 tok/s
- Peak prefill
- 48.3K tok/s
- Qwen3.5-4B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 7,196.3 tok/s
- Peak prefill
- 46.6K tok/s
- Qwen3.5-4B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 7,441.6 tok/s
- Peak prefill
- 46.8K tok/s
- Qwen3.5-35B-A3BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 7,082.0 tok/s
- Peak prefill
- 41.1K tok/s
- Qwen3.5-35B-A3BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 5,284.1 tok/s
- Peak prefill
- 39.3K tok/s
- Qwen3.5-35B-A3B-GPTQ-Int4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT46 contexts × 8 concurrency levels
- Peak decode
- 1,829.5 tok/s
- Peak prefill
- 7,807 tok/s
- Qwen3.5-35B-A3B-GPTQ-Int4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT46 contexts × 8 concurrency levels
- Peak decode
- 1,037.9 tok/s
- Peak prefill
- 8,035 tok/s
- Qwen3.5-35B-A3B-FP8H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 7,955.5 tok/s
- Peak prefill
- 44K tok/s
- Qwen3.5-35B-A3B-FP8H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 6,535.7 tok/s
- Peak prefill
- 43.5K tok/s
- Qwen3.5-35B-A3B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 5,905.2 tok/s
- Peak prefill
- 41.3K tok/s
- Qwen3.5-35B-A3B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 4,919.3 tok/s
- Peak prefill
- 39.7K tok/s
- Peak decode
- 15.2K tok/s
- Peak prefill
- 90.1K tok/s
- Peak decode
- 15.5K tok/s
- Peak prefill
- 91.7K tok/s
- Qwen3.5-2B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 11.5K tok/s
- Peak prefill
- 87K tok/s
- Qwen3.5-2B-BaseH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 15.4K tok/s
- Peak prefill
- 96K tok/s
- Qwen3.5-27BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 3,103.9 tok/s
- Peak prefill
- 10.3K tok/s
- Qwen3.5-27BH200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · BF166 contexts × 8 concurrency levels
- Peak decode
- 2,071.0 tok/s
- Peak prefill
- 10.4K tok/s
- Qwen3.5-27B-GPTQ-Int4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT46 contexts × 8 concurrency levels
- Peak decode
- 3,104.6 tok/s
- Peak prefill
- 10.1K tok/s
- Qwen3.5-27B-GPTQ-Int4H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · INT46 contexts × 8 concurrency levels
- Peak decode
- 2,283.8 tok/s
- Peak prefill
- 10K tok/s
- Qwen3.5-27B-FP8H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 3,576.0 tok/s
- Peak prefill
- 14.3K tok/s
- Qwen3.5-27B-FP8H200 NVL 141GB · vllm 0.26.1rc1.dev638+g52be12cfa · FP86 contexts × 8 concurrency levels
- Peak decode
- 2,576.1 tok/s
- Peak prefill
- 14.6K tok/s