keennay

Model

Qwen/Qwen3.5-0.8B-Base

Display name

Qwen3.5-0.8B-Base

Base model

—

Revision

main

Family

Qwen

Parameters

0.8B

Active params

—

MoE

no

Output

2,665 tok/s

Prefill

—

Total

55,847 tok/s

TTFT

143 ms

Peak VRAM

133GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

262144

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 NVL 141GB

GPU slots

—

GPU count

1

VRAM

141GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9175F

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev74+gfaab18955

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

4

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Qwen/Qwen3.5-0.8B-Base --served-model-name qwen3 --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --gpu-memory-utilization 0.95 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --speculative-config {"method":"qwen3_next_mtp","num_speculative_tokens":2} --no-enable-prefix-caching

Notes

Requires DeepGEMM installation after uv pip install vllm: git clone --recursive https://github.com/deepseek-ai/DeepGEMM.git cd DeepGEMM DG_FORCE_BUILD=1 uv pip install --force-reinstall --no-build-isolation . Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 143ms, Mean: 340ms

Reactions

—

Submitted

Apr 29, 2026, 8:18 PM

Last edited

—

Model

MiniMaxAI/MiniMax-M2.7

Display name

MiniMax-M2.7

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

339 tok/s

Prefill

—

Total

6,003 tok/s

TTFT

570 ms

Peak VRAM

278GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 NVL 242GB ×2

GPU slots

—

GPU count

2

VRAM

242GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9157F

RAM

3072GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev73+g6f20f81cb

Quantization

FP8

Backend

cuda

Tensor parallel

2

Pipeline parallel

—

GPU layers

4

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.98

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2.7 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 2 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.98 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 570ms, Mean: 974ms

Reactions

—

Submitted

Apr 29, 2026, 7:56 PM

Last edited

—

Model

MiniMaxAI/MiniMax-M2.5

Display name

MiniMax-M2.5

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

334 tok/s

Prefill

—

Total

5,522 tok/s

TTFT

375 ms

Peak VRAM

278GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 NVL 242GB ×2

GPU slots

—

GPU count

2

VRAM

242GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9157F

RAM

3072GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev73+g6f20f81cb

Quantization

FP8

Backend

cuda

Tensor parallel

2

Pipeline parallel

—

GPU layers

4

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.98

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2.5 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 2 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.98 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 375ms, Mean: 1038ms

Reactions

—

Submitted

Apr 29, 2026, 7:43 PM

Last edited

—

Model

MiniMaxAI/MiniMax-M2.1

Display name

MiniMax-M2.1

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

333 tok/s

Prefill

—

Total

5,600 tok/s

TTFT

642 ms

Peak VRAM

278GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 NVL 282GB ×2

GPU slots

—

GPU count

2

VRAM

282GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9157F

RAM

3072GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev73+g6f20f81cb

Quantization

FP8

Backend

cuda

Tensor parallel

2

Pipeline parallel

—

GPU layers

4

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.98

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2.1 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 2 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.98 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 642ms, Mean: 1026ms

Reactions

—

Submitted

Apr 29, 2026, 7:36 PM

Last edited

—

Model

MiniMaxAI/MiniMax-M2

Display name

MiniMax-M2

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

303 tok/s

Prefill

—

Total

6,492 tok/s

TTFT

727 ms

Peak VRAM

278GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 NVL 282GB ×2

GPU slots

—

GPU count

2

VRAM

282GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9157F

RAM

3072GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev73+g6f20f81cb

Quantization

FP8

Backend

cuda

Tensor parallel

2

Pipeline parallel

—

GPU layers

4

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.98

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 2 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.98 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 727ms, Mean: 1621ms

Reactions

—

Submitted

Apr 29, 2026, 7:23 PM

Last edited

—

Model

MiniMaxAI/MiniMax-M2.7

Display name

MiniMax-M2.7

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

496 tok/s

Prefill

—

Total

8,389 tok/s

TTFT

349 ms

Peak VRAM

535.8GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 SXM 564GB ×4

GPU slots

—

GPU count

4

VRAM

564GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

564GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev30+gf05f3664c

Quantization

FP8

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2.7 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.95 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 349ms, Mean: 933ms

Reactions

—

Submitted

Apr 29, 2026, 11:38 AM

Last edited

Apr 29, 2026, 7:57 PM

Model

MiniMaxAI/MiniMax-M2.5

Display name

MiniMax-M2.5

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

504 tok/s

Prefill

—

Total

9,781 tok/s

TTFT

368 ms

Peak VRAM

535.8GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 SXM 564GB ×4

GPU slots

—

GPU count

4

VRAM

564GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

564GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev30+gf05f3664c

Quantization

FP8

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2.5 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.95 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 368ms, Mean: 626ms

Reactions

—

Submitted

Apr 29, 2026, 11:32 AM

Last edited

Apr 29, 2026, 7:57 PM

Model

MiniMaxAI/MiniMax-M2.1

Display name

MiniMax-M2.1

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

499 tok/s

Prefill

—

Total

9,398 tok/s

TTFT

417 ms

Peak VRAM

535.8GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

7

Hardware class

DISCRETE_GPU

Hardware

H200 SXM 564GB ×4

GPU slots

—

GPU count

4

VRAM

564GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

564GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev30+gf05f3664c

Quantization

FP8

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2.1 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.95 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 417ms, Mean: 587ms

Reactions

—

Submitted

Apr 29, 2026, 11:23 AM

Last edited

Apr 29, 2026, 7:57 PM

Model

Qwen/Qwen3.5-122B-A10B-FP8

Display name

Qwen3.5-122B-A10B-FP8

Base model

Qwen3.5-122B-A10B

Revision

main

Family

Qwen

Parameters

122B

Active params

10B

MoE

no

Output

878 tok/s

Prefill

—

Total

18,413 tok/s

TTFT

491 ms

Peak VRAM

535.8GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

262144

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 SXM 564GB ×4

GPU slots

—

GPU count

4

VRAM

564GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

564GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev30+gf05f3664c

Quantization

FP8

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Qwen/Qwen3.5-122B-A10B-FP8 --served-model-name qwen3 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --gpu-memory-utilization 0.95 --speculative-config {"method":"qwen3_next_mtp","num_speculative_tokens":2} --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --speculative-config {method:qwen3_next_mtp,num_speculative_tokens:2} --no-enable-prefix-caching

Notes

Requires DeepGEMM installation after uv pip install vllm: git clone --recursive https://github.com/deepseek-ai/DeepGEMM.git cd DeepGEMM DG_FORCE_BUILD=1 uv pip install --force-reinstall --no-build-isolation . Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 491ms, Mean: 855ms

Reactions

—

Submitted

Apr 29, 2026, 11:08 AM

Last edited

Apr 29, 2026, 11:10 AM

Model

MiniMaxAI/MiniMax-M2

Display name

MiniMax-M2

Base model

—

Revision

main

Family

Minimax

Parameters

229B

Active params

—

MoE

yes

Output

493 tok/s

Prefill

—

Total

9,497 tok/s

TTFT

390 ms

Peak VRAM

535.8GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 SXM 564GB ×4

GPU slots

—

GPU count

4

VRAM

564GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

564GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev30+gf05f3664c

Quantization

FP8

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

env SAFETENSORS_FAST_GPU=1 vllm serve MiniMaxAI/MiniMax-M2 --served-model-name minimax_m2 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --gpu-memory-utilization 0.95 --no-enable-prefix-caching --host 0.0.0.0 --port 8000

Extra flags

--reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --no-enable-prefix-caching

Notes

Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 390ms, Mean: 718ms

Reactions

—

Submitted

Apr 29, 2026, 11:01 AM

Last edited

Apr 29, 2026, 7:58 PM

Model

lukealonso/GLM-5.1-NVFP4-MTP

Display name

GLM-5.1-NVFP4-MTP

Base model

GLM-5.1-NVFP4

Revision

main

Family

Opt

Parameters

433B

Active params

—

MoE

no

Output

197 tok/s

Prefill

—

Total

3,970 tok/s

TTFT

2,637 ms

Peak VRAM

535.8GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

202752

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 SXM 564GB ×4

GPU slots

—

GPU count

4

VRAM

564GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

564GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev30+gf05f3664c

Quantization

NVFP4

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8_ds_mla

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.97

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

export VLLM_NVFP4_GEMM_BACKEND=marlin vllm serve lukealonso/GLM-5.1-NVFP4-MTP \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.97 \ --kv-cache-dtype fp8_ds_mla \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 3 \ --trust-remote-code \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --served-model-name glm-5 \ --host 0.0.0.0 \ --port 8000 \ --no-enable-prefix-caching

Extra flags

--speculative-config.method mtp --speculative-config.num_speculative_tokens 3 --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice \ --no-enable-prefix-caching

Notes

Requires DeepGEMM installation after uv pip install vllm: git clone --recursive https://github.com/deepseek-ai/DeepGEMM.git cd DeepGEMM DG_FORCE_BUILD=1 uv pip install --force-reinstall --no-build-isolation . Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 2637ms, Mean: 4115ms

Reactions

—

Submitted

Apr 29, 2026, 10:50 AM

Last edited

Apr 29, 2026, 11:09 AM

Model

nvidia/GLM-5-NVFP4

Display name

GLM-5-NVFP4

Base model

GLM-5

Revision

main

Family

Opt

Parameters

—

Active params

—

MoE

no

Output

217 tok/s

Prefill

—

Total

3,522 tok/s

TTFT

1,803 ms

Peak VRAM

552GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

202752

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 SXM 564GB ×4

GPU slots

—

GPU count

4

VRAM

564GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

564GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.20.1rc1.dev23+gde3da0b97

Quantization

NVFP4

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8_ds_mla

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.97

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

export VLLM_NVFP4_GEMM_BACKEND=marlin vllm serve nvidia/GLM-5-NVFP4 \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.97 \ --kv-cache-dtype fp8_ds_mla \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 3 \ --trust-remote-code \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --served-model-name glm-5 \ --host 0.0.0.0 \ --port 8000 \ --no-enable-prefix-caching

Extra flags

--speculative-config.method mtp --speculative-config.num_speculative_tokens 3 --tool-call-parser glm47 --reasoning-parser glm45 --enable-auto-tool-choice \ --no-enable-prefix-caching

Notes

Requires DeepGEMM installation after uv pip install vllm: git clone --recursive https://github.com/deepseek-ai/DeepGEMM.git cd DeepGEMM DG_FORCE_BUILD=1 uv pip install --force-reinstall --no-build-isolation . Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py Benchmark Script Command: python benchmark.py --concurrency 10 --num-prompts 100 --label <label> --model <model> TTFT Results: Median: 1803ms, Mean: 3873ms

Reactions

—

Submitted

Apr 28, 2026, 7:33 PM

Last edited

Apr 29, 2026, 11:10 AM

Model

lukealonso/GLM-5.1-NVFP4-MTP

Display name

GLM-5.1-NVFP4-MTP

Base model

GLM-5.1-NVFP4

Revision

78b7fe365f3905b4e0261a85182fefdbd5137989

Family

Opt

Parameters

433B

Active params

—

MoE

no

Output

74.5 tok/s

Prefill

—

Total

1,531 tok/s

TTFT

20,956 ms

Peak VRAM

691GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 768GB ×8

GPU slots

—

GPU count

8

VRAM

768GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

sglang

Engine version

0.5.10.post2.dev584+gc314a13f7

Quantization

NVFP4

Backend

cuda

Tensor parallel

8

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8_e4m3

KV cache size

—

Prefix caching

—

Attention backend

nsa

Flash attention

—

Chunked prefill

yes

Prefill chunk

4096

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

modelopt_fp4

SGLang quant

—

GPU mem util

0.885

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -i \ --name sglang-glm51 \ --pull=missing \ --gpus all \ --ipc=host \ --network host \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ -v /workspace/models/huggingface:/workspace/models/huggingface \ -e HF_HOME=/workspace/models/huggingface \ -e HF_HUB_CACHE=/workspace/models/huggingface \ -e HUGGINGFACE_HUB_CACHE=/workspace/models/huggingface \ -e TRANSFORMERS_CACHE=/workspace/models/huggingface \ --entrypoint /opt/venv/bin/python3 \ voipmonitor/sglang:glm51-nsa-luke-0eb482465-b12x0101-20260428 \ -m sglang.launch_server \ --model-path lukealonso/GLM-5.1-NVFP4-MTP \ --served-model-name glm-5 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --tensor-parallel-size 8 \ --quantization modelopt_fp4 \ --disable-piecewise-cuda-graph \ --load-format instanttensor \ --kv-cache-dtype fp8_e4m3 \ --trust-remote-code \ --enable-pcie-oneshot-allreduce \ --disable-shared-experts-fusion \ --nsa-prefill-backend b12x \ --nsa-decode-backend b12x \ --page-size 64 \ --attention-backend nsa \ --moe-runner-backend b12x \ --fp4-gemm-backend b12x \ --cuda-graph-max-bs 8 \ --chunked-prefill-size 4096 \ --preferred-sampling-params '{"temperature": 1.0, "top_p": 0.95}' \ --json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}' \ --max-running-requests 8 \ --mem-fraction-static 0.885 \ --host 0.0.0.0 \ --port 8000 \ --api-key YOUR_API_KEY

Extra flags

--rm --i \ --name sglang-glm51 --pull missing --gpus all --ipc host --network host --ulimit memlock=-1 --ulimit stack=67108864 --e HF_HOME=/workspace/models/huggingface --e HF_HUB_CACHE=/workspace/models/huggingface --e HUGGINGFACE_HUB_CACHE=/workspace/models/huggingface --e TRANSFORMERS_CACHE=/workspace/models/huggingface --entrypoint /opt/venv/bin/python3 --model-path lukealonso/GLM-5.1-NVFP4-MTP --reasoning-parser glm45 --tool-call-parser glm47 --disable-piecewise-cuda-graph \ --load-format instanttensor --enable-pcie-oneshot-allreduce \ --disable-shared-experts-fusion \ --nsa-prefill-backend b12x --nsa-decode-backend b12x --page-size 64 --moe-runner-backend b12x --fp4-gemm-backend b12x --cuda-graph-max-bs 8 --preferred-sampling-params {"temperature": 1.0, "top_p": 0.95} --json-model-override-args {"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}

Notes

Mean TTFT: 25992ms, Median TTFT: 20956ms Benchmark script: https://github.com/keennay/gpu-cluster-setup/blob/master/benchmark.py

Reactions

—

Submitted

Apr 28, 2026, 5:33 AM

Last edited

—

Model

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Display name

NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Base model

—

Revision

main

Family

Opt

Parameters

120B

Active params

12B

MoE

yes

Output

183 tok/s

Prefill

—

Total

4,781 tok/s

TTFT

948 ms

Peak VRAM

91.0GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

256000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 96GB

GPU slots

—

GPU count

1

VRAM

96GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

1500GB

OS

Ubuntu 24.04

Power

—

Engine

vLLM

Engine version

0.18.1

Quantization

NVFP4

Backend

CUDA

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

—

Attention backend

TRITON_ATTN

Flash attention

—

Chunked prefill

yes

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

512

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path> \ --async-scheduling \ --served-model-name nemotron-3-super-nvfp4 \ --dtype auto \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 \ --pipeline-parallel-size 1 \ --data-parallel-size 1 \ --trust-remote-code \ --attention-backend TRITON_ATTN \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --max-num-seqs 512 \ --host 0.0.0.0 \ --port 8000 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path>/super_v3_reasoning_parser.py \ --reasoning-parser super_v3

Extra flags

--async-scheduling \ --data-parallel-size 1 --enable-auto-tool-choice \ --tool-call-parser qwen3_coder --reasoning-parser-plugin <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path>/super_v3_reasoning_parser.py --reasoning-parser super_v3

Notes

Mean TTFT: 1799ms, Median TTFT: 948ms

Reactions

—

Submitted

Apr 25, 2026, 8:27 PM

Last edited

—

Model

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Display name

NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Base model

—

Revision

main

Family

Opt

Parameters

120B

Active params

12B

MoE

yes

Output

175 tok/s

Prefill

—

Total

3,840 tok/s

TTFT

1,374 ms

Peak VRAM

131.5GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

256000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 NVL 141GB

GPU slots

—

GPU count

1

VRAM

141GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9175F

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vLLM

Engine version

0.18.1

Quantization

NVFP4

Backend

—

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

—

Attention backend

TRITON_ATTN

Flash attention

—

Chunked prefill

yes

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

512

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

export VLLM_USE_NVFP4_CT_EMULATIONS=1 vllm serve <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path> \ --async-scheduling \ --served-model-name nemotron-3-super-nvfp4 \ --dtype auto \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 \ --pipeline-parallel-size 1 \ --data-parallel-size 1 \ --trust-remote-code \ --attention-backend TRITON_ATTN \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --max-num-seqs 512 \ --host 0.0.0.0 \ --port 8000 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path>/super_v3_reasoning_parser.py \ --reasoning-parser super_v3 \ -cc '{"cudagraph_mode":0}'

Extra flags

--async-scheduling \ --data-parallel-size 1 --enable-auto-tool-choice \ --tool-call-parser qwen3_coder --reasoning-parser-plugin <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path>/super_v3_reasoning_parser.py --reasoning-parser super_v3 --cc {"cudagraph_mode":0}

Notes

Mean TTFT: 2849ms, Median TTFT: 1374ms, NVFP4 GEMM=Emulated

Reactions

—

Submitted

Apr 25, 2026, 8:05 PM

Last edited

—

Model

nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Display name

NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Base model

—

Revision

main

Family

Opt

Parameters

120B

Active params

12B

MoE

yes

Output

262 tok/s

Prefill

—

Total

4,720 tok/s

TTFT

1,091 ms

Peak VRAM

129.7GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

256000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

H200 NVL 141GB

GPU slots

—

GPU count

1

VRAM

141GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9175F

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vLLM

Engine version

0.18.1

Quantization

NVFP4

Backend

CUDA

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

—

Attention backend

TRITON_ATTN

Flash attention

—

Chunked prefill

yes

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

512

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path> \ --async-scheduling \ --served-model-name nemotron-3-super-nvfp4 \ --dtype auto \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 \ --pipeline-parallel-size 1 \ --data-parallel-size 1 \ --trust-remote-code \ --attention-backend TRITON_ATTN \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --max-num-seqs 512 \ --host 0.0.0.0 \ --port 8000 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path>/super_v3_reasoning_parser.py \ --reasoning-parser super_v3

Extra flags

--async-scheduling \ --data-parallel-size 1 --enable-auto-tool-choice \ --tool-call-parser qwen3_coder --reasoning-parser-plugin <NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4-Path>/super_v3_reasoning_parser.py --reasoning-parser super_v3

Notes

Median TTFT: 2281ms, Mean TTFT: 1091ms, NVFP4 GEMM=Marlin

Reactions

—

Submitted

Apr 25, 2026, 7:59 PM

Last edited

—

Model

moonshotai/Kimi-K2.5

Display name

Kimi-K2.5

Base model

—

Revision

main

Family

—

Parameters

1T

Active params

—

MoE

no

Output

74.0 tok/s

Prefill

—

Total

1,381 tok/s

TTFT

3,710 ms

Peak VRAM

748.9GB

Power draw

—

Hardware cost

—

Prompt tokens

45000

Output tokens

3000

Prefill tokens

—

Context length

200000

Batch size

10

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 768GB ×8

GPU slots

—

GPU count

8

VRAM

768GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 9555

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

SGLang

Engine version

0.5.9

Quantization

INT4

Backend

—

Tensor parallel

8

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

sglang serve --model-path moonshotai/Kimi-K2.5 --tp 8 --trust-remote-code --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --mem-fraction-static 0.90 --disable-radix-cache --host 0.0.0.0 --port $INFERENCE_PORT

Extra flags

--model-path moonshotai/Kimi-K2.5 --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --disable-radix-cache

Notes

Mean TTFT: 9001ms, Median TTFT: 3710ms

Reactions

—

Submitted

Apr 25, 2026, 7:51 PM

Last edited

—