jacklarmer

Model

deepseek-ai/DeepSeek-V4-Flash-0731

Display name

DeepSeek-V4-Flash-0731

Base model

—

Revision

main

Family

Deepseek

Parameters

304B

Active params

—

MoE

yes

Output tok/s

345.8

Prefill tok/s

618.8

Total tok/s

369.6

TTFT

137.4ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

85

Output tokens

512

Prefill tokens

—

Context length

2048

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 96GB ×4

GPU slots

—

GPU count

4

VRAM

96GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen Threadripper 9960X 24-Cores

RAM

64GB

OS

linux

Power

—

Engine

vllm

Engine version

—

Quantization

FP8

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

no

Attention backend

B12X_MLA_SPARSE

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

dspark

Spec model

—

Spec draft model

—

Spec tokens

5

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker compose up -d # vLLM 0.11.2.dev279+...b12x7bfc945.cu132 (SparkInfer/B12X 0.30.0) vllm serve /model --served-model-name DeepSeek-V4-Flash-0731 --trust-remote-code \ --tensor-parallel-size 4 --kv-cache-dtype fp8_ds_mla --moe-backend b12x \ --load-format safetensors -cc.pass_config.fuse_allreduce_rms=True \ --gpu-memory-utilization 0.90 --max-model-len 262144 --max-num-seqs 8 \ --max-num-batched-tokens 8192 --block-size 256 --max-cudagraph-capture-size 48 \ --async-scheduling --enable-chunked-prefill --enable-auto-tool-choice \ --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ --speculative-config '{"method":"dspark","num_speculative_tokens":5,"moe_backend":"b12x","draft_sample_method":"probabilistic"}' env: CUTE_DSL_ARCH=sm_120a VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_B12X_MOE=1 VLLM_USE_B12X_SPARSE_INDEXER=1 VLLM_USE_B12X_FP8_GEMM=0 B12X_DENSE_SPLITK_TURBO=1 B12X_MLA_SM120_PREFILL_F16_ROPE=1 NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=SYS NCCL_PROTO=LL,LL128,Simple VLLM_ENABLE_PCIE_ALLREDUCE=0 Notes for anyone reproducing on 4x RTX PRO 6000 Blackwell (SM120): * --block-size 16 is INVALID for the DS-MLA layout: fails several minutes into boot with assert max(sm_page_sizes) <= max(all_page_sizes) in kv_cache_utils.py. Use 256. * DSpark requires decode-context-parallel-size 1. GLM-5.2 profiles use DCP4; DeepSeek cannot. * DeepseekV4 ds_mla accepts fp8_ds_mla only; nvfp4_ds_mla asserts. NVFP4 KV is unsupported on SM120 entirely (vLLM #43562 - trtllm-gen FP4 FMHA is SM100/103 only). * VLLM_USE_B12X_FP8_GEMM must be 0 with DSpark or deep_gemm fp8_einsum hits a layout assert. * --attention-backend B12X_MLA_SPARSE is NOT used: SparkInfer 0.30.0's DSV4 topk-128 MG prefill dispatcher passes scale_format= to io_issue_gather_dsv4_nope(), which rejects it. * Threadripper PRO topology deadlocks on PCIe P2P; NCCL_P2P_DISABLE=1 is mandatory. * Tuning result: breakable_cudagraph=0, moe_force_a16=1, wo_projection=1, gpu_mem_util=0.95 and max_num_seqs=16 were each measured and each LOST to this baseline (347.8-354.8 vs 357.9 on code@temp0). The +28.6% breakable-cudagraph win reported on GB10/SM121 does not transfer to SM120. * Workload dominates every flag. Measured on this exact server with the lmx harness (2 warmups + 5 timed iterations, median) varying ONLY the prompt: default lmx prompt (25 tok, generic prose) -> 225.0 tok/s code-generation prompt (85 tok, this run) -> 345.8 tok/s A separate harness on a comparable code prompt independently measured 345.4 tok/s. With DSpark, tok/s ~= steps/s * (1 + accepted draft tokens); acceptance is far higher on deterministic code than on open-ended prose. THIS SUBMISSION USED THE CODE PROMPT. Compare against entries that used the default prompt accordingly.

Extra flags

SparkInfer/B12X 0.30.0 MoE backend; --block-size 256; --max-cudagraph-capture-size 48; --async-scheduling; CUTE_DSL_ARCH=sm_120a; NCCL_P2P_DISABLE=1 (PCIe-only host, no P2P)

Notes

—

Reactions

🚀

Submitted

Aug 1, 2026, 8:48 PM

Last edited

—

Model

deepseek-ai/DeepSeek-V4-Flash-0731

Display name

DeepSeek-V4-Flash-0731

Base model

—

Revision

main

Family

Deepseek

Parameters

304B

Active params

—

MoE

yes

Output tok/s

225

Prefill tok/s

490.1

Total tok/s

231.3

TTFT

51.0ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

25

Output tokens

512

Prefill tokens

—

Context length

2048

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 96GB ×4

GPU slots

—

GPU count

4

VRAM

96GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen Threadripper 9960X 24-Cores

RAM

64GB

OS

linux

Power

—

Engine

vllm

Engine version

—

Quantization

FP8

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

no

Attention backend

B12X_MLA_SPARSE

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

dspark

Spec model

—

Spec draft model

—

Spec tokens

5

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker compose up -d # vLLM 0.11.2.dev279+...b12x7bfc945.cu132 (SparkInfer/B12X 0.30.0) vllm serve /model --served-model-name DeepSeek-V4-Flash-0731 --trust-remote-code \ --tensor-parallel-size 4 --kv-cache-dtype fp8_ds_mla --moe-backend b12x \ --load-format safetensors -cc.pass_config.fuse_allreduce_rms=True \ --gpu-memory-utilization 0.90 --max-model-len 262144 --max-num-seqs 8 \ --max-num-batched-tokens 8192 --block-size 256 --max-cudagraph-capture-size 48 \ --async-scheduling --enable-chunked-prefill --enable-auto-tool-choice \ --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \ --speculative-config '{"method":"dspark","num_speculative_tokens":5,"moe_backend":"b12x","draft_sample_method":"probabilistic"}' env: CUTE_DSL_ARCH=sm_120a VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_B12X_MOE=1 VLLM_USE_B12X_SPARSE_INDEXER=1 VLLM_USE_B12X_FP8_GEMM=0 B12X_DENSE_SPLITK_TURBO=1 B12X_MLA_SM120_PREFILL_F16_ROPE=1 NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=SYS NCCL_PROTO=LL,LL128,Simple VLLM_ENABLE_PCIE_ALLREDUCE=0 Notes for anyone reproducing on 4x RTX PRO 6000 Blackwell (SM120): * --block-size 16 is INVALID for the DS-MLA layout: fails several minutes into boot with assert max(sm_page_sizes) <= max(all_page_sizes) in kv_cache_utils.py. Use 256. * DSpark requires decode-context-parallel-size 1. GLM-5.2 profiles use DCP4; DeepSeek cannot. * DeepseekV4 ds_mla accepts fp8_ds_mla only; nvfp4_ds_mla asserts. NVFP4 KV is unsupported on SM120 entirely (vLLM #43562 - trtllm-gen FP4 FMHA is SM100/103 only). * VLLM_USE_B12X_FP8_GEMM must be 0 with DSpark or deep_gemm fp8_einsum hits a layout assert. * --attention-backend B12X_MLA_SPARSE is NOT used: SparkInfer 0.30.0's DSV4 topk-128 MG prefill dispatcher passes scale_format= to io_issue_gather_dsv4_nope(), which rejects it. * Threadripper PRO topology deadlocks on PCIe P2P; NCCL_P2P_DISABLE=1 is mandatory. * Tuning result: breakable_cudagraph=0, moe_force_a16=1, wo_projection=1, gpu_mem_util=0.95 and max_num_seqs=16 were each measured and each LOST to this baseline (347.8-354.8 vs 357.9 on code@temp0). The +28.6% breakable-cudagraph win reported on GB10/SM121 does not transfer to SM120. * Workload dominates every flag: code@temp0 ~345-358 tok/s vs prose ~250-283 tok/s on this exact server. Draft acceptance is far higher on code.

Extra flags

SparkInfer/B12X 0.30.0 MoE backend; --block-size 256; --max-cudagraph-capture-size 48; --async-scheduling; CUTE_DSL_ARCH=sm_120a; NCCL_P2P_DISABLE=1 (PCIe-only host, no P2P)

Notes

—

Reactions

—

Submitted

Aug 1, 2026, 8:26 PM

Last edited

—

Model

runrunway/Kimi-K3-REAP-224experts

Display name

Kimi-K3-REAP-224experts

Base model

Kimi-K3

Revision

56d996a3b496df7641c428130c07174ebaa36c24

Family

—

Parameters

2780B

Active params

—

MoE

yes

Output tok/s

2.2

Prefill tok/s

97.3

Total tok/s

8.0

TTFT

1849.9ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

180

Output tokens

64

Prefill tokens

—

Context length

4096

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 768GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

GB10 Grace Blackwell

Unified memory

768GB

NPU TOPS

—

CPU

—

RAM

—

OS

Ubuntu 24.04 / Ray 2.56.1 / QSFP fabric

Power

—

Engine

vllm

Engine version

—

Quantization

MXFP4

Backend

cuda

Tensor parallel

2

Pipeline parallel

3

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.85

Max running seqs

1

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve runrunway/Kimi-K3-REAP-224experts --tensor-parallel-size 2 --pipeline-parallel-size 3 --enable-expert-parallel --max-model-len 4096 --gpu-memory-utilization 0.85 --enable-prefix-caching

Extra flags

--enable-expert-parallel --max-model-len 4096

Notes

Six NVIDIA GB10 unified-memory Spark nodes (ee76, f041, b0a8, f283, bf93, cbf2), 6x128 GB = 768 GB total, joined over QSFP. Ray 2.56.1 with vLLM image riverhouse/vllm-kimi-k3-ray:2.56.1-d61e; TP2 x PP3, expert parallel, prefix caching, GPU util 0.85, DeepGEMM disabled (Marlin MXFP4 fallback). Screening decode: 1 warmup + 3 measured trials, concurrency 1, 180 prompt tokens, 64 output-token cap, temperature 0/top_p 1, context 4096. Metrics are medians: 2.1857 output tok/s, 97.3028 prefill tok/s, 7.9537 total tok/s, 1849.9 ms TTFT. All measured trials ended at the output cap (finish_reason=length). Acceptance harness caveat: forced tool choice and cancellation cases failed; core harness status FAIL_CORE, so this is a performance result, not a full compatibility claim.

Reactions

—

Submitted

Aug 1, 2026, 7:16 PM

Last edited

—

Model

deepseek-ai/DeepSeek-V4-Flash-0731

Display name

DeepSeek-V4-Flash-0731

Base model

—

Revision

main

Family

Deepseek

Parameters

304B

Active params

—

MoE

yes

Output tok/s

231.5

Prefill tok/s

497.9

Total tok/s

237.9

TTFT

50.2ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

25

Output tokens

512

Prefill tokens

—

Context length

2048

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 96GB ×4

GPU slots

—

GPU count

4

VRAM

96GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen Threadripper 9960X 24-Cores

RAM

64GB

OS

linux

Power

—

Engine

vllm

Engine version

—

Quantization

MXFP4

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

dspark

Spec model

—

Spec draft model

—

Spec tokens

5

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker compose up -d # image riverhouse/glm52-hybrid:v1.4-cfe43b0 (vLLM 0.11.2.dev279+eldritch.enlightenment...b12x7bfc945.cu132, b12x/sparkinfer 0.30.0) vllm serve /model --served-model-name DeepSeek-V4-Flash-0731 --trust-remote-code --tensor-parallel-size 4 --kv-cache-dtype fp8_ds_mla --moe-backend b12x --load-format safetensors -cc.pass_config.fuse_allreduce_rms=True --gpu-memory-utilization 0.90 --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 8192 --max-cudagraph-capture-size 48 --async-scheduling --enable-chunked-prefill --enable-auto-tool-choice --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 --speculative-config '{"method":"dspark","num_speculative_tokens":5,"moe_backend":"b12x","draft_sample_method":"probabilistic"}' # env: CUTE_DSL_ARCH=sm_120a VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_B12X_MOE=1 VLLM_USE_B12X_SPARSE_INDEXER=1 VLLM_USE_B12X_FP8_GEMM=0 B12X_DENSE_SPLITK_TURBO=1 B12X_MLA_SM120_PREFILL_F16_ROPE=1 NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=SYS NCCL_PROTO=LL,LL128,Simple VLLM_ENABLE_PCIE_ALLREDUCE=0 # Notes: Threadripper PRO topology deadlocks on PCIe P2P, so P2P is disabled. b12x MoE is used; b12x MLA (B12X_MLA_SPARSE) is NOT usable for DeepSeek-V4 in b12x 0.30.0 (DSV4 topk-128 MG prefill dispatcher passes scale_format= to io_issue_gather_dsv4_nope(), which rejects it). VLLM_USE_B12X_FP8_GEMM must be 0 with DSpark or deep_gemm fp8_einsum hits a layout assertion during drafter warmup. DSpark requires decode-context-parallel-size 1. DeepseekV4 ds_mla accepts fp8_ds_mla only, not nvfp4_ds_mla.

Extra flags

b12x/sparkinfer 0.30.0 MoE backend; --async-scheduling; --max-cudagraph-capture-size 48; CUTE_DSL_ARCH=sm_120a; VLLM_USE_V2_MODEL_RUNNER=1; NCCL_P2P_DISABLE=1 (Threadripper PRO P2P deadlock)

Notes

—

Reactions

—

Submitted

Aug 1, 2026, 6:21 PM

Last edited

—

Model

auroter/DeepSeek-V4-Flash-0731-NVFP4

Display name

DeepSeek-V4-Flash-0731-NVFP4

Base model

DeepSeek-V4-Flash-0731

Revision

17e0f9da8257371654d458ba518659aa99954c86

Family

Deepseek

Parameters

304B

Active params

—

MoE

yes

Output tok/s

118.2

Prefill tok/s

—

Total tok/s

136.8

TTFT

32.8ms

Peak VRAM

354.1GB

Power draw

783.7W (194.1 + 184.2 + 218.3 + 187.2W)

Hardware cost

—

Prompt tokens

84

Output tokens

512

Prefill tokens

0

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 384GB ×4

GPU slots

—

GPU count

4

VRAM

384GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

—

RAM

—

OS

—

Power

—

Engine

vllm

Engine version

0.26.0

Quantization

NVFP4

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve /model --trust-remote-code --served-model-name DeepSeek-V4-Flash-0731-NVFP4 --tensor-parallel-size 4 --enable-expert-parallel --kv-cache-dtype fp8 --max-model-len 262144 --gpu-memory-utilization 0.90 --kernel-config '{"moe_backend":"marlin"}' --enable-prompt-tokens-details --reasoning-parser deepseek_v4 --tool-call-parser deepseek_v4

Extra flags

--enable-prompt-tokens-details --reasoning-parser deepseek_v4 --tool-call-parser deepseek_v4 --enable-expert-parallel --kernel-config {moe_backend:marlin} --kv-cache-dtype fp8 --max-model-len 262144 --tensor-parallel-size 4

Notes

cold fixed-prompt steady-state decode after 2 throwaway warmup run(s); server usage reported 0 prompt tokens already in KV cache per measured request.

Reactions

—

Submitted

Aug 1, 2026, 3:37 PM

Last edited

—

Model

huihui-ai/Huihui-GLM-5.2-abliterated-GGUF

Display name

Huihui-GLM-5.2-abliterated-GGUF

Base model

GLM-5.2

Revision

994200a058539c553f0977002e58a2f05de845c7

Family

—

Parameters

753B

Active params

—

MoE

yes

Output tok/s

47.9

Prefill tok/s

—

Total tok/s

81.4

TTFT

35.3ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

360

Output tokens

512

Prefill tokens

—

Context length

524288

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 96GB ×4

GPU slots

—

GPU count

4

VRAM

96GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen Threadripper 9960X 24-Cores

RAM

64GB

OS

linux

Power

—

Engine

llama.cpp

Engine version

9445 (af6528e6d)

Quantization

UD-Q2_K_MXFP4

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

layer

KV cache dtype

q8_0

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

512

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server --model GLM-5.2-UD-Q2_K_MXFP4-00001-of-00007.gguf --alias GLM-5.2 --ctx-size 524288 --parallel 1 --threads 32 --threads-batch 32 --batch-size 2048 --ubatch-size 512 --gpu-layers auto --split-mode layer --fit on --fit-target 4096 --fit-ctx 32768 --flash-attn auto --cache-type-k q8_0 --cache-type-v q8_0 --jinja --reasoning auto --reasoning-format deepseek --cont-batching

Extra flags

One slot; automatic layer fit; 4096 MiB per-GPU safety target; 32768-token fit floor

Notes

Single-stream text benchmark on four discrete 96 GB PCIe GPUs: 2 warmups followed by 5 measured streaming temperature-0/top-p-1 runs; median reported (47.80-47.93 output tok/s). Experimental abliterated Q2_K_MXFP4 split GGUF; use requires output review. Official llama.cpp CUDA build 9445 used one 524,288-token slot, Q8 K/V, layer split, continuous batching, no speculative decoding, and automatic fit with a 4 GiB per-GPU safety target. Repeated-prompt TTFT is warm-cache. Generation authentication, exact identity/context, coherent text, automatic tools, streaming, cancellation, and zero restart/OOM passed. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 7:16 AM

Last edited

—

Model

jacklarmer/GLM-5.2-Abliterated-MXFP8-NVFP4-NF3-Hybrid

Display name

GLM-5.2-Abliterated-MXFP8-NVFP4-NF3-Hybrid

Base model

GLM-5.2-MXFP8-NVFP4-NF3-Hybrid

Revision

e771827a07cbaed0e2f7c009e3b5ea7979873b8f

Family

Opt

Parameters

753B

Active params

—

MoE

yes

Output tok/s

92.0

Prefill tok/s

—

Total tok/s

147.5

TTFT

355.1ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

360

Output tokens

512

Prefill tokens

—

Context length

480000

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 96GB ×4

GPU slots

—

GPU count

4

VRAM

96GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen Threadripper 9960X 24-Cores

RAM

64GB

OS

linux

Power

—

Engine

vllm

Engine version

0.11.2.dev279+eldritch.enlightenment.v56a5c3e.kingdom.plus5.71fca68.b12x7bfc945.fi25dd814.cu132.20260706

Quantization

MXFP8-NVFP4-NF3-Hybrid

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

nvfp4_ds_mla

KV cache size

—

Prefix caching

yes

Attention backend

B12X_MLA_SPARSE

Flash attention

—

Chunked prefill

yes

Prefill chunk

3072

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.986

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

yes

Spec method

mtp

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve jacklarmer/GLM-5.2-Abliterated-MXFP8-NVFP4-NF3-Hybrid --served-model-name glm-5.2-abliterated-MXFP8-NVFP4-NF3-Hybrid --tensor-parallel-size 4 --decode-context-parallel-size 4 --dcp-comm-backend ag_rs --dcp-kv-cache-interleave-size 1 --kv-cache-dtype nvfp4_ds_mla --attention-backend B12X_MLA_SPARSE --moe-backend b12x --gpu-memory-utilization 0.986 --max-model-len 480000 --max-num-seqs 8 --max-num-batched-tokens 3072 --num-gpu-blocks-override 2490 --async-scheduling --enable-chunked-prefill --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"b12x","draft_sample_method":"probabilistic"}'

Extra flags

DCP=4; DCP backend ag_rs; PCIe P2P and B12X all-reduce/A2A disabled for the measured host topology

Notes

Single-stream text benchmark on four discrete 96 GB PCIe GPUs: 2 warmups followed by 5 measured streaming runs with temperature-0/top-p-1 target sampling and probabilistic MTP-3; median reported (80.24-94.12 output tok/s). Abliterated experimental derivative of the full GLM-5.2 hybrid checkpoint; use requires output review. Runtime is matched to the control: TP4/DCP4, NVFP4 DS-MLA KV, B12X MLA sparse attention/MoE, and prefix caching. Repeated-prompt TTFT is warm-cache. The measured host requires P2P and B12X all-reduce/A2A disabled with ag_rs. Authentication, exact identity, coherent text, automatic tools, streaming, cancellation, and zero restart/OOM passed. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 7:15 AM

Last edited

—

Model

madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid

Display name

GLM-5.2-MXFP8-NVFP4-NF3-Hybrid

Base model

GLM-5.2

Revision

68babde27a97a4c980c2494e830dd424975cd5a3

Family

Opt

Parameters

753B

Active params

—

MoE

yes

Output tok/s

100.1

Prefill tok/s

—

Total tok/s

159.9

TTFT

354.9ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

360

Output tokens

512

Prefill tokens

—

Context length

480000

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX PRO 6000 Blackwell 96GB ×4

GPU slots

—

GPU count

4

VRAM

96GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen Threadripper 9960X 24-Cores

RAM

64GB

OS

linux

Power

—

Engine

vllm

Engine version

0.11.2.dev279+eldritch.enlightenment.v56a5c3e.kingdom.plus5.71fca68.b12x7bfc945.fi25dd814.cu132.20260706

Quantization

MXFP8-NVFP4-NF3-Hybrid

Backend

cuda

Tensor parallel

4

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

nvfp4_ds_mla

KV cache size

—

Prefix caching

yes

Attention backend

B12X_MLA_SPARSE

Flash attention

—

Chunked prefill

yes

Prefill chunk

3072

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.986

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

yes

Spec method

mtp

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid --served-model-name GLM-5.2 --tensor-parallel-size 4 --decode-context-parallel-size 4 --dcp-comm-backend ag_rs --dcp-kv-cache-interleave-size 1 --kv-cache-dtype nvfp4_ds_mla --attention-backend B12X_MLA_SPARSE --moe-backend b12x --gpu-memory-utilization 0.986 --max-model-len 480000 --max-num-seqs 8 --max-num-batched-tokens 3072 --num-gpu-blocks-override 2490 --async-scheduling --enable-chunked-prefill --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"b12x","draft_sample_method":"probabilistic"}'

Extra flags

DCP=4; DCP backend ag_rs; PCIe P2P and B12X all-reduce/A2A disabled for the measured host topology

Notes

Single-stream text benchmark on four discrete 96 GB PCIe GPUs: 2 warmups followed by 5 measured streaming runs with temperature-0/top-p-1 target sampling and probabilistic MTP-3; median reported (84.01-101.34 output tok/s). Full GLM-5.2 hybrid checkpoint with TP4/DCP4, NVFP4 DS-MLA KV, and B12X MLA sparse attention/MoE. Prefix caching was enabled, so repeated-prompt TTFT is warm-cache. This host's partial-mesh PCIe topology requires P2P and B12X all-reduce/A2A disabled with the ag_rs DCP path. Authentication, exact identity, coherent text, automatic tools, streaming, cancellation, and zero restart/OOM passed. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 7:13 AM

Last edited

—

Model

empero-ai/Qwythos-9B-Claude-Mythos-5-1M

Display name

Qwythos-9B-Claude-Mythos-5-1M

Base model

Qwen3.5-9B

Revision

763f72fc2c3b186e977adcbaac0c18128f182166

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output tok/s

12.9

Prefill tok/s

—

Total tok/s

23.8

TTFT

137.1ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

438

Output tokens

512

Prefill tokens

—

Context length

262144

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

GB10 Grace Blackwell

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.23.1rc1.dev272+gb7abab517.d20260717

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

yes

Attention backend

flashinfer

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.45

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve empero-ai/Qwythos-9B-Claude-Mythos-5-1M --served-model-name qwythos-9b-claude-mythos-5-1m --tensor-parallel-size 1 --trust-remote-code --dtype auto --kv-cache-dtype fp8 --attention-backend flashinfer --gpu-memory-utilization 0.45 --max-model-len 262144 --max-num-seqs 4 --max-num-batched-tokens 8192 --enable-chunked-prefill --async-scheduling --enable-prefix-caching --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Extra flags

--max-model-len 262144 --async-scheduling --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Notes

Single-stream BF16 base configuration: 2 warmups followed by 5 measured streaming greedy runs; median reported (12.85-12.86 output tok/s). The checkpoint advertises a 1M YaRN ceiling, but this conservative qualification served the native 262K context. FP8 KV, FlashInfer attention, prefix caching, and no speculative decoding. Prefix caching was enabled, so repeated-prompt TTFT is warm-cache. The model card warns that greedy sampling can loop on some reasoning prompts; both the standardized greedy prompt and a separate recommended-sampling coherence test passed. Authentication, coherent reasoning/final text, automatic tool selection, streaming, and cancellation cleanup passed. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 4:28 AM

Last edited

—

Model

sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP

Display name

Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP

Base model

Huihui-Qwen3.6-27B-abliterated

Revision

0a3ba25350b8a48d7b78b64d68b7cd68f3cc12d8

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output tok/s

102.8

Prefill tok/s

—

Total tok/s

169.7

TTFT

687.7ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

361

Output tokens

512

Prefill tokens

—

Context length

262144

Batch size

4

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

GB10 Grace Blackwell

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.23.1rc1.dev272+gb7abab517.d20260717

Quantization

NVFP4

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

yes

Attention backend

flashinfer

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

modelopt

SGLang quant

—

GPU mem util

0.4

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

4

Spec decoding

yes

Spec method

mtp

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP --served-model-name huihui-qwen36-27b-ablit-nvfp4 --tensor-parallel-size 1 --trust-remote-code --quantization modelopt --kv-cache-dtype fp8 --attention-backend flashinfer --gpu-memory-utilization 0.4 --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 8192 --enable-chunked-prefill --async-scheduling --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Extra flags

--max-model-len 262144 --async-scheduling --speculative-config {"method":"mtp","num_speculative_tokens":3} --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Notes

Four-concurrent-request text throughput benchmark: 2 warmup waves followed by 5 measured streaming greedy waves; aggregate median reported (102.71-102.80 output tok/s). Each request used 361 prompt and 512 output tokens; batchSize is concurrent requests, while promptTokens/outputTokens are per request. This is 4.29x the same profile's 23.97 tok/s single-stream result. ModelOpt W4A4 NVFP4 checkpoint with mixed higher-precision tensors, FP8 KV, FlashInfer attention, and MTP-3. vLLM warned that FP8 KV q/prob scaling factors were not calibrated in the checkpoint and used 1.0. Prefix caching was enabled, so repeated-prompt TTFT is warm-cache. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 4:18 AM

Last edited

—

Model

sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP

Display name

Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP

Base model

Huihui-Qwen3.6-27B-abliterated

Revision

0a3ba25350b8a48d7b78b64d68b7cd68f3cc12d8

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output tok/s

24.0

Prefill tok/s

—

Total tok/s

40.3

TTFT

347.5ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

361

Output tokens

512

Prefill tokens

—

Context length

262144

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

GB10 Grace Blackwell

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.23.1rc1.dev272+gb7abab517.d20260717

Quantization

NVFP4

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

yes

Attention backend

flashinfer

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

modelopt

SGLang quant

—

GPU mem util

0.4

Max running seqs

8

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

yes

Spec method

mtp

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4-MTP --served-model-name huihui-qwen36-27b-ablit-nvfp4 --tensor-parallel-size 1 --trust-remote-code --quantization modelopt --kv-cache-dtype fp8 --attention-backend flashinfer --gpu-memory-utilization 0.4 --max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 8192 --enable-chunked-prefill --async-scheduling --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Extra flags

--max-model-len 262144 --async-scheduling --speculative-config {"method":"mtp","num_speculative_tokens":3} --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Notes

Single-stream text benchmark: 2 warmups followed by 5 measured streaming greedy runs; median reported (23.96-24.00 output tok/s). ModelOpt W4A4 NVFP4 checkpoint with mixed higher-precision tensors, FP8 KV, FlashInfer attention, and MTP-3. vLLM warned that FP8 KV q/prob scaling factors were not calibrated in the checkpoint and used 1.0. Prefix caching was enabled, so repeated-prompt TTFT is warm-cache. Authentication, coherent reasoning/final text, automatic tool selection, streaming, and cancellation cleanup passed. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 4:17 AM

Last edited

—

Model

nvidia/Qwen3.6-35B-A3B-NVFP4

Display name

Qwen3.6-35B-A3B-NVFP4

Base model

Qwen3.6-35B-A3B

Revision

491c2f1ea524c639598bf8fa787a93fed5a6fbce

Family

Qwen

Parameters

36B

Active params

3B

MoE

yes

Output tok/s

300.6

Prefill tok/s

—

Total tok/s

491.5

TTFT

341.0ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

361

Output tokens

512

Prefill tokens

—

Context length

262144

Batch size

4

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

GB10 Grace Blackwell

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.23.1rc1.dev272+gb7abab517.d20260717

Quantization

NVFP4

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

yes

Attention backend

flashinfer

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

modelopt

SGLang quant

—

GPU mem util

0.4

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

4

Spec decoding

yes

Spec method

mtp

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen36-35b-a3b-nvfp4 --tensor-parallel-size 1 --trust-remote-code --quantization modelopt --kv-cache-dtype fp8 --attention-backend flashinfer --moe-backend marlin --gpu-memory-utilization 0.4 --max-model-len 262144 --max-num-seqs 4 --max-num-batched-tokens 8192 --enable-chunked-prefill --async-scheduling --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Extra flags

--moe-backend marlin --max-model-len 262144 --async-scheduling --speculative-config {"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"} --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Notes

Four-concurrent-request text throughput benchmark: 2 warmup waves followed by 5 measured streaming greedy waves; aggregate median reported (291.40-324.31 output tok/s). Each request used 361 prompt and 512 output tokens; batchSize is concurrent requests, while promptTokens/outputTokens are per request. This is 2.42x the same profile's 124.07 tok/s single-stream result. NVIDIA ModelOpt NVFP4 with mixed higher-precision tensors, FP8 KV, Marlin MoE backend, FlashInfer attention, and MTP-3. Prefix caching was enabled, so repeated-prompt TTFT is warm-cache. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 3:06 AM

Last edited

—

Model

nvidia/Qwen3.6-35B-A3B-NVFP4

Display name

Qwen3.6-35B-A3B-NVFP4

Base model

Qwen3.6-35B-A3B

Revision

491c2f1ea524c639598bf8fa787a93fed5a6fbce

Family

Qwen

Parameters

36B

Active params

3B

MoE

yes

Output tok/s

124.1

Prefill tok/s

—

Total tok/s

205.2

TTFT

135.2ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

361

Output tokens

512

Prefill tokens

—

Context length

262144

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

GB10 Grace Blackwell

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.23.1rc1.dev272+gb7abab517.d20260717

Quantization

NVFP4

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

yes

Attention backend

flashinfer

Flash attention

—

Chunked prefill

yes

Prefill chunk

8192

Continuous batching

yes

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

modelopt

SGLang quant

—

GPU mem util

0.4

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

yes

Spec method

mtp

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen36-35b-a3b-nvfp4 --tensor-parallel-size 1 --trust-remote-code --quantization modelopt --kv-cache-dtype fp8 --attention-backend flashinfer --moe-backend marlin --gpu-memory-utilization 0.4 --max-model-len 262144 --max-num-seqs 4 --max-num-batched-tokens 8192 --enable-chunked-prefill --async-scheduling --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Extra flags

--moe-backend marlin --max-model-len 262144 --async-scheduling --speculative-config {"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"} --load-format fastsafetensors --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice

Notes

Single-stream text benchmark: 2 warmups followed by 5 measured streaming greedy runs; median reported (120.89-127.39 output tok/s). NVIDIA ModelOpt NVFP4 checkpoint with mixed higher-precision tensors, FP8 KV, Marlin MoE backend, FlashInfer attention, and MTP-3. The runtime warned that this GB10 path uses weight-only Marlin FP4 rather than native FP4 compute. Prefix caching was enabled, so repeated-prompt TTFT is warm-cache. Auth, coherent text, automatic tool selection, streaming, and cancellation cleanup passed. Metrics use server usage counters and client-observed token timing.

Reactions

—

Submitted

Jul 31, 2026, 3:03 AM

Last edited

—

Model

Intel/Qwen3.5-122B-A10B-int4-AutoRound

Display name

Qwen3.5-122B-A10B-int4-AutoRound

Base model

Qwen3.5-122B-A10B

Revision

3045d02bb737effc4581da91bddbad3be02934e4

Family

Qwen

Parameters

122B

Active params

—

MoE

no

Output tok/s

33.9

Prefill tok/s

—

Total tok/s

56.3

TTFT

439.7ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

361

Output tokens

512

Prefill tokens

—

Context length

131072

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

GB10 Grace Blackwell

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.23.1rc1.dev272+gb7abab517.d20260717

Quantization

INT4

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

fp8

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.85

Max running seqs

6

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

yes

Spec method

mtp

Spec model

—

Spec draft model

—

Spec tokens

2

Spec ngram

—

Spec draft TP

—

MTP enabled

yes

MTP draft layers

—

Temperature

0

Top P

1

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Intel/Qwen3.5-122B-A10B-int4-AutoRound --served-model-name qwen35-122b-a10b-int4 --gpu-memory-utilization 0.85 --enable-prefix-caching --trust-remote-code --max-model-len 131072 --max-num-batched-tokens 8192 --max-num-seqs 6 --kv-cache-dtype fp8 --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Extra flags

--max-model-len 131072 --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice --speculative-config {"method":"mtp","num_speculative_tokens":2}

Notes

Single-stream text benchmark: 2 warmups followed by 5 measured streaming greedy runs; median reported (33.46-34.25 output tok/s). AutoRound symmetric INT4, group size 128, auto_gptq packing; shared-expert tensors remain BF16. Prefix caching was enabled, so repeated-prompt TTFT is warm-cache. Metrics use server usage counters and client-observed token timing. Automatic tool selection passed; forced tool_choice=required hit a vLLM xgrammar error in this runtime. Public-equivalent launch command omits authentication and endpoint details.

Reactions

—

Submitted

Jul 31, 2026, 2:56 AM

Last edited

—

Model

pipenetwork/Laguna-S-2.1-MLX-4bit

Display name

Laguna-S-2.1-MLX-4bit

Base model

Laguna-S-2.1

Revision

main

Family

—

Parameters

118B

Active params

—

MoE

yes

Output tok/s

13.3

Prefill tok/s

51.8

Total tok/s

—

TTFT

15720.9ms

Peak VRAM

66GB

Power draw

—

Hardware cost

—

Prompt tokens

814

Output tokens

128

Prefill tokens

—

Context length

952

Batch size

1

Hardware class

UNIFIED

Hardware

M5 Max 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

Apple

Chip family

M5

Chip variant

M5 Max

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

—

Power

—

Engine

mlx

Engine version

mlx 0.32.0 + mlx-lm 0.31.3

Quantization

4bit-affine-gs64

Backend

metal

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

sdpa

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

python -c 'import mlx_lm; mlx_lm.generate(...)' (stock, no opts)

Extra flags

—

Notes

Stock baseline (no optimizations, no zmlx, no compile). Prompt: 814 tokens, decode: 128 tokens greedy (temp=0). For comparison: optimized version achieves 63.7 tok/s on same hardware. Code: github.com/jacklarmer/laguna-apple

Reactions

—

Submitted

Jul 27, 2026, 2:50 PM

Last edited

—

Model

pipenetwork/Laguna-S-2.1-MLX-4bit

Display name

Laguna-S-2.1-MLX-4bit

Base model

Laguna-S-2.1

Revision

main

Family

—

Parameters

118B

Active params

—

MoE

yes

Output tok/s

60.9

Prefill tok/s

842.6

Total tok/s

—

TTFT

1868ms

Peak VRAM

66GB

Power draw

—

Hardware cost

—

Prompt tokens

1574

Output tokens

128

Prefill tokens

—

Context length

1800

Batch size

1

Hardware class

UNIFIED

Hardware

M5 Max 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

Apple

Chip family

M5

Chip variant

M5 Max

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

—

Power

—

Engine

mlx

Engine version

mlx 0.32.0 + mlx-lm 0.31.3 + zmlx 0.9.0

Quantization

4bit-affine-gs64

Backend

metal

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

sdpa

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

python bench_speed.py (optimized: compile+zmlx+sigmoid+masks+bf16gate)

Extra flags

—

Notes

Optimized. One-big-function @mx.compile (48 layers) + zmlx moe_mlp sigmoid gating fix + precomputed masks + bf16 gating. Prompt: 1574 tokens, decode: 128 tokens greedy (temp=0). Quality: 0.9555 Pearson logit corr vs BF16 ref. Code: github.com/jacklarmer/laguna-apple

Reactions

—

Submitted

Jul 27, 2026, 2:49 PM

Last edited

—

Model

pipenetwork/Laguna-S-2.1-MLX-4bit

Display name

Laguna-S-2.1-MLX-4bit

Base model

Laguna-S-2.1

Revision

main

Family

—

Parameters

118B

Active params

—

MoE

yes

Output tok/s

63.8

Prefill tok/s

1015.5

Total tok/s

—

TTFT

801.6ms

Peak VRAM

66GB

Power draw

—

Hardware cost

—

Prompt tokens

814

Output tokens

128

Prefill tokens

—

Context length

1200

Batch size

1

Hardware class

UNIFIED

Hardware

M5 Max 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

Apple

Chip family

M5

Chip variant

M5 Max

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

—

Power

—

Engine

mlx

Engine version

mlx 0.32.0 + mlx-lm 0.31.3 + zmlx 0.9.0

Quantization

4bit-affine-gs64

Backend

metal

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

sdpa

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

python bench_speed.py (optimized: compile+zmlx+sigmoid+masks+bf16gate)

Extra flags

—

Notes

Optimized. One-big-function @mx.compile (48 layers) + zmlx moe_mlp sigmoid gating fix + precomputed masks + bf16 gating. Prompt: 814 tokens, decode: 128 tokens greedy (temp=0). Quality: 0.9555 Pearson logit corr vs BF16 ref. Code: github.com/jacklarmer/laguna-apple

Reactions

—

Submitted

Jul 27, 2026, 2:48 PM

Last edited

—

Model

pipenetwork/Laguna-S-2.1-MLX-4bit

Display name

Laguna-S-2.1-MLX-4bit

Base model

Laguna-S-2.1

Revision

main

Family

—

Parameters

118B

Active params

—

MoE

yes

Output tok/s

63.9

Prefill tok/s

56.3

Total tok/s

—

TTFT

1668.7ms

Peak VRAM

66GB

Power draw

—

Hardware cost

—

Prompt tokens

94

Output tokens

128

Prefill tokens

—

Context length

1200

Batch size

1

Hardware class

UNIFIED

Hardware

M5 Max 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

Apple

Chip family

M5

Chip variant

M5 Max

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

—

Power

—

Engine

mlx

Engine version

mlx 0.32.0 + mlx-lm 0.31.3 + zmlx 0.9.0

Quantization

4bit-affine-gs64

Backend

metal

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

sdpa

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

python bench_speed.py (optimized: compile+zmlx+sigmoid+masks+bf16gate)

Extra flags

—

Notes

Optimized. One-big-function @mx.compile (48 layers) + zmlx moe_mlp sigmoid gating fix + precomputed masks + bf16 gating. Prompt: 94 tokens, decode: 128 tokens greedy (temp=0). Quality: 0.9555 Pearson logit corr vs BF16 ref. Code: github.com/jacklarmer/laguna-apple

Reactions

—

Submitted

Jul 27, 2026, 2:47 PM

Last edited

—

Model

pipenetwork/Laguna-S-2.1-MLX-4bit

Display name

Laguna-S-2.1-MLX-4bit

Base model

Laguna-S-2.1

Revision

main

Family

—

Parameters

118B

Active params

—

MoE

yes

Output tok/s

69.5

Prefill tok/s

139.2

Total tok/s

—

TTFT

1710.3ms

Peak VRAM

66GB

Power draw

—

Hardware cost

—

Prompt tokens

238

Output tokens

128

Prefill tokens

—

Context length

376

Batch size

1

Hardware class

UNIFIED

Hardware

M5 Max 128GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

Apple

Chip family

M5

Chip variant

M5 Max

Unified memory

128GB

NPU TOPS

—

CPU

—

RAM

—

OS

—

Power

—

Engine

mlx

Engine version

mlx 0.32.0 + mlx-lm 0.31.3 + zmlx 0.9.0

Quantization

4bit-affine-gs64

Backend

metal

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

sdpa

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

python bench_speed.py (one-big-function @mx.compile + zmlx moe_mlp sigmoid gating fix + precomputed masks + bf16 gating)

Extra flags

—

Notes

Optimized MLX decode: one-big-function @mx.compile (48 layers), zmlx moe_mlp patch with sigmoid gating fix (zmlx defaults to softmax, Laguna uses sigmoid), precomputed attention masks, bfloat16 gating. Pure MLX, no torch/CUDA. Quality: 0.9555 Pearson logit correlation vs BF16 reference. Code: github.com/jacklarmer/laguna-apple

Reactions

—

Submitted

Jul 27, 2026, 2:43 PM

Last edited

—

Model

openai/gpt-oss-20b

Display name

gpt-oss-20b

Base model

—

Revision

main

Family

Gpt

Parameters

22B

Active params

—

MoE

yes

Output tok/s

49.1

Prefill tok/s

—

Total tok/s

66

TTFT

—

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

88

Output tokens

256

Prefill tokens

—

Context length

128000

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 121GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

Grace Blackwell

Unified memory

121GB

NPU TOPS

—

CPU

Cortex-X925

RAM

—

OS

linux

Power

—

Engine

vllm

Engine version

—

Quantization

FP16

Backend

—

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

Extra flags

—

Notes

—

Reactions

—

Submitted

Jul 27, 2026, 1:22 AM

Last edited

—

Model

poolside/Laguna-S-2.1-NVFP4

Display name

Laguna-S-2.1-NVFP4

Base model

Laguna-S-2.1

Revision

main

Family

—

Parameters

118B

Active params

—

MoE

yes

Output tok/s

20.6

Prefill tok/s

—

Total tok/s

24.9

TTFT

425.9ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

63

Output tokens

256

Prefill tokens

—

Context length

128000

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 121GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

Grace Blackwell

Unified memory

121GB

NPU TOPS

—

CPU

Cortex-X925

RAM

—

OS

linux

Power

—

Engine

vllm

Engine version

—

Quantization

NVFP4

Backend

—

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

Extra flags

—

Notes

—

Reactions

—

Submitted

Jul 27, 2026, 1:21 AM

Last edited

—

Model

QuantTrio/Qwen3.6-35B-A3B-AWQ

Display name

Qwen3.6-35B-A3B-AWQ

Base model

Qwen3.6-35B-A3B

Revision

main

Family

Qwen

Parameters

36B

Active params

3B

MoE

yes

Output tok/s

39.1

Prefill tok/s

—

Total tok/s

44

TTFT

—

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

32

Output tokens

256

Prefill tokens

—

Context length

128000

Batch size

1

Hardware class

UNIFIED

Hardware

GB10 Grace Blackwell 121GB

GPU slots

—

GPU count

1

VRAM

—

Chip vendor

NVIDIA

Chip family

GB10

Chip variant

Grace Blackwell

Unified memory

121GB

NPU TOPS

—

CPU

Cortex-X925

RAM

—

OS

linux

Power

—

Engine

vllm

Engine version

—

Quantization

AWQ

Backend

—

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

Extra flags

—