DeepSeek-V4.1-Flash on 4× DGX Spark: 76 tok/s verified single-stream

Verified speed-test run of deepseek-ai/DeepSeek-V4.1-Flash served by SGLang across four NVIDIA GB10 (DGX Spark) nodes with tensor parallelism 4 and DSPARK speculative decoding always on. Every number below comes from one submitted run measured against the live endpoint, not from a vendor datasheet.

Setup

Server launch command:

/opt/sglang/bin/python3 -m sglang.launch_server --model-path /models/DeepSeek-V4.1-Flash \
  --served-model-name deepseek-v4.1-flash --trust-remote-code --load-format safetensors \
  --tp 4 --ep-size 1 --attention-backend dsv4 --moe-runner-backend flashinfer_mxfp4 \
  --mem-fraction-static 0.78 --chunked-prefill-size 4096 --context-length 1048576 \
  --max-running-requests 16 --cuda-graph-max-bs-decode 16 --random-seed 0 \
  --enable-decoder-swa-bounded-replay --enable-cache-report \
  --tool-call-parser deepseekv41 --reasoning-parser deepseek-v41 \
  --host 0.0.0.0 --port 8888 \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 \
  --nnodes 4 --node-rank 0 --dist-init-addr 10.100.96.4:20000 \
  --max-total-tokens 6000000 --fp8-gemm-backend flashinfer_cutlass --watchdog-timeout 600

Methodology

• Harness: localmaxxing-cli (lmx) v0.1.37, remote mode against the OpenAI-compatible endpoint http://100.91.17.58:8888/v1 (served model deepseek-v4.1-flash).
• Requests: 1 untimed warm-up + 3 timed streaming requests; the median is reported, with min/max/σ recorded in the run payload.
• Prompt: the canonical reasoning-v1 mixed-reasoning task (≈330 tokens), greedy (temperature 0), max_tokens 1024, batch size 1, concurrency 1.
• Cold prefix cache: every warm-up and timed request receives a unique leading cache-bust nonce, so no measurement is served from the radix cache (cached_tokens: 0 on every request).
• Decode window: throughput is measured between the first and last streamed token (steady state, TTFT excluded). Prefill is derived from usage.prompt_tokens / TTFT; token counts come from endpoint usage.
• Evidence: promptSha256 of the nonce-stripped canonical prompt, outputSha256 of the full generated text, engineTimingsRaw from the engine's own meta_info, and per-node power/memory sampled during generation.

Results

Speculative decoding (DSPARK)

The deployment always runs DSPARK speculative decoding, so the reported decode rate is target-pass rate × accepted draft length:

Without the draft path the same server would need ≈2.85× as many target forward passes per emitted token, which is why measured decode is much higher than a raw backbone-only estimate on this hardware.

Observations

• Finding: a 512 GB unified-memory rig built from four GB10 modules sustains 76 tok/s single-stream on a 763 B-parameter MoE checkpoint at FP8/MXFP4 with a 1M-token context configured — about 12 % above the fastest previously listed GB10 run for this model on this site.
• Finding: per-node resident memory during generation is uniform (105.9 / 102.6 / 111.8 / 112.9 GB), i.e. the TP4 shard plus KV/activations dominate each node's memory budget; the configurable KV pool is mem-fraction-static 0.78 with max-total-tokens 6000000.
• Caveat: GB10 exposes no module or system power sensor through nvidia-smi (GPU memory power and power limits are N/A); the per-node watts above are GPU-domain readings only, taken as the mean while SM utilisation ≥50 % during generation. Total system draw is certainly higher.
• Caveat: peakVramGb is the sum of unified memory in use across the four nodes (free: used); on a unified-memory platform there is no separate VRAM number, and the figure includes a few GB of unrelated services on one node.
• Caveat: single-stream, batch-1, greedy. Batch-8 and multi-concurrent rows for this model already on the site (164 tok/s total at batch 8 on one GB10) are not comparable to this row.
• Metadata note: the auto-created hardware record attached to this run carries an OS string written before the kernel was checked (kernel 6.14); the verified platform string is Ubuntu 24.04.5 LTS / kernel 7.0.0-1019-nvidia on all four nodes, as stated above.

Reproduction

Client-side measurement (from a machine that can reach the endpoint):

lmx speed-test run sglang \
  --mode remote --base-url http://100.91.17.58:8888/v1 \
  --hf-id deepseek-ai/DeepSeek-V4.1-Flash --served-model deepseek-v4.1-flash \
  --quantization "FP8+MXFP4" --hardware hardware.json \
  --prompt-file prompt-reasoning-v1.txt --max-tokens 1024 \
  --warmup 1 --iterations 3 --temperature 0 \
  --spec-decoding --spec-method DSPARK --spec-num-tokens 5

hardware.json describes the server, not the client:

{
  "hwClass": "UNIFIED",
  "chipVendor": "NVIDIA",
  "chipFamily": "GB10",
  "chipVariant": "GB10 Grace Blackwell",
  "unifiedMemoryGb": 512,
  "cpu": "20-core ARM (Cortex-X925 + Cortex-A725)",
  "os": "Ubuntu 24.04.5 LTS (kernel 7.0.0-1019-nvidia, NVIDIA driver 580.178.04)",
  "powerWatts": 960
}

The server launch command is in Setup above.

Limitations

• Batch-1, single-stream, greedy decoding; throughput with concurrent requests or larger batches will differ.
• One prompt family (canonical reasoning-v1); speculative-decoding acceptance is workload dependent (37.2 % here) and moves with output distribution.
• The endpoint was shared during the measurement window; no other client traffic was issued, and cached_tokens was 0 on every request, but the box is not a dedicated benchmark rig.
• GPU-domain power readings are not a full-system energy measurement on GB10.