Speed test matrix

GLM-5.3-Flash-NVFP4 on H200 SXM 141GB ×4

sglang · 0.0.0.dev0 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Oct 20, 2026, 05:16 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.0.0.dev0
Quantization: NVFP4
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 4a0c6f2fe049

Run

Served model: nvidia/GLM-5.3-Flash-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nvidia-glm53flash-dflash2-sglang-4a0c6f2fe/bin/sglang" serve --model-path nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --trust-remote-code --tp 4 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.87 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --attention-backend dsa --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --fp4-gemm-backend marlin --moe-runner-backend marlin --speculative-algorithm DFLASH --speculative-draft-model-path incoai/GLM-5.3-Flash-DFlash2 --speculative-draft-attention-backend fa4 --quantization modelopt_fp4

Prefill

prompt tok/s by context length

8K context16.5K498 ms TTFT

16K context17.4K932 ms TTFT

32K context18.2K1,777 ms TTFT

64K context18.3K3,528 ms TTFT

128K context18K7,150 ms TTFT

256K context17.3K14,872 ms TTFT

512K context15.9K32,361 ms TTFT

1019K context13.5K75,756 ms TTFT

1024K context13.5K76,487 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	270.138 ms TTFT	458.355 ms TTFT	699.4267 ms TTFT	1,108.9147 ms TTFT	1,642.11,004 ms TTFT	2,154.7640 ms TTFT	Not run	Not run
16K	285.845 ms TTFT	438.062 ms TTFT	717.275 ms TTFT	1,083.2101 ms TTFT	1,635.1188 ms TTFT	2,143.8302 ms TTFT	Not run	Not run
32K	294.041 ms TTFT	439.655 ms TTFT	699.571 ms TTFT	1,114.486 ms TTFT	1,635.2158 ms TTFT	2,156.0263 ms TTFT	Not run	Over capacity
64K	284.060 ms TTFT	449.386 ms TTFT	710.5163 ms TTFT	1,100.7146 ms TTFT	1,619.6340 ms TTFT	2,148.6753 ms TTFT	Over capacity	Over capacity
128K	281.550 ms TTFT	201.477 ms TTFT	332.299 ms TTFT	520.6152 ms TTFT	739.4356 ms TTFT	Over capacity	Over capacity	Over capacity
256K	278.081 ms TTFT	198.2125 ms TTFT	325.9172 ms TTFT	507.0306 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
512K	296.294 ms TTFT	195.8129 ms TTFT	315.2204 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	270.3121 ms TTFT	187.4202 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	16.5K	—	498 ms	—	—	1	—	—
Prefill	16K	1	Valid	17.4K	—	932 ms	—	—	1	—	—
Prefill	32K	1	Valid	18.2K	—	1,777 ms	—	—	1	—	—
Prefill	64K	1	Valid	18.3K	—	3,528 ms	—	—	1	—	—
Prefill	128K	1	Valid	18K	—	7,150 ms	—	—	1	—	—
Prefill	256K	1	Valid	17.3K	—	14,872 ms	—	—	1	—	—
Prefill	512K	1	Valid	15.9K	—	32,361 ms	—	—	1	—	—
Prefill	1019K	1	Valid	13.5K	—	75,756 ms	—	—	1	—	—
Prefill	1024K	1	Valid	13.5K	—	76,487 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	270.1	273.3	38 ms	3.7 ms	1.0	2	8,102	30.0 s
Sustained decode	8K	2	Valid	458.3	232.3	55 ms	4.3 ms	2.0	2	13,745	30.0 s
Sustained decode	8K	4	Valid	699.4	182.3	267 ms	5.5 ms	4.0	4	20,971	30.0 s
Sustained decode	8K	8	Valid	1,108.9	145.1	147 ms	6.9 ms	7.9	8	33,257	30.0 s
Sustained decode	8K	16	Valid	1,642.1	104.9	1,004 ms	9.5 ms	16.0	0	49,261	30.0 s
Sustained decode	8K	32	Valid	2,154.7	69.1	640 ms	14.5 ms	32.0	0	64,569	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	285.8	288.0	45 ms	3.5 ms	0.9	2	8,580	30.0 s
Sustained decode	16K	2	Valid	438.0	220.6	62 ms	4.5 ms	2.0	2	13,154	30.0 s
Sustained decode	16K	4	Valid	717.2	184.4	75 ms	5.4 ms	4.0	4	21,538	30.0 s
Sustained decode	16K	8	Valid	1,083.2	142.7	101 ms	7.0 ms	7.9	8	32,494	30.0 s
Sustained decode	16K	16	Valid	1,635.1	103.7	188 ms	9.6 ms	16.0	0	49,037	30.0 s
Sustained decode	16K	32	Valid	2,143.8	69.5	302 ms	14.4 ms	32.0	0	64,304	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	294.0	296.6	41 ms	3.4 ms	1.0	2	8,824	30.0 s
Sustained decode	32K	2	Valid	439.6	221.3	55 ms	4.5 ms	2.0	2	13,183	30.0 s
Sustained decode	32K	4	Valid	699.5	187.5	71 ms	5.3 ms	4.0	4	21,007	30.0 s
Sustained decode	32K	8	Valid	1,114.4	148.3	86 ms	6.7 ms	7.9	8	33,419	30.0 s
Sustained decode	32K	16	Valid	1,635.2	105.4	158 ms	9.5 ms	16.0	0	49,053	30.0 s
Sustained decode	32K	32	Valid	2,156.0	69.2	263 ms	14.5 ms	32.0	0	64,659	30.0 s
Sustained decode	32K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	284.0	286.3	60 ms	3.5 ms	1.0	2	8,519	30.0 s
Sustained decode	64K	2	Valid	449.3	229.8	86 ms	4.4 ms	2.0	2	13,476	30.0 s
Sustained decode	64K	4	Valid	710.5	184.1	163 ms	5.4 ms	4.0	4	21,304	30.0 s
Sustained decode	64K	8	Valid	1,100.7	141.6	146 ms	7.1 ms	7.9	8	33,008	30.0 s
Sustained decode	64K	16	Valid	1,619.6	104.6	340 ms	9.6 ms	16.0	0	48,569	30.0 s
Sustained decode	64K	32	Valid	2,148.6	69.1	753 ms	14.5 ms	32.0	0	64,399	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	281.5	285.0	50 ms	3.5 ms	1.0	2	8,445	30.0 s
Sustained decode	128K	2	Valid	201.4	100.8	77 ms	9.9 ms	2.0	0	6,042	30.0 s
Sustained decode	128K	4	Valid	332.2	83.1	99 ms	12.0 ms	4.0	0	9,966	30.0 s
Sustained decode	128K	8	Valid	520.6	65.1	152 ms	15.4 ms	8.0	0	15,616	30.0 s
Sustained decode	128K	16	Valid	739.4	46.2	356 ms	21.6 ms	16.0	0	22,176	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	278.0	280.2	81 ms	3.6 ms	0.9	2	8,349	30.0 s
Sustained decode	256K	2	Valid	198.2	99.1	125 ms	10.1 ms	2.0	0	5,946	30.0 s
Sustained decode	256K	4	Valid	325.9	81.5	172 ms	12.3 ms	4.0	0	9,774	30.0 s
Sustained decode	256K	8	Valid	507.0	63.4	306 ms	15.8 ms	8.0	0	15,208	30.0 s
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	296.2	307.2	94 ms	3.3 ms	1.0	2	8,886	30.0 s
Sustained decode	512K	2	Valid	195.8	97.9	129 ms	10.2 ms	2.0	0	5,873	30.0 s
Sustained decode	512K	4	Valid	315.2	78.8	204 ms	12.7 ms	4.0	0	9,455	30.0 s
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	270.3	280.8	121 ms	3.6 ms	1.0	2	8,106	30.0 s
Sustained decode	1019K	2	Valid	187.4	93.8	202 ms	10.7 ms	2.0	0	5,622	30.0 s
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.