Speed test matrix

Nex-N2-mini on H200 SXM 141GB ×4

sglang · 0.5.19 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 12, 2026, 14:34 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.19
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2

Run

Served model: nex-n2-mini
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nex-n2-sglang-v0519/bin/sglang" serve --model-path nex-agi/Nex-N2-mini --served-model-name nex-n2-mini --tp 4 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.841297 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --mamba-ssm-dtype float32 --mamba-radix-cache-strategy extra_buffer

Prefill

prompt tok/s by context length

8K context52K158 ms TTFT

16K context59.9K271 ms TTFT

32K context63K513 ms TTFT

64K context60.9K1,058 ms TTFT

128K context55.6K2,316 ms TTFT

251K context46.1K5,469 ms TTFT

256K context45.5K5,651 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	315.394 ms TTFT	606.9107 ms TTFT	1,118.1159 ms TTFT	1,926.9274 ms TTFT	3,260.0583 ms TTFT	4,914.31,004 ms TTFT	7,600.62,466 ms TTFT	11.3K4,693 ms TTFT
16K	311.8123 ms TTFT	600.1158 ms TTFT	1,099.9214 ms TTFT	1,872.3648 ms TTFT	3,070.61,254 ms TTFT	4,525.51,514 ms TTFT	6,806.54,937 ms TTFT	9,811.510,111 ms TTFT
32K	304.7215 ms TTFT	585.6299 ms TTFT	1,058.9352 ms TTFT	1,773.4857 ms TTFT	2,804.81,521 ms TTFT	3,855.32,896 ms TTFT	5,688.410,926 ms TTFT	7,550.423,950 ms TTFT
64K	298.8400 ms TTFT	555.7744 ms TTFT	977.6732 ms TTFT	1,586.31,774 ms TTFT	2,366.73,176 ms TTFT	3,210.411,262 ms TTFT	4,161.123,131 ms TTFT	Over capacity
128K	277.1733 ms TTFT	503.9Requests queued757 ms TTFT	868.71,759 ms TTFT	1,312.63,112 ms TTFT	1,814.810,790 ms TTFT	2,313.622,606 ms TTFT	Over capacity	Over capacity
251K	249.61,359 ms TTFT	434.8Requests queued2,931 ms TTFT	702.33,558 ms TTFT	973.96,180 ms TTFT	1,255.522,148 ms TTFT	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	52K	—	158 ms	—	—	1	—	—
Prefill	16K	1	Valid	59.9K	—	271 ms	—	—	1	—	—
Prefill	32K	1	Valid	63K	—	513 ms	—	—	1	—	—
Prefill	64K	1	Valid	60.9K	—	1,058 ms	—	—	1	—	—
Prefill	128K	1	Valid	55.6K	—	2,316 ms	—	—	1	—	—
Prefill	251K	1	Valid	46.1K	—	5,469 ms	—	—	1	—	—
Prefill	256K	1	Valid	45.5K	—	5,651 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	315.3	317.4	94 ms	3.2 ms	1.0	2	9,458	30.0 s
Sustained decode	8K	2	Valid	606.9	306.7	107 ms	3.3 ms	2.0	4	18,207	30.0 s
Sustained decode	8K	4	Valid	1,118.1	282.7	159 ms	3.5 ms	4.0	8	33,542	30.0 s
Sustained decode	8K	8	Valid	1,926.9	245.3	274 ms	4.1 ms	7.9	16	57,805	30.0 s
Sustained decode	8K	16	Valid	3,260.0	207.6	583 ms	4.8 ms	15.8	16	97,793	30.0 s
Sustained decode	8K	32	Valid	4,914.3	157.8	1,004 ms	6.3 ms	31.8	32	147,432	30.0 s
Sustained decode	8K	64	Valid	7,600.6	120.8	2,466 ms	8.3 ms	63.9	10	228,092	30.0 s
Sustained decode	8K	128	Valid	11.3K	91.3	4,693 ms	11.0 ms	128.0	0	340,368	30.0 s
Sustained decode	16K	1	Valid	311.8	314.3	123 ms	3.2 ms	1.0	2	9,355	30.0 s
Sustained decode	16K	2	Valid	600.1	303.5	158 ms	3.3 ms	2.0	4	18,003	30.0 s
Sustained decode	16K	4	Valid	1,099.9	279.6	214 ms	3.6 ms	3.9	8	32,995	30.0 s
Sustained decode	16K	8	Valid	1,872.3	239.4	648 ms	4.2 ms	8.0	8	56,172	30.0 s
Sustained decode	16K	16	Valid	3,070.6	198.1	1,254 ms	5.0 ms	15.9	16	92,124	30.0 s
Sustained decode	16K	32	Valid	4,525.5	146.6	1,514 ms	6.8 ms	31.6	32	135,762	30.0 s
Sustained decode	16K	64	Valid	6,806.5	112.7	4,937 ms	8.9 ms	64.0	8	204,316	30.0 s
Sustained decode	16K	128	Valid	9,811.5	85.2	10,111 ms	11.7 ms	128.0	0	294,320	30.0 s
Sustained decode	32K	1	Valid	304.7	309.1	215 ms	3.2 ms	1.0	2	9,140	30.0 s
Sustained decode	32K	2	Valid	585.6	297.8	299 ms	3.4 ms	1.9	4	17,567	30.0 s
Sustained decode	32K	4	Valid	1,058.9	271.8	352 ms	3.7 ms	4.0	8	31,765	30.0 s
Sustained decode	32K	8	Valid	1,773.4	226.6	857 ms	4.4 ms	7.9	8	53,250	30.0 s
Sustained decode	32K	16	Valid	2,804.8	181.3	1,521 ms	5.5 ms	15.8	16	84,145	30.0 s
Sustained decode	32K	32	Valid	3,855.3	127.1	2,896 ms	7.9 ms	31.7	32	115,660	30.0 s
Sustained decode	32K	64	Valid	5,688.4	101.4	10,926 ms	9.9 ms	63.9	9	170,101	29.9 s
Sustained decode	32K	128	Valid	7,550.4	76.6	23,950 ms	13.1 ms	128.0	2	225,665	29.9 s
Sustained decode	64K	1	Valid	298.8	306.9	400 ms	3.3 ms	1.0	2	8,962	30.0 s
Sustained decode	64K	2	Valid	555.7	289.6	744 ms	3.5 ms	1.9	4	16,671	30.0 s
Sustained decode	64K	4	Valid	977.6	255.9	732 ms	3.9 ms	3.9	8	29,325	30.0 s
Sustained decode	64K	8	Valid	1,586.3	205.7	1,774 ms	4.9 ms	7.9	8	47,584	30.0 s
Sustained decode	64K	16	Valid	2,366.7	156.6	3,176 ms	6.4 ms	15.8	16	71,003	30.0 s
Sustained decode	64K	32	Valid	3,210.4	112.2	11,262 ms	8.9 ms	31.7	15	96,307	30.0 s
Sustained decode	64K	64	Valid	4,161.1	85.3	23,131 ms	11.7 ms	63.9	8	124,840	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	277.1	291.4	733 ms	3.4 ms	0.9	2	8,312	30.0 s
Sustained decode	128K	2	Requests queued	503.9	265.6	757 ms	3.8 ms	1.9	4	15,116	30.0 s
Sustained decode	128K	4	Valid	868.7	223.2	1,759 ms	4.5 ms	3.9	4	26,060	30.0 s
Sustained decode	128K	8	Valid	1,312.6	169.2	3,112 ms	5.9 ms	7.9	8	39,374	30.0 s
Sustained decode	128K	16	Valid	1,814.8	121.6	10,790 ms	8.2 ms	15.6	15	54,409	30.0 s
Sustained decode	128K	32	Valid	2,313.6	91.9	22,606 ms	10.9 ms	31.7	9	69,327	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	249.6	262.0	1,359 ms	3.8 ms	0.9	2	7,237	29.0 s
Sustained decode	251K	2	Requests queued	434.8	234.8	2,931 ms	4.3 ms	1.9	2	13,045	30.0 s
Sustained decode	251K	4	Valid	702.3	185.8	3,558 ms	5.4 ms	3.9	4	21,068	30.0 s
Sustained decode	251K	8	Valid	973.9	138.2	6,180 ms	7.2 ms	7.7	8	29,216	30.0 s
Sustained decode	251K	16	Valid	1,255.5	98.0	22,148 ms	10.2 ms	15.8	6	37,667	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.