Speed test matrix

Qwen3.8-27B-NVFP4-BF16-LMHead on RTX PRO 6000 Blackwell Server Edition 96GB

sglang · 0.5.6.post3.dev11565+gfa7784d14 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Oct 11, 2026, 00:44 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev11565+gfa7784d14
Quantization: NVFP4+FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: fa7784d140ee

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-radixark-sglang-fa7784d140ee/bin/sglang" serve --model-path RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead --served-model-name qwen --tp 1 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --page-size 64 --mamba-radix-cache-strategy extra_buffer --mamba-track-interval 64 --linear-attn-backend triton --attention-backend flashinfer --fp4-gemm-backend flashinfer_cutlass --quantization modelopt_mixed

Prefill

prompt tok/s by context length

8K context13.4K611 ms TTFT

16K context12.3K1,316 ms TTFT

32K context10.6K3,030 ms TTFT

64K context8,1367,881 ms TTFT

128K context5,55823,042 ms TTFT

251K context3,40373,728 ms TTFT

256K context3,35176,368 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	61.734 ms TTFT	116.6122 ms TTFT	225.0206 ms TTFT	435.3172 ms TTFT	746.2311 ms TTFT	1,196.1243 ms TTFT	1,179.9Requests queued243 ms TTFT	Over capacity
16K	61.034 ms TTFT	115.458 ms TTFT	221.667 ms TTFT	423.088 ms TTFT	710.7134 ms TTFT	1,084.2232 ms TTFT	Over capacity	Over capacity
32K	59.734 ms TTFT	112.960 ms TTFT	215.568 ms TTFT	401.186 ms TTFT	647.4120 ms TTFT	Over capacity	Over capacity	Over capacity
64K	57.339 ms TTFT	108.568 ms TTFT	204.094 ms TTFT	363.7147 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	52.949 ms TTFT	100.484 ms TTFT	183.3128 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	46.252 ms TTFT	87.985 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	13.4K	—	611 ms	—	—	1	—	—
Prefill	16K	1	Valid	12.3K	—	1,316 ms	—	—	1	—	—
Prefill	32K	1	Valid	10.6K	—	3,030 ms	—	—	1	—	—
Prefill	64K	1	Valid	8,136.0	—	7,881 ms	—	—	1	—	—
Prefill	128K	1	Valid	5,558.0	—	23,042 ms	—	—	1	—	—
Prefill	251K	1	Valid	3,403.0	—	73,728 ms	—	—	1	—	—
Prefill	256K	1	Valid	3,351.0	—	76,368 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	61.7	61.8	34 ms	16.2 ms	1.0	0	1,852	30.0 s
Sustained decode	8K	2	Valid	116.6	58.2	122 ms	17.2 ms	2.0	0	3,496	30.0 s
Sustained decode	8K	4	Valid	225.0	56.3	206 ms	17.8 ms	4.0	0	6,750	30.0 s
Sustained decode	8K	8	Valid	435.3	54.4	172 ms	18.4 ms	8.0	0	13,056	30.0 s
Sustained decode	8K	16	Valid	746.2	46.7	311 ms	21.4 ms	16.0	0	22,384	30.0 s
Sustained decode	8K	32	Valid	1,196.1	37.4	243 ms	26.7 ms	32.0	0	35,872	30.0 s
Sustained decode	8K	64	Requests queued	1,179.9	36.6	243 ms	27.3 ms	33.0	0	35,376	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	61.0	61.0	34 ms	16.4 ms	1.0	0	1,830	30.0 s
Sustained decode	16K	2	Valid	115.4	57.7	58 ms	17.3 ms	2.0	0	3,460	30.0 s
Sustained decode	16K	4	Valid	221.6	55.4	67 ms	18.0 ms	4.0	0	6,648	30.0 s
Sustained decode	16K	8	Valid	423.0	52.9	88 ms	18.9 ms	8.0	0	12,688	30.0 s
Sustained decode	16K	16	Valid	710.7	44.4	134 ms	22.5 ms	16.0	0	21,312	30.0 s
Sustained decode	16K	32	Valid	1,084.2	33.9	232 ms	29.5 ms	32.0	0	32,512	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	59.7	59.7	34 ms	16.8 ms	1.0	0	1,790	30.0 s
Sustained decode	32K	2	Valid	112.9	56.5	60 ms	17.7 ms	2.0	0	3,385	30.0 s
Sustained decode	32K	4	Valid	215.5	53.9	68 ms	18.6 ms	4.0	0	6,464	30.0 s
Sustained decode	32K	8	Valid	401.1	50.2	86 ms	19.9 ms	8.0	0	12,024	30.0 s
Sustained decode	32K	16	Valid	647.4	40.5	120 ms	24.7 ms	16.0	0	19,408	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	57.3	57.3	39 ms	17.5 ms	1.0	0	1,717	30.0 s
Sustained decode	64K	2	Valid	108.5	54.3	68 ms	18.4 ms	2.0	0	3,255	30.0 s
Sustained decode	64K	4	Valid	204.0	51.0	94 ms	19.6 ms	4.0	0	6,120	30.0 s
Sustained decode	64K	8	Valid	363.7	45.5	147 ms	22.0 ms	8.0	0	10,904	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	52.9	52.9	49 ms	18.9 ms	1.0	0	1,587	30.0 s
Sustained decode	128K	2	Valid	100.4	50.2	84 ms	19.9 ms	2.0	0	3,013	30.0 s
Sustained decode	128K	4	Valid	183.3	45.8	128 ms	21.8 ms	4.0	0	5,496	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	46.2	46.2	52 ms	21.6 ms	1.0	0	1,386	30.0 s
Sustained decode	251K	2	Valid	87.9	43.9	85 ms	22.8 ms	2.0	0	2,634	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.