Speed test matrix

dots3-note-prev-fp8 on H200 SXM 141GB ×8

vllm · 0.29.0 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 17, 2026, 11:18 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 8
VRAM: 1128 GB (8 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: dots-studio/dots3-note-prev-fp8
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve dots-studio/dots3-note-prev-fp8 --served-model-name dots-studio/dots3-note-prev-fp8 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser dots --max-model-len 262144 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --kernel-config.enable_flashinfer_autotune=False

Prefill

prompt tok/s by context length

8K context16.1K508 ms TTFT

16K context15.9K1,019 ms TTFT

32K context4,5317,131 ms TTFT

64K context21.1K3,058 ms TTFT

128K context9,36413,756 ms TTFT

251K context13.8K18,345 ms TTFT

256K context9,06028,421 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	117.257 ms TTFT	225.3158 ms TTFT	422.2138 ms TTFT	Looped	1,328.7280 ms TTFT	2,180.1386 ms TTFT	Looped	Over capacity
16K	116.973 ms TTFT	224.496 ms TTFT	420.6140 ms TTFT	735.0209 ms TTFT	1,319.7389 ms TTFT	2,174.8605 ms TTFT	Over capacity	Over capacity
32K	116.7105 ms TTFT	224.0144 ms TTFT	419.3206 ms TTFT	731.1330 ms TTFT	1,315.8513 ms TTFT	Over capacity	Over capacity	Over capacity
64K	116.3184 ms TTFT	222.8247 ms TTFT	416.6368 ms TTFT	723.2587 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	115.5362 ms TTFT	220.2512 ms TTFT	412.1766 ms TTFT	712.11,252 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	114.0822 ms TTFT	217.21,092 ms TTFT	403.31,744 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	16.1K	—	508 ms	—	—	1	—	—
Prefill	16K	1	Valid	15.9K	—	1,019 ms	—	—	1	—	—
Prefill	32K	1	Valid	4,531.0	—	7,131 ms	—	—	1	—	—
Prefill	64K	1	Valid	21.1K	—	3,058 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,364.0	—	13,756 ms	—	—	1	—	—
Prefill	251K	1	Valid	13.8K	—	18,345 ms	—	—	1	—	—
Prefill	256K	1	Valid	9,060.0	—	28,421 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	117.2	117.2	57 ms	8.5 ms	1.0	0	3,515	30.0 s
Sustained decode	8K	2	Valid	225.3	112.4	158 ms	8.9 ms	2.0	0	6,758	30.0 s
Sustained decode	8K	4	Valid	422.2	105.6	138 ms	9.5 ms	4.0	0	12,666	30.0 s
Sustained decode	8K	8	Looped	—	—	—	—	—	0	0	—
Sustained decode	8K	16	Valid	1,328.7	83.3	280 ms	12.0 ms	16.0	0	39,856	30.0 s
Sustained decode	8K	32	Valid	2,180.1	68.4	386 ms	14.6 ms	32.0	0	65,376	30.0 s
Sustained decode	8K	64	Looped	—	—	—	—	—	0	0	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	116.9	116.9	73 ms	8.6 ms	1.0	0	3,507	30.0 s
Sustained decode	16K	2	Valid	224.4	112.2	96 ms	8.9 ms	2.0	0	6,732	30.0 s
Sustained decode	16K	4	Valid	420.6	105.2	140 ms	9.5 ms	4.0	0	12,619	30.0 s
Sustained decode	16K	8	Valid	735.0	91.8	209 ms	10.9 ms	8.0	0	22,048	30.0 s
Sustained decode	16K	16	Valid	1,319.7	82.5	389 ms	12.1 ms	16.0	0	39,584	30.0 s
Sustained decode	16K	32	Valid	2,174.8	67.9	605 ms	14.7 ms	32.0	0	65,247	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	116.7	116.7	105 ms	8.6 ms	1.0	0	3,500	30.0 s
Sustained decode	32K	2	Valid	224.0	112.0	144 ms	8.9 ms	2.0	0	6,718	30.0 s
Sustained decode	32K	4	Valid	419.3	104.8	206 ms	9.5 ms	4.0	0	12,576	30.0 s
Sustained decode	32K	8	Valid	731.1	91.3	330 ms	11.0 ms	8.0	0	21,928	30.0 s
Sustained decode	32K	16	Valid	1,315.8	82.0	513 ms	12.2 ms	16.0	0	39,471	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	116.3	116.3	184 ms	8.6 ms	1.0	0	3,487	30.0 s
Sustained decode	64K	2	Valid	222.8	111.4	247 ms	9.0 ms	2.0	0	6,682	30.0 s
Sustained decode	64K	4	Valid	416.6	104.1	368 ms	9.6 ms	4.0	0	12,496	30.0 s
Sustained decode	64K	8	Valid	723.2	90.3	587 ms	11.1 ms	8.0	0	21,696	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	115.5	115.6	362 ms	8.6 ms	1.0	0	3,464	30.0 s
Sustained decode	128K	2	Valid	220.2	110.2	512 ms	9.1 ms	2.0	0	6,606	30.0 s
Sustained decode	128K	4	Valid	412.1	103.0	766 ms	9.7 ms	4.0	0	12,360	30.0 s
Sustained decode	128K	8	Valid	712.1	88.9	1,252 ms	11.2 ms	8.0	0	21,362	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	114.0	114.3	822 ms	8.7 ms	1.0	0	3,420	30.0 s
Sustained decode	251K	2	Valid	217.2	108.8	1,092 ms	9.2 ms	2.0	0	6,514	30.0 s
Sustained decode	251K	4	Valid	403.3	100.9	1,744 ms	9.9 ms	4.0	0	12,100	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.