Speed test matrix

Inkling-NVFP4 on H200 SXM 141GB ×8

vllm · 0.29.0 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 17, 2026, 20:13 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 8
VRAM: 1128 GB (8 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: thinkingmachines/Inkling-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve thinkingmachines/Inkling-NVFP4 --served-model-name thinkingmachines/Inkling-NVFP4 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser inkling --enable-auto-tool-choice --tool-call-parser inkling --max-model-len 1048576 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode inkling --kernel-config.enable_flashinfer_autotune=False

Prefill

prompt tok/s by context length

8K context16.1K508 ms TTFT

16K context15.9K1,019 ms TTFT

32K context15.9K2,036 ms TTFT

64K context15.8K4,077 ms TTFT

128K context14.1K9,171 ms TTFT

256K context12K21,418 ms TTFT

512K context9,10756,547 ms TTFT

1019K context6,110167,736 ms TTFT

1024K context6,073169,581 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	73.874 ms TTFT	138.5124 ms TTFT	245.5119 ms TTFT	476.9160 ms TTFT	871.7227 ms TTFT	1,485.4352 ms TTFT	2,463.3604 ms TTFT	Over capacity
16K	59.486 ms TTFT	115.5107 ms TTFT	218.4162 ms TTFT	410.1219 ms TTFT	732.9316 ms TTFT	1,280.6538 ms TTFT	Over capacity	Over capacity
32K	42.5136 ms TTFT	82.2169 ms TTFT	158.2231 ms TTFT	302.6348 ms TTFT	551.0626 ms TTFT	Over capacity	Over capacity	Over capacity
64K	27.0258 ms TTFT	52.1336 ms TTFT	101.7446 ms TTFT	198.0641 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	15.6511 ms TTFT	30.2658 ms TTFT	59.3952 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
256K	8.51,059 ms TTFT	16.31,367 ms TTFT	32.32,012 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
512K	4.42,348 ms TTFT	8.53,099 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	2.35,168 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	16.1K	—	508 ms	—	—	1	—	—
Prefill	16K	1	Valid	15.9K	—	1,019 ms	—	—	1	—	—
Prefill	32K	1	Valid	15.9K	—	2,036 ms	—	—	1	—	—
Prefill	64K	1	Valid	15.8K	—	4,077 ms	—	—	1	—	—
Prefill	128K	1	Valid	14.1K	—	9,171 ms	—	—	1	—	—
Prefill	256K	1	Valid	12K	—	21,418 ms	—	—	1	—	—
Prefill	512K	1	Valid	9,107.0	—	56,547 ms	—	—	1	—	—
Prefill	1019K	1	Valid	6,110.0	—	167,736 ms	—	—	1	—	—
Prefill	1024K	1	Valid	6,073.0	—	169,581 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	73.8	74.1	74 ms	13.5 ms	1.0	0	2,214	30.0 s
Sustained decode	8K	2	Valid	138.5	69.6	124 ms	14.4 ms	2.0	0	4,154	30.0 s
Sustained decode	8K	4	Valid	245.5	65.2	119 ms	15.3 ms	4.0	0	7,365	30.0 s
Sustained decode	8K	8	Valid	476.9	60.2	160 ms	16.6 ms	8.0	0	14,304	30.0 s
Sustained decode	8K	16	Valid	871.7	55.1	227 ms	18.1 ms	16.0	0	26,143	30.0 s
Sustained decode	8K	32	Valid	1,485.4	47.0	352 ms	21.3 ms	32.0	0	44,543	30.0 s
Sustained decode	8K	64	Valid	2,463.3	38.8	604 ms	25.7 ms	64.0	0	73,906	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	59.4	59.6	86 ms	16.8 ms	1.0	0	1,780	30.0 s
Sustained decode	16K	2	Valid	115.5	57.9	107 ms	17.3 ms	2.0	0	3,464	30.0 s
Sustained decode	16K	4	Valid	218.4	54.9	162 ms	18.2 ms	4.0	0	6,551	30.0 s
Sustained decode	16K	8	Valid	410.1	51.6	219 ms	19.4 ms	8.0	0	12,296	30.0 s
Sustained decode	16K	16	Valid	732.9	46.1	316 ms	21.7 ms	16.0	0	21,984	30.0 s
Sustained decode	16K	32	Valid	1,280.6	40.3	538 ms	24.8 ms	32.0	0	38,400	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	42.5	42.6	136 ms	23.5 ms	1.0	0	1,275	30.0 s
Sustained decode	32K	2	Valid	82.2	41.2	169 ms	24.3 ms	2.0	0	2,464	30.0 s
Sustained decode	32K	4	Valid	158.2	39.7	231 ms	25.2 ms	4.0	0	4,744	30.0 s
Sustained decode	32K	8	Valid	302.6	37.9	348 ms	26.4 ms	8.0	0	9,072	30.0 s
Sustained decode	32K	16	Valid	551.0	35.0	626 ms	28.6 ms	16.0	0	16,522	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	27.0	27.1	258 ms	37.0 ms	1.0	0	810	30.0 s
Sustained decode	64K	2	Valid	52.1	26.1	336 ms	38.3 ms	2.0	0	1,564	30.0 s
Sustained decode	64K	4	Valid	101.7	25.5	446 ms	39.3 ms	4.0	0	3,048	30.0 s
Sustained decode	64K	8	Valid	198.0	24.8	641 ms	40.4 ms	8.0	0	5,936	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	15.6	15.6	511 ms	64.0 ms	1.0	0	468	30.0 s
Sustained decode	128K	2	Valid	30.2	15.1	658 ms	66.2 ms	2.0	0	904	30.0 s
Sustained decode	128K	4	Valid	59.3	14.8	952 ms	67.5 ms	4.0	0	1,776	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	8.5	8.5	1,059 ms	118.0 ms	1.0	0	254	30.0 s
Sustained decode	256K	2	Valid	16.3	8.2	1,367 ms	121.9 ms	2.0	0	488	29.9 s
Sustained decode	256K	4	Valid	32.3	8.1	2,012 ms	123.6 ms	4.0	0	964	29.9 s
Sustained decode	256K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	4.4	4.4	2,348 ms	225.2 ms	1.0	0	132	29.9 s
Sustained decode	512K	2	Valid	8.5	4.3	3,099 ms	232.9 ms	2.0	0	256	30.0 s
Sustained decode	512K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	2.3	2.3	5,168 ms	435.3 ms	1.0	0	67	29.7 s
Sustained decode	1019K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.