Speed test matrix

Qwen3.6-27B on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 7, 2026, 07:10 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886618 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Prefill

prompt tok/s by context length

8K context10.3K797 ms TTFT

16K context10.1K1,601 ms TTFT

32K context9,4813,406 ms TTFT

64K context8,6907,413 ms TTFT

128K context7,34817,513 ms TTFT

251K context5,65144,629 ms TTFT

256K context5,62245,753 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	113.6129 ms TTFT	232.8598 ms TTFT	461.0479 ms TTFT	844.7858 ms TTFT	1,446.01,646 ms TTFT	2,247.92,699 ms TTFT	2,855.14,564 ms TTFT	Over capacity
16K	113.0174 ms TTFT	224.3259 ms TTFT	443.1518 ms TTFT	815.8961 ms TTFT	1,373.81,687 ms TTFT	2,016.43,032 ms TTFT	Over capacity	Over capacity
32K	116.8216 ms TTFT	222.6355 ms TTFT	420.2620 ms TTFT	755.91,199 ms TTFT	1,205.41,917 ms TTFT	Over capacity	Over capacity	Over capacity
64K	114.9303 ms TTFT	213.3453 ms TTFT	397.2813 ms TTFT	655.51,389 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	107.5543 ms TTFT	192.0749 ms TTFT	339.91,348 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	96.7787 ms TTFT	169.01,076 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	10.3K	—	797 ms	—	—	1	—	—
Prefill	16K	1	Valid	10.1K	—	1,601 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,481.0	—	3,406 ms	—	—	1	—	—
Prefill	64K	1	Valid	8,690.0	—	7,413 ms	—	—	1	—	—
Prefill	128K	1	Valid	7,348.0	—	17,513 ms	—	—	1	—	—
Prefill	251K	1	Valid	5,651.0	—	44,629 ms	—	—	1	—	—
Prefill	256K	1	Valid	5,622.0	—	45,753 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	113.6	115.1	129 ms	8.7 ms	1.0	0	3,408	30.0 s
Sustained decode	8K	2	Valid	232.8	117.1	598 ms	8.5 ms	2.0	0	6,982	30.0 s
Sustained decode	8K	4	Valid	461.0	117.0	479 ms	8.5 ms	4.0	1	13,830	30.0 s
Sustained decode	8K	8	Valid	844.7	106.1	858 ms	9.4 ms	8.0	0	25,338	30.0 s
Sustained decode	8K	16	Valid	1,446.0	91.1	1,646 ms	11.0 ms	16.0	0	43,367	30.0 s
Sustained decode	8K	32	Valid	2,247.9	70.0	2,699 ms	14.3 ms	32.0	0	67,440	30.0 s
Sustained decode	8K	64	Valid	2,855.1	43.7	4,564 ms	22.9 ms	64.0	0	85,675	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	113.0	114.1	174 ms	8.8 ms	1.0	0	3,392	30.0 s
Sustained decode	16K	2	Valid	224.3	113.5	259 ms	8.8 ms	2.0	0	6,727	30.0 s
Sustained decode	16K	4	Valid	443.1	112.7	518 ms	8.9 ms	4.0	0	13,284	30.0 s
Sustained decode	16K	8	Valid	815.8	101.6	961 ms	9.8 ms	8.0	0	24,458	30.0 s
Sustained decode	16K	16	Valid	1,373.8	85.7	1,687 ms	11.7 ms	16.0	0	41,194	30.0 s
Sustained decode	16K	32	Valid	2,016.4	63.1	3,032 ms	15.9 ms	32.0	0	60,470	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	116.8	117.8	216 ms	8.5 ms	1.0	0	3,502	30.0 s
Sustained decode	32K	2	Valid	222.6	112.4	355 ms	8.9 ms	2.0	0	6,675	30.0 s
Sustained decode	32K	4	Valid	420.2	106.1	620 ms	9.4 ms	4.0	0	12,610	30.0 s
Sustained decode	32K	8	Valid	755.9	95.1	1,199 ms	10.5 ms	8.0	0	22,668	30.0 s
Sustained decode	32K	16	Valid	1,205.4	76.1	1,917 ms	13.1 ms	16.0	0	36,152	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	114.9	116.4	303 ms	8.6 ms	1.0	0	3,445	30.0 s
Sustained decode	64K	2	Valid	213.3	107.8	453 ms	9.3 ms	2.0	0	6,400	30.0 s
Sustained decode	64K	4	Valid	397.2	100.0	813 ms	10.0 ms	4.0	0	11,914	30.0 s
Sustained decode	64K	8	Valid	655.5	82.2	1,389 ms	12.2 ms	8.0	0	19,655	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	107.5	108.6	543 ms	9.2 ms	1.0	0	3,223	30.0 s
Sustained decode	128K	2	Valid	192.0	97.1	749 ms	10.3 ms	2.0	0	5,757	30.0 s
Sustained decode	128K	4	Valid	339.9	84.1	1,348 ms	11.9 ms	4.0	0	10,190	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	96.7	97.9	787 ms	10.2 ms	1.0	0	2,899	30.0 s
Sustained decode	251K	2	Valid	169.0	85.9	1,076 ms	11.7 ms	2.0	0	5,066	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.