Speed test matrix

Qwen3.6-27B-FP8 on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 7, 2026, 06:01 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: qwen
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-qwen-vllm/bin/vllm" serve Qwen/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.883918 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Prefill

prompt tok/s by context length

8K context14.3K571 ms TTFT

16K context13.9K1,168 ms TTFT

32K context13.2K2,437 ms TTFT

64K context11.8K5,467 ms TTFT

128K context9,46013,604 ms TTFT

251K context6,84836,829 ms TTFT

256K context6,78737,895 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	172.2112 ms TTFT	299.9309 ms TTFT	603.1245 ms TTFT	1,013.2265 ms TTFT	1,789.41,227 ms TTFT	2,651.32,026 ms TTFT	3,315.53,425 ms TTFT	Over capacity
16K	152.9146 ms TTFT	293.2178 ms TTFT	584.8226 ms TTFT	977.7300 ms TTFT	1,662.01,380 ms TTFT	2,382.42,141 ms TTFT	Over capacity	Over capacity
32K	151.9192 ms TTFT	280.2247 ms TTFT	544.4301 ms TTFT	912.4971 ms TTFT	1,444.41,546 ms TTFT	1,940.42,943 ms TTFT	Over capacity	Over capacity
64K	147.2287 ms TTFT	271.6344 ms TTFT	493.7351 ms TTFT	808.51,284 ms TTFT	1,148.42,120 ms TTFT	Over capacity	Over capacity	Over capacity
128K	134.0509 ms TTFT	244.1522 ms TTFT	422.11,091 ms TTFT	607.72,223 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	119.9774 ms TTFT	200.2998 ms TTFT	332.01,508 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	14.3K	—	571 ms	—	—	1	—	—
Prefill	16K	1	Valid	13.9K	—	1,168 ms	—	—	1	—	—
Prefill	32K	1	Valid	13.2K	—	2,437 ms	—	—	1	—	—
Prefill	64K	1	Valid	11.8K	—	5,467 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,460.0	—	13,604 ms	—	—	1	—	—
Prefill	251K	1	Valid	6,848.0	—	36,829 ms	—	—	1	—	—
Prefill	256K	1	Valid	6,787.0	—	37,895 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	172.2	171.5	112 ms	5.8 ms	1.0	1	5,167	30.0 s
Sustained decode	8K	2	Valid	299.9	154.9	309 ms	6.5 ms	2.0	2	8,994	30.0 s
Sustained decode	8K	4	Valid	603.1	154.8	245 ms	6.5 ms	4.0	4	18,086	30.0 s
Sustained decode	8K	8	Valid	1,013.2	129.5	265 ms	7.7 ms	8.0	8	30,378	30.0 s
Sustained decode	8K	16	Valid	1,789.4	113.2	1,227 ms	8.8 ms	16.0	0	53,685	30.0 s
Sustained decode	8K	32	Valid	2,651.3	83.2	2,026 ms	12.0 ms	32.0	0	79,487	30.0 s
Sustained decode	8K	64	Valid	3,315.5	51.3	3,425 ms	19.5 ms	64.0	0	99,420	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	152.9	160.0	146 ms	6.3 ms	1.0	1	4,585	30.0 s
Sustained decode	16K	2	Valid	293.2	152.0	178 ms	6.6 ms	2.0	2	8,797	30.0 s
Sustained decode	16K	4	Valid	584.8	149.5	226 ms	6.7 ms	4.0	4	17,544	30.0 s
Sustained decode	16K	8	Valid	977.7	126.1	300 ms	7.9 ms	8.0	7	29,323	30.0 s
Sustained decode	16K	16	Valid	1,662.0	106.0	1,380 ms	9.4 ms	16.0	0	49,862	30.0 s
Sustained decode	16K	32	Valid	2,382.4	74.3	2,141 ms	13.5 ms	32.0	0	71,481	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	151.9	158.5	192 ms	6.3 ms	1.0	1	4,557	30.0 s
Sustained decode	32K	2	Valid	280.2	147.6	247 ms	6.8 ms	2.0	2	8,404	30.0 s
Sustained decode	32K	4	Valid	544.4	140.9	301 ms	7.1 ms	3.9	4	16,322	30.0 s
Sustained decode	32K	8	Valid	912.4	115.8	971 ms	8.6 ms	8.0	0	27,382	30.0 s
Sustained decode	32K	16	Valid	1,444.4	90.8	1,546 ms	11.0 ms	16.0	0	43,323	30.0 s
Sustained decode	32K	32	Valid	1,940.4	61.0	2,943 ms	16.4 ms	32.0	0	58,145	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	147.2	152.9	287 ms	6.5 ms	1.0	1	4,415	30.0 s
Sustained decode	64K	2	Valid	271.6	144.4	344 ms	6.9 ms	2.0	2	8,149	30.0 s
Sustained decode	64K	4	Valid	493.7	125.1	351 ms	8.0 ms	4.0	4	14,808	30.0 s
Sustained decode	64K	8	Valid	808.5	100.8	1,284 ms	9.9 ms	8.0	1	24,251	30.0 s
Sustained decode	64K	16	Valid	1,148.4	72.9	2,120 ms	13.7 ms	16.0	0	34,433	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	134.0	154.6	509 ms	6.6 ms	1.0	1	4,018	30.0 s
Sustained decode	128K	2	Valid	244.1	125.6	522 ms	8.0 ms	2.0	1	7,326	30.0 s
Sustained decode	128K	4	Valid	422.1	105.1	1,091 ms	9.5 ms	4.0	0	12,663	30.0 s
Sustained decode	128K	8	Valid	607.7	76.6	2,223 ms	13.1 ms	8.0	0	18,236	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	119.9	121.2	774 ms	8.2 ms	1.0	1	3,594	30.0 s
Sustained decode	251K	2	Valid	200.2	101.4	998 ms	9.9 ms	2.0	0	6,009	30.0 s
Sustained decode	251K	4	Valid	332.0	83.0	1,508 ms	12.0 ms	4.0	0	9,954	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.