Speed test matrix
Ling-3.0-tiny-base-midtrain on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-3.0-tiny-base-midtrain --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 262144 --gpu-memory-utilization 0.885259 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align
Prefill
prompt tok/s by context length8K context51.9K158 ms TTFT
16K context28.9K561 ms TTFT
32K context65.3K494 ms TTFT
64K context67.7K951 ms TTFT
128K context49.8K2,581 ms TTFT
251K context41.5K6,073 ms TTFT
256K context40.5K6,337 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 338.1103 ms TTFT | 593.4140 ms TTFT | 1,030.6172 ms TTFT | 1,390.5235 ms TTFT | 2,924.6317 ms TTFT | 4,777.6459 ms TTFT | 7,771.9Requests queued639 ms TTFT | 12.6K1,117 ms TTFT |
| 16K | 333.3102 ms TTFT | 584.2172 ms TTFT | 1,015.8205 ms TTFT | 1,365.7272 ms TTFT | 2,816.1359 ms TTFT | 4,530.8528 ms TTFT | 7,343.1818 ms TTFT | 11.2K1,358 ms TTFT |
| 32K | 323.1160 ms TTFT | 567.5202 ms TTFT | 986.4228 ms TTFT | 1,310.3303 ms TTFT | 2,662.7509 ms TTFT | 4,087.3416 ms TTFT | 6,405.71,413 ms TTFT | 9,056.32,427 ms TTFT |
| 64K | 305.0211 ms TTFT | 540.4233 ms TTFT | 941.5267 ms TTFT | 1,244.0279 ms TTFT | 2,363.2687 ms TTFT | 3,607.81,698 ms TTFT | 5,063.02,939 ms TTFT | 6,448.15,103 ms TTFT |
| 128K | 274.0328 ms TTFT | 488.2347 ms TTFT | 864.1463 ms TTFT | 1,106.9430 ms TTFT | 1,942.6386 ms TTFT | 2,762.82,835 ms TTFT | 3,530.45,714 ms TTFT | Over capacity |
| 251K | 232.6566 ms TTFT | 422.8642 ms TTFT | 748.9617 ms TTFT | 936.31,424 ms TTFT | 1,529.33,220 ms TTFT | 1,892.36,090 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 51.9K | — | 158 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.9K | — | 561 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 65.3K | — | 494 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 67.7K | — | 951 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 49.8K | — | 2,581 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 41.5K | — | 6,073 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 40.5K | — | 6,337 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 338.1 | 340.6 | 103 ms | 2.9 ms | 1.0 | 2 | 10,142 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 593.4 | 299.5 | 140 ms | 3.3 ms | 2.0 | 4 | 17,802 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,030.6 | 261.0 | 172 ms | 3.8 ms | 4.0 | 8 | 30,917 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,390.5 | 176.1 | 235 ms | 5.7 ms | 8.0 | 8 | 41,712 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,924.6 | 185.5 | 317 ms | 5.4 ms | 15.5 | 16 | 87,742 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,777.6 | 153.2 | 459 ms | 6.5 ms | 32.0 | 32 | 143,319 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 7,771.9 | 123.9 | 639 ms | 8.1 ms | 63.1 | 64 | 233,183 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.6K | 97.6 | 1,117 ms | 10.2 ms | 128.0 | 0 | 378,621 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 333.3 | 335.7 | 102 ms | 3.0 ms | 1.0 | 2 | 9,997 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 584.2 | 295.9 | 172 ms | 3.4 ms | 2.0 | 4 | 17,526 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,015.8 | 258.9 | 205 ms | 3.9 ms | 4.0 | 8 | 30,472 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,365.7 | 172.4 | 272 ms | 5.8 ms | 8.0 | 8 | 40,968 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,816.1 | 178.7 | 359 ms | 5.6 ms | 16.0 | 16 | 84,483 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,530.8 | 144.2 | 528 ms | 6.9 ms | 30.9 | 32 | 135,929 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,343.1 | 114.0 | 818 ms | 8.8 ms | 64.0 | 0 | 220,280 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 11.2K | 85.6 | 1,358 ms | 11.7 ms | 128.0 | 0 | 334,838 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 323.1 | 326.7 | 160 ms | 3.1 ms | 1.0 | 2 | 9,693 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 567.5 | 288.0 | 202 ms | 3.5 ms | 1.9 | 4 | 17,027 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 986.4 | 250.6 | 228 ms | 4.0 ms | 3.9 | 8 | 29,599 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,310.3 | 164.8 | 303 ms | 6.1 ms | 7.9 | 8 | 39,307 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,662.7 | 168.3 | 509 ms | 5.9 ms | 15.5 | 16 | 79,885 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,087.3 | 130.0 | 416 ms | 7.7 ms | 31.9 | 32 | 122,618 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,405.7 | 98.1 | 1,413 ms | 10.2 ms | 64.0 | 0 | 192,192 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 9,056.3 | 68.0 | 2,427 ms | 14.7 ms | 128.0 | 0 | 271,737 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 305.0 | 309.6 | 211 ms | 3.2 ms | 1.0 | 2 | 9,154 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 540.4 | 274.5 | 233 ms | 3.6 ms | 2.0 | 4 | 16,211 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 941.5 | 237.3 | 267 ms | 4.2 ms | 4.0 | 4 | 28,246 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,244.0 | 157.0 | 279 ms | 6.4 ms | 7.9 | 8 | 37,315 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,363.2 | 150.0 | 687 ms | 6.7 ms | 15.8 | 16 | 70,888 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,607.8 | 111.2 | 1,698 ms | 9.0 ms | 32.0 | 0 | 108,227 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,063.0 | 76.9 | 2,939 ms | 13.0 ms | 64.0 | 0 | 151,866 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 6,448.1 | 47.7 | 5,103 ms | 21.0 ms | 128.0 | 0 | 193,401 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 274.0 | 280.3 | 328 ms | 3.6 ms | 1.0 | 2 | 8,220 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 488.2 | 250.1 | 347 ms | 4.0 ms | 2.0 | 4 | 14,647 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 864.1 | 219.2 | 463 ms | 4.6 ms | 3.9 | 4 | 25,924 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,106.9 | 140.1 | 430 ms | 7.1 ms | 8.0 | 8 | 33,208 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,942.6 | 123.2 | 386 ms | 8.1 ms | 15.8 | 16 | 58,277 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,762.8 | 85.3 | 2,835 ms | 11.7 ms | 32.0 | 0 | 82,880 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 3,530.4 | 53.6 | 5,714 ms | 18.6 ms | 64.0 | 0 | 105,924 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 232.6 | 238.0 | 566 ms | 4.2 ms | 1.0 | 1 | 6,978 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 422.8 | 217.0 | 642 ms | 4.6 ms | 1.9 | 2 | 12,683 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 748.9 | 191.6 | 617 ms | 5.2 ms | 4.0 | 4 | 22,467 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 936.3 | 118.9 | 1,424 ms | 8.4 ms | 7.9 | 5 | 28,044 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,529.3 | 96.1 | 3,220 ms | 10.4 ms | 16.0 | 0 | 45,874 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,892.3 | 59.4 | 6,090 ms | 16.8 ms | 32.0 | 0 | 56,763 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.