Speed test matrix
Ling-3.0-tiny on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-3.0-tiny --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 131072 --gpu-memory-utilization 0.885275 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align
Prefill
prompt tok/s by context length8K context65.3K125 ms TTFT
16K context60.9K266 ms TTFT
32K context68.7K470 ms TTFT
64K context69.5K926 ms TTFT
123K context58.9K2,096 ms TTFT
128K context57K2,255 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 334.744 ms TTFT | 585.493 ms TTFT | 1,026.3120 ms TTFT | 1,406.6167 ms TTFT | 3,016.8187 ms TTFT | 5,061.8232 ms TTFT | 8,146.9234 ms TTFT | 13K780 ms TTFT |
| 16K | 327.990 ms TTFT | 574.0168 ms TTFT | 1,002.0202 ms TTFT | 1,376.2268 ms TTFT | 2,891.6349 ms TTFT | 4,755.3440 ms TTFT | 7,598.9773 ms TTFT | 11.4K1,372 ms TTFT |
| 32K | 318.8137 ms TTFT | 560.2193 ms TTFT | 973.6261 ms TTFT | 1,325.4342 ms TTFT | 2,700.9502 ms TTFT | 4,249.3547 ms TTFT | 6,551.31,292 ms TTFT | 9,249.42,378 ms TTFT |
| 64K | 302.6183 ms TTFT | 531.8209 ms TTFT | 930.3244 ms TTFT | 1,238.8334 ms TTFT | 2,403.5434 ms TTFT | 3,658.71,519 ms TTFT | 5,155.02,649 ms TTFT | 6,623.44,728 ms TTFT |
| 123K | 274.2268 ms TTFT | 486.8338 ms TTFT | 867.1319 ms TTFT | 1,117.8376 ms TTFT | 1,985.9438 ms TTFT | 2,872.92,669 ms TTFT | 3,677.25,236 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 65.3K | — | 125 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 60.9K | — | 266 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 68.7K | — | 470 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 69.5K | — | 926 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 58.9K | — | 2,096 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 57K | — | 2,255 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 334.7 | 336.0 | 44 ms | 3.0 ms | 1.0 | 2 | 10,041 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 585.4 | 294.6 | 93 ms | 3.4 ms | 2.0 | 4 | 17,560 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,026.3 | 258.7 | 120 ms | 3.9 ms | 3.9 | 8 | 30,789 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,406.6 | 177.4 | 167 ms | 5.6 ms | 8.0 | 8 | 42,196 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,016.8 | 190.9 | 187 ms | 5.2 ms | 15.7 | 16 | 90,497 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,061.8 | 161.0 | 232 ms | 6.2 ms | 32.0 | 32 | 151,848 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 8,146.9 | 128.2 | 234 ms | 7.8 ms | 64.0 | 64 | 244,397 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 13K | 101.3 | 780 ms | 9.9 ms | 128.0 | 0 | 389,138 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 327.9 | 331.6 | 90 ms | 3.0 ms | 1.0 | 2 | 9,841 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 574.0 | 290.4 | 168 ms | 3.4 ms | 2.0 | 4 | 17,219 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,002.0 | 255.2 | 202 ms | 3.9 ms | 4.0 | 8 | 30,059 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,376.2 | 173.9 | 268 ms | 5.8 ms | 7.7 | 8 | 41,286 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,891.6 | 183.4 | 349 ms | 5.5 ms | 15.7 | 16 | 86,740 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,755.3 | 150.9 | 440 ms | 6.6 ms | 31.0 | 32 | 142,659 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,598.9 | 118.3 | 773 ms | 8.5 ms | 64.0 | 0 | 227,973 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 11.4K | 87.9 | 1,372 ms | 11.4 ms | 128.0 | 0 | 342,409 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 318.8 | 321.9 | 137 ms | 3.1 ms | 1.0 | 2 | 9,564 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 560.2 | 284.2 | 193 ms | 3.5 ms | 1.9 | 4 | 16,804 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 973.6 | 248.2 | 261 ms | 4.0 ms | 3.9 | 8 | 29,207 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,325.4 | 168.1 | 342 ms | 6.0 ms | 7.7 | 8 | 39,764 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,700.9 | 171.8 | 502 ms | 5.8 ms | 15.9 | 16 | 81,028 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,249.3 | 134.8 | 547 ms | 7.4 ms | 31.9 | 32 | 127,466 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,551.3 | 100.3 | 1,292 ms | 10.0 ms | 64.0 | 0 | 196,539 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 9,249.4 | 69.6 | 2,378 ms | 14.4 ms | 128.0 | 0 | 277,522 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 302.6 | 306.6 | 183 ms | 3.3 ms | 1.0 | 2 | 9,082 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 531.8 | 270.0 | 209 ms | 3.7 ms | 2.0 | 4 | 15,954 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 930.3 | 234.7 | 244 ms | 4.3 ms | 4.0 | 4 | 27,909 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,238.8 | 156.5 | 334 ms | 6.4 ms | 7.8 | 8 | 37,158 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,403.5 | 151.4 | 434 ms | 6.6 ms | 15.9 | 16 | 72,102 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,658.7 | 113.4 | 1,519 ms | 8.8 ms | 32.0 | 1 | 109,758 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,155.0 | 78.2 | 2,649 ms | 12.8 ms | 64.0 | 0 | 154,632 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 6,623.4 | 48.5 | 4,728 ms | 20.6 ms | 128.0 | 0 | 198,793 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 274.2 | 279.5 | 268 ms | 3.6 ms | 1.0 | 2 | 8,230 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 486.8 | 249.8 | 338 ms | 4.0 ms | 2.0 | 4 | 14,603 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 867.1 | 218.8 | 319 ms | 4.6 ms | 4.0 | 4 | 26,012 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 1,117.8 | 141.4 | 376 ms | 7.1 ms | 8.0 | 8 | 33,530 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,985.9 | 126.1 | 438 ms | 7.9 ms | 15.8 | 16 | 59,569 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 2,872.9 | 88.8 | 2,669 ms | 11.3 ms | 32.0 | 0 | 86,180 | 30.0 s |
| Sustained decode | 123K | 64 | Valid | 3,677.2 | 55.9 | 5,236 ms | 17.9 ms | 64.0 | 0 | 110,337 | 30.0 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.