Speed test matrix
Ling-3.0-flash on H200 SXM 141GB ×4
vllm · 0.29.0 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- ling-3.0-flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-ling3-vllm/bin/vllm" serve inclusionAI/Ling-3.0-flash --served-model-name ling-3.0-flash --trust-remote-code --tensor-parallel-size 4 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 262144 --gpu-memory-utilization 0.847543 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --enable-chunked-prefill --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' --enable-prefix-caching --mamba-cache-mode align --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context25.2K325 ms TTFT
16K context32.2K503 ms TTFT
32K context38.9K830 ms TTFT
64K context41.6K1,546 ms TTFT
128K context39.8K3,229 ms TTFT
251K context35.2K7,151 ms TTFT
256K context34.1K7,522 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 289.1152 ms TTFT | 522.2159 ms TTFT | 785.9240 ms TTFT | 1,411.8231 ms TTFT | 2,300.0276 ms TTFT | 3,720.2521 ms TTFT | 5,828.71,110 ms TTFT | 8,783.51,489 ms TTFT |
| 16K | 292.9182 ms TTFT | 497.5200 ms TTFT | 830.3202 ms TTFT | 1,398.5217 ms TTFT | 2,273.5417 ms TTFT | 3,827.1677 ms TTFT | 5,777.81,274 ms TTFT | 8,399.02,431 ms TTFT |
| 32K | 290.6218 ms TTFT | 503.2242 ms TTFT | 812.2247 ms TTFT | 1,418.5260 ms TTFT | 2,166.1374 ms TTFT | 3,468.71,561 ms TTFT | 5,440.92,220 ms TTFT | 7,490.63,859 ms TTFT |
| 64K | 284.3299 ms TTFT | 516.8320 ms TTFT | 787.1337 ms TTFT | 1,370.0350 ms TTFT | 1,980.2534 ms TTFT | 3,230.72,387 ms TTFT | 4,644.54,541 ms TTFT | Over capacity |
| 128K | 254.8473 ms TTFT | 490.4503 ms TTFT | 759.4524 ms TTFT | 1,275.5726 ms TTFT | Looped | 2,691.34,574 ms TTFT | Over capacity | Over capacity |
| 251K | 235.1904 ms TTFT | 423.9969 ms TTFT | 706.81,216 ms TTFT | 1,092.91,615 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.2K | — | 325 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.2K | — | 503 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 38.9K | — | 830 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.6K | — | 1,546 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 39.8K | — | 3,229 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 35.2K | — | 7,151 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 34.1K | — | 7,522 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 289.1 | 299.7 | 152 ms | 3.3 ms | 1.0 | 2 | 8,670 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 522.2 | 271.8 | 159 ms | 3.7 ms | 2.0 | 4 | 15,665 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 785.9 | 205.9 | 240 ms | 4.9 ms | 4.0 | 4 | 23,580 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,411.8 | 186.5 | 231 ms | 5.4 ms | 8.0 | 8 | 42,353 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,300.0 | 147.8 | 276 ms | 6.8 ms | 15.9 | 16 | 69,055 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,720.2 | 118.9 | 521 ms | 8.4 ms | 31.8 | 15 | 111,888 | 30.1 s |
| Sustained decode | 8K | 64 | Valid | 5,828.7 | 95.0 | 1,110 ms | 10.5 ms | 64.0 | 0 | 174,883 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 8,783.5 | 69.4 | 1,489 ms | 14.4 ms | 128.0 | 0 | 263,450 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 292.9 | 297.7 | 182 ms | 3.4 ms | 1.0 | 2 | 8,786 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 497.5 | 269.3 | 200 ms | 3.7 ms | 2.0 | 4 | 14,924 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 830.3 | 209.8 | 202 ms | 4.8 ms | 4.0 | 4 | 24,904 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,398.5 | 182.3 | 217 ms | 5.5 ms | 8.0 | 8 | 41,954 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,273.5 | 145.6 | 417 ms | 6.9 ms | 15.9 | 16 | 68,207 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,827.1 | 120.0 | 677 ms | 8.3 ms | 31.9 | 15 | 114,869 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,777.8 | 91.6 | 1,274 ms | 10.9 ms | 64.0 | 0 | 173,300 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 8,399.0 | 64.7 | 2,431 ms | 15.4 ms | 128.0 | 0 | 252,343 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 290.6 | 298.6 | 218 ms | 3.3 ms | 1.0 | 2 | 8,718 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 503.2 | 260.9 | 242 ms | 3.8 ms | 2.0 | 4 | 15,096 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 812.2 | 210.1 | 247 ms | 4.8 ms | 4.0 | 4 | 24,367 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,418.5 | 179.2 | 260 ms | 5.6 ms | 8.0 | 8 | 42,551 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,166.1 | 138.7 | 374 ms | 7.2 ms | 15.9 | 16 | 64,986 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,468.7 | 111.5 | 1,561 ms | 9.0 ms | 32.0 | 1 | 103,998 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,440.9 | 84.2 | 2,220 ms | 11.9 ms | 64.0 | 0 | 163,252 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 7,490.6 | 56.4 | 3,859 ms | 17.7 ms | 128.0 | 0 | 224,733 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 284.3 | 296.3 | 299 ms | 3.4 ms | 1.0 | 2 | 8,528 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 516.8 | 263.9 | 320 ms | 3.8 ms | 2.0 | 4 | 15,504 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 787.1 | 202.3 | 337 ms | 4.9 ms | 4.0 | 4 | 23,614 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,370.0 | 171.0 | 350 ms | 5.8 ms | 8.0 | 8 | 41,094 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,980.2 | 123.7 | 534 ms | 8.1 ms | 15.9 | 13 | 59,391 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,230.7 | 99.4 | 2,387 ms | 10.1 ms | 32.0 | 0 | 96,894 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,644.5 | 69.8 | 4,541 ms | 14.3 ms | 64.0 | 0 | 139,647 | 30.1 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 254.8 | 275.3 | 473 ms | 3.6 ms | 1.0 | 2 | 7,642 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 490.4 | 252.4 | 503 ms | 4.0 ms | 1.9 | 3 | 14,712 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 759.4 | 195.3 | 524 ms | 5.1 ms | 4.0 | 4 | 22,780 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,275.5 | 160.5 | 726 ms | 6.2 ms | 8.0 | 8 | 38,263 | 30.0 s |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Valid | 2,691.3 | 86.2 | 4,574 ms | 11.6 ms | 32.0 | 0 | 80,768 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 235.1 | 244.0 | 904 ms | 4.1 ms | 1.0 | 1 | 7,054 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 423.9 | 226.3 | 969 ms | 4.4 ms | 2.0 | 2 | 12,716 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 706.8 | 184.2 | 1,216 ms | 5.4 ms | 3.9 | 4 | 21,203 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,092.9 | 143.5 | 1,615 ms | 7.0 ms | 7.8 | 8 | 32,777 | 30.0 s |
| Sustained decode | 251K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.