Speed test matrix
Ling-3.0-tiny-base on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-3.0-tiny-base --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 262144 --gpu-memory-utilization 0.885200 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align
Prefill
prompt tok/s by context length8K context52.9K155 ms TTFT
16K context61.2K265 ms TTFT
32K context62.3K518 ms TTFT
64K context62.2K1,034 ms TTFT
128K context57.1K2,250 ms TTFT
251K context41.5K6,069 ms TTFT
256K context40.4K6,355 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 337.477 ms TTFT | 594.3133 ms TTFT | 1,031.8170 ms TTFT | 1,409.5231 ms TTFT | 2,953.7290 ms TTFT | 4,866.9439 ms TTFT | 7,734.8Requests queued670 ms TTFT | 12.6K1,092 ms TTFT |
| 16K | 331.6132 ms TTFT | 587.3151 ms TTFT | 1,003.5225 ms TTFT | 1,359.3287 ms TTFT | 2,865.9358 ms TTFT | 4,589.6488 ms TTFT | 7,366.2823 ms TTFT | 11.2K1,377 ms TTFT |
| 32K | 322.4133 ms TTFT | 572.8169 ms TTFT | 986.4251 ms TTFT | 1,348.2337 ms TTFT | 2,661.8453 ms TTFT | 4,129.9761 ms TTFT | 6,487.11,308 ms TTFT | 8,220.8Requests queued1,022 ms TTFT |
| 64K | 304.4192 ms TTFT | 542.8218 ms TTFT | 940.4229 ms TTFT | 1,254.8461 ms TTFT | 2,413.4496 ms TTFT | 3,647.71,582 ms TTFT | 5,150.42,767 ms TTFT | 6,528.95,150 ms TTFT |
| 128K | 273.3323 ms TTFT | 492.1350 ms TTFT | 871.6457 ms TTFT | 1,110.5403 ms TTFT | 1,966.4419 ms TTFT | 2,784.02,925 ms TTFT | 3,588.75,705 ms TTFT | Over capacity |
| 251K | 231.4560 ms TTFT | 423.0679 ms TTFT | 741.7671 ms TTFT | 948.91,454 ms TTFT | 1,528.5697 ms TTFT | 1,859.5989 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 52.9K | — | 155 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 61.2K | — | 265 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 62.3K | — | 518 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 62.2K | — | 1,034 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 57.1K | — | 2,250 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 41.5K | — | 6,069 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 40.4K | — | 6,355 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 337.4 | 339.5 | 77 ms | 2.9 ms | 1.0 | 2 | 10,123 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 594.3 | 300.4 | 133 ms | 3.3 ms | 2.0 | 4 | 17,828 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,031.8 | 262.3 | 170 ms | 3.8 ms | 4.0 | 8 | 30,952 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,409.5 | 177.8 | 231 ms | 5.6 ms | 8.0 | 8 | 42,284 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,953.7 | 187.4 | 290 ms | 5.3 ms | 16.0 | 16 | 88,608 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,866.9 | 156.8 | 439 ms | 6.4 ms | 30.9 | 32 | 145,996 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 7,734.8 | 123.9 | 670 ms | 8.1 ms | 62.9 | 64 | 232,049 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.6K | 97.9 | 1,092 ms | 10.2 ms | 128.0 | 0 | 378,876 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 331.6 | 334.8 | 132 ms | 3.0 ms | 1.0 | 2 | 9,949 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 587.3 | 297.3 | 151 ms | 3.4 ms | 1.9 | 4 | 17,630 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,003.5 | 254.9 | 225 ms | 3.9 ms | 4.0 | 8 | 30,105 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,359.3 | 172.7 | 287 ms | 5.8 ms | 8.0 | 8 | 40,777 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,865.9 | 182.0 | 358 ms | 5.5 ms | 15.5 | 16 | 85,969 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,589.6 | 147.0 | 488 ms | 6.8 ms | 31.5 | 32 | 137,695 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,366.2 | 114.8 | 823 ms | 8.7 ms | 64.0 | 0 | 220,992 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 11.2K | 86.4 | 1,377 ms | 11.6 ms | 128.0 | 0 | 337,282 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 322.4 | 325.9 | 133 ms | 3.1 ms | 1.0 | 2 | 9,671 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 572.8 | 289.9 | 169 ms | 3.4 ms | 2.0 | 4 | 17,184 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 986.4 | 251.0 | 251 ms | 4.0 ms | 3.9 | 8 | 29,597 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,348.2 | 170.5 | 337 ms | 5.9 ms | 8.0 | 8 | 40,441 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,661.8 | 169.1 | 453 ms | 5.9 ms | 15.6 | 16 | 79,858 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,129.9 | 132.2 | 761 ms | 7.6 ms | 31.2 | 32 | 123,883 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,487.1 | 99.6 | 1,308 ms | 10.0 ms | 64.0 | 0 | 194,590 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 8,220.8 | 67.8 | 1,022 ms | 14.8 ms | 125.0 | 127 | 246,637 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 304.4 | 308.6 | 192 ms | 3.2 ms | 1.0 | 2 | 9,130 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 542.8 | 276.2 | 218 ms | 3.6 ms | 2.0 | 4 | 16,284 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 940.4 | 236.7 | 229 ms | 4.2 ms | 3.9 | 4 | 28,210 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,254.8 | 159.2 | 461 ms | 6.3 ms | 8.0 | 8 | 37,637 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,413.4 | 152.8 | 496 ms | 6.5 ms | 15.8 | 16 | 72,399 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,647.7 | 113.0 | 1,582 ms | 8.8 ms | 32.0 | 1 | 109,433 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,150.4 | 78.2 | 2,767 ms | 12.8 ms | 64.0 | 0 | 154,498 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 6,528.9 | 48.5 | 5,150 ms | 20.6 ms | 128.0 | 0 | 195,879 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 273.3 | 279.5 | 323 ms | 3.6 ms | 1.0 | 2 | 8,202 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 492.1 | 253.1 | 350 ms | 4.0 ms | 2.0 | 4 | 14,763 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 871.6 | 220.6 | 457 ms | 4.5 ms | 4.0 | 7 | 26,081 | 29.9 s |
| Sustained decode | 128K | 8 | Valid | 1,110.5 | 140.5 | 403 ms | 7.1 ms | 7.9 | 8 | 33,314 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,966.4 | 124.6 | 419 ms | 8.0 ms | 15.8 | 16 | 58,993 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,784.0 | 86.0 | 2,925 ms | 11.6 ms | 32.0 | 0 | 83,518 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 3,588.7 | 54.6 | 5,705 ms | 18.3 ms | 64.0 | 0 | 107,650 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 231.4 | 236.6 | 560 ms | 4.2 ms | 1.0 | 1 | 6,942 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 423.0 | 216.9 | 679 ms | 4.6 ms | 1.9 | 2 | 12,691 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 741.7 | 190.4 | 671 ms | 5.3 ms | 3.9 | 4 | 22,250 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 948.9 | 119.5 | 1,454 ms | 8.4 ms | 7.9 | 4 | 28,468 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,528.5 | 95.2 | 697 ms | 10.5 ms | 16.0 | 15 | 45,855 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,859.5 | 59.5 | 989 ms | 16.8 ms | 32.0 | 31 | 55,772 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.