Speed test matrix
Ling-3.0-flash-fp8 on H200 NVL 141GB ×2
vllm · 0.27.2rc1.dev113+g5cecfc013 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-3.0-flash-fp8 --served-model-name ling --trust-remote-code --tensor-parallel-size 2 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 262144 --gpu-memory-utilization 0.863300 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align
Prefill
prompt tok/s by context length8K context31.3K262 ms TTFT
16K context29.8K545 ms TTFT
32K context33.3K970 ms TTFT
64K context33.3K1,936 ms TTFT
128K context30K4,285 ms TTFT
251K context24.5K10,263 ms TTFT
256K context23K11,173 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 169.564 ms TTFT | 293.981 ms TTFT | 535.9103 ms TTFT | 916.4173 ms TTFT | 1,466.1247 ms TTFT | 2,490.6286 ms TTFT | 3,918.7555 ms TTFT | 5,243.0859 ms TTFT |
| 16K | 167.7132 ms TTFT | 291.0157 ms TTFT | 529.4232 ms TTFT | 904.3306 ms TTFT | 1,425.8561 ms TTFT | 2,372.6860 ms TTFT | 3,675.31,259 ms TTFT | 4,960.21,982 ms TTFT |
| 32K | 164.9169 ms TTFT | 286.4199 ms TTFT | 517.3298 ms TTFT | 881.3404 ms TTFT | 1,397.4593 ms TTFT | 2,285.6928 ms TTFT | 3,418.71,500 ms TTFT | 4,453.62,586 ms TTFT |
| 64K | 159.4226 ms TTFT | 277.2255 ms TTFT | 503.6342 ms TTFT | 848.6579 ms TTFT | 1,311.9889 ms TTFT | 2,024.11,602 ms TTFT | 2,899.52,685 ms TTFT | Over capacity |
| 128K | 149.5371 ms TTFT | 261.2418 ms TTFT | 482.3716 ms TTFT | 777.91,108 ms TTFT | 1,145.51,863 ms TTFT | 1,713.63,401 ms TTFT | Over capacity | Over capacity |
| 251K | 133.0595 ms TTFT | 239.9818 ms TTFT | 440.11,148 ms TTFT | 669.91,889 ms TTFT | 946.23,305 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.3K | — | 262 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.8K | — | 545 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.3K | — | 970 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 33.3K | — | 1,936 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 30K | — | 4,285 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 24.5K | — | 10,263 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 23K | — | 11,173 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 169.5 | 170.0 | 64 ms | 5.9 ms | 1.0 | 1 | 5,084 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 293.9 | 147.4 | 81 ms | 6.8 ms | 2.0 | 2 | 8,818 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 535.9 | 134.5 | 103 ms | 7.4 ms | 4.0 | 4 | 16,075 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 916.4 | 114.8 | 173 ms | 8.7 ms | 8.0 | 0 | 27,488 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,466.1 | 91.7 | 247 ms | 10.9 ms | 16.0 | 0 | 43,984 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,490.6 | 78.3 | 286 ms | 12.8 ms | 32.0 | 0 | 74,720 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,918.7 | 61.7 | 555 ms | 16.2 ms | 64.0 | 0 | 117,576 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,243.0 | 41.3 | 859 ms | 24.2 ms | 128.0 | 0 | 157,320 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 167.7 | 168.6 | 132 ms | 5.9 ms | 1.0 | 1 | 5,030 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 291.0 | 146.2 | 157 ms | 6.8 ms | 2.0 | 2 | 8,735 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 529.4 | 133.4 | 232 ms | 7.5 ms | 3.9 | 4 | 15,892 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 904.3 | 113.2 | 306 ms | 8.8 ms | 8.0 | 0 | 27,129 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,425.8 | 89.4 | 561 ms | 11.2 ms | 16.0 | 0 | 42,768 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,372.6 | 74.9 | 860 ms | 13.4 ms | 32.0 | 0 | 71,179 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,675.3 | 57.4 | 1,259 ms | 17.4 ms | 64.0 | 0 | 110,276 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,960.2 | 37.8 | 1,982 ms | 26.4 ms | 128.0 | 0 | 148,742 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 164.9 | 166.2 | 169 ms | 6.0 ms | 1.0 | 1 | 4,946 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 286.4 | 144.1 | 199 ms | 6.9 ms | 2.0 | 2 | 8,594 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 517.3 | 130.6 | 298 ms | 7.7 ms | 4.0 | 4 | 15,518 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 881.3 | 110.4 | 404 ms | 9.1 ms | 8.0 | 0 | 26,440 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,397.4 | 87.2 | 593 ms | 11.5 ms | 16.0 | 0 | 41,924 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,285.6 | 71.1 | 928 ms | 14.1 ms | 32.0 | 0 | 68,546 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,418.7 | 52.7 | 1,500 ms | 19.0 ms | 64.0 | 0 | 102,534 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,453.6 | 33.6 | 2,586 ms | 29.7 ms | 128.0 | 0 | 133,639 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 159.4 | 161.1 | 226 ms | 6.2 ms | 1.0 | 1 | 4,781 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 277.2 | 139.8 | 255 ms | 7.2 ms | 2.0 | 2 | 8,316 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 503.6 | 127.1 | 342 ms | 7.9 ms | 4.0 | 4 | 15,108 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 848.6 | 105.6 | 579 ms | 9.5 ms | 8.0 | 0 | 25,457 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,311.9 | 81.1 | 889 ms | 12.3 ms | 16.0 | 0 | 39,345 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,024.1 | 62.1 | 1,602 ms | 16.1 ms | 32.0 | 0 | 60,706 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,899.5 | 43.6 | 2,685 ms | 22.9 ms | 64.0 | 0 | 86,979 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 149.5 | 152.1 | 371 ms | 6.6 ms | 1.0 | 1 | 4,484 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 261.2 | 132.9 | 418 ms | 7.5 ms | 2.0 | 2 | 7,839 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 482.3 | 120.6 | 716 ms | 8.3 ms | 4.0 | 1 | 14,469 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 777.9 | 96.8 | 1,108 ms | 10.3 ms | 8.0 | 0 | 23,331 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,145.5 | 70.8 | 1,863 ms | 14.1 ms | 16.0 | 0 | 34,355 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,713.6 | 52.3 | 3,401 ms | 19.1 ms | 32.0 | 0 | 51,392 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 133.0 | 139.1 | 595 ms | 7.2 ms | 1.0 | 1 | 3,989 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 239.9 | 121.0 | 818 ms | 8.3 ms | 2.0 | 1 | 7,198 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 440.1 | 110.3 | 1,148 ms | 9.1 ms | 4.0 | 0 | 13,204 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 669.9 | 84.2 | 1,889 ms | 11.9 ms | 8.0 | 0 | 20,096 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 946.2 | 59.5 | 3,305 ms | 16.8 ms | 16.0 | 0 | 28,384 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.