Speed test matrix
Ling-2.6-flash-fp8 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_DISABLED_KERNELS=FlashInferFp8DeepGEMMDynamicBlockScaledKernel "$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-2.6-flash-fp8 --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.881035 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context22.2K368 ms TTFT
16K context21.9K742 ms TTFT
32K context22.5K1,435 ms TTFT
64K context22.7K2,832 ms TTFT
123K context21.4K5,782 ms TTFT
128K context21K6,118 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 162.468 ms TTFT | 270.7126 ms TTFT | 444.5161 ms TTFT | 648.2223 ms TTFT | 1,058.4330 ms TTFT | 1,663.2496 ms TTFT | 2,594.3692 ms TTFT | 3,739.31,172 ms TTFT |
| 16K | 161.2159 ms TTFT | 268.8239 ms TTFT | 443.2319 ms TTFT | 648.2527 ms TTFT | 1,041.7875 ms TTFT | 1,664.71,532 ms TTFT | 2,560.82,004 ms TTFT | 3,569.23,551 ms TTFT |
| 32K | 159.8178 ms TTFT | 267.0274 ms TTFT | 440.0385 ms TTFT | 638.2504 ms TTFT | 1,033.7851 ms TTFT | 1,587.71,196 ms TTFT | 2,471.51,947 ms TTFT | Over capacity |
| 64K | 156.8245 ms TTFT | 262.1357 ms TTFT | 433.1472 ms TTFT | 627.7752 ms TTFT | 1,007.51,063 ms TTFT | 1,530.51,664 ms TTFT | Over capacity | Over capacity |
| 123K | 151.3375 ms TTFT | 254.6375 ms TTFT | 422.8683 ms TTFT | 604.81,136 ms TTFT | 934.32,004 ms TTFT | 1,410.23,268 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 22.2K | — | 368 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.9K | — | 742 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.5K | — | 1,435 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.7K | — | 2,832 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 21.4K | — | 5,782 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21K | — | 6,118 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 162.4 | 163.0 | 68 ms | 6.1 ms | 1.0 | 1 | 4,871 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 270.7 | 136.8 | 126 ms | 7.3 ms | 2.0 | 2 | 8,122 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 444.5 | 111.3 | 161 ms | 9.0 ms | 4.0 | 0 | 13,336 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 648.2 | 81.4 | 223 ms | 12.3 ms | 8.0 | 0 | 19,440 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,058.4 | 67.2 | 330 ms | 14.9 ms | 16.0 | 0 | 31,743 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,663.2 | 53.5 | 496 ms | 18.7 ms | 32.0 | 0 | 49,888 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,594.3 | 41.6 | 692 ms | 24.0 ms | 64.0 | 0 | 77,822 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,739.3 | 29.7 | 1,172 ms | 33.7 ms | 128.0 | 0 | 112,128 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 161.2 | 162.3 | 159 ms | 6.2 ms | 1.0 | 1 | 4,836 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 268.8 | 136.4 | 239 ms | 7.3 ms | 2.0 | 2 | 8,064 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 443.2 | 111.1 | 319 ms | 9.0 ms | 4.0 | 0 | 13,296 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 648.2 | 81.4 | 527 ms | 12.3 ms | 8.0 | 0 | 19,440 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,041.7 | 66.2 | 875 ms | 15.1 ms | 16.0 | 0 | 31,248 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,664.7 | 53.1 | 1,532 ms | 18.8 ms | 32.0 | 0 | 49,919 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,560.8 | 40.2 | 2,004 ms | 24.9 ms | 64.0 | 0 | 76,800 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,569.2 | 27.0 | 3,551 ms | 37.0 ms | 128.0 | 0 | 107,008 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 159.8 | 161.1 | 178 ms | 6.2 ms | 1.0 | 1 | 4,793 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 267.0 | 136.0 | 274 ms | 7.4 ms | 2.0 | 2 | 8,008 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 440.0 | 110.3 | 385 ms | 9.1 ms | 4.0 | 0 | 13,204 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 638.2 | 80.2 | 504 ms | 12.5 ms | 8.0 | 0 | 19,144 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,033.7 | 65.8 | 851 ms | 15.2 ms | 16.0 | 0 | 31,008 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,587.7 | 50.5 | 1,196 ms | 19.8 ms | 32.0 | 0 | 47,612 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,471.5 | 39.0 | 1,947 ms | 25.7 ms | 64.0 | 0 | 74,108 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 156.8 | 158.7 | 245 ms | 6.3 ms | 1.0 | 1 | 4,705 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 262.1 | 134.7 | 357 ms | 7.4 ms | 2.0 | 2 | 7,866 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 433.1 | 108.4 | 472 ms | 9.2 ms | 4.0 | 0 | 12,999 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 627.7 | 78.6 | 752 ms | 12.7 ms | 8.0 | 0 | 18,824 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,007.5 | 63.3 | 1,063 ms | 15.8 ms | 16.0 | 0 | 30,224 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,530.5 | 47.8 | 1,664 ms | 20.9 ms | 32.0 | 0 | 45,919 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 151.3 | 154.1 | 375 ms | 6.5 ms | 1.0 | 1 | 4,538 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 254.6 | 129.1 | 375 ms | 7.7 ms | 2.0 | 2 | 7,638 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 422.8 | 105.6 | 683 ms | 9.5 ms | 4.0 | 0 | 12,684 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 604.8 | 75.3 | 1,136 ms | 13.3 ms | 8.0 | 0 | 18,144 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 934.3 | 58.4 | 2,004 ms | 17.1 ms | 16.0 | 0 | 28,030 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,410.2 | 43.6 | 3,268 ms | 22.9 ms | 32.0 | 0 | 42,304 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.