Speed test matrix
limite-1b-base on H200 NVL 141GB
vllm · 0.26.0 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- limite-1b-base
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_PLUGINS=limite "$HOME/env_h200-paradigma-inc-vllm-v0260/bin/vllm" serve paradigma-inc/limite-1b-base --served-model-name limite-1b-base --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context98.9K83 ms TTFT
16K context96.3K168 ms TTFT
32K context103.6K312 ms TTFT
64K context89K724 ms TTFT
123K context70.5K1,753 ms TTFT
128K context68K1,893 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 300.755 ms TTFT | 587.865 ms TTFT | 1,154.467 ms TTFT | 2,175.7120 ms TTFT | Looped | 6,879.0256 ms TTFT | Looped | Looped |
| 16K | 297.284 ms TTFT | 578.686 ms TTFT | 1,125.6135 ms TTFT | Looped | 3,766.0236 ms TTFT | Looped | Looped | Looped |
| 32K | Looped | 562.4143 ms TTFT | 1,084.3148 ms TTFT | 1,956.8201 ms TTFT | Looped | 5,210.6451 ms TTFT | Looped | Looped |
| 64K | 279.3171 ms TTFT | 535.9203 ms TTFT | 1,001.0193 ms TTFT | 1,748.8225 ms TTFT | 2,792.6261 ms TTFT | Looped | Looped | 5,989.85,535 ms TTFT |
| 123K | 254.8321 ms TTFT | 489.3343 ms TTFT | 877.6338 ms TTFT | Looped | 2,058.8384 ms TTFT | Looped | 3,323.95,520 ms TTFT | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 98.9K | — | 83 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 96.3K | — | 168 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 103.6K | — | 312 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 89K | — | 724 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 70.5K | — | 1,753 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 68K | — | 1,893 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 300.7 | 301.8 | 55 ms | 3.3 ms | 1.0 | 2 | 9,019 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 587.8 | 295.4 | 65 ms | 3.4 ms | 1.9 | 4 | 17,636 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,154.4 | 289.6 | 67 ms | 3.5 ms | 4.0 | 8 | 34,631 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,175.7 | 274.4 | 120 ms | 3.6 ms | 8.0 | 16 | 65,273 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 6,879.0 | 217.3 | 256 ms | 4.6 ms | 32.0 | 32 | 206,371 | 30.0 s |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 297.2 | 299.2 | 84 ms | 3.3 ms | 1.0 | 2 | 8,916 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 578.6 | 291.3 | 86 ms | 3.4 ms | 2.0 | 4 | 17,359 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,125.6 | 284.1 | 135 ms | 3.5 ms | 4.0 | 8 | 33,778 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 3,766.0 | 237.3 | 236 ms | 4.2 ms | 15.9 | 16 | 112,983 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Valid | 562.4 | 284.1 | 143 ms | 3.5 ms | 2.0 | 4 | 16,871 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,084.3 | 273.5 | 148 ms | 3.7 ms | 4.0 | 8 | 32,528 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,956.8 | 248.1 | 201 ms | 4.0 ms | 7.9 | 16 | 58,703 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 5,210.6 | 164.2 | 451 ms | 6.1 ms | 31.6 | 32 | 156,297 | 30.0 s |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 279.3 | 282.5 | 171 ms | 3.5 ms | 1.0 | 2 | 8,379 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 535.9 | 271.8 | 203 ms | 3.7 ms | 2.0 | 4 | 16,074 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,001.0 | 253.5 | 193 ms | 3.9 ms | 3.9 | 8 | 30,029 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,748.8 | 219.7 | 225 ms | 4.6 ms | 8.0 | 8 | 52,478 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,792.6 | 175.5 | 261 ms | 5.7 ms | 15.9 | 16 | 83,782 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Valid | 5,989.8 | 46.9 | 5,535 ms | 21.3 ms | 128.0 | 0 | 179,712 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 254.8 | 260.4 | 321 ms | 3.8 ms | 1.0 | 2 | 7,643 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 489.3 | 247.7 | 343 ms | 4.0 ms | 2.0 | 2 | 14,678 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 877.6 | 221.7 | 338 ms | 4.5 ms | 4.0 | 4 | 26,328 | 30.0 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Valid | 2,058.8 | 130.5 | 384 ms | 7.7 ms | 15.8 | 16 | 61,760 | 30.0 s |
| Sustained decode | 123K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 64 | Valid | 3,323.9 | 51.4 | 5,520 ms | 19.4 ms | 64.0 | 0 | 99,712 | 30.0 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.