Speed test matrix
Laguna-XS-2.1-FP8 on H200 NVL 141GB
vllm · 0.26.0 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 "$HOME/env_h200-poolside-laguna-xs-vllm/bin/vllm" serve poolside/Laguna-XS-2.1-FP8 --served-model-name poolside --trust-remote-code --tensor-parallel-size 1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --tool-call-parser poolside_v1 --max-model-len 262144 --gpu-memory-utilization 0.878438 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser poolside_v1 --max-num-batched-tokens 32768 --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash-FP8","num_speculative_tokens":15,"method":"dflash"}'Prefill
prompt tok/s by context length8K context42.6K192 ms TTFT
16K context39.5K410 ms TTFT
32K context24.3K1,328 ms TTFT
64K context9,7086,618 ms TTFT
128K context21.5K5,968 ms TTFT
251K context13.8K18,246 ms TTFT
256K context12.5K20,462 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 136.652 ms TTFT | 244.271 ms TTFT | 414.3118 ms TTFT | 688.9190 ms TTFT | 1,035.3262 ms TTFT | 1,348.4397 ms TTFT | 1,659.39,581 ms TTFT | 1,966.79,166 ms TTFT |
| 16K | 122.081 ms TTFT | 219.556 ms TTFT | 372.0222 ms TTFT | 592.1194 ms TTFT | 858.1314 ms TTFT | 1,091.2449 ms TTFT | 1,289.4815 ms TTFT | 1,485.07,275 ms TTFT |
| 32K | 119.3105 ms TTFT | 209.2128 ms TTFT | 340.4198 ms TTFT | 517.9287 ms TTFT | 712.7425 ms TTFT | 872.6731 ms TTFT | 994.71,335 ms TTFT | Over capacity |
| 64K | 110.5170 ms TTFT | 188.6223 ms TTFT | 290.6274 ms TTFT | 413.1525 ms TTFT | 536.4784 ms TTFT | 618.71,450 ms TTFT | Over capacity | Over capacity |
| 128K | 99.0288 ms TTFT | 159.2373 ms TTFT | 229.8532 ms TTFT | 300.3888 ms TTFT | 358.61,604 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 84.8535 ms TTFT | 126.0758 ms TTFT | 163.71,119 ms TTFT | 197.11,777 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 42.6K | — | 192 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 39.5K | — | 410 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 24.3K | — | 1,328 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,708.0 | — | 6,618 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 21.5K | — | 5,968 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.8K | — | 18,246 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.5K | — | 20,462 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 136.6 | 136.4 | 52 ms | 7.3 ms | 1.0 | 1 | 4,097 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 244.2 | 120.1 | 71 ms | 8.3 ms | 2.0 | 1 | 7,330 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 414.3 | 102.6 | 118 ms | 9.8 ms | 4.0 | 0 | 12,430 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 688.9 | 86.4 | 190 ms | 11.6 ms | 8.0 | 0 | 20,667 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,035.3 | 63.6 | 262 ms | 15.7 ms | 16.0 | 0 | 31,049 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,348.4 | 35.9 | 397 ms | 27.8 ms | 32.0 | 0 | 40,426 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,659.3 | 23.3 | 9,581 ms | 43.3 ms | 64.0 | 0 | 49,753 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,966.7 | 15.3 | 9,166 ms | 65.5 ms | 128.0 | 0 | 59,004 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 122.0 | 122.4 | 81 ms | 8.2 ms | 1.0 | 1 | 3,659 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 219.5 | 109.9 | 56 ms | 9.1 ms | 2.0 | 0 | 6,588 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 372.0 | 93.3 | 222 ms | 10.7 ms | 4.0 | 0 | 11,160 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 592.1 | 74.2 | 194 ms | 13.5 ms | 8.0 | 0 | 17,755 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 858.1 | 53.8 | 314 ms | 18.6 ms | 16.0 | 0 | 25,737 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,091.2 | 34.3 | 449 ms | 29.2 ms | 32.0 | 0 | 32,710 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,289.4 | 18.5 | 815 ms | 54.0 ms | 64.0 | 0 | 38,690 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,485.0 | 11.5 | 7,275 ms | 86.8 ms | 128.0 | 0 | 44,534 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 119.3 | 119.4 | 105 ms | 8.4 ms | 1.0 | 0 | 3,580 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 209.2 | 104.8 | 128 ms | 9.5 ms | 2.0 | 0 | 6,275 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 340.4 | 85.4 | 198 ms | 11.7 ms | 4.0 | 0 | 10,211 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 517.9 | 64.9 | 287 ms | 15.4 ms | 8.0 | 0 | 15,537 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 712.7 | 44.7 | 425 ms | 22.4 ms | 16.0 | 0 | 21,370 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 872.6 | 27.1 | 731 ms | 36.9 ms | 32.0 | 0 | 26,161 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 994.7 | 15.5 | 1,335 ms | 64.5 ms | 64.0 | 0 | 29,787 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 110.5 | 110.6 | 170 ms | 9.0 ms | 1.0 | 0 | 3,315 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 188.6 | 94.4 | 223 ms | 10.6 ms | 2.0 | 0 | 5,656 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 290.6 | 72.8 | 274 ms | 13.7 ms | 4.0 | 0 | 8,714 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 413.1 | 51.8 | 525 ms | 19.3 ms | 8.0 | 0 | 12,393 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 536.4 | 33.6 | 784 ms | 29.8 ms | 16.0 | 0 | 16,086 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 618.7 | 19.5 | 1,450 ms | 51.4 ms | 32.0 | 0 | 18,540 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 99.0 | 99.2 | 288 ms | 10.1 ms | 1.0 | 0 | 2,969 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 159.2 | 79.7 | 373 ms | 12.6 ms | 2.0 | 0 | 4,776 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 229.8 | 57.6 | 532 ms | 17.4 ms | 4.0 | 0 | 6,890 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 300.3 | 37.6 | 888 ms | 26.6 ms | 8.0 | 0 | 9,007 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 358.6 | 22.5 | 1,604 ms | 44.4 ms | 16.0 | 0 | 10,750 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 84.8 | 85.0 | 535 ms | 11.8 ms | 1.0 | 0 | 2,543 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 126.0 | 63.2 | 758 ms | 15.8 ms | 2.0 | 0 | 3,780 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 163.7 | 41.2 | 1,119 ms | 24.3 ms | 4.0 | 0 | 4,908 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 197.1 | 24.8 | 1,777 ms | 40.3 ms | 8.0 | 0 | 5,906 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.