Speed test matrix
Laguna-XS-2.1-FP8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9269+g834400705 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9269+g834400705
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
834400705f2d
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_SHARED_EXPERT_TP1=1 "$HOME/env_h200-poolside-sglang/bin/sglang" serve --model-path poolside/Laguna-XS-2.1-FP8 --served-model-name poolside --trust-remote-code --tp 1 --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --context-length 262144 --mem-fraction-static 0.850588 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context41.7K197 ms TTFT
16K context41.5K391 ms TTFT
32K context37.7K855 ms TTFT
64K context30.1K2,135 ms TTFT
128K context20.9K6,131 ms TTFT
251K context13K19,283 ms TTFT
256K context12.9K19,822 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 189.875 ms TTFT | 230.8102 ms TTFT | 605.999 ms TTFT | 865.2132 ms TTFT | 1,567.4196 ms TTFT | 2,409.8336 ms TTFT | Over capacity | Over capacity |
| 16K | 186.7100 ms TTFT | 212.1140 ms TTFT | 585.2146 ms TTFT | 783.2186 ms TTFT | 1,447.2329 ms TTFT | 2,045.3636 ms TTFT | Over capacity | Over capacity |
| 32K | 180.7125 ms TTFT | 125.1156 ms TTFT | 494.2213 ms TTFT | 666.6379 ms TTFT | 1,125.1680 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 170.6173 ms TTFT | 161.2290 ms TTFT | 450.9402 ms TTFT | 724.6791 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 153.8292 ms TTFT | 222.8439 ms TTFT | 417.31,035 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 128.5510 ms TTFT | 208.8902 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.7K | — | 197 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 41.5K | — | 391 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 37.7K | — | 855 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.1K | — | 2,135 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.9K | — | 6,131 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13K | — | 19,283 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.9K | — | 19,822 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 189.8 | 190.4 | 75 ms | 5.3 ms | 1.0 | 1 | 5,693 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 230.8 | 114.7 | 102 ms | 8.7 ms | 2.9 | 0 | 6,924 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 605.9 | 152.3 | 99 ms | 6.6 ms | 4.0 | 4 | 18,176 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 865.2 | 107.7 | 132 ms | 9.3 ms | 10.8 | 0 | 25,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,567.4 | 98.2 | 196 ms | 10.2 ms | 16.0 | 0 | 47,024 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,409.8 | 74.1 | 336 ms | 13.5 ms | 33.6 | 0 | 72,288 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 186.7 | 187.4 | 100 ms | 5.3 ms | 1.0 | 1 | 5,600 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 212.1 | 100.6 | 140 ms | 9.9 ms | 21.2 | 0 | 6,364 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 585.2 | 146.5 | 146 ms | 6.8 ms | 4.0 | 4 | 17,556 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 783.2 | 97.8 | 186 ms | 10.2 ms | 11.3 | 0 | 23,496 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,447.2 | 88.4 | 329 ms | 11.3 ms | 16.5 | 0 | 43,408 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,045.3 | 63.3 | 636 ms | 15.8 ms | 37.9 | 0 | 61,344 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 180.7 | 181.6 | 125 ms | 5.5 ms | 1.0 | 1 | 5,421 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 125.1 | 61.7 | 156 ms | 16.2 ms | 31.9 | 0 | 3,752 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 494.2 | 127.3 | 213 ms | 7.9 ms | 4.7 | 4 | 14,826 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 666.6 | 83.6 | 379 ms | 12.0 ms | 12.0 | 0 | 19,992 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,125.1 | 68.8 | 680 ms | 14.5 ms | 20.6 | 0 | 33,744 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 170.6 | 171.8 | 173 ms | 5.8 ms | 1.0 | 1 | 5,117 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 161.2 | 78.5 | 290 ms | 12.7 ms | 14.3 | 0 | 4,834 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 450.9 | 109.3 | 402 ms | 9.1 ms | 4.6 | 0 | 13,524 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 724.6 | 91.3 | 791 ms | 11.0 ms | 8.0 | 0 | 21,736 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 153.8 | 155.4 | 292 ms | 6.4 ms | 1.0 | 1 | 4,614 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 222.8 | 107.0 | 439 ms | 9.3 ms | 4.4 | 0 | 6,684 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 417.3 | 102.5 | 1,035 ms | 9.8 ms | 4.0 | 0 | 12,516 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 128.5 | 131.0 | 510 ms | 7.6 ms | 1.0 | 1 | 3,854 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 208.8 | 100.2 | 902 ms | 10.0 ms | 2.2 | 0 | 6,264 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.