Speed test matrix
Laguna-S-2.1 on H200 SXM 141GB ×4
sglang · 0.5.19 · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.19
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
Run
- Served model
- poolside
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nex-n2-sglang-v0519/bin/sglang" serve --model-path poolside/Laguna-S-2.1 --served-model-name poolside --trust-remote-code --tp 4 --reasoning-parser poolside_v1 --tool-call-parser poolside_v1 --context-length 1048576 --mem-fraction-static 0.779676 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype auto --default-chat-template-kwargs '{"enable_thinking":true}' --attention-backend fa3Prefill
prompt tok/s by context length8K context15.2K540 ms TTFT
16K context29.8K544 ms TTFT
32K context29.4K1,094 ms TTFT
64K context29.1K2,205 ms TTFT
128K context29.2K4,399 ms TTFT
256K context24.4K10,492 ms TTFT
512K context17.2K29,886 ms TTFT
1019K context10.5K96,699 ms TTFT
1024K context10.6K96,787 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 175.1121 ms TTFT | Looped | 527.0159 ms TTFT | Looped | Looped | Looped | Looped | Over capacity |
| 16K | 173.9139 ms TTFT | Looped | Looped | Looped | 1,335.6633 ms TTFT | Looped | Looped | Over capacity |
| 32K | 172.0169 ms TTFT | Looped | Looped | Looped | Looped | Looped | Over capacity | Over capacity |
| 64K | 168.7236 ms TTFT | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 128K | 162.8362 ms TTFT | 272.6546 ms TTFT | 471.51,052 ms TTFT | 744.62,062 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | Looped | 248.21,087 ms TTFT | 422.02,108 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 133.41,256 ms TTFT | 218.12,295 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 115.52,769 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.2K | — | 540 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.8K | — | 544 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.4K | — | 1,094 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.1K | — | 2,205 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29.2K | — | 4,399 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 24.4K | — | 10,492 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 17.2K | — | 29,886 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 10.5K | — | 96,699 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 10.6K | — | 96,787 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 175.1 | 175.9 | 121 ms | 5.7 ms | 1.0 | 1 | 5,252 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 527.0 | 133.3 | 159 ms | 7.5 ms | 3.9 | 4 | 15,808 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 173.9 | 174.8 | 139 ms | 5.7 ms | 1.0 | 1 | 5,216 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 1,335.6 | 84.0 | 633 ms | 11.9 ms | 16.0 | 0 | 40,064 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 172.0 | 173.0 | 169 ms | 5.8 ms | 1.0 | 1 | 5,158 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 168.7 | 170.1 | 236 ms | 5.9 ms | 1.0 | 1 | 5,060 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 162.8 | 164.9 | 362 ms | 6.1 ms | 1.0 | 1 | 4,884 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 272.6 | 138.7 | 546 ms | 7.2 ms | 2.0 | 2 | 8,175 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 471.5 | 118.7 | 1,052 ms | 8.4 ms | 4.0 | 0 | 14,144 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 744.6 | 94.7 | 2,062 ms | 10.6 ms | 8.0 | 0 | 22,336 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 2 | Valid | 248.2 | 129.2 | 1,087 ms | 7.7 ms | 1.9 | 2 | 7,446 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 422.0 | 107.6 | 2,108 ms | 9.3 ms | 4.0 | 0 | 12,661 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 133.4 | 139.5 | 1,256 ms | 7.2 ms | 1.0 | 1 | 4,002 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 218.1 | 110.9 | 2,295 ms | 9.0 ms | 2.0 | 0 | 6,542 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 115.5 | 115.5 | 2,769 ms | 8.7 ms | 1.0 | 0 | 3,464 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.