Speed test matrix
INTELLECT-3 on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- intellect
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-primeintellect-sglang/bin/sglang" serve --model-path PrimeIntellect/INTELLECT-3 --served-model-name intellect --tp 2 --reasoning-parser deepseek-r1 --tool-call-parser qwen3_coder --context-length 131072 --mem-fraction-static 0.850148 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context16.3K503 ms TTFT
16K context14.7K1,100 ms TTFT
32K context11.9K2,720 ms TTFT
64K context8,4457,634 ms TTFT
123K context5,49122,543 ms TTFT
128K context5,35824,035 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 141.885 ms TTFT | 223.0101 ms TTFT | 344.2113 ms TTFT | 443.0138 ms TTFT | 607.4203 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 137.597 ms TTFT | 70.8120 ms TTFT | 140.7134 ms TTFT | 326.2207 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 130.6116 ms TTFT | 81.7171 ms TTFT | 149.7215 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 119.2165 ms TTFT | 86.2244 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 102.1270 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.3K | — | 503 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.7K | — | 1,100 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.9K | — | 2,720 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,445.0 | — | 7,634 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,491.0 | — | 22,543 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,358.0 | — | 24,035 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 141.8 | 142.7 | 85 ms | 7.0 ms | 1.0 | 1 | 4,254 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 223.0 | 105.9 | 101 ms | 9.4 ms | 2.0 | 0 | 6,686 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 344.2 | 84.6 | 113 ms | 11.8 ms | 4.0 | 0 | 10,320 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 443.0 | 54.9 | 138 ms | 18.2 ms | 9.9 | 0 | 13,280 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 607.4 | 38.3 | 203 ms | 26.1 ms | 24.0 | 0 | 18,223 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 137.5 | 138.3 | 97 ms | 7.2 ms | 1.0 | 1 | 4,121 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 70.8 | 35.3 | 120 ms | 28.3 ms | 22.0 | 0 | 2,122 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 140.7 | 35.2 | 134 ms | 28.4 ms | 22.0 | 0 | 4,220 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 326.2 | 40.0 | 207 ms | 25.0 ms | 14.0 | 0 | 9,776 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 130.6 | 131.6 | 116 ms | 7.6 ms | 1.0 | 1 | 3,918 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 81.7 | 40.9 | 171 ms | 24.5 ms | 13.7 | 0 | 2,450 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 149.7 | 37.3 | 215 ms | 26.8 ms | 13.2 | 0 | 4,488 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 119.2 | 119.2 | 165 ms | 8.4 ms | 1.0 | 0 | 3,575 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 86.2 | 43.2 | 244 ms | 23.1 ms | 7.7 | 0 | 2,586 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 102.1 | 102.1 | 270 ms | 9.8 ms | 1.0 | 0 | 3,065 | 30.0 s |
| Sustained decode | 123K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.