Speed test matrix
INTELLECT-3.1 on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- intellect
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-primeintellect-sglang/bin/sglang" serve --model-path PrimeIntellect/INTELLECT-3.1 --served-model-name intellect --tp 2 --reasoning-parser deepseek-r1 --tool-call-parser qwen3_coder --context-length 131072 --mem-fraction-static 0.850148 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context16.4K500 ms TTFT
16K context14.8K1,094 ms TTFT
32K context11.9K2,708 ms TTFT
64K context8,4837,600 ms TTFT
123K context5,48622,562 ms TTFT
128K context5,34624,091 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 142.182 ms TTFT | 224.4111 ms TTFT | 346.8119 ms TTFT | 442.5143 ms TTFT | 604.2211 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 137.991 ms TTFT | 70.9137 ms TTFT | 142.1126 ms TTFT | 325.8225 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 130.7114 ms TTFT | 79.6167 ms TTFT | 143.4219 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 119.4172 ms TTFT | 85.6243 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 102.2247 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.4K | — | 500 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.8K | — | 1,094 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.9K | — | 2,708 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,483.0 | — | 7,600 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,486.0 | — | 22,562 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,346.0 | — | 24,091 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 142.1 | 143.0 | 82 ms | 7.0 ms | 1.0 | 1 | 4,262 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 224.4 | 106.6 | 111 ms | 9.4 ms | 2.0 | 0 | 6,728 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 346.8 | 85.4 | 119 ms | 11.7 ms | 4.0 | 0 | 10,404 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 442.5 | 54.7 | 143 ms | 18.3 ms | 9.9 | 0 | 13,272 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 604.2 | 38.0 | 211 ms | 26.3 ms | 24.0 | 0 | 18,128 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 137.9 | 138.8 | 91 ms | 7.2 ms | 1.0 | 1 | 4,137 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 70.9 | 35.3 | 137 ms | 28.3 ms | 22.3 | 0 | 2,126 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 142.1 | 35.5 | 126 ms | 28.1 ms | 22.0 | 0 | 4,260 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 325.8 | 40.0 | 225 ms | 25.0 ms | 14.0 | 0 | 9,760 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 130.7 | 131.8 | 114 ms | 7.6 ms | 1.0 | 1 | 3,917 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 79.6 | 39.8 | 167 ms | 25.1 ms | 13.9 | 0 | 2,386 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 143.4 | 35.8 | 219 ms | 27.9 ms | 13.5 | 0 | 4,300 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 119.4 | 119.4 | 172 ms | 8.4 ms | 1.0 | 0 | 3,578 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 85.6 | 43.1 | 243 ms | 23.2 ms | 7.7 | 0 | 2,568 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 102.2 | 102.3 | 247 ms | 9.8 ms | 1.0 | 0 | 3,066 | 30.0 s |
| Sustained decode | 123K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.