Speed test matrix
DeepSeek-V4-Flash-nvfp4-DSpark on H200 NVL 141GB ×2
sglang · 0.5.16 · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.16
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
Run
- Served model
- deepseek
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-deepseek-sglang/bin/sglang" serve --model-path nvidia/DeepSeek-V4-Flash-nvfp4-DSpark --served-model-name deepseek --trust-remote-code --tp 2 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --context-length 1048576 --mem-fraction-static 0.876000 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --disable-cuda-graph --max-running-requests 16 --moe-runner-backend marlin --speculative-algorithm DSPARK
Prefill
prompt tok/s by context length8K context13K631 ms TTFT
16K context14.3K1,139 ms TTFT
32K context14.7K2,196 ms TTFT
64K context14.6K4,419 ms TTFT
128K context13.8K9,360 ms TTFT
256K context12.1K21,346 ms TTFT
512K context9,61753,605 ms TTFT
1019K context6,815150,530 ms TTFT
1024K context6,706153,723 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 41.4176 ms TTFT | 90.9208 ms TTFT | 176.21,559 ms TTFT | 425.41,576 ms TTFT | 740.83,386 ms TTFT | Not run | Not run | Not run |
| 16K | 33.2197 ms TTFT | 118.3232 ms TTFT | 161.2254 ms TTFT | 406.0346 ms TTFT | 632.6419 ms TTFT | Not run | Not run | Not run |
| 32K | 37.0222 ms TTFT | 110.2225 ms TTFT | 177.6336 ms TTFT | 368.3447 ms TTFT | 661.2Requests queued811 ms TTFT | Not run | Not run | Over capacity |
| 64K | 42.5289 ms TTFT | 111.1350 ms TTFT | 224.2471 ms TTFT | 361.5865 ms TTFT | 757.8Requests queued1,627 ms TTFT | Not run | Over capacity | Over capacity |
| 128K | 47.8392 ms TTFT | 74.4504 ms TTFT | 169.7826 ms TTFT | 322.61,605 ms TTFT | 586.7Requests queued3,172 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 35.6590 ms TTFT | 102.1928 ms TTFT | 204.12,052 ms TTFT | 414.53,194 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 38.61,037 ms TTFT | 77.11,736 ms TTFT | 162.23,178 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 39.62,030 ms TTFT | 73.63,435 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13K | — | 631 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.3K | — | 1,139 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.7K | — | 2,196 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.6K | — | 4,419 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.8K | — | 9,360 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.1K | — | 21,346 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,617.0 | — | 53,605 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,815.0 | — | 150,530 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,706.0 | — | 153,723 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 41.4 | 42.3 | 176 ms | 23.6 ms | 1.0 | 0 | 1,242 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 90.9 | 45.2 | 208 ms | 22.4 ms | 2.0 | 0 | 2,721 | 29.9 s |
| Sustained decode | 8K | 4 | Valid | 176.2 | 41.6 | 1,559 ms | 24.2 ms | 4.0 | 0 | 5,278 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 425.4 | 43.7 | 1,576 ms | 22.9 ms | 8.0 | 0 | 12,758 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 740.8 | 42.4 | 3,386 ms | 23.6 ms | 16.0 | 0 | 22,209 | 30.0 s |
| Sustained decode | 8K | 32 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 33.2 | 34.3 | 197 ms | 29.2 ms | 1.0 | 0 | 996 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 118.3 | 56.1 | 232 ms | 17.9 ms | 2.0 | 0 | 3,545 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 161.2 | 40.6 | 254 ms | 24.6 ms | 4.0 | 0 | 4,827 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 406.0 | 45.3 | 346 ms | 22.1 ms | 8.0 | 0 | 12,164 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 632.6 | 40.7 | 419 ms | 24.6 ms | 16.0 | 0 | 18,970 | 30.0 s |
| Sustained decode | 16K | 32 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 37.0 | 37.9 | 222 ms | 26.4 ms | 1.0 | 0 | 1,109 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 110.2 | 52.1 | 225 ms | 23.4 ms | 2.0 | 0 | 3,301 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 177.6 | 43.0 | 336 ms | 23.3 ms | 4.0 | 0 | 5,315 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 368.3 | 41.3 | 447 ms | 24.2 ms | 8.0 | 0 | 11,028 | 29.9 s |
| Sustained decode | 32K | 16 | Requests queued | 661.2 | 42.6 | 811 ms | 23.5 ms | 13.9 | 8 | 19,833 | 30.0 s |
| Sustained decode | 32K | 32 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 42.5 | 41.9 | 289 ms | 23.8 ms | 1.0 | 0 | 1,273 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 111.1 | 53.4 | 350 ms | 19.6 ms | 2.0 | 0 | 3,329 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 224.2 | 48.9 | 471 ms | 20.6 ms | 4.0 | 0 | 6,714 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 361.5 | 43.6 | 865 ms | 23.0 ms | 8.0 | 0 | 10,824 | 29.9 s |
| Sustained decode | 64K | 16 | Requests queued | 757.8 | 45.9 | 1,627 ms | 21.8 ms | 14.6 | 15 | 22,709 | 30.0 s |
| Sustained decode | 64K | 32 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 47.8 | 46.7 | 392 ms | 21.4 ms | 1.0 | 0 | 1,432 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 74.4 | 37.0 | 504 ms | 27.4 ms | 2.0 | 0 | 2,230 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 169.7 | 41.6 | 826 ms | 24.0 ms | 4.0 | 0 | 5,092 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 322.6 | 41.6 | 1,605 ms | 24.1 ms | 8.0 | 0 | 9,667 | 30.0 s |
| Sustained decode | 128K | 16 | Requests queued | 586.7 | 42.0 | 3,172 ms | 23.8 ms | 14.2 | 25 | 17,563 | 29.9 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 35.6 | 37.1 | 590 ms | 27.0 ms | 1.0 | 0 | 1,066 | 29.9 s |
| Sustained decode | 256K | 2 | Valid | 102.1 | 49.2 | 928 ms | 20.5 ms | 2.0 | 0 | 3,058 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 204.1 | 49.5 | 2,052 ms | 20.2 ms | 4.0 | 0 | 6,123 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 414.5 | 44.4 | 3,194 ms | 22.6 ms | 8.0 | 0 | 12,414 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 38.6 | 39.8 | 1,037 ms | 25.2 ms | 1.0 | 0 | 1,156 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 77.1 | 38.1 | 1,736 ms | 26.4 ms | 2.0 | 0 | 2,308 | 29.9 s |
| Sustained decode | 512K | 4 | Valid | 162.2 | 41.3 | 3,178 ms | 24.2 ms | 4.0 | 0 | 4,858 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 39.6 | 40.4 | 2,030 ms | 24.8 ms | 1.0 | 0 | 1,187 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 73.6 | 37.8 | 3,435 ms | 26.5 ms | 2.0 | 0 | 2,204 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.