InferCap
A CLI toolkit for LLM inference preflight, serving configuration, deployment verification, benchmarking, telemetry, and capacity analysis.
What InferCap covers
Hardware preflight
Check Python, PyTorch, vLLM, system memory and NVIDIA GPUs.
Model-family discovery
Filter compatible architectures and rank models likely to fit your GPU.
Serving configuration
Generate vLLM commands with safe, balanced, latency or throughput profiles.
Endpoint verification
Check an OpenAI-compatible endpoint and the model ID being served.
Structured benchmarks
Run concurrency, burst or request-rate tests with TTFT, latency, TPS and RPS.
Saturation analysis
Find the operating point before throughput, TTFT or error rate crosses a threshold.
One workflow from model to capacity
- Choose an exact model or search by family.
- Check compatibility with the available hardware.
- Review and adjust the generated vLLM command.
- Verify the endpoint, then run a benchmark.
- Read telemetry and find a sustainable load.
uv run infercap check --model <model> --jsonThe problem
Deploying an LLM often starts with uncertainty: will the model fit, which serving settings are reasonable, and where does the GPU reach its limit?
The approach
InferCap checks Python, PyTorch, vLLM, system memory, NVIDIA GPUs, model architecture, and estimated model-weight memory; it can recommend vLLM commands, verify endpoints, benchmark load, and analyze saturation.
Available materials
The public repository includes a Python 3.12+ CLI, tests, documentation, quick-start and architecture guides, JSON/PNG benchmark artifacts, and a live web interface.
Contribution & context
Designed and built the project as an open-source toolkit.
Project scope & limitations
The current implementation targets NVIDIA GPUs and vLLM. Memory estimates are feasibility checks and do not guarantee runtime stability.