Self-hosted Rust proxy for LLM caching and failover.

OpenAI-compatible reverse proxy with exact cache hits, local semantic cache, reactive failover, Docker deployment, and Prometheus metrics.
✗ cold routed fallback request 53.6ms measured locally
✓ exact cache hit 2.3ms measured locally · $0 upstream cost
→ semantic cache hit: 50.1ms with local BGE-small-en-v1.5
2.3ms
exact cache hit
133 MB
local BGE model, no API
v0.3.0
stable release
$
Drop-in. Point OpenAI-compatible clients at the proxy base_url; keep your existing app flow.
$
Reactive failover. Retry selected transport errors and 5xx/429 responses through a configured fallback provider.
$
Semantic cache. BGE-small-en-v1.5 runs locally on CPU via Candle. Cosine similarity match across conversation buckets. No external embedding API.
$
Prometheus metrics. Scrape /admin/metrics/prometheus for cache hits, misses, upstream calls, and reactive failovers.
$
Docker or binary. Run with Docker Compose for quick testing, or use pre-built Linux and Windows release artifacts.

Download v0.3.0

Pre-built binaries, checksums, Docker instructions, and the Prometheus/failover configuration are available in the release docs.

↓ GitHub Release Docker Quickstart