LLMix
A production LLM call layer for AI agents and tools: keep your existing SDKs, hot-swap models via MDA presets, and add cache, retries, circuit breakers, and key rotation.
- Source repo
- sno-ai/llmix
- Stars
- ★ 132
- Last updated
- 21d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 67/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Multilingual teams: a Python service, a TypeScript service, and a Rust worker must follow the same model-call contract (canonical cache keys, retry, and key-pool behavior aligned across runtimes).
- Not a fit if
- Single-script projects with one API key
- Teams needing streaming or a prompt framework
- Runtimes older than Python 3.14+, TypeScript 5.0+, or Rust 1.83+
- Source review
- 67/100 · Some gaps
What does this agent do, and when should you use it?
LLMix (repository sno-ai/llmix) is the call layer between your product and the provider SDK, shipped for Python, TypeScript, and Rust. It does not ask you to rewrite OpenAI, Anthropic, Gemini, LiteLLM, or AI SDK client code; it wraps the call and adds response caching (L1 memory plus optional Redis L2), circuit breakers, key-pool rotation, singleflight request collapsing, jittered exponential-backoff retries, and AIMD adaptive concurrency. Model configuration becomes data: MDA presets are signed and published as a compiled registry that services load at runtime through a current.json pointer, so a model swap needs no redeploy. Core components include CallPipeline, KeyPool, TwoTierCache, and ConfigRegistryManager. The project is Apache-2.0 licensed and distributed as npm, PyPI, and crates.io packages.
LLMix wraps each provider call inside a CallPipeline: it builds the request from config (provider, model, temperature, etc.), checks the TwoTierCache, then passes through a circuit breaker scoped by provider and base URL, singleflight deduplication, KeyPool rotation (429 rotation, 401/403 dead-key eviction), a retry loop that honors Retry-After, and finally a dispatch helper such as openaiDispatch, anthropicDispatch, or geminiDispatch. MDA presets are validated, integrity-checked, signed (did:web), and released via the mda CLI; llmix publish-registry generates current.json and the compiled/ directory, and runtime code loads it with ConfigRegistryManager using an external trust anchor for tamper rejection. The Rust build exposes the same pipeline contract behind llmix-rs feature gates (providers-openai, redis).
- Multilingual teams: a Python service, a TypeScript service, and a Rust worker must follow the same model-call contract (canonical cache keys, retry, and key-pool behavior aligned across runtimes).
- Rate-limited teams: need a round-robin key pool that rotates on 429 and evicts dead keys on 401/403.
- High-traffic products: many users ask the same thing at once, so singleflight collapses identical concurrent requests into one upstream call.
- Services that swap models frequently without redeploying: publish a signed registry release, reload, and the next request runs a different provider or model.
- Teams needing secure config governance: MDA CLI signing, integrity checks, and doctor runs give auditable, tamper-rejecting production configuration.
- Tool builders needing one call shape across providers: the same pipeline adapts to OpenAI, Anthropic, Gemini, OpenRouter, DeepInfra, Novita, Together, and Sno GPU.
How do you install or deploy this agent?
Install per runtime: TypeScript via npm install @snoai/llmix; Python via pip install sno-llmix (PyPI name is sno-llmix, import path is still llmix; Redis support with pip install "sno-llmix[redis]"); Rust via cargo add llmix-rs (OpenAI helper and Redis cache via cargo add llmix-rs --features providers-openai,redis). Requires Python 3.14+, TypeScript 5.0+, or Rust 1.83+. Provider SDKs are optional dependencies — install only the clients you call — and set environment variables such as OPENAI_API_KEY / ANTHROPIC_API_KEY / GEMINI_API_KEY (plural _KEYS accepts comma-separated pools; set REDIS_URL for the Redis cache).
How do you use this agent?
TypeScript example: create new CallPipeline({ dispatch: openaiDispatch(), responseCache: new TwoTierCache("memory") }), register a key pool with pipeline.setKeyPool("openai", new KeyPool([process.env.OPENAI_API_KEY!])), then call pipeline.call({ config: { provider: "openai", model: "gpt-4o-mini", common: { temperature: 0.2, maxOutputTokens: 512 }, caching: { strategy: "memory" } }, messages: [...] }) and read response.content. Python and Rust expose the same CallPipeline / KeyPool / TwoTierCache contract. For production, use the registry: initialize presets with the mda CLI (mda init --template llmix-preset ...), sign them, run llmix publish-registry --root config/llm --release-plan release/plan.json ..., and load at runtime via ConfigRegistryManager.open("config/llm", { signedRoot: ... }), verifying with llmix check-registry --tamper-proof.
What are this agent's strengths and limitations?
- Aligned pipeline contract across Python, TypeScript, and Rust with canonical cross-runtime cache keys — a concrete fit for polyglot microservices sharing one call layer.
- Operational primitives (response cache, circuit breaker, key pools, singleflight, adaptive concurrency) ship out of the box as composable classes like CallPipeline, KeyPool, and TwoTierCache.
- The signed MDA registry with did:web / Sigstore/Rekor trust chains provides a complete release, verification, and tamper-rejection flow for production config.
- Model switching happens through data (presets and registry) with no redeploy, positioned explicitly as a harness rather than a router.
- High version floors: Python 3.14+, TypeScript 5.0+, Rust 1.83+ — older runtimes cannot adopt it.
- Rust provider helpers are explicitly beta, limited to OpenAI, Anthropic, Gemini, and Sno GPU — less complete than the other two languages.
- No streaming support (streaming stays with your SDK) and no prompt framework — you must supply those layers yourself.
- The registry release flow involves mda CLI signing, trust policies, did:web keys, and an external trust anchor — nontrivial setup and ops cost; low value for a single-script, single-key project.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| LLMix This agent | 67 · Some gaps | Library / SDKFree + model costs | ★ 132 | 21d ago | Python | OpenAI API · Claude API |
| Adaline Gateway | 69 · Some gaps | Library / SDKFree + model costs | ★ 607 | 2mo ago | TypeScript | OpenAI API · Claude API |
| Cersei — Rust SDK for Coding Agents | 60 · Some gaps | Library / SDKFree + model costs | ★ 461 | 2mo ago | Rust | OpenAI API · Claude API |
| Instructor | 55 · Major gaps | Library / SDKFree + model costs | ★ 14k | 5d ago | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Least-privilege design is well evidenced: caller-supplied dispatch, keys held in memory only, kill-switch state dir permissions delegated explicitly; key rotation and dead-key eviction documented. Deductions: no user-confirmation mechanism (acceptable for a library, but scored 1); dependency-security evidence is thin — no lockfile or dependency manifests in the provided files; source attribution is only an org name with unverified publisher; the INTERNAL_SERVICE_SECRET alias for SNO_LLM_API_KEY in test code is a minor covert fallback path (test-only, but must be flagged).
CI builds, type-checks and tests all three languages (ci.yml); tests assert cache hits, error classification, and fast failure. Deductions: integration tests depend on real keys and external services (Redis, GPU endpoint), so much of the suite silently skips without them; README examples use inconsistent model names (gpt-5-mini vs gpt-4o-mini), slightly weakening contract credibility.
Audience and scenarios are clearly framed ('What This Is Not'); capability boundaries are honest (Rust helpers marked beta, not a streaming framework). Deductions: the env-var matrix is broad but trigger precision (rotation timing, breaker thresholds) is described only qualitatively without defaults; the Python 3.14+ floor is unusually high and restricts environment fit.
Information architecture is excellent: README → runtime guides → secure-config runbook reading order, translations, install table; full Apache-2.0 license text present. Deductions: no CHANGELOG, monorepo package. carries no version, version parity is only asserted via a script; maintenance responsibility rests on a 48h/14d SLA promise in SECURITY.md with no governance files to back it.
Tri-language parity Quick Starts and a response object with content/usage/cache_hit/thinking_content give above-average output usability; the marginal-value claim (cache, singleflight, breaker around each call) is clear. Deductions: no execution evidence for performance or cost benefit; core implementation source was not in the evidence set, so doc–code agreement is unconfirmed.
Strong security claims (MDA signing, Rekor/Sigstore, tamper rejection) are documented but not demonstrated in the provided evidence, scored 1; cross-source corroboration between README, SECURITY.md, CI, and tests on key handling and test scope is mostly consistent (2); test comments separate fact from design intent (e.g., the total_tokens note), so fact/inference separation is acceptable.
- MDA signing and tamper rejection are documentation claims only; no implementation evidence was in this static review — verify the registry verification flow before adoption.
- Integration tests silently skip without real API keys, REDIS_URL, or GPU_BASE_URL; green CI does not prove end-to-end behavior.
- The test code lets SNO_LLM_API_KEY fall back to INTERNAL_SERVICE_SECRET; do not repurpose that env var name in deployment.
- No CHANGELOG and no version in the root package.; verify published package versions against docs before upgrading.
- The Python 3.14+ floor is aggressive; confirm runtime compatibility before adopting.
- Publisher identity is unverified by the curated registry; assess supply-chain trust independently.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
v2.2.0 v2.2.0 — gateway-addressed models keep their effort and their price
Release notes excerpt
Model capability detection and pricing lookup now strip a gateway vendor prefix before matching. A model addressed as openai/gpt-5.6-luna or z-ai/glm-5.2 previously failed every anchored rule, so it was classified as a standard model, reasoningEffort was silently deleted from the request, and the pricing lookup returned nothing. Setting an effort level had no effect and produced no warning. Fixed …View on GitHub ↗ -
v2.0.0 LLMix 2.0.0
Release notes excerpt
LLMix 2.0.0 is the production-ready rewrite of the LLM orchestration layer for AI agents, AI tools, and config-driven model workflows. Highlights - Config-driven model swaps with MDA presets, so provider, model, and runtime parameters can move out of application code. - Keep the SDK you already use: OpenAI, Anthropic, Gemini, AI SDK v6, LiteLLM, OpenRouter, DeepInfra, Novita, Together, Sno GPU, or any async callable. …View on GitHub ↗