ODS Private AI Server
Deploy private inference, chat, agents, voice, RAG, image generation, and automated workflows on your own computer.
The evidence shows localhost-bound defaults, a recommendation to install from a non-administrator Windows session, narrowly scoped GitHub workflow permissions, commit-pinned Actions, and APE tests for API-key enforcement, per-session and per-intent limits, hard denials, circuit breaking, and one-shot human approvals keyed to the exact session, tool, intent, and arguments. The README also distinguishes local, cloud, and hybrid paths, masks secrets in configuration output, and documents uninstall and recovery commands. Deductions apply because these controls are supported mainly by documentation and one policy-engine test area, not by a stack-wide permission inventory, per-service destination map, complete secret lifecycle, or universal confirmation policy. The prominent installer executes code from main, forced uninstall can remove labelled Docker resources and the runtime directory, and broad recovery is not demonstrated. Components are named, but no complete third-party attribution or licensing inventory is supplied.
APE tests thoroughly cover legacy response contracts, persisted state, corrupt-state recovery, concurrency, strict mode, circuit breaking, and approval retries, while workflows emit a clear error when required secrets are missing. Deductions apply because the README recommends v2.6.0, tags, or audited commits for stable use while its most prominent Linux/macOS and Windows quickstarts still consume main. Claims such as sub-two-minute chat, broad platform support, and release-grade validation depend on referenced files that were not supplied. Docker prerequisites are stated, but concrete versions and availability behavior for the many services, models, and Python tools are not shown; scanner dependencies are installed without versions, and Bandit findings are prevented from failing the job.
The README clearly addresses home, lab, workstation, newcomer, maintainer, and downstream-fork scenarios. It describes local, cloud, hybrid, CPU fallback, port overrides, model profiles, hardware tiers, and manifest-based extensions, with especially detailed Linux, Windows, Apple Silicon, NVIDIA, AMD, and Intel Arc guidance. Deductions apply because some capability boundaries remain broad marketing assertions and important backend limitations are deferred to an absent support matrix. Model selection varies with architecture, memory, downloads, and overrides, while trigger precision is evidenced only locally through APE intent mapping and tightly keyed approvals rather than across Hermes, n8n, and every extension.
The root README has strong organization: repository layout, quickstarts, component catalog, hardware tables, CLI examples, extension structure, and a documentation index. Prerequisites, ports, platform differences, uninstall, and recovery commands are concrete. The complete Apache-2.0 text matches the metadata, and v2.6.0, main, release/2.6.x, release notes, and update channels are explicitly distinguished. Deductions apply because the OpenClaw-to-Hermes migration introduces some naming and product-surface instability, and many limitations are delegated to documents outside the supplied evidence. Maintainer runbooks, private reporting, and issue paths are identified, but no named responsible party, response target, or verifiable release-signing identity is shown; publisher identity remains unknown.
The product offers directly usable operations for installation, opening the UI, checking status, switching models, managing services, viewing masked configuration, extending the stack, and uninstalling it. Pre-wiring inference, chat, voice, workflows, RAG, image generation, and operations would provide substantial value over manual assembly. Deductions apply because the competitor comparison and claims such as two-minute startup and universal out-of-box integration are repository assertions without supplied execution results or independent comparisons. The material mentions no subscription requirement, optional paid APIs, and CI budgets, but does not quantify downloads, disk use, energy, hardware acquisition, maintenance burden, or extension resource costs.
Many claims point to specifically named support, validation, installer-trust, forkability, and release documents, while detailed APE tests and the security policy corroborate several public API and governance claims. The README also distinguishes stable from development channels, acknowledges variable model selection, and states that throughput requires a local benchmark. Deductions apply because most referenced documents are absent from the evidence bundle and therefore cannot be checked here. Workflows and tests establish intended behavior and checking mechanisms, not successful execution. Broad platform, performance, privacy, and competitive claims remain largely self-reported by one repository, with only partial separation between established facts, goals, and promotional inference.
- The most prominent quickstart tracks main. For production-like or high-trust deployment, prefer the documented pinned tag or audited commit and inspect the installer before execution.
- ODS can manage Docker, network exposure, models, secrets, and autonomous tool calls. Review ports, credentials, volume mounts, and tool permissions separately before enabling LAN/public exposure, cloud APIs, n8n integrations, or new extensions.
- Do not assume that APE test coverage means every agent and workflow path is obligatorily mediated by the same governance layer; verify enforcement for Hermes, workflows, and each extension.
- Forced uninstall removes Docker resources labelled for the ODS Compose project and deletes the runtime directory. Back up persistent data, configuration, workflows, and secrets before using it.
- This was a static review of the supplied text only. Installation, tests, scanners, inference, release validation, and referenced but omitted documents were not executed or independently checked.
What does this agent do, and when should you use it?
ODS, the Osmantic Deployment System, turns a Linux, Windows, or Apple Silicon machine into a self-hosted AI server with local execution as the default. It installs and connects components including llama-server, Open WebUI, LiteLLM, Hermes Agent, n8n, Whisper, Kokoro, Qdrant, SearXNG, and ComfyUI. Users interact through the browser-based chat UI and control dashboard, while the ods CLI manages models, modes, extensions, logs, and service health. Its installer detects the hardware and memory envelope, selects a GGUF model from a versioned catalog, and can start a small bootstrap model before downloading and hot-swapping the full model. The normal deployment boundary is the user's machine and Docker environment, with optional cloud or hybrid operation through OpenAI, Anthropic, and Together APIs.
The installer detects NVIDIA, AMD Strix Halo, Apple Silicon, Intel Arc, or CPU environments and selects an appropriate GGUF model by reading ods/config/model-library.json through select-model.py or the Windows PowerShell tier map. It generates credentials and .env settings, then starts llama-server, Open WebUI, the Dashboard, the LiteLLM gateway, and enabled extensions; bootstrap mode first downloads a 1.5B model and later hot-swaps to the full model downloaded in the background. Hermes Agent supplies a local-first agent with memory, skills, and browser operation, n8n runs integration workflows, and APE audits and governs autonomous tool calls. Whisper performs speech-to-text, Kokoro performs text-to-speech, Qdrant and TEI Embeddings support retrieval, SearXNG and Perplexica provide search and research services, and ComfyUI runs image-generation workflows. The Dashboard and ods CLI expose service health, GPU information, logs, model controls, extension management, and switches among local, cloud, and hybrid inference modes.
- An individual or small team handling sensitive prompts and documents can host chat, retrieval, and agent services without making a cloud provider the default execution boundary.
- A workstation owner with NVIDIA, AMD Strix Halo, Intel Arc, or Apple Silicon hardware can avoid manually wiring an inference server, chat UI, vector database, and Docker services.
- A user building voice-driven automation can connect Hermes Agent, Whisper, Kokoro, and n8n for listening, speaking, tool use, and integration workflows.
- A homelab or research operator can build private document retrieval and search using Qdrant, TEI Embeddings, SearXNG, and Perplexica.
- A creator or developer can run local models and ComfyUI while managing service health, GPU metrics, models, and token usage from one stack.
- Someone without sufficient local compute can retain the broader ODS stack while using OpenAI, Anthropic, or Together APIs in cloud mode.
What are this agent's strengths and limitations?
- It covers substantially more than model serving: chat, agents, n8n workflows, voice, retrieval, search, image generation, privacy tooling, and observability are assembled into one stack.
- The installers explicitly handle several GPU and unified-memory families and choose a GGUF model, context size, and hardware tier from a shared catalog.
- Bootstrap mode offers an early working model while the larger selection downloads, followed by a documented hot-swap rather than requiring the user to wait for the full download.
- Local, cloud, and hybrid modes are documented, with OpenAI, Anthropic, and Together API paths rather than dependence on one model provider.
- Services use discoverable manifest.yaml and compose.yaml extensions that integrate with the Dashboard, CLI, health checks, and Compose stack.
- Docker is a core operational dependency; Windows specifically requires Docker Desktop with WSL2, while documented macOS support is limited to Apple Silicon with Docker Desktop.
- The large collection of services increases storage, memory, GPU, port, image-download, and maintenance demands, and the source says throughput still requires a local benchmark.
- Runtime behavior differs by platform: macOS runs llama-server natively while other services remain in Docker, and AMD Strix Halo uses a platform-specific accelerated path.
- The main branch moves quickly, so appliance or production-like adopters must pin a stable tag or audited commit and retain their own validation record.
- Some facilities depend on optional extensions or external services, including Langfuse, Brave Search, and cloud APIs, which may add credentials, configuration, or paid usage.
How do you install or deploy this agent?
Docker must be installed and running. On Linux or macOS, run:
curl -fsSL https://install.osmantic.com/ods.sh | bashA clone-based installation is also documented:
git clone https://github.com/Osmantic/ODS.git
cd ODS/ods
./install.shmacOS additionally requires Apple Silicon (M1 or newer) and Docker Desktop. Windows requires Docker Desktop with the WSL2 backend; from a normal, non-Administrator PowerShell session, run:
$ProgressPreference = "SilentlyContinue"
$odsSrc = Join-Path $env:TEMP ("ods-install-" + [guid]::NewGuid().ToString("N"))
$odsZip = Join-Path $odsSrc "ods-main.zip"
New-Item -ItemType Directory -Path $odsSrc | Out-Null
Invoke-WebRequest "https://github.com/Osmantic/ODS/archive/refs/heads/main.zip" -OutFile $odsZip
Expand-Archive -LiteralPath $odsZip -DestinationPath $odsSrc -Force
cd (Get-ChildItem -LiteralPath $odsSrc -Directory | Select-Object -First 1).FullName
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1Local mode needs no cloud credentials. Cloud or hybrid operation requires credentials for the selected OpenAI, Anthropic, or Together API. The source identifies v2.6.0 as the stable release and recommends pinning a tagged release or audited commit for appliance, lab, or production-like use instead of tracking main.
How do you use this agent?
After installation, open http://localhost:3000 and start a conversation in Open WebUI. Linux Docker installations expose llama-server at http://localhost:11434 by default; native macOS Metal and native Windows or Lemonade paths default to http://localhost:8080. Representative management commands are:
ods status
ods list
ods logs llm
ods restart llm
ods model current
ods model list
ods model swap T3
ods enable n8n
ods disable whisper
ods config showUse ods mode local, ods mode cloud, or ods mode hybrid to change the inference source. To use an existing single-file GGUF model, place it in data/models/ and load it from Dashboard → Models; older or headless installations can instead update GGUF_FILE and LLM_MODEL in .env and run ods restart llm. For a cloud-first installation, run ./install.sh --cloud. Ports can be overridden at installation time, for example WEBUI_PORT=9090 ./install.sh.
How does this agent compare with similar options?
Compared with Ollama or llama.cpp alone, ODS adds the surrounding chat, dashboard, voice, retrieval, workflow, agent, privacy, and service-management layers. Compared with deploying Open WebUI by itself, it supplies an installer, control plane, and pre-wired local services. The source positions AnythingLLM as more RAG-centered, while ODS also encompasses inference, voice, workflows, image generation, and operations. It treats an n8n self-hosted AI starter kit as workflow automation within a narrower scope, whereas n8n is one component of ODS. Its comparison table also characterizes LocalAI as primarily model-serving software, while ODS includes hardware-aware model selection, autonomous agents, voice, RAG, and an extension system.