LiveKit Agents: Framework for Realtime Voice AI
Build realtime, programmable voice agents that can see, hear, and understand.
- Source repo
- livekit/agents
- Stars
- ★ 15k
- Last updated
- today
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 43/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Developers quickly building WebRTC voice assistants for customer service or virtual assistants.
- Not a fit if
- Developers unwilling to run or connect to a LiveKit server
- Teams that need a stable, non-evolving API
- Source review
- 43/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
LiveKit Agents is a Python framework for building server-side realtime voice agents. It provides AgentSession for managing conversations, Agent for defining instructions and tools, and AgentServer for coordinating job scheduling. It supports flexible STT, LLM, and TTS integrations via plugins or LiveKit Inference, and includes features like RPC, data APIs, MCP support, and a built-in test framework.
The framework runs an AgentServer that listens for room sessions and initiates AgentSession. Developers define an entrypoint with @server.rtc_session, configure VAD, STT, LLM, TTS (e.g., deepgram/nova-3, google/gemma-4-31b-it, cartesia/sonic-3), and define agent behavior via instructions and tools (@function_tool). Sessions support reply generation, multi-agent handoff, structured output, and can be run via CLI (console, dev, start).
- Developers quickly building WebRTC voice assistants for customer service or virtual assistants.
- Interactive voice response systems, e.g., phone bots via SIP integration.
- Multi-agent workflows where one agent collects info then hands off to another.
- Scenarios requiring structured output from the LLM to guide TTS tone.
- Integrating MCP tools to interact with external systems.
- Automated testing of voice agents using the built-in test framework and judges.
How do you install or deploy this agent?
Install the core library and plugins via pip: pip install "livekit-agents[openai,deepgram,cartesia]". Requires Python.
How do you use this agent?
Set environment variables LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET. Run python myagent.py dev for a development server with hot reload. For production, use python myagent.py start. To test in terminal, use python myagent.py console.
What are this agent's strengths and limitations?
- Constantly updated with integrations for OpenAI, Deepgram, Cartesia, etc.
- Built-in dispatch APIs for job scheduling and distribution.
- Extensive WebRTC client SDK support across platforms.
- Native MCP support for easy tool integration.
- Built-in test framework to verify agent behavior.
- Requires a LiveKit server (cloud or self-hosted), adding deployment complexity.
- Needs API keys and network connectivity.
- Dependent on model providers, which may incur costs.
- Rapidly evolving API may change frequently.
How does this agent compare with similar options?
Unlike chatbot-only libraries, this framework focuses on realtime voice interactions and integrates tightly with the LiveKit ecosystem, requiring a LiveKit server.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| LiveKit Agents: Framework for Realtime Voice AI This agent | 43 · Major gaps | CLIFree + model costs | ★ 15k | today | Python | OpenAI API · Claude API |
| OpenAI Agents SDK (JavaScript/TypeScript) | 55 · Major gaps | Library / SDKFree + model costs | ★ 3.9k | 3d ago | TypeScript | OpenAI API |
| TanStack AI | 76 · Some gaps | Library / SDKFree + model costs | ★ 3.2k | 4d ago | TypeScript | OpenAI API · Claude API |
| AgentScope 2.0 | 70 · Some gaps | Library / SDKFree + model costs | ★ 33k | today | Python | OpenAI API · Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows the framework requires environment variables (LIVEKIT_URL etc.) and API keys, but no explicit least-privilege guidance; user confirmation mechanism not mentioned; data flow transparency partially shown in docs mentioning RPC and data APIs, but not detailed; sensitive data handling not explicit; dependency security has lock file (uv.lock) and CI builds, but no vulnerability scanning mentioned; external effects: framework connects to external services (STT/LLM/TTS), but side effects not documented; rollback not mentioned; source attribution: code and docs clearly attribute to LiveKit, but publisher unverified. Deductions: lack of security best practices documentation and user confirmation mechanism.
Self-consistency is good, README and examples align, but no version compatibility notes; dependency availability: many dependencies and some require external API keys, but no offline or fallback options; failure messages: error handling not detailed in docs. Deductions: external service dependencies may cause unavailability, and error handling documentation insufficient.
Audience clearly developers, scenarios cover voice, video, text; capability boundaries shown via feature list and examples, but no explicit limitations; trigger precision: tool calls and event handling have examples, but trigger conditions not detailed; environment fit: supports multiple platforms and deployment modes, but resource requirements not specified. Deductions: trigger precision and resource requirements insufficient.
Information architecture clear, with README, docs links, examples directory; install notes detailed with pip command; naming stability: API names consistent in examples, but no naming convention docs; examples and FAQ rich, but FAQ not separate; known limitations not explicit; license Apache-2.0 clear; versioning changelog not provided; maintenance responsibility: contribution guide and community channels, but maintainers not explicit. Deductions: missing changelog and known limitations.
Output usability: multiple examples and test framework provided, but no output format spec; marginal value: feature-rich, but no comparison with alternatives; cost-benefit: no performance or cost data. Deductions: lack of cost-benefit analysis.
Claim traceability: feature claims in README supported by examples, but no detailed doc links; cross-source corroboration: multiple examples and tests, but no independent verification; fact-inference separation: docs do not clearly distinguish facts from inferences. Deductions: lack of independent verification and fact-inference separation.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Publisher identity unverified; be cautious about supply chain risks.
- Framework relies on many external services and API keys; ensure key security.
- No changelog provided; check compatibility manually when upgrading.
- Known limitations not explicit; evaluate before production use.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
GPT-Live through LiveKit Inference You can now run GPT-Live with your LiveKit credentials instead of a separate OpenAI key. inference.DuplexModel handles its protocol, and you can adopt it directly with a model string: ```python session = AgentSession(llm="openai/gpt-live-1") ``` New providers and models - OpenAI GPT-Live 1: now available through LiveKit Inference. - Google Gemini Live: adds Inference model-string support for google/gemini-3.8-live, google/gemini-3.8-live-extended-thinking, and google/gemini-live. …View on GitHub ↗ -
Release notes excerpt
Realtime E2E latency You can now measure realtime replies from the last detected user speech to first assistant playback when you configure local VAD. Read the latency from assistant message metrics: ```python @session.on("conversation_item_added") def on_message(event): if event.item.type == "message" and event.item.role == "assistant": print(event.item.metrics.get("e2e_latency")) ``` New providers and models - Claude Opus 5, Opus 5.5 and Sonnet 5.5: model support through the Anthropic SDK upgrade. Breaking changes …View on GitHub ↗ -
Release notes excerpt
What's Changed * fix(sarvam): close the streams the STT collects by @dhruvladia-sarvam in https://github.com/livekit/agents/pull/7058 * fix(aws): keep realtime session active after stream restart by @uuzzrm in https://github.com/livekit/agents/pull/6811 * fix(telemetry): format a loop stall report without reading source files by @zdurm in https://github.com/livekit/agents/pull/7616 * update deps by @davidzhao in https://github.com/livekit/agents/pull/7655 * fix(deps): bump bithuman in uv.lock for Windows wheel support by @abhijeetgaur-dev in https://github.com/livekit/agents/pull/7559 …View on GitHub ↗