Stream Vision Agents
A Python SDK for low-latency, real-time voice and video agents.
- Source repo
- GetStream/Vision-Agents
- Stars
- ★ 8.2k
- Last updated
- today
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 43/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API · Claude API
- Cost
- Free software; you pay for model usage
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- A sports or fitness product team building live pose-aware coaching with YOLO Pose and Gemini Live.
- Not a fit if
- Developers building text-only chat apps with no realtime audio or video
- Users without GPU access or willingness to pay multiple model API providers
- Source review
- 43/100 · Major gaps 2 safety controls not found
What does this agent do, and when should you use it?
Vision Agents is Stream’s open-source Python project for multimodal agents that watch, listen to, and understand live video. Its central Agent combines an edge connection, a realtime model, and pluggable video processors to run audio-video interactions. The README demonstrates getstream.Edge(), gemini.Realtime(), and ultralytics.YOLOPoseProcessor in one agent definition. Documented capabilities span WebRTC streaming, STT, TTS, turn detection, tool calling and MCP, phone calling, RAG, and memory across turns and sessions. The project also describes HTTP serving, Prometheus metrics, horizontal scaling, and Kubernetes deployment, while its getting-started flow requires Stream API credentials.
A developer installs the SDK with uv add vision-agents and defines an Agent. In the documented example, the agent receives edge=getstream.Edge(), agent_user, instructions, llm=gemini.Realtime(fps=10), and processors=[ultralytics.YOLOPoseProcessor(model_path="yolo11n-pose.pt", device="cuda")]. It routes realtime audio and video to model providers and can run YOLO, Roboflow, or custom PyTorch/ONNX processing pipelines before or after LLM calls, producing interactive responses. The project also states that agents can call code, external APIs, or MCP servers during a conversation, and can support bidirectional phone audio through Twilio or Telnyx.
- A sports or fitness product team building live pose-aware coaching with YOLO Pose and Gemini Live.
- A support team building a low-latency voice assistant that can search knowledge and files while speaking.
- A security or moderation team applying Roboflow, YOLO, or custom vision models to live video streams.
- A telephony automation team handling inbound and outbound calls through Twilio or Telnyx with bidirectional audio.
- A virtual try-on or stylized-video team using video-restyling models such as Decart Lucy.
How do you install or deploy this agent?
In a Python project, run uv add vision-agents. For the optional integrations named in the README, run uv add "vision-agents[getstream, openai, elevenlabs, deepgram]". Then obtain Stream API credentials, which the documented setup explicitly requires.
How do you use this agent?
Install the package and configure Stream API credentials. The supplied example creates Agent(edge=getstream.Edge(), agent_user=agent_user, instructions="Read @golf_coach.md", llm=gemini.Realtime(fps=10), processors=[ultralytics.YOLOPoseProcessor(model_path="yolo11n-pose.pt", device="cuda")]). The snippet is from examples/02_golf_coach_example/golf_coach_example.py; the supplied material does not include imports, agent_user setup, or a start invocation, so a complete first runnable invocation requires that example file or the quickstart content.
What are this agent's strengths and limitations?
- It brings realtime WebRTC, video models, speech capabilities, and LLM calls together through an
Agentand processor pipeline. - It names multiple LLM, realtime, STT, TTS, and vision providers while exposing native OpenAI, Gemini, and Claude SDK methods.
- YOLO, Roboflow, PyTorch, and ONNX processors can be placed around LLM calls, supporting workloads that need specialized vision models.
- The README documents paths for phone calling, RAG, MCP, HTTP serving, Prometheus, horizontal scaling, and Kubernetes deployment.
- The documented onboarding flow requires Stream API credentials, despite the claim that the project can work with any video edge network.
- The README reports context degradation in continuous video-understanding sessions of roughly 30 seconds or longer.
- Small text, scores, and signs in video may cause model hallucinations.
- Most use cases need specialized models such as YOLO or Roboflow alongside larger LLMs, and realtime models need audio or text to trigger a response; video alone does not do so.
How does this agent compare with similar options?
The repository does not compare itself with a single direct competitor, but its integration list provides model-path choices including OpenAI, Gemini, xAI, OpenRouter, Hugging Face, Kimi AI, and MiniMax. Its named realtime paths include OpenAI Realtime, Gemini Live, AWS Nova Sonic, Qwen, and Inworld.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Stream Vision Agents This agent | 43 · Major gaps | Library / SDKFree + model costs | ★ 8.2k | today | Python | OpenAI API · Claude API |
| Voice AI Learning Path | 63 · Some gaps | Library / SDKFree | ★ 338 | 8d ago | Astro | — |
| Patter Voice SDK | 52 · Major gaps | Library / SDKFree + model costs | ★ 1.1k | 2mo ago | Python | OpenAI API · Claude API |
| Android Voice AI SDK (MVVM) | 47 · Major gaps | Library / SDKFree + model costs | ★ 2.6k | 5mo ago | Kotlin | Claude API |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence: README mentions tool calling and MCP, which can execute code and APIs, but does not mention least privilege or user confirmation mechanisms; data flow transparency is partially addressed by describing video streaming and processor pipelines, but data flow and storage are not explicit; sensitive data handling is not mentioned; dependency security is partially addressed by listing dependencies in pyproject.toml, but no security audit or vulnerability scanning evidence is provided; external effects are possible via tool calling and phone integration, but no restrictions or confirmations are described; rollback is not mentioned; source attribution is partially addressed by providing contact emails in README and SECURITY.md, but publisher identity is not verified. Deductions: lack of specific evidence for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, and source attribution.
Evidence: README and test files (e.g., test_anam_avatar.py) show internal consistency, but failure messages are not detailed; dependency availability is partially addressed by listing many dependencies in pyproject.toml, but no version pinning or availability guarantees are provided; failure messages are partially covered by error message assertions in tests, but not all scenarios are covered. Deductions: insufficient evidence for dependency availability and failure messages.
Evidence: README describes multiple scenarios (voice, video, phone, RAG, etc.) with clear audience; capability boundaries are partially addressed by listing integrations and features, but limitations are not explicit; trigger precision is partially addressed by mentioning real-time triggers, but not detailed; environment fit is partially addressed by mentioning Kubernetes deployment and horizontal scaling, but no specific configurations are provided. Deductions: insufficient evidence for trigger precision.
Evidence: README provides clear information architecture including quickstart, features, integrations, documentation, and examples; install notes are clear (uv add); naming stability is partially addressed by consistent naming in README and pyproject.toml, but no version history is provided; examples and FAQ are partially addressed by providing multiple examples, but no FAQ is present; known limitations are explicitly listed; license is Apache-2.0 and complete; versioning changelog is not provided; maintenance responsibility is partially addressed by providing contact email, but maintainers are not explicit. Deductions: insufficient evidence for naming stability, versioning changelog, and maintenance responsibility.
Evidence: README demonstrates output usability, such as real-time video and voice agents; marginal value is provided by multiple integrations and features; cost-benefit is partially addressed by mentioning free API keys and Maker program, but no detailed cost analysis is provided. Deductions: insufficient evidence for cost-benefit.
Evidence: Claims in README (e.g., low latency) are not backed by verifiable test results; cross-source corroboration is partially addressed by referencing external docs and examples, but no independent verification is provided; fact-inference separation is weak as README mixes facts and marketing claims. Deductions: insufficient evidence for claim traceability, cross-source corroboration, and fact-inference separation.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Publisher identity is unverified; treat source with caution.
- Many dependencies without security audit; check dependency security yourself.
- Tool calling and phone integration may have external side effects; ensure user confirmation mechanisms.
- Data flow and sensitive data handling are not transparent; review data flow and storage.
Recent releases
Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.
-
Release notes excerpt
What's Changed * [Speechify] Add support for Speechify as a TTS provider by @DaemonLoki in https://github.com/GetStream/Vision-Agents/pull/615 * fix(speechify): send Speechify-Caller attribution header by @luke-speechify in https://github.com/GetStream/Vision-Agents/pull/617 * [xAI] Add support for grok-voice-think-fast-2.0 by @DaemonLoki in https://github.com/GetStream/Vision-Agents/pull/624 * fix(nvidia): default VLM to llama-3.2-11b-vision-instruct by @Nash0x7E2 in https://github.com/GetStream/Vision-Agents/pull/625 …View on GitHub ↗ -
Release notes excerpt
What's Changed * Support the new stateful AudioStreamTrack from getstream by @dangusev in https://github.com/GetStream/Vision-Agents/pull/612 Full Changelog: https://github.com/GetStream/Vision-Agents/compare/v0.6.7...v0.6.8View on GitHub ↗ -
Release notes excerpt
What's Changed * CI: add least-privilege permissions to GitHub Actions workflows by @peter-matkovski in https://github.com/GetStream/Vision-Agents/pull/609 * fix(twelvelabs): wait for asset ready and stamp clip duration for Pegasus by @Nash0x7E2 in https://github.com/GetStream/Vision-Agents/pull/610 * chore: clean up open-model TTS plugins (Kokoro warmup, Pocket deps, drop moonshine extra) by @Nash0x7E2 in https://github.com/GetStream/Vision-Agents/pull/613 * chore(anam): bump Anam SDK to 0.6.0 by @Nash0x7E2 in https://github.com/GetStream/Vision-Agents/pull/614 …View on GitHub ↗
FAQ
Is Stream required?
getstream.Edge(). The README also says the project works with any video edge network, but the supplied material does not provide steps for replacing the edge network.