VideoAgent
Natural-language video understanding, editing, and creative remaking in one framework.
- Source repo
- HKUDS/VideoAgent
- Stars
- ★ 2.1k
- Last updated
- 4mo ago
- License
- MIT
- Primary language
- Python
- FA score
- 39/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changesOpenAI API · Claude API (Partial support)
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A researcher analyzing a podcast or video corpus who needs question answering and summaries can use the Video QA/Summarization path after preparing Whisper.
- Not a fit if
- Users without an 8GB GPU or on unsupported operating systems
- Teams wanting zero-config setup; multiple models and API keys must be configured
- Website
- arxiv.org/abs/2606.23327 ↗ set on GitHub
- Maintenance
- Commits in the last 6 monthslast commit 2026-07-22+2.1 ★/day over 7 days
- Source review
- 39/100 · Major gaps 1 safety controls not found
What does this agent do, and when should you use it?
VideoAgent is an MIT-licensed framework for video understanding, editing, and remaking. It processes requests through intent analysis, intent-to-agent mapping, graph-powered workflow construction, and a documented two-step self-evaluation feedback loop. Its Storyboard Agent analyzes pre-captioned video banks and turns requests into fine-grained, visually and semantically aligned queries for retrieval. The documented feature set includes video Q&A, summarization, movie edits, commentary videos, video overviews, meme remaking, music videos, and cross-lingual adaptations. It runs locally through a Python command-line entry point, main.py, and relies on downloaded speech, voice-conversion, transcription, and multimodal models.
After python main.py starts, the user enters a request at User Requirement:. The system decomposes it into explicit and implicit sub-intents, uses an intent-to-agent mapping to select capabilities, and creates an executable graph-based workflow with an execution sequence; self-evaluation feedback iteratively refines planning. The Storyboard Agent analyzes a pre-captioned video-material bank and produces fine-grained visual queries for retrieval. Depending on the task, the framework can perform video Q&A or summarization and produce movie edits, commentary videos, video overviews, meme videos, music videos, and cross-lingual comedy adaptations. Provider settings are stored in environment/config/config.yml, including API keys and base URLs for DeepSeek, Claude, GPT, and Gemini.
- A researcher analyzing a podcast or video corpus who needs question answering and summaries can use the Video QA/Summarization path after preparing Whisper.
- A creator who wants to rewrite speech in an existing video while retaining the original visuals and speaker voice can use the documented video remixing, TTS, or SVC-related workflows.
- A producer with a stand-up script, reference script, and footage who needs a completed comedy video with timing, audience reactions, and matched footage can use the documented stand-up scenario.
- A bilingual comedy creator adapting English stand-up into Chinese crosstalk, or Chinese crosstalk into English stand-up, can use the Cross Talk or Talk Show scenarios.
- A music-video maker using DiffSinger, seed-vc, Whisper, and ImageBind for an AI music video can use the MAD SVC scenario.
- A content team that needs to retrieve clips matching textual descriptions from a video bank before producing a commentary video or overview can use the Storyboard Agent retrieval flow.
Key setup facts
Pulled from the install and usage notes below. The project README has the current values.
- Install / run
pip install -r requirements.txt
- Config files
environment/config/config.yml- Install notes cover
- Windows · Linux
How do you install or deploy this agent?
The documented environment is Linux or Windows with 8GB GPU memory. Run:
git clone https://github.com/HKUDS/VideoAgent.git
conda create --name videoagent python=3.10
conda activate videoagent
conda install -y -c conda-forge pynini==2.1.5 ffmpeg
pip install -r requirements.txtDownload the models needed for the intended feature: the README supplies huggingface-cli download or wget commands for CosyVoice, fish-speech, seed-vc, DiffSinger, Whisper, and ImageBind; ImageBind setup also calls for git lfs install. Edit environment/config/config.yml with the relevant deepseek_api_key, claude_api_key, gpt_api_key, and gemini_api_key, plus their base URLs. The README explicitly states that Claude is required for the Agentic Graph Router.
How do you use this agent?
After downloading models and configuring environment/config/config.yml, run:
python main.pyEnter a video task when the console displays User Requirement:. Documented examples include rewriting dialogue in an existing video while retaining the original speaker voice and visuals, or turning a stand-up script, reference script, and footage into a complete video. Model requirements vary by scenario: Video QA/Summarization lists Whisper, while Cross Talk, Talk Show, Comm, and News list CosyVoice, Whisper, and ImageBind.
What are this agent's strengths and limitations?
- One framework documents video Q&A, summarization, editing, commentary, overviews, meme remaking, music videos, and cross-lingual adaptation instead of a single video task.
- Its graph-powered workflow construction combines user-intent mapping with a documented two-step self-evaluation loop for multi-step video requests.
- The Storyboard Agent explicitly covers the retrieval stage by converting a pre-captioned material bank into visually and semantically aligned fine-grained queries.
- Feature-specific model requirements let adopters download only the models relevant to their intended workflows.
- The README explicitly requires Claude for the Agentic Graph Router, so operation is not fully independent of external model providers.
- Different features also call for DeepSeek, GPT, or Gemini configuration and for models such as CosyVoice, fish-speech, seed-vc, DiffSinger, Whisper, and ImageBind, increasing setup complexity.
- The documented runtime requires Linux or Windows, 8GB GPU memory, Python 3.10, FFmpeg, and Pynini 2.1.5.
- The supplied material documents only the
python main.pycommand-line entry point; it provides no evidence of an HTTP API, container deployment, or hosted service. - Demo audio and video assets are described as internet-sourced and for research and demonstration purposes, so commercial adopters need to review content rights independently.
How does this agent compare with similar options?
The README compares VideoAgent with Director, Funclip, NarratoAI, and NotebookLM. In that table, VideoAgent is marked for beat-synced edits, storytelling video, video overview, meme remaking, song remixes, cross-lingual adaptations, video Q&A, and sound-effects tools, while the named alternatives cover only subsets of those rows.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| VideoAgent This agent | 39 · Major gaps | CLIFree + model costs | ★ 2.1k | 4mo ago | Python | — |
| Natively AI Assistant | 50 · Major gaps | Desktop appFreemium | ★ 3.4k | 1d ago | TypeScript | OpenAI API · Claude API |
| Kineo (Kineo AI) | 34 · Major gaps | — | ★ 137 | 5d ago | TypeScript | — |
| PPT Master — AI-native PowerPoint generation | 70 · Some gaps | Agent plugin / skillFree + model costs | ★ 79k | 1d ago | Python | Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The README shows API key configuration and several model download sources, and says demo media comes from the internet, giving partial visibility into data and external dependencies. Deductions: it does not say which services receive user videos, audio, or prompts, and provides little on sensitive data handling, permission boundaries, per-action external effects, confirmation, or rollback. Its rights statement for demo assets is general.
The README gives Python 3.10 setup steps, a model checklist, and a launch command; pyproject.toml lists a project version and dependencies. Deductions: the dependency set is large, contains duplicates and unpinned items, and platform-specific availability is not explained despite claiming Linux and Windows support. The supplied code offers limited evidence about failure handling.
The README describes natural-language video QA, editing, summarization, and generation, with a feature-to-model table and example requests. Deductions: capability descriptions are broad and do not define boundaries, eligibility conditions, or when a capability should not be selected; environment requirements are limited to brief GPU and OS notes.
The README has a contents section, quick start, model downloads, configuration, usage, demos, and acknowledgments, including installation steps and examples. Deductions: there is no FAQ or concrete limitations section; the repository LICENSE says MIT while pyproject.toml says Apache; only version 0.1.0 is shown, with no changelog or clear maintenance owner.
The product targets video understanding, editing, and generation, and the README provides demos, example requests, and guidance for choosing models by feature. Deductions: output formats, quality constraints, and human review steps are not specified in the supplied material; downloading several large models and configuring multiple LLM services carries substantial resource and setup costs.
The README links demos, describes evaluations with referenced figures, and acknowledges models and projects, making some claims traceable to its own materials. Deductions: key performance claims are mostly self-reported, and the supplied files provide no test evidence or independent corroboration; claims and inference are not clearly separated.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Before installation, reconcile the license declarations in LICENSE and pyproject.toml, and check dependency and model-download compatibility with the target environment.
- Before adding API keys or processing personal video or audio, determine which services receive the data and how they handle it; the README does not specify these flows.
- The README says demo assets come from the internet and are for research and demonstration, but does not itemize their licenses.
FAQ
Which model-provider credentials are required?
Can I download only a subset of local models?
How do I start it and submit a task?
python main.py and enter the request at the User Requirement: console prompt.Is there a web service or SDK?
Is VideoAgent free?
Is VideoAgent open source?
Is VideoAgent safe to use?
Is VideoAgent still maintained?
Who makes VideoAgent?
What is VideoAgent's official website?
Thanks — we use this to improve the page.