CogAgent GUI Operator
A screenshot-driven VLM that produces grounded GUI actions from natural-language tasks.
- Source repo
- zai-org/CogAgent
- Stars
- ★ 1.2k
- Last updated
- 26d ago
- License
- Apache-2.0
- Primary language
- Python
- FA score
- 42/100 · Major gaps
At a glance
- How it runs
- Works with
- Portable with changes
- Cost
- Free, no paid service needed
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A developer prototyping local, screenshot-driven GUI automation on macOS 14 or 15.
- Not a fit if
- Users who need text-only chat without screenshots
- Individual developers without A100/H100-class GPUs
- Teams wanting a turnkey setup with no local inference build
- Source review
- 42/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
CogAgent is an open-source vision-language-model project for GUI operation, centered in this repository on CogAgent-9B-20241220. Built on GLM-4V-9B, the model takes screenshots, a task, execution history, a platform marker, and a requested response format. It is an execution-oriented model rather than a continuous conversational model, so callers provide prior history when starting a new session. Its strict string output can include Action, Status, Plan, Grounded Operation, and sensitivity markers; Grounded Operation supplies an operation type and screen coordinates. The repository offers transformers-based CLI inference, a web demo, and a locally deployed Agent APP example, while the hosted demo displays inference results only and cannot control a computer.
A caller concatenates a task, History steps, a platform field (WIN, Mac, or Mobile), and a format field, then submits that prompt with a screenshot to CogAgent-9B-20241220. The model reads the visible interface and prior steps, returning Grounded Operation strings such as CLICK, TYPE, SCROLL_DOWN, or RIGHT_CLICK, with a box identifying the target coordinate region. inference/cli_demo.py interactively accepts image paths and saves annotated output images as {your_input_image_name}_{round}.png. inference/web_demo.py supports repeated image uploads for interactive inference, and the example documented in app/README.md is intended for local GPU-server deployment and automated GUI operations.
- A developer prototyping local, screenshot-driven GUI automation on macOS 14 or 15.
- A researcher who needs click, typing, and scrolling coordinates from Windows 10 or 11 screenshots.
- A mobile-testing team collecting parseable operation instructions for Android 13, 14, or 15 interfaces.
- An engineer validating screen grounding and output parsing with a single-step task such as marking all emails as read.
- An evaluation team with A100- or H100-class GPUs assessing a GUI-agent model locally.
How do you install or deploy this agent?
Install Python 3.10.16 or later, then run:
pip install -r requirements.txtThe documented local commands do not require an API key. Obtain a model directory; the examples use THUDM/cogagent-9b-20241220. BF16 inference requires at least 29GB of VRAM.
How do you use this agent?
Prepare screenshots from the platform selected in the command, then run:
python inference/cli_demo.py --model_dir THUDM/cogagent-9b-20241220 --platform "Mac" --max_length 4096 --top_k 1 --output_image_path ./results --format_key status_action_op_sensitiveDuring interaction, construct the prompt in the documented order: Task, History steps, Platform, then format. To run the web demo:
python inference/web_demo.py --host 0.0.0.0 --port 7860 --model_dir THUDM/cogagent-9b-20241220 --format_key status_action_op_sensitive --platform "Mac" --output_dir ./resultsWhat are this agent's strengths and limitations?
- Consumes screenshots and natural-language tasks directly, then emits coordinate-bearing Grounded Operation output that a downstream executor can parse.
- Explicitly supports Windows, macOS, and Android platform fields, with Chinese and English interaction.
- Includes CLI inference, a web demo, and a local Agent APP example for evaluation and deployment entry points.
- Offers several strict output formats so integrations can request status, plan, actions, and sensitivity labels as needed.
- The model requires images; text-only conversation cannot complete GUI-agent tasks.
- It is not a continuous conversational model, so the caller must provide prior execution history for every new session.
- BF16 inference needs at least 29GB of VRAM, while SFT and LoRA fine-tuning carry substantial A100 memory and multi-GPU requirements.
- The project does not guarantee safe AI behavior, and its Agent APP example is for academic reference; adopters need their own review and execution controls.
How does this agent compare with similar options?
The project’s technical blog compares the model with GPT-4o-20240806, Claude-3.5-Sonnet, GPT-4o + UGround, GPT-4o + OS-ATLAS, and the open models Qwen2-VL, ShowUI, and SeeClick. The repository reports strong results on Screenspot, OmniAct, CogAgentBench-basic-cn, and OSWorld; these are project-reported results and should be retested in the intended environment.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| CogAgent GUI Operator This agent | 42 · Major gaps | CLIFree | ★ 1.2k | 26d ago | Python | — |
| Clawd Cursor | 74 · Some gaps | MCP serverFree + model costs | ★ 406 | today | TypeScript | Codex · Claude Code |
| macOS Harness | 55 · Major gaps | CLIFree + model costs | ★ 1k | 23d ago | Python | Codex · Claude Code |
| Open Computer Use (MCP desktop service) | 44 · Major gaps | CLIFree + model costs | ★ 2.9k | 4d ago | Swift | Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: The repository provides no explicit mechanisms for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. README only mentions sensitive operation markers in output, but no permission control or user confirmation during execution. Therefore all trust criteria score 0.
Evidence shows: README consistently describes input/output formats, platform support, and inference costs, so self-consistency scores 2. Dependency list is explicit but lacks version pinning or integrity checks, so dependency availability scores 1. For failure messages, README provides no error handling or troubleshooting guidance, only a caution to use carefully, so scores 1.
Evidence shows: README clearly identifies target audience (researchers and developers), use cases (GUI operations), capability boundaries (no dialogue, requires image input), and environment requirements (Python 3.10.16+, GPU memory, etc.), so each criterion scores 2.
Evidence shows: README is well-structured with installation instructions, examples, known limitations (e.g., no dialogue, requires image input), license (Apache-2.0), and version info (20241220). However, no explicit changelog is provided, and versioning info is limited, so versioning/changelog scores 1, others score 2.
Evidence shows: README provides detailed input/output formats and examples, so output usability scores 2. The model claims significant improvements in GUI tasks, so marginal value scores 2. Cost-benefit is addressed with inference and fine-tuning costs, but no comparison with alternatives, so scores 2.
Evidence shows: README cites papers, technical blogs, and benchmarks, but lacks specific data or reproduction steps, so claim traceability scores 2. Cross-source corroboration relies solely on self-claims without independent verification, so scores 1. Fact-inference separation is clear as README distinguishes model capabilities from speculation, so scores 2.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository provides no permission control or user confirmation mechanisms; executing GUI operations may have irreversible effects, so use with caution.
- Dependencies are not version-pinned, posing supply chain risks; consider using a virtual environment and pinning versions.
- The model output format is strict; parsing errors may cause operation failures, so follow the documentation precisely.
Maintain this project? Add the FARS badge to your README
[](https://followagents.com/en/agents/cogagent)<a href="https://followagents.com/en/agents/cogagent"><img src="https://followagents.com/badge/cogagent.svg" alt="FARS review"></a>