CogAgent GUI Operator
A screenshot-driven VLM that produces grounded GUI actions from natural-language tasks.
Per-dimension scores and reasoning
Evidence shows: The repository provides no explicit mechanisms for least privilege, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. README only mentions sensitive operation markers in output, but no permission control or user confirmation during execution. Therefore all trust criteria score 0.
Evidence shows: README consistently describes input/output formats, platform support, and inference costs, so self-consistency scores 2. Dependency list is explicit but lacks version pinning or integrity checks, so dependency availability scores 1. For failure messages, README provides no error handling or troubleshooting guidance, only a caution to use carefully, so scores 1.
Evidence shows: README clearly identifies target audience (researchers and developers), use cases (GUI operations), capability boundaries (no dialogue, requires image input), and environment requirements (Python 3.10.16+, GPU memory, etc.), so each criterion scores 2.
Evidence shows: README is well-structured with installation instructions, examples, known limitations (e.g., no dialogue, requires image input), license (Apache-2.0), and version info (20241220). However, no explicit changelog is provided, and versioning info is limited, so versioning/changelog scores 1, others score 2.
Evidence shows: README provides detailed input/output formats and examples, so output usability scores 2. The model claims significant improvements in GUI tasks, so marginal value scores 2. Cost-benefit is addressed with inference and fine-tuning costs, but no comparison with alternatives, so scores 2.
Evidence shows: README cites papers, technical blogs, and benchmarks, but lacks specific data or reproduction steps, so claim traceability scores 2. Cross-source corroboration relies solely on self-claims without independent verification, so scores 1. Fact-inference separation is clear as README distinguishes model capabilities from speculation, so scores 2.
- The repository provides no permission control or user confirmation mechanisms; executing GUI operations may have irreversible effects, so use with caution.
- Dependencies are not version-pinned, posing supply chain risks; consider using a virtual environment and pinning versions.
- The model output format is strict; parsing errors may cause operation failures, so follow the documentation precisely.
What does this agent do, and when should you use it?
CogAgent is an open-source vision-language-model project for GUI operation, centered in this repository on CogAgent-9B-20241220. Built on GLM-4V-9B, the model takes screenshots, a task, execution history, a platform marker, and a requested response format. It is an execution-oriented model rather than a continuous conversational model, so callers provide prior history when starting a new session. Its strict string output can include Action, Status, Plan, Grounded Operation, and sensitivity markers; Grounded Operation supplies an operation type and screen coordinates. The repository offers transformers-based CLI inference, a web demo, and a locally deployed Agent APP example, while the hosted demo displays inference results only and cannot control a computer.
A caller concatenates a task, History steps, a platform field (WIN, Mac, or Mobile), and a format field, then submits that prompt with a screenshot to CogAgent-9B-20241220. The model reads the visible interface and prior steps, returning Grounded Operation strings such as CLICK, TYPE, SCROLL_DOWN, or RIGHT_CLICK, with a box identifying the target coordinate region. inference/cli_demo.py interactively accepts image paths and saves annotated output images as {your_input_image_name}_{round}.png. inference/web_demo.py supports repeated image uploads for interactive inference, and the example documented in app/README.md is intended for local GPU-server deployment and automated GUI operations.
- A developer prototyping local, screenshot-driven GUI automation on macOS 14 or 15.
- A researcher who needs click, typing, and scrolling coordinates from Windows 10 or 11 screenshots.
- A mobile-testing team collecting parseable operation instructions for Android 13, 14, or 15 interfaces.
- An engineer validating screen grounding and output parsing with a single-step task such as marking all emails as read.
- An evaluation team with A100- or H100-class GPUs assessing a GUI-agent model locally.
What are this agent's strengths and limitations?
- Consumes screenshots and natural-language tasks directly, then emits coordinate-bearing Grounded Operation output that a downstream executor can parse.
- Explicitly supports Windows, macOS, and Android platform fields, with Chinese and English interaction.
- Includes CLI inference, a web demo, and a local Agent APP example for evaluation and deployment entry points.
- Offers several strict output formats so integrations can request status, plan, actions, and sensitivity labels as needed.
- The model requires images; text-only conversation cannot complete GUI-agent tasks.
- It is not a continuous conversational model, so the caller must provide prior execution history for every new session.
- BF16 inference needs at least 29GB of VRAM, while SFT and LoRA fine-tuning carry substantial A100 memory and multi-GPU requirements.
- The project does not guarantee safe AI behavior, and its Agent APP example is for academic reference; adopters need their own review and execution controls.
How do you install or deploy this agent?
Install Python 3.10.16 or later, then run:
pip install -r requirements.txtThe documented local commands do not require an API key. Obtain a model directory; the examples use THUDM/cogagent-9b-20241220. BF16 inference requires at least 29GB of VRAM.
How do you use this agent?
Prepare screenshots from the platform selected in the command, then run:
python inference/cli_demo.py --model_dir THUDM/cogagent-9b-20241220 --platform "Mac" --max_length 4096 --top_k 1 --output_image_path ./results --format_key status_action_op_sensitiveDuring interaction, construct the prompt in the documented order: Task, History steps, Platform, then format. To run the web demo:
python inference/web_demo.py --host 0.0.0.0 --port 7860 --model_dir THUDM/cogagent-9b-20241220 --format_key status_action_op_sensitive --platform "Mac" --output_dir ./resultsHow does this agent compare with similar options?
The project’s technical blog compares the model with GPT-4o-20240806, Claude-3.5-Sonnet, GPT-4o + UGround, GPT-4o + OS-ATLAS, and the open models Qwen2-VL, ShowUI, and SeeClick. The repository reports strong results on Screenspot, OmniAct, CogAgentBench-basic-cn, and OSWorld; these are project-reported results and should be retested in the intended environment.