CogAgent GUI Operator
A screenshot-driven VLM that produces grounded GUI actions from natural-language tasks.
What does this agent do, and when should you use it?
CogAgent is an open-source vision-language-model project for GUI operation, centered in this repository on CogAgent-9B-20241220. Built on GLM-4V-9B, the model takes screenshots, a task, execution history, a platform marker, and a requested response format. It is an execution-oriented model rather than a continuous conversational model, so callers provide prior history when starting a new session. Its strict string output can include Action, Status, Plan, Grounded Operation, and sensitivity markers; Grounded Operation supplies an operation type and screen coordinates. The repository offers transformers-based CLI inference, a web demo, and a locally deployed Agent APP example, while the hosted demo displays inference results only and cannot control a computer.
A caller concatenates a task, History steps, a platform field (WIN, Mac, or Mobile), and a format field, then submits that prompt with a screenshot to CogAgent-9B-20241220. The model reads the visible interface and prior steps, returning Grounded Operation strings such as CLICK, TYPE, SCROLL_DOWN, or RIGHT_CLICK, with a box identifying the target coordinate region. inference/cli_demo.py interactively accepts image paths and saves annotated output images as {your_input_image_name}_{round}.png. inference/web_demo.py supports repeated image uploads for interactive inference, and the example documented in app/README.md is intended for local GPU-server deployment and automated GUI operations.
- A developer prototyping local, screenshot-driven GUI automation on macOS 14 or 15.
- A researcher who needs click, typing, and scrolling coordinates from Windows 10 or 11 screenshots.
- A mobile-testing team collecting parseable operation instructions for Android 13, 14, or 15 interfaces.
- An engineer validating screen grounding and output parsing with a single-step task such as marking all emails as read.
- An evaluation team with A100- or H100-class GPUs assessing a GUI-agent model locally.
What are this agent's strengths and limitations?
- Consumes screenshots and natural-language tasks directly, then emits coordinate-bearing Grounded Operation output that a downstream executor can parse.
- Explicitly supports Windows, macOS, and Android platform fields, with Chinese and English interaction.
- Includes CLI inference, a web demo, and a local Agent APP example for evaluation and deployment entry points.
- Offers several strict output formats so integrations can request status, plan, actions, and sensitivity labels as needed.
- The model requires images; text-only conversation cannot complete GUI-agent tasks.
- It is not a continuous conversational model, so the caller must provide prior execution history for every new session.
- BF16 inference needs at least 29GB of VRAM, while SFT and LoRA fine-tuning carry substantial A100 memory and multi-GPU requirements.
- The project does not guarantee safe AI behavior, and its Agent APP example is for academic reference; adopters need their own review and execution controls.
How do you install or deploy this agent?
Install Python 3.10.16 or later, then run:
pip install -r requirements.txt
The documented local commands do not require an API key. Obtain a model directory; the examples use THUDM/cogagent-9b-20241220. BF16 inference requires at least 29GB of VRAM.
How do you use this agent?
Prepare screenshots from the platform selected in the command, then run:
python inference/cli_demo.py --model_dir THUDM/cogagent-9b-20241220 --platform "Mac" --max_length 4096 --top_k 1 --output_image_path ./results --format_key status_action_op_sensitive
During interaction, construct the prompt in the documented order: Task, History steps, Platform, then format. To run the web demo:
python inference/web_demo.py --host 0.0.0.0 --port 7860 --model_dir THUDM/cogagent-9b-20241220 --format_key status_action_op_sensitive --platform "Mac" --output_dir ./results
How does this agent compare with similar options?
The project’s technical blog compares the model with GPT-4o-20240806, Claude-3.5-Sonnet, GPT-4o + UGround, GPT-4o + OS-ATLAS, and the open models Qwen2-VL, ShowUI, and SeeClick. The repository reports strong results on Screenspot, OmniAct, CogAgentBench-basic-cn, and OSWorld; these are project-reported results and should be retested in the intended environment.