JARVIS / HuggingGPT
A ChatGPT-controlled system that plans tasks and orchestrates Hugging Face expert models for multi-step AI requests.
- Source repo
- microsoft/JARVIS
- Stars
- ★ 26k
- Last updated
- 1y ago
- License
- MIT
- Primary language
- Python
- FA score
- 36/100 · Major gaps
At a glance
- How it runs
- Works with
- Platform-specificOpenAI API
- Cost
- Free software; you pay for model usage
- Setup effort
- High · needs real infrastructure
- You'll need
- Typical use
- A user combining the pose from one image with the content of another can have JARVIS plan tasks such as
openpose-control,image-to-text, andopenpose-text-to-image. - Not a fit if
- Users without 24GB+ VRAM and 284GB+ disk for local deployment
- Anyone wanting a quick trial without API keys and large model downloads
- Source review
- 36/100 · Major gaps 8 safety controls not found
What does this agent do, and when should you use it?
JARVIS, presented in its paper as HuggingGPT, is a collaborative system with an LLM controller and expert-model executors from Hugging Face Hub. It plans a user request, selects models from their descriptions, executes them, and uses ChatGPT to synthesize the predictions into a response. models_server.py supplies local or hybrid inference endpoints, while awesome_chat.py supports server and CLI modes; a Web client and Gradio demo are also documented. Its POST endpoints are /hugginggpt for the full service, /tasks for planning output, and /results for model-selection and execution results. Local and hybrid operation require downloaded expert models and substantial default hardware capacity, whereas the lite configuration avoids local model deployment but is limited to stable Hugging Face Inference Endpoints.
JARVIS accepts conversational requests, including requests that reference local file paths. In Task Planning, ChatGPT analyzes intent and decomposes the request into solvable tasks; in Model Selection, it selects Hugging Face expert models from their descriptions; in Task Execution, it invokes those models; and in Response Generation, it integrates their predictions. With models_server.py running, POST /tasks returns Stage #1 task plans, POST /results exposes results through model selection and execution, and POST /hugginggpt accesses the complete service. Users can work interactively through awesome_chat.py --mode cli, or use the Web client or run_gradio_demo.py after starting the relevant services.
- A user combining the pose from one image with the content of another can have JARVIS plan tasks such as
openpose-control,image-to-text, andopenpose-text-to-image. - A researcher counting objects across a collection of local image files can use the system’s image-captioning and object-detection workflow to obtain a consolidated answer.
- A user extracting named entities from prose can have the system select a token-classification model and return its processed result.
- An application developer needing HTTP access to a staged task workflow can inspect planning through
/tasksor call/hugginggptfor the complete service. - A team operating NVIDIA Jetson AGX Orin hardware can use the documented Jetson Dockerfile path to run the model server, chat service, and Web application under the stated resource constraints.
How do you install or deploy this agent?
Put a personal OpenAI Key and Hugging Face Token in server/configs/config.default.yaml, or set OPENAI_API_KEY and HUGGINGFACE_ACCESS_TOKEN. For the server: run cd server, conda create -n jarvis python=3.8, conda activate jarvis, conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia, and pip install -r requirements.txt. For local or hybrid inference, change to models and run bash download.sh; git-lfs must be installed. The documented default configuration requires Ubuntu 16.04 LTS, at least 24GB VRAM, more than 12GB RAM, and more than 284GB disk. configs/config.lite.yaml avoids downloading local expert models.
How do you use this agent?
From server, start python models_server.py --config configs/config.default.yaml when using local or hybrid, then start python awesome_chat.py --config configs/config.default.yaml --mode server. Send a POST request to http://localhost:8004/tasks with a JSON messages array, for example {"role":"user","content":"based on pose of /examples/d.jpg and content of /examples/e.jpg, please show me a new image"}. Use /hugginggpt for the full service and /tasks or /results for intermediate outputs. For a terminal session, run python awesome_chat.py --config configs/config.default.yaml --mode cli. For the browser client, run npm install and npm run dev from web.
What are this agent's strengths and limitations?
- Implements a defined four-stage flow—planning, model selection, execution, and response generation—and exposes intermediate stages through
/tasksand/results. - Documents HTTP, CLI, Web, and Gradio interfaces rather than a single interaction surface.
- Offers
local,huggingface, andhybridendpoint modes, plusminimal,standard, andfulllocal deployment scales. - Documents Azure OpenAI service and GPT-4 support, and includes an experimental Docker route for NVIDIA Jetson devices.
- Its controller depends on ChatGPT/OpenAI credentials, while expert execution depends on a Hugging Face token and Hugging Face models or Inference Endpoints.
- The documented default local deployment is resource-intensive: at least 24GB VRAM and more than 284GB disk, with several large model downloads.
- The lite configuration depends on stable Hugging Face Inference Endpoints; the documentation also notes possible instability of remote endpoints in hybrid use.
- The documented configuration lists
text-davinci-003as the current LLM; additional open-source LLM integration is described as work in progress without a usable configuration.
How does this agent compare with similar options?
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| JARVIS / HuggingGPT This agent | 36 · Major gaps | Self-hosted serviceFree + model costs | ★ 26k | 1y ago | Python | OpenAI API |
| Cache-to-Cache | 43 · Major gaps | Library / SDKFree | ★ 770 | 9d ago | Python | — |
| AutoResearch | 52 · Major gaps | CLIFree + model costs | ★ 99k | 7mo ago | Python | Codex · Claude Code |
| AgileRL | 71 · Some gaps | Library / SDKFreemium | ★ 965 | 1d ago | Python | — |
How does FollowAgents rate this agent?
Why each dimension lost points
Evidence shows: The repository does not provide explicit mechanisms for permission management, user confirmation, data flow transparency, sensitive data handling, dependency security, external effects, rollback, or source attribution. The README only requires users to provide OpenAI Key and Hugging Face Token but does not explain their usage scope or protection. Therefore, all trust criteria score 0.
Evidence shows: The README and configuration instructions are consistent in several places, but detailed error handling or failure messages are not provided. Dependency availability (e.g., PyTorch, Hugging Face models) is not explicitly guaranteed, but a download script is provided. Thus, self-consistency scores 1, dependency availability scores 1, and failure messages score 1.
Evidence shows: The README clearly describes multiple usage scenarios (server, web, Gradio, CLI) and configuration options (inference_mode, local_deployment), and provides system requirements. Capability boundaries are somewhat reflected through configuration descriptions, but trigger precision (e.g., specific rules for task planning) is not detailed. Therefore, audience and scenarios score 2, capability boundaries score 2, trigger precision scores 1, and environment fit scores 2.
Evidence shows: The README provides clear installation steps, configuration instructions, and examples, but lacks explicit FAQ and known limitations. The license is MIT, but versioning/changelog is not provided. Maintenance responsibility is indicated through Microsoft's SECURITY.md. Thus, information architecture scores 2, install notes score 2, naming stability scores 1, examples and FAQ score 2, known limitations score 1, license scores 2, versioning/changelog scores 1, and maintenance responsibility scores 2.
Evidence shows: The README provides detailed output examples (e.g., CLI dialogues), indicating good output usability. Marginal value is demonstrated by connecting LLMs and Hugging Face models, but cost-benefit is not explicitly discussed (e.g., computational resource requirements). Therefore, output usability scores 2, marginal value scores 2, and cost-benefit scores 1.
Evidence shows: The README cites papers and arXiv links, but does not provide detailed verification methods. Cross-source corroboration is limited, and separation of facts and inferences is not clear. Thus, claim traceability scores 2, cross-source corroboration scores 1, and fact-inference separation scores 1.
- Not found in source: least-privilege scopingGrant only what the task needs: a dedicated account or read-only token, scoped to specific directories and repos.
- Not found in source: confirmation before actingTurn on (or add) a confirmation step before it acts, and try it in a sandbox or test environment before real data.
- Not found in source: data-flow disclosureWatch which external services it contacts (proxy or firewall logs) and keep sensitive data out until you know where it goes.
- Not found in source: sensitive-data handlingUse dedicated, low-privilege, revocable API keys — never production credentials — and keep secrets out of logs.
- Not found in source: dependency securityPin versions and run a dependency audit (npm audit, pip-audit) before installing; prefer running it in a container.
- Not found in source: disclosed external effectsEstablish which external systems it writes to, sends to or changes, and verify with test accounts or repos before production.
- Not found in source: rollback or recovery pathBack up first, or work on a git branch or snapshot, so its changes can be undone.
- Not found in source: verifiable attributionInstall from the official repo or registry and check the publisher and URL to avoid look-alike packages.
- The repository requires users to provide OpenAI Key and Hugging Face Token but does not explain their usage scope or protection, posing a risk of sensitive information leakage.
- Dependency availability (e.g., PyTorch, Hugging Face models) is not explicitly guaranteed, and detailed error handling or failure messages are not provided.
- Versioning/changelog and known limitations are not provided, which may affect users' judgment of project stability.
FAQ
Must I host expert models locally?
configs/config.lite.yaml requires no local expert-model download or deployment, but it is restricted to stable Hugging Face Inference Endpoints. local and hybrid modes require the model server and downloaded models.Which credentials are required?
server/configs/config.default.yaml or supplied through OPENAI_API_KEY and HUGGINGFACE_ACCESS_TOKEN.Can I inspect the planned tasks before the full response?
POST /tasks exposes Stage #1 planning output, and POST /results exposes intermediate results through model selection and execution.Is Jetson deployment supported?
Dockerfile.jetson. Its documented target is the Jetson AGX Orin family, preferably with 64GB onboard RAM, using inference_mode: local and local_deployment: standard.