webclaw Web Extraction Engine
Turn websites into clean Markdown, JSON, and model-ready context with a local-first Rust toolchain.
The README distinguishes the pure extraction core, local fetching, hosted API, proxies, and LLM providers, and enumerates the relevant environment variables; the local core and layered architecture provide some least-privilege evidence. Secrets are shown through environment variables and contributors are told to remove keys, cookies, tokens, and customer data from logs, but there is no documented credential storage, implemented redaction, retention policy, or hosted-service data policy. The installer detects clients and writes MCP configuration, while scraping, crawling, and batching contact external sites, yet the evidence shows no per-action confirmation, destination authorization controls, or configuration preview. A dependency-sync workflow offers limited update handling, but no vulnerability scanning, dependency audit, or strict pinning is shown. Content snapshots support comparison, but configuration changes and external requests lack a general rollback mechanism. Repository, contributor, and partner attribution is visible, although publisher identity remains unverified.
The README, Cargo workspace, and CI are broadly consistent about the Rust workspace, module boundaries, WASM constraint, and local-first positioning. CI covers tests, formatting, Clippy, documentation, and WASM compilation. Installation options and native prerequisites are documented, but npx, the moving stable toolchain, and broad major-version dependencies remain exposed to upstream availability and change. Failure-handling evidence is thin: it includes native-tool troubleshooting, an extraction bug-report template, and limited operational guidance, but does not show structured CLI, REST, or MCP errors, retry and timeout behavior, or partial-failure semantics.
Audience and scenario coverage is thorough across terminal use, MCP agents, RAG, SDKs, monitoring, structured extraction, and brand analysis. The tool table, local-versus-hosted split, architecture, and configuration variables define useful capability boundaries, though hosted fallback behavior, protected-site handling, crawl limits, and LLM-provider selection rules remain incomplete. Tool names and example prompts give reasonable trigger guidance, but MCP parameter schemas and conflict conditions are absent. Environment fit is strongly supported through macOS, Linux, Windows, Docker, Cargo, Homebrew, several MCP clients, and a dedicated WASM check.
The README has strong information architecture, with installation, quick start, tools, SDKs, formats, architecture, configuration, and contribution paths clearly organized; the multiple installation routes and platform prerequisites justify full marks. Naming is broadly stable across crates, CLI, MCP, and SDKs, but compatibility between OSS components, hosted services, and separately named packages is not explained. Examples are numerous and workflow directories are linked, but there is no real FAQ and the linked example contents were not supplied. Some limitations are acknowledged, including protected sites, JavaScript rendering, and poor extraction cases, but there is no systematic treatment of limits, quotas, robots/compliance concerns, scale ceilings, or unsupported cases. AGPL-3.0 is consistent across the README, Cargo metadata, and license file. A workspace version and Releases path exist, but no changelog or compatibility policy is provided. Contribution, issue, discussion, and automated dependency-update paths are clear, while named maintainers and release/support commitments remain unclear.
Markdown, LLM text, plain text, JSON, and cleaned HTML outputs directly serve agents and RAG workflows; selectors, main-content filtering, batching, structured extraction, and diffing strengthen usability. The combination of CLI, REST, MCP, SDKs, and a local extraction layer offers clear marginal value over raw HTML handling. The cost-benefit story distinguishes account-free local capabilities from hosted needs, but no verifiable benchmarks for performance, resource use, accuracy, pricing, or alternatives are supplied, preventing full credit.
Some claims trace to Cargo metadata, the module layout, CI commands, and configuration tables, but promotional claims concerning speed, blocked-page handling, production use, and partner performance lack code references, benchmarks, or test evidence. README, Cargo, and CI provide some corroboration for version, license, Rust architecture, and the WASM target. Facts, illustrative demonstrations, proposed use cases, and sponsor marketing are not consistently separated; in particular, third-party reliability, coverage, and scale figures are unsupported by the supplied evidence.
- The installer detects clients and writes MCP configuration; inspect its proposed file changes and back up existing configuration before use.
- Scraping, crawling, search, proxies, and hosted fallback create outbound traffic. Independently constrain destination domains, rate, depth, and page count, and verify site terms, robots rules, and applicable law.
- Do not assume that every feature remains local merely from the local-first description. Search, research, protected-site handling, JavaScript rendering, and explicit --cloud paths use hosted services; review their data policy before supplying API keys.
- The supplied material does not demonstrate dependency vulnerability scanning, a structured error contract, retry policy, or performance and accuracy benchmarks. Production adoption needs an independent audit and static review; this assessment did not execute the software.
- AGPL-3.0 can impose source-availability obligations for distribution and modified network services; obtain appropriate legal advice before proprietary-service integration.
What does this agent do, and when should you use it?
webclaw is a local-first Rust project for scraping, crawling, and extracting web content for AI assistants and RAG pipelines. The repository contains a CLI, an MCP server, a self-hostable REST server, and separate crates for extraction, fetching, LLM integration, and PDF text extraction. It produces Markdown, compact LLM context, plain text, JSON, or cleaned HTML, while also supporting crawling, URL mapping, batch scraping, structured extraction, summarization, snapshot comparison, and brand-asset extraction. Its core extraction path runs locally without an account; protected-site access, JavaScript rendering, search, asynchronous research jobs, and production usage tracking are provided by the hosted webclaw.io service. TypeScript, Python, and Go SDKs are available, and the MCP setup explicitly covers clients including Codex CLI, Claude Code, Cursor, Windsurf, and OpenCode.
A user submits one or more URLs through the webclaw CLI, REST API, an SDK, or webclaw-mcp. webclaw-fetch handles retrieval, same-origin crawling, batching, and URL mapping; webclaw-core converts HTML into Markdown, text, JSON, LLM format, or cleaned HTML; and webclaw-pdf extracts PDF text. Optional webclaw-llm support connects a local or configured model for extract and summarize. The scrape operation processes one URL, crawl follows same-origin links, map discovers URLs without extracting every page, batch processes multiple URLs in parallel, diff compares stored snapshots, and brand returns colors, fonts, logos, and metadata. The normal extraction path is local, while --cloud and the https://api.webclaw.io/v1/scrape endpoint use hosted infrastructure for protected pages, JavaScript rendering, search, and research workflows.
- A developer using Claude Code, Codex CLI, Cursor, or another MCP client needs clean website context that can be passed directly to a model.
- A data team building a RAG index needs to crawl documentation, help centers, blogs, or knowledge bases and retain structured Markdown.
- A product intelligence team wants to snapshot pricing pages, changelogs, or product pages and use
diffto detect later changes. - An automation engineer needs typed JSON from irregular pages and wants to integrate through REST, TypeScript, Python, or Go.
- A brand researcher needs to collect logos, colors, fonts, and social metadata from a company website.
- A team that prefers infrastructure control wants to run extraction locally, in Docker, or on a self-hosted server while reserving hosted features for difficult sites.
What are this agent's strengths and limitations?
- The core CLI and MCP extraction path can run locally without an account, and the server is self-hostable.
- One toolkit covers single-page scraping, crawling, URL mapping, parallel batches, structured extraction, summaries, diffs, and brand discovery.
- It exposes CLI, MCP, REST, TypeScript, Python, and Go interfaces for different integration styles.
webclaw-coreis separated from network fetching and can be used independently as pure extraction logic.- Markdown, LLM, text, JSON, and cleaned HTML outputs support RAG ingestion, automation, and custom post-processing.
- Protected-site access, JavaScript rendering, web search, asynchronous research jobs, and production usage tracking depend on the hosted webclaw.io API and its key.
extractandsummarizerequire a local or configured model, potentially adding an Ollama, OpenAI-compatible, or Anthropic-compatible dependency.- Building from source carries native setup costs, including OpenSSL development packages, CMake, Clang, and platform compiler tools.
- The supplied material does not document hosted pricing, quotas, or service levels, so operating cost and capacity cannot be assessed from it.
- A Firecrawl-compatible API example is listed, but there is no evidence that every Firecrawl endpoint and behavior is interchangeable.
How do you install or deploy this agent?
For the quickest MCP setup, run npx create-webclaw; the installer detects supported clients such as Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, and Codex CLI and writes their configuration. On macOS, use brew tap 0xMassi/webclaw followed by brew install webclaw. Cargo users can run cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli and cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp. A container can be tested with docker run --rm ghcr.io/0xmassi/webclaw https://example.com. Source builds require the documented platform build tools, including items such as pkg-config, OpenSSL development libraries, CMake, Clang, Git, and a compiler toolchain. Core local extraction requires no account or API key; hosted features require WEBCLAW_API_KEY.
How do you use this agent?
After installing the CLI, verify it with webclaw https://example.com --format markdown. For compact model context, run webclaw https://docs.anthropic.com --format llm; to crawl documentation, run webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50; add --only-main-content when page chrome should be removed. An MCP client can launch the server with command npx and arguments ["-y", "@webclaw/mcp"]. For hosted processing, run export WEBCLAW_API_KEY=wc_your_key and then webclaw https://example.com --cloud. Local model features can use OLLAMA_HOST, while OpenAI-compatible and Anthropic-compatible providers use their documented API-key and base-URL variables.
How does this agent compare with similar options?
The project explicitly positions itself as a Firecrawl alternative and includes a Firecrawl-compatible API example. Its stated distinction from a hosted-only extractor is a Rust-based local core and a self-hosting path, although protected sites, JavaScript rendering, search, and research jobs still rely on webclaw.io. The material does not establish complete Firecrawl parity, so adopters should test the endpoints and response shapes their migration requires.
FAQ
Do I need an account for local use?
WEBCLAW_API_KEY.Can it handle JavaScript-rendered or bot-protected sites?
Can structured extraction and summarization run offline?
extract and summarize can use a local or configured model. Ollama is supported through OLLAMA_HOST; OpenAI-compatible and Anthropic-compatible services are also configurable, so offline operation depends on the provider selected.Will a crawl follow links onto other domains?
crawl behavior follows same-origin links. Cross-domain crawling is not established by the supplied material.