Spider - High-Performance Web Crawler and Scraper in Rust

What does this agent do, and when should you use it?

Spider is a concurrency-first web crawling engine written in Rust, designed to quickly crawl and stream web data. It prioritizes HTTP requests and only launches headless Chrome when a page requires JavaScript, improving efficiency. The project offers a Rust library, CLI, Node.js and Python packages, and supports managed crawling via Spider Cloud. The engine includes built-in concurrency, rate limiting, retries, proxies, and stealth features, and can scale from a single script to a distributed fleet. Spider is suitable for building data pipelines for LLM and RAG, monitoring, data export, and AI browsing agents.

Spider asynchronously crawls websites, automatically discovering links and respecting crawl boundaries. It uses HTTP clients first and launches headless Chrome when JavaScript is needed. Pages are streamed back as soon as they are fetched, and users can subscribe to receive them. Configuration options include concurrency limit, crawl depth, delay, robots.txt respect, subdomain handling, user agent, and stealth mode. It can export content in Markdown, JSON, or WARC formats. It runs locally or via Spider Cloud, which provides proxy and auto-unblocking. All operations are performed through the Rust API or CLI.

1. AI/LLM developers extracting web data from large volumes of pages to feed vector stores for RAG pipelines.
2. SEO teams monitoring website changes, tracking rankings, or watching competitor updates.
3. Price monitoring systems that regularly scrape product pages and compare prices.
4. Researchers or analysts exporting web content as Markdown or JSON for processing or archiving.
5. AI browsing agents driving headless Chrome to interact with dynamic websites.
6. Teams needing high-concurrency crawling, scaling from a single script to a distributed cluster without code changes.

How do you install or deploy this agent?

To use as a Rust library, run cargo add spider in your project. For JavaScript rendering, enable the 'chrome' feature in Cargo.toml. Alternatively, install the CLI with cargo install spider_cli, use Node.js via npm i @spider-rs/spider-rs, or Python via pip install spider_rs.

How do you use this agent?

In Rust code, create a Website instance, set the target URL and configuration (such as concurrency and depth), then call crawl() or crawl_smart() to start crawling. Subscribe to receive streamed page data. Example:

use spider::{tokio, website::Website};

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    let mut rx = website.subscribe(16);
    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{}  {}", page.status_code, page.get_url());
        }
    });
    website.crawl().await;
    website.unsubscribe();
}

What are this agent's strengths and limitations?

How does this agent compare with similar options?

Key facts side by side with the most closely related agents.

How does FollowAgents rate this agent?

Recent releases

Latest entries from the project's GitHub Releases. Notes are quoted from upstream as written, not rewritten or tested by FollowAgents.

1. v2.52.2 v2.52.2 — CLI authenticate for Spider Cloud 2026-03-31
   Release notes excerpt

   Changes since vv1.7.5 - fix: concurrency/memory/panic audit hardening (v2.52.7) (#401) - chore(release): bump workspace to v2.52.6 - browser: per-crawl control map for optional chrome enhancements (#400) - chore(release): bump workspace to v2.52.5 - browser: genericize internal gateway topology/metrics in doc comments - browser: harden in-request gateway re-acquire (jitter, opt-out, timeouts) - security: fix cargo-audit failures (quick-xml DoS + lopdf) - chore(deps): bump actions/checkout from 4 to 7 (#399) - security: bump quinn-proto to 0.11.15 (RUSTSEC-2026-0185) …

   View on GitHub ↗
2. v2.48.4 2026-03-25
   Release notes excerpt

   New spider_mcp crate — MCP server for Spider. ```bash cargo install spider_mcp ``` Setup (Claude Code ~/.claude/settings.json or Claude Desktop config): ```json { "mcpServers": { "spider": { "command": "spider-mcp" } } } ``` Usage examples: ``` Scrape a page: "Fetch https://example.com as markdown" Crawl a site: "Crawl https://example.com up to 5 pages" Extract links: "Get all links from https://example.com" Transform HTML: "Convert this HTML to markdown: <h1>Hello</h1>" ``` Tools: spider_scrape · spider_crawl · spider_links · spider_transform

   View on GitHub ↗
3. v2.48.2 v2.48.2 — Parallel Crawl Backends 2026-03-25
   Release notes excerpt

   Race alternative browser engines alongside your primary crawl. Best HTML wins. ```rust use spider::configuration::{BackendEndpoint, BackendEngine, ParallelBackendsConfig}; let mut website = Website::new("https://example.com"); website.configuration.parallel_backends = Some(ParallelBackendsConfig { backends: vec![BackendEndpoint { engine: BackendEngine::LightPanda, endpoint: Some("ws://127.0.0.1:9222".to_string()), binary_path: None, protocol: None, }], ..Default::default() }); website.crawl().await; …

   View on GitHub ↗

FAQ