Scrapling Web Scraping Framework
Handle page requests, browser rendering, extraction, and concurrent crawls in one Python framework.
- Source repo
- D4Vinci/Scrapling
- Stars
- ★ 89k
- Last updated
- 1d ago
- License
- BSD-3-Clause
- Primary language
- Python
- FA score
- 66/100 · Some gaps
At a glance
- How it runs
- Works with
- Universal · cross-platformClaude.ai (Partial support)
- Cost
- Free, no paid service needed
- Setup effort
- Medium · a few setup steps
- You'll need
- Typical use
- Python developers extracting structured fields from static pages can use Fetcher with CSS or XPath selectors.
- Not a fit if
- Users who only need a simple static-page fetch
- Environments that cannot run Python or access the network
- Source review
- 66/100 · Some gaps
What does this agent do, and when should you use it?
Scrapling is a Python web scraping framework with HTTP fetchers, a Playwright-based StealthyFetcher, a parser, and Spider crawlers. It can fetch static pages, render JavaScript pages, extract content with CSS or XPath, and capture API responses from browser sessions. Spiders support concurrency, proxy rotation, retries, automatic throttling, pause and resume, and JSON, JSONL, CSV, or XML exports. You can import it as a Python library or use its command-line tools; it also offers an MCP server and Agent Skill for AI clients. It runs in a local Python environment and needs network access, with browser dependencies required for browser-based fetching.
After installing scrapling, call Fetcher.get() to make an HTTP request, then use the returned page's css(), xpath(), text, or regex search methods to extract content. For dynamic pages, use StealthyFetcher or StealthySession to fetch through a browser. For larger collections, define a Spider subclass with start_urls and an asynchronous parse() callback that yields Requests or data items, then save results with the built-in exporters. Spiders can configure concurrency, proxy rotation, retries, AutoThrottle, and crawldir checkpoints. The CLI provides scrapling extract get, scrapling extract fetch, and an interactive shell. The optional MCP server lets AI clients browse pages, extract data, take screenshots, and access saved login state and background API responses.
- Python developers extracting structured fields from static pages can use Fetcher with CSS or XPath selectors.
- Researchers collecting JavaScript-rendered content can use StealthyFetcher to render pages and inspect the DOM.
- Teams collecting data across many pages can configure Spider concurrency, retries, proxy rotation, and pause/resume.
- Developers collecting public Shopify catalogs can use ShopifySpider to retrieve product variant data.
- Terminal users who want to save page content without writing a script can use scrapling extract to produce text, Markdown, or HTML.
- Users who want Claude, Cursor, or another AI client to browse and extract web data can configure the Scrapling MCP server.
How do you install or deploy this agent?
Python 3.10 or higher is required. The base install provides the parser; install the fetcher extras for Fetcher or Spider use, then install browser dependencies:
pip install "scrapling[fetchers]"
scrapling installInstall the optional MCP server, shell, or all extras with:
pip install "scrapling[ai]"
pip install "scrapling[shell]"
pip install "scrapling[all]"HTTP fetching does not require browser installation. The README content provided here does not specify the MCP client configuration file format or complete setup steps.
How do you use this agent?
Single-page HTTP fetch:
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)Fetch a page from the terminal and save it as Markdown:
scrapling extract get 'https://quotes.toscrape.com/' quotes.mdFor a crawl, define a Spider subclass and call start(); save results with a method such as result.items.to_json(). The MCP server is an optional interface for AI clients and requires scrapling[ai]; the README does not include a client configuration example.
What are this agent's strengths and limitations?
- One framework covers HTTP requests, browser rendering, page parsing, and concurrent Spider crawls.
- Parsing supports CSS, XPath, text, and regex search, plus adaptive element tracking for page changes.
- Spider includes proxy rotation, retries, AutoThrottle, checkpoint recovery, and several export formats.
- Offers CLI, interactive shell, MCP server, and Agent Skill entry points.
- Requires Python 3.10 or higher, with extra dependencies for Fetcher and Spider use.
- Browser fetching downloads browser binaries and system dependencies, adding setup beyond the parser-only install.
- The provided README content does not explain MCP client configuration details.
- Scraping targets requires network access; browser mode also requires running Playwright browsers.
How does this agent compare with similar options?
The README describes the Spider API as Scrapy-like and says the parser API resembles Scrapy/Parsel and BeautifulSoup. It also documents using the scrapling_response decorator with existing Scrapy callbacks. Teams already using Scrapy can assess that integration; teams looking for one framework spanning HTTP, browser fetching, and crawler scheduling can assess Scrapling.
Key facts side by side with the most closely related agents.
| Agent | Source review | Form / cost | Stars | Updated | Language | Full support on |
|---|---|---|---|---|---|---|
| Scrapling Web Scraping Framework This agent | 66 · Some gaps | Library / SDKFree | ★ 89k | 1d ago | Python | — |
| wigolo | 54 · Major gaps | CLIFree + model costs | ★ 5.5k | 23d ago | TypeScript | Codex · Claude Code · OpenAI API · Claude API |
| CamoFox Browser Server | 72 · Some gaps | Self-hosted serviceFree | ★ 425 | 28d ago | JavaScript | Claude Code |
| Uni-CLI | 72 · Some gaps | CLIFree | ★ 235 | 10d ago | TypeScript | ChatGPT · Codex · Claude Code |
How does FollowAgents rate this agent?
Why each dimension lost points
The repository documents controls such as allowed_domains, optional robots.txt compliance, per-domain delays, and session closing. Robots.txt is off by default, and proxy rotation, stealth browsing, and anti-bot bypass make external requests consequential, so least privilege and external-effects controls are only partial. The README says the MCP server supports authentication options, saved login state, and API-response capture, but the provided material does not explain credential storage, retention, or access controls; requests can be made without per-action confirmation. Pause/resume and session closing offer limited recovery. License and maintainer fields identify the repository author, while publisher identity remains unknown. Dependencies are declared, but no lock or vulnerability mitigation is shown in the supplied files.
The spider guide and README broadly agree on parsing, follow-up requests, and exports, and the supplied test source includes MCP behavior assertions; static review cannot establish runtime results. Dependency versions mostly use lower bounds, with no lockfile or availability guarantee shown. MCP tests demonstrate errors returned as text and spider documentation describes statistics, but evidence for ordinary user-facing failure handling is limited.
The material covers single-page scraping, concurrent spiders, browsers, HTTP, MCP, CLI, streaming, and multiple export formats, for Python developers and AI tools. It describes boundaries such as domain limits, robots.txt, and session types, but gives limited guidance on the appropriate boundaries and authorized use of anti-bot bypass features. Spider settings permit selection of domains, delays, and concurrency; Python 3.10+ and several usage modes are documented, while deployment details are limited.
The README organizes fetching, parsing, crawling, AI, and developer features and links to focused documentation; the spider guide includes prerequisites, a progressive tutorial, examples, and next steps. Installation instructions are not present in the supplied README excerpt; pyproject exposes required and optional dependencies, but evidence for installation steps is limited. Names and API terms are broadly consistent, though stability across the multiple interfaces cannot be established from this material. The docs explain some limitations, including robots.txt being off by default, but no comprehensive known-limitations list is shown. The BSD-3-Clause license text is complete. Version 0.5 and changelog/release-note links are present, but the changelog contents were not supplied. pyproject names an author and maintainer, but does not establish a maintenance response commitment or update process.
Examples provide usable Python code, field extraction, and JSON/JSONL/CSV/XML exports; the framework combines requests, parsing, concurrency, retries, and export with clear value over a one-page script. Browser, MCP, proxy, and dependency options add deployment and operating costs; no quantified cost or resource benchmarks are supplied, and README performance and coverage claims cannot be independently confirmed from these files.
Specific README capabilities can be compared with the spider guide, pyproject, license, and MCP test excerpts; however, only partial test source is supplied, and static review is not execution verification. Performance, coverage, and usage-scale claims lack independent support in the supplied material. The docs specify conditions such as robots.txt being disabled by default; some promotional claims do not distinguish measured facts from assertions, so full marks are not justified.
- The framework sends requests to target websites and offers proxy rotation, browser automation, and anti-bot bypass; users should establish authorization, applicable rules, and request scope themselves.
- The material mentions MCP login state and response capture but does not explain sensitive-data retention or access controls; review those data flows before configuring them.
FAQ
Does Scrapling require a paid service or API key?
Can I use Fetcher and Spider with the minimal installation?
pip install scrapling includes only the parser and its dependencies. Fetcher and Spider require scrapling[fetchers]; browser fetching also requires scrapling install.Can it handle JavaScript pages?
Can a Spider pause and resume a crawl?
crawldir to save pending requests and seen URLs. Restart with the same directory to resume; scraped items must be saved separately.Does Scrapling automatically obey robots.txt?
robots_txt_obey flag for Disallow, Crawl-delay, and Request-rate directives. Whether it is enabled depends on configuration.