Crawl4AI Review

Crawl4AI is a Python library and Docker service for crawling the web into data models can use. The core call is AsyncWebCrawler.arun(), configured with a BrowserConfig (headless mode, user agent, JavaScript toggles) and a CrawlerRunConfig per crawl (caching, extraction, timeouts, hooks, js_code, wait_for, and session_id). Markdown generation returns raw Markdown plus a fit_markdown version after a content filter, and the docs pair the default generator with a pruning filter (threshold 0.4 in the example) that adds about 50 ms of processing. Extraction supports JsonCssExtractionStrategy with a schema of a baseSelector and fields that needs no LLM, a generate_schema() helper that uses an LLM once to create a reusable schema, and LLMExtractionStrategy with a Pydantic schema and any provider LiteLLM supports. Browser control offers persistent profiles, remote browsers over CDP, sessions, proxies, stealth mode, and Chromium, Firefox, and WebKit, and crawling includes deep crawl (BFS, DFS, best-first) with resume state, adaptive crawling, sitemap URL discovery, screenshots, PDFs, and caching.

Install with pip install -U crawl4ai, then run crawl4ai-setup to install the browser and crawl4ai-doctor to check it, and if browser setup fails run python -m playwright install --with-deps chromium. The default cache mode is CacheMode.BYPASS (fresh content), and you opt into caching with CacheMode.ENABLED. arun_many() crawls multiple URLs in parallel with a MemoryAdaptiveDispatcher that adjusts concurrency to system resources, and with stream=True you consume results as they finish. The Docker API server uses the image unclecode/crawl4ai on port 11235, requires a CRAWL4AI_API_TOKEN, exposes endpoints such as /md, /html, /crawl, /crawl/stream, /screenshot, /pdf, and /execute_js, and includes a monitoring dashboard, a playground, MCP support, and AMD64 and ARM64 images (the deployment docs page we tried was unavailable, so check it for resource recommendations). The README shows a Python version badge but states no minimum in text, and the project asks for attribution through a Powered by Crawl4AI badge or a text credit line.

For developers, the library offers an async API you call from scripts or pipelines, and the Docker server gives the same capability over HTTP for other languages. For dynamic pages, js_code runs JavaScript before extraction, wait_for waits for content, and session_id with js_only continues an existing page session until kill_session() cleans it up. An AdaptiveCrawler.digest() mode crawls until it has enough information for a query and reports a confidence score. The hosted Crawl4AI Cloud runs browsers for you and exposes /scrape, /extract, /search, /serp (beta), and /answer (experimental), with a free tier of 10 requests per minute, 2 concurrent, and 2,000 starting credits and a paid tier from $10 with 120 requests per minute and 10 concurrent. The README says cloud pricing is in a soft launch and may change. This section reflects the README and quickstart; we did not run crawls.

• Clean Markdown with a fit_markdown filter, numbered citations, and custom generators
• CSS and XPath extraction without an LLM, a one-time LLM schema generator, and LLM-based extraction via LiteLLM
• Deep crawling (BFS, DFS, best-first) with resume state, plus adaptive crawling with a confidence score
• Persistent browser profiles, CDP remote browsers, proxies, and stealth mode
• Parallel crawling with arun_many and a memory-adaptive dispatcher
• Screenshots, PDFs, caching modes, and JavaScript execution for dynamic pages
• Docker API server with MCP support, a playground, and a monitoring dashboard

• No forced API keys or paywalls on the open-source version
• Cheap structured extraction: generate a CSS schema once with an LLM and then run it without one
• Active, popular project: about 85,000 stars and commits within hours
• Both a library and a ready-to-run Docker service

• You run the browsers, proxies, and scaling yourself, and anti-bot handling is up to you unless you use the cloud
• The quickstart's code snippets omit some imports, so examples need a little fixing before they run
• The README states no minimum Python version in text and has no known-limitations section
• The hosted cloud is in a soft launch, with some endpoints beta or experimental and pricing subject to change
• Attribution is requested, which may matter for some commercial products

The open-source library has no usage metering. Crawl4AI Cloud has a free tier (10 requests per minute, 2 concurrent, 2,000 starting credits) and a paid tier from $10 (120 requests per minute, 10 concurrent), and the README warns the pricing may change during its soft launch.

Jina AI offers hosted APIs for reading and searching the web and for embeddings, and Browserbase provides cloud browsers, while Apify and Tavily are commercial scraping and search platforms. Crawl4AI is the do-it-yourself, free option focused on crawling and extraction: you get more control and no per-request fees, in exchange for operating browsers. Firecrawl is the nearest open-source peer, with a hosted service and a community edition that omits some cloud-only features, so the choice often comes down to whether you prefer a Python library with deep crawl controls or a hosted API.

• AI engineers building RAG pipelines that need clean Markdown from many pages
• Teams that want free, self-hosted crawling with fine browser control
• Developers who want deep crawls with resumable state and cheap structured extraction

• Teams that need managed anti-bot infrastructure out of the box (use the cloud or a commercial service)
• Non-developers who want a no-code scraper
• Organizations that cannot accept an attribution request