WebCrawler API

About WebCrawler API

WebCrawler API crawls a whole site from a single request and returns clean Markdown built for LLM consumption — menus, footers, cookie banners and ads stripped before the content comes back, so what lands in a prompt or a vector index needs no second cleanup pass. Three endpoints cover the range: scrape for one page, crawl for a whole site with an item limit, and an agent mode for tasks that need navigation. The infrastructure it absorbs is the actual value: residential proxies, automatic retries, rate-limit handling, real headless browsers, JavaScript rendering, CAPTCHA solving and anti-bot bypass, with each request routed down the fastest path likely to succeed. Smart caching returns frequently requested pages in under a second instead of several, with a max_age parameter to force a fresh fetch. The feature worth singling out is change detection: you set up a feed for a site and receive only what changed — added pages, removed entries, structural changes and full diffs — instead of running polling loops that re-fetch and re-tokenise unchanged content, which is the expensive default for anyone keeping a documentation index current. SDKs cover Node, Python, PHP, .NET and Java, plus cURL and an AI-agent path, and there are no-code guides for Zapier, Make, n8n and Integrately. Billing is per successful request only.

Key Features

Tags