Crawl API

Crawl entire websites, following links and extracting content from each page.

Start a Crawl

POST /api/v1/crawl
Content-Type: application/json
x-api-key: YOUR_API_KEY

{
  "url": "https://example.com",
  "limit": 100,
  "maxDepth": 3,
  "includePaths": ["/docs/*", "/blog/*"],
  "excludePaths": ["/admin/*"],
  "scrapeOptions": {
    "formats": ["markdown"],
    "onlyMainContent": true
  }
}

Parameters

Field	Type	Default	Description
`url`	string	-	Starting URL
`limit`	number	50	Max pages to crawl
`maxDepth`	number	3	Max link depth
`includePaths`	string[]	-	Only crawl matching paths (glob patterns)
`excludePaths`	string[]	-	Skip matching paths
`allowBackwardLinks`	boolean	false	Follow links to parent pages
`allowExternalLinks`	boolean	false	Follow links to other domains
`ignoreSitemap`	boolean	false	Skip sitemap discovery
`scrapeOptions`	object	-	Scrape config for each page
`webhook`	string	-	URL to receive results when complete

Response (Sync)

{
  "success": true,
  "data": [
    {
      "url": "https://example.com",
      "markdown": "# Home\n...",
      "metadata": { "title": "Home", "statusCode": 200 }
    },
    {
      "url": "https://example.com/about",
      "markdown": "# About\n...",
      "metadata": { "title": "About", "statusCode": 200 }
    }
  ],
  "total": 2,
  "creditsUsed": 4
}

Async Crawl

POST /api/v1/crawl/async
Content-Type: application/json
x-api-key: YOUR_API_KEY

{
  "url": "https://example.com",
  "limit": 500,
  "maxDepth": 5
}

Response:

{
  "success": true,
  "data": { "id": "crawl_abc123" }
}

Check Crawl Status

GET /api/v1/crawl/async/:id

Check Crawl Errors

GET /api/v1/crawl/async/:id/errors

Cancel Crawl

DELETE /api/v1/crawl/async/:id