Scrape API

Extract content from web pages in multiple formats.

Scrape a URL

POST /api/v1/scrape
Content-Type: application/json
x-api-key: YOUR_API_KEY

{
  "url": "https://example.com",
  "formats": ["markdown", "html", "links"],
  "onlyMainContent": true,
  "timeout": 30000
}

Parameters

Field	Type	Required	Description
`url`	string	Yes	URL to scrape
`formats`	string[]	No	Output formats: markdown, html, rawHtml, links, screenshot, json
`onlyMainContent`	boolean	No	Extract main content only (strips nav, footer, etc.)
`includeTags`	string[]	No	Only include these HTML tags
`excludeTags`	string[]	No	Exclude these HTML tags
`headers`	object	No	Custom request headers
`waitFor`	number	No	Wait ms after page load
`timeout`	number	No	Request timeout in ms

Response

{
  "success": true,
  "data": {
    "markdown": "# Page Title\n\nContent...",
    "html": "<h1>Page Title</h1>...",
    "links": ["https://example.com/page1", "https://example.com/page2"],
    "metadata": {
      "title": "Example Page",
      "description": "A description",
      "language": "en",
      "statusCode": 200
    }
  }
}

Batch Scrape

POST /api/v1/scrape/batch
Content-Type: application/json
x-api-key: YOUR_API_KEY

{
  "urls": [
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3"
  ],
  "formats": ["markdown"]
}

Credits: 1 per URL in the batch.

Async Scrape

POST /api/v1/scrape/async
Content-Type: application/json
x-api-key: YOUR_API_KEY

{
  "url": "https://example.com",
  "formats": ["markdown"]
}

Response:

{
  "success": true,
  "data": {
    "id": "scrape_abc123"
  }
}

Check Status

GET /api/v1/scrape/async/:id

AI Extraction

Use AI to extract structured data from pages:

POST /api/v1/scrape
Content-Type: application/json
x-api-key: YOUR_API_KEY

{
  "url": "https://example.com/products",
  "formats": ["json"],
  "extract": {
    "prompt": "Extract all product names and prices",
    "schema": {
      "type": "object",
      "properties": {
        "products": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "price": { "type": "number" }
            }
          }
        }
      }
    }
  }
}