Universal Scraper

The Universal Scraping API provides advanced page unlocking and JavaScript rendering. Use it to bypass anti-bot protections, render JavaScript-heavy pages, and execute custom instructions.

Basic Usage

import { ScrayleClient } from '@scrayle/sdk';

const client = new ScrayleClient();

const result = await client.universal.scrape({
  actor: 'unlocker.webUnlocker',
  input: {
    url: 'https://protected-site.com',
    js: true,
  },
});

console.log(result);

JavaScript Instructions

Execute a sequence of actions on the page before extracting content:

const result = await client.universal.scrape({
  actor: 'unlocker.jsRender',
  input: {
    url: 'https://example.com/products',
    js: true,
    instructions: [
      { type: 'wait', value: 2000 },
      { type: 'click', selector: '.load-more' },
      { type: 'waitFor', selector: '.extra-items', timeout: 5000 },
      { type: 'fill', selector: '#search', value: 'laptop' },
      { type: 'evaluate', script: 'document.title' },
    ],
  },
});

Instruction Types

Type	Parameters	Description
`wait`	`value` (ms)	Wait for a fixed duration
`waitFor`	`selector`, `timeout?`	Wait for an element to appear
`click`	`selector`	Click an element
`fill`	`selector`, `value`	Type text into an input
`evaluate`	`script`	Execute JavaScript on the page

Available Scrapers

Scraper ID	Description
`unlocker.webUnlocker`	Bypass anti-bot protection and return page content
`unlocker.jsRender`	Full JavaScript rendering with instructions
`unlocker.akamaiWebCookie`	Generate Akamai web cookies
`unlocker.akamaiWebSensor`	Generate Akamai sensor data

Proxy Configuration

const result = await client.universal.scrape({
  actor: 'unlocker.webUnlocker',
  input: { url: 'https://example.com' },
  config: {
    proxy: {
      country: 'US',
      type: 'residential',
    },
    retries: 3,
    timeout: 30000,
  },
});

Content Output Types

The universal scraper can extract specific content types:

phone_numbers - Detect phone numbers on the page
headings - Extract all headings
links - Extract all links
images - Extract all images
tables - Extract table data