# decrawl API reference

The decrawl API turns websites into clean content and structured data.

- **[Scrape](#scrape)** — read one URL and get Markdown, HTML, links, images, records, a summary, or JSON. Add browser actions or Agent Mode for interactive pages.
- **[Crawl](#crawl)** — start from one URL, follow its links, and scrape every page it finds.
- **[Map](#map)** — list a website's URLs from its links and sitemap, without scraping them.

Download the [OpenAPI specification](openapi.json) to import every endpoint into your API client.

## Base URL

All requests go to this host. Append the endpoint path, for example `POST /v1/scrape`.

```text
https://api.decrawl.sixteenlabs.space
```

## Authentication

Send your API key as a bearer token in the `Authorization` header of every request. Create a key from your dashboard.

```text
Authorization: Bearer <YOUR_API_KEY>
Content-Type: application/json
```

Keep your key on your server. A missing, invalid, or expired key returns `401`.

## Response codes

decrawl uses standard HTTP status codes. A `2xx` means success, a `4xx` means the request needs fixing, and a `5xx` means the target site or decrawl failed.

| Status | Meaning |
| --- | --- |
| `200` | The request succeeded. |
| `400` | The body is invalid: a bad URL, selector, field value, or failed browser action. |
| `401` | The API key is missing, invalid, revoked, or expired. |
| `403` | The URL points to a private or non-public address. |
| `404` | The crawl ID is unknown or its results have expired. |
| `408` | The scrape timed out. |
| `429` | Your account has no requests left. |
| `5xx` | The target site blocked or failed the request, or decrawl had an error. |

Errors return a JSON body. Include `request_id` when you report a problem.

```json
{
  "success": false,
  "error": "Invalid request body: invalid character '}' looking for beginning of object key string",
  "code": "BAD_REQUEST",
  "request_id": "ab966ca3539c9868"
}
```

## Scrape — POST /v1/scrape

Scrape one URL and get it back in the formats you ask for. Pages are fetched directly; `waitFor`, `actions`, or `agent` load them in a real browser. HTML, PDFs with a text layer, text, JSON, and XML are supported, up to 50 MiB.

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links"]
  }'
```

### Body

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string, **required** | — | The page to scrape. A URL without a scheme uses HTTPS. |
| `formats` | array | `["markdown"]` | Outputs to return; see [formats](#scrape-formats). |
| `prompt` | string | — | Describe the data you want; adds `json` output. Also the Agent Mode task when `agent.prompt` is empty. |
| `agent` | object | — | Turns on Agent Mode; see [Agent Mode](#scrape-agent-mode). |
| `onlyMainContent` | boolean | `true` | Drop navigation, banners, footers, and sidebars. |
| `includeTags` | string[] | `[]` | CSS selectors to keep. |
| `excludeTags` | string[] | `[]` | CSS selectors to remove. |
| `actions` | object[] | `[]` | Browser actions to run first; see [actions](#scrape-actions). |
| `waitFor` | integer (ms) | `0` | Wait after the page loads in the browser, 0–60,000. |
| `timeout` | integer (ms) | `120000` | Deadline for the whole scrape, including model work; 1,000–300,000. Omit or use `0` for 120 seconds. |
| `headers` | object | — | Headers to send to the page. Turns off caching. |
| `maxAge` | integer (ms) | `172800000` | Return a cached result up to this old (2 days). `0` always scrapes fresh. |
| `storeInCache` | boolean | `true` | Set `false` to not cache this result. |

The cache stores both the complete successful response (including LLM output) and the scraped source page. Identical requests return the saved response directly. A different prompt or format reuses the source page and generates new output. Failed scrapes, failed extraction, and timeouts never enter the cache. Entries last up to two days, are limited to 8 MiB each, and can take time to propagate between locations.

Agent Mode checks cached content with the model first and opens a live browser if the task needs more navigation or interaction. Explicit `actions` and custom `headers` bypass both caches. Set `maxAge: 0` to force a fresh scrape and model call; `storeInCache: false` permits cache reads but prevents writes. `X-Cache: hit` indicates a reused response or source page.

For product images, include `"formats": ["markdown", "images"]`. Extraction receives the discovered gallery URLs along with the page content, even when the visible Markdown shows only one image. Images available only after an interaction may require browser actions. Set `"timeout": 120000` for a two-minute deadline.


### Formats

Each entry is a string such as `"markdown"`, or an object with `type`, plus optional `prompt` and `schema` (a JSON Schema) for `json`.

| Format | Returns |
| --- | --- |
| `markdown` | `data.markdown`: the page as clean Markdown. |
| `html` | `data.html`: cleaned HTML. |
| `rawHtml` | `data.rawHtml`: the page's original HTML. |
| `links` | `data.links`: every link URL on the page. |
| `images` | `data.images`: all discovered image URLs, including lazy/responsive images, gallery zoom URLs, product JSON-LD image arrays, and embedded gallery data. |
| `records` | `data.records`: tables, lists, and repeated cards as rows and columns. |
| `summary` | `data.summary`: a short summary written by the model. |
| `json` | `data.json`: structured data shaped by your prompt and schema. |

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": [{
      "type": "json",
      "prompt": "Extract product names and prices.",
      "schema": {"type": "object", "properties": {"products": {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "string"}}}}}}
    }]
  }'
```

### Actions

Actions run in the browser, in order, before the page is read. A failed action returns `400 ACTION_FAILED`.

| `type` | Fields | Does |
| --- | --- | --- |
| `wait` | `milliseconds` (1–60,000) or `selector` | Waits for a time or for an element. |
| `click` | `selector`, optional `all` | Clicks the first match, or every match. |
| `write` | `text` | Types into the focused element. |
| `press` | `key` | Presses a key such as `Enter` or `Escape`. |
| `scroll` | `direction` (`up` / `down`), optional `selector` | Scrolls one screen. |
| `scrape` | — | Saves the HTML at this point to `data.actions.scrapes`. |
| `executeJavascript` | `script` | Runs JavaScript; the result goes to `data.actions.javascriptReturns`. |

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "actions": [{"type": "click", "selector": "#load-more"}, {"type": "wait", "milliseconds": 1000}, {"type": "scrape"}]
  }'
```

### Agent Mode

Give the agent a goal and it clicks, types, scrolls, and navigates until it finds the answer, then returns your formats.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `agent.prompt` | string | top-level `prompt` | What the agent should find. |
| `agent.maxSteps` | integer | `10` | Most actions the agent may take, 1–25. |

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "prompt": "Find the latest article and give its title and date.",
    "agent": {"maxSteps": 10}
  }'
```

`data.agent.steps` lists what the agent did. `data.agent.finished` is `false` if it ran out of steps.

### Response

```json
{
  "success": true,
  "data": {
    "markdown": "# Example Domain\nThis domain is for use in documentation examples.",
    "links": ["https://www.iana.org/domains/example"],
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com",
      "url": "https://example.com/",
      "statusCode": 200,
      "contentType": "text/html",
      "engine": "http",
      "elapsedMs": 123
    }
  }
}
```

Only the formats you asked for are returned.

| Field | Type | Description |
| --- | --- | --- |
| `success` | boolean | `true` when the scrape worked. |
| `data.markdown` | string | Page content as Markdown. |
| `data.html` | string | Cleaned HTML. |
| `data.rawHtml` | string | Original HTML. |
| `data.links` | string[] | Link URLs. |
| `data.images` | string[] | All discovered, deduplicated image URLs, including gallery variants. The list can include other page images such as logos. |
| `data.records` | object[] | Each has `type` (`table` or `list`), `columns`, and `rows`. |
| `data.summary` | string | Model summary. |
| `data.json` | any | Model output for your prompt or schema. |
| `data.warning` | string | Set when a format was skipped or input was shortened. |
| `data.actions` | object | `scrapes` and `javascriptReturns` from your actions. |
| `data.agent` | object | Agent Mode `steps` and `finished`. |
| `data.metadata` | object | Always present. |
| `data.metadata.sourceURL` | string | The URL you sent. |
| `data.metadata.url` | string | The final URL after redirects. |
| `data.metadata.statusCode` | integer | The page's HTTP status. |
| `data.metadata.engine` | string | `http` or `browser`. |
| `data.metadata.elapsedMs` | integer | Time taken in milliseconds. |
| `data.metadata.title` | string | Page title. |
| `data.metadata.description` | string | Page description. |
| `data.metadata.language` | string | Page language. |
| `data.metadata.contentType` | string | Content type of the page. |
| `data.metadata.numPages` | integer | Page count, for PDFs. |
| `data.metadata.cacheState` | string | `hit` when the complete response or its source page came from the cache. |
| `data.metadata.cachedAt` | string | When the cached result was saved. |

## Crawl — POST /v1/crawl

Start a crawl from one URL. decrawl follows links and scrapes each page in the background; the response returns a crawl ID right away. Poll [crawl status](#crawl-get-crawl-status) for results.

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/crawl \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 10,
    "scrapeOptions": {"formats": ["markdown"]}
  }'
```

### Body

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string, **required** | — | Where the crawl starts. |
| `limit` | integer | `10000` | Most pages to scrape. |
| `includePaths` | string[] | `[]` | Only crawl paths matching these regular expressions. |
| `excludePaths` | string[] | `[]` | Skip paths matching these regular expressions. |
| `regexOnFullURL` | boolean | `false` | Match the path filters against the full URL. |
| `maxDiscoveryDepth` | integer | unlimited | How many links deep to go from the start page. |
| `sitemap` | string | `include` | `include` the sitemap, `skip` it, or use `only` the sitemap. |
| `ignoreQueryParameters` | boolean | `false` | Treat URLs that differ only by query string as one page. |
| `crawlEntireDomain` | boolean | `false` | Also crawl pages outside the start URL's path. |
| `allowExternalLinks` | boolean | `false` | Follow links to other sites. |
| `allowSubdomains` | boolean | `false` | Follow links to subdomains. |
| `ignoreRobotsTxt` | boolean | `false` | Crawl pages that robots.txt blocks. |
| `robotsUserAgent` | string | `*` | Which robots.txt rules to follow. |
| `delay` | number (s) | `0` | Wait between pages, 0–60. Forces one page at a time. |
| `maxConcurrency` | integer | `10` | Pages scraped at once, 1–50. |
| `scrapeOptions` | object | Markdown | [Scrape](#scrape-body) options for every page, including `formats`, `prompt`, `actions`, and `agent`. |

### Response

```json
{
  "success": true,
  "id": "f6445ae4-f73d-42e0-a769-e74a4fe0b92d",
  "url": "https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d"
}
```

### Get crawl status — GET /v1/crawl/{id}

Returns progress and the pages scraped so far. Crawls run in the background, so poll this endpoint until `status` is `completed`, `failed`, or `cancelled`. Status checks don't count against your account.

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d \
  -H "Authorization: Bearer $API_KEY"
```

Poll every few seconds until the crawl finishes:

```bash
CRAWL_ID=f6445ae4-f73d-42e0-a769-e74a4fe0b92d
while :; do
  STATUS=$(curl -sS "https://api.decrawl.sixteenlabs.space/v1/crawl/$CRAWL_ID" \
    -H "Authorization: Bearer $API_KEY" | jq -r .status)
  echo "$STATUS"
  [ "$STATUS" != "scraping" ] && break
  sleep 3
done
```

Large results are split into responses of about 10 MiB. When `next` is not `null`, request that URL with the same `Authorization` header, or pass `?skip=N` to leave out the first N scraped pages.

```json
{
  "success": true,
  "status": "completed",
  "total": 1,
  "completed": 1,
  "creditsUsed": 1,
  "createdAt": "2026-11-01T10:00:00Z",
  "completedAt": "2026-11-01T10:00:02Z",
  "expiresAt": "2026-11-01T11:00:02Z",
  "duration": 2,
  "next": null,
  "data": [
    {
      "markdown": "# Example Domain\nThis domain is for use in documentation examples.",
      "metadata": { "sourceURL": "https://example.com", "url": "https://example.com/", "statusCode": 200, "engine": "http", "elapsedMs": 240 }
    }
  ]
}
```

| Field | Description |
| --- | --- |
| `status` | `scraping`, `completed`, `failed`, or `cancelled`. |
| `total` | Pages found so far. |
| `completed` | Pages scraped successfully. |
| `creditsUsed` | Pages scraped successfully. |
| `createdAt`, `completedAt` | When the crawl started and finished. |
| `expiresAt` | When the results are deleted. Save them before this. |
| `duration` | Seconds the crawl has run. |
| `next` | URL of the next page of results, or `null`. |
| `data` | Scraped pages, each shaped like a [scrape response](#scrape-response)'s `data`. |

### Get crawl errors — GET /v1/crawl/{id}/errors

Lists the pages that failed and the URLs robots.txt blocked.

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d/errors \
  -H "Authorization: Bearer $API_KEY"
```

```json
{
  "errors": [
    { "id": "1", "timestamp": "2026-11-01T10:00:01Z", "url": "https://example.com/broken", "error": "target returned HTTP 404" }
  ],
  "robotsBlocked": ["https://example.com/private"]
}
```

### Cancel a crawl — DELETE /v1/crawl/{id}

Stops a running crawl. Pages already scraped stay available from crawl status.

```bash
curl -sS -X DELETE https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d \
  -H "Authorization: Bearer $API_KEY"
```

```json
{ "status": "cancelled" }
```

## Map — POST /v1/map

Get a website's URLs fast. Map reads the start page's links and the sitemap; it does not scrape the pages it finds.

```bash
curl -sS https://api.decrawl.sixteenlabs.space/v1/map \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "search": "pricing",
    "limit": 100
  }'
```

### Body

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string, **required** | — | The website to map. |
| `search` | string | — | Put URLs containing these words first. |
| `sitemap` | string | `include` | `include` page links and the sitemap, `skip` the sitemap, or use `only` the sitemap. |
| `includeSubdomains` | boolean | `true` | Keep URLs on subdomains. |
| `ignoreQueryParameters` | boolean | `true` | Treat URLs that differ only by query string as one. |
| `limit` | integer | `5000` | Most URLs to return, 1–100,000. |
| `timeout` | integer (ms) | `120000` | Deadline for the map. |

### Response

```json
{
  "success": true,
  "links": [
    { "url": "https://example.com/" },
    { "url": "https://example.com/pricing" }
  ]
}
```

| Field | Description |
| --- | --- |
| `success` | `true` when the map worked. |
| `links` | URLs found, each as an object with `url`. |