BUILD WITH DECRAWL

The API reference.

A page or an entire website. Turn the web into clean content, structured data, and answers your applications can use.

YOUR BASE URLhttps://api.decrawl.sixteenlabs.space

01Scrape a pageContent, JSON, and Agent Mode 02Crawl a websiteDiscover, scrape, and collect 03Map a websiteEvery URL, without scraping

The decrawl API turns websites into clean content and structured data.

Scrape — read one URL and get Markdown, HTML, links, images, records, a summary, or JSON. Add browser actions or Agent Mode for interactive pages.
Crawl — start from one URL, follow its links, and scrape every page it finds.
Map — list a website's URLs from its links and sitemap, without scraping them.

Download the OpenAPI specification to import every endpoint into your API client.

Base URL#

All requests go to this host. Append the endpoint path, for example POST /v1/scrape.

text

https://api.decrawl.sixteenlabs.space

Authentication#

Send your API key as a bearer token in the Authorization header of every request. Create a key from your dashboard.

text

Authorization: Bearer <YOUR_API_KEY>
Content-Type: application/json

Keep your key on your server. A missing, invalid, or expired key returns 401.

Response codes#

decrawl uses standard HTTP status codes. A 2xx means success, a 4xx means the request needs fixing, and a 5xx means the target site or decrawl failed.

Status	Meaning
`200`	The request succeeded.
`400`	The body is invalid: a bad URL, selector, field value, or failed browser action.
`401`	The API key is missing, invalid, revoked, or expired.
`403`	The URL points to a private or non-public address.
`404`	The crawl ID is unknown or its results have expired.
`408`	The scrape timed out.
`429`	Your account has no requests left.
`5xx`	The target site blocked or failed the request, or decrawl had an error.

Errors return a JSON body. Include request_id when you report a problem.

json

{
  "success": false,
  "error": "Invalid request body: invalid character '}' looking for beginning of object key string",
  "code": "BAD_REQUEST",
  "request_id": "ab966ca3539c9868"
}

ScrapePOST /v1/scrape#

Scrape one URL and get it back in the formats you ask for. Pages are fetched directly; waitFor, actions, or agent load them in a real browser. HTML, PDFs with a text layer, text, JSON, and XML are supported, up to 50 MiB.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links"]
  }'

Body#

Field	Type	Default	Description
`url`	string, required	—	The page to scrape. A URL without a scheme uses HTTPS.
`formats`	array	`["markdown"]`	Outputs to return; see formats.
`prompt`	string	—	Describe the data you want; adds `json` output. Also the Agent Mode task when `agent.prompt` is empty.
`agent`	object	—	Turns on Agent Mode; see Agent Mode.
`onlyMainContent`	boolean	`true`	Drop navigation, banners, footers, and sidebars.
`includeTags`	string[]	`[]`	CSS selectors to keep.
`excludeTags`	string[]	`[]`	CSS selectors to remove.
`actions`	object[]	`[]`	Browser actions to run first; see actions.
`waitFor`	integer (ms)	`0`	Wait after the page loads in the browser, 0–60,000.
`timeout`	integer (ms)	`120000`	Deadline for the whole scrape, including model work; 1,000–300,000. Omit or use `0` for 120 seconds.
`headers`	object	—	Headers to send to the page. Turns off caching.
`maxAge`	integer (ms)	`172800000`	Return a cached result up to this old (2 days). `0` always scrapes fresh.
`storeInCache`	boolean	`true`	Set `false` to not cache this result.

The cache stores both the complete successful response (including LLM output) and the scraped source page. Identical requests return the saved response directly. A different prompt or format reuses the source page and generates new output. Failed scrapes, failed extraction, and timeouts never enter the cache. Entries last up to two days, are limited to 8 MiB each, and can take time to propagate between locations.

Agent Mode checks cached content with the model first and opens a live browser if the task needs more navigation or interaction. Explicit actions and custom headers bypass both caches. Set maxAge: 0 to force a fresh scrape and model call; storeInCache: false permits cache reads but prevents writes. X-Cache: hit indicates a reused response or source page.

For product images, include "formats": ["markdown", "images"]. Extraction receives the discovered gallery URLs along with the page content, even when the visible Markdown shows only one image. Images available only after an interaction may require browser actions. Set "timeout": 120000 for a two-minute deadline.

Formats#

Each entry is a string such as "markdown", or an object with type, plus optional prompt and schema (a JSON Schema) for json.

Format	Returns
`markdown`	`data.markdown`: the page as clean Markdown.
`html`	`data.html`: cleaned HTML.
`rawHtml`	`data.rawHtml`: the page's original HTML.
`links`	`data.links`: every link URL on the page.
`images`	`data.images`: all discovered image URLs, including lazy/responsive images, gallery zoom URLs, product JSON-LD image arrays, and embedded gallery data.
`records`	`data.records`: tables, lists, and repeated cards as rows and columns.
`summary`	`data.summary`: a short summary written by the model.
`json`	`data.json`: structured data shaped by your prompt and schema.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": [{
      "type": "json",
      "prompt": "Extract product names and prices.",
      "schema": {"type": "object", "properties": {"products": {"type": "array", "items": {"type": "object", "properties": {"name": {"type": "string"}, "price": {"type": "string"}}}}}}
    }]
  }'

Actions#

Actions run in the browser, in order, before the page is read. A failed action returns 400 ACTION_FAILED.

`type`	Fields	Does
`wait`	`milliseconds` (1–60,000) or `selector`	Waits for a time or for an element.
`click`	`selector`, optional `all`	Clicks the first match, or every match.
`write`	`text`	Types into the focused element.
`press`	`key`	Presses a key such as `Enter` or `Escape`.
`scroll`	`direction` (`up` / `down`), optional `selector`	Scrolls one screen.
`scrape`	—	Saves the HTML at this point to `data.actions.scrapes`.
`executeJavascript`	`script`	Runs JavaScript; the result goes to `data.actions.javascriptReturns`.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "actions": [{"type": "click", "selector": "#load-more"}, {"type": "wait", "milliseconds": 1000}, {"type": "scrape"}]
  }'

Agent Mode#

Give the agent a goal and it clicks, types, scrolls, and navigates until it finds the answer, then returns your formats.

Field	Type	Default	Description
`agent.prompt`	string	top-level `prompt`	What the agent should find.
`agent.maxSteps`	integer	`10`	Most actions the agent may take, 1–25.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "prompt": "Find the latest article and give its title and date.",
    "agent": {"maxSteps": 10}
  }'

data.agent.steps lists what the agent did. data.agent.finished is false if it ran out of steps.

Response#

json

{
  "success": true,
  "data": {
    "markdown": "# Example Domain\nThis domain is for use in documentation examples.",
    "links": ["https://www.iana.org/domains/example"],
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com",
      "url": "https://example.com/",
      "statusCode": 200,
      "contentType": "text/html",
      "engine": "http",
      "elapsedMs": 123
    }
  }
}

Only the formats you asked for are returned.

Field	Type	Description
`success`	boolean	`true` when the scrape worked.
`data.markdown`	string	Page content as Markdown.
`data.html`	string	Cleaned HTML.
`data.rawHtml`	string	Original HTML.
`data.links`	string[]	Link URLs.
`data.images`	string[]	All discovered, deduplicated image URLs, including gallery variants. The list can include other page images such as logos.
`data.records`	object[]	Each has `type` (`table` or `list`), `columns`, and `rows`.
`data.summary`	string	Model summary.
`data.json`	any	Model output for your prompt or schema.
`data.warning`	string	Set when a format was skipped or input was shortened.
`data.actions`	object	`scrapes` and `javascriptReturns` from your actions.
`data.agent`	object	Agent Mode `steps` and `finished`.
`data.metadata`	object	Always present.
`data.metadata.sourceURL`	string	The URL you sent.
`data.metadata.url`	string	The final URL after redirects.
`data.metadata.statusCode`	integer	The page's HTTP status.
`data.metadata.engine`	string	`http` or `browser`.
`data.metadata.elapsedMs`	integer	Time taken in milliseconds.
`data.metadata.title`	string	Page title.
`data.metadata.description`	string	Page description.
`data.metadata.language`	string	Page language.
`data.metadata.contentType`	string	Content type of the page.
`data.metadata.numPages`	integer	Page count, for PDFs.
`data.metadata.cacheState`	string	`hit` when the complete response or its source page came from the cache.
`data.metadata.cachedAt`	string	When the cached result was saved.

CrawlPOST /v1/crawl#

Start a crawl from one URL. decrawl follows links and scrapes each page in the background; the response returns a crawl ID right away. Poll crawl status for results.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/crawl \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "limit": 10,
    "scrapeOptions": {"formats": ["markdown"]}
  }'

Body#

Field	Type	Default	Description
`url`	string, required	—	Where the crawl starts.
`limit`	integer	`10000`	Most pages to scrape.
`includePaths`	string[]	`[]`	Only crawl paths matching these regular expressions.
`excludePaths`	string[]	`[]`	Skip paths matching these regular expressions.
`regexOnFullURL`	boolean	`false`	Match the path filters against the full URL.
`maxDiscoveryDepth`	integer	unlimited	How many links deep to go from the start page.
`sitemap`	string	`include`	`include` the sitemap, `skip` it, or use `only` the sitemap.
`ignoreQueryParameters`	boolean	`false`	Treat URLs that differ only by query string as one page.
`crawlEntireDomain`	boolean	`false`	Also crawl pages outside the start URL's path.
`allowExternalLinks`	boolean	`false`	Follow links to other sites.
`allowSubdomains`	boolean	`false`	Follow links to subdomains.
`ignoreRobotsTxt`	boolean	`false`	Crawl pages that robots.txt blocks.
`robotsUserAgent`	string	`*`	Which robots.txt rules to follow.
`delay`	number (s)	`0`	Wait between pages, 0–60. Forces one page at a time.
`maxConcurrency`	integer	`10`	Pages scraped at once, 1–50.
`scrapeOptions`	object	Markdown	Scrape options for every page, including `formats`, `prompt`, `actions`, and `agent`.

Response#

json

{
  "success": true,
  "id": "f6445ae4-f73d-42e0-a769-e74a4fe0b92d",
  "url": "https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d"
}

Get crawl statusGET /v1/crawl/{id}#

Returns progress and the pages scraped so far. Crawls run in the background, so poll this endpoint until status is completed, failed, or cancelled. Status checks don't count against your account.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d \
  -H "Authorization: Bearer $API_KEY"

Poll every few seconds until the crawl finishes:

bash

CRAWL_ID=f6445ae4-f73d-42e0-a769-e74a4fe0b92d
while :; do
  STATUS=$(curl -sS "https://api.decrawl.sixteenlabs.space/v1/crawl/$CRAWL_ID" \
    -H "Authorization: Bearer $API_KEY" | jq -r .status)
  echo "$STATUS"
  [ "$STATUS" != "scraping" ] && break
  sleep 3
done

Large results are split into responses of about 10 MiB. When next is not null, request that URL with the same Authorization header, or pass ?skip=N to leave out the first N scraped pages.

json

{
  "success": true,
  "status": "completed",
  "total": 1,
  "completed": 1,
  "creditsUsed": 1,
  "createdAt": "2026-11-01T10:00:00Z",
  "completedAt": "2026-11-01T10:00:02Z",
  "expiresAt": "2026-11-01T11:00:02Z",
  "duration": 2,
  "next": null,
  "data": [
    {
      "markdown": "# Example Domain\nThis domain is for use in documentation examples.",
      "metadata": { "sourceURL": "https://example.com", "url": "https://example.com/", "statusCode": 200, "engine": "http", "elapsedMs": 240 }
    }
  ]
}

Field	Description
`status`	`scraping`, `completed`, `failed`, or `cancelled`.
`total`	Pages found so far.
`completed`	Pages scraped successfully.
`creditsUsed`	Pages scraped successfully.
`createdAt`, `completedAt`	When the crawl started and finished.
`expiresAt`	When the results are deleted. Save them before this.
`duration`	Seconds the crawl has run.
`next`	URL of the next page of results, or `null`.
`data`	Scraped pages, each shaped like a scrape response's `data`.

Get crawl errorsGET /v1/crawl/{id}/errors#

Lists the pages that failed and the URLs robots.txt blocked.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d/errors \
  -H "Authorization: Bearer $API_KEY"

json

{
  "errors": [
    { "id": "1", "timestamp": "2026-11-01T10:00:01Z", "url": "https://example.com/broken", "error": "target returned HTTP 404" }
  ],
  "robotsBlocked": ["https://example.com/private"]
}

Cancel a crawlDELETE /v1/crawl/{id}#

Stops a running crawl. Pages already scraped stay available from crawl status.

bash

curl -sS -X DELETE https://api.decrawl.sixteenlabs.space/v1/crawl/f6445ae4-f73d-42e0-a769-e74a4fe0b92d \
  -H "Authorization: Bearer $API_KEY"

json

{ "status": "cancelled" }

MapPOST /v1/map#

Get a website's URLs fast. Map reads the start page's links and the sitemap; it does not scrape the pages it finds.

bash

curl -sS https://api.decrawl.sixteenlabs.space/v1/map \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "search": "pricing",
    "limit": 100
  }'

Body#

Field	Type	Default	Description
`url`	string, required	—	The website to map.
`search`	string	—	Put URLs containing these words first.
`sitemap`	string	`include`	`include` page links and the sitemap, `skip` the sitemap, or use `only` the sitemap.
`includeSubdomains`	boolean	`true`	Keep URLs on subdomains.
`ignoreQueryParameters`	boolean	`true`	Treat URLs that differ only by query string as one.
`limit`	integer	`5000`	Most URLs to return, 1–100,000.
`timeout`	integer (ms)	`120000`	Deadline for the map.

Response#

json

{
  "success": true,
  "links": [
    { "url": "https://example.com/" },
    { "url": "https://example.com/pricing" }
  ]
}

Field	Description
`success`	`true` when the map worked.
`links`	URLs found, each as an object with `url`.

The API reference.

Base URL#

Authentication#

Response codes#

ScrapePOST /v1/scrape#

Body#

Formats#

Actions#

Agent Mode#

Response#

CrawlPOST /v1/crawl#

Body#

Response#

Get crawl statusGET /v1/crawl/{id}#

Get crawl errorsGET /v1/crawl/{id}/errors#

Cancel a crawlDELETE /v1/crawl/{id}#

MapPOST /v1/map#

Body#

Response#

No matching sections