# Deepcrawl (Beta)

> API endpoints for initiating and monitoring asynchronous deep crawl tasks. Starting a task costs 20 credits; status checks are free.

Canonical URL: https://s1.dev/docs/advanced/deepcrawl

A deepcrawl first discovers the site's URLs (its sitemap, or the links on the start page when there is none), then crawls them and packages the pages.

Scope the crawl with `include_paths` and `exclude_paths`, and cap it with `max_pages`:

- A rule without `*` is a path prefix: `/docs` matches `/docs` and `/docs/api`, but not `/docsearch`.
- A rule with `*` must match the whole path, and `*` matches any characters including `/`: `/blog/*` matches every page under `/blog/`.
- Exclusion wins over inclusion. `max_pages` (1–1000, default 1000) applies after the rules, in discovery order.

Choose **One section, capped** in the request example selector below for a crawl of at most 20 pages under `/docs/`. When the task finishes, the status response's `stats` reports how many URLs were discovered, matched the rules, were left out by `max_pages`, and were crawled or failed; `failed` lists the URLs that could not be crawled and why.

By default the pages are packaged as Markdown files in a ZIP behind `zipUrl`. Set `format` to `"json"` (the **Pages as JSON** example) to get them directly in the completed status response as `pages: [{ "url", "title", "content" }]`, with `content` in Markdown. `resultUrl` points to the same JSON file; when the result is larger than 5 MB, `pages` is omitted and only `resultUrl` is returned.

> **info:** Starting a task costs 20 credits, however many pages it crawls. Status checks are free.

Complete machine-readable schema: https://api.search1api.com/openapi.json