Deepcrawl (Beta)
API endpoints for initiating and monitoring asynchronous deep crawl tasks. Starting a task costs 20 credits; status checks are free.
A deepcrawl first discovers the site's URLs (its sitemap, or the links on the start page when there is none), then crawls them and packages the pages.
Scope the crawl with include_paths and exclude_paths, and cap it with max_pages:
- A rule without
*is a path prefix:/docsmatches/docsand/docs/api, but not/docsearch. - A rule with
*must match the whole path, and*matches any characters including/:/blog/*matches every page under/blog/. - Exclusion wins over inclusion.
max_pages(1–1000, default 1000) applies after the rules, in discovery order.
Choose One section, capped in the request example selector below for a crawl of at most 20 pages under /docs/. When the task finishes, the status response's stats reports how many URLs were discovered, matched the rules, were left out by max_pages, and were crawled or failed; failed lists the URLs that could not be crawled and why.
By default the pages are packaged as Markdown files in a ZIP behind zipUrl. Set format to "json" (the Pages as JSON example) to get them directly in the completed status response as pages: [{ "url", "title", "content" }], with content in Markdown. resultUrl points to the same JSON file; when the result is larger than 5 MB, pages is omitted and only resultUrl is returned.
Starting a task costs 20 credits, however many pages it crawls. Status checks are free.
bearerAuthAuthorizationBearer <token>application/json- body
url*stringuritype?string"sitemap""all"max_pages?integerMaximum number of pages to crawl, 1–1000; default 1000. Applied after include_paths/exclude_paths, in discovery order.
1 <= value <= 1000include_paths?array<>Only crawl URLs whose path matches one of these rules (up to 20). A rule without * is a path prefix on segment boundaries: "/docs" matches /docs and /docs/api but not /docsearch. A rule with * must match the whole path, and * matches any characters including /, e.g. "/blog/*". Empty or omitted means every discovered URL.
items <= 20exclude_paths?array<>Skip URLs whose path matches one of these rules (up to 20), using the same syntax as include_paths. Exclusion wins over inclusion.
items <= 20format?stringResult format; default "zip". "zip" packages the pages as Markdown files behind zipUrl. "json" returns pages: [{url, title, content}] in the completed status response, plus resultUrl for the same JSON file.
"zip""json"Successful response
application/json- response
taskId*stringstatus*"queued""queued"curl -X POST 'https://api.search1api.com/deepcrawl' \ -H 'Content-Type: application/json' \ -d '{ "url": "https://example.com"}'{ "taskId": "string", "status": "queued"}bearerAuthAuthorizationBearer <token>taskId*string1 <= lengthSuccessful response
application/json- response
taskId*stringstatus?string"queued""waiting""processing""completed""failed""not_found"success?booleanmessage?stringerror?string|nullr2Key?stringzipUrl?|uristats?Page counts for a finished task: discovered links, how many matched the path rules, how many max_pages left out, and how many pages were crawled or failed.
format?string"zip""json"failed?array<>Pages that could not be crawled, with the reason.
resultUrl?|format=json only: the JSON result file, {"pages": [...], "failed": [...]}.
uripages?array<>format=json only: the crawled pages in discovery order. Omitted when the result is larger than 5 MB; download resultUrl instead.
[key: string]?anycurl -X GET 'https://api.search1api.com/deepcrawl/status/string'{ "taskId": "string", "status": "queued", "success": true, "message": "string", "error": "string", "r2Key": "string", "zipUrl": "http://example.com", "stats": { "discovered": 0, "matched": 0, "skipped": 0, "crawled": 0, "failed": 0 }, "format": "zip", "failed": [ { "url": "http://example.com", "error": "string" } ], "resultUrl": "http://example.com", "pages": [ { "url": "http://example.com", "title": "string", "content": "string" } ]}