GitHub scraper API for repos and files
Send a public GitHub URL to the Crawl endpoint and get back clean Markdown — repo pages return the README, file views return fenced source, raw URLs return the file as-is. No GitHub account needed.
Endpoint — bash
A dedicated converter maps each public page shape to the artifact that matters — the document, not the chrome.
The repo page returns its README as clean Markdown — what a reader came for, without the file tree and nav.
blob URLs return the file body in a fenced code block — real source, syntax fence included.
raw.githubusercontent.com returns the file verbatim — Markdown stays Markdown.
Useful for
README ingestion / Docs corpora / Dependency research
What goes in, what comes back
Three URL types. Left: the public page as GitHub actually serves it to anonymous visitors — a real capture through the same egress the crawler uses. Right: the complete results.content the API returns, byte for byte.
github.com/anthropics/claude-codeWhat GitHub serves
335 KB of HTML — file tree, sidebar modules, nav chrome
What the API returns
markdown
The README as Markdown — 2.8 KB, ready to index
github.com/facebook/react/blob/main/package.jsonWhat GitHub serves
351 KB of HTML — line numbers, blame bar, edit buttons
What the API returns
markdown
The file in a fenced code block — 8.6 KB of source
raw.githubusercontent.com/anthropics/claude-code/main/README.mdWhat GitHub serves
3 KB — the raw file, already plain text
What the API returns
markdown
Verbatim — 2.8 KB, byte-identical Markdown
What is covered, by URL pattern
GitHub support is measured per URL pattern, not claimed per domain. Every row below is verified against the live crawler.
| URL pattern | Status | What you get |
|---|---|---|
/<owner>/<repo> | Supported | The repository’s README as clean Markdown — the document, not the page chrome. |
/<owner>/<repo>/blob/<ref>/<path> | Supported | The file body in a fenced code block — real source text, not rendered HTML. |
raw.githubusercontent.com/* · gist raw | Supported | Raw file content returned verbatim — already text, stays text. |
GitHub content without the page chrome
Point the Crawl endpoint at a GitHub URL and the request is routed to a converter that knows what the page is for. A repo page returns its README; a file view returns the file; a raw URL returns bytes as served. The nav, file tree, and sidebar chrome never reach your pipeline.
The artifact, not the page
Repo pages exist to show files — the converter returns the README a visitor actually reads, minus the chrome around it.
Source stays source
Blob views come back as fenced code blocks and raw URLs come back verbatim — indentation and formatting survive the round trip.
Honest boundaries
Issues, PRs, and profile pages are not claimed — where the dedicated path has no clean answer, the matrix simply does not list the pattern.
Verified per URL pattern
Every row in the matrix is reproduced from a live crawl of a real URL — support is measured per pattern, not claimed per domain.
What is a GitHub scraper API?
A GitHub scraper API is an HTTP interface that takes a GitHub URL and returns the page’s real content in a structured, model-ready format. GitHub’s pages are heavy — a repo view is a file tree, a sidebar, and a nav stack around one README. Search1API’s Crawl endpoint uses a converter that maps each URL shape to its artifact: README for repos, fenced source for file views, verbatim text for raw files. The result is content you can index or feed to a model directly, without writing per-site cleanup rules.
Typical workflow
Send a public GitHub URL to the Crawl endpoint and get back clean Markdown — repo pages return the README, file views return fenced source, raw URLs return the file as-is. No GitHub account needed.
POST a GitHub URL to the Crawl endpoint — no GitHub account needed.
The URL is matched against the pattern rules and routed to the GitHub converter.
Repo pages return README Markdown; blob views return fenced source; raw URLs return the file verbatim.
Patterns without a clean answer are not claimed — the matrix is the boundary, verified on live requests.
Where teams use it
Ingest READMEs across a dependency tree or an org’s public repos for docs search.
Pull source files into code-aware pipelines as fenced, syntax-marked blocks.
Track upstream projects — a repo URL returns the current README on every call.
Feed raw config and docs files straight into retrieval without HTML cleanup.
FAQ
Do I need a GitHub account or token?
No — everything here reads public pages an anonymous visitor can open. Private repos and anything behind login are simply out of scope.
Which GitHub URLs are supported?
Repository roots, blob file views, and raw.githubusercontent.com URLs. Issues, pull requests, releases, and profile pages are not part of the verified matrix.
What does a repo page return?
The README as clean Markdown — the artifact a reader came for. The file listing, sidebar, and navigation chrome are stripped by the dedicated converter.
How is this different from the GitHub API?
The REST API needs a token for useful rate limits and returns JSON you still have to render or fetch file-by-file. This path takes the URL you already have and returns the readable artifact in one call — Markdown in, Markdown out.
Does it handle monorepos and large files?
Blob views return the file body regardless of repo layout. Very large files follow the same content-size limits as every other Crawl response.
Is scraping GitHub legal?
We only fetch content GitHub serves to anonymous visitors — no login, no session sharing, no circumvention of access controls. Whether a specific use complies with applicable law and GitHub’s terms depends on your jurisdiction and use case, so confirm that with your own counsel.