PyPI project pages to Markdown
PyPI scraper API for project pages
Send a PyPI project URL to the Crawl endpoint and get back structured Markdown — the project description plus license, requires-python, and the requirements list. Pinned-release URLs included.
Endpoint — bash
What you can crawl
The adapter reads pypi.org/pypi/<name>/json — the project README and metadata without the release-history sidebar the web page wraps it in.
1
Project pages
The description as Markdown — plus license, requires-python, and release date in metadata.
2
Pinned releases
/project/<name>/<version>/ URLs return that exact release — its description and its requirements.
3
Requirements as a list
Install requirements render as a clean list — extras and environment markers preserved.
Useful for
Dependency research / Package discovery / Version auditing
What goes in, what comes back
Two URL types. Left: the public page as PyPI actually serves it — a real anonymous capture. Right: the complete results.content the API returns, byte for byte.
Project page
pypi.org/project/fastapi/What PyPI serves
472 KB rendered HTML — the description wrapped in release history, project links, and stats sidebars
What the API returns
markdown
Project card + description — 24 KB · fastapi 0.141.1 · MIT · 32 requirements
Pinned release
pypi.org/project/django/5.2/What PyPI serves
581 KB rendered HTML — the same layout pinned to release 5.2
What the API returns
markdown
Exact release — 2.7 KB · Django 5.2 · requires-python >=3.10 · that release’s requirements
What is covered, by URL pattern
PyPI support is measured per URL pattern, not claimed per domain. Every row below is verified against the live crawler.
| URL pattern | Status | What you get |
|---|---|---|
/project/<name>/ | Supported | Latest release — description as Markdown plus license, requires-python, release date, and the requirements list. |
/project/<name>/<version>/ | Supported | A pinned release returns that exact version — its description and its requirements, useful for audits and diffs. |
Project cards without the sidebar noise
PyPI web pages wrap the project description in a release-history sidebar — tens of thousands of characters where half are version links. Point the Crawl endpoint at a project URL and the request routes to a dedicated adapter that reads pypi.org/pypi/<name>/json — just the README and the metadata record, nothing else.
JSON API, not the sidebar
The adapter reads pypi.org’s JSON API — the project description and metadata record without the version-link chrome around it.
Card plus description
The Markdown starts with the project card — name, version, license, requires-python — then the full description below.
Requirements as a list
Install requirements render as a clean list — extras and environment markers survive intact.
Verified per URL pattern
Every row in the matrix is reproduced from a live crawl of a real URL — support is measured per pattern, not claimed per domain.
What is a PyPI scraper API?
A PyPI scraper API is an HTTP interface that takes a PyPI project URL and returns the project page in a structured, model-ready format. The web page embeds the description in navigation and a release-history sidebar; the substance is the project record — README, license, requires-python, requirements. Search1API’s Crawl endpoint reads pypi.org’s JSON API directly, so project and pinned-release URLs return their card as Markdown without the sidebar noise.
Implementation path
Typical workflow
Send a PyPI project URL to the Crawl endpoint and get back structured Markdown — the project description plus license, requires-python, and the requirements list. Pinned-release URLs included.
1
POST a PyPI project URL to the Crawl endpoint — no account needed.
2
The URL is matched against the pattern rules and routed to the PyPI adapter.
3
The adapter reads pypi.org/pypi/<name>/json — plain HTTP, no challenge.
4
Projects return as Markdown plus results.metadata — version, license, requiresPython.
Where teams use it
Vet dependencies before adopting — description, license, and requires-python in one call.
Audit versions — pinned-release URLs return a specific release’s requirements for diffing.
Feed package discovery — the same card shape across every project on the index.
Pair with the npm adapter — one pipeline shape for both package registries.
FAQ
Do I need a PyPI account?
No. The adapter reads the public pypi.org JSON API — the same record the site renders. No token or session anywhere in the read path.
Which PyPI URLs are supported?
Project pages (/project/<name>/) and pinned releases (/project/<name>/<version>/). Search and listing pages are not on the dedicated path.
What is the difference from the web page?
The web page wraps the description in a release-history sidebar — tens of thousands of characters of version links. The JSON API returns just the README and the metadata record.
Do pinned versions really differ?
Yes — a pinned release returns that version’s own description and requirements list, including the requires-python bound and extras as they were at that release.
What metadata comes back?
platform, package, version, license, requiresPython, and descriptionContentType — enough to route or filter projects without reading the description.
Is scraping PyPI legal?
We only fetch what PyPI serves anonymously through its public JSON API — no login, no session sharing, no circumvention of access controls. Whether a specific use complies with applicable law and PyPI’s terms depends on your jurisdiction and use case, so confirm that with your own counsel.