Convert a web page
POST /v1/fetch opens a page in a headless browser (Cloudflare Browser Run), waits until the network is quiet (networkidle2, at most 30 seconds), and converts the page as it is after its JavaScript has run. It costs 5 credits per fetch ($1.00 per 1,000 fetches), whatever the length of the page. For a PDF link, use /v1/fetch-pdf.
curl -X POST https://api.md.tlelabs.com/v1/fetch \ -H "Authorization: Bearer <YOUR_API_KEY>" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com/article"}'Request
Section titled “Request”Content-Type: application/json, a JSON object of at most 2 MiB:
| Field | Required | Value |
|---|---|---|
url |
yes | An http or https URL without user name or password, on a public host (not localhost or a private address). |
preserve_images |
no | true (default) or false, as for /v1/convert. |
A field of the wrong type gets 400 invalid_request with details.field; a URL that is refused gets 400 invalid_url.
What happens
Section titled “What happens”- 5 credits are reserved before the page is opened; with fewer available you get
402 insufficient_creditsand nothing is fetched. - The browser opens the page without cookies or a login. It identifies itself as a bot, so a site that blocks bots may show a challenge page, and that page is what you get.
- A page that answers an HTTP error (
404,500, …) gets422 fetch_failedwithdetails.http_status. - A page that is not HTML gets
415 unsupported_content; a PDF names/v1/fetch-pdf. A page without aContent-Typeis read as HTML. - Scripts and styles are removed. Relative links and image sources are made absolute from the final URL (after redirects) and the page’s
<base>;#anchors,mailto:anddata:values are kept as written. - The page is converted like an HTML file sent to
/v1/convertwithocr=off.
Any failure after the reservation releases it: a failed fetch is not charged.
Response
Section titled “Response”200 OK:
{ "markdown": "# Article title\n\n…", "format": "web", "usage": { "pages": 3, "ocr_pages": 0, "credits_charged": 5, "credits_remaining": 495 }, "metadata": { "url": "https://example.com/article", "final_url": "https://example.com/article", "http_status": 200, "byte_size": 81234, "duration_ms": 5400 }, "warnings": []}| Field | Meaning |
|---|---|
usage.pages |
Pages of the converted HTML (how pages are counted); the price does not depend on it. |
metadata.final_url |
The page URL after redirects; null when the browser did not report it, in which case relative links are kept as in the page and a warning says so. |
metadata.http_status |
HTTP status of the page. |
metadata.byte_size |
UTF-8 size of the page HTML after rendering, before scripts and styles are removed. |
With Accept: text/markdown the body is the Markdown, with the headers X-Credits-Charged, X-Credits-Remaining, X-Http-Status, X-Duration-Ms and X-Final-Url (when known).
Errors
Section titled “Errors”| Status | Code | When |
|---|---|---|
| 400 | invalid_request |
Not application/json, invalid JSON, missing url, or a field of the wrong type. |
| 400 | invalid_url |
The URL is not http/https, has credentials, or points to a local or private address. |
| 402 | insufficient_credits |
Fewer than 5 credits available. |
| 413 | request_too_large |
JSON body over 2 MiB. |
| 413 | file_too_large |
The rendered page is larger than the HTML converter accepts. |
| 415 | unsupported_content |
The page is not HTML (a PDF names /v1/fetch-pdf). |
| 422 | fetch_failed |
The page answered an HTTP error (details.http_status). |
| 502 | fetch_failed |
The browser could not open the page (DNS, connection, refused URL). |
| 503 | service_unavailable |
Web fetch is busy (retry after Retry-After) or unavailable. |
| 504 | fetch_timeout |
The page did not finish loading within 30 seconds. |
For all of them the reservation is released: an error is not charged. Sending the same URL again is a new fetch and is charged again.