# Convert a web page

> POST /v1/fetch opens a web page in a browser, runs its JavaScript and returns the page as Markdown, for a fixed price per fetch.

`POST /v1/fetch` opens a page in a headless browser (Cloudflare Browser Run), waits until the network is quiet (`networkidle2`, at most 30 seconds), and converts the page as it is after its JavaScript has run. It costs 5 credits per fetch ($1.00 per 1,000 fetches), whatever the length of the page. For a PDF link, use [`/v1/fetch-pdf`](https://md.tlelabs.com/docs/fetch-pdf/).

```sh
curl -X POST https://api.md.tlelabs.com/v1/fetch \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/article"}'
```

## Request

`Content-Type: application/json`, a JSON object of at most 2 MiB:

| Field             | Required | Value                                                                                                            |
| ----------------- | -------- | ---------------------------------------------------------------------------------------------------------------- |
| `url`             | yes      | An `http` or `https` URL without user name or password, on a public host (not `localhost` or a private address). |
| `preserve_images` | no       | `true` (default) or `false`, as for [`/v1/convert`](https://md.tlelabs.com/docs/convert/).                       |

A field of the wrong type gets `400 invalid_request` with `details.field`; a URL that is refused gets `400 invalid_url`.

## What happens

1. 5 credits are reserved before the page is opened; with fewer available you get `402 insufficient_credits` and nothing is fetched.
2. The browser opens the page without cookies or a login. It identifies itself as a bot, so a site that blocks bots may show a challenge page, and that page is what you get.
3. A page that answers an HTTP error (`404`, `500`, …) gets `422 fetch_failed` with `details.http_status`.
4. A page that is not HTML gets `415 unsupported_content`; a PDF names `/v1/fetch-pdf`. A page without a `Content-Type` is read as HTML.
5. Scripts and styles are removed. Relative links and image sources are made absolute from the final URL (after redirects) and the page’s `<base>`; `#anchors`, `mailto:` and `data:` values are kept as written.
6. The page is converted like an HTML file sent to `/v1/convert` with `ocr=off`.

Any failure after the reservation releases it: a failed fetch is not charged.

## Response

`200 OK`:

```json
{
  "markdown": "# Article title\n\n…",
  "format": "web",
  "usage": { "pages": 3, "ocr_pages": 0, "credits_charged": 5, "credits_remaining": 495 },
  "metadata": {
    "url": "https://example.com/article",
    "final_url": "https://example.com/article",
    "http_status": 200,
    "byte_size": 81234,
    "duration_ms": 5400
  },
  "warnings": []
}
```

| Field                  | Meaning                                                                                                                                              |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `usage.pages`          | Pages of the converted HTML ([how pages are counted](https://md.tlelabs.com/docs/formats-and-pricing/#pages)); the price does not depend on it.      |
| `metadata.final_url`   | The page URL after redirects; `null` when the browser did not report it, in which case relative links are kept as in the page and a warning says so. |
| `metadata.http_status` | HTTP status of the page.                                                                                                                             |
| `metadata.byte_size`   | UTF-8 size of the page HTML after rendering, before scripts and styles are removed.                                                                  |

With `Accept: text/markdown` the body is the Markdown, with the headers `X-Credits-Charged`, `X-Credits-Remaining`, `X-Http-Status`, `X-Duration-Ms` and `X-Final-Url` (when known).

## Errors

| Status | Code                   | When                                                                                     |
| ------ | ---------------------- | ---------------------------------------------------------------------------------------- |
| 400    | `invalid_request`      | Not `application/json`, invalid JSON, missing `url`, or a field of the wrong type.       |
| 400    | `invalid_url`          | The URL is not `http`/`https`, has credentials, or points to a local or private address. |
| 402    | `insufficient_credits` | Fewer than 5 credits available.                                                          |
| 413    | `request_too_large`    | JSON body over 2 MiB.                                                                    |
| 413    | `file_too_large`       | The rendered page is larger than the HTML converter accepts.                             |
| 415    | `unsupported_content`  | The page is not HTML (a PDF names `/v1/fetch-pdf`).                                      |
| 422    | `fetch_failed`         | The page answered an HTTP error (`details.http_status`).                                 |
| 502    | `fetch_failed`         | The browser could not open the page (DNS, connection, refused URL).                      |
| 503    | `service_unavailable`  | Web fetch is busy (retry after `Retry-After`) or unavailable.                            |
| 504    | `fetch_timeout`        | The page did not finish loading within 30 seconds.                                       |

For all of them the reservation is released: an error is not charged. Sending the same URL again is a new fetch and is charged again.
