Skip to main content

Confirm

Are you sure?

Java · Code recipe

Convert a web page to Markdown in Java

Turn any URL into Markdown you can hand to a model, in Java 17+ with java.net.http.HttpClient. Every program on this page runs as it stands — each one was run against a stub of the API before it was published.

By · Last updated: September 2026

TL;DR

To convert a web page to Markdown in Java, POST its URL to https://urlpipe.dev/markdown with sync set to true; the response body is the Markdown, read with HttpResponse.BodyHandlers.ofString(). The page is rendered in real Chrome first, then converted by a deterministic walk over the DOM, not a model, for 1 credit a page.

Free plan, no credit card. 1,000 credits a month.

One POST to /markdown renders the page in real Chrome, then walks the rendered DOM and writes the main content as Markdown, with navigation, footers and boilerplate left out. No model is involved, so the same page gives the same Markdown every time, in about 20 ms after the page has loaded.

In Java the response is plain text, read with HttpResponse.BodyHandlers.ofString() — there is no JSON envelope to unwrap. Measured on 33 pages, the output covered 87.2% of the text a visitor sees, and 11.0% of it was text the visitor never saw; the guide to Markdown for LLMs has the comparison.

Setup

Before you start

Java 17 or later: java.net.http is in the JDK. The async and error-handling programs read JSON, and the JDK has no parser, so those use Gson — one jar, no dependencies of its own. In a Maven or Gradle build add com.google.code.gson:gson:2.13.1; to try a file on its own, download the jar next to it.

Terminal
java -version   # 17 or later
curl -sSLO https://repo1.maven.org/maven2/com/google/code/gson/gson/2.13.1/gson-2.13.1.jar
export URLPIPE_API_KEY="your_api_key"

The request

Print a page as Markdown

"sync": true keeps the request open until the result is ready. BodyHandlers.ofString() reads the whole body as text, decoded with the charset the response names. HttpClient does not throw on a 4xx or 5xx, so check statusCode(). The body is the Markdown itself, so the String is the whole job.

HtmlToMarkdown.java
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class HtmlToMarkdown {
    public static void main(String[] args) throws Exception {
        var request = HttpRequest.newBuilder(URI.create("https://urlpipe.dev/markdown"))
            .header("Authorization", "Bearer " + System.getenv("URLPIPE_API_KEY"))
            .header("Content-Type", "application/json")
            // A sync call can take up to 60 s; HttpClient would otherwise wait forever.
            .timeout(Duration.ofSeconds(90))
            .POST(HttpRequest.BodyPublishers.ofString("""
                {"url": "https://example.com", "sync": true}
                """))
            .build();

        var response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        if (response.statusCode() != 200) {
            System.err.println("URLpipe answered " + response.statusCode() + ": " + response.body());
            System.exit(1);
        }
        String body = response.body();

        // Plain text: pipe it into a file, a chunker or a prompt.
        System.out.println(body);
    }
}

Run it: java HtmlToMarkdown.java

Async

The async variant: a token, a webhook and a poll

Leave out sync and the answer is a token, straight away. One HttpClient is shared by every call — it holds the connection pool, so build it once — and a switch expression covers the answers GET /result/:token can give.

HtmlToMarkdownAsync.java
import com.google.gson.JsonParser;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class HtmlToMarkdownAsync {
    static final String API = "https://urlpipe.dev";
    static final HttpClient CLIENT = HttpClient.newHttpClient();

    static HttpResponse<String> send(HttpRequest.Builder builder) throws Exception {
        var request = builder
            .header("Authorization", "Bearer " + System.getenv("URLPIPE_API_KEY"))
            .timeout(Duration.ofSeconds(30))
            .build();
        return CLIENT.send(request, HttpResponse.BodyHandlers.ofString());
    }

    static void fail(String message) {
        System.err.println(message);
        System.exit(1);
    }

    public static void main(String[] args) throws Exception {
        // No "sync": the request is accepted at once and the work carries on without you.
        var accepted = send(HttpRequest.newBuilder(URI.create(API + "/markdown"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString("""
                {
                  "url": "https://example.com",
                  "report_to": "https://your-app.com/webhooks/urlpipe",
                  "labels": {"customer": "acme"}
                }
                """)));
        if (accepted.statusCode() != 200) {
            fail("URLpipe answered " + accepted.statusCode() + ": " + accepted.body());
        }
        String token = JsonParser.parseString(accepted.body()).getAsJsonObject().get("token").getAsString();
        System.out.println("Accepted " + token);

        // The result is POSTed to report_to when it is ready. Polling by token is the
        // other way to collect it: no endpoint needed, and a backup for the webhook.
        HttpResponse<String> response = null;
        for (int attempt = 0; attempt < 60; attempt++) {
            response = send(HttpRequest.newBuilder(URI.create(API + "/result/" + token)).GET());
            if (response.statusCode() != 202) { // 202 means still processing
                break;
            }
            Thread.sleep(2000);
        }
        switch (response.statusCode()) {
            case 200 -> { }
            case 202 -> fail("Still processing after two minutes; try the token again later.");
            case 422 -> fail("The analysis failed: "
                + JsonParser.parseString(response.body()).getAsJsonObject().get("error").getAsString());
            case 410 -> fail("The result is past the 30-day window; send the request again.");
            default -> fail("URLpipe answered " + response.statusCode() + ": " + response.body());
        }
        String body = response.body();

        // Plain text: pipe it into a file, a chunker or a prompt.
        System.out.println(body);
    }
}

Run it: java -cp gson-2.13.1.jar HtmlToMarkdownAsync.java

Errors

Handle errors and retries

A checked URLpipeException makes the caller decide what a refusal means. firstValueAsLong reads Retry-After without a parse of your own, and a body that is not JSON (a 401 answers in plain text) falls back to an empty object.

HtmlToMarkdownErrors.java
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class HtmlToMarkdownErrors {
    static class URLpipeException extends Exception {
        URLpipeException(String message) {
            super(message);
        }
    }

    static final HttpClient CLIENT = HttpClient.newHttpClient();

    // POST a sync request and return the result body; retry the two 429s that clear by themselves.
    static String urlpipe(String path, String payload) throws Exception {
        final int attempts = 5;
        for (int attempt = 0; attempt < attempts; attempt++) {
            var request = HttpRequest.newBuilder(URI.create("https://urlpipe.dev" + path))
                .header("Authorization", "Bearer " + System.getenv("URLPIPE_API_KEY"))
                .header("Content-Type", "application/json")
                .timeout(Duration.ofSeconds(90))
                .POST(HttpRequest.BodyPublishers.ofString(payload))
                .build();
            var response = CLIENT.send(request, HttpResponse.BodyHandlers.ofString());
            int status = response.statusCode();
            if (status == 200) {
                return response.body();
            }
            if (status == 401) {
                throw new URLpipeException("401: the API key is missing or wrong. Check URLPIPE_API_KEY.");
            }

            JsonObject error;
            try {
                error = JsonParser.parseString(response.body()).getAsJsonObject();
            } catch (RuntimeException notJson) {
                error = new JsonObject();
            }
            String code = error.has("error") ? error.get("error").getAsString() : "";
            String detail = error.has("message") ? code + ": " + error.get("message").getAsString() : code;

            if (status == 429 && code.equals("rate_limited")) {
                // Sending too fast: Retry-After says how long the window has left.
                Thread.sleep(response.headers().firstValueAsLong("Retry-After").orElse(1) * 1000);
            } else if (status == 429 && code.equals("concurrency_limit")) {
                // Every parallel slot on your plan is busy with your own requests.
                Thread.sleep((1L << attempt) * 1000);
            } else if (status == 504) {
                // Still running on our side; the token collects it from GET /result/:token.
                throw new URLpipeException(
                    "504 processing_timeout: collect it later with token " + error.get("token").getAsString());
            } else {
                // 403 email_unverified, 422 (a bad parameter, or a page that would not load),
                // 429 quota_exceeded: sending the same request again gets the same answer.
                throw new URLpipeException(status + ": " + detail);
            }
        }
        throw new URLpipeException("429: still refused after " + attempts + " attempts");
    }

    public static void main(String[] args) throws Exception {
        String body;
        try {
            body = urlpipe("/markdown", """
                {"url": "https://example.com", "sync": true}
                """);
        } catch (URLpipeException e) {
            System.err.println("URLpipe: " + e.getMessage());
            System.exit(1);
            return;
        }

        // Plain text: pipe it into a file, a chunker or a prompt.
        System.out.println(body);
    }
}

Run it: java -cp gson-2.13.1.jar HtmlToMarkdownErrors.java

Details

What to know about /markdown

  • It reads no CSS, so text hidden only by a stylesheet can end up in the Markdown.
  • Pages over 10 MB of HTML are refused with The page is too big to be processed.
  • page_options.remove_selectors drops elements before conversion when a site's chrome survives the boilerplate rules.
  • A repeat request inside max_age (7 days by default) is served from the cache and costs nothing.

Other languages

Convert a web page to Markdown in another language

More Java: every Java recipe

FAQ

Frequently asked questions

Does the Markdown conversion use an LLM?
No. It is a deterministic walk over the rendered DOM, so it is fast, costs 1 credit a page, and gives the same output for the same page. Only /meta, /summarize and /keywords call a model.
Does it work on pages built with JavaScript?
Yes. The page is loaded in real Chrome and its scripts run before the DOM is read, so React, Vue and other client-rendered pages convert like static ones.
How do I keep the Markdown for later?
Write the body to a .md file, or rely on the cache: the same URL within max_age (7 days by default) is answered from storage for free, and any result can be fetched again by its token for 30 days.
Do I need an SDK to call URLpipe from Java?
No SDK. HttpClient is in the JDK; the programs that read JSON add Gson, one jar, because the JDK has no parser.

Get a key and run it.

Free plan, no card. Paste your key into URLPIPE_API_KEY and every program on this page runs as it is.