Use cases

Know when a page actually changed

Compare the text a reader sees, not the HTML around it — so a new ad slot or a rotated CSS hash is not a change.

By Roger Campos · Last updated: September 2026

TL;DR

To monitor a web page for changes, fetch its readable content on a schedule, compare it with the last version, and diff only when it differs. URLpipe's /markdown is a good input for that: it renders the page, drops the chrome, and gives the same Markdown for the same page every time. Send max_age: 0 so every check is a fresh fetch.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The job

Why comparing HTML is the wrong test

Competitor pricing pages, a supplier's terms, a government notice, the changelog of an API you depend on: you want to know when the words change. Comparing raw HTML tells you something changed on every check — a build hash in a script tag, a CSRF token, a rotated ad. Comparing a screenshot flags a carousel. What you want to compare is the text a reader sees.

/markdown gives you that. The conversion is a deterministic walk of the rendered DOM — no model, so no paraphrase drift between runs — and the same page produces the same Markdown. Hash it; if the hash matches the last one, nothing happened. If it moved, diff the two texts and send the diff.

Turn any URL into clean Markdown

Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.

Try it free

Watching one value rather than the whole page

Sometimes the page changes all the time and only one thing matters — a price, a version number, a stock level. Fetch /html instead: it returns the rendered DOM after the page's scripts have run, so you can pull that one element out with a CSS selector in your own code and compare just its text. It costs the same 1 credit as the Markdown.

Settings

The three options that matter

max_age: 0 — skip the cache and fetch the page as it is now. Without it, a check inside seven days of the last one is served from the cache: free, but it cannot show a change. A max_age: 0 request also never shares a run already in flight, because that run may have fetched its page before you asked.
page_options — take out what changes without meaning anything: block_ads, block_cookie_banners, and remove_selectors for a "latest posts" box or a live counter. What they remove is gone from the Markdown, not hidden in it. See page options.
report_to and labels — run the batch async and have each result delivered with your watch id, so the comparison happens in a webhook rather than in a loop that holds connections open.

Flow

A check, end to end

1
Your scheduler — cron, a queue, a GitHub Action — sends one request per watched URL, async, with max_age: 0 and a watch label.
2
The webhook delivers the Markdown. Normalise whitespace, hash it, and compare with the stored hash for that watch.
3
Same hash: record the check and stop. Different: store the new version, diff it against the old, and notify.
4
A failed delivery is retried, and can be re-sent by hand from the dashboard; the result also stays at GET /result/:token for 30 days.

Check a page, and compare it when it arrives

import difflib, hashlib, json, os, requests

def check(watch_id: str, url: str):
    requests.post("https://urlpipe.dev/markdown",
        headers={"Authorization": f"Bearer {os.environ['URLPIPE_API_KEY']}"},
        json={
            "url": url,
            "max_age": 0,
            "page_options": {"block_ads": True, "block_cookie_banners": True,
                             "remove_selectors": [".latest-posts"]},
            "report_to": "https://your-app.com/webhooks/urlpipe",
            "labels": {"watch": watch_id},
        }, timeout=30).raise_for_status()

def on_delivery(body: bytes):          # after verifying the signature
    d = json.loads(body)
    if not d["success"]:
        return
    text = "\n".join(line.rstrip() for line in d["result"].splitlines())
    digest = hashlib.sha256(text.encode()).hexdigest()
    watch = db.get(d["labels"]["watch"])
    if digest != watch.digest:
        diff = difflib.unified_diff(watch.text.splitlines(), text.splitlines(),
                                    "before", "after", lineterm="")
        notify(watch, "\n".join(diff))
        db.save(watch.id, digest=digest, text=text)

import { createHash } from "node:crypto";

export async function check(watchId, url) {
  const res = await fetch("https://urlpipe.dev/markdown", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.URLPIPE_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url,
      max_age: 0,
      page_options: { block_ads: true, block_cookie_banners: true,
                      remove_selectors: [".latest-posts"] },
      report_to: "https://your-app.com/webhooks/urlpipe",
      labels: { watch: watchId },
    }),
  });
  if (!res.ok) throw new Error(`URLpipe ${res.status}`);
}

export const fingerprint = (markdown) =>
  createHash("sha256")
    .update(markdown.split("\n").map((l) => l.trimEnd()).join("\n"))
    .digest("hex");

curl -X POST https://urlpipe.dev/markdown \
  -H "Authorization: Bearer $URLPIPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor.example/pricing",
    "max_age": 0,
    "page_options": {"block_ads": true, "block_cookie_banners": true},
    "report_to": "https://your-app.com/webhooks/urlpipe",
    "labels": {"watch": "competitor-pricing"}
  }'

Cadence

Choosing a schedule

Check each page as often as it is worth knowing about, not as often as you can. A reasonable starting point:

What you watch	Check	max_age
Competitor pricing and plans	Daily	<code>0</code>
Terms, privacy policies, legal notices	Daily or weekly	<code>0</code>
An upstream API's changelog or status page	Hourly	<code>0</code>
Documentation you ingest into a RAG index	Weekly	<code>"7 days"</code> (the default)
Pages where "changed today" is enough	Every few hours	<code>"6 hours"</code>

Stagger the checks through the hour rather than firing them all at :00. The API allows 60 requests a minute and 15 per 10 seconds per project, and each async request holds one of your plan's parallel slots from the moment it is accepted until it finishes.

A page that fails to load is not a change. Record failures separately — the delivery's success is false and error says why — and only alert on content after a successful fetch.

Credits

What it costs

Every check with max_age: 0 does real work, so the schedule is the bill. At list price, 500 pages checked once a day:

What	A month	Credits each	Credits
500 pages, checked daily	15,000	1	15,000

15,000 credits; the cheapest plan that covers it is Starter: $19 a month for 20,000 credits. Hourly checks multiply quickly, so keep them for the few pages that need them:

What	A month	Credits each	Credits
50 pages, checked hourly	36,000	1	36,000

36,000 credits; the cheapest way to buy it is Starter ($19, 20,000 credits) plus 16,000 credits of overage at $1.50 per 1,000 credits — $43.00 a month. If "changed in the last few hours" is good enough, send max_age: "6 hours" instead of 0: any check inside that window is served from the cache for free.

Limits

What URLpipe does not do for change monitoring

Schedule, store or diff. It fetches the page. Keeping the last version and deciding what changed is your code — a dozen lines, above.
Watch a whole site. One URL per request, and no discovery of new pages. List what you watch.
See what the Markdown drops. Images, styling and layout are not in the text. For visual changes, compare screenshots instead.
Ignore noise it cannot see. A rotating testimonial or a live price ticker is a real text change. Remove it with remove_selectors.

Endpoints

The endpoints on this page

/markdown API docs /html API docs

FAQ

Frequently asked questions

Why Markdown rather than HTML for change detection?

HTML changes on almost every load — build hashes, tokens, ad slots. Markdown keeps only the readable content, and URLpipe's conversion is deterministic, so identical text gives identical output and a hash comparison is meaningful.

Does max_age=0 cost more?

No more than any other fetch: 1 credit. It simply never uses the cache, so every check does real work and is billed. A failed fetch is still free.

Can I get notified by webhook when a page changes?

URLpipe delivers every check's result to your webhook, signed with HMAC-SHA256. Deciding whether it changed — comparing with the last version — happens in your handler.

How do I ignore parts of the page that always change?

Use page_options.remove_selectors with CSS selectors for those elements, plus block_ads and block_cookie_banners. Removed elements are absent from the Markdown entirely.

Keep reading

page_options: clean up the page before anything is read What stops a headless browser in 2026 Cursor Each project chooses whether to follow robots.txt Pacing a site that can't keep up URL features inside a multi-tenant app All of use cases

Build it on the free plan.

Free plan, no card. Confirm your email and your API key is live — you'll be making real requests in minutes.

Start free Email the team