Guide
Why your AI agent reads empty pages and how to fix it
You ask an agent to read a page and it answers from a "please enable JavaScript" notice. We fetched three real client-rendered pages the way the reference MCP fetch server does, and the way a browser does, and counted what each one got.
By Roger Campos · Last updated: September 2026
TL;DR
Most agent web tools, including the reference MCP fetch server, make one plain HTTP GET and convert the HTML they get back. On a client-rendered page that HTML is a shell: the content is built later by JavaScript. On three real pages, the fetch server returned an error twice and a browser-compatibility notice once, where a browser showed 842 to 2,201 words. The fix is to render the page before reading it.
Free plan, no credit card. 1,000 credits a month.
The symptom
Your agent read the page. Why does it know nothing about it?
You paste a link into Claude, Cursor or your own agent and ask about it. The answer is vague, or it tells you the page "requires JavaScript". The agent did fetch the page — it just never saw the content.
Most tools that give a model web access work the same way: make one HTTP request, take the HTML that comes back, strip it down and hand the text to the model. That is exactly what the reference fetch server from the Model Context Protocol project does, and it is the default web tool in a great many agent setups. It is fast, it needs no browser, and on a server-rendered page it works well.
On a client-rendered page it cannot work. A single-page app built with React, Vue, Svelte or similar sends the browser a near-empty HTML document and a bundle of JavaScript. The browser runs the script, the script fetches the data and builds the page. A tool that makes a request and reads the response stops before any of that happens. What it hands the model is the shell: a title, some meta tags and, if the site is polite, a <noscript> message asking you to turn JavaScript on.
The measurement
What a plain fetch sees on three real pages
We picked three public, client-rendered pages and fetched each one twice: the way the reference fetch server does, and in a real browser. Measured on 24 September 2026.
| Page | What the fetch server returned | Words in the raw HTML | Words after rendering |
|---|---|---|---|
| mastodon.social/@Gargron (a Mastodon profile) | An error: page failed to be simplified | 21 | 842 |
| bsky.app/profile/bsky.app (a Bluesky profile) | An error: page failed to be simplified | 47 | 2,130 |
| soundcloud.com/discover (SoundCloud's discovery page) | 16 words: "Your current browser isn't compatible with SoundCloud" | 58 | 2,201 |
In the body of the raw HTML, every word on all three pages is a fallback notice: "To use the Mastodon web application, please enable JavaScript", "JavaScript Required — this is a heavily interactive web application", "JavaScript is disabled — you need to enable JavaScript to use SoundCloud". Bluesky's shell also carries the account's one-line bio. None of them contains a single post, track or follower count.
The fetch server's own output is worse than the raw HTML, because its readability step correctly decides there is no article in a JavaScript notice. For two of the three pages the model receives an error string; for SoundCloud it receives a browser-compatibility warning and reasons from that.
How we measured
Raw: mcp-server-fetch version 2026.8.18, installed from PyPI, calling its own fetch_url function with its default autonomous user agent — the exact code path an agent triggers. Word counts for the raw HTML are the text of the document body with scripts and styles removed and <noscript> kept, because that is all a non-browser client can read. Rendered: headless Google Chrome with a 1350 × 797 window and up to 15 seconds for the page to settle, then the same count over the rendered DOM with <noscript> removed, since a browser running JavaScript never shows it. A word is a whitespace-separated token. Counts move as these pages change; the gap does not.
Why it matters
An empty page is worse than an error
If the tool stopped the conversation, you would notice. It doesn't. The model gets an error string or a compatibility notice as the tool's result, the turn carries on, and a language model with a URL and a hunch will answer anyway. It describes the account from what it remembers, guesses at recent posts from its training data, or tells you confidently that the page has no pricing information. Unless you read the tool call, nothing in the reply says it never saw the page.
The pages most likely to be client-rendered are also the ones agents are most often asked about: dashboards, app stores, social profiles, product configurators, documentation sites built on SPA frameworks, search results and anything behind a "load more" button.
- Research agents summarize the fallback notice, or fill the gap from memory.
- Coding agents reading a docs page get the navigation shell and hallucinate the API.
- Monitoring agents report "no change" on a page they never actually loaded.
Diagnose
How to tell whether your agent's tool can read a page
- 1Pick a sentence you can see on the page in your browser — a post, a price, a heading halfway down.
- 2Run
curl -sL URLand search the output for it. If it's there, the page is server-rendered and any fetch tool will do. - 3If it's missing and the response is mostly
<script>tags, the page is built in the browser. A tool that doesn't render will give your agent the shell. - 4Ask your agent to quote that sentence from the page. An agent that can't quote it can't read it — whatever it says about the page is coming from somewhere else.
curl -sL https://example.com/some/page | grep -c "a sentence you can see in the browser"
# 0 → the page is client-rendered; a plain fetch hands your agent the shellFix
Three ways to give an agent the rendered page
All three fixes have one thing in common: something runs a real browser, lets the page's JavaScript execute, and only then reads the document.
| Approach | What the model gets | Trade-off |
|---|---|---|
| Browser-automation MCP server (Playwright, Puppeteer) | A browser to drive: navigate, snapshot, click | Many tool calls and a large accessibility snapshot per page; a browser on your machine |
| Your own render step | Whatever you build: rendered HTML → Markdown | You run and maintain headless Chrome, its memory and its failures |
| A hosted MCP server that renders | Clean Markdown of the rendered page, in one call | A network call to a third party, and a per-page cost |
Browser automation is the right tool when the agent has to do something — log in, fill a form, click through a flow. For reading, it is an expensive shape: the model spends turns steering a browser and tokens parsing its snapshots, when what it wanted was the text.
For reading, the efficient shape is one tool call that renders the page, converts it to Markdown and returns it. The conversion matters as much as the rendering: a rendered page is several hundred kilobytes of markup — the three above rendered to 247 KB, 554 KB and just over 1 MB of HTML — and the model needs a few thousand words of it.
With URLpipe
A remote MCP server that renders first
URLpipe's MCP server loads every page in real Chrome on our own custom-built rendering engine, lets it run, and returns what a visitor would see. fetch_markdown is the call for reading: 1 credit per page, converted without a model, and free when your organization fetched the same page in the last seven days. There is nothing to install — it is a URL and a bearer token, and the setup guide has the configuration for Claude Code, Claude Desktop and Cursor.
For pages whose content arrives late, the agent can pass page_options.wait_for_selector to hold the read until an element appears. For the three pages above, rendering alone was enough.
{
"mcpServers": {
"urlpipe": {
"type": "http",
"url": "https://urlpipe.dev/mcp",
"headers": { "Authorization": "Bearer YOUR_TOKEN" }
}
}
}What it does not do: log in, click, or scroll through infinite feeds. A page that needs an account returns what an anonymous visitor sees. If your agent needs to operate a site, pair it with a browser-automation server and use rendering-plus-Markdown for everything it only has to read.
FAQ
Frequently asked questions
Why does my AI agent say a page is empty or asks me to enable JavaScript?
Does the MCP fetch server run JavaScript?
How do I know if a page needs rendering for my agent?
Is rendering every page slower and more expensive?
Can I just give my agent a headless browser instead?
Try it yourself
Free tools for this
No signup — run these on a real page right now, then call the same endpoint from your code.
- Turn any URL into clean Markdown
Paste a link and get the page's main content as tidy Markdown — headings, lists, links and code kept, navigation and cookie banners stripped. The format LLMs and RAG pipelines work best with.
Try it free - Get the rendered HTML of any URL
Paste a link and get the page's HTML after JavaScript has run and redirects have been followed — the DOM a real browser sees, not the empty shell curl returns.
Try it free - Everything about a URL, from one visit
Paste a link and get a screenshot of the page, its title, description and share image, the main content as Markdown and the console errors it threw — all from the same page load.
Try it free
Put this into practice.
Each of the eight kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.