Turn the web into clean data

Why Markdown is the right format for language models, what a clean conversion keeps and drops, and how to handle client-rendered pages.

TF-IDF, RAKE and YAKE vs model-based extraction, a competitor's page as a keyword brief, clustering, and the limits.

Markdown first, heading-aware splitting, chunk size and overlap, metadata, and keeping chunks fresh.

The format section by section, a real example, and an honest look at which tools read it.

The reference fetch server does a plain GET, so client-rendered pages reach your agent empty — measured on three real pages.

Exact configuration for Claude Code, Claude Desktop and Cursor, token scopes, first prompts and costs.

The three metadata standards, why they conflict in practice, and how to reconcile them into one clean object.

The og:image size and file limits each network documents, checked against its own docs, and the one size that works everywhere.

Puppeteer vs Playwright vs an API, and the full-page gotchas: lazy loading, sticky headers, banners and size limits.

What the four categories and the Performance score mean, how Core Web Vitals are measured, and how to prioritize fixes.

LCP, INP and CLS, their thresholds, and why a Lighthouse run and real-user data tell different stories.

DevTools, headless Chrome and load-time captures — and what each can and cannot see on a site you don't own.

Why modern sites return an empty shell to curl, how client-side rendering works, and how to get the post-JavaScript DOM.

The signed string, the raw body, timestamp tolerance, constant-time compares and rotation — with code in five languages.

What RFC 9309 says about robots.txt, how to pace requests per host, and how to back off when a site struggles.

What the header does, how servers implement it, and the mistakes that make a retry charge twice.

Why automated visits get every banner and ad, and how to remove them without consenting on anyone's behalf.