Guide

How to get keywords from a URL and what to do with them

Paste a URL, get the terms the page is about. This guide explains the ways to extract keywords — statistical and model-based — what each is good for, and how to turn a list of keywords into a brief, a tag set or a cluster of pages.

By Roger Campos · Last updated: September 2026

TL;DR

To get keywords from a URL, fetch the page, reduce it to its main text, then extract the terms it is about. Statistical methods (TF-IDF, RAKE, YAKE) are free and fast but favour frequent words; a language model picks topics the way a reader would, at a cost. Either way you learn what the page covers — not search volume, difficulty or rankings.

Start free See the API

Free plan, no credit card. 1,000 credits a month.

The short answer

What "get keywords from a URL" means

Keyword extraction reads a page and returns the words and phrases that best describe what it is about. From a URL, that is three steps: fetch the page, keep only its content, extract.

People search for this for different reasons — an SEO brief, auto-tagging a content library, sorting a list of links, feeding a classifier — but the pipeline is the same. What changes is which extraction method you use, and what you do with the list afterwards.

One thing to settle up front: extracting keywords from a page tells you what the page covers. It does not tell you what people search for, how many of them, or where the page ranks. Those come from search data — Search Console for your own site, a rank-tracking tool for anyone else's — and no amount of reading the page will produce them.

Step 1

Get the page's real text first

Extraction is only as good as its input, and a web page's HTML is mostly not content. Run a keyword extractor over raw HTML and the top terms are "cookie", "subscribe", "menu" and the site's name, because they appear on every page and in every banner.

1
Render the page if it's built with JavaScript — otherwise there is nothing to extract from. The rendered vs. raw HTML guide shows how to tell.
2
Drop the chrome — navigation, footer, sidebars, consent banners, related-post lists. Converting to Markdown with a converter that isolates the main content does this in one step.
3
Keep the structure — the title and headings are the strongest signal on the page. Many extractors weight them, and you should give them the chance.

Step 2

The ways to extract keywords, compared

There are two families of methods. Statistical ones count: which words appear often here but rarely elsewhere, which words keep company with each other. Model-based ones read: a language model, or an embedding model, judges which phrases the text is about.

Method	How it works	Good at	Weak at
TF-IDF	Scores a term by how often it appears in this page, discounted by how many pages in a collection contain it	Comparing pages within a set; fast, free, explainable	Needs a collection; one page alone has nothing to compare against
RAKE	Splits text at stop words and punctuation into candidate phrases, scores words by frequency and co-occurrence	Multi-word phrases from a single document; no training	Long, clunky phrases; English-centric stop lists
YAKE	Scores candidates on position, casing, frequency and spread through the text	Single documents in many languages; no training	Still surface statistics — misses synonyms and topics never named outright
Embedding-based (e.g. KeyBERT)	Embeds the document and candidate phrases, keeps the phrases closest to the whole	Phrases that capture the topic, not just frequent words	Candidates still come from the text; needs a model to run
Language model	Reads the text and names the topics, in instructions you write	Topics as a reader would put them; dedupes near-duplicates; any language	Costs per page; output varies unless you pin it down

Statistical methods are the right default when you have many pages and want to compare them — which terms make this page different from the rest of the site. A model is the right choice when you want the list a human editor would write: "server-side rendering" rather than "server" and "rendering" separately, no site name, no near-duplicates, and the page's own language.

TF-IDF over a set of pages, with scikit-learn

from sklearn.feature_extraction.text import TfidfVectorizer

# texts: the main content of each page, e.g. its Markdown with links stripped
vectorizer = TfidfVectorizer(stop_words="english", ngram_range=(1, 2), max_df=0.8, min_df=1)
matrix = vectorizer.fit_transform(texts)
terms = vectorizer.get_feature_names_out()

for url, row in zip(urls, matrix):
    scores = row.toarray().ravel()
    top = scores.argsort()[::-1][:10]
    print(url, [terms[i] for i in top if scores[i] > 0])

max_df=0.8 is doing quiet work there: it drops any term that appears in more than 80% of the pages, which is how site-wide boilerplate that survived step 1 gets filtered out.

Use 1

A competitor's page as a keyword brief

The most common reason to pull keywords from someone else's URL is to write a better page on the same subject. The page that ranks for your target query is, among other things, a statement of what Google currently thinks a good answer covers. Its keywords are the table of contents of that answer.

1
Search your target query and take the top three to five results that are the same kind of page as yours — guides with guides, product pages with product pages.
2
Extract keywords from each one.
3
Merge the lists. Topics that appear on most pages are the table stakes: a page without them looks incomplete. Topics on only one page are that author's angle.
4
Compare with your own page's keywords. What's missing is your brief; what nobody covers yet is your chance to be the only page that does.

What this can't tell you

Which of those topics people actually search for, or how much traffic the competitor gets from them. The brief tells you what to cover; search data tells you what to prioritize.

Use 2

Tagging and clustering a list of URLs

Keywords per page turn a pile of links into something you can sort. Tag each URL with its keywords, and pages that share keywords are about the same thing. For a quick clustering without any machine learning, compare keyword sets directly:

Group URLs whose keyword sets overlap

def jaccard(a, b):
    a, b = set(a), set(b)
    return len(a & b) / len(a | b) if a | b else 0.0

# keywords: {url: ["keyword", ...]}
clusters = []
for url, kws in keywords.items():
    for cluster in clusters:
        if jaccard(kws, cluster["keywords"]) >= 0.3:
            cluster["urls"].append(url)
            cluster["keywords"] |= set(kws)
            break
    else:
        clusters.append({"urls": [url], "keywords": set(kws)})

Lowercase and trim keywords before comparing them. For fuzzier grouping — "headless browser" and "headless Chrome" as the same topic — embed each page's keywords and cluster the vectors instead. Either way, the useful outputs are the same: duplicate pages competing for one topic (candidates to merge), topics with a single thin page (candidates to expand), and pages that match no cluster at all.

Content audits — find cannibalization and gaps across a site.
Auto-tagging — suggest tags for a CMS, a bookmark library or a knowledge base.
Routing — send incoming links to the right team or category by topic.
Related content — link pages that share keywords, without hand-curation.

Limits

What keyword extraction can't do

No search volume, difficulty or rankings. Extraction reads the page; it has no idea how the web searches.
No hidden keywords. The meta keywords tag has been ignored by Google for ranking since 2009, and what a page is about is in its content anyway.
One page is one page. A site's home page rarely represents its content. Extract per page and aggregate.
Garbage in. A page behind a login, a consent wall or a JavaScript shell yields the keywords of the wall. Check the text you extracted from before trusting the list.

With URLpipe

Keywords from any URL, in one call

The keyword extractor does all of the above for one URL, free and without signing up: it renders the page in real Chrome, keeps the readable text, and has a language model return between 5 and 15 keywords, most relevant first, in the page's own language, keeping one of any near-duplicates and leaving out the site's name. Fewer keywords for a page that covers less — it doesn't pad.

From code, that's the /keywords endpoint, at 15 credits a page because it runs a model. If you want the text as well — to run your own TF-IDF over a set of pages, say — /markdown costs 1 and runs no model, and /scrape returns both from a single page visit.

POST /keywords

curl -X POST https://urlpipe.dev/keywords \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/guide", "sync": true}'

# → a JSON array of 5–15 strings, most relevant first

FAQ

Frequently asked questions

How do I get the keywords from a website?

Fetch a page (rendering it if it is built with JavaScript), strip the navigation, footer and banners, and run a keyword extractor over the remaining text. Do it per page: a site's home page is rarely representative of what its other pages are about.

Can I see a competitor's keywords from their URL?

You can see the keywords their page is about — the topics and phrases it covers, which is a good brief for your own page. You cannot see which keywords it ranks for or how much traffic they bring; that needs a rank-tracking tool with search data.

How many keywords should a page have?

There is no right number. A focused article is usually about a handful of topics; a long guide covers more. An extractor that returns a fixed 20 terms for every page is padding the short ones.

Do meta keywords tags still matter?

No. Google has ignored the meta keywords tag for web ranking since 2009. Extracting keywords from the content itself tells you what the page is about far more reliably than any tag the author wrote.

Does keyword extraction work in other languages?

Statistical methods need a stop-word list and tokenizer for each language. Model-based extraction works in most languages out of the box; a good one returns keywords in the page's own language rather than translating them.

Try it yourself

Free tools for this

No signup — run these on a real page right now, then call the same endpoint from your code.

/keywords API docs /scrape API docs

Keep reading

HTML to Markdown for LLMs Chunking web pages for RAG What llms.txt is Reading a Lighthouse audit Rendered vs. raw HTML All guides

Put this into practice.

Each of the eight kinds of data URLpipe returns has a free, no-signup tool — try the ideas from this guide on a real page, then grab an API key to run them from your code. 1,000 credits a month, no card.

Start free Email the team