My duplicate-content check passed the page it should have blocked

I run a site of free calculators, and I wrote a gate that stops me from building a page that competes with one I already have. On its first real day it was wrong twice, once in each direction. Here is the code, the two failures, the fix and the test table.

What the gate is for

Before a new page gets built, the research script compares its target keyword with every page on the site. If an existing page already covers the topic, the run fails and I either extend that page or explain, in one line, why the intent differs. Two pages chasing one query split their rankings and neither does well. I found that out the slow way: Search Console showed 63 queries (4 or more impressions each) in 28 days where two of my own URLs appeared.

Version 1 was the obvious thing. Lowercase, drop stopwords and the word "calculator", then compare word sets.

const toks = (t) => new Set(String(t).toLowerCase().replace(/[^a-z0-9 ]+/g, ' ')
  .split(/\s+/).map((w) => w.replace(/s$/, '')).filter((w) => w && !SW.has(w)));
const jac = (a, b) => { let i = 0; for (const x of a) if (b.has(x)) i++; return i / (a.size + b.size - i); };
const subset = (a, b) => a.size > 0 && [...a].every((x) => b.has(x));
// strong overlap: jac >= 0.67, or one set contains the other and jac >= 0.5

Each keyword was compared with each page's primary keyword, title and slug.

Failure A: it blocked a page that does not exist

The first keyword I ran was "quebec paycheck calculator". The gate failed it: strong overlap with /paycheck-calculators/ and /take-home-pay-calculator/. I have no Quebec page. Both existing pages are generic, so their word sets are subsets of the keyword's, and "subset in either direction" counted that as a match. The extra word, a place, is exactly what makes the query different.

The fix is to allow the subset test in one direction only. An existing page covers a keyword only when the page already contains every word of the keyword:

const covered = pageSets.some((x) => subset(kt, x) && jac(kt, x) >= 0.5);

A generic page no longer blocks a specific keyword. A page that says everything the keyword says still does.

Failure B: it passed a page I had already written

I then planned a "BC income tax calculator" (6,600 searches a month in Semrush's Canada data). The gate said no strong overlap. It was wrong. The day before, I had published a BC paycheck calculator, and it already carried the phrase: in an FAQ question, in bold in a section about what people call the tool, and again as "income tax calculator BC". Its meta description said "income tax calculator" too. Four windows of text in total.

The gate looked at titles and slugs. The page titled "BC Paycheck Calculator" shares only the word "BC" with "bc income tax calculator", so the sets looked different. The page text said otherwise.

So the gate now also reads the built pages. For each keyword it looks for a window, a few words longer than the keyword, in which all the keyword's words appear in any order (this version keeps the word "calculator", because here it matters). Two or more windows on one page counts as the page already targeting that phrase:

const need = [...toksKeep(kw)];
let hits = 0, i = 0;
while (i < words.length) {
  const win = words.slice(i, i + need.length + 3);
  if (need.every((n) => win.includes(n))) { hits++; i += need.length + 3; } else i++;
}
if (hits >= 2) textHits.push({ keyword: kw, page, occurrences: hits });

Run against the real pages, it flags the BC page (4 windows) and the Alberta one (3 windows). I dropped both planned pages and retitled the existing ones so the title leads with "income tax calculator", the phrase people actually search.

One bug nearly slipped through even then. I removed a trailing "s" from keyword words but not from page words, so "vs" became "v" on one side only and "RRSP vs TFSA" never matched. Normalise both sides the same way.

The test table

The last row is the honest one. "Overtime pay calculator" against my overtime page is real cannibalization. The word "pay" is an extra token, so the gate lets it through. Word overlap cannot see near-synonyms.

What it cannot see, and what covers it

For that gap I rely on Search Console. A query with query and page as dimensions, grouped by query, lists every query where two of my URLs show up. I re-run it every Monday and log the count. It was 63 this week, and the list is the to-do for merging or separating pages.

Takeaways

• A check that compares titles and slugs only sees the page's label. Read the page text too.
• Subset tests need a direction. "The page contains all of the keyword" is a different claim from "the keyword contains all of the page".
• Normalise both sides of a comparison the same way.
• Keep a table of cases that must pass and cases that must fail, including one you know it will get wrong.
• Use the search engine's own data to check the checker.

This post was drafted by an AI assistant that helps run this account. The code and numbers are from the real project and were checked against it.