Duplicate content: causes, myths and how to consolidate
Duplicate content is the same or near-identical content at more than one URL, or titles and descriptions repeated across pages. Causes, myths and fixes.
Duplicate content is the same or nearly the same content reachable at more than one URL, and by extension the same title or meta description repeated across pages that are meant to be different.
Most of it is technical. One page answers with and without www, with and without a trailing slash, in mixed case and with tracking parameters, and each variant is a separate URL to a crawler. Google’s URL guidance points out that URLs are case-sensitive, so /APPLE and /apple are two URLs. The other kind is editorial, such as a template that gives a whole product range one title.
Category: Content
Why it matters
There is no duplicate content penalty. Google’s 2008 post on the subject says duplication is not grounds for action unless it looks meant to deceive and manipulate results. The cost is different. Google picks one version of each set to show, and it may not pick the one you want. Links and other signals split across the copies, and crawlers spend requests on URLs that add nothing. Repeated titles and descriptions make pages look interchangeable in results, and Google’s own guidance asks for distinct text in each page’s <title> and description.
How squirrelscan checks it
content/duplicate-title and core/title-unique group crawled pages by title and report every title held by more than one page. The second also collapses runs of whitespace, so a double space does not hide a duplicate. content/duplicate-description does the same for meta descriptions. All three leave noindexed pages out on a site that is itself indexed, and count reciprocal hreflang alternates in the same language, such as en-gb and en-us, as one page when neither canonicalises elsewhere.
The URL rules flag where duplicates come from. core/canonical checks each page for an absolute canonical. url/trailing-slash reports each path’s slash state, url/lowercase warns on uppercase in a path, and url/parameters counts query parameters and names tracking ones. squirrelscan does not compare page bodies for near-duplicate text.
| Rule | What it flags | Severity |
|---|---|---|
| Duplicate title tags: the same title on several pages | Two pages with the same title compete with each other in search and look identical in results. squirrel groups every crawled page by its title. | warning |
| Duplicate meta descriptions across pages | One description reused across a section describes none of those pages well. squirrel groups every crawled page by its meta description. | warning |
| Duplicate page titles: the site-wide uniqueness check | Two or more pages sharing a title after case and whitespace are normalised. squirrel groups them and lists every URL. | warning |
| rel=canonical: presence, absolute form and self-reference | A page with no canonical link, a relative one, or one that points somewhere else. squirrel checks every crawled page. | warning |
| Trailing slash: /page and /page/ are two different URLs | A file URL with a trailing slash is a mistake; a directory URL either has one or does not, consistently. squirrel reports the state of each path. | info |
| Uppercase in a URL: the duplicate content it creates | Most servers treat /Page and /page as two URLs serving the same content. squirrel flags every crawled path containing a capital letter. | warning |
| URL parameters: how many is too many, and what to do | Every parameter combination is a separate URL to a crawler. squirrel counts the query parameters on each page and names the tracking ones. | info |
How to fix it
Pick one URL per page, redirect the other forms to it, and declare it:
<link rel="canonical" href="https://example.com/shoes/running" />
<title>Running shoes, page 2 | Acme</title>Choose one slash convention and one case, and redirect the rest permanently (the trailing slash fix, the lowercase fix). Point tracking and filter variants at the clean URL with a canonical (the canonical fix, the parameters fix). Give each page its own title, with the page number on paginated listings (the duplicate title fix). A page with nothing distinct to say is better with no meta description than a copied one (the duplicate description fix).
Run the check
squirrel audit https://example.com --rule-include content/duplicate-title,content/duplicate-description,core/title-unique,core/canonical,url/trailing-slash,url/lowercase,url/parametersA signed-out local audit costs nothing.
Related terms
- Canonical URL: The URL a page declares, with
<link rel="canonical">or a Link header, as the preferred address for its content. - Title tag: The
<title>element in the page head; browsers show it in the tab and search engines usually use it as the result headline. - Meta description: A
<meta name="description">summary of the page that search engines may use as the result snippet. - Trailing slash: Whether URLs end in /; /page and /page/ are different URLs, so a site should pick one and redirect the other.
- Thin content: A page with too little substantive content to be useful, judged here against word count and the site’s own norm.