« Web Scraping

Canonical tags & meta robots directives

robots.txt controls whether a crawler fetches a page at all. These two mechanisms work at the page level instead, after a page has already been fetched: a canonical tag says which URL a piece of content should be attributed to, and a robots directive says what to do with this specific page once it's been read.

Canonical tags

• Self-referencing — the recommended default: a page's canonical points at itself
• Duplicate content, page A and page B — two different URLs, both canonical to A
• Tag vs. header conflict — the <link> tag names one URL, the HTTP Link header names another
• Cross-domain — canonical points at a URL on a different domain entirely

Meta robots & X-Robots-Tag

• noindex, follow — keep this page out of the index, but still crawl its links
• index, nofollow — index this page, but don't follow its outbound links
• X-Robots-Tag header — the same noindex directive, delivered as an HTTP response header instead of a meta tag (the only option for non-HTML resources like PDFs and images)
• noarchive, nosnippet — index this page, but don't cache a copy of it or show a snippet in search results