« Web Scraping

hreflang

hreflang tells a crawler that several URLs are language or region variants of the same page, so it can serve the right one to the right reader instead of treating them as duplicate content. The rule that trips people up: every variant has to link to all the others, including itself, or the tags aren't considered valid.

Each variant also declares an x-default, the fallback for a reader whose language doesn't match any listed variant.