Robots.txt, Sitemaps and Canonical Tags: What Each One Does

A page disappears from search after a site update. Another ranks under an address with tracking parameters on the end. These problems usually trace back to three small pieces of technical SEO that look alike but do different jobs: the robots.txt file, the XML sitemap and the canonical tag. This guide explains what each controls, the mistakes that do the most damage and how to write and check them with free tools.

Three pieces, three different jobs

• robots.txt is a text file at the root of your site that asks crawlers which URLs they may fetch. It manages crawling.
• An XML sitemap is a list of URLs you would like search engines to know about. It helps discovery.
• A canonical tag is a line in a page's head that names the preferred address when several URLs show the same or very similar content. It helps consolidate duplicates.

None of them is a command, and none is a switch for indexing, which is where the most common mistake begins. Each is a signal that well-behaved crawlers usually follow.

robots.txt: crawling, not indexing

The file sits at the root of the host, for example https://example.com/robots.txt, and is plain text. A few lines cover most needs:

• User-agent: * starts a group of rules for every crawler.
• Disallow: /admin/ asks crawlers not to fetch addresses that begin with that path.
• Allow: /admin/help/ makes an exception inside a blocked folder.
• Sitemap: https://example.com/sitemap.xml points to your sitemap with a full address.

Paths are case-sensitive, and in Google's implementation the more specific rule wins. Google ignores the crawl-delay line.

The Robots.txt Generator builds the file from blocked paths, optional exceptions and sitemap addresses, plus an optional switch that adds rules for twelve named AI crawlers, which work only if the crawler chooses to obey them. Copy the output or download robots.txt and upload it to the site root. Its presets for WordPress, Next.js and online stores are starting points, so read every line. The Next.js preset blocks only /api/ and /admin/. It deliberately leaves /_next/static/ alone, because that is where a Next.js site keeps its compiled JavaScript and CSS, and Google needs to fetch those files to render your pages. If you add a rule of your own, check that it does not block scripts, styles or images a page needs.

The mistakes that cost the most

1. Using robots.txt to hide a page. A blocked URL can still appear in results, without a description, if other pages link to it. To keep a page out of search, use a noindex robots meta tag or password protection.
2. Blocking a page you want noindexed. Google has to fetch a page to see its noindex tag, so a robots.txt block means the tag is never read.
3. Blocking CSS, JavaScript or images. Google renders pages much as a browser does, and blocked files can make it misread the layout and content.
4. Leaving a full block in place. A rule that disallows the whole site suits a staging copy and is disastrous after launch. The generator shows a red warning when you choose it.
5. Putting the file in the wrong place. It works only at the root of the exact host. A file in a subfolder is ignored, and a subdomain needs its own.

XML sitemaps: a list of pages you want found

A sitemap helps search engines find pages, especially new or rarely linked ones, but it does not guarantee crawling or indexing. One file can hold up to 50,000 URLs or 50 MB uncompressed, and larger sites split the list across several sitemaps.

List only the canonical, indexable version of each page: addresses that return a normal 200 response, do not redirect, are not blocked and do not carry noindex.

The Sitemap Generator takes one URL per line. It removes duplicates, reports invalid lines, assumes https when you leave out the scheme, and warns when the list mixes hosts or passes 50,000 URLs. Change frequency and priority are optional and off by default, because Google ignores them. Add a last modified date only when it reflects a real content change, since Google relies on it only if it proves accurate. The tool builds the file from the URLs you give it and does not crawl your site or check status codes. Upload sitemap.xml, add a Sitemap line to robots.txt and submit the address in Google Search Console.

Canonical tags: choosing one address

The same page can be reachable at several addresses: with and without www, over http and https, with tracking parameters, or through filters and sort orders. The canonical tag, a link element with rel set to canonical, names the one you prefer. Most pages should point to themselves. Google treats the tag as a hint and may pick a different canonical.

Common mistakes:

• Pointing every page at the home page, which tells Google they are all duplicates of it.
• Pointing at an address that redirects, returns an error or carries noindex.
• Two tags that disagree, or a tag placed outside the head, where it is ignored.
• Relative paths, http addresses, #fragments or tracking parameters in the canonical URL.
• Combining a canonical tag with noindex on the same page, which sends mixed signals.

The Canonical Tag Checker tests pasted HTML for a missing or repeated tag, a tag outside the head, empty, relative or http addresses, fragments, tracking parameters and a noindex on the same page. Enter the page's own URL and it also says whether the tag is self-referencing or points elsewhere. It reads only what you paste and does not fetch the target, so use the status tools below for that. If JavaScript adds the tag, copy the rendered HTML from your browser's developer tools instead of using view-source.

How to test all three

1. Open yoursite.com/robots.txt in a browser. It should load as plain text. The Server Status Checker confirms the status code, and the Redirect Checker shows whether the file, the sitemap or a canonical address passes through a redirect.
2. In Google Search Console, the robots.txt report lists the files Google found and any problems. URL Inspection shows whether a URL is blocked and which canonical Google chose next to the one you declared.
3. Submit the sitemap in the Sitemaps report and read its status.
4. Paste key templates into the Canonical Tag Checker, and repeat everything after a migration or redesign.

If a check fails and the cause is unclear, Website not loading? Read DNS, status codes and redirects walks through the layers in order. For the snippet side of search, see meta titles, descriptions and schema markup.