The Robots.txt Generator writes a robots.txt file for your website and shows it as you type. Choose whether to allow crawling or block the whole site, list the paths you want crawlers to skip, add exceptions and your sitemap addresses, and optionally block 12 named AI crawlers or set a crawl delay. Presets cover WordPress, Next.js apps and online shops. Copy the result or download it as robots.txt. It runs in your browser.
Made for marketers and seo. Typical uses:
Blocking /admin/ and /api/ and entering the sitemap as example.com produces a file with "User-agent: *", "Disallow: /admin/", "Disallow: /api/" and a final line "Sitemap: https://example.com/sitemap.xml". Choosing "Allow everything" with no paths writes an empty Disallow line, which blocks nothing.
Robots.txt controls crawling, not indexing: a blocked page can still appear in search results if other sites link to it, so use a noindex tag on a crawlable page to keep it out. The file is public and is only a request to well-behaved crawlers, so it does not hide anything. Google ignores the crawl-delay line, and paths are case-sensitive.
Accuracy: Results show what the tool can measure at the moment you run it and can differ from what search engines see. Use them as guidance.
Expert Tip
After you upload the file, open yourdomain.com/robots.txt to check it loads, and never leave "Disallow: /" on a live site. Do not block your CSS or JavaScript folders, because search engines need them to see the page.
A robots.txt file is a plain text file at the root of a website, for example example.com/robots.txt, that tells crawlers which paths they may request. Search engines read it before they crawl. It controls crawling, not indexing: a page blocked in robots.txt can still appear in search results as a bare link if other sites link to it, because the crawler never sees the page to read a noindex tag. To keep a page out of search, leave it crawlable and add a noindex directive. The Robots.txt Generator writes the file for you and shows it as you type.
You first choose whether to allow crawling and block only some paths, or to block the whole site. Then you can fill in these settings.
With paths /admin/ and /api/ blocked and the sitemap entered as example.com, the file reads: User-agent: * on the first line, then Disallow: /admin/ and Disallow: /api/ on separate lines, then a blank line and Sitemap: https://example.com/sitemap.xml. With "Allow everything" and no paths, the tool writes an empty Disallow line, which means nothing is blocked. With "Block the whole site", it writes Disallow: /, and shows a red warning, because that removes a site from search results over time. Use it only for staging sites.
Paths are case-sensitive and match from the start of the address. Each host needs its own file, so blog.example.com does not use the file of example.com. The file must be plain UTF-8 text at the root, and Google reads only the first 500 KiB. The commonest mistake is leaving Disallow: / on after launch. Another is blocking CSS and JavaScript folders, which stops Google from seeing the page as visitors do. Robots.txt is also public, so do not list secret paths in it, and do not rely on it for privacy, because it only asks well-behaved crawlers to stay away. Google ignores the crawl-delay line, while some other search engines follow it.
The AI option is a choice, not a default. Well-behaved crawlers respect it, but it cannot stop those that ignore the file. Blocking Google-Extended and similar training crawlers affects how content is used for AI models, while blocking search-style bots such as OAI-SearchBot or ChatGPT-User can also stop your pages being cited in AI answers. Decide based on whether you want to be quoted or kept out.
Upload the file to the root and open yourdomain.com/robots.txt to confirm that it loads. Create a sitemap with the Sitemap Generator, then confirm that important pages return 200 with the HTTP Status Checker and that moved pages redirect with the Redirect Checker. The guide to robots.txt, sitemaps and canonical tags shows how the three fit together. The generator runs in your browser.
Three small technical SEO pieces that are easy to get wrong: what robots.txt, XML sitemaps and canonical tags each control, common mistakes, and how to test them.
Explore more in SEO Tools, read the guides and articles, or see how we test our tools.
Generate XML sitemaps for better search engine indexing.
Best for creating a sitemap for a small or new site
Check canonical tags on any webpage to prevent duplicate content issues.
Best for verifying a migrated site's pages canonicalize to the correct new URLs
Check HTTP status codes for any URL.
Best for confirming a page returns 200 OK after deploying a fix for a broken link
Check URL redirects and redirect chains.
Best for auditing a site's redirects for SEO after a domain migration
Generate SEO-optimized meta tags for your web pages.
Best for creating SEO meta tags for new web pages