Robots.txt strategy: what to block, what to allow and why most sites get it wrong

Key takeaways

• Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it.
• Google ignores anything past 500 KiB in a robots.txt file and generally caches it for up to 24 hours, so changes are not instant.
• Rules apply only to the exact host, protocol and port, so a subdomain or staging host needs its own file and, better, authentication.
• Overly broad disallow rules that block CSS, JavaScript or whole sections are the most damaging common mistake.
• Google retired the Search Console robots.txt tester in late 2023; use the robots.txt report and third-party testers instead.

Your robots.txt file tells crawlers which URLs they may request. One wrong line can stop Google from crawling an entire section, while a missing rule can let it waste time on endless low-value URLs. This Emulent guide covers what to block, what never to block, how to structure the file and how to test changes.

What should you block in robots.txt?

URL patterns that offer no search value and can multiply without limit.

• Internal search results, such as /search?q= or /?s=, which create a near-infinite URL space.
• Cart, checkout, account, wishlist and session ID URLs.
• Admin and backend paths that are not already protected. Note that robots.txt is public, so it is not a security control; sensitive areas need authentication.
• Print or alternate-view URLs that duplicate the main pages.
• Faceted filter combinations you have decided should never be crawled.

Block staging and development hosts with HTTP authentication. Google’s robots.txt specification says rules apply only to the host, protocol and port where the file sits, so the main site’s file does not cover a staging subdomain, and a disallow rule alone does not keep URLs out of the index.

What should you never block in robots.txt?

Anything Google needs to render or rank your pages.

• CSS and JavaScript files. Google renders pages to understand them, and blocking these resources can make pages look broken or empty to it.
• Image directories, unless you deliberately want images kept out of Google Images.
• Any page you want to rank. Leftover migration rules and loose wildcards are frequent culprits.
• Pages you want removed from the index. To deindex a page, let Google crawl it and see a noindex tag; a robots.txt block hides that tag.
• Pages with strong backlinks, since Google cannot crawl them to understand where those links lead.

How should you structure robots.txt for an enterprise site?

Keep it short, documented and version-controlled.

• Comment each rule with its purpose and the date it was added.
• Group rules by purpose, such as crawl traps, duplicates and temporary blocks.
• Add a Sitemap: line pointing to your sitemap index.
• Stay well under Google’s 500 KiB limit by consolidating rules with wildcards.
• Test wildcards carefully. Disallow: /*?sort= blocks every URL containing that parameter; $ anchors a pattern to the end of a URL.
• Decide how to treat AI crawlers. Tokens such as GPTBot, ClaudeBot and Google-Extended can be allowed or disallowed separately from Googlebot, and blocking Google-Extended does not affect Google Search.

WordPress generates a basic virtual robots.txt that disallows /wp-admin/ while allowing admin-ajax.php. Emulent replaces it with a documented file. Shopify, which Emulent recommends for ecommerce, lets you edit robots.txt through a robots.txt.liquid template.

How do you test robots.txt changes before deploying them?

Test against real URLs, compare crawls, then watch the logs.

• Check proposed rules against lists of important and blocked URLs with a robots.txt testing tool; Google’s old tester was replaced by the Search Console robots.txt report in November 2023, which shows fetch status and errors.
• Deploy to staging, crawl it and confirm important pages remain reachable.
• Compare crawl results before and after to see exactly which URLs changed status.
• After release, watch server logs and Crawl Stats for several days. A sudden drop in crawling of important sections signals a bad rule.

The robots.txt report also lets you request a recrawl of the file after an urgent fix, since Google otherwise caches it for up to a day.

How can the Emulent team help?

Emulent audits robots.txt files as part of technical work within enterprise SEO programs, removing rules that block important content and adding rules for crawl traps. A caveat: robots.txt mistakes can remove large sections from search quickly, so make changes through version control with review. Contact the Emulent team to review your robots.txt.