robots.txt Mistakes That Accidentally De-Index a Whole Site
robots.txt is a short, plain-text file, and most of what goes wrong with it comes down to a handful of small, specific mistakes rather than any real complexity in the format itself. A few of them are common enough that anyone who's worked in SEO for a while has a story about finding one live on a client's site.
The one-character disaster: Disallow: /
A single forward slash after Disallow blocks the entire site — every path starts with /, so this rule matches everything. This usually happens by accident: a staging-site robots.txt (deliberately blocking everything so search engines don't index the test environment) gets copied over to production during a deploy and never gets changed back. It's the single most common way a site quietly vanishes from search results with no warning, since nothing about the site itself looks broken — it just stops getting crawled.
Blocking the CSS and JS your pages need to render
Years ago, blocking asset folders like /wp-content/themes/ or /assets/ was common practice, on the theory that search engines only needed the HTML. That's no longer true — modern search engines render pages the way a browser does, including CSS and JavaScript, to judge layout, mobile-friendliness, and content that only appears after scripts run. Blocking those folders means a crawler sees a broken, unstyled version of your page, which can genuinely hurt how it's understood and ranked.
Case-sensitivity nobody expects
Paths in robots.txt are matched case-sensitively. Disallow: /Private/ does nothing to protect a folder actually named /private/ — to the parser, those are two entirely different paths. This one is sneaky because it produces no error and no warning; the rule is syntactically valid, it just silently doesn't match what you meant it to.
Listing a URL as both disallowed and in your sitemap
This isn't invalid syntax, but it's a contradiction: you're telling crawlers "don't fetch this" in one file and "here's this, please index it" in another. It usually happens because sitemap generation and robots.txt rules get maintained separately, by different tools or on different schedules, so a newly-blocked path doesn't automatically get pulled from the sitemap. Google's typical behavior is to respect the block and mostly ignore the sitemap entry — but it's a sign the two files aren't being kept in sync, which is worth fixing even when it isn't causing visible harm yet.
Trailing slash confusion
Disallow: /blog and Disallow: /blog/ aren't the same rule. Without the trailing slash, the rule matches any path starting with those characters — /blog, /blog/, and also /blog-archive or /blogger-outreach, which is almost certainly not intended. Being precise about the trailing slash, and about whether a rule should match a whole folder versus a single file, avoids accidentally blocking (or failing to block) more than intended.
Forgetting it's not access control
A Disallow rule is a request that well-behaved crawlers honor voluntarily — nothing stops a browser, a script, or a less scrupulous bot from ignoring it and fetching the page anyway. Treating robots.txt as a security measure for genuinely sensitive content (an internal tool, unpublished data) is a mistake in the other direction: the fix there is authentication or a firewall rule, not a text file that only asks nicely.
Generate one correctly
FreeToolDev's robots.txt generator builds a syntactically correct file from a simple form — paths to block, paths to explicitly allow, your sitemap URL, and optional AI-crawler blocking — so the common mistakes above (a stray slash, wrong case assumptions, forgetting the sitemap line) don't have a chance to creep in from hand-editing the file.