Set which paths to block, add your sitemap, and optionally block AI crawlers like GPTBot and ClaudeBot — all in one generated file, ready to copy or download.
Upload the file to your site root as robots.txt — it needs to be reachable at exactly yoursite.com/robots.txt, not in a subfolder, or crawlers won't find it. Test it afterward in Google Search Console's robots.txt report to confirm it's being read correctly.
Disallowing GPTBot, ClaudeBot, Google-Extended, and similar user-agents opts your content out of that specific company's AI training crawls — it's separate from, and doesn't affect, whether your site shows up in ordinary search results or in that company's live AI-answer features, which often use a different crawler. This is a preference call, not a fix for a technical problem: leaving them unblocked means your public content may be used to train or ground AI models; blocking them opts out of that specific use, at no cost to your regular SEO.
Every rule here is a request that well-behaved crawlers choose to honor — nothing enforces it. A disallowed page can still appear in search results if other sites link to it, and robots.txt is not a substitute for real access control on anything genuinely private (use authentication or a noindex tag instead). See this post for a fuller breakdown of what robots.txt does and doesn't control.
No — Disallow only stops future crawling, it doesn't retroactively remove a page from the index. To get an already-indexed page removed, use a noindex meta tag (which requires the page to still be crawlable so Google can see the tag) or return a real 404/410 status.
No, generally. Search engines need to load your CSS and JS to render the page the way a visitor sees it — blocking asset folders was a common practice years ago, but today it can hurt rendering-dependent ranking signals. Only disallow folders that hold genuinely non-public content: admin panels, internal search results, staging directories.
Not necessarily — a company's training crawler and its live-answer crawler are sometimes the same user-agent and sometimes different ones, and this varies by provider and changes over time. Blocking the training-specific bot listed here targets training use; check that provider's current documentation if you specifically want to affect live citations.