robots.txt vs llms.txt vs humans.txt: Three Files, Three Audiences
All three sit at the same kind of location — a site's root, a plain-text file, no build step required. That surface similarity is where the resemblance ends. Each one answers a completely different question, for a completely different reader, and none of the three can substitute for either of the others.
robots.txt — can an automated visitor request this page at all
robots.txt is a decades-old, widely-honored convention that answers a permission question: is this specific crawler allowed to fetch this specific path? Search engine crawlers, AI training crawlers (GPTBot, ClaudeBot, and others can be named individually), and other bots all read the same file and get a yes-or-no per path based on their user-agent. It says nothing about what a page contains — only whether a well-behaved automated visitor is allowed to look.
llms.txt — what is this site actually about, for an AI model
llms.txt is a much newer, community-driven convention aimed specifically at AI models trying to understand a site well enough to answer questions about it or cite it accurately. Instead of a permission list, it's a structured markdown summary — a name, a short description, and a categorized index of key pages, each with a one-line description. It doesn't grant or restrict access to anything; a page can be fully crawlable and still absent from an llms.txt index, or vice versa. It exists purely to make comprehension cheaper for a model with a limited context window.
humans.txt — who actually built this, for a person who checks
humans.txt is the odd one out: no automated system reads it. It's a plain-text credits file — team members, technologies used, a short note — for the rare visitor who knows the convention exists and goes looking on purpose. It has zero effect on crawling, indexing, rankings, or AI answers. Where the other two exist because some machine reads them, humans.txt exists specifically because no machine does.
Side by side
- Reader: robots.txt → crawlers and bots. llms.txt → AI models. humans.txt → people, occasionally.
- Question answered: robots.txt → "can you fetch this?" llms.txt → "what is this site, quickly?" humans.txt → "who made this?"
- Effect on SEO or AI visibility: robots.txt → direct (controls crawl access). llms.txt → indirect (helps comprehension, not ranking). humans.txt → none.
- How enforced: robots.txt → honored voluntarily by well-behaved crawlers. llms.txt → read voluntarily by supporting AI systems. humans.txt → not read by any system at all.
Do you need all three?
robots.txt is worth having on essentially every site regardless of size or purpose, since it governs basic crawl access. llms.txt is worth adding once AI comprehension of your content specifically matters — documentation, knowledge bases, API references. humans.txt is optional in a different sense entirely: it's not solving a technical problem for anyone, it's a small courtesy for a category of reader that may never show up. Having all three isn't redundant — they simply don't overlap.
Generate all three
FreeToolDev has a free, no-signup generator for each: robots.txt (with one-click AI crawler blocking), llms.txt (turns a pasted page list into a structured index), and humans.txt (team credits and tech stack, no email required) — all generated entirely in your browser.