Robots.txt is a plain-text file placed at a site’s root that tells crawlers which URL paths they are and are not welcome to fetch. It expresses access preferences per user-agent, not a security boundary.
| Term | Robots.txt |
|---|---|
| Category | Page Structure and Technical SEO |
| Also known as | Robots File |
| Where it appears | Technical factors and Site Builder context |
What it means in RankGear
RankGear treats robots.txt as a technical signal: it checks whether the file exists at the site root and what its directives allow or disallow for the paths in play. Those observations feed the technical factors, and the Site Builder surfaces the file so you can see and manage crawl rules alongside the pages you are working on. The point is not the file in isolation but whether its rules are consistent with the pages you actually want indexed and compared.
How to interpret it
Read both presence and implementation, not presence alone. A robots.txt can exist yet still block a directory you meant to keep open, sit empty, or contradict itself across user-agent groups. So the useful question is whether the directives match intent: are the URLs you care about reachable, and are the ones you want kept out actually disallowed. Remember too that Disallow governs crawling, not indexing — a blocked URL can still appear in results if it is linked elsewhere — and that the file is advisory, honored by mainstream crawlers but not enforced against ones that ignore it.
Example
You are auditing a page that should be ranking but never gets crawled. RankGear reads the site’s robots.txt and finds a Disallow: /blog/ line left over from a staging setup, sitting under User-agent: *. Every article beneath /blog/ is being told not to fetch. You remove the stray rule, confirm the paths are now allowed, and re-check so the content can be crawled and measured against the ranking set.
Important considerations
- Robots.txt controls crawling, not indexing — to keep a URL out of results, a noindex directive or authentication is the reliable tool, not a Disallow line.
- The file is a request, not an enforcement mechanism; well-behaved crawlers respect it, but it is not a substitute for access controls on private content.
- A single misplaced rule can hide a whole section of a site, so check the paths that matter rather than assuming a present file is a correct one.
- Having a clean, well-formed robots.txt improves crawl efficiency and machine readability, but it does not by itself cause a page to rank.
- Technical factors are comparative indicators of how your setup differs from the ranking pages, not scores Google assigns.
Related terms
Part of the RankGear glossary · how RankGear measures · the 870 factors.