What Is robots.txt and How Sites Accidentally Block Google With It
A surprising number of "why did my traffic suddenly disappear" situations trace back to a single file that almost nobody checks until something's already gone wrong: robots.txt. It's a small, plain-text file that quietly controls what search engines are allowed to crawl, and a single misplaced line in it can block an entire site from Google without throwing a single visible error anywhere.
This guide covers what robots.txt actually does, the specific mistakes that cause real damage, and how to check whether yours is actually doing what you think it is.
What robots.txt Actually Is
Robots.txt is a plain-text file, located at the root of your domain (yourdomain.com/robots.txt), that gives crawlers instructions about which parts of your site they're allowed to access. It's one of the oldest web standards still in active use, and nearly every legitimate crawler, including Google's, checks it before crawling anything else on your site.
A basic file looks like this:
User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xml
This tells every crawler (User-agent: *) not to crawl anything under /admin/ or /api/, and points them to your sitemap, covered in more depth in our guide to XML sitemaps, for discovering everything else.
The Mistake That Deindexes an Entire Site
This is the one that causes the most damage, and it's almost always accidental:
User-agent: *
Disallow: /
That single line tells every crawler to avoid the entire site, every page, with no exceptions. This configuration is genuinely common on staging and development environments, specifically to keep search engines from indexing a site that isn't ready for public visibility yet. The damage happens when that exact file gets deployed to production unchanged, often because a deployment script copies the staging environment's configuration by default, or because a developer disabled crawling during development and simply forgot to revert it before launch.
The effect is brutal and often slow to notice: Google respects the instruction, stops crawling the site entirely, and existing indexed pages gradually drop out of search results as Google periodically re-checks them and keeps finding the same block. There's no error message anywhere in your own site's logs or dashboard. The first visible symptom is usually a slow, unexplained decline in organic traffic over subsequent weeks, by which point real damage has already accumulated.
Other Common Mistakes Worth Checking For
Blocking resources crawlers need to render your page properly. Blocking /css/ or /js/ directories can prevent Google from rendering your page the way an actual visitor sees it, which can affect how it's evaluated, especially for pages that rely heavily on client-side rendering.
Confusing "noindex" with "disallow." Robots.txt controls crawling, not indexing directly. A page can still get indexed from external links even if robots.txt blocks crawling it, just without Google having actually read its content, which often produces a worse result (a bare URL with no title or description in search results) than a dedicated noindex meta tag would.
Case sensitivity and exact path matching. Robots.txt rules are case-sensitive and match paths literally. Disallow: /Blog/ does not block /blog/, a subtle typo that can leave content unintentionally exposed or unintentionally blocked depending on which direction the mismatch goes.
How to Actually Check Your Own robots.txt
The fastest check is also the most overlooked one: just visit yourdomain.com/robots.txt directly in a browser and read it. It's a plain text file, there's no interface to navigate, just an honest look at exactly what instructions are currently live on your actual production site, not what you assume is there from when it was last set up.
Google Search Console also includes a dedicated way to test specific URLs against your current robots.txt rules, confirming whether a given page is actually crawlable rather than guessing based on reading the raw file yourself.
Setting This Up Correctly in Next.js
The App Router handles this with a dedicated file, generated dynamically rather than maintained as a static text file that can silently drift:
// app/robots.ts
import type { MetadataRoute } from 'next';
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: '*',
allow: '/',
disallow: ['/admin/', '/api/'],
},
sitemap: 'https://yourdomain.com/sitemap.xml',
};
}
Generating it this way means it's defined once in your actual codebase, reviewed the same way any other code change would be, rather than existing as a static file someone configured once and nobody has looked at since.
Frequently Asked Questions
Does robots.txt guarantee a page won't appear in search results?
No. It only controls crawling. A blocked page can still appear in results, typically without a proper title or description, if other pages link to it. A noindex meta tag is the correct tool for genuinely preventing a specific page from appearing in results.
How often should I check my robots.txt?
At minimum, right after any deployment pipeline change, hosting migration, or staging-to-production launch, since these are exactly the moments a staging-environment block can accidentally carry over.
Can robots.txt block specific search engines but not others?
Yes. Rules can target a specific crawler by name instead of using the wildcard *, allowing different access rules for different bots if genuinely needed.
Is a missing robots.txt file a problem?
Not inherently. No robots.txt generally means crawlers are free to crawl everything, which is often fine for a small site with nothing that needs restricting.
Conclusion
Robots.txt is a small, easy-to-ignore file that can cause outsized damage when it's wrong, and the failure mode is specifically silent, no error, no warning, just a slow traffic decline that takes real investigation to trace back to its actual cause. A five-minute check of your live file is one of the cheapest, highest-leverage things worth doing periodically, especially after any deployment or hosting change.
Not sure if your own robots.txt is quietly working against you? Get in touch.
Anas, full-stack Next.js developer building SaaS products and premium templates.
