Technical SEO · 2026-08-10
Robots.txt Mistakes That Quietly Kill Your Rankings
Robots.txt controls crawling, not indexing. See the small mistakes—blocking CSS, a stray Disallow: /, hiding pages—that quietly tank your rankings.
TL;DR
The most damaging robots.txt mistakes are blocking CSS and JavaScript, accidentally disallowing your whole site with 'Disallow: /', using it to hide pages from search (it doesn't), and blocking URLs you also want indexed. Robots.txt controls crawling, not indexing—one wrong line can quietly drop rankings for weeks.
The mistake we see most often on small sites is a single leftover line: 'Disallow: /'. It usually survives from a staging or development environment and gets pushed live with the theme. That one line tells Google to stop crawling every page. Rankings don't crash the same day—they erode over two to four weeks as cached pages go stale, so most founders never connect the cause to the effect.
The second common error is blocking CSS, JavaScript, or image folders (like 'Disallow: /assets/' or '/scripts/'). Google renders your pages like a browser before ranking them. If it can't load your styles and scripts, it sees a broken layout, misjudges mobile usability, and may treat content loaded by JavaScript as missing. Never block resources your pages need to render.
The biggest misconception is that robots.txt hides pages from search results. It doesn't. It only asks bots not to crawl a URL. If that URL is linked from anywhere, Google can still index it—showing a bare title with 'No information is available for this page.' To actually keep a page out of results, allow crawling and use a 'noindex' meta tag or X-Robots-Tag header instead.
Watch for the contradiction trap: blocking a URL in robots.txt while also expecting it to rank. If you 'Disallow' a page, Google can't crawl it, so it can't read your 'noindex' tag or your canonical either. The two directives cancel out. Pick one job per URL—block it from crawling, or let it be crawled and control indexing on the page.
Shopify and other hosted platforms add their own quirks. Shopify ships a default robots.txt that blocks '/cart', '/checkout', and internal search URLs—that's correct, leave it. The risk starts when you edit 'robots.txt.liquid' to block '/collections' or filtered URLs and accidentally cut off pages you rank for. On WordPress, check that 'Discourage search engines' under Settings > Reading is off; it silently injects a site-wide block.
Audit it in ten minutes. Open yoursite.com/robots.txt directly and read every 'Disallow' line out loud—if you can't explain why it's there, it's a suspect. Then use Google Search Console's URL Inspection tool to confirm key pages are 'Allowed', and run a crawl in Screaming Frog or Ahrefs to see which URLs your own file is blocking. Check it after every theme update, migration, or platform change, because that's when the bad line sneaks back in.
Frequently asked
Does robots.txt stop a page from appearing in Google?
No. It only stops crawling. A blocked URL can still be indexed if it's linked elsewhere—use a 'noindex' tag on a crawlable page to actually keep it out of results.
Should I block CSS and JavaScript files in robots.txt?
No. Google renders pages before ranking them, so blocking these breaks how it sees your layout and content. Leave your assets, scripts, and images fully crawlable.
Where do I find and check my robots.txt file?
It lives at yourdomain.com/robots.txt. Read it directly, then verify pages with Google Search Console's URL Inspection tool and a crawler like Screaming Frog.
How fast do robots.txt mistakes hurt rankings?
Usually not instantly. Traffic erodes over two to four weeks as cached pages go stale, which is why the damage often goes unnoticed until it's significant.
Free, concrete, yours to keep
Want this done for you — properly?
Get a free audit of your rankings and AI-answer presence. Real findings within days, no sales deck.