Robots.txt mistakes that block Google from crawling your site

Your site's robots.txt file is one of the smallest files you have, and one of the easiest to get wrong. A single stray character can ask search engines to ignore your whole website. This guide is for small business owners and the person who looks after their website. It explains the mistakes that cause the most damage and shows you how to check your own file in a few minutes, without breaking anything.

In short: robots.txt controls what compliant crawlers may fetch, not what appears in search results. Check that it doesn't contain Disallow: /, doesn't block the CSS and JavaScript your pages need, and isn't being used to hide pages from Google. Then confirm what Google actually sees in Search Console.

What robots.txt does (and doesn't do)

robots.txt is a plain text file at the root of your site, such as https://yourbusiness.com/robots.txt. Google, Bing and other well-behaved crawlers read it before fetching pages and follow its requests.

Five goals matched to the right control: robots.txt for wasted crawling, noindex on a crawlable page, login for private content, 404 or 410 for deleted pages, canonical tag for duplicates.

Three points explain most of the trouble below:

It controls crawling, not indexing. If a URL is blocked, Google can't read the page. But the URL can still show up in results if other pages link to it, often with no description.
It is not security. Anyone can read the file, and badly behaved bots ignore it. Never use it to hide admin areas or private files.
Google must crawl a page to see its noindex tag or canonical link. Block the page and Google can't see either.

Pick the control that matches your goal, and note why it fits:

Goal	Right control	Why
Stop compliant bots wasting time on low-value URLs	`robots.txt`	It stops the fetch, so crawlers spend their time elsewhere
Keep a public page out of search results	`noindex` on a crawlable page	Google has to read the page to see the instruction
Protect private content	Login or other authentication	Only access control actually keeps people out
Remove a page that no longer exists	`404` or `410` status	It tells crawlers the page is gone
Say which duplicate is the main one	Canonical tag	It points Google to the version you prefer

What a healthy file looks like

For most small business sites, this is enough:

A healthy robots.txt with three lines: User-agent star, an empty Disallow, and a Sitemap line, each with a note: Googlebot group overrides, empty Disallow blocks nothing, Sitemap points to your pages.

User-agent: *
Disallow:

Sitemap: https://yourbusiness.com/sitemap.xml

An empty Disallow: means nothing is blocked. The Sitemap: line tells crawlers where your list of pages lives. It isn't required, but it is a useful hint. Use the full URL, and check that the sitemap itself loads and isn't blocked by a rule.

If your file contains a separate User-agent: Googlebot group, Googlebot follows only that group and ignores the * group, so read the group that applies to the crawler you care about.

The six mistakes

Checklist of the six robots.txt mistakes: Disallow slash on the live site, blocking CSS and JavaScript, using it to remove pages, case and slash slips, blocking parameters, wrong location or response.

1. `Disallow: /` on the live site

User-agent: *
Disallow: /

That one slash asks every compliant crawler to stay out of the entire site. It is often a leftover from a staging or "coming soon" version that was copied across when the site went live.

Fix: change it to an empty Disallow: (or remove the line), then check how the file is generated so the next deployment doesn't bring it back. Afterwards, submit your sitemap in Google Search Console. Recovery depends on when Google chooses to crawl again, so don't expect an instant change.

2. Blocking CSS, JavaScript or images

Older files sometimes block folders such as /wp-includes/, /wp-content/plugins/ or /assets/js/. Google renders pages much as a browser does. If it can't load the files that build your layout and content, it may see a stripped-down page. This matters even more for sites that build content with JavaScript, which we cover in why your React site might be invisible to Google.

Fix: don't block folders that hold files your public pages need. Before deleting an old rule, check which URLs it covers, in case it was added for another reason.

3. Using robots.txt to remove pages from Google

Adding a page to Disallow feels like "hide this page". It isn't. Google no longer supports a noindex rule inside robots.txt (support ended in September 2019), so that doesn't work either. A blocked page that is already known can stay in results, and Google can no longer read your content or any noindex instruction on it.

The right way to remove a public page:

Make sure the page is not blocked in robots.txt.
Add <meta name="robots" content="noindex"> to the page's <head>, or send an X-Robots-Tag: noindex HTTP header.
Leave the page crawlable until it has dropped out of results.

For deleted pages, return a 404 or 410. For anything private, use a login. The Removals tool in Search Console can hide a URL temporarily while you apply the permanent fix.

Watch out: if a page holds private information, noindex is not protection. Anyone with the link can still open it. Put it behind a login or take it offline.

4. Case and trailing-slash slips

Paths in robots.txt are case-sensitive and match from the start of the URL.

Disallow: /Private/

This blocks /Private/ but not /private/. And this rule is broader than it looks:

Disallow: /thanks

It matches /thanks, /thanks.html, /thanks-you and /thanks/order-123. Use /thanks/ for the folder only. Google also supports * (wildcard) and $ (end of URL), so Disallow: /thanks$ blocks just that exact path. Not every crawler understands * and $.

5. Blocking parameters you want crawled

Filter and sort options can create thousands of near-duplicate URLs, so owners are tempted to write:

Disallow: /*?

That blocks every URL containing a question mark, including campaign links and paginated pages such as ?page=2. Be specific instead, and remember that a parameter can appear first or later in the query string:

Disallow: /*?sort=
Disallow: /*&sort=

6. Wrong location or wrong response

Rules apply only to the exact protocol, host and port that serve the file:

https://yourbusiness.com/robots.txt covers https://yourbusiness.com/ only.
https://www.yourbusiness.com/ needs its own file (or a redirect to your main version).
A file at https://yourbusiness.com/blog/robots.txt is ignored.

The status code matters too. A working file should return 200. Google treats a missing file (404) as "no restrictions", but server errors (5xx) or 429 responses may make Google pause crawling temporarily. If your file is slow or erroring, ask your host to look at it.

A copy-ready example

This is a starting point, not something to paste blindly. Folder names differ between platforms and themes, so check the real URLs on your own site first.

A shop that keeps internal search results, cart and checkout pages out of the crawl:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=
Disallow: /*&s=

Sitemap: https://yourbusiness.com/sitemap.xml

Notice what it doesn't block: product pages, category pages, images, CSS and JavaScript.

Warning: test every rule against real URLs before you publish it. Open a few URLs you want crawled and a few you don't, and confirm each rule affects only what you intended. If your CMS or an SEO plugin generates the file, make the change there, or your edit may be overwritten.

How to check yours safely

Save a copy of the current file and find out where it comes from: written by hand, your CMS, an SEO plugin or your hosting. Fix the source, not just the output.
Read it at https://yourbusiness.com/robots.txt. Check the www and non-www versions too.
Open the robots.txt report in Google Search Console (under Settings). It shows the version Google last fetched and any problems.
Test real pages with the URL Inspection tool: your home page, a service page, a blog post and a product or category page. Look for whether crawling is allowed.
Open the Page indexing report (under Indexing) and look for the status "Blocked by robots.txt". Make sure nothing important is listed.
After any change, reload the file in your browser and confirm the live contents.

Tip: "Crawl allowed" doesn't mean "will be indexed". If a page still isn't showing up, look for noindex tags, redirects, canonical tags pointing elsewhere and login walls. Our guide to how to read an SEO audit helps you decide which findings matter first.

Key takeaways

robots.txt manages crawling. It doesn't reliably remove pages from search and it isn't security.
Look for Disallow: / first, especially after a redesign, migration or launch.
Don't block CSS, JavaScript or images that your public pages use.
To keep a page out of search, leave it crawlable and add noindex.
Test every rule against real URLs, remembering case, slashes and parameters.
Save a copy before editing, and confirm the result in Search Console.

FAQ

Do I need a robots.txt file at all?

No. Without one, compliant crawlers assume nothing is restricted. It's useful if you want to steer crawlers away from certain URLs or point to your sitemap.

Will fixing robots.txt bring my pages back straight away?

Not necessarily. Google has to crawl the file and your pages again, and that happens on its own schedule. Fix the problem, confirm it in Search Console, and give it time.

Can I block AI crawlers or scrapers with robots.txt?

You can ask compliant ones to stay away by naming their user agents, but it's a request, not enforcement. Bots that ignore it need blocking at the server or firewall level.

For the official details, see Google's guides to robots.txt and to blocking indexing with noindex.