Blocked by robots.txt in Search Console: What to Do

What does "Blocked by robots.txt" actually mean?

Here's the short version. Your site has a small text file, robots.txt, that gives crawlers instructions about which parts of the site they may visit. Somewhere in that file, a rule matched a URL, and Googlebot obeyed it and stayed away. Search Console reports that as "Blocked by robots.txt."

That's the whole message. It says nothing about whether the page is good, bad, indexed or hidden. It says only that Google was told not to fetch it. Whether that's a problem depends entirely on whether you meant it.

Where do you see the status?

In Search Console, open Indexing, then Pages, and look under the reasons pages aren't indexed. You'll see "Blocked by robots.txt" and, sometimes, a related entry, "Indexed, though blocked by robots.txt." Click either to see sample URLs. You can also inspect an individual URL to see whether crawling is allowed.

The robots.txt report in Search Console's settings shows which robots.txt files Google found, when they were last fetched and any problems parsing them. It's the quickest way to confirm Google is reading the version you think it is.

Check the live file yourself

Type your domain followed by /robots.txt into a browser. The file is public. Read every line. It takes a minute and often explains the whole report.

Is this status always a problem?

No. Plenty of sites block things on purpose: internal search results, shopping cart steps, admin areas, endless filter combinations. Seeing those in the "blocked" list means the rules are working. The status becomes a problem only when valuable pages, such as service pages, products or articles, are caught by a rule that was too broad or left over from development.

So the first job isn't fixing. It's sorting. Which blocked URLs did you intend to block, and which are casualties?

What is robots.txt actually for?

To fix the report sensibly, you need to understand what the file can and can't do. Google's documentation spells this out, and the details surprise a lot of people.

The page, titled "Introduction to robots.txt," was last updated on 10 December 2025.

What does Google say it's for?

Google describes robots.txt as a way to manage crawler traffic, mainly to stop crawlers overloading your server, and to keep crawlers away from unimportant or near-duplicate pages. It can also stop image, video and audio files from appearing in search results, though it won't stop other sites linking to them. And it can block unimportant resource files, with a caution that Google may struggle to understand pages that depend on resources you block.

In other words, it's a traffic and efficiency tool. It's about where crawlers spend their time, not about what appears in search.

The short list of good uses

Keeping crawlers out of internal search results, cart and checkout steps, endless parameter combinations and admin sections. Reducing load from aggressive bots. Pointing to your sitemap.

What does Google say it's not for?

The same page is explicit: robots.txt is not a mechanism for keeping a web page out of Google. A page blocked by robots.txt can still appear in search results if it's linked from elsewhere, usually as a bare URL with no description. If you want a page removed from Search, Google recommends password protection or noindex.

It's also not a security measure. Compliance is voluntary, so well-behaved crawlers follow it and others don't. Anyone can read your robots.txt, which means a line like "Disallow: /secret-offers/" can advertise exactly what you hoped to hide.

Never hide sensitive content with robots.txt

If content must stay private, put it behind a login. Treat robots.txt as a polite request, not a lock.

Why can a blocked page still be indexed?

This is the part that confuses people most. If Google can't crawl a page, how can it be in the index? The answer lies in how Google builds its picture of the web.

Crawling and indexing are separate steps, and links matter to both. The difference between crawling versus indexing explains why a blocked page can still appear.

How does a URL get indexed without being crawled?

Google discovers URLs by following links. If other pages, on your site or elsewhere, link to a blocked URL, Google knows it exists. It can't fetch the content, but it can still record the address and some information from the links pointing to it, such as anchor text. In some cases that's enough for the URL to appear in results, typically without a description, because Google couldn't read the page.

That's the status Search Console calls "Indexed, though blocked by robots.txt." It's a warning that the block isn't doing what you may have wanted.

What it looks like in search

A bare URL or a title taken from link text, with a note saying no description is available because of robots.txt. It's not a good look for a business.

Why does blocking plus noindex fail?

Because noindex is something Google reads on the page. If you block crawling, Google can't fetch the page, so it never sees the noindex tag. The instruction is invisible, and the URL can stay in the index indefinitely.

Google's documentation warns that crawling and indexing rules can work against each other when combined. The sensible sequence for removing a page is the opposite of what many people try: allow crawling, add noindex, wait until Google drops the page, and only then, if you want to, block crawling.

The order matters

First noindex, then wait, then block. Doing it the other way round leaves a bare URL hanging around.

How do you diagnose a block you didn't intend?

Accidental blocks are one of the most damaging and most preventable technical SEO problems. Fortunately, they're also among the easiest to diagnose, because the evidence is in a small text file.

Work through it in a calm, structured way.

What should you check in the file?

Read every rule. Look for "Disallow: /" on its own, which blocks the entire site. Look for broad folder rules such as "Disallow: /blog/" or "Disallow: /services/". Look for wildcard patterns that match more than you expect, like "Disallow: /*?" which blocks every URL with a query string. And look at which user agents each group of rules applies to.

Compare the file with your site structure. Does any rule match a folder that holds pages you want in search? Does a rule date from a staging period and never get removed? A leftover "Disallow: /" from a pre-launch build is one of the oldest SEO accidents in existence.

Common causes of unintended blocks

A development rule copied to production. A CMS setting such as "discourage search engines from indexing this site." A security plugin that edits robots.txt. A migration that overwrote the file. A rule that was fine for one section and accidentally catches another.

How do you test specific URLs?

Use the robots.txt report and the URL inspection tool in Search Console to see whether Google is allowed to crawl a given URL and which rule applies. You can also test URLs with third-party robots.txt testers or a crawler configured to respect your rules.

Test a sample from each important section: a service page, a product, a blog post, a location page, your CSS and JavaScript files. If any are blocked and shouldn't be, you've found the damage.

Don't forget the resources

Blocking scripts or stylesheets can stop Google rendering pages properly. Google's guidance cautions that it may struggle to analyse pages that depend on blocked resources.

How do you unblock pages you want indexed?

Once you've found an unintended block, fixing it is usually a one-line change. The care goes into making sure you don't create a new problem.

Take it step by step.

What's the safe way to edit robots.txt?

Back up the current file first. Make the smallest change that achieves the goal: remove the offending rule or narrow it so it no longer matches important URLs. If you're replacing a broad rule with a narrow one, test the new version against a list of URLs you want crawled and a list you want blocked.

Upload the change, then fetch the live file in your browser and confirm it's correct. Check the robots.txt report in Search Console to see that Google has picked up the new version. Google caches the file, so the update may take a little time to be seen.

Keep a change log

Record the date, the change and the reason. Robots.txt edits are tiny and powerful, and a short note saves hours when something goes wrong later.

How do you get the pages crawled and indexed afterwards?

Once Google can fetch the pages, give it a reason to do so soon. Make sure they're in your sitemap, linked from relevant pages, and returning a normal 200 response. For key pages, use the URL inspection tool to request indexing. Then be patient: crawling and indexing don't happen instantly.

Watch the page indexing report over the following weeks. The "Blocked by robots.txt" count should fall, and the affected pages should move into the index. If they land in crawled, currently not indexed instead, that's a separate question about page value.

Recheck after any deployment

Robots.txt files get overwritten by deployments, plugins and migrations. Add a robots.txt check to your release routine.

How do you remove pages from Google properly?

Plenty of people land on this topic because they blocked a page hoping to hide it, and it's still showing. Here's how to do it properly.

The right tool depends on how sensitive the content is and how permanent the removal should be.

What are the proper options?

For pages that should never be public, put them behind authentication, such as a login. That's the only genuinely secure option. For pages that can be public but shouldn't appear in search, use a noindex robots meta tag or an X-Robots-Tag HTTP header, and allow Google to crawl the page so it can see the instruction. For pages that no longer exist, return a 404 or 410. For urgent removals, Search Console has a removals tool that temporarily hides URLs while you make the permanent fix.

Each has a purpose. A temporary removal request doesn't delete anything permanently. It buys time. The permanent solution still has to be in place.

A word on staging sites

Protect staging with a password or IP restriction. A robots.txt "Disallow: /" on staging is a courtesy, not protection, and it's easy to push to production by accident, which blocks your live site.

What if the page is already indexed as a bare URL?

If you've blocked a page and it's showing in results, remove the block, add noindex, make sure the page is crawlable and wait. Google needs to recrawl the page, see the noindex and drop it. That can take days to weeks. Once it's gone, you can decide whether to block crawling again, though often there's no need.

For pages with sensitive content already exposed, act faster: use the removals tool for a quick temporary hide, move the content behind authentication and then work through the permanent steps. Treat any leak of personal or confidential information as an urgent security issue, not just an SEO one.

Check for other copies

Content that has been indexed may also exist in cached versions, in links from other sites or in sitemaps. Clean up each source.

What are the common robots.txt mistakes?

The file is short, which makes errors easy to make and easy to miss. A handful of mistakes account for most of the trouble.

Knowing them helps you check your own file in minutes.

Which syntax mistakes cause the most harm?

A blanket "Disallow: /" left from development. Missing or misplaced wildcards that catch too much. Case mismatches, because paths are case-sensitive. Conflicting Allow and Disallow rules where you haven't considered which wins. Putting rules under the wrong user-agent group. And typos in paths so that a rule silently matches nothing, leaving content crawlable that you meant to block.

Another subtle one: forgetting that robots.txt applies per host and protocol. The rules at example.com don't govern a subdomain, which needs its own file.

Test, don't assume

A tester that shows exactly which rule matches each URL is worth its weight in gold. Use one every time you edit the file.

Which strategic mistakes cause the most harm?

Blocking CSS and JavaScript files that pages need to render. Blocking whole sections to "save crawl budget" on small sites that don't need it. Using robots.txt to try to consolidate duplicates when canonical tags or redirects are the right tool. Blocking pages that have valuable links pointing to them, which stops Google passing value through them.

Another is relying on robots.txt for private content. It's the single most repeated misunderstanding, and it can expose exactly what you wanted hidden. Better ways to handle duplication include canonical tags and duplicate URLs rather than blocking.

The question to ask before every rule

"What problem am I solving, and is crawling the right place to solve it?" If the answer is "I want it out of Google," then robots.txt isn't the answer.

How do you build a safe robots.txt for a business site?

Most small business sites need very little. A short, careful file beats a long, clever one.

Start from the principle that the less you block, the less you can break.

What does a simple, safe file look like?

For a typical small business site, the file may allow everything except a few areas that offer no value in search: admin paths, internal search results, cart and checkout steps. It points to the XML sitemap. It doesn't try to micro-manage individual pages or hide private content.

Keep the structure readable, with comments explaining why each rule exists. A rule without a reason is a future mystery. If you're unsure about a rule, leave it out and handle the issue another way.

Always include the sitemap line

A line pointing to your sitemap helps crawlers find it. It's a simple, harmless addition that most sites benefit from.

How do you maintain it over time?

Review it when the site changes: after a redesign, platform change, migration or new plugin. Compare the live file with your documented version. Test a handful of key URLs. Add the file to your launch checklist so it's checked before and after every significant release.

If several people can edit the file, agree who owns it. Many incidents come from well-meaning edits by someone who didn't realise the consequences. A short, written owner and process prevent most of them.

What does a real-world robots.txt incident look like?

Nothing teaches like a story, so here's a composite. It's hypothetical, but every element has happened to real sites.

The names are invented, and the pattern is familiar.

What happened?

A professional services firm launches a redesign over a weekend. On Monday, enquiries are normal. By the following Friday, organic traffic has dropped sharply and a handful of key service pages no longer appear in search. Nobody has touched the content. The marketing manager checks Search Console and finds hundreds of URLs reported as "Blocked by robots.txt."

A look at the live file shows a single line, "Disallow: /", left from the staging site and copied across during launch. Google had been politely staying away since the day the site went live.

The cost

A week of lost visibility during a busy period, plus the time it takes for Google to recrawl and reassess once the block is removed.

How was it fixed?

The developer removed the rule, uploaded the corrected file and confirmed it by loading the live URL. The team checked the robots.txt report in Search Console, requested indexing for the five most important pages and watched the blocked count fall over the following days. They added a robots.txt check to the launch checklist and set up an alert for changes to the file.

Traffic recovered over the next few weeks. The lesson isn't that robots.txt is dangerous. It's that a tiny file with enormous power deserves a place in every launch checklist, and in the hands of someone who understands what each line does. A technical SEO consultant can review yours as part of robots.txt, crawling and indexing checks.

What to take from it

Check robots.txt before and after every release. It takes thirty seconds, and it can save a month.

How does robots.txt interact with noindex, canonicals and sitemaps?

These tools get mixed up constantly, and the confusion is the root of many indexing headaches. They don't compete. They do different jobs, and they can contradict one another if you aren't careful.

A simple way to hold it in your head: robots.txt says where crawlers may go, noindex says what may appear, canonical says which version to prefer, and the sitemap says what you'd like found.

What happens when the signals conflict?

If a URL is in your sitemap but blocked in robots.txt, you're asking Google to find a page and forbidding it from visiting. Search Console will often flag the mismatch. If a page is blocked and also carries a canonical tag pointing to another URL, Google can't read the canonical because it can't fetch the page. If a page is noindexed and linked prominently from every menu, you're telling Google it matters and shouldn't appear at once.

Conflicts don't usually cause disasters. They cause unpredictability, and unpredictability is exactly what you don't want when you're trying to diagnose a problem. Clean, consistent signals make Google's choices easier and your reports easier to read.

A consistency checklist

Every URL in the sitemap should be crawlable, indexable, canonical to itself and return a 200 status. Every URL you want out of the index should be crawlable until it has gone, and then either blocked, removed or protected. Every blocked URL should have a reason.

Where do the Google tools fit in?

Search Console's robots.txt report shows which version Google fetched and any errors. The URL inspection tool shows whether crawling is allowed and which canonical Google chose. The page indexing report shows the outcome across the whole site. Together they let you check whether your signals are being read the way you intended.

Use them as a loop: make a change, check the tools, wait, check again. Knowing how crawling versus indexing relate makes the reports easier to interpret.

What should you do with specific kinds of blocked URL?

Not all blocked URLs deserve the same treatment. Group them first, then decide what each group needs.

Here are the groups we see most often.

What about internal search, filters and parameters?

Usually leave them blocked. These pages create near-infinite combinations of low-value URLs and are exactly what robots.txt is good at keeping out of the crawl path. Make sure no important page depends on them, and don't link to them prominently from your navigation. If some filter pages have real search demand, handle those deliberately with proper landing pages instead of relying on the filters.

For shops, faceted navigation and filter indexing decides which combinations deserve crawling.

The cart and checkout rule

Cart, checkout and account pages have no place in search. Blocking crawling is sensible, and adding noindex to those pages as well is harmless where the platform allows it.

What about CSS, JavaScript and images?

Be careful. Google's guidance warns that blocking resource files can make it harder to understand pages that depend on them. If a stylesheet or script is blocked, Google may not render the page as users see it, which can affect how it interprets layout, content and mobile friendliness.

The safe default is to let Google crawl the resources your pages need. If you want to keep some images out of search results, robots.txt can do that, but remember it won't stop other sites linking to them.

Test with rendering

Use the URL inspection tool's live test to see the rendered page and any resources Google couldn't load. If something important is missing, review your rules.

What about old URLs and removed sections?

Don't block them. Let them return a 404 or 410, or redirect them properly, so Google can see the status and update its index. Blocking a removed URL stops Google seeing that it's gone, and the stale entry can linger. Removed content is a job for status codes, not robots.txt.

If you've blocked an old section during a migration, revisit it. A rule that made sense during the launch can quietly keep the old URLs alive in the index long after you meant them to disappear. Moved content is handled cleanly with 301 redirects and redirect mapping.

When blocking is genuinely right

When a section generates enormous numbers of useless URLs and nothing else is practical. In that case, block it, make sure nothing important links there and monitor the result.

How do you monitor robots.txt so it doesn't bite you later?

A file this small should be easy to watch, and yet most sites don't. Ten minutes of monitoring setup is cheap insurance.

The goal is to find out about an unwanted change within a day, not a month.

What can you automate?

Save a copy of the approved file in version control or a shared document. Use a monitoring tool or a simple scheduled check to fetch the live file daily and alert you if it changes. Set up Search Console email alerts and check the robots.txt report after deployments. Add the file to your release checklist so a person looks at it before and after every launch.

For agencies and multi-site owners, a lightweight dashboard that shows the current status of each client's file can catch a rogue "Disallow: /" within hours.

What a useful alert says

"robots.txt on example.com changed at 14:02. Old version attached. New version attached. Difference highlighted." It tells you what changed, when and how, which is all you need to act.

Who should own it?

One named person or team, with a documented change process. Everybody who can edit the file should know the rules: back up first, change one thing at a time, test, log. If a plugin or platform modifies the file automatically, understand when and why, and check it after updates.

If you're not sure where your site stands, an independent SEO audit will include a review of robots.txt alongside sitemaps, indexing and crawl behaviour, so you know exactly what's blocked and why.

What should a quarterly robots.txt review cover?

Four things, in about fifteen minutes. Read the live file line by line and compare it with your approved copy. Test five or six important URLs, such as the homepage, a service page, a product or article, and your main stylesheet. Check the robots.txt report in Search Console for fetch errors or warnings. And scan the page indexing report for any new "Blocked by robots.txt" entries that look important.

Write the result down, even if the answer is "all fine." A short log of reviews makes the next investigation faster, and it shows anyone who inherits the site that someone has been paying attention. It also builds a habit, which is what really protects a small file with a very large effect.

What to do when you find something odd

Don't rush to edit. Note it, work out why it's there, check who added it and what it affects, and then decide. Some strange-looking rules turn out to protect something you'd forgotten about, and removing them blindly can create a fresh problem.