Google Search Console Crawl Stats Report: How to Read It and Act on It

What is the Crawl Stats report, and why does it exist?

Most Search Console reports tell you what Google decided: indexed, not indexed, ranked, clicked. The Crawl Stats report is different. It tells you what Google did. It's a log of Googlebot's visits to your site, summarised into charts, and it answers a question no other report does: how is Google actually treating your server?

For most small business sites, the answer is "fine, thanks," and you can safely ignore it. For larger sites, or sites with odd traffic patterns and mysterious indexing delays, it's one of the most valuable free diagnostics available.

What does Google say about it?

Google's help page, Crawl Stats report, describes the report as showing Google's crawling history for your site: how many requests were made and when, what responses your server gave and any availability problems. It describes the report as aimed at advanced users, and says sites with fewer than about 1,000 pages generally don't need it.

It's also limited to root-level properties. If your Search Console property is a Domain property, or a URL-prefix property at the root of the site, you'll see it. If it's a subfolder, you won't.

How to open it

In Search Console, go to Settings and open Crawl stats. The report shows the last 90 days of data, with charts you can expand.

Who should use it?

Owners of larger sites, ecommerce stores, publishers and anyone dealing with indexing delays, server problems or suspected crawl waste. Developers and technical SEOs use it to verify that fixes have worked and that releases haven't caused trouble. Smaller sites might look at it once after a migration or a hosting change, as a health check.

If you're just starting out, treat it as a reference tool, not a daily dashboard. Learn what normal looks like, so that you can spot abnormal.

A simple rule

If your site has a few dozen pages and indexing is healthy, the page indexing report is the better place to spend your time.

What are the main sections of the report?

The report has a summary at the top and several breakdowns below. Each shows a different slice of the same crawling activity.

Learn the sections and the report becomes easy to read.

What does the top-level summary show?

Three headline charts. Total crawl requests counts every crawl attempt, successful or not, including failed DNS lookups, connectivity failures, unavailable robots.txt fetches and abandoned redirect loops. Total download size shows the bytes Google downloaded from your site, with cached resources counted only on first request. Average response time is the mean response time across the resources fetched.

Beneath those sits host status, which indicates availability problems over the past 90 days across robots.txt fetching, DNS resolution and server connectivity. A healthy site shows green. Warnings point to infrastructure issues that need attention.

Read the charts together

A spike in requests with a rising response time can mean Google is hammering a slow server. A drop in requests with rising errors can mean Google backed off.

What do the breakdowns show?

By response: the percentage of responses by type, such as 200 OK, redirects, 404s, 5xx errors and robots.txt unavailable. Most should be successful. By file type: HTML, images, JavaScript, CSS, PDF, video and others. By purpose: discovery, meaning a URL Google hadn't crawled before, versus refresh, meaning a recrawl of a known URL. By Googlebot type: smartphone, desktop, image, video, page resource load, AdsBot or StoreBot.

Each breakdown includes sample URLs, which are representative rather than complete. Click through to see examples of what Google fetched, and use them as leads.

How do you read host status?

Host status is the early-warning light on the dashboard. It summarises whether Google has had trouble reaching your site in three areas that all have to work: robots.txt, DNS and server connectivity.

A problem in any of them can stall crawling, so it's the first thing to check.

What are the three checks?

Robots.txt fetching: can Google retrieve your robots.txt file? If the file is unavailable for a long time, Google may become cautious about crawling. DNS resolution: can Google translate your domain name into an address? Failures here make the whole site unreachable. Server connectivity: can Google connect to the server and get a response without timing out?

Google's help page describes host status as an indicator of availability issues over the past 90 days, assessed across those three. Each is shown with a status, and clicking in shows details and dates.

What the colours mean

Green indicates no significant issues. Amber or red indicates problems in the period, with details about when and how often.

What do you do about a warning?

Look at the dates. Did the problem coincide with a deployment, a hosting change, a DNS update or a traffic surge? Check your hosting provider's status and logs for outages. Make sure robots.txt returns a 200 status and loads quickly. Confirm that DNS records are correct and that your provider is reliable.

If problems are recurring, discuss capacity and stability with your host. A site that sometimes disappears from view is hard for Google to rely on, and consistent availability is the foundation of consistent crawling. The robots.txt side of things is covered in the guide to writing a robots.txt file.

Transient blips are normal

A single short outage rarely matters. Patterns and long durations do.

What do the response codes tell you?

The "By response" breakdown is the most actionable part of the report for most people. It shows what kinds of answers your server gave Google, and what share of crawling went to each.

A healthy site is dominated by 200 responses.

Which codes should worry you?

A rising share of 5xx server errors suggests the server is failing under load or misconfigured. A high share of 404s suggests Google is requesting URLs that don't exist, perhaps from old links, a crawl trap or a sitemap listing dead pages. A lot of 301 and 302 redirects suggests the site links to old URLs or has chains. A large share of 304 not modified is good news, meaning unchanged pages were served efficiently.

Also watch for "robots.txt not available" and 429 or 503 responses, which Google may treat as overload signals and respond to by crawling less.

Sample URLs are your best clue

Click into each category and read the examples. Patterns, such as parameter URLs, old paths or a particular folder, show where the waste or the failure is.

How do you act on what you see?

For 404s: fix or remove internal links, clean the sitemap and decide whether to redirect valuable old URLs. The right handling of missing pages is covered in the guide to soft 404 errors. For redirects: update internal links to point straight to final URLs. For 5xx errors: investigate server logs, capacity and plugins. For robots.txt issues: ensure the file is stable and fast.

After changes, watch the report over the next weeks. The mix should shift toward 200 responses, and error spikes should disappear.

Don't chase every 404

A few 404s are normal. Google's documentation treats 404s as a clean signal that a page is gone. Focus on patterns and on pages you want to exist.

What do response time and download size reveal?

Speed and size are two sides of server efficiency. They influence how much Google is willing to crawl, and they affect the visitors who use the same server.

Look for trends, not single numbers.

Why does average response time matter?

Google adjusts its crawling to what your server can cope with. Its crawl documentation explains that capacity rises when a site responds quickly and consistently and falls when responses are slow, when 5xx errors appear or when the server signals overload. A rising response time in the report is therefore worth investigating, because it can lead to less crawling.

Compare the chart with your own monitoring. If response times spike at the same moments as releases, backups or traffic peaks, you've found a cause. Caching, database optimisation, a content delivery network and better hosting are the usual remedies.

A useful benchmark

Record your normal average response time when things are healthy, so that you can recognise drift early.

What about download size?

Total download size shows how many bytes Googlebot pulled from your site. Large spikes can come from heavy files, such as big images, PDFs or scripts, or from a surge in the number of URLs crawled. Use the file type breakdown to see which categories dominate.

If images or PDFs account for most of the bandwidth, ask whether they're optimised and whether all of them need to be crawled. If JavaScript or CSS dominates, check for unnecessary or duplicated resources. Smaller, well-cached files mean faster pages and cheaper crawling.

What do file type, purpose and Googlebot type add?

These three breakdowns give context, and they're often where unusual patterns reveal themselves. They answer "what is Google fetching, why and as whom?"

A quick look can save hours of guessing.

How do you read the file type breakdown?

The report splits requests by resource type: HTML, images, JavaScript, CSS, PDF, video and others. For most sites, HTML should make up a healthy share of requests, with page resources such as scripts and styles alongside. If JavaScript or image requests dominate unexpectedly, look at how pages load resources and whether something is generating needless fetches.

A site where PDF requests dwarf HTML, for instance, may have a large archive of documents that Google is crawling heavily. That may be fine, or it may be wasted effort, depending on whether those PDFs matter to you.

Check rendering implications

Many requests for scripts and styles are normal for rendering. If they fail or are blocked, pages can render badly. The guide to JavaScript SEO and rendering checks explains how to test.

What do purpose and Googlebot type tell you?

Crawl purpose splits requests into discovery, meaning URLs Google hadn't crawled before, and refresh, meaning recrawls of known URLs. A healthy mature site is mostly refresh. A surge in discovery can mean you've launched a lot of new content, or that your site is generating new URLs, perhaps through filters or parameters.

Googlebot type shows which crawler made the requests. Because Google primarily uses mobile crawling for indexing, a large share from the smartphone crawler is normal. If you see a lot from a type you don't expect, check why.

Sudden discovery spikes

If discovery spikes without a corresponding content launch, look at sample URLs for parameters, session IDs or calendar-like patterns. It may be a crawl trap.

How do you spot a crawl problem?

Beyond individual charts, certain patterns point to problems. Learning the shapes helps you diagnose faster.

Here are the common ones and what they usually mean.

What do spikes and drops indicate?

A spike in total requests after a launch, migration or sitemap update is normal. A spike with no obvious cause suggests new URLs have appeared, often from filters, parameters or a bug. A sudden drop in requests alongside a rise in errors or response time suggests Google has backed off because of server trouble. A steady decline with no errors may reflect reduced crawl demand, which links to perceived quality and freshness.

Look at the sample URLs and the response breakdown for each shift. The cause is usually visible within a few clicks.

Compare against your change log

Overlay the dates of releases, plugin updates, migrations and hosting changes. Most crawl anomalies line up with something you did.

How does this connect to indexing statuses?

If many URLs are stuck as "Discovered, currently not indexed," the crawl stats may show slow response times, errors or a flood of low-value URLs consuming Google's attention. The explanation in the guide to discovered, currently not indexed shows how crawl capacity and demand interact, and the Crawl Stats report is the place to see capacity in action.

Likewise, if important pages are slow to refresh, check whether crawl is dominated by unimportant URLs. Cleaning up the clutter is often the most effective fix.

Not every delay is a crawl issue

Some indexing delays reflect quality or duplication rather than crawling. The Crawl Stats report helps rule capacity in or out.

What are the report's limitations?

Like every tool, the report has limits. Knowing them stops you drawing the wrong conclusions.

Google's help page is open about them.

Where does the data fall short?

Data covers only the selected domain. Requests to other domains, and resources hosted elsewhere, such as a content delivery network on a different domain, are excluded. Example URLs are representative samples, not a complete list. Some requests may go uncounted, so figures can differ from server logs. Totals can also run higher than logs when robots.txt is unavailable for an extended period.

The report also covers a limited window, so you'll need to export or screenshot data if you want a long-term record. Don't treat it as an audit trail.

Use logs for detail

If you need exact counts and full URL lists, server logs are the reliable source. The report is for trends and orientation.

How do you combine it with other data?

Pair it with the page indexing report, the Sitemaps report and your server logs. The indexing report shows outcomes. The Crawl Stats report shows behaviour. Logs show every request. Together they give a complete picture, as described in the guide to the technical SEO audit checklist.

For larger investigations, a specialist can analyse logs alongside the report to see which sections Googlebot favours, which it ignores and where crawl is being wasted, and then brief developers on the fixes.

How do you use the report in a routine?

You don't need to open the report daily. A light routine catches problems and gives you a baseline.

Fit it into your monthly technical checks.

What does a monthly check look like?

Open the report and look at the last 90 days. Check host status for any new warnings. Note total requests, average response time and the response code mix. Scan for sudden spikes or drops. Click into any category that looks different from last month, and read the sample URLs. Record the numbers in a log with notes on events such as releases.

Ten minutes is enough. If something looks odd, dig in the same week, while the cause is fresh.

Take screenshots or exports

Save the charts each month. The report's rolling window means older data disappears, and a record lets you compare year-on-year.

When should you act immediately?

When host status shows a failure and key pages are unreachable. When 5xx errors spike. When response times jump and stay high. When requests collapse suddenly. When a flood of unexpected URLs appears. In any of those cases, check server health, recent changes and the sample URLs, and involve your developer or host straight away.

Quick action matters because Google adapts to what it sees. A site that recovers promptly usually returns to normal crawling quickly, while a site that stays broken may see crawl reduce for longer.

How does the report help after migrations and big changes?

Major changes are when crawl behaviour matters most. A migration, redesign or platform change can transform the URL set overnight, and the Crawl Stats report is a window into how Google reacts.

Use it as part of your post-launch monitoring.

What should you watch after a migration?

Expect a rise in discovery and in total requests as Google finds new URLs and follows redirects. Watch the response codes: a healthy migration shows a burst of 301s, then a shift toward 200s on the new URLs as old ones fade. Watch for 404s, which can indicate missing redirects. Watch response times, because a new platform may be slower under crawl load.

Compare against your pre-launch baseline and your redirect map. If you see lots of 404s on URLs that should redirect, fix the map straight away. The full sequence is in the website migration SEO checklist.

Give it time

Crawl patterns settle over weeks. Don't over-interpret day-to-day swings in the first days.

What about new content and large launches?

If you publish many pages at once, expect a discovery spike, and watch whether response times hold up. Make sure the sitemap is updated with accurate dates and that internal links point to the new pages, so Google finds them efficiently. If response times rise sharply, consider caching or staggering the launch.

For publishers and shops that add content constantly, a steady pattern of discovery and refresh with stable response times is the sign of a healthy site.

What does a worked example look like?

An example shows how the pieces fit together. This one is hypothetical, built from typical patterns, with illustrative details.

It follows a mid-sized online shop.

What was the problem?

An online shop with several thousand products noticed that new items took weeks to appear in search. Search Console showed many URLs as discovered but not indexed. The owner opened the Crawl Stats report for the first time.

Total crawl requests were high, but average response time had been climbing for a month. The response breakdown showed a large share of 301 and 404 responses, and the sample URLs were filter and sort combinations. The purpose chart showed a high proportion of discovery requests, meaning Google was constantly finding new URLs.

The picture

Google was spending its crawl on endless filter URLs from a slow server, leaving little attention for new products.

How did they fix it?

The developer limited which filter combinations generated crawlable links, blocked low-value patterns in robots.txt after confirming they weren't needed in the index, fixed internal links that pointed to redirecting URLs and cleaned the sitemap. The host upgraded the server plan and added caching, which brought response times down.

Over the following weeks, the share of 200 responses rose, discovery requests fell, response time stabilised and new products began to be crawled and indexed within days. The Crawl Stats report didn't fix anything itself, but it pointed to exactly where to look. If you'd like help interpreting it for a larger site, a technical SEO consultant can combine the report with logs and write the developer tickets, and an independent SEO audit can place it alongside the rest of the technical picture.

The takeaway

Watch host status, response codes and response time. Investigate spikes with the sample URLs. Fix the cause, then check the charts return to normal.

Which mistakes lead people to misread crawl stats?

Because the report looks like a performance dashboard, it invites over-interpretation. A few common misreadings account for most of the wasted effort.

Keep these in mind before you act on a chart.

Why isn't "more crawling" always better?

More requests aren't a goal. A site that's crawled heavily isn't necessarily ranked more highly, and a site crawled lightly isn't necessarily neglected. Crawling is a means to an end: keeping Google's picture of your site fresh and complete. Small, stable sites often have modest crawl and perfectly healthy indexing.

The question to ask is whether important pages are discovered and refreshed promptly. If they are, the absolute numbers matter little. If they aren't, look for the reason in the response codes, response times and URL patterns, not just the request count.

Don't compare with other sites

Crawl levels depend on site size, update frequency and perceived value. Another site's numbers say nothing about yours.

Why shouldn't you read every spike as a crisis?

Crawl volume naturally varies. Launches, sitemap changes, algorithm updates and seasonal content can all move it. A spike that clears within days, with no rise in errors, is usually harmless. A spike that persists, comes with growing response times or fills with strange URLs deserves attention.

Compare against your change log and look at the sample URLs before deciding. The cause is usually something you did, such as publishing a batch of pages or changing a plugin.

Look for patterns over weeks

A single day's chart is noisy. Trends across a month tell the real story.

Why isn't the report a substitute for logs?

Google's help page notes that figures can differ from server logs, because the report covers only the selected domain, example URLs are representative samples and some requests may go uncounted. If you need exact numbers, or want to see every URL Googlebot requested, server logs are the authoritative source.

Use the report for orientation and trends, then go to the logs for detail when something needs proof. The two are complementary.

How can small business sites use it without overcomplicating things?

If your site has a few hundred pages, the report won't change your life. It can still be a useful occasional health check, and it teaches you what normal looks like.

Keep the use light and purposeful.

What's a sensible light-touch approach?

Open it once a quarter and after any hosting change or major release. Check that host status is healthy, that response times are steady and that most responses are 200s. Skim the file type and purpose charts for anything strange. Note the numbers in a simple log. That's it.

If everything looks normal, close the tab and go back to the things that move the needle for a small site: strong service pages, good internal links, accurate local details and honest reviews. The page indexing report and the guide to checking whether a page is indexed by Google will tell you far more about day-to-day visibility.

When a small site should look closer

After a migration, if the site feels slow, if your host reports problems or if new pages suddenly stop being discovered. Those are the moments the report earns its keep.

What should a small site do with unusual findings?

If the report shows frequent server errors or slow responses, talk to your host. If it shows lots of requests for URLs you don't recognise, check for plugins, parameters or a security issue. If it shows a surge in redirects or 404s after a change, review the redirect map and internal links.

Write down what you find, what you changed and what happened next. A short note per incident gives you a history to learn from, and it makes any future conversation with a developer or consultant far more productive. When the cause isn't obvious, an independent SEO audit can untangle it.

How do you explain the report to non-technical colleagues?

Use plain comparisons. Total requests are how often Google knocked on the door. Response time is how quickly someone answered. Response codes are what they said when they answered: here's the page, it moved, it's gone or sorry, we're broken. Host status is whether the door was reliably open.

Framed that way, a manager can follow a short update such as "Google knocked on our door about the same number of times as last month, we answered a little slower, and a small share of answers were 'it's gone'." Then you can explain what you're doing about the slowness and the missing pages. Clear explanations get decisions made and budgets approved, which matters more than perfect technical vocabulary.

Keep a one-slide summary

Three numbers, one chart and one action. That's all a busy decision maker needs from a crawl review.