Independent SEO consultant serving London & the UK+44 7472 849630info@seoconsultantlondon.onlineGoogle Business Profile
  1. Home
  2. Guides
  3. Crawling vs Indexing: Diagnose Missing Pages
SEO guide

Crawling vs Indexing: Diagnose Missing Pages

Crawling is when Google fetches a page; indexing is when it decides to store that page so it can appear in results. A page can be discovered but not crawled, crawled but not indexed, or indexed but not ranking. This guide shows how to tell which stage is failing and how to fix it.

  • Evidence-led recommendations
  • Plain-English reporting
  • Built for London and UK markets
Entity relationship diagram for Crawling vs Indexing: Diagnose Missing Pages

Quick answer

  • Crawling fetches a URL; indexing stores and understands it; ranking orders indexed pages for a query.
  • Use the Google URL Inspection tool and Search Console Pages report to find the failing stage.
  • “Discovered, currently not indexed” and “Crawled, currently not indexed” mean different things.
  • Fixes range from removing blocks to improving content quality and internal links.

What crawling is in SEO

Google crawling is the process by which Googlebot finds URLs and fetches them. Discovery happens through links, sitemaps and previous knowledge of your site. Once Google has a URL, it decides whether and when to fetch it, based on factors such as site speed, server responses and how important the page appears.

If a page is blocked by robots.txt, returns an error or is too slow, crawling may fail or be limited. Understanding what is crawling in SEO helps you check the first link in the chain before looking for deeper problems.

Checklist graphic for What crawling is in SEO (Crawling vs Indexing: Diagnose Missing Pages)
What crawling is in SEO

What indexing is in SEO

Indexing is what happens after a crawl. Google processes the content, renders it where needed, understands the topic and decides whether to add the page to its index. Only indexed pages can appear in search results. Google may choose not to index pages it considers duplicate, low value or blocked by a noindex directive.

So crawling and indexing are separate steps, and a page can pass one and fail the other. Ranking is a third step, which applies only to indexed pages.

Phased roadmap timeline for What indexing is in SEO (Crawling vs Indexing: Diagnose Missing Pages)
What indexing is in SEO

Reading Search Console index statuses

The Pages report in Google Search Console gives reasons. These are the common ones and what they usually mean.

StatusMeaningTypical action
Discovered, currently not indexedGoogle knows the URL but has not crawled it yetImprove internal links, site speed and page importance
Crawled, currently not indexedGoogle fetched the page but chose not to index itImprove uniqueness, quality and usefulness
Excluded by noindexA noindex tag or header prevents indexingRemove it if the page should rank
Duplicate, Google chose a different canonicalGoogle treats another URL as the main versionReview canonicals; see canonical tags
Blocked by robots.txtCrawling is disallowedAllow crawling if the page should be indexed
Search results page mock-up for Reading Search Console index statuses (Crawling vs Indexing: Diagnose Missing Pages)
Reading Search Console index statuses

Why is my page not indexed? A diagnostic sequence

Work through these steps in order.

  1. Check the page returns a 200 status and is not a redirect or error.
  2. Check robots.txt and meta robots or X-Robots-Tag for blocks.
  3. Use URL Inspection to see Google’s view, including the rendered page.
  4. Confirm the canonical is the page itself, or a deliberate alternative.
  5. Check discoverability: is the URL in your XML sitemap and linked from other pages?
  6. Assess quality: is the content unique, useful and not a near-duplicate?
Internal link network diagram for Why is my page not indexed? A diagnostic sequence (Crawling vs Indexing: Diagnose Missing Pages)
Why is my page not indexed? A diagnostic sequence

How to check if a page is indexed

The most reliable method is the Google URL Inspection tool within Search Console, which reports whether a URL is on Google, the last crawl date and the canonical Google selected. A site: search gives only a rough impression. For many URLs, use the Pages report and filter by status, then export examples to find patterns by template or section.

A live test in URL Inspection shows whether Google can fetch and render the current version, which is useful after you have fixed an issue and want to request indexing.

Reporting dashboard graphic for How to check if a page is indexed (Crawling vs Indexing: Diagnose Missing Pages)
How to check if a page is indexed

Fixing website indexing issues

Fixes depend on the cause. Remove accidental blocks, correct canonicals, repair server errors and add internal links from relevant, strong pages so Google can find and value the URL. For quality-related exclusions, improve the page so it adds something that existing indexed pages do not, or merge it with a stronger page. Large sites should also reduce crawl waste from parameters and filters, as described in faceted navigation and filter indexing.

Requesting indexing in Search Console can help speed up recrawls but does not override quality decisions.

Line chart showing organic growth for Fixing website indexing issues (Crawling vs Indexing: Diagnose Missing Pages)
Fixing website indexing issues

Indexing, crawling and Google’s own process

To understand what is indexing in SEO, picture a library: crawling is sending a scout to find books, indexing is cataloguing them so they can be found. Google crawling and indexing are therefore linked but not identical, and a failure at either stage means invisibility. When Search Console reports discovered currently not indexed, it is telling you that the scout knows the book exists but has not yet fetched it.

The usual response is patience combined with better internal links and a faster server. If a large share of your site sits in that status for months, treat it as a signal about priority, and strengthen the pages that matter most.

Structured data code illustration for Indexing, crawling and Google’s own process (Crawling vs Indexing: Diagnose Missing Pages)
Indexing, crawling and Google’s own process

When to call in technical help

If many pages remain unindexed, the problem is probably structural: templates, rendering, architecture or content duplication. A technical review will find the cause faster than page-by-page fixes. See our technical SEO consultant service, and our SEO audit for a broad diagnosis. For JavaScript-driven sites, also read JavaScript SEO and rendering checks. The overview of our services is on SEO consultant London.

Frequently asked questions

What is the difference between crawling and indexing?

Crawling is Google fetching a page. Indexing is Google processing and storing it so it can appear in search results.

What does “Crawled, currently not indexed” mean?

Google visited the page but decided not to include it in the index, often because it is thin, duplicate or not seen as valuable enough.

What does “Discovered, currently not indexed” mean?

Google knows the URL exists but has not yet crawled it, often due to crawl priorities, site speed or weak internal linking.

How do I request indexing?

Use the URL Inspection tool in Search Console and select Request Indexing. It may help recrawl but does not guarantee indexing.

Can a page be indexed but not rank?

Yes. Indexing only makes a page eligible. Ranking depends on relevance, quality and competition.

Talk to an SEO consultant

Questions about this topic or your own site? Send a few details and you will get a clear, practical reply.

Request your SEO consultation

Tell us about your website. You will get a straightforward reply by email, with no obligation.

We use your details only to reply to this enquiry. Privacy policy.

Related pages

Get missing pages indexed

Send your website and goals. You will get a clear next step, not a generic sales pitch.

Book a free consultation
Get a quoteEmail us