Quick Definition #
Crawling, indexing and ranking are three distinct stages in organic search. Crawling is how a search engine fetches a page, indexing is how it processes and may store the page, and ranking is how it orders eligible results for a particular query.
Learning Objectives #
- Explain the difference between crawling, indexing and ranking.
- Recognise common technical issues at each stage.
- Use a practical troubleshooting sequence when a page is missing from search.
1. Crawling #
Crawling occurs when an automated program, commonly called a crawler or bot, requests a URL and retrieves its available content. Search engines discover URLs through links, sitemaps and other signals. Crawling may be affected by server errors, access restrictions, robots.txt rules, URL complexity and crawl prioritisation.
Example: A website publishes a new guide and links to it from its SEO category page. A crawler can discover the URL by following that internal link and then request the page.
2. Indexing #
After fetching a page, a search engine processes its content and decides whether and how to include it in its index. Processing can involve text, images, canonical signals, duplicate-content assessment and other page information. A crawled page is not automatically indexed.
Common reasons a page may not be indexed include a noindex directive, duplicate or low-value content, access problems, conflicting canonical signals or the search engine choosing not to include it.
3. Ranking #
When a user enters a query, the search engine selects potentially relevant pages from its index and orders results using many systems and signals. Relevance to the query, usefulness, context, content quality and other factors may matter. No single checklist or score guarantees a position.
A page can be indexed yet receive few impressions because it does not adequately satisfy the query, faces strong competition or targets a topic with limited demand.
Comparison Table #
| Stage | Core question | Typical issue | Useful check |
|---|---|---|---|
| Crawling | Can the engine fetch the URL? | Blocked access or server error | HTTP response, robots.txt, server logs |
| Indexing | Was the page processed and selected for the index? | Noindex, duplicate URL, canonical conflict | Page source and Search Console reports |
| Ranking | How well does the page meet this query? | Weak relevance or competing pages | Query data, intent and competing results |
Practical Troubleshooting Workflow #
- Confirm the page loads correctly and returns the expected HTTP status.
- Check whether important pages link to it and whether the preferred URL appears in the XML sitemap.
- Review robots.txt and page-level indexing directives.
- Inspect the canonical URL and look for duplicate versions.
- Use Google Search Console URL Inspection to review reported crawl and indexing information.
- If indexed, assess query intent, content usefulness, internal links and search performance before changing the page.
Common Mistakes #
- Assuming crawling guarantees indexing.
- Assuming indexing guarantees rankings.
- Using robots.txt as a substitute for a noindex directive.
- Adding more keywords before diagnosing the actual problem.
- Changing several technical settings at once without recording the original state.
Key Takeaway #
Diagnose SEO visibility in stages: first discovery and crawling, then indexing, and finally ranking. Identifying the correct stage prevents unnecessary changes.
FAQs #
Can a page rank without being indexed? #
A page generally needs to be in the search engine’s index to appear as a standard organic result for a query.
Does a sitemap guarantee indexing? #
No. A sitemap helps communicate preferred URLs but does not guarantee crawling, indexing or rankings.
What should I check first if a page is missing? #
Confirm that the URL works, then inspect crawl access, indexing directives and canonical signals before investigating ranking performance.
Related Topics #
Continue with How Search Engines Work and Search Engine Results Pages (SERPs).
