How search engines actually work: crawling, indexing, and ranking
Most SEO advice gets followed as a checklist — add this tag, hit this word count — without the underlying mechanism ever being explained. This series builds that understanding first, then the techniques, across 22 parts covering on-page, technical, off-page, and AI-answer-engine SEO, with real, working examples throughout rather than generic advice. Part 1 is the mechanism itself: what a search engine actually does between "a page exists on the internet" and "a page shows up in results."
Stage one: crawling
A search engine's crawler (Google's is called Googlebot) discovers pages by following links — starting from pages it already knows, and from XML sitemaps site owners submit directly (part 9 covers building one). It has no innate knowledge that a new page exists; it finds pages the same way a person would, by following a link to them, or being told directly via a sitemap.
This is why a page with zero links pointing to it — an "orphan" page — can go completely unindexed even if it's technically published and reachable by typing the exact URL. Crawling is a discovery problem before it's anything else.
Stage two: indexing
Once crawled, a page's content gets parsed and stored in the search engine's index — a database, at massive scale, of every page it's decided is worth keeping. Not every crawled page gets indexed: a page can be crawled and then explicitly excluded (a noindex tag, a duplicate of another page, content judged too thin to be useful).
<meta name="robots" content="noindex" />This single tag is the most direct control a page has over whether it enters the index at all — present, and a search engine will crawl the page but deliberately leave it out of results.
Stage three: ranking
For any given search query, ranking is the process of ordering every indexed page that could plausibly answer it, from most to least relevant. This is the stage that gets the most attention in SEO discussions, and it's driven by a genuinely large number of signals — Google has never published an exact count, but has confirmed the algorithm weighs hundreds of factors, not a handful. This series groups them into the categories that actually matter in practice: on-page content relevance (parts 2-8), technical crawlability and performance (parts 9-15), off-page authority signals (parts 16-18), and — increasingly — how well content serves AI-driven answer generation on top of traditional ranking (part 19).
Why this pipeline matters to how you work
Every SEO technique in this series maps to one of these three stages, and knowing which one is what tells you whether a technique is even solving the right problem:
- No organic traffic, page not showing up in search at all — likely a crawling or indexing problem (part 9), not a ranking one. No amount of keyword optimization helps a page that was never indexed.
- Indexed, but consistently on page 3+ of results — a ranking problem — content relevance, technical performance, or authority signals not yet strong enough (parts 2-18).
- Ranking, but nobody clicks — not a ranking problem at all. The title tag and meta description (part 2) are what a searcher actually sees and decides whether to click on, independent of position.
Treating every traffic problem as a "need more keywords" problem. A huge share of real SEO issues are crawling or indexing failures — a page genuinely unreachable, or explicitly excluded — that no amount of on-page optimization touches at all, since the page was never in the running to rank in the first place.
Next: the two tags every searcher actually sees before ever clicking through — the title tag and meta description.