SEO Fundamentals: How Search Engines Crawl, Index and Rank

A second-hand bicycle shop launches a website with forty bikes. Three weeks later the owner searches for the exact name of one of her own listings and finds nothing, while a rival's page shows up. She suspects a penalty. She is wrong: the page was never even seen. Search is a pipeline with several stages, and a page can drop out at any of them. Knowing the stages turns "why am I not on Google?" from a mystery into a checklist.

The pipeline: discover, store, order

Crawling is done by automated programs, often called crawlers or bots, that follow links from known pages to new ones and fetch what they find. They also read sitemaps you submit. They are polite about it: how often and how deeply a site is crawled depends on its speed, its size and how much the engine thinks the site changes. Indexing means analysing the page and storing it in a huge database, keyed by what it is about. A crawled page is not guaranteed to be indexed. Ranking happens at the moment someone searches: the engine picks relevant pages from its index and orders them using many signals.

Why links are how pages get found

The simplest reason a page is missed is that nothing links to it. Crawlers discover pages mainly through links, so a page with no inbound links is invisible until something points to it. The script below models the bike shop site as a graph of pages and links and walks it breadth-first from the home page, the way a crawler would.

from collections import deque

# page -> pages it links to (a tiny second-hand bicycle shop)
site = {
    "/": ["/bikes", "/about"],
    "/bikes": ["/bikes/road", "/bikes/kids"],
    "/bikes/road": ["/bikes/road/trek-520"],
    "/bikes/kids": [],
    "/bikes/road/trek-520": [],
    "/about": [],
    "/bikes/gravel/orphan-grit": [],   # exists, but nothing links to it
}

def crawl(start: str) -> list[str]:
    seen, queue, order = {start}, deque([start]), []
    while queue:
        page = queue.popleft()
        order.append(page)
        for link in site.get(page, []):
            if link not in seen:
                seen.add(link)
                queue.append(link)
    return order

found = crawl("/")
print("discovered:", len(found), "of", len(site))
print("never discovered:", sorted(set(site) - set(found)))

discovered: 6 of 7
never discovered: ['/bikes/gravel/orphan-grit']

The orphan page exists on the server but no path from the home page reaches it. In real sites, orphan pages come from forgotten category pages, search results that are the only route to an item, and links produced only by JavaScript that bots do not run. The fix is structural: link to important pages from places crawlers already visit, and list them in a sitemap.

What ranking is and is not

Search engines publish broad principles about ranking but not the full recipe. In general terms they look for pages that match the words and meaning of the query, that satisfy the intent behind it, that come from sources that appear trustworthy, and that work well on the device being used. Links from other sites that point to your page act as a signal that others consider it useful. Beyond that, treat anyone who claims to know an exact formula, or who guarantees a position, with suspicion. Results also vary by country, language, device and personal context, so "my ranking" is an average over many searches rather than one number.

Check your own site in five minutes

You can test each stage without any paid tool. First, search for your page's exact title in quotes, or use the site: operator followed by your domain to see which pages the engine has indexed. Second, open the page's source (not just the inspector) and confirm the main text is there; if you only see an empty container and a script tag, a bot that cannot run scripts sees nothing. Third, look at the raw HTML head for stray noindex or a canonical link that points to a different URL. Fourth, check that the page responds with a normal success status, because a page that returns an error code for bots will not be stored. Fifth, in Search Console, use the URL Inspection tool, which reports whether a URL is known to Google, whether it was crawled and whether it is indexed.

<head>
  <title>Refurbished road bikes | Spoke &amp; Pedal Cycles</title>
  <meta name="robots" content="index, follow">
  <link rel="canonical" href="https://www.example.com/bikes/road">
</head>

The canonical tag tells the engine which URL is the preferred version when the same content appears at several addresses (for example with tracking parameters). A page whose canonical points to a different URL is telling the engine to index that other one instead, which is a common accidental cause of "my page is not indexed".

Common mistakes

• Blocking the staging fix in production. Symptom: a noindex tag or a blanket Disallow: / in robots.txt copied from staging, and the whole site slowly vanishes from results. Fix: add a release check for robots directives.
• Assuming a crawled page will be indexed. Symptom: Search Console reports the URL as discovered or crawled but not indexed. Fix: improve the page's uniqueness and value and link to it from relevant pages, instead of resubmitting it repeatedly.
• Content only behind scripts or interactions. Symptom: view-source shows an empty shell, and the snippet in results is blank or generic. Fix: server-render or pre-render the main content.
• Creating many near-identical pages. Symptom: dozens of colour or size variants competing against each other and being folded into one. Fix: canonicalise to one main page.
• Chasing a single ranking number. Symptom: celebrating a first-page position for a term nobody searches. Fix: track clicks and impressions for the queries that matter to your goals.

How this shows up in real projects and interviews

Developers meet SEO as rendering and structure questions: is the content in the initial HTML, are URLs stable, do pages return correct status codes? Frameworks that render on the server make this easier, and the Next.js course covers the rendering modes involved. In interviews for web or growth roles, a good answer to "how does Google find a new page?" describes links and sitemaps, then the crawl, render, index and rank sequence, and ends with the honest line that ranking itself is not something you can control. Portfolio builders should also make sure their own pages can be found when employers look; see LinkedIn profile optimisation tips for the profile-side equivalent.