SEO Fundamentals: Technical SEO, Sitemaps and Search Console

A board-game cafe redesigns its site over a weekend. On Monday it is happy: faster, prettier. By Friday, organic visits have halved. Nothing about the content changed. The developer had copied a robots.txt from the staging server that blocked everything, and the new URLs no longer matched the old ones, with no redirects. None of that is visible in the design. Technical SEO is the layer that makes a site reachable, stable and measurable, and most of it is checks you can run in an afternoon.

robots.txt: crawl rules, not a lock

A robots.txt file at the root of a domain tells well-behaved crawlers which paths they may fetch. It controls crawling, not indexing: a blocked URL can still appear in results, without a snippet, if other pages link to it. To keep a page out of results, use a noindex directive and leave the page crawlable so the bot can see it. And robots.txt is not security. Anyone can read it, and it will not stop a hostile visitor, so never use it to hide private paths.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /

Sitemap: https://www.example.com/sitemap.xml

You can test rules before shipping them. Python's standard library includes a parser, so this check can run in your release pipeline.

from urllib.robotparser import RobotFileParser

robots = """\
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /

Sitemap: https://www.example.com/sitemap.xml
""".splitlines()

rp = RobotFileParser()
rp.parse(robots)
for path in ["/", "/events/trivia-night", "/cart/checkout", "/search?q=chess"]:
    print(f"{path:<24} crawlable: {rp.can_fetch('*', 'https://www.example.com' + path)}")

/                        crawlable: True
/events/trivia-night     crawlable: True
/cart/checkout           crawlable: False
/search?q=chess          crawlable: False

XML sitemaps

A sitemap lists the URLs you want indexed, optionally with a last-modified date. It does not force indexing but it helps discovery, especially for new sites, large sites and orphan pages. Include only canonical, indexable URLs that return a success status, keep them in one consistent host and protocol, and use accurate modification dates (a date that changes on every deploy teaches the engine to ignore it). Remember to escape special characters such as the ampersand.

from datetime import date
from xml.sax.saxutils import escape

pages = [
    ("https://www.example.com/", date(2025, 3, 1)),
    ("https://www.example.com/events/trivia-night", date(2025, 2, 20)),
    ("https://www.example.com/menu?type=a&b", date(2025, 1, 5)),
]
lines = ['<?xml version="1.0" encoding="UTF-8"?>',
         '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
for url, modified in pages:
    lines.append(f"  <url><loc>{escape(url)}</loc><lastmod>{modified.isoformat()}</lastmod></url>")
lines.append("</urlset>")
print("\n".join(lines))

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://www.example.com/</loc><lastmod>2025-03-01</lastmod></url>
  <url><loc>https://www.example.com/events/trivia-night</loc><lastmod>2025-02-20</lastmod></url>
  <url><loc>https://www.example.com/menu?type=a&amp;b</loc><lastmod>2025-01-05</lastmod></url>
</urlset>

Page experience and Core Web Vitals

Core Web Vitals are three field-measured metrics describing real users' experience of loading, interaction and visual stability. As of this writing Google's published guidance describes them as follows, but check the current documentation for exact thresholds and any changes.

Two things to keep in perspective. First, these are one input among many, and a fast page does not outrank a far more relevant one; their larger effect is on real visitors, who leave slow or jumpy pages. Second, the data that matters is field data from real users (shown in Search Console's Core Web Vitals report), while lab tools such as Lighthouse are for diagnosing. Typical fixes are to compress and size images, set explicit width and height on media, defer non-critical scripts, use a CDN and cache, and trim third-party widgets.

A Search Console routine

Search Console is free and shows how the search engine sees your site. Verify the domain, submit the sitemap, then adopt a short monthly habit.

1. INDEXING / PAGES REPORT
   - Did "not indexed" counts jump? Open the biggest reason group.
   - Spot-check three important URLs with URL Inspection.

2. SITEMAPS
   - Status is "Success"; submitted count roughly equals indexable pages.

3. PERFORMANCE REPORT (last 3 months vs previous period)
   - Queries with high impressions and low click-through: rewrite the
     title and description.
   - Queries with positions just below the first page: strengthen the
     page, add internal links.
   - Pages that lost clicks: check recent changes before assuming
     an algorithm change.

4. CORE WEB VITALS AND PAGE EXPERIENCE
   - Note URL groups marked "poor"; hand them to a developer.

5. MANUAL ACTIONS AND SECURITY ISSUES
   - Both should be empty. If not, fix and request a review.

6. WRITE DOWN: what changed on the site this month, and what you will
   change next. Later you can connect cause and effect.

Spammy tactics to avoid

Search engines publish spam policies, and violating them can reduce visibility or cause a manual action. The recurring offenders are worth knowing so you can recognise them when an agency proposes them.

• Keyword stuffing: repeating phrases unnaturally in text, titles or hidden spots.
• Hidden text and links: white text on white, off-screen positioning or tiny fonts for bots only.
• Cloaking and sneaky redirects: showing crawlers one page and users another.
• Link schemes: buying links, large-scale exchanges or automated link-building.
• Scaled low-value content: mass-producing thin, automatically generated or scraped pages, including doorway pages made only to funnel visitors.
• Expired-domain and reputation abuse: hosting unrelated third-party content to borrow a site's standing.

Common mistakes

• Shipping staging robots rules to production. Symptom: traffic drops sharply after launch and Search Console reports "Blocked by robots.txt". Fix: automated release check on the live robots.txt.
• Changing URLs without 301 redirects. Symptom: a spike of "Not found (404)" for formerly strong pages and lost rankings. Fix: map every old URL to its new equivalent with a permanent redirect.
• Blocking a page in robots.txt to deindex it. Symptom: the URL stays in results with no snippet. Fix: allow crawling and use noindex, or remove it properly.
• Sitemaps full of redirected, noindexed or error URLs. Symptom: Search Console lists many submitted URLs as excluded. Fix: generate the sitemap from the list of indexable canonical pages only.
• Optimising only the lab score. Symptom: a green Lighthouse run but poor field data. Fix: use field data from real users to prioritise.

How this shows up in real projects and interviews

Technical SEO is most often a launch-week concern: migrations, redesigns and framework changes are when sites lose visibility, so developers who add redirect maps and robots checks to the release process are valued. Page-speed work overlaps with general front-end performance, and CSS fundamentals and JavaScript fundamentals cover things like layout stability and script cost. In interviews, "how would you migrate a site without losing traffic?" has a clear structure: inventory URLs, map redirects, keep content equivalent, check robots and sitemaps, monitor Search Console afterwards.