Technical SEO Explained

Technical SEO is the least discussed and most consequential part of the discipline, for one reason: its failures are silent.

Write a bad article and you can see it is bad. Ship a noindex header on a template, or a canonical pointing at the wrong URL, and everything looks perfect. The page renders, the content is good, nothing errors. It simply never appears in search, and nothing tells you.

This is the layer that determines whether any of your other work is visible at all. It is also, encouragingly, the most finite: a checklist you can complete, rather than an ongoing effort.

Glossary

Everything this post uses, defined before it is used. Skip it if the terms are already familiar, or come back when one of them trips you up.

The one sentence the rest of this post expands: before a search engine can judge your writing it has to reach the page, run whatever code is needed to see it, decide which address owns it, and find it usable, and each of those can fail without any visible sign.

Terms

Term	Meaning
Crawler	A program that fetches pages and follows their links. Googlebot is one.
Crawling	Fetching a page.
Indexing	Storing an understood version of it so it can be returned for a query.
`robots.txt`	A text file at the site root telling crawlers which paths they may fetch.
`Disallow`	A robots.txt rule blocking a path from being fetched at all.
`noindex`	A tag or header telling engines not to store a page they have fetched.
Sitemap	A machine-readable list of your URLs and when they last changed.
Rendering	Running a page's JavaScript so the crawler sees what a user would see.
Canonical	The address you declare as the real home of a piece of content.
Self-referencing canonical	A page whose canonical points at itself. The normal, correct case.
Structured data	A block of machine-readable labels stating what kind of page this is.
Rich result	A search result with extra detail: author, date, ratings, an FAQ dropdown.
Core Web Vitals	Three measurements of loading, responsiveness and visual stability.
Main thread	The single lane where a browser runs your JavaScript and updates the page.
View source	Looking at the raw HTML the server sent, before any JavaScript ran.
URL Inspection	The Search Console tool that reports how Google crawled, rendered and indexed one page.
Index	The engine's own store of pages it has understood.

Abbreviations

Short	Full form	In plain words
SEO	Search Engine Optimisation	Making a site easier for engines to find, understand and rank
URL	Uniform Resource Locator	A web address
HTML	HyperText Markup Language	The text a server sends describing a page
JS	JavaScript	Code that runs in the browser and can change the page after it loads
SSG	Static Site Generation	Pages built into HTML files ahead of time
SSR	Server-Side Rendering	HTML built on the server for each request
CSR	Client-Side Rendering	An empty shell that JavaScript fills in on the reader's device
DOM	Document Object Model	The browser's live picture of the page, after JavaScript has changed it
CWV	Core Web Vitals	The three page-experience measurements below
LCP	Largest Contentful Paint	When the biggest thing on screen finishes appearing
INP	Interaction to Next Paint	How quickly the page responds after you tap or click
CLS	Cumulative Layout Shift	How much the page jumps about while loading
JSON-LD	JavaScript Object Notation for Linked Data	The usual format for structured data
XML	eXtensible Markup Language	The format a sitemap file is written in
HTTPS	HyperText Transfer Protocol Secure	The encrypted version of the web's transport protocol
AMP	Accelerated Mobile Pages	A stripped-down page format Google once required for top stories
LLM	Large Language Model	The kind of model behind AI assistants, several of which crawl the web
4G	fourth-generation mobile network	An ordinary, not especially fast, phone connection
FAQ	Frequently Asked Questions	The question section near the end

Can a crawler reach your pages?

Everything starts here. Nothing else matters if the answer is no.

robots.txt: the file that can quietly delete your site

robots.txt tells crawlers where they may go. It is four lines of text and one of the easiest ways to remove a site from search entirely:

User-agent: *
Disallow: /

That says "nobody crawl anything". It appears in production more often than you would think, usually copied from a staging config.

What you normally want:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

One subtlety worth understanding: Disallow prevents crawling, not indexing. A blocked URL can still appear in results if other sites link to it, the engine knows it exists and cannot see what is on it. To keep a page out of the index, let it be crawled and serve noindex. Blocking it in robots.txt actually prevents the engine from seeing the noindex you added.

Sitemaps: discovery without luck

A sitemap lists your URLs and when they last changed. It does not guarantee indexing. It makes discovery reliable rather than dependent on link structure.

Generate it, a hand-maintained sitemap is a stale sitemap. Generating means the list is produced from the same data your pages come from, so it cannot drift out of date. In Next.js it is one file:

export default function sitemap(): MetadataRoute.Sitemap {
  return getAllPosts().map((post) => ({
    url: `${base}/blog/${post.slug}`,
    lastModified: new Date(post.date),
  }));
}

Because it derives from the same data the pages use, a new post appears automatically and an unpublished one never leaks.

Internal links are the real crawl path

Crawlers follow links. A page reachable only by typing its URL is effectively invisible, sitemap or not.

The practical failure: a blog index that paginates with JavaScript-only controls, so page two never gets crawled and everything on it disappears from search.

Can it understand and index the page?

Reached is not the same as understood.

Going deeper: rendering, the JavaScript problem

Two versions of your page exist. There is the text the server sends, and there is what appears after the browser runs your code. A reader only ever sees the second. A crawler starts with the first, and getting to the second costs it real money.

Google renders JavaScript, and that fact gets quoted as though the problem were solved. The detail matters: rendering happens in a second pass, queued and resourced separately. Content in the initial HTML is processed immediately. Content requiring JS execution waits, sometimes days, sometimes not at all. Other search engines and most social preview crawlers do far less.

Which is why the rendering strategy decides your SEO ceiling:

Static (SSG), HTML at build time. Best case. Every crawler sees everything.
Server-rendered (SSR), HTML per request. Equally visible.
Client-rendered (CSR), an empty shell plus JS. Worst case. You are relying on the second pass.

For content that must rank, static or server-rendered is not an optimisation, it is the requirement.

The quickest check: view source (not DevTools' Elements panel, which shows the DOM after JS). If your article text is not in the source, crawlers are not reliably seeing it either.

Going deeper: canonicals and duplicates

The same content reachable at several URLs splits its own signals and competes with itself. Common causes: www and non-www, http and https, trailing slashes, tracking parameters, and print views.

A canonical tag names the authoritative version:

<link rel="canonical" href="https://example.com/blog/technical-seo-explained" />

Set it on every page, including the canonical one, self-referencing canonicals are correct and prevent parameterised copies competing.

The classic bug is a canonical hardcoded in a shared layout, so every page declares the homepage as its canonical. The site looks fine and only the homepage gets indexed.

Metadata and structured data

Each page needs a unique <title> and description. Duplicates across a site are a strong signal that the pages are near-identical.

Structured data (JSON-LD) states explicitly what a page is, an article, a product, an FAQ, rather than leaving it to inference:

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Technical SEO Explained",
  "datePublished": "2026-03-20",
  "author": { "@type": "Person", "name": "Nawal Kattel" }
}

It does not raise rankings directly. It makes rich results, author, date, ratings, FAQ accordions, possible, and those raise click-through.

Speed, stability, and the rest

The final layer is about whether the page is pleasant enough to keep someone there.

Going deeper: Core Web Vitals, briefly

These are three attempts to measure what "the page felt fine" means, in numbers. One for how long you stare at nothing, one for how long a tap takes to do anything, one for how much the page moves while you are trying to read it.

Three measurements Google uses:

LCP, when the largest element appears. Target under 2.5s. Usually an image or a webfont.
INP, responsiveness to interaction. Target under 200ms. Usually long-running JavaScript on the main thread.
CLS, how much the layout jumps. Target under 0.1. Usually images without dimensions, or ads and banners injected above content.

They are a modest ranking factor and a large user-experience one. CLS in particular is worth fixing for its own sake: content shifting under a reader's finger is infuriating.

The most common wins are unexciting, size your images, set explicit width/height, self-host fonts with font-display: swap, and ship less JavaScript.

The silent-failure checklist

Worth running before assuming a ranking problem is about content:

Is the page in the index? Search site:example.com/your-page.
Does view-source contain the content?
Is there an accidental noindex header or meta tag?
Does the canonical point at this page?
Is it reachable by clicking from the homepage?
Is it in the sitemap, and does Search Console report the sitemap as read?
Does it load acceptably on a mid-range phone on 4G?

Search Console's URL Inspection answers most of these directly, including how Google rendered the page. It is the single most useful free tool in SEO and most people never open it.

Key takeaways

Technical SEO fails silently. The page looks perfect and never appears.
Disallow blocks crawling, not indexing, and blocking a page prevents engines from seeing its noindex.
Generate the sitemap from your data, so it cannot go stale.
Content in the initial HTML is indexed reliably. JS-rendered content waits for a second pass that may not come.
Set a self-referencing canonical on every page, and check it is not hardcoded to the homepage.
Structured data does not rank you higher. It makes rich results possible, which raises clicks.
Use Search Console's URL Inspection before theorising about rankings.

FAQ

My page is not indexed. What is the first thing to check?

URL Inspection in Search Console. It tells you whether Google has crawled it, whether it was indexed, what canonical it chose, and how it rendered, which usually identifies the cause immediately.

Does Google really run JavaScript?

Yes, in a separate, later, resource-limited pass. Rely on it and you accept a delay and a risk. Other crawlers, Bing, social previews, LLM crawlers, do substantially less.

Do I need AMP?

No. It has been deprecated as a requirement for top stories, and fast ordinary pages achieve the same thing without the constraints.

How important is HTTPS?

Non-negotiable. It is a ranking signal, and browsers warn users away from sites without it.

Should every page be indexed?

No. Tag pages, filtered listings, and paginated archives often should not be. They dilute the index with near-duplicates. Use noindex deliberately.

How often should I audit?

Check Search Console monthly and run a full audit after any framework upgrade, redesign, or URL change. Those three are when silent breakage is introduced.

Conclusion

Technical SEO is unglamorous and finite. There is a defined set of things that can be wrong, most are checkable in an afternoon, and once fixed they stay fixed until you change something structural.

That combination makes it the highest-return work in SEO, not because it will lift you above a better page, but because it decides whether you are in the race at all. Content and links compete. Crawlability is a gate.

References

Official documentation for the topics covered here.

Google Search Central: Introduction to robots.txt - what the file can and cannot stop, including the indexing trap
Google Search Central: Sitemaps overview - formats, size limits and how to submit one
Google Search Central: Block indexing with noindex - the correct way to keep a reachable page out of the index
Google Search Central: How to specify a canonical URL - canonical tags, duplicate clusters and the signals Google weighs
Google Search Central: Understand JavaScript SEO basics - how client rendered pages are queued, rendered and indexed
web.dev: Web Vitals - the definitions and thresholds behind LCP, INP and CLS
Google Search Central: Introduction to structured data - how to add machine readable metadata without breaking eligibility

What Is SEO? for the wider picture, On-Page SEO vs Off-Page SEO for what sits above this layer, and How Google Search Works for what the crawler does with the pages it collects.