Technical SEO has a reputation for being impenetrable. Some of it is. But the most important part is simple to grasp: if Google can’t find, access and index your pages, nothing else you do for SEO matters.
A surprising number of small business websites have pages that Google isn’t indexing, often because of a setting nobody remembers changing. The good news is that these problems are usually easy to spot once you know where to look.
This guide covers the technical SEO basics a business owner can check without a developer.
Crawling and indexing in one paragraph
Google’s crawlers visit your site, follow links and read your pages. That’s crawling. Google then decides whether to store each page in its index so it can be shown in search results. That’s indexing. A page can be crawled but not indexed, and only indexed pages can appear in results.
Your main tool: Google Search Console
Search Console is free, and it’s the most direct view you’ll get of how Google sees your site. If you haven’t set it up, do that first. Your web developer or host can help with verification if needed.
Two parts are especially useful for indexing:
- The page indexing report. This shows how many of your pages are indexed and lists reasons others aren’t. Some reasons are perfectly normal, such as redirected pages or deliberate duplicates. Others need attention.
- The URL inspection tool. Paste in any page from your site and it tells you whether that page is indexed, when it was last crawled and whether there are problems. You can also request indexing after making changes, though that’s a request, not a guarantee.
Common reasons pages aren’t indexed
A “noindex” setting left switched on
Websites are often built with search engines blocked, so a half-finished site doesn’t appear in results. If nobody switches that off at launch, the whole site stays hidden. In WordPress, for example, there’s a setting under Reading that discourages search engines. SEO plugins can also set individual pages to noindex. Check both.
Blocked by robots.txt
The robots.txt file tells crawlers which parts of a site they shouldn’t visit. A single misplaced line can block important sections. You can view yours at yourdomain.co.uk/robots.txt. If you see Disallow: / on its own under a general user-agent, that’s blocking everything and needs looking at.
Duplicate or near-duplicate pages
If several pages are very similar, Google will usually pick one to index and ignore the others. That’s why templated location pages and product variants often show as “duplicate” in Search Console.
Thin or low-value content
Search Console sometimes reports pages as “crawled, currently not indexed” or “discovered, currently not indexed”. There can be several causes, but on small sites it often suggests Google didn’t see enough value in the page to store it. Improving the content and linking to it from relevant pages is usually the right response.
Orphan pages
Pages with no internal links pointing to them are hard for Google to find and look unimportant. Make sure every page you care about is linked from somewhere sensible.
Indexing problems are rarely dramatic. They’re usually one forgotten tick box. That’s exactly why they’re worth checking.
Sitemaps and canonical tags
Sitemaps: helpful, not magic
An XML sitemap lists the pages you want Google to know about. Most modern website platforms and SEO plugins create one automatically. Submit it in Search Console so Google knows where it is.
A sitemap helps discovery, but it doesn’t force indexing. Keep it clean: it should include the pages you want in search, not redirected, deleted or noindexed ones.
Canonical tags in plain English
A canonical tag tells Google which version of a page is the main one when similar versions exist. For example, the same product might be reachable through several URLs. Most platforms handle this automatically. Problems arise when canonicals are set incorrectly, for instance every page pointing to the homepage as the “main” version. If Search Console reports lots of pages where Google chose a different canonical from yours, it’s worth asking your developer to take a look.
A quick technical SEO health check
- Is Search Console set up and your sitemap submitted?
- Are your most important pages indexed? Check each one with URL inspection.
- Is the site accidentally set to discourage search engines?
- Does robots.txt block anything important?
- Does your site load on one version only, for example with https and consistently with or without www, with the others redirecting?
- Are there pages you don’t want in search, such as thank-you pages or test pages, that are indexed?
Running this technical SEO check every few months, and always after a website change, catches most problems before they cost you.
When to bring in help
If Search Console shows lots of errors you don’t understand, or indexing has dropped suddenly, it’s worth a proper look. Our technical SEO service covers exactly this, and explains what’s wrong and why in plain terms before fixing it.
