One Setting Can Hide Your Whole Website From Google

I’ll start with something embarrassing.

Earlier this year, the website of a marketing agency barely existed on Google. Three keywords with any impressions in ninety days. All three below position 50, which is page five and beyond, where nobody goes.

The agency was mine.

When I finally audited it properly, I found the homepage and twenty-one other pages had no focus keyword at all. And sitting in a tiny file called robots.txt was a line telling every search crawler to wait 20 seconds between pages. Google, as it turns out, ignores that particular instruction. Bing doesn’t.

None of it was visible from the front of the site. The design was fine. The copy was fine. The settings underneath were cancelling both.

So before you spend another naira on content, check the boring stuff.

Below is the audit I should have run, in the order the checks should be done, using free tools. You don’t need to be a developer to run it, though you may need one to fix some of what it turns up.

Got Ten Minutes?

These five checks catch most of the serious problems.

1. Is any important page set to noindex? (See 1.2.)
2. Does robots.txt block your site or slow crawlers down? (See 1.3.)
3. Is your sitemap submitted, and does the indexed count come close to the submitted count? (See 2.1.)
4. Is every key page linked from somewhere else on the site? (See 2.2.)
5. Do your key pages pass Core Web Vitals on mobile? (See 4.1.)

Your Toolkit, All Free to Start

Google Search Console, which is free, is the only reliable view of how Google sees your site, including indexing, errors and search queries. Google PageSpeed Insights, also free, measures speed and Core Web Vitals on mobile and desktop. A site crawler such as Screaming Frog (free tier available) crawls your site the way a search engine does and lists every page and problem. Your SEO plugin, such as Rank Math or Yoast, handles page-level titles, descriptions, focus keywords and settings. And Google’s Rich Results Test checks structured data.

If you only set up one of these, make it Search Console. Every stage of the audit leans on it.

Stage 1. Can Google Index Your Pages?

Nothing else matters until this is right. A page that isn’t indexed can’t rank for anything, so start here and fix what you find before moving on.

1.1 What Google has actually indexed

In Search Console, open the Pages report under Indexing. It splits your pages into indexed and not indexed, with a reason for every exclusion. Read every reason. Some exclusions are correct, like tag pages, duplicate URLs and admin pages. Others are disasters hiding in a list, like your services page excluded by a noindex tag.

1.2 The noindex trap

A stray noindex works like Harry’s Invisibility Cloak. Everything is still there, perfectly intact, and Google walks straight past it. It’s the most common serious technical SEO error on WordPress sites, and it nearly always starts innocently. A developer builds the site on a staging server and ticks “discourage search engines from indexing this site” so the half-finished version stays out of Google. The site goes live and the setting stays on, either across the whole site or on individual pages that were copied across. Check four places.

• WordPress → Settings → Reading. The “Discourage search engines” box must be unticked on your live site.
• Your SEO plugin’s settings for each important page. Confirm each one is set to index.
• The page source. View the source of your key pages and search for noindex. If a robots meta tag contains it, that page is asking search engines to leave it out.
• The server response headers. A noindex can also arrive as an X-Robots-Tag header, set in server or security configuration, and it won’t appear in the page source. Search Console’s URL Inspection tool tells you whether a page is blocked from indexing, whichever way the instruction arrives.

Check your highest-value pages twice, especially home, services, pricing and case studies. A noindex on your services page hides the one page most likely to bring in customers.

1.3 robots.txt

Visit yourdomain.com/robots.txt. This file tells crawlers what they may and may not access. Look for four things.

• Disallow: / under User-agent: *, which blocks the entire site. It’s sometimes left over from staging.
• A Crawl-delay line. Google ignores it, but Bing and some other search engines honour it and crawl your site very slowly. On a site that publishes regularly, a long delay slows how quickly those engines find new pages. Remove it unless your server can’t cope.
• Blocked resources. Blocking CSS or JavaScript folders stops search engines rendering your pages properly.
• A missing Sitemap: line. Add one pointing to your sitemap’s full address.

Keep one distinction straight. robots.txt controls crawling, and noindex controls indexing. A page blocked in robots.txt can still appear in results if other sites link to it. To keep a page out of results, use noindex and leave the page crawlable, otherwise search engines will never see the noindex instruction.

1.4 AI crawlers

AI assistants send their own crawlers, each named in robots.txt, and which one you block matters. OpenAI’s OAI-SearchBot decides whether your pages can appear in ChatGPT’s search answers, while GPTBot collects data for model training, and blocking it doesn’t affect search appearance. Anthropic splits its crawlers the same way, with Claude-SearchBot for search quality, ClaudeBot for training and Claude-User for pages a user asks Claude to fetch. Perplexity’s search crawler is PerplexityBot.

Security plugins sometimes block all of these by default, so check what yours does. If you want AI assistants to recommend you, allow the search crawlers. Whether to allow training is a separate decision. Crawler names change, so confirm them against each company’s current documentation. The AI search visibility guide covers the wider picture.

1.5 Canonical tags

A canonical tag tells search engines which version of a page is the main one when several URLs show similar content. Each important page’s canonical should point to itself. If it points to a different page, the staging domain or an old URL, you’re telling Google to ignore the page you’re looking at.

Stage 2. Can Google Find and Understand Them?

2.1 XML sitemap

Your SEO plugin usually builds a sitemap automatically. Open it, make sure it lists your important pages and leaves out junk like tag archives and thin pages, then submit it in Search Console under Sitemaps. Search Console will report how many of the listed pages were indexed. A big gap between submitted and indexed is worth chasing.

2.2 Internal links and orphan pages

Search engines find and weigh pages largely through links. A page that nothing on your site links to, an orphan page, is hard to find and tells Google you don’t think it matters. Run a crawl and compare the pages the crawler found against the pages in your sitemap. Any page in the sitemap that the crawler couldn’t reach through links is an orphan. Link to it from relevant pages.

Check the opposite direction too. Your most commercially important pages should be your most linked-to pages. If your blog posts link to each other all day and never to your services page, you’re telling Google the blog matters more than the business.

2.3 Broken links and redirect chains

Your crawler will list internal links that return errors (404s) and redirects. Point broken internal links at the correct page. Where one redirect leads to another and then another, point links straight at the final destination. After a rebuild, check that every important old URL redirects (with a 301) to its new equivalent. Lost redirects are one of the main reasons rankings drop after a redesign, as the redesign guide explains.

2.4 URL structure

Good URLs are short, readable and descriptive, like /services/seo/ rather than /?page_id=482. Check that your permalink settings produce readable URLs, and never change the URL of an established page without a redirect.

Stage 3. Does Each Page Say What It’s About?

3.1 One page, one purpose

Each important page should target one clear search intent, with a focus keyword that reflects it. Pages with no focus keyword in your SEO plugin are a sign that nobody decided what they should rank for. Decide, page by page.

3.2 Titles and meta descriptions

Every page needs a unique title tag that includes its main topic and a unique meta description that makes a searcher want to click. Your crawler will flag titles and descriptions that are missing, duplicated, too long or too short. Fix duplicates first, because two pages with the same title leave search engines guessing which to show.

3.3 Heading structure

Each page should have one H1 describing its main topic, with H2s and H3s organising the content beneath it. Elementor and theme-built sites often turn decorative elements into headings purely for styling, like statistic counters, section labels and buttons. The result is dozens of meaningless headings that blur what the page is actually about. Change the element’s HTML tag in its settings where you can. Where a widget doesn’t offer that setting, you may need a developer.

3.4 Keyword cannibalisation

When several of your pages target the same search term, they compete with each other, and search engines may rank none of them well. Search Console’s Performance report shows this. Filter by a query and see which pages appear for it. If two pages keep swapping places, merge them or give each a distinct focus.

3.5 Thin and duplicate content

Pages with very little unique content, or near-copies of each other, drag your site down. The usual suspects are tag archives, location pages that change only the city name, and placeholder pages left over from a theme demo. Improve them, merge them, or noindex the ones with no search value.

Stage 4. Does It Work on a Phone in Lagos?

4.1 Core Web Vitals

Google measures real-user experience through Core Web Vitals, which you’ll find in Search Console and PageSpeed Insights.

Check mobile results first. Look at your own analytics, but on most Nigerian sites the majority of visitors are on phones, often on mobile data, and Google mainly assesses the mobile version of your site. If Search Console shows no Core Web Vitals data because your traffic is too low, use the lab results in PageSpeed Insights instead. The launch checklist lists the specific speed fixes.

4.2 Mobile usability

Open your key pages on a real mid-range phone. Text should be readable without zooming, buttons easy to tap, nothing sliding off the side of the screen, and menus working. Desktop emulators miss problems a real phone shows you in seconds.

4.3 When speed plugins break things

Speed plugins help, but aggressive settings can break layouts, animations and forms. Settings that strip “unused” CSS or delay JavaScript are the usual culprits. Switch on one change at a time and check your key pages after each, especially forms and anything interactive.

Stage 5. Is It Secure?

Every page should load over HTTPS, with the HTTP version redirecting to it. The www and non-www versions should redirect to a single preferred domain. Secure pages shouldn’t load images or scripts over plain HTTP. And check Search Console’s Security issues report, because a hacked site can be flagged and lose visibility fast.

Stage 6. Structured Data

Structured data, or schema markup, describes your business and content to search engines in a standard format. It helps them understand who you are and can make your pages eligible for richer results. Check your existing markup with Google’s Rich Results Test, fix any errors, and add the essentials if they’re missing. The schema markup guide has copy-and-paste templates.

What to Fix First

A thorough audit usually produces a long list. Resist the urge to work through it from the top. Score each issue on impact and effort, then work in this order.

Put It in the Calendar

Run the full audit after any redesign, migration or major plugin change. Run a lighter version every quarter, which means checking the Pages report in Search Console, rerunning a crawl for new errors and rechecking Core Web Vitals. Set a monthly reminder to glance at Search Console’s indexing and security reports.

Most serious technical problems arrive without warning, through a plugin update or a well-meant change. The businesses that catch them early are simply the ones that keep looking.

And go and open that robots.txt file today. It takes thirty seconds.

If your technical SEO turns out fine and the traffic still isn’t coming, the problem may sit somewhere else. The free 2-minute diagnostic shows which of your four walls is weakest, and search visibility usually lands under Attract.

Find Your Weakest Wall →

Spending ₦1M or more a month on marketing? Book the free 30-minute Leak Check as well, and we’ll show you three places your money is leaking.

Search engine behaviour, Search Console reports and tool features change regularly. These checks reflect common practice at the time of writing. This is general technical information and no guarantee of rankings.

This article covers one part of a bigger system. Our guide to digital marketing in Nigeria shows how the four walls fit together, and the order to build them in.