The site Google could not see
Search Console said 23 pages. The site had 86. The missing ones included the home page, the services and the contact page — and nothing on the site looked wrong.
My consultancy's website looked fine. It loaded fast, it worked on a phone, it had a blog, a services section and a contact page. Then I opened Google Search Console and read one line: 23 pages discovered.
The site had 86.
The 63 missing ones were not obscure. They included the home page, the services, the products, the about page, the contact page and almost the entire blog. Google had never been told about any of the pages that explain what the company does.
How a site goes missing without anything breaking
The cause was a single file: sitemap.xml, the list a website hands to search engines saying "these are my pages". Ours had been written by hand at some point, and then the site kept growing while the list stayed where it was.
A sitemap is not the only way Google finds pages — it also follows links. But on a young site that few other sites link to, the sitemap is most of what Google has to go on. Leave a page off it and you are relying on luck.
Nothing about this shows up when you use the site. Every page opens. Every link works. The problem only exists from Google's side, which is exactly why nobody had noticed.
The fix that lasts
Adding the missing 63 URLs by hand would have fixed it for a month. The real fix was to stop writing the list by hand at all.
Now a small script builds the sitemap from the site itself. It reads the canonical address each page declares about itself — so it never invents a URL — skips any page marked noindex, and leaves out the handful of private pages and the ones that redirect elsewhere. It runs every time a page is added.
The result was 66 URLs, every one of them indexable and every one of them returning a proper 200. Not 86, and that is correct: legal pages and a few placeholder pages are deliberately kept out of search, so they have no business being in the list either.
What else turned up once I looked
A sitemap that is wrong is rarely the only thing that is wrong. Looking properly turned up a few more quiet problems, here and on other sites I run:
- Translations that only pointed one way. On a site in two languages, each version has to name the other, and the other has to name it back. If only one side does it, Google ignores the pair entirely. There were 15 broken pairs.
- One company, counted three times. The structured data — the part of a page written for machines — described the business in three slightly different ways, so search engines read it as three different companies.
- Every page "changed today". On this site the sitemap stamped today's date on every page, every day. A date that is always today tells Google nothing, so it stops trusting it. It now uses the real date each page last changed.
- A fix of mine that made things worse. On another project I tried to give 32 pages a canonical address in one go. The addresses it produced redirected somewhere else, and a canonical pointing at a redirect is worse than none. I caught it, reverted it, and did it page by page instead.
None of these broke anything a visitor could see. All of them decided what Google could see.
Check yours in ten minutes
You do not need to be technical for most of this:
- Open
yoursite.com/sitemap.xmlin a browser. Count the addresses and compare them with the pages you know you have: the menu, each service, the blog posts. If the numbers are far apart, that is your answer. - In Google Search Console, open Sitemaps and look at how many pages it says were discovered. Then open Pages and see how many are actually indexed.
- Pick your three most important pages and look for
rel="canonical"in the page source. It should point to that same page, at its final address — same domain, samehttps, no redirect in between. - If your site is in more than one language, check that each version lists the others and that they list it back.
- Pages you hide from search on purpose (legal notices, thank-you pages) should not appear in the sitemap at all.
- If every date in the sitemap is today's, it is being generated wrong.
Keep it from coming back
The deeper lesson is the one from every quiet failure: it came back the moment nobody was looking. So the check does not depend on anyone remembering it anymore. A small script audits my own sites every morning — pages that are linked but missing from the sitemap, sitemap entries that are hidden from search or do not load, canonicals that point somewhere else — and tells me what it found. It does not change anything on its own; it just makes sure the next problem is noticed in a day, not in a year.
If you would like me to look at yours, tell me what you have and I will tell you honestly whether there is anything worth fixing.
Keep reading
One laptop, $6.3 billion: what the Binance hack teaches small teams
Binance says the biggest crypto theft on record started with an engineer's compromised laptop. You are not an exchange, but the lessons about access, limits and alarms scale all the way down.
Read →Claude Duet and the agent that asks first
Anthropic's new proactive agent works in the background and waits for a yes before it sends, pays, signs or deletes anything. That design choice is the real news, and it is a good rule for any agent you let near your business.
Read →