Google Was Only Indexing 28 of Our 400+ Pages. A Single DNS Setting Was the Cause.
We pulled up Google Search Console and saw a number that stopped us cold.
28 pages indexed.
Our site had over 400 pages. Thousands of photos. Blog posts going back over a year. Vendor profiles for 75+ local wedding professionals. Google knew about all of it. It just was not indexing most of it.
The breakdown in GSC made it look even worse:
- 302 pages: "Discovered, currently not indexed" (Google found them but chose not to crawl)
- 70 pages: "Crawled, currently not indexed" (Google visited but decided not to index)
- 15 pages: Soft 404s
- 7 pages: Actual 404 errors
We had been publishing new content, optimizing pages, building internal links. All of it was essentially invisible to Google. And the worst part was that nothing looked wrong from the outside.
The Problem Was Completely Invisible in a Browser
When you visit scarboroughhouse.com in Chrome or Safari, the experience is seamless. You type the address, you land on the site, everything loads. No error messages. No warnings. No broken redirects you can see.
That is exactly what makes this class of problem so dangerous. The bug was not in the content, the design, or the server code. It was one level deeper, in the infrastructure layer that handles how your domain name resolves to your website.
To find it, we had to pull the raw HTTP response headers directly.
curl -I http://scarboroughhouse.comThat command tells you exactly what a server returns before it sends any page content. And in that output, buried in a line most people would scroll past, was the problem:
Location: https://scarboroughhouse.com:443/Notice the :443 at the end of the URL.
Port 443 is the default port for HTTPS. It is so standard that it is normally invisible. When you type https://scarboroughhouse.com, your browser is connecting to port 443 whether it says so or not. Browsers are smart enough to strip that out and show you a clean URL.
Google's crawler is not as forgiving.
How Google Reads a Redirect URL
When Google's crawler follows a redirect, it reads the Location header literally. To Google, these are two completely different URLs:
https://scarboroughhouse.com/https://scarboroughhouse.com:443/
Same content. Same server. Same everything from a human perspective. But Google's indexing system treats them as separate addresses.
What happens when Google finds the same content at two different URLs? It has to pick one as the canonical version and either hold or discard the other. While it is sorting that out, it stops indexing. It puts pages in a "discovered, not indexed" queue. It waits.
That queue had 302 of our pages sitting in it.
We had been running this way for long enough that Google built up a significant backlog of pages it had found through our sitemap and internal links, but refused to commit to indexing because the redirect was telling it two different canonical addresses for the root domain.
Why This Happened: Cloudflare Proxy Mode
We use Cloudflare for DNS and security. Cloudflare is excellent and we recommend it for any venue managing their own domain. But it has a configuration detail that is easy to miss.
In Cloudflare's DNS panel, every record has two states:
- Proxied (orange cloud): Traffic flows through Cloudflare's network. Cloudflare handles SSL, caching, redirects, and DDoS protection before the request ever reaches your hosting server.
- DNS only (grey cloud): Cloudflare just tells browsers where your server lives. Traffic goes directly to your host. Cloudflare's protections and redirect handling are bypassed entirely.
Our www subdomain record was set to proxied. That is why www.scarboroughhouse.com redirected cleanly, every time, with a perfect Location: https://scarboroughhouse.com/ header.
Our apex domain record (the root scarboroughhouse.com, no www) was set to DNS only. So when a browser or Google requested http://scarboroughhouse.com, that request went straight to our hosting server, bypassing Cloudflare entirely. And our hosting server, when it generated the HTTPS redirect, included the explicit port number in the URL.
One orange cloud versus grey cloud toggle on one DNS record. That was the entire root cause.
The Fix Took About Ten Minutes
Once we knew what to look for, the fix was straightforward: flip the apex A record from grey cloud to orange cloud in Cloudflare. One API call, confirmed within seconds.
After that change:
- All HTTP requests to
scarboroughhouse.comnow route through Cloudflare - Cloudflare's "Always Use HTTPS" rule (which was already enabled) generates the redirect
- The redirect goes to
https://scarboroughhouse.com/with no port number appended - Google sees a single canonical URL for our root domain
We confirmed it immediately by running the same curl command again. Clean redirect, no port, every time.
What We Found While We Were Digging
A DNS audit tends to surface more than just one issue. While we had the records open, we found two more problems that were not causing the indexing issue but were creating risk elsewhere.
DMARC was set to "none"
DMARC is a DNS record that tells email servers what to do with messages that claim to come from your domain but fail authentication checks. Our record existed, but it was set to p=none, which means: do nothing, just report.
In practice, p=none means anyone could send a spoofed email appearing to come from @scarboroughhouse.com and most email providers would deliver it. We updated this to p=quarantine, which routes unauthenticated messages to spam instead of inboxes. The next step would be p=reject once we have confirmed all our legitimate sending sources are authenticated.
Our SPF record was missing two major email services
SPF (Sender Policy Framework) is another DNS record that lists which servers are authorized to send email on behalf of your domain. Ours only listed Cloudflare Email Routing. It was missing:
- Resend (Amazon SES infrastructure), which we use for transactional emails like tour confirmations and booking notifications
- SendGrid, which is used by our CRM for automated follow-up sequences
Emails from those services were technically unauthenticated, which increases the chance of landing in spam, especially now that Gmail and Yahoo enforce stricter rules for high-volume senders. Updated both in the same DNS session.
The Final Step: Getting Every URL to Google at Once
Fixing the DNS problem stops the bleeding, but it does not immediately tell Google to go back and re-evaluate every page it had been sitting on. Google recrawls sites on its own schedule, which could mean weeks before it works through our 302-page backlog.
We used IndexNow to skip the wait.
IndexNow is a protocol supported by Google, Bing, and other major search engines that lets you submit a list of URLs for immediate crawl prioritization. Instead of waiting for Google to discover that our pages are now clean and consistent, we sent a single API call with all 2,383 URLs from our sitemap at once.
That batch included:
- 119 blog posts
- 76 vendor profiles
- 62 photo collection pages
- Lodging, grounds, pricing, and content pages
- 2,023 individual photo pages
The response came back HTTP 200. All 2,383 URLs submitted in one call. Google, Bing, and every other IndexNow participant received the notification simultaneously.
We expect the indexed page count to climb noticeably within a week as Google works through the queue now that the canonical confusion is resolved.
What to Check on Your Own Domain
This problem is not specific to our hosting setup. Any venue with a custom domain, especially one using Cloudflare or another CDN in front of their hosting server, could have this exact issue and never know it.
Here is how to check:
Step 1: Run a curl command on your own domain
curl -I http://yourvenuedomain.comLook at the Location: line in the response. It should read something like:
Location: https://yourvenuedomain.com/If it reads:
Location: https://yourvenuedomain.com:443/You have this problem.
Step 2: Check your Cloudflare DNS records
Log into Cloudflare, go to DNS, and find your apex A record (the one with just @ in the name column, pointing to your server IP). The cloud icon next to it should be orange (proxied), not grey (DNS only).
If it is grey, click the toggle to enable proxying. Your HTTPS redirect will immediately start going through Cloudflare, which generates clean redirect URLs without the port suffix.
Step 3: Check your Google Search Console indexing report
Go to Search Console, click Indexing, then Pages. Look at the "Discovered, currently not indexed" count. If that number is large relative to the total pages Google knows about, a canonical issue is often the cause. The content quality problems that lead to "Crawled, not indexed" are a separate issue, but a big "Discovered" backlog almost always points to infrastructure.
Step 4: Verify your DMARC and SPF records
You can check both at MXToolbox for free. Run a DMARC lookup and an SPF lookup for your domain. If DMARC shows p=none or is missing entirely, you have no protection against email spoofing. If SPF is missing the services you actually send email through, your deliverability is at risk.
The Bigger Lesson
We had been doing the right things on the content side: writing in-depth pages, building internal links, submitting a sitemap, maintaining consistent NAP information. None of it moved the indexing needle because the infrastructure underneath it was broken in a way that was completely invisible to us.
Technical SEO is not glamorous. It does not produce immediate visible results the way a new blog post does. But it is the foundation that everything else sits on. If Google cannot reliably determine what your canonical URL is, it does not matter how good your content is. It will sit in a queue.
The fix was one DNS toggle. The audit that found it took a few hours of methodical investigation into records we had never looked at closely. The result will be hundreds of pages seeing their first real chance at indexing.
If you manage your own domain and have not done a DNS and GSC audit recently, this is a good reminder that the problem you cannot see is often the one doing the most damage.
We will follow up with actual indexing numbers once Google works through the backlog. Check back in a few weeks.