Can search engines find your members-only pages?
Your sitemap tells Google which pages matter. We check which members pages it lists, which ones only robots.txt is hiding, and which have no noindex.
The short answer
Can search engines find your members-only pages?
They can if your sitemap lists them or nothing tells search engines not to index them. A members-only page should be left out of your sitemap and carry a noindex, in a robots meta tag or an X-Robots-Tag header; a robots.txt block stops crawling, not indexing. The checker reads your sitemap and robots.txt and checks up to 25 members-only pages, logged out, for all three.
Questions about sitemaps and members pages
What this checker looks at, what it can't see from outside, and how to fix what it finds in Webflow.
It reads your robots.txt and your sitemap, then works out which of the listed pages are members-only. If your homepage loads Memberstack, it reads the page and folder gating rules from your Memberstack app, the same rules Memberstack's script uses in a visitor's browser. You can also type in paths or folders you've gated. Each members-only page is then checked three ways: is it in your sitemap, does robots.txt block Googlebot from it, and does it carry a noindex in a robots meta tag or an X-Robots-Tag header. It fetches up to 25 of those pages, logged out.
Yes, for pages it finds in your sitemap. If your robots.txt asks crawlers like ours to stay out, we don't fetch those pages and say so in the report. Paths you type in yourself are fetched regardless, because a members area is exactly what robots.txt tends to disallow. The checker reads robots.txt a second time as Googlebot, to report what Google is told.
No. That's between your site and Google, and we can't see it from the outside. This checker reports what your site tells search engines: which members pages your sitemap lists, which ones robots.txt blocks, and which ones ask not to be indexed.
Google describes a sitemap as the file that tells search engines which pages you think are important. Listing a members page asks for it to be found, and anyone can open the file and see where your members area lives. Memberstack's gating checklist recommends removing gated pages from your sitemap.
No. Google says robots.txt is not a way to keep a page out of Google: a disallowed page can still be indexed if it's linked from other sites. It also means Google can't read a noindex on that page, because a noindex only works on a page that robots.txt doesn't block. Webflow's docs give the same advice: don't use robots.txt to hide a page, use the Sitemap indexing toggle.
A robots meta tag with noindex or none in its content, or the same rule in a googlebot meta tag or an X-Robots-Tag response header. Google treats none as noindex plus nofollow, reads the names and rules case-insensitively, and applies the most restrictive rule when several are present. Rules addressed to other crawlers, like bingbot, aren't counted.
Open the page, go to Page settings > SEO settings, turn Sitemap indexing off, and publish. Webflow adds a noindex meta tag to the page and removes it from your sitemap. The toggle only works when the auto-generated sitemap is on, under Site settings > SEO > Sitemap. For a CMS item, open the item and use the Sitemap Indexing toggle in Item details, which needs a paid Site plan.
Webflow keeps a page in the auto-generated sitemap when the noindex comes from custom code, and when robots.txt blocks it. Only the Sitemap indexing toggle takes it out. Google will still drop a noindexed page from its results, but the sitemap goes on listing it.
That isn't what gating is for. Memberstack gates pages with JavaScript in the browser, and your sitemap and noindex settings live in Webflow, so you set them there. Memberstack's checklist covers both sides: remove gated pages from your sitemap, and never link to gated content from a public page.
More free tools.
No signup, no catch. Each one answers a question builders keep asking about their membership.
Attribute Generator
Which data attributes do I need for this?
A signup form is four attributes across three elements, and that's the simple one. Pick what you're building and get them as a checklist, on the right elements, in the order you'd select them in Webflow.
- Free
- No signup
- Runs in your browser
Content Flash Checker
Does your gated content flash before it's hidden?
On a slow connection, members-only content can show for a moment before Memberstack hides it. We load your page on a throttled browser and show you exactly what appeared, and for how long.
- Free
- No signup
- Real browser, slow 4G
Membership X-ray
How is that membership site actually built?
Paste any membership or paywalled site. We read the page a logged-out visitor gets and show the membership platform, payments, login flow, pricing tiers and gated content behind it, plus how you'd rebuild it on Memberstack.
- Free
- No signup
- Any site