Locked password store page - robots - to follow or not?

Hi Lisa,
Sorry I missed your reply on this. Did you get this figured out?
I think the problem your having with google flagging the 401 paages is in that it can not read the pages (htaccess blocking them) to see your no index meta tag.
Since the store pages are in your navigation (and sitemap) search engines will try to read them.
You might want to put a robots.txt file in the top level directory (root) of your site:
It is a simple text file must be called:
robots.txt
musst be placed in the root directory (were you index.html for your home page is.
It is case-sensitive.

looking at your site (assuming all pages to be blocked are in client-store or below that) this should work:

User-agent: *
Disallow: /client-store*

You can find more about robots.txt files here:
https://support.google.com/webmasters/answer/6062608?hl=en

You can also test it here:
https://www.google.com/webmasters/tools/robots-testing-tool/

1 Like