Locked password store page - robots - to follow or not?

(Lisa Sandler) #1

Getting my SEO ducks in a row… I have my store and store sub pages locked for clients to enter only with a user name and password…
As a good practice, should I uncheck these robot boxes for all my locked pages so Google doesn’t show 401 errors?

(Oliver) #2

Hello @LSPhoto,

I believe at this point this is the best you can do.

Another possibility is to use the RapidBot Plugin by Foreground to directly disallow trying to crawl this pages.

Happy weaving,

(Doug Bennett) #3

Password protection should prevent any search engine from accessing content on that page. The major engines won’t try to guess passwords or bypass protection.
Now they may index links to that page so that they will know the page exist and show that in results. Chances of anyone finding that in a search would be slim, and if they did it would come up 401.

That being said I would uncheck them won’t harm anything.

I also would do a custom 401 page, maybe one that has links to a login or signup.

(Lisa Sandler) #4

Thanks @Noridge and @teefers.

The custom 401 page… how do I create that? If anyone clicks on any of the store pages, they get a login. Then, I guess if they hit cancel, they get the 401 page you are referring to? Do I really need the 401 page?

(Oliver) #5

Its always good to have some custom error-sites.

You can create them in RW, of course without being shown in the menu or the sitemap.
To use them you need to create in your root directory a .htaccess file with this content:

ErrorDocument 400 /error/400.php
ErrorDocument 401 /error/401.php
ErrorDocument 403 /error/403.php
ErrorDocument 404 /error/404.php
ErrorDocument 500 /error/500.php 

In my case /error is the folder where I stored my error pages, of course you also can have your error pages wherever you want, just be sure for the path being correct. You also don’t need to have error pages for all of those errors shown above, but those are the most common…

Happy weaving,

(Lisa Sandler) #6

Right, I know how to create the pages and not show in menu, but I don’t know how to redirect? Is that the right word?
Can you give me a little more direction? I have an .htacces file already… that is how I locked my pages. Do I open the .htaccess file and put the code in there? Is “ErrorDocument” the name you gave your page? or is it code?


(Doug Bennett) #7

The code in @Noridge’s post is htaccess file code. The ErrorDocument statement is a htaccess directive.
the number following that is the error code followed by the address to the page you want the redirect to go.

Yes put it in your current htaccess file.

(Lisa Sandler) #8

Thanks. I’ll take a pass at this when I have some time.

(Doug Bennett) #9

Is your locked down store page for testing?
that is what I was thinking it was for.
If not then how does a customer get a username and password.
That was was I was suggesting a custom 401 page. If it is testing then you would want to tell the customer that its not available yet. If they need to setup an account first then have a link to were the set one up.

(Lisa Sandler) #10

No. It’s not for testing. Each client gets a user name and pass to order from the store when I send them their proofing gallery. I am the one who sets up the user/pass for them.

I asked this question because I see that Google is flagging the 401 pages.

(Doug Bennett) #11

Hi Lisa,
Sorry I missed your reply on this. Did you get this figured out?
I think the problem your having with google flagging the 401 paages is in that it can not read the pages (htaccess blocking them) to see your no index meta tag.
Since the store pages are in your navigation (and sitemap) search engines will try to read them.
You might want to put a robots.txt file in the top level directory (root) of your site:
It is a simple text file must be called:
musst be placed in the root directory (were you index.html for your home page is.
It is case-sensitive.

looking at your site (assuming all pages to be blocked are in client-store or below that) this should work:

User-agent: *
Disallow: /client-store*

You can find more about robots.txt files here:

You can also test it here:

Easy way to prevent robots indexing resource folders and content?
RapidWeaver SEO Articles
(Lisa Sandler) #12


Yes, thank you. I understand that it can’t read the 401 pages, and I have heard about robot.txt files. I might try that one of these days. I can probably use Text Edit to create that file?

I’m still trying to decide if I should approach the store from a different way… maybe have a real “joiner” kind of sign in for the store, where the client creates the user name and password.

Do you know if I can do that in Chillidog or with another stack? I’ve not researched this at all yet.

thanks, Lisa

(Doug Bennett) #13

You can use any text editor (plan text) to create a robots.txt file. You can also add it to RW diretly.

  1. Add a HTML page
  2. Remove formatting by unchecking Apply Theme
  3. Remove from navigation
  4. Name it robots.txt
  5. Set the directory to /

Then just add the text.
Its important to test your robots.txt file. Google has a test area in the Webmaster search console:

As for membership type software you might want to check out Sitelock:

(Lisa Sandler) #14

wow. I had no idea I could create a page in RW for the robots file. Thank you!

and I’ll check out Sitelock. I’ve heard of it but haven’t looked at it.