Easy way to prevent robots indexing resource folders and content?

[teefers (Doug Bennett)]

If what you want is to stop search engines from indexing a folder, the robots.txt file should help.
Some things about a robots.txt file:

• It provides no protection, if they have the link, they can get the resource
• A link from anywhere (without a nofollow) may still get indexed
• The robots.txt file is in the open and is often targeted by hackers to find “hidden” resources

Locked password store page - robots - to follow or not?

Hi Lisa, Sorry I missed your reply on this. Did you get this figured out? I think the problem your having with google flagging the 401 paages is in that it can not read the pages (htaccess blocking them) to see your no index meta tag. Since the store pages are in your navigation (and sitemap) search engines will try to read them. You might want to put a robots.txt file in the top level directory (root) of your site: It is a simple text file must be called: robots.txt musst be placed in t…