# Easy way to prevent robots indexing resource folders and content? **URL:** \ **Category:** RapidWeaver Classic\ **Created:** [January 15, 2018, 12:41pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072 "2018-01-15T12:41:42Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![manofdogz](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/manofdogz/32/26314_2.png) [@manofdogz](https://forums.realmacsoftware.com/u/manofdogz)\ **Post date:** [January 15, 2018, 12:41pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/1 "2018-01-15T12:41:42Z")
I’m sure I have protected RW Resources in the past using .htaccess, but before I go trawling through stuff remembering how to do it I’m just wondering if anyone knows a simple way to do this via a stack or plugin. I’m sure it is a common problem where password secured pages are rendered partly useless because linked resource files in the page are indexed by Google and available to search ---
**Author:** ![teefers](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/teefers/32/8173_2.png) [@teefers](https://forums.realmacsoftware.com/u/teefers)\ **Post date:** [January 15, 2018, 1:14pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/2 "2018-01-15T13:14:22Z")
If what you want is to stop search engines from indexing a folder, the robots.txt file should help. Some things about a robots.txt file: - It provides **no protection** , if they have the link, they can get the resource - A link from anywhere (without a nofollow) may still get indexed - The robots.txt file is in the open and is often targeted by hackers to find “hidden” resources > [@Locked password store page - robots - to follow or not?](http://forums.realmacsoftware.com/t/locked-password-store-page-robots-to-follow-or-not/11974/11): > > Hi Lisa, Sorry I missed your reply on this. Did you get this figured out? I think the problem your having with google flagging the 401 paages is in that it can not read the pages (htaccess blocking them) to see your no index meta tag. Since the store pages are in your navigation (and sitemap) search engines will try to read them. You might want to put a robots.txt file in the top level directory (root) of your site: It is a simple text file must be called: robots.txt musst be placed in t… ---
**Author:** ![teefers](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/teefers/32/8173_2.png) [@teefers](https://forums.realmacsoftware.com/u/teefers)\ **Post date:** [January 15, 2018, 1:23pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/3 "2018-01-15T13:23:43Z")
> [@manofdogz](#): > > password secured pages are rendered partly useless because linked resource files in the page are indexed BTW Password protected pages are **not** index by search engines. ---
**Author:** ![manofdogz](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/manofdogz/32/26314_2.png) [@manofdogz](https://forums.realmacsoftware.com/u/manofdogz)\ **Post date:** [January 15, 2018, 1:35pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/4 "2018-01-15T13:35:15Z")
I’m aware of that but in this particular case there are resource files that are linked to from a password protected page. The links aren’t followed but the ‘private’ files are still found by search engines because the resource folders are being indexed ---
**Author:** ![teefers](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/teefers/32/8173_2.png) [@teefers](https://forums.realmacsoftware.com/u/teefers)\ **Post date:** [January 15, 2018, 2:03pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/5 "2018-01-15T14:03:33Z")
You should if not already be blocking directory listings on your site. That would prevent any ligament crawler from “finding” files, or directories not referenced directly. This should be set for general security anyway. Add this line to your .htaccess file. ``` Options -Indexes ``` A robots.txt file should prevent any folder from being indexed, but as I said above it’s not really a security thing. Even htaccess files have weaknesses. It really depends on how sensitive what you’re securing is. If it’s downloads being protected then there’s a lot of options for secure cloud files (paid). if you’re securing through Sitlok they offers some hidden links options. [https://vibralogix.helpscoutdocs.com/article/19-display-a-secure-tree-view-of-files-for-download-in-sitelok](https://vibralogix.helpscoutdocs.com/article/19-display-a-secure-tree-view-of-files-for-download-in-sitelok) ---
**Author:** ![Rovertek](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/rovertek/32/223_2.png) [@Rovertek](https://forums.realmacsoftware.com/u/Rovertek)\ **Post date:** [January 15, 2018, 4:23pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/6 "2018-01-15T16:23:43Z")
If you are using Sitelok for password-protected areas, you may be interested in Linklok plugin that specifically protects links to resources. You may want to double-check this info, because I do not use Linklok myself. ---
**Author:** ![manofdogz](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/manofdogz/32/26314_2.png) [@manofdogz](https://forums.realmacsoftware.com/u/manofdogz)\ **Post date:** [January 20, 2018, 4:27pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/7 "2018-01-20T16:27:24Z")
Moving to Sitelok this coming week so yes that is helpful to know. ---
**Author:** ![system](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/system/32/25301_2.png) [@system](https://forums.realmacsoftware.com/u/system)\ **Post date:** [January 26, 2018, 4:27pm UTC](https://forums.realmacsoftware.com/t/easy-way-to-prevent-robots-indexing-resource-folders-and-content/18072/8 "2018-01-26T16:27:32Z")
This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.