Easy way to prevent robots indexing resource folders and content?

I’m sure I have protected RW Resources in the past using .htaccess, but before I go trawling through stuff remembering how to do it I’m just wondering if anyone knows a simple way to do this via a stack or plugin.

I’m sure it is a common problem where password secured pages are rendered partly useless because linked resource files in the page are indexed by Google and available to search

If what you want is to stop search engines from indexing a folder, the robots.txt file should help.
Some things about a robots.txt file:

  • It provides no protection, if they have the link, they can get the resource
  • A link from anywhere (without a nofollow) may still get indexed
  • The robots.txt file is in the open and is often targeted by hackers to find “hidden” resources

BTW
Password protected pages are not index by search engines.

I’m aware of that but in this particular case there are resource files that are linked to from a password protected page. The links aren’t followed but the ‘private’ files are still found by search engines because the resource folders are being indexed

You should if not already be blocking directory listings on your site. That would prevent any ligament crawler from “finding” files, or directories not referenced directly. This should be set for general security anyway.
Add this line to your .htaccess file.

Options -Indexes

A robots.txt file should prevent any folder from being indexed, but as I said above it’s not really a security thing. Even htaccess files have weaknesses.
It really depends on how sensitive what you’re securing is.
If it’s downloads being protected then there’s a lot of options for secure cloud files (paid). if you’re securing through Sitlok they offers some hidden links options.
https://vibralogix.helpscoutdocs.com/article/19-display-a-secure-tree-view-of-files-for-download-in-sitelok

2 Likes

If you are using Sitelok for password-protected areas, you may be interested in Linklok plugin that specifically protects links to resources. You may want to double-check this info, because I do not use Linklok myself.

1 Like

Moving to Sitelok this coming week so yes that is helpful to know.

This topic was automatically closed 6 days after the last reply. New replies are no longer allowed.