How do you prevent a page form beeing seen by any search engines

RapidWeaver Classic
1

Hi

I am building af community site in RW6, Stacks 3, and Foundation.
Within that site I have some private pages meant to be completely private for one member user and a group og members that this user chooses.

I am using Sitelok to handle the login security matters, which it does absolutely perfect :slight_smile:

My question is how I, in order to keep the pages totally private, can protect these private pages from beeing seen by any search engines.

Thanks in advance :slight_smile:

2

Hi, I think the better way to do that is using disallow command in your htaccess file…

3

Thanks @Persetcom

How do I do that?
You should know that I am a not “code-experienced” guy, so I don’t know where to find the htaccess file, and I don’t know what to put in it where :slight_smile:

Can you help me out there too?

4

That must be a file called robots.txt in the roots (the same location like htaccess file) of your webspace, with this content.
I have blocked this 5 folders from search engines.

User-Agent: *
Allow: /

Disallow: /testrw5/
Disallow: /testsites/
Disallow: /testsites5/
Disallow: /kochkurs13/
Disallow: /wordpress/

5

Thanks,

But I don’t have a file called robots.txt

6

You can easily create and manage your robots.txt file with RapidBot.

7

You must create this file with Textedit, Text Wrangler etc. and bring it to the server with Cyberduck and Co.

8

Sorry, it is on “robots.txt” (not htaccess) :blush:
ex :
User-agent: *
Disallow: /folder_name/

robots.txt is in you site root (access by /www in your server)
If you have not, you’ll have to create one in your Text Edit, name it “robots.txt” and place it in your root server (or the root folder of your website)

9

Thanks everybody.

@rob I have just bought and downloaded RapidBot, but I never got an registration e-mail so am not able to register the product. Do you know how to get the e-mail?

10

License has been promptly sent to your email address.
Please check your Spam folder.
If you still can’t see it, you can retrieve it from here.
https://4gnd.com/licenses

If still nothing, please contact us at http://help.4gnd.com

11

Haven’t recieved anything - not even in the spam folder :slight_smile:
Tried the licenses link - it says that everything is sent.
The help link give an certificate error an reccommends that I move back …

12

Got through via the help link and have posted a note there. No reaction

Still absolutely nothing in my mailbox.

13

Still haven’t got or heard anything form anybody. Strange …

14

You can learn about this and other .htaccess ideas by watching these two WeaverCasts Joe Workman did


15

Sorry for the delay.
Lunch time here in Italy.
I’ve just replied to your support request attaching your license that has been sent on purchase.

16

Thanks @rob

Haven’t head or recieved anything yet.
Could you please provide them with a coupe of alternate e-mai adresses - just in case :slight_smile:

pkr@mail.dk
kroman_peter@live.dk
pkr828gmail.com
peter_kroman@genealogiskforum.dk

17

@peterkroman Just resent to all your four email addresses.

18

Thanks @rob
Appreciate it very much

19

But is it correct that RapidBot does not create a robots.txt file?

I have one in the root , but it is quite empty. also after registrating RapidBot and uploading again.

What to do?

20

We’ve got a couple of video tutorials on the RapidWeaver Community that show you how to do this.

First one is Control how Google indexes your site — this one is good if you anyone to be able to visit the page, but have google exclude it from their search results. I cover a couple of different techniques (robots.txt, meta tags, and RapidBot).

And the other one is Basic Authentication using htaccess — this one is good if you want to password protect a page(s). Password protecting the page will also stop google from indexing the page(s), as the google bots won’t be able to access the page.

Note: You’ll need a RapidWeaver Community subscription to watch both of those videos.

Hope that helps :slight_smile: