Blocking Google; remaking website on an empy website

RapidWeaver Classic
1

I am in the process of redoing my website.
I prefer to redo my current website on a completely different, clean website because I have a mess on the original site.

I want to allow places like Screaming frog access my website.
I do Not want Google or Bing to be able to access this hidden website.

Is this possible???
Thank you!
Kim

2

@kim do you want the site to be publicly viewable or private for you only? Do you want to maintain the current site live and make a second development site or is it okay to stop access to a live site?

3

I really don’t care if others can see the development site. I just don’t want google or bing to index it and get penalized on my active site. I want the second to be a development site. I must keep my active site active.
Thank you Barchard

4

Hi @kim,

Assuming your new dev website is a directory called /dev of your primary website, you can setup your robots.txt as follows:

User-agent: Googlebot
Disallow: /dev/

This will tell Google to not index whatever is in the directory /dev on your webserver, so what you do in there doesn’t impact what Google thinks of the rest of your site.

You can set up robots.txt by putting a file with that name in the root of your website (where the main index file is located). The file should (in your case) only contain the example text that I gave you, but you can find numerous tutorials online which explain what else you can put in there.

Cheers,
Erwin

5

Thank you Erwin
That is great advice.
Thank you for your help. :smile:
Kim

6

Yes, what @Heroic_Nonsense said with a couple of caveats,

  1. this only covers Google, you’ll want to expand it to include Bing. Here is a list of Bing’s UserAgents
  2. here are all of google’s user agent strings (for future reference), just the 1 should suffice.
7

Thank you Gregory!
I needed this too. :smile:
Kim