# What do I need to specify in my robots.txt fie

**URL:** <https://forums.realmacsoftware.com/t/what-do-i-need-to-specify-in-my-robots-txt-fie/20087>\
**Category:** RapidWeaver Classic\
**Created:** [May 9, 2018, 3:41pm UTC](https://forums.realmacsoftware.com/t/what-do-i-need-to-specify-in-my-robots-txt-fie/20087 "2018-05-09T15:41:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![kentuckienne](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/kentuckienne/32/9463_2.png) [@kentuckienne](https://forums.realmacsoftware.com/u/kentuckienne)\
**Post date:** [May 9, 2018, 3:41pm UTC](https://forums.realmacsoftware.com/t/what-do-i-need-to-specify-in-my-robots-txt-fie/20087/1 "2018-05-09T15:41:45Z")

</div>

I received an AdGrant from Google, but my site ([www.wikibeaks.org](http://www.wikibeaks.org)) keeps getting disallowed. One of the complaints from the Google console was that the crawler couldn’t find a robots.txt file, so it can’t index the site - that’s the default now, so the crawler doesn’t expose private links. I thought that checking the “(robots)index this page” box would take care of this, but it seems it doesn’t. So I created a simple robots.txt file. I’m not sure what I need to put in it … I want the crawler to only see the published pages, and it seems too much hassle to explicitly allow them. I need to get this working so I can get re-indexed and re-approved.

# What directories/files should I exclude? Here’s what I have now:

> # robots.txt file created for [http://www.wikibeaks.com/](http://www.wikibeaks.com/)
> 
> # May 8, 2018
> 
> # Exclude Files From All Robots:
> 
> User-agent: \*  
> Disallow: /resources  
> Disallow: /rw\_common  
> Disallow: /blog\_files  
> Disallow: /markdown  
> Disallow: /Wikibeaks\_readme.txt
> 
> # End robots.txt file

==================

---

<div class="post-metadata">

**Author:** ![NeilUK](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/neiluk/32/23434_2.png) [@NeilUK](https://forums.realmacsoftware.com/u/NeilUK)\
**Post date:** [May 9, 2018, 3:48pm UTC](https://forums.realmacsoftware.com/t/what-do-i-need-to-specify-in-my-robots-txt-fie/20087/2 "2018-05-09T15:48:37Z")

</div>

You should allow everything, and then specify the pages you don’t want to be crawled/indexed. It looks like you’re blocking the a lot of important stuff that would cause issues.

If you REALLY don’t want something seen on the internet, you shouldn’t publish it.

---

<div class="post-metadata">

**Author:** ![TechBill](https://avatars.discourse-cdn.com/v4/letter/t/73ab20/32.png) [@TechBill](https://forums.realmacsoftware.com/u/TechBill)\
**Post date:** [May 9, 2018, 7:56pm UTC](https://forums.realmacsoftware.com/t/what-do-i-need-to-specify-in-my-robots-txt-fie/20087/3 "2018-05-09T19:56:49Z")

</div>

Why not go for the whole shebang and enter this in your robot.txt 😄

User-agent: \*  
Disallow: /

---

<div class="post-metadata">

**Author:** ![kentuckienne](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/kentuckienne/32/9463_2.png) [@kentuckienne](https://forums.realmacsoftware.com/u/kentuckienne)\
**Post date:** [May 9, 2018, 8:01pm UTC](https://forums.realmacsoftware.com/t/what-do-i-need-to-specify-in-my-robots-txt-fie/20087/4 "2018-05-09T20:01:40Z")

</div>

I didn’t know how much stuff - such as page drafts would be indexed and searchable. I only want to block the minimum amount of stuff. If I don’t need to block the RW general files, I’ll just block the readme - which is just a note reminding me not to delete the Google authentication file.

\*\* Just got off the phone with Google AdWords. They had me modify my robots.txt file as below, and it’s now submitted for review. Apparently needs the specific google permissions … asterisk isn’t good enough.

# robots.txt file created for [http://www.wikibeaks.com/](http://www.wikibeaks.com/)

# May 8, 2018

# Exclude Files From All Robots:

User-agent: \*  
Disallow:

User-agent: Googlebot  
Disallow:

User-agent: Googlebot-image  
Disallow:

# End robots.txt file

---

<div class="post-metadata">

**Author:** ![system](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/system/32/25301_2.png) [@system](https://forums.realmacsoftware.com/u/system)\
**Post date:** [June 9, 2018, 6:01am UTC](https://forums.realmacsoftware.com/t/what-do-i-need-to-specify-in-my-robots-txt-fie/20087/5 "2018-06-09T06:01:40Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.