# Implementing robots.txt for my existing site within RW 8.7 **URL:** \ **Category:** RapidWeaver Classic\ **Created:** [December 3, 2020, 4:12pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040 "2020-12-03T16:12:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Vlask0vec2020](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/vlask0vec2020/32/21036_2.png) [@Vlask0vec2020](https://forums.realmacsoftware.com/u/Vlask0vec2020)\ **Post date:** [December 3, 2020, 4:12pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/1 "2020-12-03T16:12:58Z")
After an extensive review by an SEO site , besides some other minor things, they suggested I download a generator for robots and do it myself. I did that and got a whole bunch of “text/code”?..but have no idea where to put it. Or does RW have a way of generating it itself? R ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 3, 2020, 4:27pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/2 "2020-12-03T16:27:23Z")
You’ll have to log into your server and edit the file there. RW doesn’t support files outside of its control except for an `Edit .htaccess File` button on the publishing settings page. I’ve always wished that it would add support for this kind of stuff. Just an edit and publish functionality would be nice. @dan ---
**Author:** ![joeworkman](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/joeworkman/32/23780_2.png) [@joeworkman](https://forums.realmacsoftware.com/u/joeworkman)\ **Post date:** [December 3, 2020, 4:30pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/3 "2020-12-03T16:30:36Z")
- Add an HTML page to your project. - Set the page to not use a theme in the inspector. - Name the page robots.txt - Populate the [robots.txt rules](https://www.robotstxt.org) - Publish 🙂 ---
**Author:** ![joeworkman](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/joeworkman/32/23780_2.png) [@joeworkman](https://forums.realmacsoftware.com/u/joeworkman)\ **Post date:** [December 3, 2020, 4:34pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/4 "2020-12-03T16:34:19Z")
I spent a week trying to build a robots.txt builder with Stacks. Sadly I hit some roadblocks that were out of my control. Maybe one day. 🙂 ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 3, 2020, 4:45pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/5 "2020-12-03T16:45:48Z")
> [@joeworkman](#): > > - Add an HTML page to your project. > - Set the page to not use a theme in the inspector. > - Name the page robots.txt > - Populate the [robots.txt rules](https://www.robotstxt.org) > - Publish > > 🙂 Yea, I get that this works, but it feels kinda clutzy to me. A text file page that doesn’t allow a theme would have a cleaner feel to it, but sometimes you have to work around various software limitations. ---
**Author:** ![Vlask0vec2020](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/vlask0vec2020/32/21036_2.png) [@Vlask0vec2020](https://forums.realmacsoftware.com/u/Vlask0vec2020)\ **Post date:** [December 3, 2020, 4:47pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/6 "2020-12-03T16:47:19Z")
Great Joe…I did all up to populate the robot.txt rules…I took on the link and then I stopped…populate with what? Probably a dumb question, but…This is what SEO site gave me when I downloaded their suggestion: robots.txt generated by [www.seoptimer.com](http://www.seoptimer.com) User-agent: Googlebot Disallow: User-agent: googlebot-image Disallow: User-agent: googlebot-mobile Disallow: User-agent: MSNBot Disallow: User-agent: Slurp Disallow: User-agent: yahoo-mmcrawler Disallow: User-agent: yahoo-blogs/v3.9 Disallow: User-agent: \* Disallow: Disallow: /cgi-bin/ ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 3, 2020, 4:48pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/7 "2020-12-03T16:48:58Z")
All but the `generated by` line would be what you’d put in that page. ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 3, 2020, 4:51pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/8 "2020-12-03T16:51:00Z")
I kinda semi-hijacked this thread, but I’m thinking .htaccess, robots.txt, all the appcast stuff, sendy (or any other thirdparty code/tools), error pages, … ---
**Author:** ![Vlask0vec2020](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/vlask0vec2020/32/21036_2.png) [@Vlask0vec2020](https://forums.realmacsoftware.com/u/Vlask0vec2020)\ **Post date:** [December 3, 2020, 4:54pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/9 "2020-12-03T16:54:58Z")
Hey Rodney…don’t worry about semi-hijacking. So what you’re saying is copy and paste everything starting with User-agent, etc… ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 3, 2020, 4:56pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/10 "2020-12-03T16:56:45Z")
yes, that is correct you could add a # before the generated by line to make it a comment if you want to keep the credit there. ---
**Author:** ![joeworkman](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/joeworkman/32/23780_2.png) [@joeworkman](https://forums.realmacsoftware.com/u/joeworkman)\ **Post date:** [December 3, 2020, 5:00pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/11 "2020-12-03T17:00:26Z")
> [@forbesrodney](#): > > Yea, I get that this works, but it feels kinda clutzy to me. A text file page that doesn’t allow a theme would have a cleaner feel to it, but sometimes you have to work around various software limitations. If there was a text page type, the only step that would be removed is clicking one checkbox. As a dev, I am not sure the extra overhead of maintaining another page type is worth it. You cannot use this for htaccess though. RW does not allow you to publish pages as dot files. That could be pretty cool though. ---
**Author:** ![Vlask0vec2020](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/vlask0vec2020/32/21036_2.png) [@Vlask0vec2020](https://forums.realmacsoftware.com/u/Vlask0vec2020)\ **Post date:** [December 3, 2020, 5:04pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/12 "2020-12-03T17:04:09Z")
OK…when I preview and go to my Menu it is “not” there…so it is “hidden” ? Always scared it will publish it as a page. I’m publishing…crossed fingers. Thanx ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 3, 2020, 5:05pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/13 "2020-12-03T17:05:35Z")
If `Show in navigation` is unchecked in the page inspector, you should be good. ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 3, 2020, 5:07pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/14 "2020-12-03T17:07:55Z")
> [@joeworkman](#): > > If there was a text page type, the only step that would be removed is clicking one checkbox. As a dev, I am not sure the extra overhead of maintaining another page type is worth it. The developer in me just doesn’t like things feeling clunky, but I get what you’re saying. ---
**Author:** ![Vlask0vec2020](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/vlask0vec2020/32/21036_2.png) [@Vlask0vec2020](https://forums.realmacsoftware.com/u/Vlask0vec2020)\ **Post date:** [December 3, 2020, 5:49pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/15 "2020-12-03T17:49:22Z")
I’m good. All seems OK…have other issues but that is for another day. I’m stuck in Europe because of Covid restrictions so it’s Netflix time…going googly eyed working on this all day, ha. Thanx guys. Ciao for now. ---
**Author:** ![Vlask0vec2020](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/vlask0vec2020/32/21036_2.png) [@Vlask0vec2020](https://forums.realmacsoftware.com/u/Vlask0vec2020)\ **Post date:** [December 4, 2020, 1:57pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/16 "2020-12-04T13:57:26Z")
Gentlemen, Thanx for everything yesterday…but, fresh analyses of the SEO tells me that no sign of robots.txt is seen yet. I should be patient? I assume it takes a bit of time…or? Thanx in advance, RM ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 4, 2020, 4:15pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/17 "2020-12-04T16:15:41Z")
double check for me that in the page inspector, - `Filename` is `robots.txt` - `Show in navigation` is unchecked - `Draft` is unchecked - `Folder` is `/` ![Screen Shot 2020-12-04 at 11.18.38 AM](https://europe1.discourse-cdn.com/flex005/uploads/realmacsoftware1/original/3X/7/7/776056edd5566443c7bf4ddad2ca414a3d02e377.png) ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 4, 2020, 4:38pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/18 "2020-12-04T16:38:01Z")
oh, and to get rid of the theme, I think you want this ![Screen Shot 2020-12-04 at 11.36.18 AM](https://europe1.discourse-cdn.com/flex005/uploads/realmacsoftware1/original/3X/1/e/1ed8a0001d38cffede50f52341799818bdfdc10d.png) ---
**Author:** ![forbesrodney](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/forbesrodney/32/20067_2.png) [@forbesrodney](https://forums.realmacsoftware.com/u/forbesrodney)\ **Post date:** [December 4, 2020, 4:38pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/19 "2020-12-04T16:38:57Z")
And I just tried this in my current project, and was successful ---
**Author:** ![joeworkman](https://dub1.discourse-cdn.com/flex005/user_avatar/forums.realmacsoftware.com/joeworkman/32/23780_2.png) [@joeworkman](https://forums.realmacsoftware.com/u/joeworkman)\ **Post date:** [December 4, 2020, 7:12pm UTC](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040/20 "2020-12-04T19:12:26Z")
The trick is that you need to make sure that this is not checked… ![CleanShot 2020-12-04 at 11.11.42](https://europe1.discourse-cdn.com/flex005/uploads/realmacsoftware1/original/3X/5/e/5e669fc6409307982e8f85d8f2ffe9ecff575bff.jpeg) It was off by default for me. [Next page](https://forums.realmacsoftware.com/t/implementing-robots-txt-for-my-existing-site-within-rw-8-7/36040.md?page=2)