After an extensive review by an SEO site , besides some other minor things, they suggested I download a generator for robots and do it myself. I did that and got a whole bunch of “text/code”?..but have no idea where to put it. Or does RW have a way of generating it itself?
R
You’ll have to log into your server and edit the file there.
RW doesn’t support files outside of its control except for an Edit .htaccess File button on the publishing settings page. I’ve always wished that it would add support for this kind of stuff. Just an edit and publish functionality would be nice. @dan
- Add an HTML page to your project.
- Set the page to not use a theme in the inspector.
- Name the page robots.txt
- Populate the robots.txt rules
- Publish
3 Likes
I spent a week trying to build a robots.txt builder with Stacks. Sadly I hit some roadblocks that were out of my control. Maybe one day.
1 Like
Yea, I get that this works, but it feels kinda clutzy to me. A text file page that doesn’t allow a theme would have a cleaner feel to it, but sometimes you have to work around various software limitations.
Great Joe…I did all up to populate the robot.txt rules…I took on the link and then I stopped…populate with what? Probably a dumb question, but…This is what SEO site gave me when I downloaded their suggestion:
robots.txt generated by www.seoptimer.com
User-agent: Googlebot
Disallow:
User-agent: googlebot-image
Disallow:
User-agent: googlebot-mobile
Disallow:
User-agent: MSNBot
Disallow:
User-agent: Slurp
Disallow:
User-agent: yahoo-mmcrawler
Disallow:
User-agent: yahoo-blogs/v3.9
Disallow:
User-agent: *
Disallow:
Disallow: /cgi-bin/
All but the generated by line would be what you’d put in that page.
I kinda semi-hijacked this thread, but I’m thinking .htaccess, robots.txt, all the appcast stuff, sendy (or any other thirdparty code/tools), error pages, …
Hey Rodney…don’t worry about semi-hijacking. So what you’re saying is copy and paste everything starting with User-agent, etc…
yes, that is correct
you could add a # before the generated by line to make it a comment if you want to keep the credit there.
If there was a text page type, the only step that would be removed is clicking one checkbox. As a dev, I am not sure the extra overhead of maintaining another page type is worth it.
You cannot use this for htaccess though. RW does not allow you to publish pages as dot files. That could be pretty cool though.
OK…when I preview and go to my Menu it is “not” there…so it is “hidden” ? Always scared it will publish it as a page. I’m publishing…crossed fingers. Thanx
If Show in navigation is unchecked in the page inspector, you should be good.
The developer in me just doesn’t like things feeling clunky, but I get what you’re saying.
1 Like
I’m good. All seems OK…have other issues but that is for another day. I’m stuck in Europe because of Covid restrictions so it’s Netflix time…going googly eyed working on this all day, ha. Thanx guys. Ciao for now.
Gentlemen,
Thanx for everything yesterday…but, fresh analyses of the SEO tells me that no sign of robots.txt is seen yet. I should be patient? I assume it takes a bit of time…or?
Thanx in advance,
RM
double check for me that in the page inspector,
Filenameisrobots.txtShow in navigationis uncheckedDraftis uncheckedFolderis/
oh, and to get rid of the theme, I think you want this
And I just tried this in my current project, and was successful
The trick is that you need to make sure that this is not checked…
It was off by default for me.
1 Like