Hi everyone,
Could anyone advise how to add a robots.txt file to a RapidWeaver site? It’s not generating one automatically.
Your responses would be appreciated.
Best, Natasha
Why do you think you need one?
The default for all ligament search engines to index everything.
The only use for a robots.txt file is to ask search engines or a specific robot not to index part or all of a site.
Quite honestly Google has been know to accidentally index pages that the robots.txt file said not to.
If some SEO “tool” is telling you that you need one to help your ranking, then it’s not a good tool.
Add an HTML page you can set the page props to not use a theme. Then you can simple call the page name robots.txt
This is my actual setup on my Weaver’s Space project
I also have a bunch of videos that you can learn from if you are wanting to learn some more… https://www.youtube.com/c/WeaversSpace/search?query=robots
4 Likes
I checked through an SEO tool which automatically suggested that both a robot.txt and sitemap.xml are needed.
Thank you!
The SEO tool is totally incorrect about the robots.txt file. Google, Bing, Yahoo and most every other search engine out there have confirmed that they only use robots.txt for disallow (not to index) and they default to searching everything.
In fact both Google and Bing documentation shows that, and they both have clearly stated that disallowon the robots.txt file as well as the nofollow meta tag are only considered “suggestions” and they may index the page anyway.
Google makes it vary clear that the only sure way to not get indexed is to put what you want not index behind a password. They don’t try and hack your password protected area.
The sitemap.xml file does help Google, Bing and other bots scan your site. They use it as a starting point, so even pages that aren’t in the sitemap.xml will still get indexed as long as there’s a link to it. That’s why these bots are sometimes referred to as crawlers. They “crawl” every link they find.
My concern is that any SEO tool that tell you that a robots.txt file will help your ranking is full of crap.
As you can see from Joe’s robots.txt file, he’s only using it to disallow certain pages from being index. I bet the SEO tool is telling you to add “allows” and that will help your ranking.
Search engines will look for the sitemap.xml file in the root directory. The robots.txt file can tell a different or additional locations for site maps.
Btw: you can look at anybody’s robot.txt file. They are simple text files open to anyone.
Amazon’s https://www.amazon.com/robots.txt
Apple’s https://www.apple.com/robots.txt
3 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.