Implementing robots.txt for my existing site within RW 8.7

forbesrodney · December 4, 2020, 7:24pm

Good to know.

Vlask0vec2020 · December 4, 2020, 9:22pm

Oops…Thanx Rodney. It was wrong. Devil in the details and all that. Draft was unchecked and so was show in navigation -good, file was not “robots.txt.”, folder was “code” not “/” . We live in hope, ha. Just in case here is the screen capture.
Will undertake analysis soon.
R

Vlask0vec2020 · December 4, 2020, 9:38pm

Yes it was off by default for me as well. Oh boy here we go…

Panans · December 5, 2020, 11:50am

There is an add-on that does the job for you: Rapidbot by 4ground (https://4gnd.com/store/index.php?product/page/39/RapidBot+3). Very easy to allow or disallow certain search engines. Just add a Rapidbot page to your project.

Hans