When I create a new webpage in RW I am prompted to decide whether or not to Enable Robot Meta Tags.
As a subset to allowing the Robot Meta Tags I additionally have the option to “Index this Page” and/or “Follow Links”. I can either select one or the other, both or none and all.
What do these various options accomplish for me?
The robot meta tag, is a simple HTML meta tag placed in the header of a page.
Its purpose is to request that search engines follow the parameters on the meta tag.
The default behavior for search engines is index everything they find and to follow every link they locate. After all, it why we use them, to find things.
The options RapidWeaver provides are no meta tag at all, index or no index, and follow or no follow.
Index or no index, request the search engines included or not include this page in their database index. Follow or no follow requests the search engines to either follow(crawl) or not follow all the links they find on the page.
So you can have a page that you don’t want to show up in SERP’s (noindex), but still want all the links on the page to be crawled(follow).
You can have a page that you want to show up on results pages(index), but don’t want the search engines to follow the links(nofollow) it finds on the page.
Or you can have a page that you tell the search engines to do what they were going to do anyway and index the page and follow the links on the page they find.
Year’s back some idiot SEO self proclaimed expert Declared that you would get better rankings if you placed the meta tag on each page telling the crawlers to do their job and index the page and follow the links, even though it is the default.
So even though all the search engines (Bing, Google, Yahoo!, etc) have told everyone you don’t need these tags unless you don’t want the default behavior (index and follow), a lot of people still put them on.
There’s also other parameters (RapidWeaver doesn’t offer) and a robots.txt file that can be used to request non-default behavior at the site level. There are SEO self-proclaimed experts that’ll tell you that you need one of those as well, don’t buy it.
Thanks for that really thorough response Doug. You really explained this well.
For purposes of simplicity I think I will close my eyes and just enable the Robot Meta Tags. The concept of crawlers going across the web is a little creepy. Probably something I don’t really want to know about.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.