Robots Meta Tag

The robots meta tag is used to help web authors control the access of web robots to specific pages. This is a quick way to exclude pages from search engines even when the robots.txt file says that that directory is accessible.

There are four possible content values for the robots meta tag:

• index — allow this page to be indexed by the search engine
• noindex — do not allow this page to be indexed by the search engine
• follow — allow search engine spiders to traverse the links on this page
• nofollow — do not allow search engine spiders to traverse the links on this page
• noarchive — do not allow search engines to store a cached copy of this page
• nosnippet — do not allow search engines to display a description of the page in the search results
• noodp — do not allow search engines to use the Open Directory Project (ODP) description of the page as the description in the search results
• noydir — do not allow search engines to use the Yahoo! Directory description of the page as the description in the search results

You can include multiple values by separating them with commas, such as:

"index,nofollow"

The Robots meta tag is a named meta tag.

<meta name="robots" content="noindex,nofollow">

Placing a nofollow attribute on a page will prevent search engines from following the links on this page, but if they are linked elsewhere (such as on an external site), the search engine will still be able to find the page. If you need a page blocked from search engines, then you should modify your robots.txt file.

The robots meta tag is most effective when it’s used on every page of your website.

You can also modify this meta tag to name the exact search engine you would like to block. Simply change the name="" value to one of the following:

• name="teoma" — to block Ask.com
• name="msnbot" — to block Bing
• name="googlebot" — to block Google
• name="slurp" — to block Yahoo!