Home » ON PAGE SEO » Robots Meta Tag

Robots Meta Tag

Follow Us

Robots Meta TagA number of SEO beginners are unfamiliar with a tag that says <meta name=”robots” content=”index”>. Asking themselves, “what is its function?” and “why is it in HTML content?” to state a few.  Here, we will explain what  this  Tag means and what it does.

 

What is Robots Meta Tag?

Robots <META> tag is a tag that instructs a search engine robot or user agent a specific directive on how your website or a specific page should be crawled. It’s a piece of HTML code that is located in the <head> section of your webpage.

 <html>
    <head>
        <meata name=”robots” content=”noindex” />
            (….)
    </head>
        <body>(…)</body>
</html>

 

 

One of the functions of the robots <META> tag is it allows you to use unprocessed, page-only style to manage the way individual pages are indexed and presented to online searchers.

Telling user agents what links to follow and what links to stop is another function of the meta robots tag. Having a number of outgoing links from your website, this will result to losing some link juice that would result to a lower page rank. To avoid losing any of your link juice, we tell search engine crawlers not to follow the links that are coming out of your site through Meta robots tag.

 

Why should we use Robots Meta Tag?

 

By default, when a new website is launched on the World Wide Web and the search engine crawlers take notice of this, they automatically crawl and index it.  With no Meta robot tag found on your HTML content, it will crawl all of the websites pages — which is not a bad thing at all.

 

Nevertheless, when a website has been on the World Wide Web and already has tons of outgoing links and the website has no Robots Meta Tag, that is another story.

 

Due to default settings of how a crawler crawls this website, all outgoing links will receive link juice from your website and that is a bad thing.

 

How to write a Robot Meta Tags?

 

Like most of <META> tag, it should always be placed on the <HEAD> section of your HTML page.

<HEAD>
 	<title>Crawlers must not index and follow</title>
 	<META name="robots" content="NOINDEX,NOFOLLOW" />
</HEAD>
<HEAD>
 	<title>Frequently changing content, search result might be misleading</title>
 	<META name="googlebot" content="NOARCHIVE, NOODP, NOSNIPPET" />
	<META name="slurp" content="NOARCHIVE, NOYDIR, NOSNIPPET" />
</HEAD>

 

When using the robots<META> tag, you must take note of the following:

  • Malware Robots can scan the web for security vulnerabilities that can ignore your <META> tag. An email address harvester used by spammers will ignore it.
  •  When a robot finds  similar links on other pages or other websites without a NOFOLLOW directive, it will still arrive at your undesired page. Meaning, the NOFOLLOW only applies to links on that page.

Note: “NOFOLLOW” and rel=”nofollow” are two different things.

 

Below are a list of different Directives .

 

ROBOTS META TAG DIRECTIVES

image from. https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

When do we use these Robot <Meta> Tag Directives?

 

Now that you have a list of different directives. The question now is. When and where do we use these directives?

META Entry

Where to Put

<META name=”ROBOTS” content=”NOINDEX”>

 

Use this directive entry for pages with many links on them, but no useful data,

Sitemap Page.

<META name=”ROBOTS” content=”NOFOLLOW, INDEX “>

 

Use this directive for pages, which have useful, content but outdated or problematic links.

Contact Us Page.

<META name=”ROBOTS” content=”NOINDEX, NOFOLLOW”>

 For Pages like Blogs, Forums and Guestbook’s

<META name=”ROBOTS” content=”NOARCHIVE”>

Useful if the content changes frequently: headlines, auctions, etc. The search engine still archives the information, but won’t show it in the results.

<META name=”ROBOTS” content=”NOODP”>

Telling search engines to use the page title tag, and match term in context, or META Description tag content instead of the Open Directory Project or ODP content, which may be misleading or outdated.

<META name=”ROBOTS” content=”NOYDIR”>

(Yahoo Slurp robot only)

Same as above, only for the Yahoo directory, and the other search indexers will ignore it.

<META name=”ROBOTS” content=”NOSNIPPET>

Tells the search engines to use the title only, and to suppress the “cache” link. Might be useful if the site has special plus box listings in search results, but otherwise, not so much.

 

 

serena

More Posts

Follow Me:
TwitterFacebookPinterestGoogle PlusYouTube