Using Meta Robots

Using noindex,nofollow Meta Robots

HTTP Collection crawler can be controlled using Meta robots tags.

• Rules for crawling based on meta robots would be considered after robots.txt and path settings specified in HTTP collections.
• Using meta robots
  • User can avoid indexing a page, but crawl the same.
  • User can index a page, but avoid crawling the same.
  • User can avoid indexing as well as crawling the page.
  • User can index and crawl the page.

Meta Robots

These meta tags which direct the indexing and crawling of a page has to be specified in the header section of HTML code.

noindex, follow

• To avoid indexing a page but allow crawling the following meta tag has to be specified in the page that is crawled:
  <meta name="robots" content="noindex, follow">

index, nofollow

• To avoid crawling but allow indexing the following meta tag has to be specified:
  <meta name="robots" content="index, nofollow">

noindex, nofollow

• To avoid both indexing and crawling the following meta tag has to be provided:
  <meta name="robots" content="noindex, nofollow">

index, follow

• To index as well as crawl either you can avoid giving the meta robots or specify the same as shown:
  <meta name="robots" content="index, follow">

Using Content Exclusion Meta Tags

In HTTP collections, it might be required to exclude content from sections of an HTML page from being indexed, such as headers, footers, and navigation.

• Noindex tags
• Stopindex, Startindex tags
• Googleon, Googleoff tags

Content Exclusion Metatags Supported

• noindex tags
  <noindex> Content to Exclude</noindex>
• stopindex, startindex tags
  <!--stopindex-->Content to Exclude <!--startindex-->
• googleon,googleoff tags
  <!--googleoff: all-->Content to Exclude<!--googleon: all-->

Rules for noindex,stopindex,google on-off tags

• Body content that is enclosed with stopindex/startindex tags or noindex tags or googleon/googleoff tags will not be included in the index
• These tags are not applicable in the head section or meta tags.
• These tags should not be nested.
• stopindex should be followed by startindex tags, noindex start tag should be followed by noindex end tag and googleoff tag should be followed by googleon tag.
• Please check the standards for these tags in Wikipedia or Google.