## **Using noindex,nofollow Meta Robots**
HTTP Collection crawler can be controlled using Meta robots tags.
Rules for crawling based on meta robots would be considered after robots.txt and path settings specified in HTTP collections.
Using meta robots
User can avoid indexing a page, but crawl the same.
User can index a page, but avoid crawling the same.
User can avoid indexing as well as crawling the page.
User can index and crawl the page.
### Meta Robots
These meta tags which direct the indexing and crawling of a page has to be specified in the header section of HTML code.
**noindex, follow**
To avoid indexing a page but allow crawling the following meta tag has to be specified in the page that is crawled: `
<meta name="robots" content="noindex, follow">
`
**index, nofollow**
To avoid crawling but allow indexing the following meta tag has to be specified: `
<meta name="robots" content="index, nofollow">
`
**noindex, nofollow**
To avoid both indexing and crawling the following meta tag has to be provided: `
<meta name="robots" content="noindex, nofollow">
`
**index, follow**
To index as well as crawl either you can avoid giving the meta robots or specify the same as shown: `
<meta name="robots" content="index, follow">
`
## **Using Content Exclusion Meta Tags**
In HTTP collections, it might be required to exclude content from sections of an HTML page from being indexed, such as headers, footers, and navigation.
Noindex tags
Stopindex, Startindex tags
Googleon, Googleoff tags
### Content Exclusion Metatags Supported
noindex tags `
<noindex> Content to Exclude</noindex>
`stopindex, startindex tags `
<!--stopindex-->Content to Exclude <!--startindex-->
`googleon,googleoff tags `
<!--googleoff: all-->Content to Exclude<!--googleon: all-->
`
### Rules for noindex,stopindex,google on-off tags
Body content that is enclosed with stopindex/startindex tags or noindex tags or googleon/googleoff tags will not be included in the index
These tags are not applicable in the head section or meta tags.
These tags should not be nested.
stopindex should be followed by startindex tags, noindex start tag should be followed by noindex end tag and googleoff tag should be followed by googleon tag.
Please check the standards for these tags in Wikipedia or Google.