Exclude Web Pages from Search Engines Using Meta Robots Tags

You can use Meta robots tags to prevent a search engine spider from crawling specific pages in your Web site. You can include a special tag in the HTML code of the page to tell robots not to index that page, or not to follow the links on that page. You would place this tag after the other Meta tags, which are part of the HTML code located in the head section of a Web page.

Using Meta robots tags is less efficient than using a site-wide robots text file for two reasons. First, robots sometimes ignore Meta robots tags, and second, these tags slow down the robots reading your pages, which may decrease the number of pages they’re willing to crawl. Also, this method can give your Webmaster headaches because the tags have to be maintained on the individual pages, rather than in a central file.

The Meta robots tag below tells the search engine robot not to index the page and not to follow any of the links on the page:


You could use this tag to tell the robot to read and index the page’s content, but not to follow any of the links:


The next tag instructs the robot to ignore the page’s content, but follow the links: