<title>Should Not Be Indexed (Meta Robots noindex nofollow)
<META name="robots" content="NOINDEX,NOFOLLOW" />
above meta tag tells the search engines to "do not index our pages on your search engine"
Below is the list of options you can follow.
- Robots Information
- Introduction to web crawling robots for search indexing and other purposes
- Robot Exclusion Protocol (REP)
- Information on the original protocol and the June 2008 search engines extensions
- Elements of Robots.txt
- Robots.txt Details
- Practical notes on implementing robots.txt
- META Robots Tag Page
- Describes the META Robots tag contents and implications for search indexing robots.
- Indexing Robot Checklist
- A list of important items for those creating robots for search indexing.
- List of Robot Source Code
- Links to free and commercial source code for robot indexing spiders
- List of Robot Development Consultants