Do you know you can tell the search engine robots to whether your pages are indexed or not, enable the web site cache, isuues on Open Directory Project, and tell other things to search engines by just changing your meta tag.
This is how to do it.
In JUNE 2008 the revised Robots Exclusion Protocol added NOARCHIVE, NOODP and NOSNIPPET to the list of supported values supported by Google and MSN Live Search, and Yahoo added NOYDIR.
Why do you need to add meta tags to support as search engines ?
Because of it is better to direct (tell) the search engines, for what you need.
The default values are now assumed to be INDEX, FOLLOW, ARCHIVE, ODP, SNIPPET and YDIR.
There is no actual need to include these,
unless someone on the web team needs reminding.
So let's begin.
I think you know what is the meta tag.
It is like as follows.
<title>Should Not Be Indexed (Meta Robots noindex nofollow)
<META name="robots" content="NOINDEX,NOFOLLOW" />
above meta tag tells the search engines to "do not index our pages on your search engine"
Below is the list of options you can follow.
- Robots Information
- Introduction to web crawling robots for search indexing and other purposes
- Robot Exclusion Protocol (REP)
- Information on the original protocol and the June 2008 search engines extensions
- Elements of Robots.txt
- Robots.txt Details
- Practical notes on implementing robots.txt
- META Robots Tag Page
- Describes the META Robots tag contents and implications for search indexing robots.
- Indexing Robot Checklist
- A list of important items for those creating robots for search indexing.
- List of Robot Source Code
- Links to free and commercial source code for robot indexing spiders
- List of Robot Development Consultants