what is the Robots.txt?

[Danielnash]

Hi everybody,


tell me about Robots.txt file?


Thanks

[Antra]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

[mhasan]

robot.txt file is created in order to prevent the google cralwer crawling the file or page which you dont want to index by google

[jason830]

it is a file that tell spider which page they can crawl,which page they could not crawl.

[seostellar]

Hi dear,

robot.txt file is used to allow or disallow user agents(search engine spider) like googlebot, msnbot for crawling a particular page in a website.


seostellar

[lopezobrador]

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

http://en.wikipedia.org/wiki/Robots_exclusion_standard

Lopez Obrador