How do I create a robots.txt file?

Question

Help!! Every time I submit my site to http://www.scrubtheweb.com/ I get the following message:

This site does NOT provide the robots.txt file or the robots.txt file is not constructed properly. It is recommended all sites provide this file.

Is this another way of adding tags?

Cheers

Answer 1

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

You need to create text file in the top-level directory of your web server.

Robot looks for the "example.com/robots.txt" file

robots.txt

To exclude all robots from the entire server, add next rows:

User-agent: *

Disallow: /

To allow all robots complete access:

User-agent: *

Disallow:

Answer 2

To create a simple robots.tht you can use this service here: http://bestseoadmin.com/tool.php?id=generate-robots-txt

(I use this service all the time, conveniently and quickly)

Answer 3

http://en.wikipedia.org/wiki/Robots_exclusion_standard