How to edit robots.txt in wordpress?

Question

I need help locating my robots.txt file in FTP for my blog site lifetimestocks.com, I believe one was generated by a plug in and I google won't crawl my site so it needs to be fixed. the robots file is at lifetimestocks.com/robots.txt

Answer 1

You should be able to see the robots.txt as a rule - but I have a plugin that "hides" mine which is ridiculous. I have to turn the plugin off in order to unlock the robots.txt file so I suggest you do that.



If it's not a security plugin doing this, you might want to rethink using the plugin.

Answer 2

Hi Jake,



Since your robots.txt file is reached via lifetimestocks.com/robots.txt you should be able to see that file through FTP in your document root directory (usually public_html). For more information on working with and editing your robots.txt file:



http://www.webhostinghub.com/support/website/how-to/using-robotstxt



Hope this helps!

Answer 3

You just need all in one SEO plugin to sort it.



However, there is nothing wrong with your current robots.txt. It's not blocking Google...

Answer 4

I think you are getting confused,



robots.txt only tells search engine bots which pages they can and cannot visit, you dont submit it to google or any other search engine



here is some of the option you should have in your robots.txt



User-agent: *



Disallow: /wp-content/

Disallow: /wp-icludes/

Disallow: /trackback/

Disallow: /wp-admin/

Disallow: /archives/

Disallow: /category/

Disallow: /tag/*

Disallow: /tag/

Disallow: /wp-*

Disallow: /login/

Disallow: /*.js$

Disallow: /*.inc$

Disallow: /*.css$

Disallow: /*.php$



User-agent: All

Allow: /



User-agent: ia_archiver

Disallow: /



User-agent: duggmirror

Disallow: /



you may be referring to a sitemap, a sitemap needs to be submitted to google and other search engines its really called a sitemap.xml as its an xml file



you can generate one with this plugin http://wordpress.org/extend/plugins/google-sitemap-generator/