robots.txt.file

ashleydmurray · Member · Jan 3, 2010 at 6:54 pm
Copy link

Add topic to favorites
google webmaster tools is telling me my site is restricted from access due to a robots.txt.file. this might be a stupid question but how do i fix this? is there something i can change on my settings that i’m not aware of? or is there another reason why it would be restricted?

The blog I need help with is: (visible only to logged in users)
monsoonpk · Member · Feb 7, 2010 at 3:20 pm
Copy link
I have the same problem. I think we should learn how to change tag in robots.txt to allow.
raincoaster · Member · Feb 7, 2010 at 4:18 pm
Copy link
Have you checked your Settings->Privacy to ensure you’re allowing search engines?
mark · Member · Feb 7, 2010 at 4:52 pm
Copy link
http://ashleydoniellemurray.wordpress.com/robots.txt
AND
http://iamasocialanimal.wordpress.com/robots.txt

User-agent: IRLbot
Crawl-delay: 3600

User-agent: *
Disallow: /next/

# har har
User-agent: *
Disallow: /activate/

User-agent: *
Disallow: /signup/

User-agent: *
Disallow: /related-tags.php

User-agent: *
Disallow:

See that last line?

http://www.robotstxt.org/robotstxt.html

To allow all robots complete access
User-agent: *
Disallow:

Google is wrong.
mark · Member · Feb 7, 2010 at 4:55 pm
Copy link
The above will ONLY appear is you choose the top of the 3 options in Settings > Privacy.

(and Google is still wrong).
monsoonpk · Member · Feb 7, 2010 at 9:40 pm
Copy link
Thanks @mark and @raincoaster
I have generated new robots.txt through google webmaster. I want to upload that file to replace the old one.
File code is as follows:

User-agent: *
Allow: /

Please help me to get rid of this problem. I don’t understand what’s going on, since in setting>privacy I have allowed search engines.

Thanks.
timethief · Member · Feb 7, 2010 at 9:45 pm
Copy link
@monsoonpk
For interest sake please read this report: http://ismyblogworking.com/iamasocialanimal.wordpress.com

The important stuff:

* Your web server [76.74.254.123] is working fine.
* Your RSS feed is available.
* Your robots.txt file looks ok.
monsoonpk · Member · Feb 7, 2010 at 10:08 pm
Copy link
@timethief
Thanks for quick response.
My aim is very simple i.e. to index my blog with search engines including google.

When I use option “Fetch a googlebot” on google webmaster the result is “Denied by robots.txt” please see the result of tries below:

URL Status Date submitted
http://iamasocialanimal.wordpress.com/ Denied by robots.txt 2/7/10, 02:00 PM

http://iamasocialanimal.wordpress.com/ Denied by robots.txt 2/6/10, 03:46 PM

Google demands change in robots.txt from “Disallow to Allow”.
I am aware that google indexing is not wordpress.com’s responsibility, I appreciate you help in this regard.
Thanks.
tellyworth · Staff · Feb 7, 2010 at 11:43 pm
Copy link
@monsoonpk your robots.txt file already includes an equivalent line to what you are requesting (the last 2 lines as Mark pointed out). The “denied by robots.txt” message is incorrect.

Here’s how you can test for yourself:

1. Go to Google Webmaster Tools.

2. Choose the Site Configuration / Crawler Access in the menu.

3. Leave the “text of … robots.txt” box intact. It will contain the robots.txt file contents as pasted by Mark above.

4. In the “URLs” text box, paste some URLs from your site like these:

http://iamasocialanimal.wordpress.com/
http://iamasocialanimal.wordpress.com/about/
http://iamasocialanimal.wordpress.com/2010/02/01/roskilde-fjord-is-frozen/
http://iamasocialanimal.wordpress.com/tag/english/

5. Click the Test button, then scroll down to the results.

In every case you’ll see the result: Allowed by line 20: Disallow:

That means your robots.txt file is allowing Googlebot to crawl.

The earlier “denied by robots.txt” message probably occurred because you had your blog set to block search engines at that point.
tellyworth · Staff · Feb 8, 2010 at 12:13 am
Copy link
ismyblogworking.com now does the same test automatically:

http://ismyblogworking.com/iamasocialanimal.wordpress.com

Your robots.txt file is valid and allows search engines to index your content.

It will warn you if robots.txt is preventing Googlebot or MSNBot (Bing) from reaching your content.
dreamsburnred · Member · Feb 8, 2010 at 3:09 am
Copy link
It’s working just fine for my 3 blogs w/ google webmaster.

It can be up to 24 hours before Google wakes up to a new robots.txt if you had privacy enabled.
monsoonpk · Member · Feb 8, 2010 at 2:00 pm
Copy link
Hi all guys and @tellyworth and @timethief,

My site is indexed by google and can come up in search.
Thanks for your support.. I highly appreciate.
Thanks..

No items found.