robots.txt file is stopping google from accessing my blog – how do I delete it?
-
This file is on my wordpress blog: http://holfordwatch.info/robots.txt. It’s stopping google from indexing the blog, and I presume other search engines too. Any idea how I get rid of, or change, this file.
My privacy options are set so that the blog should be indexable by search engines.
-
-
sorry – the file’s at http://holfordwatch.info/robots.txt
The full stop I put in broke the link. One of those days…
-
-
-
thanks barry – google doesn’t seem to have realised there’s a new robots.txt file yet :( but when it does it should work OK?
-
he-he…
PS
everything will be ok, john. don’t worry. next time when bot visit your blog it’ll crawl whole site and reindex it. -
I receive this same file-type name and have this message at my google account –
User-agent: IRLbot
Crawl-delay: 3600User-agent: *
Disallow:is that not normal? I didn’t ever set up the – http://todayyesterdayandtomorrow.wordpress.com/robots.txt
I’ve noticed some “not found'” responses in response to some of my posts in the google account when viewing the stats –
Thanks much –
-
Google will read the changes soon and it should be fine then
Yes, that is the default robots.txt file on all WordPress.com blogs. Since I don’t have access to your Google webmaster account (which is I assume what you are talking about), I can’t really comment on the “not found” messages.
-
thanks all for the help – very much appreciated. I was tearing my hair out trying to figure out why the robots.txt file was there…
-
Mine’s got a strange
disallow: /next/I wonder why…(???)
Why is there anything behind ‘disallow:’???
Anyone? -
It appears that the directory /next/ on our WP.com sites contains whatever is necessary for the Next button in the bar at the top of your screen when you are surfing wp.com. It shouldn’t be crawled and therefore is blocked.
Just for general robots.txt info
http://www.robotstxt.org/wc/exclusion-admin.html -
OK, I’m very new to all this, so this will probably be a no-brainer for you guys. My file http://eppe.wordpress.com/robots.txt says:
User-agent: IRLbot
Crawl-delay: 3600
User-agent: *
Disallow: /next/
User-agent: *
Disallow:Since my site is invisible to google even though its been up for more than 6 weeks, I thought this may be the source of the problem… Anyone? Oh, and thanks in advance for helping a poor newbie out!
EDIT: Ok, should have had a look at some documentation BEFORE posting… If I understood correctly, my robot.txt is fine, not disallowing anything. Right?
-
I am not staff and am only making an educated answer based on past experience. From what it says in the robots.txt file, the “next” directory is blocked to all Bots. The IRLbot apparently crawls sites too frequently, so it’s been told that it can’t crawl more than once every 3600 seconds.
There’s really not much we can do in changing our robots.txt file other than setting up our preferences to allow our blogs to be searched by search engines, etc. under the Options/Privacy tab in our Dashboard. Make sure yours is set to allow it. All the rest is under the hood and not accessible to users.
Forum volunteers have provided some much needed information on how to promote your blog, including submitting to search engines. Check out these threads:
https://en.forums.wordpress.com/topic.php?id=15804&page&replies=4
https://en.forums.wordpress.com/topic.php?id=16349&page&replies=5 -
-
It’s not a matter of trusting anyone. You post asking a question, we do our best to answer. We’re volunteers; we don’t claim to know everything but most of us have been using WP for a while and like to help newer users. And generally we don’t post in a thread unless we have a good idea of what we’re talking about. If you’re not sure about something we say, wait to hear back from staff instead. That’s your choice.
- The topic ‘robots.txt file is stopping google from accessing my blog – how do I delete it?’ is closed to new replies.
WordPress.com Forums 