robot txt is blocking search
-
robot txt is blocking search
WP.com: Yes
Jetpack: No
Correct account: YesThe blog I need help with is: (visible only to logged in users)
-
Just wanted to add to my query above, that my site is a free wordpress site, for a charity, and I do not want to pay for plug ins or to upgrade the free site. It seems that wordpress put the block on automatically when I created the site. How do I change it please?
-
@whitemosshouse Can you check your site’s Privacy settings to ensure that they are set to Public and that “Discourage search engines” is unchecked. Save any changes you make.https://wordpress.com/support/privacy-settings/
Let us know if you need more help with this.
@bixapk your site is running the open source WordPress software hosted with Cloudflare. It also looks like you’re using the Yoast plugin for SEO. I’d check with their support directly about how to configure your robots.txt.
These forums assist sites hosted on WordPress.com, the hosting provider. To learn about the differences, check out this guide: https://wordpress.com/support/com-vs-org/#are-wordpress-org-and-wordpress-com-the-same-thing
-
Yes, I’ve done that .
This is the problem https://grasmerewi.wordpress.com/robots.txt
User-agent: * Disallow: / # This file was generated on Wed, 10 Apr 2024 16:18:49 +0000
-
Hello again @whitemosshouse I did look at your site’s robots.txt file before replying. :) Was your site previously set to Private and you’ve only recently changed it to Public?
Regardless, please wait for staff to look in on this thread and assist. I’ve flagged your thread for their attention. Thanks for your patience.
-
Thank you very much.
I’ve always had it set to public.
I’ve been trying to get more links in and lots of links out in the hopes that Google will find it anyway, but sadly nothing seems to be working.
If your team could unblock it for us Grasmere Women’s Institute ( In Grasmere, England, home to poet William Wordsworth) would be very grateful!
I do have other older blogs that are accessible and searchable, so this has been an unhappy surprise for me.
-
Do free WordPress.com sites automatically put a robots.txt file on that doesn’t allow search engine indexing to make people take out paid plans?
-
Are you familiar with the Internet Archive’s Wayback Machine? I thought it only saved our site webpages, but it turns out that it also saves our robots.txt.
Here is a screenshot of the robots.txt as it appeared on your site on 31 March 2021, which is also looks to be the first time that the IA crawled your site.
https://web.archive.org/web/20210331143720/https://grasmerewi.wordpress.com/robots.txt
My guess is that something is stuck and needs support help in shaking it loose. Thanks again for your patience while they get back to you. (Best to be patient. Each time we add to this forum thread, we push it down in the very long queue waiting for staff.)
-
Thanks for your patience! This is a weird bug that pops up sometimes and we’ve not been able to figure out the root cause.
If this happens in the future, setting the site to Private, then setting it back to Public usually shakes this bug out. I did that for your site @whitemosshouse and it’s showing the correct robots.txt information now.
I hope that helps!
-
Thank you ! I’ve requested re indexing on my google search console.
How long will it take to be indexed and can you help?
I wish I’d asked for your help sooner! It’s been a mystery why we weren’t appearing in searches https://grasmerewi.wordpress.com/ still not coming up, but fingers crossed for this all to change soon.
Thanks, Sue Dixon aka @whitemosshouse
-
How long will it take to be indexed and can you help?
Unfortunately, there isn’t anything we can do – it’s up to Google now. If you’ve requested an indexing already that’s really all you can do at this point and wait for them to process the request.
Let us know if you have any questions or need help with anything else.
- The topic ‘robot txt is blocking search’ is closed to new replies.
WordPress.com Forums 