Request Recrawl, Modify Robot.txt

alixcalon · Member · Mar 21, 2022 at 3:54 am
Copy link

Add topic to favorites
Hi! I need help reindexing my site on Google. The last crawl was in January so the title and subtitle are outdated. Google said my recrawl request was blocked by a Robot.txt file. How can I modify this to request a recrawl? What are my other options to get my title and subtitle updated on Google? Thanks!

WP.com: Yes
Correct account: Yes

The blog I need help with is: (visible only to logged in users)
amiteshg93 · Member · Mar 21, 2022 at 7:10 am
Copy link
Hi there,

For re-indexing related queries, please reach out to the related forum.

This forum is only for sites hosted or issues related to the WordPress.com platform.
amiteshg93 · Member · Mar 21, 2022 at 7:44 am
Copy link
You can also refer to the below link for more support:

https://developers.google.com/search/docs/advanced/crawling/ask-google-to-recrawl
staff-ozmodiar · Staff · Mar 21, 2022 at 2:47 pm
Copy link
Generally, this is out of our hands here at WordPress.com.

The recrawl link above is likely a good place to start.
alixcalon · Member · Mar 21, 2022 at 4:27 pm
Copy link
Thanks all! I’ve requested the recrawl but got the “Blocked by Robot.txt” error from Google. Is there a way to disable or modify the Robot.txt file on my WordPress site?
supernovia · Staff · Mar 21, 2022 at 10:15 pm
Copy link
Here’s what’s in your robots.txt file:
https://alixcalon.wordpress.com/robots.txt

It links to your site map to index your content, then blocks any robots from trying to index anything related to core WordPress files.

Is there any chance the message you’re getting mentions one of those disallowed directories or files?
supernovia · Staff · Mar 21, 2022 at 10:25 pm
Copy link
@alixcalon actually, I took another look and it seems you changed your privacy settings to discourage search engines, and you made the site private at some point, too.

That will indeed change your robots.txt file to disallow Google and others. If you didn’t make those changes, let us know.

You’ve changed it back to public recently, though, so it should start working again soon. I’d just have Google check again tomorrow once it’s had a minute for cache to clear.

Cheers!
alixcalon · Member · Mar 21, 2022 at 10:52 pm
Copy link
Thanks so much, @supernova! I set my site to private thinking it would at least remove the old content from Google (which it didn’t). I updated my site title and tag, and even my homepage thinking it would initiate a refresh on Google (which it also didn’t do). My hope is that it refreshes on its own soon so I don’t have to worry about requesting a manual recrawl because I don’t know how to modify the Robot.txt file from the online version of wordpress or without a plug-in. I will follow up in a few days/when it refreshes.
staff-blorbo · Staff · Mar 21, 2022 at 11:47 pm
Copy link
Google doesn’t operate in real-time.

It can take days or even weeks for them to update their index, but you can track the progress by verifying your site in their search console: https://wordpress.com/support/site-verification-services/#google-search-console

No items found.