blocked by robots error

  • carmelitequotes · Member ·

    I got a message Google Search Console telling me that there’s a ‘New reason preventing your pages from being indexed. Search Console has identified that some pages on your site are not being indexed due to the following new reason: Blocked by robots.txt’

    I give up, what is this and what can I do to restore the post or page to be fully indexed again?

    WP.com: Yes
    Jetpack: No
    Correct account: Yes

    The blog I need help with is: (visible only to logged in users)

  • umair4342fa0bcb · Member ·

    I understand your frustration. Getting the “Blocked by robots.txt” message in Google Search Console can be confusing and disheartening. But don’t give up!
    Understanding the error:mm
    robots.txt: This is a file on your website that tells search engine robots which pages they can and cannot crawl and index. If Googlebot tries to access a page but is blocked by your robots.txt file, it won’t be indexed and won’t appear in search results.
    Finding the blocked page:
    Google Search Console: Go to the “Coverage” section in Google Search Console. Check the “Excluded” tab and look for the specific URL that’s blocked by robots.txt. This will give you a starting point for investigating the issue.
    Disallow: /your-blocked-page/
    Remove the blocking rule: If you identified the blocking rule, simply remove it from your robots.txt file. Remember to save the changes and upload the updated file to your website.
    Validate your robots.txt: Use the robots.txt Tester tool in Google Search Console to validate your robots.txt file and ensure there are no syntax errors or other issues that might be blocking your pages.

    Wait for Googlebot to recrawl: After fixing the issue, it might take some time for Googlebot to recrawl your pages and update the indexing status. Be patient and check back in Google Search Console after a few days.

  • staartmees · Member ·

    @umair4342fa0bcb you forgot 1 important thing: you don’t have access to robots.txt on the wordpress.com platform. Your answer is only valid for a self-hosted site.

  • umair4342fa0bcb · Member ·

    You’re correct, and I appreciate the clarification. On the WordPress.com platform, users don’t have direct access to the robots.txt file, as the management of the robots.txt file is handled by WordPress.com itself. This limitation is due to the hosted nature of WordPress.com, where certain server-level configurations are managed by the platform.
    If you are using WordPress.com and have specific requirements or concerns regarding search engine indexing, you may want to explore available options within the WordPress.com settings or contact WordPress.com support for guidance. They may offer features or settings that allow you to control search engine visibility and indexing for your site.

  • The topic ‘blocked by robots error’ is closed to new replies.