My website is blocked by robots.txt

ludivine93260 · Member · May 13, 2019 at 5:39 pm
Copy link

Add topic to favorites
Hi!

When I go to “www.bitumeetmimosa.blog/robots.txt”, it’s mentioned:

User-agent: *
Disallow: /

# This file was generated on Wed, 08 May 2019 10:15:15 +0000

AND in Google Search Console they said they can’t index my blog because of robots.

BUT, I choosed “public” when I created the website.

Can you help me please?

Thanks, Ludivine

The blog I need help with is: (visible only to logged in users)
supernovia · Staff · May 13, 2019 at 7:21 pm
Copy link
Hi Ludivine,

User-agent: *
Disallow: /

Actually it says this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /wp-signup.php
Disallow: /press-this.php
Disallow: /remote-login.php
Disallow: /activate/
Disallow: /cgi-bin/
Disallow: /mshots/v1/
Disallow: /next/
Disallow: /public.api/

So it’s allowing Google to index everything except those directories — which really wouldn’t help your ranking anyway.

Are you seeing disallow / with only a slash?
ludivine93260 · Member · May 13, 2019 at 7:37 pm
Copy link
Yes only with one slash!
supernovia · Staff · May 13, 2019 at 8:25 pm
Copy link
I’m not seeing that. Can you please get a screenshot and upload it to your media library, then update us here? Be sure to include your browser address bar as well.

This is for https://bitumeetmimosa.blog/robots.txt
ludivine93260 · Member · May 13, 2019 at 10:03 pm
Copy link
Hi, it’s in my media library, how can i send it to you?

Best regards,

Ludivine
kokkieh · Staff · May 14, 2019 at 5:04 am
Copy link
Hi there,

I saw the same as you, so I set your site to private and back to public, and now the file appears as it should on my end. Can you please clear your browser cache and then check the file again on your end?
ludivine93260 · Member · May 14, 2019 at 5:40 am
Copy link
Yes it’s ok!

Thank you so much!
hostnetindiacom · Member · May 14, 2019 at 6:41 am
Copy link
update your robot txt file using this code

User Agent:*
Disallow:
kokkieh · Staff · May 14, 2019 at 6:47 am
Copy link
@ludivine93260,

Thanks for confirming :)

Happy blogging!

@hostnetindiacom, it’s not possible to edit the robots.txt file directly on WordPress.com, and the OP has already indicated that what I did above fixed the issue.

Please review our volunteer guidelines if you haven’t done so already:

WordPress.com Volunteers

No items found.