robots.txt

en WordPress.com Forums robots.txt

• 

  jonpalanis · Member · Nov 8, 2023 at 10:11 am

  • Copy link
  • Add topic to favorites

  Hi, for some reason the default robots.txt blocks the following bots, I don’t want to block any bot. Please don’t block bots unless I specifically asked you to.

  User-agent: GPTBot
  Disallow: /

  User-agent: CCBot
  Disallow: /

  User-agent: SentiBot
  Disallow: /

  User-agent: sentibot
  Disallow: /

  User-agent: Google-Extended
  Disallow: /

  User-agent: FacebookBot
  Disallow: /

  User-agent: omgili
  Disallow: /

  User-agent: omgilibot
  Disallow: /

  User-agent: Amazonbot
  Disallow: /

  User-agent: Bingbot
  Disallow: /

  WP.com: Yes
  Jetpack: No
  Correct account: Yes

  The blog I need help with is: (visible only to logged in users)

• 

  danielmewes7081 · Member · Dec 10, 2023 at 8:26 am

  • Copy link

  +1 . I’d also like to allow all bots to crawl my site. Especially bingbot. My page is amongai.com

• 

  justjennifer · Member · Dec 10, 2023 at 9:29 am

  • Copy link

  Hi there, this issue is being discussed in another forum thread at https://wordpress.com/forums/topic/blocking-bingbot-in-robots-txt-why/ Staff have replied there so you may wish to follow that discussion. Thanks.

• The topic ‘robots.txt’ is closed to new replies.