~~~ Please read before posting. ~~~

Important: We need your Club Number at a minimum, and as many details as possible.
For further info please read This page before posting.

Index Coverage Issue

  • mmaurer
  • mmaurer's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 6
  • Thanks: 2

Index Coverage Issue

7 years 3 weeks ago
#71193
Hello!

I just received an Index Coverage Issue alert from Google for our club website.

Club address is crossroadstm.ca/

This is the current robots.txt that it's showing.
User-agent: Googlebot
Crawl-delay: 10
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/ckeditor_smileys/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: bingbot
Crawl-delay: 20
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: MSNBot
Crawl-delay: 20
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: Slurp
Crawl-delay: 10
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: MJ12bot
Disallow: /

User-agent: pmoz.info
Disallow: /

User-agent: Yak
Disallow: /

User-agent: AlphaSeoBot
Disallow: /

User-agent: AlphaSeoBot-SA
Disallow: /

User-agent: omgili
Disallow: /

User-agent: linkdexbot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: coccocbot-web
Disallow: /

User-agent: Pinterest
Disallow: /

User-agent: Uptimebot
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: stremorbot
Disallow: /

User-agent: YandexBot
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: Sosospider+
Disallow: /

User-agent: wonderbot/JS 1.0
Disallow: /

User-agent: HubSpot
Disallow: /

User-agent: SurdotlyBot/1.0
Disallow: /

User-agent: spbot/5.0.3
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: moget
User-agent: ichiro
Disallow: /

User-agent: NaverBot
User-agent: Yeti
Disallow: /

User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: linkdexbot/2.0
Disallow: /

User-agent: BUbiNG
Disallow: /
Crossroads Toastmasters
Club #1396 | District 64
CrossroadsTM.ca
The topic has been locked.
  • SteveTheTechie
  • SteveTheTechie's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 11526
  • Thanks: 3050

Re: Index Coverage Issue

7 years 3 weeks ago - 7 years 3 weeks ago
#71194
What is the actual issue reported?

We ultimately control the robots.txt file--individual clubs/districts do not get to change that. That is because every website is sharing the same single server, and we have to make sure that search spiders/bots do not drag server performance down to a stand-still. (We have actually had that happen in the past.)

Therefore, you are welcome to post indexing issues and suggestions for improvement, but ultimately, we must make the final decision on what we will allow to be indexed based on a broader perspective than just your website.
Last edit: 7 years 3 weeks ago by SteveTheTechie.
The topic has been locked.
  • mmaurer
  • mmaurer's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 6
  • Thanks: 2

Re: Index Coverage Issue

7 years 3 weeks ago
#71195
Steve,

It just gave the following description.
New issue found:
Indexed, though blocked by robots.txt

It might not be a problem. I tried indexing the site again and it didn't show any errors. I just got the email about an hour ago from Google and thought it might have been related to the website changes.

Not sure why else I would have been sent the email today.

Everything appears to be fine though.
Crossroads Toastmasters
Club #1396 | District 64
CrossroadsTM.ca
The topic has been locked.
  • SteveTheTechie
  • SteveTheTechie's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 11526
  • Thanks: 3050

Re: Index Coverage Issue

7 years 3 weeks ago
#71196
You are probably correct about it being related to the changes...
I went ahead and updated the robots.txt file where it makes sense. Thanks!
The topic has been locked.
Moderators: Brian, Jane Atkinson, Pam, rhtaylor3, marc33, NotLiable, jgavin, Lcala305, peterb323
Time to create page: 0.133 seconds

Copyright © 2025 FreeToastHost 3 Support. All Rights Reserved.