Welcome, Guest
Username: Password: Remember me
1. The "search..." box above searches the Docs & Forum Posts. The "Search" tab above just searches the Forum Posts. :side:
Please use these to search for your issue *before* creating a new message topic, as your issue may have been previously solved.
2. Please put your Club # and Club Web Address in your Forum Signature (best) OR in each post to get faster support from us.
Click here to edit your signature at the bottom of the Profile Information tab.
3. Our user and admin docs are available at: support.toastmastersclubs.org/doc "There's a doc for that!" ;)
4. There is an "Opt In" Feature for newly added members. The Opt In document explains the strikethrough member information. Click Here to View the Post
5. When posting a New Topic , please include all relevant details and be specific. When did your issue 1st occur? What operating system, browser, & browser version are you using? Did you refresh your browser cache? Are your cookies enabled? Lastly, a screen shot is often helpful.
6. Please abide by the Terms of Use . We are volunteers contributing our spare time. We are happy to assist you, so long as you are respectful and courteous.
7. We are always looking for new FreeToastHost Ambassadors to join our team and support fellow Toastmasters in their use of the FreeToastHost website system. If you are familiar with the system and have some interest, send a Send Us a Private Message.
  • Page:
  • 1

TOPIC:

Index Coverage Issue 6 years 11 months ago #71193

  • mmaurer
  • mmaurer's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 6
  • Thank you received: 2
Hello!

I just received an Index Coverage Issue alert from Google for our club website.

Club address is crossroadstm.ca/

This is the current robots.txt that it's showing.

User-agent: Googlebot
Crawl-delay: 10
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/ckeditor_smileys/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: bingbot
Crawl-delay: 20
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: MSNBot
Crawl-delay: 20
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: Slurp
Crawl-delay: 10
Disallow: /jquery/
Disallow: /fthadmin/
Disallow: /logfiles/
Disallow: /OLD_FILES/
Disallow: /text/
Disallow: /js/
Disallow: /json/
Disallow: /css/custom-theme3
Disallow: /css/custom-theme1
Disallow: /css/custom-theme2

User-agent: MJ12bot
Disallow: /

User-agent: pmoz.info
Disallow: /

User-agent: Yak
Disallow: /

User-agent: AlphaSeoBot
Disallow: /

User-agent: AlphaSeoBot-SA
Disallow: /

User-agent: omgili
Disallow: /

User-agent: linkdexbot
Disallow: /

User-agent: SemrushBot-SA
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: spbot
Disallow: /

User-agent: coccocbot-web
Disallow: /

User-agent: Pinterest
Disallow: /

User-agent: Uptimebot
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: stremorbot
Disallow: /

User-agent: YandexBot
Disallow: /

User-agent: Ezooms
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: Baiduspider
Disallow: /

User-agent: Sosospider
Disallow: /

User-agent: Sosospider+
Disallow: /

User-agent: wonderbot/JS 1.0
Disallow: /

User-agent: HubSpot
Disallow: /

User-agent: SurdotlyBot/1.0
Disallow: /

User-agent: spbot/5.0.3
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: moget
User-agent: ichiro
Disallow: /

User-agent: NaverBot
User-agent: Yeti
Disallow: /

User-agent: Baiduspider
User-agent: Baiduspider-video
User-agent: Baiduspider-image
Disallow: /

User-agent: sogou spider
Disallow: /

User-agent: YoudaoBot
Disallow: /

User-agent: linkdexbot/2.0
Disallow: /

User-agent: BUbiNG
Disallow: /

Crossroads Toastmasters
Club #1396 | District 64
CrossroadsTM.ca
The topic has been locked.

Index Coverage Issue 6 years 11 months ago #71194

  • SteveTheTechie
  • SteveTheTechie's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 11526
  • Thank you received: 3795
What is the actual issue reported?

We ultimately control the robots.txt file--individual clubs/districts do not get to change that. That is because every website is sharing the same single server, and we have to make sure that search spiders/bots do not drag server performance down to a stand-still. (We have actually had that happen in the past.)

Therefore, you are welcome to post indexing issues and suggestions for improvement, but ultimately, we must make the final decision on what we will allow to be indexed based on a broader perspective than just your website.
Last edit: by SteveTheTechie.
The topic has been locked.

Index Coverage Issue 6 years 11 months ago #71195

  • mmaurer
  • mmaurer's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 6
  • Thank you received: 2
Steve,

It just gave the following description.

New issue found:
Indexed, though blocked by robots.txt


It might not be a problem. I tried indexing the site again and it didn't show any errors. I just got the email about an hour ago from Google and thought it might have been related to the website changes.

Not sure why else I would have been sent the email today.

Everything appears to be fine though.
Crossroads Toastmasters
Club #1396 | District 64
CrossroadsTM.ca
The topic has been locked.

Index Coverage Issue 6 years 11 months ago #71196

  • SteveTheTechie
  • SteveTheTechie's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 11526
  • Thank you received: 3795
You are probably correct about it being related to the changes...
I went ahead and updated the robots.txt file where it makes sense. Thanks!
The topic has been locked.
  • Page:
  • 1
Moderators: Pamjliumarc33NotLiabledeedubbleyooNSBPhyllis Kirouac
Time to create page: 0.038 seconds
Powered by Kunena Forum