~~~~~~~~~~~~ IMPORTANT INFORMATION -- Please read! ~~~~~~~~~~~~

1. The "search..." box above searches the Docs & Forum Posts. The "Search" tab above just searches the Forum Posts. :side:
Please use these to search for your issue *before* creating a new message topic, as your issue may have been previously solved.
2. Please put your Club # and Club Web Address in your Forum Signature (best) OR in each post to get faster support from us.
Click here to edit your signature at the bottom of the Profile Information tab.
3. Our user and admin docs are available at: support.toastmastersclubs.org/doc "There's a doc for that!" ;)
4. There is an "Opt In" Feature for newly added members. The Opt In document explains the strikethrough member information. Click Here to View the Post
5. When posting a New Topic , please include all relevant details and be specific. When did your issue 1st occur? What operating system, browser, & browser version are you using? Did you refresh your browser cache? Are your cookies enabled? Lastly, a screen shot is often helpful.
6. Please abide by the Terms of Use . We are volunteers contributing our spare time. We are happy to assist you, so long as you are respectful and courteous.
7. We are always looking for new FreeToastHost Ambassadors to join our team and support fellow Toastmasters in their use of the FreeToastHost website system. If you are familiar with the system and have some interest, send a Send Us a Private Message.

Regarding the club websites

  • pratik411
  • Topic Author
  • Offline
  • New Member
  • New Member
More
8 years 10 months ago #58900 by pratik411
Regarding the club websites was created by pratik411
Hi,

I was going through few club websites and came across a robots.txt file which is used by the search engines to index a website. Why do we have a Disallow for all the files ?

User-agent: *
Disallow: /

This blocks the entire website from getting indexed in google. Can I change it myself for the club website?

Thanks
The topic has been locked.
More
8 years 10 months ago - 8 years 10 months ago #58904 by SteveTheTechie
Replied by SteveTheTechie on topic Regarding the club websites
We also use Allows that will allow indexing on specific folders that we deem important.

You cannot change that as it directly impacts overall system performance. We are not going to grant access to that.
Last edit: 8 years 10 months ago by SteveTheTechie.
The topic has been locked.
  • pratik411
  • Topic Author
  • Offline
  • New Member
  • New Member
More
8 years 10 months ago #58906 by pratik411
Replied by pratik411 on topic Regarding the club websites
Hi Sir,

Thanks for the reply.

Kindly find below my robots.txt file. Having the below robots file won't facilitate indexing as there is no
Code:
allow
parameter specified.

Help appreciated.
Code:
User-agent: Googlebot Crawl-delay: 10 Disallow: /jquery/ Disallow: /fthadmin/ Disallow: /logfiles/ Disallow: /OLD_FILES/ Disallow: /text/ Disallow: /js/ckeditor_smileys/ Disallow: /js/ Disallow: /json/ User-agent: bingbot Crawl-delay: 10 Disallow: /jquery/ Disallow: /fthadmin/ Disallow: /logfiles/ Disallow: /OLD_FILES/ Disallow: /text/ Disallow: /js/ Disallow: /json/ User-agent: MSNBot Crawl-delay: 10 Disallow: /jquery/ Disallow: /fthadmin/ Disallow: /logfiles/ Disallow: /OLD_FILES/ Disallow: /text/ Disallow: /js/ Disallow: /json/ User-agent: Slurp Crawl-delay: 10 Disallow: /jquery/ Disallow: /fthadmin/ Disallow: /logfiles/ Disallow: /OLD_FILES/ Disallow: /text/ Disallow: /js/ Disallow: /json/ User-agent: * Disallow: / Crawl-delay: 60 User-agent: magpie-crawler Disallow: / User-agent: rogerbot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: stremorbot Disallow: / User-agent: YandexBot Disallow: / User-agent: Ezooms Disallow: / User-agent: omgilibot Disallow: / User-agent: SeznamBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Sosospider Disallow: / User-agent: Sosospider+ Disallow: / User-agent: wonderbot/JS 1.0 Disallow: /
The topic has been locked.
  • GeorgeMarshall
  • FreeToastHost Ambassador
  • FreeToastHost Ambassador
More
8 years 10 months ago #58907 by GeorgeMarshall
Replied by GeorgeMarshall on topic Regarding the club websites
Google is certainly indexing clubs on FTH.
The topic has been locked.
More
8 years 10 months ago #58908 by SteveTheTechie
Replied by SteveTheTechie on topic Regarding the club websites

pratik411 wrote: Kindly find below my robots.txt file.


Ok, thanks, I will look into that when I get a chance this evening. We may have temporarily tweaked it a few months ago during the officer changeover when system performance was suffering.

However, keep in mind the following:

It is not your robots.txt file, per se. Any idea that that FTH websites are totally independent of one another and having their own robots.txt file is a bit of an illusion that the system promotes. The system is essentially really only one website (template) with many "personalities" (content), each one which results in "an individual club website".

We manage the robots.txt file(s) for the entire system to keep the *system* performance optimal.
The topic has been locked.
More
8 years 10 months ago - 8 years 10 months ago #58909 by SteveTheTechie
Replied by SteveTheTechie on topic Regarding the club websites

GeorgeMarshall wrote: Google is certainly indexing clubs on FTH.


Understood. However, as Brian will confirm, we have had instances where bad bots have just brought the system performance "to its knees" (not googlebot), so we tend to be very picky about what bots we allow and where we allow them to venture. In any case, in some cases, robots.txt is ignored anyway.

We do want to enable SEO, but we tend to focus on the major search providers.

I would refer you to the following article on semalt as an example: www.incapsula.com/blog/semalt-botnet-spam.html This *killed* the system performance a few years ago.

It is probably time for us to tweak our robots.txt file anyway for current bad bots, so this thread is probably a good thing. www.botreports.com/badbots/

I know that Google is looking at more parts of the websites (including javascript), so we may need to relax that a bit. However, I will be very wary of any resulting system performance impacts.
Last edit: 8 years 10 months ago by SteveTheTechie.
The topic has been locked.
Moderators: GeorgeMarshall, Pam, jliu, marc33, NotLiable, deedubbleyoo, NSB
Time to create page: 0.109 seconds

Copyright © 2025 FreeToastHost 3 Support. All Rights Reserved.