TOPIC:

Hi Toastmasters, this is an update to my earlier thread support.toastmastersclubs.org/4-general-...ot-indexing-our-site (which has since been locked). Yesterday, I sent a mail to Bing reporting the issue, and I received a reply from them today (see attached image).
According to them, their bots are not able to crawl the site, probably because of a block or some other reason.
Please look into the issue, and let me know if there's anything you'd like me to ask them.

If the IP were blocked it was because the access was abusive, this has happened many times with Bing and there is nothing we can do to prevent Bing from being abusive to the server.

If Bing wants to present their IP, I will have a look at what happened but I an not wasting my time trying to figure out which IP Bing bot is using this week.

Yeah but as a fallout of you not wanting to waste your time, many of the freetoasthost-powered websites aren't discoverable on Bing. In fact, among all the FTH sites that I've tried out, I've not seen a single one appearing in their results, except this one.

So I guess the short answer is no, it doesn't and won't show up on Bing ever. Thanks anyway.

No as a result of Bing not having a well behaved spider. The Bing spider kills the performance of the FTH server. The Bing does follow the industry standards of following the instructions set in the robots.txt. when they fix their spider we will let the back in.

We have no issues with the Google and Yahoo spiders just Bing.

I think we may want to try using a crawl delay directive so that at least we can show up on Bing:

blogs.bing.com/webmaster/2012/05/03/to-c...is-bingbots-question
www.siteground.com/kb/how_to_decrease_th...s_search_engine_bot/

I am committed to good server performance, also.

However, even though I think I understand the performance concerns, not showing up on Bing seems a bit problematic, since there really is no way that we can conceivably tell a complete stranger who is just looking for a Toastmasters meeting to attend that they must not use Bing to find an appropriate club's website. They are going to use what ever search engine they desire to use, and if their preferred search engine is Bing, then we are essentially impeding their ability to find and look at the website of a prospective club that they may want to visit and/or join.

Therefore, I think that trying to degrade Bing's rate of crawling FTH websites via a directive is probably worth investigating, since it would at least allow FTH websites to show up on Bing, albeit even if they are a bit outdated due to setting a high crawl delay.

We need to seek an acceptable compromise, an acceptable "middle ground" that does not kill the server while supporting all the various popular search engines of the current moment.

Brian wrote:

No as a result of Bing not having a well behaved spider. The Bing spider kills the performance of the FTH server. The Bing does follow the industry standards of following the instructions set in the robots.txt. when they fix their spider we will let the back in.

We have no issues with the Google and Yahoo spiders just Bing.

I wrote this reply to Brian's message before Steve posted.

I really don't think we should expect Bing to change their behavior. They index billions of other pages without an issue, so if anything needs to change, I think it's something on the FreeToastHost end. That could be the robots.txt, the rules you use to ban IPs or perhaps something else.

You're really doing a disservice to Toastmasters if you're letting thousands of club sites not be indexed by the second largest search engine in the world. It's a significant issue that you should spend some time looking into rather than brushing it off. I know you site admins do this for free, and we appreciate it, but you should seriously consider open sourcing the FTH project if you can't be bothered to seriously look into getting this issue resolved.

Bing does index the server in December there were over 31,000 hits from the Bing bot.

We do not block BING as an entity. We block IP addresses that attack the server in an uncontrolled manner.

We have all the search engines specified with the same control parameters in the system. Google and Yahoo have no issues obeying the parameters set, and index the server successfully without overwhelming the server.

The Bing bot and the MSN bot both do not follow the rules and consistently get blocked for bad behaviors, the blocks are temporary and expire after 60 minutes.

At this time we are not blocking any IP designated as BING or MSN bot IP

We do know what we are doing and do not need any arm chair quarterbacks. If your club is not happy with the FTH service your club is always permitted to find and manage its own hosting.

Brian wrote:

Bing does index the server in December there were over 31,000 hits from the Bing bot.

We do not block BING as an entity. We block IP addresses that attack the server in an uncontrolled manner.

We have all the search engines specified with the same control parameters in the system. Google and Yahoo have no issues obeying the parameters set, and index the server successfully without overwhelming the server.

The Bing bot and the MSN bot both do not follow the rules and consistently get blocked for bad behaviors, the blocks are temporary and expire after 60 minutes.

At this time we are not blocking any IP designated as BING or MSN bot IP

We do know what we are doing and do not need any arm chair quarterbacks. If your club is not happy with the FTH service your club is always permitted to find and manage its own hosting.

I understand you don't explicitly block Bing, but as you state, it's being blocked due to other rules you have in place.

When you do the following search on Bing:
site:toastmastersclubs.org -site:support.toastmastersclubs.org
as of now, only 2 results are found. On Google, that same search has 35,700 results.

So clearly, just about all the FTH club sites can't be found with Bing.

If there are so many other sites out there that can be indexed by Bing without an issue, there must be some change that can be made on the FTH side to make this work. As Steve said, perhaps you just need to find the correct settings to allow it to index the site at a rate that does not impact performance. I don't think simply saying "no" and not looking into a resolution is the right direction to go.