Online documentation working again!

FreeToastHost documentation is back online.

Bing is not indexing my site (Reply from Bing)

More
7 years 6 months ago - 7 years 6 months ago #67306 by arindam.roy
Hi Toastmasters, this is an update to my earlier thread support.toastmastersclubs.org/4-general-...ot-indexing-our-site (which has since been locked). Yesterday, I sent a mail to Bing reporting the issue, and I received a reply from them today (see attached image).
According to them, their bots are not able to crawl the site, probably because of a block or some other reason.
Please look into the issue, and let me know if there's anything you'd like me to ask them. :)

IFIM Toastmasters Club (05016997)
ifim.toastmastersclubs.org
Attachments:
Last edit: 7 years 6 months ago by arindam.roy. Reason: added the image since attachment quality is bad
The following user(s) said Thank You: FreeToastHostUser
The topic has been locked.
More
7 years 6 months ago #67309 by Brian
If the IP were blocked it was because the access was abusive, this has happened many times with Bing and there is nothing we can do to prevent Bing from being abusive to the server.

If Bing wants to present their IP, I will have a look at what happened but I an not wasting my time trying to figure out which IP Bing bot is using this week.
The topic has been locked.
More
7 years 6 months ago - 7 years 6 months ago #67363 by arindam.roy
Yeah but as a fallout of you not wanting to waste your time, many of the freetoasthost-powered websites aren't discoverable on Bing. In fact, among all the FTH sites that I've tried out, I've not seen a single one appearing in their results, except this one.

So I guess the short answer is no, it doesn't and won't show up on Bing ever. Thanks anyway.

IFIM Toastmasters Club (05016997)
ifim.toastmastersclubs.org
Last edit: 7 years 6 months ago by arindam.roy. Reason: .
The topic has been locked.
More
7 years 6 months ago #67364 by Brian
No as a result of Bing not having a well behaved spider. The Bing spider kills the performance of the FTH server. The Bing does follow the industry standards of following the instructions set in the robots.txt. when they fix their spider we will let the back in.

We have no issues with the Google and Yahoo spiders just Bing.
The topic has been locked.
More
7 years 6 months ago - 7 years 6 months ago #67365 by SteveTheTechie
I think we may want to try using a crawl delay directive so that at least we can show up on Bing:

blogs.bing.com/webmaster/2012/05/03/to-c...is-bingbots-question
www.siteground.com/kb/how_to_decrease_th...s_search_engine_bot/

I am committed to good server performance, also.

However, even though I think I understand the performance concerns, not showing up on Bing seems a bit problematic, since there really is no way that we can conceivably tell a complete stranger who is just looking for a Toastmasters meeting to attend that they must not use Bing to find an appropriate club's website. They are going to use what ever search engine they desire to use, and if their preferred search engine is Bing, then we are essentially impeding their ability to find and look at the website of a prospective club that they may want to visit and/or join.

Therefore, I think that trying to degrade Bing's rate of crawling FTH websites via a directive is probably worth investigating, since it would at least allow FTH websites to show up on Bing, albeit even if they are a bit outdated due to setting a high crawl delay.

We need to seek an acceptable compromise, an acceptable "middle ground" that does not kill the server while supporting all the various popular search engines of the current moment.
Last edit: 7 years 6 months ago by SteveTheTechie.
The following user(s) said Thank You: arindam.roy, FreeToastHostUser
The topic has been locked.
More
7 years 6 months ago - 7 years 6 months ago #67366 by FreeToastHostUser

Brian wrote: No as a result of Bing not having a well behaved spider. The Bing spider kills the performance of the FTH server. The Bing does follow the industry standards of following the instructions set in the robots.txt. when they fix their spider we will let the back in.

We have no issues with the Google and Yahoo spiders just Bing.


I wrote this reply to Brian's message before Steve posted.

I really don't think we should expect Bing to change their behavior. They index billions of other pages without an issue, so if anything needs to change, I think it's something on the FreeToastHost end. That could be the robots.txt, the rules you use to ban IPs or perhaps something else.

You're really doing a disservice to Toastmasters if you're letting thousands of club sites not be indexed by the second largest search engine in the world. It's a significant issue that you should spend some time looking into rather than brushing it off. I know you site admins do this for free, and we appreciate it, but you should seriously consider open sourcing the FTH project if you can't be bothered to seriously look into getting this issue resolved.

Club #3203719
Last edit: 7 years 6 months ago by FreeToastHostUser.
The following user(s) said Thank You: arindam.roy
The topic has been locked.
Time to create page: 0.104 seconds