TOPIC:

Hi

What's the best way to add a Custom Web page, so that Google will index it?
I ask this because it seems to me, the method I have used (which I think is pretty standard), prevents the page from being found.

Months ago, I added a Custom Web page to our FreeToastHost site (flindersuni.toastmastersclubs.org), called "Club News". It's public, has keyworks, is in the site menu and frequently updated. It's also linked to from other pages. But Google never indexes it, despite repeated requests.
I think the reason is, that all pages have a robots meta tag which explicitly asks search engines to ignore all links:<meta name="robots" content="index,nofollow,noimageindex" /> 
I can imagine good reasons for doing this for some links (e.g. the generic links to TMI content, which are already well-known).
But for custom pages, this gets in the way of promoting the club website content.
Is there a way of making sure that custom pages don't get caught by this meta tag?
Or is this "by design" for some reason?
developers.google.com/search/docs/advanc...alify-outbound-links

thanks
Chris Simmons

Well, it is true that given we only have one donated server for the entire system to handle all 12,000+ websites, we do place a number of restrictions on what we will allow bots to do and not do, in order to prevent them from using up too much server processing power.  However, it is a bit of a subjective balancing act on our part regarding on what restrictions we put in place--we have access to overall performance data for the system, so we are certainly better positioned than individual users or club admins.  (That is part of our role in maintaining the system to keep it viable and performant.)

Let me take a look into why I had put those specifics in place for bots with that meta tag.   Keep in mind though, that my conclusion may well be that we are not going to change it.   Because of the nature of how resource constrained the FTH system is, there are just some things we will not support.

The team must be doing some real magic, to be able to support all this on one server!
The need to keep consistency must be very high. And keeping the right crawlers crawling no more than necessary takes work.
(I can relate to the situation: at the University where I was in charge of the main website, we were running for years on just two servers. It took quite a bit of careful tuning and monitoring to keep it going. And everyone wanted it to run *their way*!).