~~~ Please read before posting. ~~~

Important: We need your Club Number at a minimum, and as many details as possible.
For further info please read This page before posting.

Adding custom pages which can be found by Google

  • chrissimmonsFUTC
  • chrissimmonsFUTC's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 12
  • Thanks: 1

Adding custom pages which can be found by Google

3 years 10 months ago
#85517
Hi

What's the best way to add a Custom Web page, so that Google will index it?
I ask this because it seems to me, the method I have used (which I think is pretty standard), prevents the page from being found.

Months ago, I added a Custom Web page to our FreeToastHost site ( flindersuni.toastmastersclubs.org ), called "Club News". It's public, has keyworks, is in the site menu and frequently updated. It's also linked to from other pages. But Google never indexes it, despite repeated requests.
I think the reason is, that all pages have a robots meta tag which explicitly asks search engines to ignore all links:<meta name="robots" content="index,nofollow,noimageindex" /> 
I can imagine good reasons for doing this for some links (e.g. the generic links to TMI content, which are already well-known).
But for custom pages, this gets in the way of promoting the club website content.
Is there a way of making sure that custom pages don't get caught by this meta tag?
Or is this "by design" for some reason?
developers.google.com/search/docs/advanc...alify-outbound-links

thanks
Chris Simmons
The topic has been locked.
  • SteveTheTechie
  • SteveTheTechie's Avatar
  • Offline
  • Administrator
  • Administrator
  • Posts: 11526
  • Thanks: 3050

Re: Adding custom pages which can be found by Google

3 years 10 months ago - 3 years 10 months ago
#85518
Well, it is true that given we only have one donated server for the entire system to handle all 12,000+ websites, we do place a number of restrictions on what we will allow bots to do and not do, in order to prevent them from using up too much server processing power.  However, it is a bit of a subjective balancing act on our part regarding on what restrictions we put in place--we have access to overall performance data for the system, so we are certainly better positioned than individual users or club admins.  (That is part of our role in maintaining the system to keep it viable and performant.)

Let me take a look into why I had put those specifics in place for bots with that meta tag.   Keep in mind though, that my conclusion may well be that we are not going to change it.   Because of the nature of how resource constrained the FTH system is, there are just some things we will not support.
Last edit: 3 years 10 months ago by SteveTheTechie.
The following user(s) said Thank You: chrissimmonsFUTC
The topic has been locked.
  • chrissimmonsFUTC
  • chrissimmonsFUTC's Avatar Topic Author
  • Offline
  • New Member
  • New Member
  • Posts: 12
  • Thanks: 1

Re: Adding custom pages which can be found by Google

3 years 10 months ago
#85520
The team must be doing some real magic, to be able to support all this on one server!
The need to keep consistency must be very high. And keeping the right crawlers crawling no more than necessary takes work.
(I can relate to the situation: at the University where I was in charge of the main website, we were running for years on just two servers. It took quite a bit of careful tuning and monitoring to keep it going. And everyone wanted it to run *their way*!).
The topic has been locked.
Moderators: BrianPamrhtaylor3marc33NotLiablejgavinLcala305Jane Atkinsonpeterb323
Time to create page: 0.380 seconds