Earlier attempts at internationalisation worked fairly well for western European languages. which use just a small set of accented letters. But they were problematic for users of non-Latin characters, such as Chinese, Japanese, and many others. Sometimes some tweaks would make them work properly, but not always.

FreeToastHost (FTH) was previously using this sort of approach. It wasn't too bad for websites, especially if they don't get changed often, but it did not work well for email. As additional security features, such as DKIM, were added to email, it became more difficult.

Server upgrade

The FTH server operating system was upgraded recently, to a new version of Amazon Linux (2023). This was a jump of about 15 years in terms of the age of the programs, and not surprisingly, some issues showed up which previously hadn't been too much of a problem.

In particular, Japanese members using their native language in emails ran into difficulties. There were several clubs informing us of email issues. They couldn't use non-Latin characters in email subject lines. DKIM, which is an essential security item these days, didn't work and messages weren't getting delivered.

But it didn't only affect non-European languages. Emojis are also extended characters and they are very popular among speakers of English. Emails were also choking on these.

Bringing our system up to date

The modern approach to internationalisation uses a standard known as utf8mb4, which allows a vast array of characters to be handled. Some adjustments were needed to make sure that the FTH email system uses this reliably. The underlying software doesn't do this naturally, and so a number of special program routines needed to be written to take care of it. This took some working out. We wish to thank the people who helped test the changes and who reported bugs.†

This is a great start, but it is only a beginning. There are other parts of the FTH websites that need to be brought up to date to comply with this standard. In particular, there are the agenda and the basic email features (that is, where emails are sent from the actual site rather than your email client), which still require quite a bit of work. Ultimately, we wish to have sites which use the utf8mb4 standard in all areas and are truly international.

†Of course, there could still be some other sneaky bugs hiding out. If you find any, please report them to us so we can fix them.