Non-English Characters Omitted in Emails – Discussion

A year ago, I reported a problem on this forum about accented characters omitted in emails sent using FTH tools such as the agenda and the new member welcoming message. Original thread: support.toastmastersclubs.org/...accented-characters-omitted-from-the-new-member-welcome-message

Since then, the same issue has been reported at least twice by other people:

• support.toastmastersclubs.org/...special-characters-missing-in-e-mails-problem-with-code-page
• support.toastmastersclubs.org/...french-characters

Unfortunately, the FTH staff has repeatedly dismissed it as a user problem. I would like to revisit and discuss the 2 claims made by FTH to justify their decision.

Claim 1: FTH only supports UTF-8 encoding and does not support double-byte or multiple-byte characters.

This is self-contradictory.

UTF-8 is a variable-width encoding, not a single-byte encoding (see reading suggestions below). Logically, if a system supports UTF-8 encoding, then it supports multiple-byte characters. Conversely, if it does not support multiple-byte characters, then it does not support UTF-8 encoding.

Here are examples of how certain characters are encoded using UTF-8 ( www.fileformat.info/info/charset/UTF-8/list.htm ):

Symbol	Hex Code (2 digits = 1 byte)	Binary Code (8 bits = 1 byte)	Number of Bytes
Dollar sign ($)	24	00100100	1
Pound sign (£)	C2 A3	11000010 10100011	2
Euro sign (€)	E2 82 AC	11100010 10000010 10101100	3

Perhaps the confusion comes from UTF-8’s backward-compatibility with ASCII, a single-byte character set (which, not coincidentally, contains all the essential characters for communication in US English).

To be clear, FTH obviously supports UTF-8: multiple-byte characters (accented, Chinese, emoji, etc.) can be written and displayed without any problem on FTH web pages, including the agenda. However, multiple-byte characters are not displayed in emails sent using the FTH emailing tools; they simply vanish. Thus it can be asserted that the FTH emailing tools do not currently support UTF-8.

Claim 2: Special characters in emails sent from (or received by) a Windows operating system will not be displayed properly because FTH uses UTF-8 while Windows uses UTF-16 by default.

This is false.

When composing a message to be emailed, the proper encoding is set by the emailing service, not by the computer’s operating system. A “Content-Type” tag is added in the email header (usually hidden) that indicates which encoding standard has been used to compose the message. When receiving an email, the email client determines how to decode the message by looking at the Content-Type setting ( www.emailonacid.com/blog/article/email-d...ding_in_html_emails/ ).

UTF-8 is universally recognized as the preferred encoding standard for web pages. Also, practically all email clients, including Microsoft Outlook and Gmail (the client I use), support UTF-8 by default.

In the past, due to the many character sets available on different systems and in different parts of the world, certain characters or entire messages would sometimes come out garbled due to applying the wrong character set. With the dominance of Unicode and UTF-8, this has become a rare event. I am a French-speaker and a long-time user of Internet services and, in the last several years, I have never experienced any problem with the use of accented characters, FTH being the unique and glaring exception.

Reading suggestions:

• UTF-8 (Wikipedia)
• Unicode, UTF-8 & Character Sets: The Ultimate Guide
• The Absolute Minimum Every Software Developer Absolutely Positively Must Know about Unicode and Character Sets (No Excuses!)
• An Encoding Primer

Considering how the FTH staff is usually caring and attentive to the users' needs and issues, I find it baffling that the problem I and other people have reported is getting completely and utterly ignored. I remember that, not so long ago, FTH expressed some interest in providing versions of FTH in different languages. If that interest is still there, then we should be worried about non-English characters vanishing from emails sent using FTH tools regardless of which operating system is involved.

I remain at your disposal and would be happy to help if you need a test user.

Sincerely,

To anyone out there having the same problem as I do:

I just discovered a workaround. As it turns out, it is possible to get non-English characters to show up in emails… by using HTML codes. Two kinds are presented here.

HTML Named Character Entities
A classic. Named character entities include accented characters in the Latin alphabet. Examples:

e with an acute accent (é)	Code: é
a with a grave accent (à)	Code: à
o with a circumflex accent (ô)	Code: ô

Roles titles in the agenda can be rewritten using these codes. Example: “Évaluateur général" can be rewritten as follow: Note that once you click “Save”, the FTH system will replace the codes with their corresponding characters. Or so it seems. In reality, they are saved internally as HTML codes.

Since HTML codes are written in standard ASCII (i.e. single-byte characters), they are preserved through the emailing process and their corresponding characters therefore show up properly in emails.

HTML Numeric Character Reference
Almost any Unicode character can be invoked using an HTML code starting with &#x and followed by a hex reference. Examples:

Character	Unicode	HTML Reference
Dollar sign ($)	U+0024	Code: $ (also $)
Pound sign (£)	U+00A3	Code: £ (also £)
Euro sign (€)	U+20AC	Code: € (also €)
Grinning Face	U+1F600	Code: 😀

Limitations
Based on my tests, HTML codes work on role/item titles in the agenda, but not the role/item descriptions (see below). They also work on member names in the Membership Management section.

Alas, they do NOT seem to work on the New Member Welcome Message.

Role/Item Description
This part has an embedded text editor (see attached image). Certain special characters, when typed directly into the editor, are automatically rewritten internally as HTML codes. This is why, for example, accented characters typed in role/item descriptions are properly displayed in emails.

Note
The workaround described above is just that: a workaround. It is rather cumbersome to use, but at least my French-speaking club can finally email the agenda and expect it to look decent again! Hopefully, it will stay that way.

Credit to Manon from Toastmasters club Les Mots Arts for finding out about this HTML code trick.

I've run into the same problem, and came to the same conclusions as you have. I didn't know about the html code trick - thanks.

Hello Daniel: Thank you so much for doing this research and sharing what you've found with all of us. I was just discussing this issue with other District 61 members in a workshop yesterday, and lamenting the fact that the problem has not been addressed. I found this site that lists more of the HTML character entities: dev.w3.org/html5/html-author/charref
Will try re-do my bilingual club's agenda template using these. If they work, will share with my workshop participants!

Thanks again!
Kimman Chan
Laurier Toastmasters, Ottawa Toastmasters

I re-did my agenda template using the HTML character entities. I'd say it's improved by 75% - 80%. Some characters still got stripped after I had sent the agenda via the server. For some reason, o and i with accent circonflexe would take turns getting stripped. I've replaced them at least four times, but one of them would disappear after the agenda's been sent. Rather frustrating.

I'm hoping that FTH would kindly work on this on their end, so that clubs in regions that do use accents in their languages would not have to work so hard to try to produce an un-garbled agenda for their members. Personally, I think it's in Toastmasters' interests to accommodate members that write in languages with accents. I understand FTH is run by volunteers. When you have a chance to turn your attention to this issue, your efforts would be greatly appreciated by members in a multitude of countries! Many thanks in advance for your work!

Thank you for your feedback, Kimman!

Note that the special characters must all be replaced with HTML codes at the same time. Every time changes are made to a role/item, the special characters must be replaced with codes again, otherwise the previously inserted codes are lost.