Which is better UTF-8 or ISO-?

Someone asked today on a mailing list whether they should be using UTF-8 or “ISO” encoding for sending email. What’s the best choice depends on some of the details of the situation, but here’s the answer I gave:
UTF-8 will work for pretty much anything, as it’s just an 8 bit encoding scheme for Unicode (which is supposed to be the one character encoding to rule them all). It’s well supported in most languages and development environments – Windows has been native UTF-16 under the covers since the mid 90s, for instance – and typical messages that use mainstream glyphs should render well from utf-8 in most western MUAs and browsers.
There are still a very few old or broken clients out there that will not handle UTF-8 well but (outside the asian language market, where there’s still some non-ASCII, non-Unicode legacy usage) they’re typically ones that don’t really handle any character set encoding well and the only thing safe to send to them is either plain ASCII or whichever ASCII superset their OS happens to support natively (which is probably an argument for sending Windows-1252 codepage, but not a terribly strong one).
The various extended ASCIIs (such as ISO-8859-*) will only work for messages that are written solely using characters from that character set. If you have even one character in a message that cannot be expressed in ISO-8859-1, then you can’t use ISO-8859-1 to send that message.
ISO-8859-1 (aka Latin1) is fairly sloppy in some respects – it has no apostrophe, nor single quotes, for instance – but it can handle an awful lot of languages, from Kurdish to Swahili. It can’t handle Dutch, Estonian, Finnish, Hungarian and Welsh particularly well, nor can it show the Euro symbol (ISO-8859-14 or -15 are needed for some characters there).
A common problem is that many people (and the software they write) think that Windows uses Latin1. It doesn’t, it uses Windows-1252. If you accept messages written on Windows, using the Windows-1252 code page, and throw them out on the wire as ISO-8859-1 what you end up with is not quite right. It mostly works, as the two codepages overlap quite a bit, but they have different glyphs in the 0x80-0x9f range. So if you use single or double quotes (“smart quotes”), or the Euro symbol, or ellipses, or bullet, or the trademark symbol in your message they’ll be garbled. This is so common that some mail clients and web browsers will actually treat a document that claims to be ISO-8859-1 as Windows-1252, but that’s a bug workaround and not something it’s really safe to rely on.
If you’re doing personalized messages, and you’re sending one of them to Győző and one of them to Eiður then you may have to use different character sets for the two messages. If you’re talking about Győző and personalizing it for Eiður then you might find things break horribly.
Someone probably has some concrete data on mail client character set support, broken down by region and language, but my understanding is that this is a reasonable approach:

  1. If you’re sending any entirely non-ASCII messages (asian languages, primarily) then it’s way more complex than it should be, and you need to find someone who really understands the deployment of BIG5 and ShiftJIS and so on if you want to dial in to perfection. If you don’t have someone like that, UTF-8 is your best bet.
  2. If your tool chain supports non-ASCII messages, and you want to choose a single encoding, go with UTF-8. If the UTF-8 you end up sending is entirely, or almost entirely, ASCII then this will render well even on the tiny fraction of mail clients that don’t support character sets. (This is what I’d do).
  3. If your tool chain supports non-ASCII, but almost all your messages are plain ASCII and you want to be clever, encode each message separately. That’ll mean storing the templates and mailmerge data as UTF-8 probably, then converting the resulting message to ASCII if possible. That’s actually a lot simpler to do than it sounds, as a valid UTF-8 message that can be converted to ASCII is already an ASCII message, so all you need to do is set the charset to UTF-8 if there’s any byte bigger than 127 in the message or to us-ascii otherwise.
  4. If your toolchain was written to support only truly 8 bit character sets, or if that’s all you ever expect to send, then ISO-8859-15 is about as good as you can do.

If you’re accepting messages via a web front end, or any other path that could lead to stuff being copy and pasted or uploaded from Windows, look out for the Windows-1252 issues, and convert the provided message to whatever format you’re storing templates as (probably utf-8).
Then, once you have the message, be careful to send it over the wire as 7 bit ASCII, probably using quoted-printable encoding, to make sure it isn’t thrown away or transcoded to base64 en-route.

Related Posts

What is an email address? (part one)

Given we deal with email addresses every day, dozens or thousands or millions of them, it seems a bit strange to ask what an email address is – but given some of the problems people have with the grubbier corners of address syntax it’s actually an interesting question.
There are two real standards that define what is a valid email address and what isn’t. The most complex is RFC 5322 – Internet Message Format, which describes all sorts of things about the structure of an email, including what’s valid to put in From: and To: headers. It’s really too liberal in what it allows an email address to look like to be terribly useful, but it does provide for one very commonly used feature – the friendly from where the name that’s displayed to the recipient is not just the email address.

Read More

Troubleshooting the simple stuff

I was talking with one of my Barry pals recently and was treated to a rant regarding deliverability experts that can’t manage simple things. We’ve been having an ongoing conversation recently about the utterly stupid and annoying questions some senders ask. Last week, I was ranting about a delivery person asking what “5.7.1. Too many receipts this session” meant. This morning I got an IM.

Read More

Email is store and forward

Many of us are so used to email appearing instantaneous, we forget that the underlying protocol was never designed for instant messaging. When the SMTP protocol was originally proposed it was designed to support servers that may have had intermittent connectivity. The protocol allowed for email to be spooled to disk and then sent when resources were available. In fact, almost everyone who was around more than 10 years ago knows of a case where an email took weeks, months or even years to deliver.
These days we’re spoiled. We expect the email we send to friends and relatives to show up in their mailbox within moments of sending it. We expect that sales receipt or e-ticket to show up in our mailbox within instants of a purchase. We expect that our ISPs will get us email immediately, if not sooner.
But there are a lot of things that can slow down email delivery. At several points in the process an email may be spooled to disk. It stays on the spool until the next part of the delivery process can happen. Other points of slowdown include the various anti-spam, anti-virus and anti-phishing protections that ISPs must implement. Then add in the extreme volume of email (around 10 billion messages a day) and all of a sudden email delivery is slower than many senders and recipients expect it to be. This delay is not ideal, but the system is designed so that mail is not silently discarded.
While individual emails may be delayed, most users will rarely see that delay in the email that they send. Bulk senders, who may be sending thousands or hundreds of thousands of emails a day, may see more delays in a single send than the average user sees in years of sending one-to-one email.
Email is store and forward, not instant. Sometimes that means there is a delay in getting email into the recipients inbox. And, sometimes there isn’t anything anyone can do to speed up delivery, except to adjust expectations of how email works.

Read More