UTF-8 good, UTF-16 bad

Lately I’ve been programming in Go. I started by writing a few popular Unix tools in Go, before moving on to a lexer that supports structural regular expressions and UTF-8.

Unicode support was easy to add because Go lives and breathes UTF-8. But why did the Go designers favour UTF-8, and not UTF-16? Widespread software such as Windows, Java, and Python use UTF-16 internally, so why not Go?

Further reading revealed the answer. UTF-16 is a historical accident that persists mainly due to inertia. UTF-16 has no practical advantages over UTF-8, and it is worse in some ways. The Go designers knew what they were doing, so they chose UTF-8.

Down with UTF-16

Before, from vague hunches, I had thought that UTF-16 was simpler because each character took exactly 2 bytes (so it handled only a strict subset of Unicode), and that UTF-16 was more compact for certain languages.

The first statement is patently false. UTF-16 is a variable-length encoding, and hence not much simpler than UTF-8. As for the second: consider HTML. The amount of whitespace and tags in a typical document is high enough that UTF-8 is more compact than UTF-16. This generalizes to other formats, such as Go source code.

So there’s no upside. But there are plenty of downsides to UTF-16:

1. We must worry about byte ordering because UTF-16 deals with 16 bits at a time. A related drawback is the loss of self-synchronization at the byte level.

2. We lose backwards compatibility with ASCII.

3. We lose backwards compatibility with code treating NUL as a string terminator.

4. We cannot losslessly convert from byte streams.

5. The encoding is inflexible: hopefully 0x110000 code points should be enough, but if it were extended (again), UTF-16 would need major surgery. In contrast, by simply allowing up to 6 bytes, UTF-8 can handle up to 2³¹ code points.

6. Ugly encoding: it’s the reason why U+D800 to U+DFFF is reserved (though thankfully UTF-16 is self-synchronizing with 16-bit granularity).

I tired of thinking about the drawbacks of UTF-16, but there are likely more. Thus even if we only worked with a strange set of documents where UTF-16 is more efficient, the savings are too small to justify the complexity.

A living dead standard

How did so many projects settle on such a horrible standard? It’s a sad story. The original Unicode supported exactly 0x10000 code points, making UCS-2 (the precursor to UTF-16) attractive. Each character neatly corresponded to one 16-bit word. Although UTF-8 might still be preferable thanks to byte ordering and ASCII compatibility, UCS-2 is certifiably sane. People built systems based on UCS-2, and rejoiced that the codepage Stone Age was over.

Then an unnatural disaster struck: the Unicode Consortium decided more code points were needed. The supple UTF-8 got away unscathed, but there’s no way to squeeze more than 16 bits into UCS-2. Apart from abandoning UCS-2, the only recourse was to reserve a chunk of values for encoding the higher code points. Thus UTF-16 was born. Hopefully, developers could simply tweak masses of existing UCS-2 code for the new standard. Sadly, various UTF-16 bugs suggest that much code remains unaware UTF-16 is a variable-length encoding.

Despite the trauma it would cause, they should have killed UCS-2 when they had the chance. Now we have a grotesque encoding scheme that shambles alongside UTF-8.

Gauss was here

According to Wikipedia, UTF-8 was built on ASCII by Rob Pike and Ken Thompson. ASCII was developed from telegraph codes by a committee. The telegraph codes stemmed from a Western Union code, which evolved from a code by Donald Murray, who modified a 5-bit code invented by Émile Baudot, who based his work on a code designed by Carl Friedrich Gauss and Wilhelm Weber. It’s awe-inspiring that the Prince of Mathematicians was involved. And if he were alive today, I bet he’d disparage UTF-16!

Inkscape and Valgrind

I spend a lot more time at the PBC (Pairing-Based Cryptography) library site than I would like since I maintain those pages. I bet most webmasters have similar complaints.

I began to tire of seeing page after page of text, so I broke the monotony by adding a site logo. It serves no other purpose, but who knows? Maybe one day I'll be hawking coffee mugs and T-shirts emblazoned with this graphic!

I went through several ideas before selecting a shape, during which I gained respect for professional logo designers. Some famous logos seem so simple that I feel I could have designed them. But on further examination, they insiduously invade and reside in one's memory, are pleasing to the eye (or at least are not eyesores), and in fact satisfy many constraints.

After a few attempts I discovered that it is extremely diffcult to find symbols with the necessary properties, and in my case, to even judge how effective a given image is. Thus I settled on a design as soon as I sketched one that I could tolerate.

I recall the path I took. At one point I was experimenting with a stylized "P(B,C)". I removed the letters to get "(,)" and played around with the parentheses and comma until I arrived at the current logo.

As for the colour scheme, to pay homage to my current university, I let Stanford's design guidelines on colour dictate my choices. Thus the logo is predominantly cardinal, with sandstone and black playing secondary roles. On most desktops, many screen elements are white and hence all four Stanford "Identity Colors" appear.

The result reminds me of a symbol featured in the movie "The Incredibles". I hope this causes people to think the PBC library is incredible, at least subconsciously!

I used Inkscape to realize my idea. I had never tried Inkscape before. I was surprised at how easy it was for a newbie to use and how quickly I became comfortable and efficient with its interface.

(Strictly speaking, this is false. Many moons ago I fired up Inkscape's ancestor Sodipodi, but I was unable to do much without reading documentation. Either Sodipodi has since made great advances, or the fork was worth the trouble.)

While I'm on my soapbox lauding software, let me also praise Valgrind. I first heard of this program years ago. If only I had tried it years ago.

Finding memory leaks in my C projects had always been troublesome. Thanks to geek machismo [isn't this an oxymoron?], I chose the hard way, namely, to code wrappers around standard memory allocation functions that recorded statistics which I would then analyze to detect leaks.

Since my leak detection code would need to be bug-free itself for this to work, I would restrict myself to primitive designs to simplify implementation details, ultimately leaving much of the sleuthing to the developer. Furthermore, toggling leak detection was so annoying I rarely checked for leaks.

As I recently discovered, Valgrind magically gives detailed reports on where the leak is and what is being leaked [insert HP joke here]. And to use Valgrind one only needs to invoke gcc with different options during compilation.

Valgrind has other features too, but I haven't tried them yet.

Disabling Links to the Current Page

According to Jakob Nielsen's list of The Ten Most Violated Homepage Design Guidelines, a link should never point to the current page.

I've seen high-profile sites that break this rule so I'd expect most users not to get too confused. Nevertheless, I whipped up some javascript to do disable links to the current page for this site. Ideally I'd like the links to be removed in the raw HTML and not require Javascript, but the way I've built my site makes this difficult.

A brief search revealed that some people do this by changing the onclick callback, and also the mouse cursor, but I prefer to replace the whole anchor tag with a span tag of a particular class.

Elements of Style

I had heard of William Strunk's Elements of Style long ago. I finally saw what all the fuss is about when I found this online HTML version of the book [alternate link].

Much of this 1918 work is still relevant today. I was surprised to find that many misused and/or hackneyed words and phrases have been around for many years.

When I write English, I borrow techniques I use to write code. I start with a syntactically correct body of text that gets the job done. I then revise and optimize: I reword, rephrase, replace and remove lines, aiming for clean, concise, efficient and effective writing.

The analogy isn't perfect. With prose, I'll throw in extra words if I like the way it sounds, but for code, I'll sacrifice style for performance.

Some of the rules in Elements of Style seem to achieve the same goals, though they are explained from a nonprogrammer's viewpoint!

More Redesign

Less PHP, CSS code duplication throughout site. Changed the menu style. No more submenus, and now there are unclickable headings. Found a nice colour scheme generation tool.

I shrunk the validation buttons as I felt they were getting in the way. Forgot to handle the alpha channel the first time around. They now sit humbly in the bottom right hand corner.

I experimented with a pink/purple colour scheme with a horizontal menu for the Bliss pages. Most of my site is back up, though I got rid of some of the old content. The xmmspipe site is the only one left with the original look-and-feel (not counting the old Toshiba pages), though my mathematics notes still look the same as when I first started using CSS.

Major Homepage Renovation

When I first wrote my homepage, I modeled my site on Debian's site which at the time used HTML tables for layout. I had never used CSS and I also felt that plain HTML would give maximum exposure.

A while later, I had to learn CSS when I wrote pages using MathML. The XHTML/CSS approach seemed cleaner and easier, and sometimes when I added new sections to my site I'd use XHTML and CSS. But I never converted the rest of my site out of laziness, and also I was worried that my site would look wrong in IE with its numerous CSS bugs.

I now feel the time is right to use CSS on my whole site [10 reasons why tables should not be used for layout]. I've spent a while trying to learn CSS properly. Additionally I've been reading about webpage design. Apparently, sans-serif fonts are preferred to serif fonts when on screen. The best widely available on-screen sans-serif font is Verdana while for serif fonts it's Georgia[1, 2].

Note I'm still sticking to HTML for now, as XHTML is not handled properly by IE or Google yet.

Now my default CSS file gives my site a kind of bluish Slashdot.org look. My only regret is that w3m will no longer be able to render my page layout (perhaps that's the only advantage of using tables for layout?), but then again, how often do I use text mode browsers?

My site is also long overdue for a general cleanup. So far I've been working on the presentation of my page but the content will also be reorganized.