Encoding videos for the G1

Soon after Android's debut, initial reviews were complaining about the G1's inability to play videos. At the time I felt it was a trifling matter. Since YouTube already works on the G1, a more general video player ought to require only minor tinkering. And why is it a big deal anyway? Are people in a rush to experience Hollywood blockbusters on a tiny screen and tinny speakers?

Sure enough, video playing apps appeared quickly on Android Market. I found at least three in my last search. As for their utility: now that I've played with them, I understand why some see a video player as a must-have, and not merely yet another vehicle for demonstrating the G1's capabilities.

While breathtaking cinematography, award-winning soundtracks and state-of-the-art special effects translate poorly to a handheld device, movies that derive their strength chiefly from factors such as a witty dialogue lose little. Informational clips are another good use case. And even with scenes that only work well with a big screen: if they're my favourite scenes of all time, it's still comforting to have them at my fingertips, even if they are greatly reduced.

Which leaves the problem of converting videos to a G1-friendly format. I knew almost nothing about the black art of video encoding, but after hours of false starts and trawling the net for help, I still know almost nothing. However, I picked up enough to get started.

An unofficial FAQ states the G1 currently supports the H.264, 3GP and MPEG4 video codecs. I skimmed a few Wikipedia entries to decode this alphanumeric soup, and my interpretation is that this really means the G1 recognizes .MP4 and .3GP files (which are container formats and not video encoding methods) and understands the H.263, MPEG-4 ASP ("roughly similar" to H.263) and H.264 (aka MPEG-4 AVC) video compression standards. [Trivia: ASP expands to the somewhat oxymoronic "Advanced Simple Profile".]

The G1 only speaks the BP (Baseline Profile) dialect of H.264; videos that use more advanced features of H.264 play incorrectly, or not at all. [At last this post makes sense to me!]

On Ubuntu, the easiest way to encode videos was to use avidemux (sudo apt-get install avidemux). For audio, I only tested the AAC codec (usually at 96kbps), but several others should also work. For video, selecting MPEG-4 ASP produced videos that seemed to tax the CPU heavily on playback, often to the point of unwatchability. Perhaps this could be remedied by lowering the number of frames per second via an appropriate filter.

In any case, I prefer selecting the MPEG-4 AVC video codec and dumbing it down for the G1 by disabling the appropriate H.264 options: I changed the Max Consecutive B-Frames to 0, unchecked all options related to B-frames, and disabled CABAC along with the 8x8 Transform and 8x8 Intra search options. As before, I added the MPlayer variant of the resize filter: the G1 has a 480x320 screen in landscape mode. At 384kbps, this produced videos that played smoothly though higher bit rates should be feasible.

Now that I appreciate video more, I eagerly await the "Cupcake" Android update which is rumored to have video recording. I also hope the next generation of hardware will have video out and enough power to handle any popular codec.

G1 Miscellany

What fortuitous timing! Intending to start from a clean slate, I performed a factory reset on my G1 recently. I had forgotten that, unlike the Developer Edition of the G1, even an unlocked standard issue G1 requires a SIM card with a data plan for the initial setup.

I was going to wait until I could borrow a suitable SIM from a friend for a few seconds (as after registration, I'm happy to use just WiFi for my G1 net needs for the time being), but I just heard about a new jailbreak for all G1s. It works by flashing the G1 back to RC29 where the old jailbreak can be exploited. Once done, one can register the G1 via WiFi, without a data plan or even a SIM.

Some Linux-specific notes. The G1 is touchy about the SD card for the RC29 image. The time it worked for me, I had reformatted it with "mkfs -t vfat /dev/foo", and used uppercase exclusively for the filename. To get the "adb" tool from the Android SDK working, I had to create an appropriate /etc configuration file.

I only needed two of the incantations. On the G1, I enabled adb via:
<enter>setprop persist.service.adb.enable 1<enter>

and once in the adb shell, I opened the Settings menu before registration:
$ am start -a android.intent.action.MAIN -n com.android.settings/.Settings

From here it was trivial to setup WiFi and sign in.

Some unrelated tips. When typing, Alt + Delete clears the whole line. RemoteDroid turns the G1 into a wireless keyboard and mouse; a fun application to show off.

I remain easily amused by the digital compass. The Orienteer app is often the first app I demonstrate. Also, the Sky Map app is perfect for those interested in picking out constellations in the night sky, but not interested enough to actually memorize them and figure out when and where they can be seen.

The T-Mobile G1

I had intended to code more for my GP2X, starting with tweaking my version of Netwalk. But I almost certainly never will, now that I recently received a T-Mobile G1 phone. Though less suitable for gaming, its smaller size along with its phone and WiFi mean I'll actually carry it everywhere and run its apps.

I had thought it likely that I'd end up reverting to my old simple phone and only access the Internet with my laptop, as I have always done, but the G1 has enough right that I finally see why so many people are effectively surgically grafted to their smartphones. Judging by the last few days, I've become one of those addicts I formerly mocked. I've even caught myself scrolling the home view (or whatever it's called) back and forth, just to enjoy the smooth parallax action at my fingertips.

Even now I get that "shiny new toy" delight from discovering a new G1 trick. Some "Wow!"-worthy free apps:

• Twisty: Interactive fiction on the go. (Not on Android Market at the moment; I installed this by allowing unknown sources and running ApkInstaller.)
• ShopSavvy, CompareEverywhere: When did you last own a phone that could scan barcodes for price comparisons?
• WikiTude: Try this futuristic app in a photogenic location.
• StreetView: Digital-compass-enhanced deja vu on demand.
  
• Amazed: Hone your tilting skills.
• Voice Dialer: I have to put on an American accent to get digits working, but I'm impressed it recognizes my atrocious pronunciation of certain names. When will we see more voice control and text-to-speech apps?
• Flashlight: Unexpectedly handy. If only I could find a colour that brings out the printer dot codes. Or even better, an app that could decode the patterns for the secret agent appeal.
  

Full disclosure: I work for Google, though not on Android. Not that it matters as you should approach everything you read with healthy skepticism! Anyway, I'm not completely happy with the G1...

The n things I hate about Java

If it weren't for Java, I'd already be writing Android applications instead of writing about them. I've never gotten along well with the language. I'm old enough to remember the hype surrounding its release. I instinctively distrust excessive marketing (which indirectly caused an unrelated language to be named "JavaScript") and my experiences have only reinforced my initial disdain.

When I first examined Java, I was dismayed that autoboxing was unsupported (now fixed) resulting in the "int versus Integer" mess. I also felt they had missed a golden opportunity to leave the most confusing parts of C syntax behind, instead of superficially pandering to veteran programmers. Also, my pro-multiple-inheritance stance obviously clashed with Java. It seemed to me Java was made public before it was ready. Resources spent on marketing may have been better spent on improving the language before release.

Years later, another encounter cemented my opinion. In a project where I had to use Java, I wanted a thread to perform a nonblocking read on a channel. I was mildly perturbed when I found the Java threading API describing certain functions as deprecated because they were impossible to implement safely. How could they get this wrong? Surely they could have studied any of the many previous threading APIs?

More distressing was my lengthy trial-and-error discovery that my desired nonblocking read was impossible, due to some API design detail which I've gladly forgotten. I do remember documentation claiming the issue was addressed in a newer version, but unfortunately I was constrained to an older version. When I eventually worked around the problem, I felt especially frustrated since I could do exactly what I wanted in a few lines of C.

Shifts in my beliefs have further lessened Java's appeal. I was once an object-oriented fanatic, even submitting an object-oriented entry to a programming contest in a futile attempt to "prove" it was the right way to design software. (To clarify, I'm not saying the attempt was futile because it was object-oriented; the blame lies with me.) In those days I shunned Java because it wasn't object-oriented enough. Since I renounced my object-oriented ways years ago, I now feel Java is too object-oriented!

Now, to develop for Android, I must face Java again. That I'm willing to consider this is itself a glowing endorsement of the G1. I remain unimpressed: Java's primary advantage appears to be its fame, not its features. In fact, developers are urged to avoid common object-oriented practices for performance reasons.

Additionally, the downloads are heavyweight; a good net connection is a must. The "Hello, Android" example somehow requires a nontrivial amount of code as well as an XML file. In contrast, for C, a kinda sorta C compiler fits in a few hundred lines, and "Hello, World" fits right here: int puts(const char *);int main(){return puts("Hello, World!");}

This comparison is grossly unfair, as C libraries for such a sophisticated phone would also be nontrivial, but all the same, if the Bell Labs researchers of yore were behind the drawing board, the programming interface would be terse and elegant.

One bright spot is Android's register-based Dalvik virtual machine. A prevailing attitude was (and still is?) that Java could replace C/C++ in most situations: how can this be when the standard JVM is a stack-based snail? This is yet another detail Java should have got right from the start; compare with the design of Inferno's Dis virtual machine.

I hope Dalvik bytecode compilers for more agreeable languages will spring forth soon, as that will eliminate my biggest gripe.

Hardware Hankerings

Some deficiencies can be resolved with a little money. At the top of my wishlist is a standard stereo jack so I can use it as a stand-alone music player; I'll probably get an HTC 3-in-1 USB adapter for this purpose. (New G1 buyers will get an adapter free, but that's little consolation for me.) I'll probably also fork over dough for a bigger micro SDHC card, preferably at least 8GB so I can laugh at standard 4.7GB DVDs.

Other features will have to wait for the next generation. For gaming, I'd love to have an Android device with a nice D-pad or two; see the Pandora. While I'm at it, a video camera as well. One that can face the user so you could videophone others like in many a sci-fi flick. And then I may as well ask for a video out too, so it could be a stand-alone video player. Rather than build in all these gadgets, it may be better to have a few standard ports (perhaps USB) to attach oddball devices. I can't wait to see what they'll come up with.

Notes on my notes on Git

I was mildly shocked, but honoured, to find that the venerable (or at least popular!) Wikipedia references my tutorial in its article on Git. Interestingly, the entry states that Git tips and tricks are 'popularly referred to as "magic"'. I named my guide Git Magic as a homage to Arthur C. Clarke's quip featured in the preface, and also because magic happens to be a hobby of mine. If the Wikipedia comment is true, then either my title was fortuitously appropriate, or I have unwittingly coined a phrase.

More pleasant surprises followed. The guide appeared on the first page when I Googled for "git" (though sadly much lower on Yahoo). I found several pages linking to mine, whose recommendations were probably responsible. Thanks to everyone who praised and promoted my site. [I do work for Google, but I have no control over the ranking of my content.]

It almost makes me feel guilty that my enthusiasm for maintaining the guide and for source control in general has waned. But this is inevitable because of Git's power and convenience: I use Git so frequently that it has faded into the background. Fascinating new commands have gradually transformed into boring old reflexes. Documenting them has as much appeal and utility to me as documenting my exploits with, say, the ls command, or sleeping and eating. Also, there are many more Git tutorials these days. But as long as my site ranks well, I'll attempt to keep it updated.

I use Google Analytics to keep an eye on site traffic as I lack access to the logs on the Stanford server hosting Git Magic. Some observations:

On 20080412, traffic to my site, which includes Git Magic, spiked at over 10000 hits. Previously few days saw more than 100 hits. The spike rapidly diminished to 100 to 200 hits per day until 20080805, when a larger spike of 16000 hits occurred. Again, this activity tapered off quickly to a few hundred hits per day until 20081102, which featured a small spike of over 3500 hits.

It seems the first and third spikes were caused by reddit.com, and the second spike was caused by an article at LWN.net.

Since mid-September 2007, roughly when the guide was released, the top three browsers were Firefox (67.43%), Safari (12.48%) and Internet Explorer (6%). The top three operating systems were Windows (36.53%), Linux (35.15%) and Macintosh (26.9%).

A handful of hits identified themselves as mobile devices: iPhone (489), iPod (113), SymbianOS (19), DangerHiptop (11), PalmOS (4), Android (3). Some intriguing curiosities: Playstation 3 (8), Nintendo Wii (6) and OS/2 (1).

During the last spike, the top three browsers were Firefox (69.42%), Safari (12.33%) and Chrome (5.15%), and the top three operating systems were Windows (49.45%), Macintosh (27.23%) and Linux (21.73%).

Restoring GRUB after reinstalling XP

For me, a common system setup is dual-boot Windows and Linux, and a common problem is an overwritten MBR (Master Boot Record) after reinstalling Windows XP, making it impossible to boot to Linux. Most sites recommend using a Linux install or rescue CD to fix this, but I prefer to install the GRUB bootloader on the Windows partition, and then use GRUB to boot into Linux to repair the boot record:

1. Download the latest zip file from GRUB for DOS and extract the file grldr to C:\
2. Run cmd and type:
   cd \
   attrib +r +s +h boot.ini
   edit boot.ini
   
3. Append the line
   c:\grldr="GRUB"
   to the operating systems section, which is usually last. Save and quit.
4. Type
   attrib -r -s -h boot.ini
5. Reboot and select GRUB from the boot menu.
6. From the grub prompt we can repair the system:
   find /boot/grub/stage1
   root (hdX, Y)
   setup (hd0)
   reboot

Take care with boot.ini as invalidating this file can render the system unbootable. In this case you must boot by another method, e.g. network, USB, CD or floppy.

AsciiDoc and MathML

Nowadays I use AsciiDoc to author HTML. I'm even converting old pages to use this convenient and presentable text format.

My mathematics notes present an interesting challenge. For MathML the AsciiDoc user guide recommends using double-dollar passthroughs around equations, but this implies at least three characters are required to delimit every equation, since ASCIIMathML or equivalent itself needs at least one character.

I'd like to have something like:

= Introduction =

Let $E: Y^2 = X^3 + a X + b$ be an elliptic curve.

to just work, so I settled on the following solution. I use itex2MML because

• My old-fashioned, superstitious, purist side prefers bare-bones JavaScript-free static documents.
• Familiarity.
  
• It has a handy syntax for equations in display mode, i.e. equations in their own center-justified paragraphs as seen in mathematics texts. An invaluable feature, as I can't figure out how to center a paragraph with AsciiDoc.

It should be simple to substitute an alternative such as ASCIIMathML. Some may prefer its approach because documents are smaller, and the page source is intelligible. But I argue that most never bother viewing the source, the savings are slight, and one can always provide the AsciiDoc source if these factors matter.

Write these rules to a file named "macros":

[miscellaneous]
newline=\n

[blockdef-passthrough]
delimiter=^@{4,}$
subs=none

The first is optional. I just abhor the CR LF abomination.

Then feed the source through these commands to produce the final product, which should be a file with an .xhtml extension:

sed 's/\$\([^$]*\)\$/+++$\1$+++/g' \
| sed '/\\\[/i@@@@' \
| sed '/\\\]/a@@@@' \
| asciidoc -b xhtml11 -f macros - \
| itex2MML | sed '/<!DOCTYPE/c \
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1 plus MathML 2.0//EN"\
"http://www.w3.org/TR/MathML2/dtd/xhtml-math11-f.dtd" [\
<!ENTITY mathml "http://www.w3.org/1998/Math/MathML"> ]>
/xhtml11.dtd/d'

The above gobbledygook [careful with some of those newlines; Blogger may have put extra cuts in my lines] performs the following:

1. Put the "+++" AsciiDoc inline passthrough around equations delimited by "$".
2. Surround display-mode equations, delimited by "\[" and "\]", with our newly defined "@@@@" block passthrough macro.
3. Put the resulting mess through AsciiDoc, which converts everything but our equations to an HTML 1.1 document.
   
4. Run itex2MML to convert the equations to MathML.
5. Fix the DOCTYPE declaration.

Caveats:

• Inline equations must be closed on the line they are opened.
• Expressions such as "$i$th" must be written as "$i$#th#", since AsciiDoc "+++" quotes are constrained.

I'm content with this setup, but I'm considering extending my script to detect equations automatically, a recent feature of ASCIIMathML, so that even the dollar signs are unnecessary.

Memory Mapped Files

I've only recently learned about memory-mapped files in Linux. Until recently, to perform file I/O, I followed the textbook examples of using read and write calls on file descriptors or streams. It wasn't until I browsed the git source (to help with my git clone) that I came across a mysterious mmap() call.

Not only is it more efficient for the reasons given in the Wikipedia article, but it is easier to work with file contents as if they were already in memory. There's no need to learn the syntax and semantics of various read and write calls.

The only tough part is remembering the mmap syntax. But the call can be readily encapsulated. For example, to read a file, I define a function like the following:

void *map_file_to_memory(size_t *len, const char *path) {
  int fd = open_or_die(path, O_RDONLY);
  *len = get_size_of(fd);
  if (!*len) {
    close(fd);
    return NULL;
  }
  void *map = mmap(NULL, *len, PROT_READ, MAP_PRIVATE, fd, 0);
  if (MAP_FAILED == map) die("mmap failed");
  close(fd);
  return map;
}

There are concerns with large files, but hopefully madvise() is well-written and can prevent excessive page faults.