Planet LCA2010

Drupal Down Under

2010-09-02T04:41:42+00:00

Applied:

Monday, 16 August 2010

Granted:

Wednesday, 1 September 2010

Grantee:

Marmalade Soul

Original:

http://lists.linux.org.au/pipermail/linux-aus/2010-August/018210.html

Dear Linux Australia Council,

We're planning a National Drupal conference called DrupalDownunder. We're backing off LCA 2011 as per discussions with Shaun Nykvist — with Donna Benjamin being our liaison between both parties.

We're looking at 200 delegates to attend and we're currently in discussions with several well-known Drupal speakers enticing them to submit proposals for LCA and in turn DDU (DrupalDownunder).

We'd love to be a subcommittee of LA, as has been the recommended course of action.

Explaining MVCC in Postgres: system defined columns

2010-09-01T15:00:18+00:00

I’m playing around with some diagrams for explaining MVCC that I’ll be posting here over the next few days. Not sure if I’ll end up giving up on slides and just use a whiteboard for the talk. I made an illustrated shared buffers deck to go along with Greg Smith’s excellent talk on shared buffers a while back. This is the beginning of a talk that I hope will emulate that.

Here are my first few slides, showing the system-defined columns. The next few slides will describe optimizations PostgreSQL has for managing the side effects of our pessimistic rollback strategy, and reducing IO during vacuuming and index updates.

New Google Maps Imagery Of Auckland

2010-09-01T10:47:19+00:00

At some point relatively recently, Google added much higher-resolution images of Auckland to Google Maps. These images appear to have been taken by aerial photography and they're quite amazing.

Best (Scott Pilgrim) meta film review ever

2010-09-01T06:39:10+00:00

Here’s what I’m saying: I’m a woman, I’m in my late thirties, I can’t handle first-person shooters, I’m afraid of Comic-Con, and I really, really liked Scott Pilgrim vs. The World.

I hope I’m not, you know, blowing your mind.

– Linda Holmes in her incredibly cool meta-film-review, ‘Scott Pilgrim’ Versus The Unfortunate Tendency To Review The Audience

Code Quality

2010-08-31T15:01:17+00:00

Morten, it’s actually my fault.

the patch that was submitted was the incorrect one; I’ve been fixing the implementation since then, mostly because the API is correct — and went through different eyeballs to ensure that — and we wanted to land it before the GLib freeze. the implementation, alas, has issues — and not just the ones you saw.

as I said, it’s definitely my fault: I reviewed the original in a way that was less than acceptable. I sincerely apologize for that.

okay, having said that, and resuming my usual self: bitching on planet GNOME is not going to get anything fixed. insulting the maintainer when the issue lies mostly in a “patch lieutenant” (such as myself) is also not how polite people deal with errors.

finally, saying that the code needs to be taken out back is a lame cop-out and it is an untenable position for anyone contributing. the code went in, there were issues, so it needs to be fixed. if we waited for perfect code the amount of merged we’d get would be zero. that code has been in the bug for the past 6 months, and it’s been discussed in the past IRC meetings, with public minutes and logs.

so, please: file bugs. I’ll try to close them faster than you can open them.

Using the Grid plugin in Compiz

2010-08-31T09:09:51+00:00

Recently I’ve been introduced to the Grid plugin in Compiz – this is a fantastic little plugin that snaps your windows to predefined positions on the screen. After using it for the last couple of weeks I can’t imagine going back to not using it.

To get access to it you will need to install the compiz settings manager, and load Compiz’s “Fusion Plugins Extra”:

sudo apt-get install \
    compizconfig-settings-manager \
    compiz-fusion-plugins-extra

Once you’ve done this simply enable it:

System, Preferences, CompizConfig Settings Manager
In the Filter box in the top left, type in “grid”
Tick the box to enable it and click “Close” (bottom left)

Now try using Ctrl-Alt and numbers on your keypad. CA-9 will snap a window into the top right corner (press it a few more times to get different sizes) and CA-8 will make it snap to the top half of the screen (the number is the position on the screen, you’ll get the gist of it).

Check out this demo video:

Happy Ubuntuing!

Going Solo

2010-08-31T08:09:06+00:00

I'm going solo, working for myself at Horms Solutions, a boutique free and open source software consultancy.

The Naked and Famous

2010-08-31T05:46:12+00:00

If your eclectic tastes include legendary modern music-as-art heroes such as Radiohead, MGMT, Florence and the Machine, Nine Inch Nails and How to Destroy Angels, you simply must check out upcoming New Zealand band The Naked and Famous.

Their debut album will be released in early September, but they’ve put out some mind-blowingly good singles already… here’s Young Blood:

Update: Okay, I can’t just link the one. Here are my other favourites so far…

rbd (rados block device) status

2010-08-30T19:44:59+00:00

The rados block device (rbd) is looking pretty good at this point. The basic feature set:

network block device backed by objects in the Ceph distributed object store (rados)
thinly provisioned
image resizing
image export/import/copy/rename
read-only snapshots
revert to snapshot
Linux and qemu/kvm clients

Main items on the the to-do list:

TRIM
image layering/copy-on-write
locking

The server side components are in place in both the v0.21 releases and the unstable branch. On the client side, there are two options.

First, qemu/kvm can be patched to map an rbd image as a block device. The code is available in git from

git://ceph.newdream.net/git/qemu-kvm.git (branch rbd)

Alternatively, Wido has built some patched Ubuntu 10.4 packages for both qemu and libvirt, available from

deb http://pcx.apt-get.eu/ubuntu lucid unofficial

The qemu/kvm patches will likely be included in the next major qemu release.

The native Linux kernel rbd kernel driver is also quite stable, but did not make it upstream for the 2.6.36 release cycle. We hope to have it in 2.6.37. The code for that is available at

git://ceph.newdream.net/git/ceph-client.git (branch rbd)

The main hold up there is that the addition of rbd involves refactoring a lot of the common Ceph file system client code into a libceph module that is shared by both rbd and the file system client. This makes rebasing more difficult, so that branch may not have the most recent fixes in the master branch or the current -rc kernels. Also, the code reorganization completely breaks my semi-automated ceph-client-standalone.git updates, so for now you can’t clone and build it as a standalone module.

For more information, see the rbd and kvm-rbd wiki pages.

More web site revisions

2010-08-30T19:17:00+00:00

I've made some more revisions to the book web site. The changes include include:

Slight tweaks to the structure of the source code tarballs (thanks to Jens Thoms Toerring).
A new page with pointers to slides from various conference presentations I've made in the last few years.

Meta-contacts in Empathy (and beyond!)

2010-08-30T17:34:49+00:00

We've finally closed Gnome Bug #460647 (Empathy: Support metacontacts)!

Let's get right to the pictures:

This is our "Link Contacts" dialog. Selecting a number of contacts on the left side designates which contacts will be included in the Individual (read: meta-contact) at right.

We currently auto-link contacts with the same service ID (foo@example.org) contained in multiple accounts. In the future, we're going to push as much into auto-linking as possible. My goal is that most users would never need to use this dialog. Until then, this dialog is how you designate meta-contacts in Empathy.

If you accidentally link together contacts you didn't mean to, you can simply open the Individual's info dialog and hit Unlink to start over.

One of the first benefits of meta-contacts is pulling together multiple modes of contact for a person into a single place. So if you've got a friend with separate work and personal IM, you can be an upstanding citizen and only send funny cat pictures to their personal IM address. (If they sign on to both at work, at least you tried your best).

We've kept the 1-click Chat/Audio Call/Send File/etc. actions for the (I think, more-common) case that you just want to communicate and you don't care about the specifics. This just uses any valid (ie, online) account for that Individual.

Thanks

This meta-contacts functionality is all possible thanks to the Folks project. It's a collection of libraries (so far, all distributed together) that collect contact information through various backends (Telepathy so far; many more to come) and aggregate Individuals (ie, People). Empathy was our starting point, but we really plan to integrate very well into the Gnome desktop and beyond. Think: typing -Alice- to start a chat with Alice through the Gnome Shell.

I'll discuss Folks in greater detail in future posts.

I'd like to thank my colleagues Philip "code factory" Withnall for helping with Folks development/Empathy integration and Empathy maintainer Guillaume Desmottes for letting us know exactly what we broke in each commit. And, of course, a special thanks to our employer, Collabora, for sponsoring us to work on this full-time.

Try it now!

Since you're no doubt very excited about these new features in Empathy, you can use them Right Now™ in version 2.31.91 (also coming to Debian experimental shortly). Be sure to file any and all bugs you run into, so this will be as solid a stable release as any other!

If software bugs have you leaping onto the nearest elevated furniture, no problem -- Empathy will, of course, by included in Gnome 2.32, coming to a friendly Linux distribution near you!

Variable substitution with psql

2010-08-30T16:00:50+00:00

Updated: Thanks @johto for s/:bar/:foo/.

A coworker asked about variable substitution with psql using \set, and so I looked into it a bit further.

You definitely can do things like this:
16:55 sdeckelmann@[local]:5432|postgres=> \set test 'select * from :foo limit 10;' 16:56 sdeckelmann@[local]:5432|postgres=> \set foo 'test' 16:56 sdeckelmann@[local]:5432|postgres=> :test myint ------- 1 2 3 4 5 6 7 8 9 10 (10 rows)

But, what about something like this:

=> \set test 'select * from :var limit 10;' => :test mytable

Unfortunately, this isn’t supported.

The best you could do is something pathological like:

=> \set s 'select * from ' => \set pr ' limit 10;' => :s mytable :pr => :s test :pr myint ------- 1 2 3 4 5 6 7 8 9 10 (10 rows)

A Night Out

2010-08-30T12:04:23+00:00

I had a very nice dinner at "Wildfire Churrascaria" courtesy of Microsoft. Lots of yummy, fatty, salty meat.

I went to catch my bus and missed it by one minute. At 10:41pm, the next bus was at 11:10pm.

Being impatient, instead of waiting for the next bus I ran home ... well, half-ran, half-walked ... 5.5km In 42 minutes. Not fast, but then I had a backpack full of Microsoft swag plus my laptop, and a belly full of meat. Faster than the bus anyway.

Cc and Bcc field size in Gmail

2010-08-30T10:13:34+00:00

The Cc and Bcc field sizes in Gmail are too small to work with if you’re sending to more than about 20 email addresses. The following bookmarklet will increase their height when used:

javascript:(function(){var%20e=window.frames[3].document.getElementsByTagName('textarea');%20for(i=0;i&lt;e.length;i++){%20if(e[i].name=='cc'%20of%20e[i].name=='bcc')%20{%20e[i].style.height='150px';%20}%20};})();

To install (in Firefox).

Right click on your Bookmarks toolbar and choose “New Bookmark”
Enter a name for the Bookmark, e.g. “Big Cc Field”
Into the location, paste the above code
Save the bookmark

Now when at Gmail and you can see the Cc or Bcc fields you can click the new Bookmark link to make them bigger.

To find out more about Bookmarklets, check out the Bookmarklet page on Wikipedia.

TechEd

2010-08-30T07:08:05+00:00

I'm at Microsoft TechEd in Auckland today and tomorrow --- they reached out to invite a few open source people, so I thought I'd go and check out their messages. It's fun too. One of the highlights of today was a talk about geolocation APIs ... the presenter showed the creation of a toy HTML geolocation app, but couldn't get it working in Chrome and had to switch to Firefox :-). And apparently IE9 isn't going to support the geo APIs at all as far as anyone knows. Funny sort of demo for a Microsoft conference!

In fact so far I haven't seen anything about IE9 at all. There's a lot more Silverlight/WIndows Phone 7. I wonder if that's accidental (this is just the first day after all) ... or not.

Taxonomy in Drupal

2010-08-30T04:21:32+00:00

The taxonomy module is a very useful utility for Drupal sites, especially for those of any size. Primarily summarised from the Drupal Handbook on the subject, the following gives a basic summary of the subject.

Installing Molden on a 64-bit AMD Opteron Cluster running CentOS 5 Linux

2010-08-30T00:51:19+00:00

Molden displays molecular density from the ab initio packages GAMESS and GAUSSIAN and others. Molden reads all the required information from the GAMESS / GAUSSIAN outputfile. Molden is capable of displaying molecular orbitals, electron density and molecular minus atomic density.

Using logger with pg_standby

2010-08-29T16:00:26+00:00

Piping logs to syslog is pretty useful for automating log rotation and forwarding lots of different logs to a central log server.

To that end, the command-line utility ‘logger’ is nice for piping output from utilities like pg_standby without having to add syslogging code to the utility itself. Another thing is that logger comes by default with modern packages of syslog.

Here’s an easy way to implement this:

restore_command = 'pg_standby -d -s 2 -t /pgdata/trigger /shared/wal_archive/ %f %p %r 2>&1 | logger -p local3.info -t pgstandby'

Online aggregation paper from 1997 and PSU’s database reading group

2010-08-28T23:03:36+00:00

A couple weeks ago, Mark Wong and I took a field trip over to the Database Reading Group at Portland State University. It’s a group of students and professors that meet weekly throughout the school year to go over research papers. The papers are picked by the participants, and vary in topic from obscure to very practical.

This week’s paper reading was led by Professor Len Shapiro, and titled “Online Aggregation“. The paper is considered a foundational paper about SQL aggregates (like COUNT() or AVERAGE), and was published in 1997 by researchers from UC Berkeley and IBM. It’s also precursor to research into query parallelization and streaming databases. It was also awarded the SIGMOD “Test of Time” award in 2007, and is cited by over 170 other papers in the ACM archive.

The basic idea behind the paper centered around how to improve user experience in reporting results of aggregate queries – asking questions about how to solve three key problems when solving aggregates: blocking, fairness and control (from a user’s perspective). Roughly: Blocking is what happens when some part of the system waits and doesn’t return results to the user as a result of the waiting. Fairness concerns whether certain types of operations prevent certain groups of data from being processed (the example given had to do with GROUP BY and groups being processed one at a time). Control concerns whether or not a user can exert control over the speed of computation applied to a group (example given being a lever that “speeds up” processing of a set).

One insight from the paper is how online aggregates should be treated differently than traditional query processing – which might favor expensive plans involving sorts so that the output is ordered. When you’re dealing with online aggregates, you prefer unordered, or ideally random order, because your intermediate results will be more representative of the ultimate result. I guess that’s probably obvious once you think about it, but the paper provided some concrete examples.

Another interesting thought experiment involving the planner is how you pick plans that favor non-blocking, fairness and user control. Each of those properties is not narrowly defined, and changes based on individual user expectation. Professor Kristen Tufte mentioned that she’d be interested in how the ideas presented in this paper would be applied today, and Professor David Meier brought up that we might most be interested in applications involving managing Hadoop.

Prof Meier also brought up an interesting paper involving alternating nested loop joins during a discussion about optimizing JOIN algorithms for online aggregates. Another cool thing about the paper is that it involved modifications to Postgres! Granted, it was Postgres95, which doesn’t resemble the modern PostgreSQL 9.0 very much. But it was nice to revisit research that used Postgres that’s still relevant today.

v0.21.2 released

2010-08-28T16:49:28+00:00

This is a second bugfix release for the v0.21 series. Changes include:

osd: less log noise
osd: mark down old heartbeat peers
filejournal: clean up init sequence, less confusing errors on startup
msgr: fix throttler leak (fixes deadlock)
osdmaptool: don’t crash on corrupt input
mds: error to client on invalid opcode
mds: fix ENOTEMPTY checks on rmdir
osd: fix race between reads and cloned objects
auth: fix keyring search path when $HOME not defined
client: fix xattr writeback
client: fix snap vs metadata writeback
osd: fix journal, btrfs throttling
msgr: fix memory leak on closed connections

Relevant URLs:

Direct download at http://ceph.newdream.net/download/ceph-0.21.2.tar.gz
For Debian packages see http://ceph.newdream.net/wiki/Debian

Walk the line

2010-08-28T16:08:06+00:00

I’ve noticed that I haven’t blogged about JSON-GLib in a while.

since the last time, I release version 0.10, which had a lot of fixes in the JSON ⬌ GObject transformation code, thanks to the contributions of Tristan Van Berkom. the 0.10 release allowed to transform all GObject properties of fundamental types — and to register transformation functions for boxed types as well. it also had bugs fixes coming from Clutter and other projects using JSON-GLib.

now we’re near the 0.12 release — I released the first release candidate on August 2nd, and I plan to do another release candidate along with the GNOME 2.31.90 snapshot.

for this cycle I’ve had the contribution of Luca Bruno, who wrote a JsonBuilder class which provides a simple API for building JSON trees. JsonBuilder has been inspired by JSONWriter and by GVariantBuilder:

JsonBuilder *builder = json_builder_new ();

json_builder_begin_object (builder);

json_builder_set_member_name (builder, "url");
json_builder_add_string_value (builder, "http://www.gnome.org/img/flash/two-thirty.png");

json_builder_set_member_name (builder, "size");
json_builder_begin_array (builder);
json_builder_add_int_value (builder, 652);
json_builder_add_int_value (builder, 242);
json_builder_end_array (builder);

json_builder_end_object (builder);

JsonGenerator *gen = json_generator_new ();
json_generator_set_root (json_builder_get_root (builder));
char *str = json_generator_to_data (generator);

g_object_unref (generator);
g_object_unref (builder);

/* "str" now contains the string:
 * { "url" : "http://www.gnome.org/img/flash/two-thirty.png", "size" : [ 652, 242 ] }
 */

in the same spirit, I had a fun two hours hacking session that resulted in an XmlReader-like API for JSON trees, called JsonReader:

/* str contains the JSON from the example above */
JsonParser *parser = json_parser_new ();
json_parser_load_from_data (parser, str, -1, NULL);

JsonReader *reader = json_reader_new (json_parser_get_root (parser));

json_reader_read_member (reader, "url");
const char *url = json_reader_get_string_value (reader);
json_reader_end_member (reader);

json_reader_read_member (reader, "size");

json_reader_read_element (reader, 0);
int width = json_reader_get_int_value (reader);
json_reader_end_element (reader);

json_reader_read_element (reader, 1);
int height = json_reader_get_int_value (reader);
json_reader_end_element (reader);

json_reader_end_member (reader);

g_object_unref (reader);
g_object_unref (parser);

I also added support for parsing (synchronously and asynchronously) JSON from a GInputStream and generating stringified JSON to a GOutputStream (in this case, only synchronously). this means that JSON-GLib now depends on GIO.

for the next cycle, which is going to be fairly relaxed in terms of duration, I’m planning some new features like support for the JSON Schema draft in the JsonParser class; and a sensible implementation of a Path API for direct access of members.

if you have bug reports, feature requests, or code you want to contribute, feel free to hop in Bugzilla and file a new ticket.

have fun!

More Dell Fail (Or Maybe NVidia)

2010-08-28T10:10:16+00:00

I bought a Dell ST2410 monitor for my new home computer (a Dell XPS 8100 shipped with an NVidia GTX260 graphics card, which has two DVI ports). Dell shipped a VGA cable and a DVI-VGA converter, so I thought I'd go out and buy a digital connector. Dell shipped me a DVI-HDMI converter dongle (made by NVidia apparently), the Dell monitor has HDMI and my TV has HDMI so I thought I'd get an HDMI cable and this would be easy. Wrong. The monitor completely fails to detect any signal from the computer. I tried everything I can think of. Even my old Macbook Pro can drive the monitor through its DVI port, the DVI-HDMI dongle and the HDMI cable. So apart from the cable, which obviously works, we have here three parts all shipped by Dell that don't work together. Sigh. I guess I'll try a DVI cable next...

Sunshine Rises Again

2010-08-28T07:20:54+00:00

As previously reported, the wonderful "Sunshine" Chinese restaurant in Market Place near the Viaduct Harbour suffered a tragic demise. And also as previously reported, it has been reincarnated. Today our family visited the new incarnation, "Crystal Harbour". I am very pleased to report that the new version is very similar indeed to the old "Sunshine". The decor is the same, the layout is the same, the lack of queues is the same, and most importantly the food is very much as it was. Where Sunshine excelled --- the unique barbeque pork buns, the seaweed plate, the ice cream dumplings --- so does Crystal Harbour. Crystal Harbour's promotional material claims there's a new chef, but clearly (and fortunately) a lot has been preserved. One change is that there were a lot more people there today than I ever saw at Sunshine. It could be the novelty factor, but I hope Crystal Harbour does well. I certainly plan to contribute as often as I can!

Returning to normal life

2010-08-27T04:56:40+00:00

Now that production of the book is done, life has started returning somewhat more to normal. For the first time in well over a year, I found time to read someone else's book: Antony Beevor's excellent D-Day: The Battle for Normandy. Another reminder that the USA once helped save Europe from itself seems timely.

Free linux.conf.au swag for Software Freedom Day

2010-08-25T14:27:16+00:00

Hi All,

Over the years Linux Australia has collected a number of swag items
that have become left over from conferences and events. To help combat
this problem of where to store it and what to do with it Linux
Australia would like to help supply registered Software Freedom Day[1]
teams with extra goodies to give away and help promote free software!

Since SFD is less than a month away we need responses as quickly as
possible! If you are running a free software promotion type event and
would like some extra swag to give away please email the Linux

iOS4 Background Audio Revisited

2010-08-25T11:08:44+00:00

The other day I went ahead and revisited streaming audio apps for iOS4. I blogged previously about how it is the biggest pain in the ass to get working….properly. And it still is. Background buffering was a pain to do, but I finally worked out how to apply it to AudioStreamer (a github project by mattgallagher). The solution in this StackOverflow post is incredibly useful.

So, I decided I’d checkout AudioStreamer from git, take a look at other people’s forks, try to merge all the fixes and features then add my own. That’s what I did the other day.

My current github fork of AudioStreamer is here: http://github.com/DigitalDJ/AudioStreamer (and it’s pretty damn stable, just sayin’).

Basically, my fork cherry-picks some commits from other users, notably:

- Shoutcast metadata [jfricker]
- MIME type detection [andybee]
- HE-AACv2 [idevsoftware]
- Level Metering [idevsoftware]
- NSThread memory leak [mattgallagher]

Then I set out to try and fix the bugs I had encountered…and fix up the sample app so that it worked fully with iOS4 multitasking:

- Fixed interruption crashes (crashes when you use iPod app then re-use the sample app, or receive a phone call)
- Background buffering (!!! AudioStreamer now buffers properly in the background)
- Play/pause from iPod controls (The sample app now works properly with the iPod controls)
- Stop all UI updating and timers while backgrounded (The sample app now correctly follows guidelines to stop all UI updating and unnecessary timers to save battery life)

- Retina display example (The sample app now has a retina display example…just to be iPhone 4 compatible)

- Support for Pause button in UI (Pausing now works properly, even for streams that don’t support it)
- Local Notifications (outside app) on error (Notification popup if there is a lack of connectivity when playing music outside the application)

There’s just one more bug for me to squash, and frankly I can’t figure it out. Notifications are sent from AudioStreamer to the UI in case of an error, the UI then handles the notification with a UIAlertView. Sometimes the alerts refuse to show up, or instead, multiple show up, or…..sometimes it just works as intended (i.e. one alert popup). I’m not sure if it’s an iOS bug or something in my code. It’s rather odd as it isn’t easily reproducible and often a reboot of the device will fix the issue. Very very odd.

A Possible Fix: Xen HVM & Windows 2008

2010-08-25T11:01:51+00:00

Recently, I switched my VPS from VMware ESX to a provider with Xen HVM since my original host disappeared off the face of the planet. However, the Xen HVM VPS has decided to randomly BSOD every 24-48 hours or so. Here’s how I troubleshooted and possibly fixed the issue.

The Issue:
BSODs randomly occur with the error “A clock interrupt was not received on a secondary processor within the allocated time interval.” and STOP code 0×00000101.

Caused by:
Well, the exact issue is unknown. I have a feeling that this occurs when the host has a higher than usual load, which causes some sort of delay of the RTC and eventual crashes. Examining 3 crash dumps showed that the Realtek networking driver (Rtnic64.sys) is always near the top of the stack trace with something to do with the RTC in the kernel.

The solution:
There are a few solutions. I’m not sure which one fixes the issue exactly, but here’s what you can try:

First try modifying your VM config to add these options (the first ensures time is always “wall clock time” and the second exposes the Hyper-V interface to the VM):
timer_mode = 2 viridian = 1

You can then also try
os_variant = "vista"

And if worse comes to worse, limit to 1 CPU core
vcpus = 1

Finally, if you run Windows you should install the GPL PV drivers. If you want to run Windows, you MUST use Xen HVM (Xen PV is unavailable). The default drivers when using HVM are designed to work with QEMU’s emulated devices and therefore carry a performance degrading overhead. Luckily, there’s been an effort to create PV drivers for Windows, so the VM can work more directly with the host’s hardware, creating less overheads and a significant performance boost.

Download the latest drivers (32 or 64bit, 2003/2008) from: http://www.meadowcourt.org/downloads/

If you run a 64-bit version of Windows later than 2003 (i.e. Vista, 7, 2008, 2008 R2) you need to disable driver signature integrity checks. You can also create a secondary boot option to disable the PV drivers and revert back to the less optimized drivers. For example:

Disable Driver Integrity Checks
bcdedit /set loadoptions DISABLE_INTEGRITY_CHECKS bcdedit /set testsigning ON bcdedit /set nointegritychecks ON

Create secondary boot option
bcdedit /enum /v Note the GUID string next to "identifier" under "Windows Boot Loader" bcdedit /copy {string from above} /d "Windows Server 2008 NOGPLPV" Description can be whatever you want bcdedit /set {new GUID from above} LOADOPTIONS "NOGPLPV" Use the GUID outputted from the 2nd command

Now install the MSI, restart, open up Device Manager and you should find you have no more “Unknown Devices” and that you now have a different network card (among other things) installed.

Cloudy Horizons

2010-08-25T05:32:20+00:00

I was asked to write this article for an IT publication a month ago but missed the publishing deadline. Due to the wonders of modern technology the world is still able to share and acclaim my wise words:

Photo Google Schwag (CC) Attribution-ShareAlike 3.0 Some rights reserved

The Cloud application of the moment is Facebook. Facebook represents the web within the web. It is a privately owned world that is more populated than most countries. It allows individuals and businesses to create their own identities, manage their contacts, find new contacts, conduct business and develop applications specific to Facebook’s technologies.

It seems everyone is “on” Facebook, whether you are a government department, Telecommunications business or simply someone with funny cat photos to share.

But there is a cloud on the horizon. The Bloomburg Business Week is reporting that Facebook founder, Mark Zuckerberg, may have signed a contract which if upheld, would see the company transfer ownership (http://www.businessweek.com/news/2010-07-20/facebook-lawyer-unsure-zuckerberg-signed-contract.html). In addition to its ownership uncertainty, Facebook has a nasty habit of making unilateral changes to the usage policies, effectively the contract it has with each Facebook user. This is a risky proposition for businesses relying on Facebook for their future, and for individuals putting significant parts of their life on one platform.

While I was in the UK recently I heard about a musical instrument supplier. They have sold their goods through Amazon for many years, with Amazon taking about a 7 percent commission. That week Amazon announced that it was getting into the musical instrument vending business itself. Co-incidentally Amazon was also unilaterally doubling the commission they were charging to third party music instrument vendors.

So what do these anecdotes mean for people who have invested everything in “the cloud”. The people who rely on Facebook, or Google apps, or Amazon, to be the “internet of everything”, for their livelihood?

How do you shift from one vendor to another? If their infrastructure is cutting out, or the terms of service becoming more and more unfavourable, or your ISP is favouring a competitor’s traffic, or your clients are moving from one application to another (Bebo anyone?) how do you move? How do you get your data and applications transferred? Is the data you created even yours in the first place?

The success of “the cloud” is also its massive failing – scale. The scale of the cloud has pushed down costs and increased convenience significantly. But, as Google’s Vint Cerf points out, there are no accepted standards or protocols for cloud services and systems to store and exchange information and systems.

With the huge scale that the internet has enabled comes an equally huge imbalance in the nature of the relationship between cloud service vendors and users. The magnitude of the lock-in that users of cloud applications find themselves committed to far outweighs anything that has preceded the current phenomenon.

So, what do we do? If the convenience of cloud services is impossible to ignore then the pitfalls and potential for “all of business” disasters are should be evaluated and mitigated.

To start with, your data should be available to you on a device of your choosing at any time and in an open format that can be easily recognised by different software systems. This means that you should be able to backup and download your data easily and at regular intervals.

Secondly, you should easily be able to transfer from one platform to another. The best way of achieving this is to ensure that whatever cloud service you are using is based on software that is free and open source. This is easier than you might imagine. For every Google app, Twitter, Amazon Web Service, there is an open source alternative. Most, if not all, of these also run as cloud services. Examples include Wordpress, Status.net, Teambox, RedMine, FengOffice and WikiMedia. Using open source cloud services ensures that transferring from one provider to another is not just possible, but straightforward. It also guarantees that your data can be processed if you decide you have to make the shift.

To conclude, the advantages of cloud services have been well sold, I would say oversold. They represent a privatisation of what we used to call “The Internet”. The pitfalls are less well expressed but they exist.

The trick for businesses and government in particular is to ensure that they avoid becoming enslaved on a scale that has never previously been possible. Fortunately free and open source software proves, yet again, to be an effective way of getting the best of all worlds – access to high quality technology and services without the proprietary capture and other business risks that are a feature of many cloud services.

-moz-element Landed

2010-08-25T02:43:27+00:00

Markus Stange picked up the work I did in 2008 on the "-moz-element()" CSS extension (which was later extended by Ryo Kawaguchi), made some major improvements and got it reviewed and landed. Check out his blog post. -moz-element lets you render the contents of an element as the background image for any other element. This is a very powerful tool that can be used in very interesting ways; check out Markus' amazing demos. This feature is on trunk now and will be in Firefox 4. We will also propose this to the CSS WG.

Note for browser UI and extension authors: eventually -moz-element will be the preferred way to render "live" copies of Web page contents (insteading of using MozAfterPaint/drawWindow). Right now, -moz-element can be used to render the contents of a <browser> element elsewhere, although it's less well-tested and is less tweakable for performance. Post-FF4, we can tie -moz-element into the layers framework so that in many cases --- such as tab thumbnails --- rendering -moz-element just recomposites a layer subtree, fully GPU-accelerated.

Vinge

2010-08-23T21:17:48+00:00

The first Vernor Vinge books I read were A Fire Upon The Deep and A Deepness In The Sky ... not surprising, since they're the most famous, and also the best. I was a bit disappointed by Rainbows End. Just recently I read two of his earlier novels (written in the 80s), The Peace War and Marooned In Realtime --- the latter being a sequel to the former --- and I think they're excellent, perhaps not as good as Fire and Deepness, but Marooned in particular I found more intriguing and even quite moving.

Warning: if you haven't read these books, go out now and read them before you come back to the rest of this post, because spoilers are ahead...

Marooned and Rainbows End present two rather different visions of human development, and Rainbows End is far closer to my own thinking even though I like the book less. I'm ignoring the bobbles here --- they're a wonderful plot device, but I think the real themes of Marooned are the technological Singularity and a yearning for anarcho-capitalism. At heart I think Marooned is fundamentally an optimistic view of human progress to the Singularity. Rainbows End, on the other hand, seems to me to be a much darker view, a view of humanity lurching from one potential planet-killing catastrophe to the next at decreasing intervals, with no Singularity-salvation in sight. Now, Vinge may make a liar of me yet, since he's said he'd like to do a sequel to Rainbows End, but based on what he's written so far I guess in the twenty years between the books he's become more pessimistic. Although curiously, he may have become a lot more optimistic about governments --- in Marooned he hates governments, in Rainbows End we see a benign totalitarian state.

Personally I think Rainbows End is too optimistic :-). I wrote about this a while ago and I stand by it: the technology that could eventually lead to some kind of Singularity (very eventually; this stuff is way, way harder than most techo-futurists imagine), leads much sooner to either the total elimination of cognitive freedom or the destruction of all intelligent life. It's just not realistic to imagine we can walk the ever-thinning razor's edge for long. Man is fallen, but he still has a long way to fall. God is going to have to save us from ourselves, again.

Time to stop. Excessive futurist navel-gazing is definitely a sin :-).

Brisbane Pics

2010-08-22T07:12:57+00:00

The Great White Out The ferry service was canceled till about lunch time, this placed a huge stress on the busses picking up the slack on the ferry services.. Botanical Gardens A few pics from the botanical gardens next to the Queensland University of Technology. On the river Rock Climbing Kangaroo Point [...]

The other half of the hottest 100 songs

2010-08-21T23:59:52+00:00

As many people have pointed out, the recent Triple-J Hottest 100 Songs of All Time had a baffling bias. Fortunately, Naomi Eve is proactively restoring the balance, with a Hottest 100 Songs by Women poll and list. My list of favorite women artis...

Read and post comments | Send to a friend

Proposing a mentoring partnership framework for Ubuntu Women

2010-08-21T14:11:57+00:00

Back in December while starting to scurry about to pull together the competitions framework which I was hoping would make a few splashes, I was already thinking ahead to after the competitions, thinking of ways to kick the mentoring programme in to gear through which the target audience attention could be catalysed to participation.

A few weeks ago, as Alan Bell was poking me wondering about the code we’d used for the competition voting polls, it dawned on me. The result of that epiphany is my new pet project, Pollka, a php project that aims to build a quickly deployable, standalone, simple polling web application.

Pollka is not part of Ubuntu, and is not an Ubuntu project. It is however a real project which intends to partner with the Ubuntu Women Project to mentor interested members in the purposeful use of tools such as bzr, launchpad tools and packaging, and concepts such as reporting bugs and documenting. The aim being to give prospective women contributors a chance to gain confidence in methodologies that will ease their integration path in to activities that are considered “contributions to Ubuntu”.

I believe this real world project sandpit concept is a missing link in the current UW mentoring structure. Jumping straight in to a massive project with commercial responsibilities like Ubuntu is daunting, especially so for inexperienced people who feel an extra burden in proving themselves. I want UW to be able to help interested women break out of that self-fulfilling prophecy cycle.

I am proposing that a mentoring partnership framework be established to compliment the current UW mentoring pathways, and I intend to use Pollka as a proof of concept partner project. I’ve added the blueprint to the next UW meeting agenda and hopefully we can use this as a deliverable for the Natty cycle.

OpenOffice.org is the most frustrating piece of software I use

2010-08-21T01:41:03+00:00

No, really.

I have recently been constructing a 100 page document going over a whole bunch of the details for the Monorail we’re building at Burning Man this year.

Apart from randomly freezing, and then suddenly not displaying images until I had restarted it – it’s also really slow.

The last straw was when leafing through the document before getting it printed. I had inserted a bunch of pages before this last section. But now, there was this empty page in the last section of the document. The part that I hadn’t touched for days. If I tried to remove the blank page, all the images on nearby pages moved so that they were on top of each other.

I ended up just printing it. There is a blank page that I can’t get rid of.

It is a piece of software that worries me. Is this really meant to be an alternative? It has NEVER worked well for me. Basic tasks sure, but I continually find myself pining for Word 5.1a on the Mac (System 7 that is) or Nisus Writer or even ClarisWorks.

If opening Microsoft Word documents fairly accurately is your only good feature, how do you expect to survive in the free (software) world?

So, while my twitter stream may suggest desires for punning the developers in the face or their early demise through painful methods….. I really just wish that sometime in the past 10 years you had made it not shit me to tears.

Certainly another failure of Sun Microsystems and I don’t expect Oracle to do any better at all (especially considering recent actions).

Source code tarball now available

2010-08-20T22:55:00+00:00

I've made some more revisions to the web site of my book. The most notable change is that you can now download the source code for the sample programs. (It's also possible to browse individual program files online here.)

The source code tarball is provided in two versions:

Distribution version: a version of the source code that includes extra material not shown in the book. Probably, this is the version of the code that you want.
Book version: the source code as it appears in the book.

The web site explains why there are two versions.

I'd like to hear of any problems that you encounter building (either version of) the code on various Linux systems, especially older distributions. I'll use feedback to build out the source code FAQ.

SHOW CREATE TABLE using table protobuf message

2010-08-20T03:37:45+00:00

… and really testing the replication code path for CREATE TABLE.

So, for a very long time now, Drizzle has been using a protobuf based structure to describe the structure of tables. The idea was to be able to have engines rather painlessly generate this structure themselves (which several now do). A secondary idea was to use this structure itself for CREATE TABLE (in progress, and embedded_innodb does in fact does only use the table message for its CREATE TABLE codepath). The third idea was to just put the table protobuf message into the replication stream instead of the CREATE TABLE statement (i.e. a SQL string). This means that you could (easily) write a replicator to a DBMS with different SQL syntax, or to a system that doesn’t speak SQL at all.

The final step, to reduce duplicated code functionality, would be to have the code that does SHOW CREATE TABLE to use a common bit of code for turning a table protobuf message back into a SQL string.

We now have that.

Just now, my branch to replace the old SHOW CREATE TABLE code (that was using TableShare and friends) with the statement_transform code (that we also use in converting a replication log to SQL commands) hit trunk.

Yay.

CSS Units Changes Landed

2010-08-19T23:36:27+00:00

The CSS units changes that I blogged about in January have landed and will be in the next Firefox milestone. With these changes, 1in = 96px always. Likewise 3pt = 4px, 25.4mm = 96px, etc. This matches the behaviour of Internet Explorer, Safari and Chrome.

By default, when printing, 1in is rendered as one physical inch. For other output media, all these units are scaled in a medium-dependent and platform-dependent way by default. One goal of this scaling is to give results consistent with user expectations and other applications on the system. For example, standard form controls such as checkboxes should look the same in Web pages as in other applications, by default. Another goal is to choose default scaling so that a document designed to print well on normal-sized paper will be readable on the output device, e.g., a phone. So, the advice for authors using CSS physical units is to set lengths so the document looks good when printed without scaling; the browser will then scale those lengths to display the document suitably on different kinds of screens.

There are some rare cases where it makes sense to include true physical measurements in a Web document --- for example, "life size" diagrams, or elements in a touch interface. For these cases we have introduced a new experimental unit, "mozmm". For media such as screens that can be touched, 1mozmm is rendered as one physical millimetre (or as close as we can get based on what we know about the medium). For other media, such as contact lens displays, brain-implanted electrodes, or lasers projecting into the sky, we reserve the right to treat 'mozmm' similarly to 'mm'. Authors should only use mozmm for elements which really need the same physical size on, for example, a 4" phone screen and a 24" monitor. This is hardly ever going to be what you want.

Internally, our DPI code has been overhauled. Everything is now controlled by two parameters: for each window, the number of device pixels per inch (returned by nsIWidget::GetDPI), and also for each window, the default scale (returned by nsIWidget::GetDefaultScale). The 'layout.css.dpi' about:config pref overrides the result of nsIWidget::GetDPI, if present. nsIWidget::GetDPI only affects the interpretation of mozmm (unlike before, where on some platforms, some DPI values would trigger automatic scaling). We set CSS 1px to one device pixel times GetDefaultScale times the current zoom factor. Currently GetDefaultScale always returns 1.0 on all platforms, although on Mac we should set it to the system "default UI scale" (and change some other code to compensate), and on Windows we should set it based on the "system font DPI", which is essentially a user preference that controls scaling of all applications on the system. It's important that the default scale be based on a system-wide setting; that will keep Firefox consistent with the rest of the system, and ensure that the user doesn't get a surprise.

Franklin Street Statement PDF

2010-08-19T17:00:48+00:00

The Franklin Street Statement can be found on the web in the form of a fairly ugly blog post, which does not befit a document of such importance… and for that matter, neither does the printed form of that page.

I wanted a nice paper copy of the statement to put in a prominent position on my desk, and figured it would make sense to share my rendering. It may be an interesting thing to hand out at developer conferences or Software Freedom Day events.

The document is set in Free typefaces, Liberation Serif and League Gothic — appropriate, given that League is a wonderful Free alternative to Franklin Gothic! I know, I know, you see what I did there. Now… Download the Franklin Street Statement PDF!

Note: I wish the source ODF had survived the wrath of OpenOffice.org’s “eat my document” crash recovery feature, but at least the resulting PDF is fairly easy to edit.

Some people march to a different drummer – and some people polka

2010-08-19T15:14:49+00:00

You may remember that over a few months earlier this year, the Ubuntu Women Project ran some competitions in which people voted; firstly a story contest, then a photo one.

The voting websites were powered by some (really quite messy) php that I put together over a few wee-hours sessions, after being unsuccessful in finding something that was functional but at the same time not more complicated than the purpose required.

After a few months and some prodding by Alan Bell as he wanted to build on it for deciding on a new logo for Ubuntu-UK, I did some refactoring of the code a few weeks back and pushed up to LP.

Originally I put it up in my junk directory, but then I had an idea <insert ominous music here>.

Why not use the project to mentor Ubuntu Women members who’ve benefited from and found the UW project via the competitions. I suggested this to others in #ubuntu-women and they agreed that it was a good idea, and decided it should be called “Pollka”. Awesome sauce.

The aim of Pollka is to have a quickly deployable simple web application for polling between a number of choices. It currently uses php and mysql, but ideally other database options will be catered for in the future. If you want to help out, more information can be found at the following locations:

https://launchpad.net/~pollka-devs is the development team, and
https://launchpad.net/pollka is the project page on LP.
#pollka on the same network as Ubuntu’s channels, irc.freenode.net

19 Aug 2010

2010-08-19T00:32:13+00:00

I wanted to update a Nokia E71 to the latest firmware. So I booted a Windows 7 VM, went to the Nokia web site and downloaded the update tool. It took about quarter of an hour to download 33MiB over my crappy ADSL line.

When it finally finished downloading, I installed it and rebooted as it asked me to, then tried to start it using the icon it had installed on the desktop.

It told me there was an update available, and I couldn't use it until I updated. I muttered darkly at this idiocy, but let it update. It spent another quarter of an hour downloading, and only then did it check and tell me that it couldn't proceed because it needed to be run as Administrator.

So I right-clicked on it and used the 'run as Administrator' option, and watched it download itself for the third time. But still it failed, complaining that I had to run it as Administrator.

So I logged out completely and logged in using the Administrator account, and I ran it again. After downloading the entire thing for a fourth time it failed again, still complaining that it has insufficient privileges and needed to be run as the Administrator.

I am stunned — Nokia really ought to be ashamed at this crap.

Storage Engine independent test suite

2010-08-18T01:57:16+00:00

This is something that has never existed in the MySQL world. Nothing to help you start developing your engine.

Sure, you could start running the whole test suite against your engine…. but where it wasn’t specifically MyISAM, you’d certainly hit things that were.. well… as simple as having to change the result file so that SHOW CREATE TABLE masked out the ENGINE= part.

Also, if you were just starting out and were trying to incrementally write your engine, instead of just hacking away for 6 months on “everything” and then testing “everything” and hoping that anything at all worked – you were out of luck.

So when working on the embedded_innodb engine for Drizzle I wanted to finally fix this problem. I was not going to fix it perfectly, or completely. What I wanted was a set of simple tests, that were very short and that tested distinct bits of the engine.

So… if you look in the Drizzle tree, in plugin/embedded_innodb/tests/ you’ll find a bunch of test cases beginning with basic_ (they’re also now in plugin/pbxt/tests). These check very basic functionality and should be useful when starting to write your own storage engine.

They should also port easily to MySQL too (possibly without changes), but I haven’t tried it. Porting to PBXT was simple.. a regex to replace InnoDB with PBXT.

Enjoy.

Show them that you care

2010-08-18T01:51:11+00:00

There’s just 29 days left to get your nominations in for this year’s New Zealand Open Source Awards.

There are 8 categories this year: Open Source Use in Government, Open Source Use in Business, Open Source Use in Education, Open Source Use in The Arts, Open Source Software Project, Open Source Contributor, Open Source Advocate, Open Source People’s Choice Award

Think about the people in the New Zealand open source community whose contributions have made a difference to your life or your business over the last couple of years – the last awards ceremony was in 2008 – and send in your nominations now.

Remember that these awards aren’t just for the coders, but also for the supporting cast of people contributing, using and making a difference with open source software. Don’t forget to nominate your own contributions, projects, or even yourself if you’ve been doing great work that you like to see more widely recognised.

Wired? Tired? Expired.

2010-08-17T20:57:27+00:00

Glyn Moody pointed out an awful article written by Chris Anderson (of Long Tail infamy) and Michael Wolff, “The Web is Dead. Long Live the Internet”. It is almost Nick Carr-esque in its pursuit of overwrought claims with little to no basis in fact.

The lunacy of this article will resonate with anyone remotely connected with the technology industry, let alone those involved in software and web development.

But here are my favourite bits of abject cluelessness:

JavaScript then, Objective-C now? HTML then, XML now? What the fuck have you been smoking, Chris Anderson? … and don’t tell me it was some “junior staffer” subbing the living crap out of your stupid lack of perspective.

… and then this little gem to finish things off, ignoring years of evidence that we enjoy temporary dalliances with convenience before returning to (often quite) revolutionary openness:

The Internet is the real revolution, as important as electricity; what we do with it is still evolving. As it moved from your desktop to your pocket, the nature of the Net changed. The delirious chaos of the open Web was an adolescent phase subsidized by industrial giants groping their way in a new world. Now they’re doing what industrialists do best — finding choke points. And by the looks of it, we’re loving it.

I’ll make my own bold, unsubstantiated claim… Wired: Tired and expired.

Glad I have that off my chest. Thank you, ball boys.

* * *

Update: Check out this hilarious take-down from Boing Boing … with facts! … and analysis! … and pretty graphs!

embedded_innodb engine (HailDB Engine)

2010-08-17T18:45:06+00:00

A whole bunch of stuff got merged to the embedded_innodb engine recently. The end game is to have the embedded_innodb engine be just linking to HailDB (where we’re continuing maintenance of Embedded InnoDB).

So, what made it in? A lot of bug fixes (especially around error reporting and tables without an explicit primary key), SELECT FOR UPDATE, support for temporary tables and support of prefix indexes.

We also now, if it’s available, link to HailDB instead of Embedded InnoDB. At some point “soon” we shall just require HailDB.

The mozRequestAnimationFrame Frame Rate Limit

2010-08-16T21:49:24+00:00

A few people have been playing with mozRequestAnimationFrame and noticed that they can't get more than 50 frames per second. This is intentional, and it's a good feature.

On modern systems an application usually cannot get more than 50-60 frames per second onto the screen. There are multiple reasons for this. Some of them are hardware limitations: CRTs have a fixed refresh rate, and LCDs are also limited in the rate at which they can update the screen due to bandwidth limitations in the DVI connector and other reasons. Another big reason is that modern operating systems tend to use "compositing window managers" which redraw the entire desktop at a fixed rate. So even if an application updates its window 100 times a second, the user won't be able to see more than about half of those updates. (Some applications on some platforms, typically games, can go full-screen, bypass the window manager and get updates onto the screen as fast as the hardware allows, but obviously desktop browsers aren't usually going to do that.)

So, firing a MozBeforePaint event more than about 50 times a second is going to achieve nothing other than wasting CPU (i.e., power). So we don't. Apart from saving power, reducing animation CPU usage helps overall performance because we can use the free time to perform garbage collection or other house-cleaning tasks, reducing the incidence or length of frame skips.

We need to do some followup work to make sure that on each platform we use the optimal rate; modern platforms have APIs to tell us the window manager's composition rate. But 50Hz is almost always pretty close.

This all means that measuring FPS is a bad way to measure performance, once you're up to 50 or more. At that point you need to increase the difficulty of your workload.

v0.21.1 released

2010-08-16T15:02:36+00:00

We’ve made a bugfix release for v0.21 last week. Changes include:

debian and rpm packaging fixes
mds: fixed crash on some mds->client messages
mds: fix snaprealm behavior on readdir (occasional client misbehavior)
monmaptool: man page typo
rados: usage fix
osd: fix heartbeat to/from osds (fixing osd up/down flapping)
osd: fix replies to dup/committed requests (fixes client hangs)
librados: .hpp fix
cclass: fix .so loading
cauthtool: fix man page example for fs clients
fix log rotation

Relevant URLs:

Direct download at http://ceph.newdream.net/download/ceph-0.21.1.tar.gz
For Debian packages see http://ceph.newdream.net/wiki/Debian

Web site revisions

2010-08-15T23:44:08+00:00

I've started making some more substantial additions and changes to the website for my book. With the latest revisions, you can now see:

A more detailed description of the book
The short and long table of contents
The preface
The index
A sample chapter from the book (Chapter 4: File I/O: The Universal I/O Model)
The source code of all of the example programs in the book, with each file shown as a separate web page (a tarball of the source code will appear later).

Packet and Denby

2010-08-15T13:53:33+00:00

This is Packet.

This is Denby.

I mention in the video that it’ll end up being “like TweetDeck”. I breezed past that a tad too quickly — it’s definitely not going to be an HTML5 clone of TweetDeck! (Turns out they’re working on one of those already.)

Instead, it will take inspiration from the multi-column approach, but hopefully improve the user experience on a number of levels:

It’ll be 100% Free Software / Open Source… and one hopes, peer produced.
JavaScript on the server, JavaScript on the client, with liberal doses of “HTML5″ (the platform your platform could be), whatever that means to you.
No Adobe AIR, and thus, no vicious memory and CPU abuse! Seriously: ~~Firefox~~ TweetDeck is wasting more CPU time sitting “idle” than Chromium and node.js are using to run Denby. Oh, and node.js is 12MB resident.
It can work a bit like a desktop app… use the “web page as application” tool your browser provides. Firefox has Prism, Chromium has… a menu item. Denby will support things like desktop notifications, audio bleeps, drag-n-drop (for media uploads), inline display of media, etc.
Run local or hosted. Once I’m happy with the user experience, I’ll start thinking about cool things the server could do while you’re not connected to it! For now, it only maintains the connection to Twitter while a client is connected.
I want to build delicious multiple account support, without complicating the single account experience. It’ll merge streams, detect the context you’re acting upon (click reply and you’ll be replying from the appropriate account), etc.

Please comment if you have any thoughts or suggestions… crimes committed by other Twitter clients, ideas for lovely web/desktop integration, and so on. Thanks!

HELLO DENBY.

* * *

Note that I’ve already had questions about StatusNet and identi.ca support. Thus far, they don’t have user streams, which was one of the main reasons behind building Denby. That said, it is entirely possible for the Denby server to poll the REST API (given that it already talks to Twitter’s) and send the results down the WebSockets tube… so, we’ll see.

Hmm. Perhaps this is the best way to build a user streams API for StatusNet anyway? The web app could send JSON messages to node.js, which could relay them to the intended users… via multiple protocols! Long-running HTTP, WebSockets, whatever. If anyone is inspired to do this, StatusNet could have a bi-directional WebSockets API before Twitter does!

* * *

Update: I made another quick video, showing a few improvements (including update bleeps — no chicken noises as yet), Denby in Firefox’s Prism environment, and sharing some thoughts about the web as a development platform. Enjoy!

Auckland Food

2010-08-15T07:42:38+00:00

For cheap tasty food, BBQ King on Wyndham Street West is hard to beat. Today I went there with family and based on past experience, we ordered only two dishes for four people, "BBQ pork and crispy pork stirred noodle" for $13.50 and "seafood fried rice" for $12. After eating the free soup and then dividing the first dish among four of us, we were pretty much satisfied, so boxed the second dish and brought it home. Crazy!

Tragically, my favourite Chinese restaurant in the city --- Sunshine --- closed several months ago. I can understand when a bad restaurant goes under, but not a good one; the imperfections of a market economy! But I have heard rumours that a new Chinese restaurant has taken its place. This needs investigation.

Daikoku Teppanyaki on Quay St is now open for lunch seven days a week. The $13.50 lunch special is still great value.

Around Newmarket: Happy Valley, the Chinese cafe, has closed down. That's sad, since they'd been serving pretty good food since the early 90s.

The Organic Pizza Co.'s $10 lunch specials are pretty good. Their pizzas are about as good as Archie's, but you get a free drink and the place is far less crowded.

Selera, Night Spice, Crazy Noodle and Dee Jai are frequent targets of visits from the Mozilla office, as is the food court under the Rialto carpark. The actual Rialto food court next to the cinema seems to be declining --- two outlets have closed recently --- but they still have the only Subway in the area. We'd go to Hansan more but we're too lazy to walk there except for special occasions.

Pearl Garden still the best yum cha in Newmarket, followed by Sun World and Sunnytown. There's a new place whose name escapes me over near Davis Crescent; not bad, but not great. I need to try it again.

Standards and Portability in TLPI

2010-08-15T07:17:37+00:00

As Linux becomes more and more prevalent, many programmers are targeting just one system: Linux. For these programmers, The Linux Programming Interface is ideal.

However, all the world is not Linux, and many of the most interesting UNIX applications need to be truly portable across systems: Linux, the BSDs, Solaris, HP-UX, AIX, and so on. For these applications, following portable programming practices and adhering to formal standards such as POSIX and the Single UNIX Specification are important.

Although I've focused on Linux in the book, I've also given close attention to formal and implementation standards, and clearly distinguished details that are Linux-specific from the features that are common across most or all UNIX implementations. This means that TLPI will be useful to programmers working on any UNIX system (but it won't provide you with too many of the nonstandard and system-specific details of those other systems). That's why we subtitled the book A Linux and UNIX System Programming Handbook.

A few statistics illustrate TLPI's emphasis on standards and portability. Of the just over half a million words in the book, here are a few word counts:

POSIX, 714
SUSv3/SUSv4, 730
standard(s)/nonstandard/standardized, 685
portable/nonportable/portability/portably, 231
UNIX, 1039
BSD, 254
System V (excluding discussion of System V IPC), 108

Firefox Sync

2010-08-15T05:32:12+00:00

I just tried using Firefox Sync to synchronize data between my main Firefox profile and my newly-installed home computer's Firefox profile. It was easy to set up and worked perfectly --- and it was fast too! I have to confess my expectations were not high for a feature that just got turned on for beta 4 :-). Well done everyone! This is definitely going to make my life a little easier.

unittest, part 3

2010-08-14T15:34:41+00:00

So far, we've talked about TestSuites, TestCases and TestResults. We've seen how these objects interact with each other and how they can generally be thought about as having more than one interface. TestResult has an interface for the TestCase and an interface used for querying the results, TestCase has an interface for test runners and an interface for test authors, and so forth.

Now we need to give some time to the bits that glue everything together: the test runner and the test loader.

TestRunner

You will not find a class in unittest.py called TestRunner. A test runner is simply something that takes user input about a test run – what tests to run, what manner to run them in, how to display the results – and does it.

Essentially, it does something like this:

  test = TestLoader().loadTests(user_specified_test_string)
  result = makeTestResult(options_specified_by_user)
  result.startTestRun()
  try:
    test.run(result)
  finally:
    result.stopTestRun()

And that's it.

You see that the test runner is responsible for instantiating the test loader and the test result. It's perhaps excusable for a test runner to be tightly bound to particular implementations of test loader and test result. Certainly, before TestResult grew startTestRun and stopTestRun it was inevitable: since the test runner was responsible for summarizing the results of a test run, overall responsibility for displaying the results was split between the runner and the result.

Nowadays, the tight coupling can be limited. If your test runner has an option to display stack traces as it gets them, then that's pretty much going to force you to use a particular result. However, you can still write your code internally such that someone could pass in a different result that still works, even though it doesn't do exactly what the user asked for.

TestLoader

From the point of view of interfaces and compatibility, this is a pretty boring class, and that's a good thing. The test loader's job is to find tests based on some user input and construct a single ITest object for them.

When it does more than this, one runs the risk of having the behaviour of a test suite depend too much on the runner itself. The ideal is to have the test suite run in any runner: trial, nose, unittest2, py.test, whatever.

Some TestLoaders provide hooks so that users with complicated test suites can customize the way their tests are loaded. Whenever the Trial TestLoader sees a test_suite() function in a module, it lets that function take charge of the loading.

The standard library in 2.7 has a new hook, inspired by an innovation in bzrlib, but slightly different. load_tests(loader, standard_tests, pattern) is given the loader used by the test runner, the tests that the loader would have loaded, and if appropriate, a glob used for matching test module files. The advantage of this hook is that it reduces the danger of customizations made to the loader, since the test suite has access to the same loader. It also makes custom loading easier by giving the standard tests as a starting point. bzrlib uses this to run the same set of tests against many implementations.

I think that's all I have to say about these two, which means that's pretty much all I have to say about unittest's API for test frameworks. Still one more post to go though: interfaces for test authors.

Let me know if I've missed anything, if anything here surprises you or contradicts something I said in the past or if things are unclear. The comments on the previous two posts have really helped!

mozRequestAnimationFrame

2010-08-14T10:13:53+00:00

In Firefox 4 we've added support for two major standards for declarative animation --- SVG Animation (aka SMIL) and CSS Transitions. However, I also feel strongly that the Web needs better support for JS-based animations. No matter how rich we make declarative animations, sometimes you'll still need to write JS code to compute ("sample") the state of each animation frame. Furthermore there's a lot of JS animation code already on the Web, and it would be nice to improve its performance and smoothness without requiring authors to rewrite it into a declarative form.

Obviously you can implement animations in JS today using setTimeout/setInterval to trigger animation samples and calling Date.now() to track animation progress. There are two big problems with that approach. The biggest problem is that there is no "right" timeout value to use. Ideally, the animation would be sampled exactly as often as the browser is able to repaint the screen, up to some maximum limit (e.g., the screen refresh rate). But the author has no idea what that frame rate is going to be, and of course it can even vary from moment to moment. Under some conditions (e.g. the animation is not visible), the animation should stop sampling altogether. A secondary problem is that when there are multiple animations running --- some in JS, and some declarative animations --- it's hard to keep them synchronized. For example you'd like a script to be able to start a CSS transition and a JS animation with the same duration and have agreement on the exact moment in time when the animations are deemed to have started. At each paint you'd also like to have them sampled using the same "current time".

These problems have come up from time to time on mailing lists, for example on public-webapps. A while ago I worked out an API proposal and Boris Zbarsky just implemented it; it's in Firefox 4 beta 4. Here's the API, it's really simple:

window.mozRequestAnimationFrame(): Signals that an animation is in progress, requests that the browser schedule a repaint of the window for the next animation frame, and requests that a MozBeforePaint event be fired before that repaint.
The browser fires a MozBeforePaint event at the window before we repaint it. The timeStamp attribute of the event is the time, in milliseconds since the epoch, deemed to be the "current time" for all animations for this repaint.
There is also a window.mozAnimationStartTime attribute, also in milliseconds since the epoch. When a script starts an animation, this attribute indicates when that animation should be deemed to have started. This is different from Date.now() because we ensure that between any two repaints of the window, the value of window.mozAnimationStartTime is constant, so all animations started during the same frame get the same start time. CSS transitions and SMIL animations triggered during that interval also use that start time. (In beta 4 there's a bug that means we don't quite achieve that, but we'll fix it.)

That's it! Here's an example; the relevant sample code:

var start = window.mozAnimationStartTime;
function step(event) {
  var progress = event.timeStamp - start;
  d.style.left = Math.min(progress/10, 200) + "px";
  if (progress  2000) {
    window.mozRequestAnimationFrame();
  } else {
    window.removeEventListener("MozBeforePaint", step, false);
  }
}
window.addEventListener("MozBeforePaint", step, false);
window.mozRequestAnimationFrame();

It's not very different from the usual setTimeout/Date.now() implementation. We use window.mozAnimationStartTime and event.timeStamp instead of calling Date.now(). We call window.mozRequestAnimationFrame() instead of setTimeout(). Converting existing code should usually be easy. You could even abstract over the differences with a wrapper that calls setTimeout/Date.now if mozAnimationStartTime/mozRequestAnimationFrame are not available. Of course, we want this to become a standard so eventually such wrappers will not be necessary!

Using this API has a few advantages, even in this simple case. The author doesn't have to guess a timeout value. If the browser is overloaded the animation will degrade gracefully instead of uselessly running the step script more times than necessary. If the page is in a hidden tab, we'll be able to throttle the frame rate down to a very low value (e.g. one frame per second), saving CPU load. (This feature has not landed yet though.)

One important feature of this API is that mozRequestAnimationFrame is "one-shot". You have to call it again from your event handler if your animation is still running. An alternative would be to have a "beginAnimation"/"endAnimation" API, but that seems more complex and slightly more likely to leave animations running forever (wasting CPU time) in error situations.

This API is compatible with browser implementations that offload some declarative animations to a dedicated "compositing thread" so they can be animated even while the main thread is blocked. (Safari does this, and we're building something like it too.) If the main thread is blocked on a single event for a long time (e.g. if a MozBeforePaint handler takes a very long time to run) it's obviously impossible for JS animations to stay in sync with animations offloaded to a compositing thread. But if the main thread stays responsive, so MozBeforePaint events can be dispatched and serviced between each compositing step performed by the compositing thread, I think we can keep JS animations in sync with the offloaded animations. We need to carefully choose the animation timestamps returned by mozAnimationStartTime and event.timeStamp and dispatch MozBeforePaint events "early enough".

mozRequestAnimationFrame is an experimental API. We do not guarantee to support it forever, and I wouldn't evangelize sites to depend on it. We've implemented it so that people can experiment with it and we can collect feedback. At the same time we'll propose it as a standard (minus the moz prefix, obviously), and author feedback on our implementation will help us make a better standard.

President's Report August 2010

2010-08-14T07:34:14+00:00

It’s been just over a month since my last presidents report, which according to past presidents means that I’m doing well. Apparently the first report is the easy one, maintaining momentum is the key!

LCA2011

We are now well and truly into the run up to linux,conf.au 2011. The LCA Call for Papers, Miniconfs, Posters and Tutorials (that is quite a mouthful!), has been open since the 13th of July. The CFP closes at midnight tonight, so it’s still not too late to get a proposal in.

Google vs Oracle

2010-08-13T22:46:39+00:00

I don't know much about the Google/Oracle dispute so I'll limit my remarks, but here they are:

I don't understand why Oracle is doing this. They may wish Google was using Java ME, but I would have thought having more developers using Java was good for Java overall. Probably there are important background discussions we are not privy to.

Dalvik is open source, but it's very much a Google project that Google happens to release under an open source license, rather than a community project. So I think of this as two big companies scrapping rather than Oracle launching an attack on the open source community.

However, this extends a disturbing trend of large mainstream companies using software patents to attack competitors, especially prominent in the mobile space. Observers of software patents, including myself --- and even Bill Gates in his infamous 1991 memo --- have always seen that volumes of easily obtained software patents on straightforward ideas could be a powerful weapon to crush competition; software development is so inventive that most programmers daily write code that someone patented somewhere. Fortunately, for a long time, other than "patent trolls", serious industry players declined to use that power. But now that grace has departed and I fear patent armageddon is upon us. In the end the open source community is likely to be particularly hard-hit, since it's easy to detect infringement, and open source communities have limited funds for defense. People have argued that open source communities are less of a target because they have less money to extract, but the most dangerous suits are about shutting down competition, not about extracting licensing fees --- like this Google/Oracle suit, apparently.

Overall I'm extremely gloomy about the situation. A world where each programmer has to be shepherded by a dozen lawyers through patent minefields is not one I will enjoy working in, and it will be disastrous for the progress of software. I call on employees of Oracle, Apple and other litigating companies to protest to their management in the strongest possible terms, including resignation. Google and Mozilla are hiring :-).

It's little consolation that some enlightened countries --- like New Zealand, apparently --- will hopefully remain free of software patents. A software company --- or an open source project --- that can't do business or get distribution in the USA or many other countries (including most of Europe, given the 'method patent' regime) is somewhat crippled.

Chapter 4 available online

2010-08-11T23:15:03+00:00

No Starch Press now has a sample chapter from my book available online: Chapter 4: File I/O: The Universal I/O Model. You can find the chapter here.

Dell Fail

2010-08-11T09:19:33+00:00

Separate from the laptop discussion, I just bought a new home machine. I just wanted a generic PC, high-ish end for longevity and in case I (or someone else) wants to hack on it. This machine will definitely run Linux, but I'm going to keep the Windows 7 install in a partition in case we ever need it. So I'm going through Dell's Windows 7 first-run experience, and it's not great.

The initial Microsoft setup screens are pretty good, although it all seems to take longer than it should. Then you get to a Dell screen asking you to opt into some Dell stuff, which for some unfathomable reason is rendered in the Windows 95/2000 "Classic" theme, gray box scrollbars and all. It's ugly, jarring and totally mystifying.

Soon you're offered the chance to burn system recovery DVDs. I don't understand why they ask users to obtain blank DVDs and burn them instead of just shipping those DVDs; shipping them with every system would add a few dollars to the system cost, but probably save more in support calls and give a much better user experience.

The application that burns the recovery DVDs has one crazy screen that shows you some information and asks you to click "Next". But there is no "Next" button visible. But there's a vertical scrollbar! Scrolling down, you can get to a "Next" button. Of course, the window is not resizable, and it contains lots of blank vertical space so there is no possible reason why the "Next" button should not be visible.

Microsoft's initial Windows network setup asks you whether you're on a "Home", "Work" or "Public" network, which I bet is often hard for people to answer. I wonder how Windows uses that information. But right after choosing that option, the (preinstalled) McAfee antivirus software pops up an ugly little box in which you have to choose those same options again.

Of course I still have to analyze the system for the paid-to-be-there crapware (including McAfee) and uninstall most of it.

I'm genuinely curious about what motivates system vendors like Dell to sully what could have been a better experience. It's not apathy, since they obviously paid people to develop many of these "extras". Whatever it is, it's no surprise platform vendors want to sell directly to the customer instead of working through partners like Dell.

Kodak Portra

2010-08-10T20:57:45+00:00

I started to realise that I was liking the look of photos shot on Kodak Portra. I wanted to shoot some of it to see what I thought. I bought a pack of 5 rolls of 160VC from Glazer’s just before heading to OSCON.

Here are some of the shots I got:

(granted I didn’t press the shutter release, but I like it)

Is the one that sealed it for me. This was the “ahh… I can use this for all sorts” shot.

All of these were developed and scanned at the Walgreens down the street. I could probably do better scans of some… but this was awfully less work for me.

I am really liking the skin tones from it. The vividness of colours also comes through while retaining excellent skin tones (certainly not always the case). I may even end up shooting some at Burning Man (did buy more rolls yesterday!)

There is (of course) more being added to my Kodak Portra 160VC set on flickr.

Choosing Sides

2010-08-10T11:36:08+00:00

My Macbook Pro is 3.5 years old and still works pretty well, apart from the disk being full, frequently spontaneous wake-ups in my backpack which heat everything up alarmingly, and flaky wireless connection. Plus the turn of Moore's Law means I can now get a quad-core laptop and a lot more RAM and disk. So it's finally time to upgrade. I'm opting for non-Apple hardware; Apple have gone beyond the pale pursuing patent warfare and platform lockdown, and I can no longer live with buying their products. A Lenovo W510 is probably in my future.

Now I'm faced with a somewhat difficult decision: Linux vs Windows. There are good reasons on both sides. The best thing about developing in Windows is that it's good to have developers on the platform that most of our users use. VMWare record and replay is also very attractive. The Mozilla build and tools situation on Windows used to be terrible --- very slow builds, horrible profiling --- but it's gotten a lot better thanks to pymake and xperf. But Microsoft, while not as dangerous as Apple at the moment, still aspires to be, and I won't embrace them gladly.

Linux, of course, has the virtue of freedom, and a chance to regain Miguel's love. On Linux you get valgrind. But not many of our users use Linux, and VMWare's record and replay doesn't really work there. I'd have to use X11, which I loathe with passion.

Tough call. Another thing to consider is that whichever way I go, I'll end up using the other in a VM quite often. Dual-booting is also an option.

We Melted Gold(ish)!

2010-08-09T05:24:00+00:00

Kate had a bunch of copper wire clippings spare, so I tried melting them in the furnace. Using a sawn-off steel CO2 container with its bottom flattened as a crucible and the sawn-off bit poked into the top to reduce oxidation. I also added a bit of glass to float any slag.

As you can see, I managed to get it up to a fair old temperature, glowing yellow hot and some. The copper melted inside the container with an amazing amount of outgassing.

So the picture makes it look like gold, but it was copper. Not that there's much of a difference in melting point. The big surprise is that it came out as bead-like globules. Hopefully a clean re-melt will fix it.

This melt was done with homemade charcoal, which produced an excellent heat. Next I'll be using smaller chunks, and mould a sloping trench to direct the charcoal into the path of the incoming air blast. Out into the rain we go to mould the clay while it's damp...

"The Trouble with Multicore" by David Patterson in July 2010 IEEE Spectrum

2010-08-09T00:06:04+00:00

Patterson's article is quite interesting, and brings out some good points. I was of course happy that he mentioned Sequent, my old employer, despite the fact that the mention was on a list of “long-gone parallel hopefuls.” An unflattering mention, perhaps, but undeniably true.

However, I was especially happy to see the following sentence:

So rather than working on general programming languages or computer designs, we are instead trying to create a few important applications that can take advantage of many-core microprocessors.

Focusing on parallelization in the large is a great improvement over the traditional academic focus on parallelization in the small. All else being equal, the larger the software artifact, the larger the units of work, and the smaller the fraction of computational resources spent on communication. The less the communication, the better the performance, and usually the greater the scalability. So Patterson's pronouncement is a welcome change, especially given his group's earlier focus on small-scale computational kernels. I hope that the fact that Patterson has now joined the growing group of academics focused on parallelization in the large will encourage other academics to do the same.

Of course, I could raise a number of quibbles with the paper:

The analogy of parallel processing with journalism (last full paragraph of the last column on page 30) misses the mark. Patterson notwithstanding, the fact is that most writers do in fact use parallel processing: there will be a reporter, a copy-editor, and so on. It is in fact quite common for authors of large works to acknowledge those who did research, fact-checking, and other tasks. Of course, to Patterson's point, there must be a limit to the degree of parallelism that can be achieved. But the success of things like Wikipedia indicates that the potential for parallelism is much larger than has been commonly thought.
Patterson argues that desktop applications rarely have sufficient intellectual horsepower behind them to make good use of multicore systems (last sentence of page 31). History has shown, however, that it is not raw intellectual horsepower that is required, but rather experience and proper training.
Patterson also seems to believe that parallel programmers should start small and work their way up to larger systems (last sentence of first paragraph of page 32). Sequent's experience indicates otherwise: by starting off with 30-CPU systems from the get-go, Sequent avoided the typical parallel-programming experience, which is to rewrite the program from scratch multiple times, first to accommodate parallelism at all, next to scale beyond two CPUs, next to get beyond the 16-32-CPU level, and so on. Diving into the deep end of the parallel-programming pool can be quite a bit cheaper and easier than gingerly paddling out from the shallows.
Patterson complains that large systems (128 cores) are not being manufactured, and that software emulation is painfully slow (middle of third column on page 32). Such large systems have in fact been available for quite some time from a number of manufacturers. Of course, they are still quite expensive, which can certainly render them unavailable to most developers. However, there is little need for universities to fabricate them, unless of course they are conducting research on the hardware itself.

Finally, the box on page 31 entitled “Easy as Pi” deserves special attention. In this box, Patterson contrasts a sequential method for calculating the quantity π/4, namely summing the infinite series for the arctangent of one radian, with a parallel Monte Carlo method, which generates pairs of random floating-point numbers between -1 and +1, then counts the fraction that lie within the unit circle.

How good are these algorithms?

Kicking Javascript to the Footer in Drupal 8?

2010-08-08T09:14:28+00:00

As a platform, Drupal has excellent javascript support. Drupal 7 will ship with jQuery 1.4.2 and jQuery UI 1.8, which will make it even easier to build rich user interactions with Drupal.

Drupal supports aggregating javascript files to reduce the number of network connections a browser must open to load a page. It is common practice for Drupal themes to put the <script> tag in the <head> section of the page. Unfortunately this has a performance impact, as all browsers will stop processing the page and start loading and processing the referenced javascript file. For this reason, both Yahoo! and Microsoft recommend placing all javascript just before the closing </body> tag in a page so it is loaded and processed after the content.

Making this change in Drupal is a pretty straight forward process. It is already possible to do this in Drupal 6 or 7. My site places the $scripts variable at the end of the page. Unfortunately some modules rely on javascript being in the <head>er, and some even place <script>s in the body to allow inline function calls.

It is too late to implement this change in Drupal 7, but the transition can occur now. Documentation can be updated to inform theme developers that they can place the $script variable at the end of the page, just above where the $closure variable is placed. The module development guide can be updated to strongly recommend against relying on the value 'header' for the 'scope' element of the $options array for drupal_add_js() meaning that the javascript will end up in the header and to not place any inline javascript code in themes or modules. In Drupal 8 the scope element for the $options array can be dropped.

If theme and module developers adopt this best practice approach for their Drupal 7 releases there should be minimal transition work for this change in the version 8 release cycle.

I am hoping to discuss this at the Core Developers Summit at DrupalCon Copenhagen later this month.

WebSRT and HTML5 media accessibility

2010-08-07T01:09:13+00:00

On 23rd July, Ian Hickson, the HTML5 editor, posted an update to the WHATWG mailing list introducing the first draft of a platform for accessibility for the HTML5 <video> element. The platform provides for captions, subtitles, audio descriptions, chapter markers and similar time-synchronized text both in-band (inside the video resource) and out-of-band (as external text files). Right now, the proposal only regards <video>, but I personally believe the same can be applied to the <audio> element, except we have to be a bit more flexible with the rendering approach. Anyway…

What I want to do here is to summarize what was introduced, together with the improvements that I and some others have proposed in follow-up emails, and list some of the media accessibility needs that we are not yet dealing with.

For those wanting to only selectively read some sections, here is a clickable table of contents of this rather long blog post:

THE WebSRT TIMED TEXT FORMAT

The first and to everyone probably most surprising part is the new file format that is being proposed to contain out-of-band time-synchronized text for video. A new format was necessary after the analysis of all relevant existing formats determined that they were either insufficient or hard to use in a Web environment.

The new format is called WebSRT and is an extension to the existing SRT SubRip format. It is actually also the part of the new specification that I am personally most uncomfortable with. Not that WebSRT is a bad format. It’s just not sufficient yet to provide all the functionality that a good time-synchronized text format for Web media should. Let’s look at some examples.

WebSRT is composed of a sequence of timed text cues (that’s what we’ve decided to call the pieces of text that are active during a certain time interval). Because of its ancestry of SRT, the text cues can optionally be numbered through. The content of the text cues is currently allowed to contain three different types of text: plain text, minimal markup, and anything at all (also called “metadata”).

In its most simple form, a WebSRT file is just an ordinary old SRT file with optional cue numbers and only plain text in cues:

  1
  00:00:15.00 --> 00:00:17.95
  At the left we can see...

  2
  00:00:18.16 --> 00:00:20.08
  At the right we can see the...

  3
  00:00:20.11 --> 00:00:21.96
  ...the head-snarlers

A bit of a more complex example results if we introduce minimal markup:

  00:00:15.00 --> 00:00:17.95 A:start
  Auf der <i>linken</i> Seite sehen wir...

  00:00:18.16 --> 00:00:20.08 A:end
  Auf der <b>rechten</b> Seite sehen wir die....

  00:00:20.11 --> 00:00:21.96 A:end
  <1>...die Enthaupter.

  00:00:21.99 --> 00:00:24.36 A:start
  <2>Alles ist sicher.
  Vollkommen <b>sicher</b>.

and add to this a CSS to provide for some colors and special formatting:

    ::cue { background: rgba(0,0,0,0.5); }
    ::cue-part(1) { color: red; }
    ::cue-part(2, b) { font-style: normal; text-decoration: underline; }

Minimal markup accepts <i>, <b>, <ruby> and a timestamp in <>, providing for italics, bold, and ruby markup as well as karaoke timestamps. Any further styling can be done using the CSS pseudo-elements ::cue and ::cue-part, which accept the features ‘color’, ‘text-shadow’, ‘text-outline’, ‘background’, ‘outline’, and ‘font’.

Note that positioning requires some special notes at the end of the start/end timestamps which can provide for vertical text, line position, text position, size and alignment cue setting. Here is an example with vertically rendered Chinese text, right-aligned at 98% of the video frame:

  00:00:15.00 --> 00:00:17.95 A:start D:vertical L:98%
  在左边我们可以看到...

  00:00:18.16 --> 00:00:20.08 A:start D:vertical L:98%
  在右边我们可以看到...

  00:00:20.11 --> 00:00:21.96 A:start D:vertical L:98%
  ...捕蝇草械.

  00:00:21.99 --> 00:00:24.36 A:start D:vertical L:98%
  一切都安全.
  非常地安全.

Finally, WebSRT files can be authored with abstract metadata inside cues, which practically means anything at all. Here’s an example with HTML content:

  00:00:15.00 --> 00:00:17.95 A:start
  <img src="pic1.png"/>Auf der <i>linken</i> Seite sehen wir...

  00:00:18.16 --> 00:00:20.08 A:end
  <img src="pic2.png"/>Auf der <b>rechten</b> Seite sehen wir die....

  00:00:20.11 --> 00:00:21.96 A:end
  <img src="pic3.png"/>...die <a href="http://members.chello.nl/j.kassenaar/
elephantsdream/subtitles.html">Enthaupter</a>.

  00:00:21.99 --> 00:00:24.36 A:start
  <img src="pic4.png"/>Alles ist <mark>sicher</mark>.<br/>Vollkommen <b>sicher</b>.

Here is another example with JSON in the cues:

  00:00:00.00 --> 00:00:44.00
  {
    slide: intro.png,
    title: "Really Achieving Your Childhood Dreams" by Randy Pausch,
             Carnegie Mellon University, Sept 18, 2007
  }

  00:00:44.00 --> 00:01:18.00
  {
    slide: elephant.png,
    title: The elephant in the room...
  }

  00:01:18.00 --> 00:02:05.00
  {
    slide: denial.png,
    title: I'm not in denial...
  }

What I like about WebSRT:

it allows for all sorts of different content in the text cues – plain text is useful for texted audio descriptions, minimal markup is useful for subtitles, captions, karaoke and chapters, and “metadata” is useful for, well, any data.
it can be easily encapsulated into media resources and thus turned into in-band tracks by regarding each cue as a data packet with time stamps.
it is not verbose

Where I think WebSRT still needs improvements:

break with the SRT history: since WebSRT and SRT files are so different, WebSRT should get its own MIME type, e.g. text/websrt, and file extensions, e.g. .wsrt; this will free WebSRT for changes that wouldn’t be possible by trying to keep conformant with SRT
introduce some header fields into WebSRT: the format needs
- file-wide name-value metadata, such as author, date, copyright, etc
- language specification for the file as a hint for font selection and speech synthesis
- a possibility for style sheet association in the file header
- a means to identify which parser is required for the cues
- a magic identifier and a version string of the format
allow innerHTML as an additional format in the cues with the CSS pseudo-elements applying to all HTML elements
allow full use of CSS instead of just the restricted features and also use it for positioning instead of the hard to understand positioning hints
on the minimum markup, provide a neutral structuring element such as <span @id @class @lang> to associate specific styles or specific languages with a subpart of the cue

Note that I undertook some experiments with an alternative format that is XML-based and called WMML to gain most of these insights and determine the advantages/disadvantages of a xml-based format. The foremost advantage is that there is no automatism with newlines and displayed new lines, which can make the source text file more readable. The foremost disadvantages are verbosity and that there needs to be a simple encoding step to remove all encapsulating header-type content from around the timed text cues before encoding it into a binary media resource.

ASSOCIATING EXTERNAL TIMED TEXT RESOURCES WITH A VIDEO

Now that we have a timed text format, we need to be able to associate it with a media resource in HTML5. This is what the <track> element was introduced for. It associates the timestamps in the timed text cues with the timeline of the video resource. The browser is then expected to render these during the time interval in which the cues are expected to be active.

Here is an example for how to associate multiple subtitle tracks with a video:

  <video src="california.webm" controls>
    <track label="English" kind="subtitles" src="calif_eng.wsrt" srclang="en">
    <track label="German" kind="subtitles" src="calif_de.wsrt" srclang="de">
    <track label="Chinese" kind="subtitles" src="calif_zh.wsrt" srclang="zh">
  </video>

In this case, the UA is expected to provide a text menu with a subtitle entry with these three tracks and their label as part of the video controls. Thus, the user can interactively activate one of the tracks.

Here is an example for multiple tracks of different kinds:

  <video src="california.webm" controls>
    <track label="English" kind="subtitles" src="calif_eng.wsrt" srclang="en">
    <track label="German" kind="captions" src="calif_de.wsrt" srclang="de">
    <track label="French" kind="chapter" src="calif_fr.wsrt" srclang="fr">
    <track label="English" kind="metadata" src="calif_meta.wsrt" srclang="en">
    <track label="Chinese" kind="descriptions" src="calif_zh.wsrt" srclang="zh">
  </video>

In this case, the UA is expected to provide a text menu with a list of track kinds with one entry each for subtitles, captions and descriptions through the controls. The chapter tracks are expected to provide some sort of visual subdivision on the timeline and the metadata tracks are not exposed visually, but are only available through the JavaScript API.

Here are several ideas for improving the <track> specification:

<track> is currently only defined for WebSRT resources – it should be made generic and then browsers can compete on the formats for which they provide support. WebSRT could be the baseline format. A @type attribute could be added to hint at the MIME type of the provided resource.
<track> needs a means for authors to mark certain tracks as active, others as inactive. This can be overruled by browser settings e.g. on @srclang and by user interaction.
karaoke and lyrics are supported by WebSRT, but aren’t in the HTML5 spec as track kinds – they should be added and made visible like subtitles or captions.

EXPOSING A LIST OF TimedTracks TO JAVASCRIPT

This is where we take an extra step and move to a uniform handling of both in-band and out-of-band timed text tracks. Futher, a third type of timed text track has been introduced in the form of a MutableTimedTrack – i.e. one that can be authored and added through JavaScript alone.

The JavaScript API that is exposed for any of these track type is identical. A media element now has this additional IDL interface:

interface HTMLMediaElement : HTMLElement {
...
  readonly attribute TimedTrack[] tracks;
  MutableTimedTrack addTrack(in DOMString label, in DOMString kind,
                                 in DOMString language);
};

A media element thus manages a list of TimedTracks and provides for adding TimedTracks through addTrack().

The timed tracks are associated with a media resource in the following order:

The <track> element children of the media element, in tree order.
Tracks created through the addTrack() method, in the order they were added, oldest first.
In-band timed text tracks, in the order defined by the media resource’s format specification.

The IDL interface of a TimedTrack is as follows:

interface TimedTrack {
  readonly attribute DOMString kind;
  readonly attribute DOMString label;
  readonly attribute DOMString language;
  readonly attribute unsigned short readyState;
           attribute unsigned short mode;
  readonly attribute TimedTrackCueList cues;
  readonly attribute TimedTrackCueList activeCues;
  readonly attribute Function onload;
  readonly attribute Function onerror;
  readonly attribute Function oncuechange;
};

The first three capture the value of the @kind, @label and @srclang attributes and are provided by the addTrack() function for MutableTimedTracks and exposed from metadata in the binary resource for in-band tracks.

The readyState captures whether the data is available and is one of “not loaded”, “loading”, “loaded”, “failed to load”. Data is only availalbe in “loaded” state.

The mode attribute captures whether the data is activate to be displayed and is one of “disabled”, “hidden” and “showing”. In the “disabled” mode, the UA doesn’t have to download the resource, allowing for some bandwidth management.

The cues and activeCues attributes provide the list of parsed cues for the given track and the subpart thereof that is currently active.

The onload, onerror, and oncuechange functions are event handlers for the load, error and cuechange events of the TimedTrack.

Individual cues expose the following IDL interface:

interface TimedTrackCue {
  readonly attribute TimedTrack track;
  readonly attribute DOMString id;
  readonly attribute float startTime;
  readonly attribute float endTime;
  DOMString getCueAsSource();
  DocumentFragment getCueAsHTML();
  readonly attribute boolean pauseOnExit;
  readonly attribute Function onenter;
  readonly attribute Function onexit;
  readonly attribute DOMString direction;
  readonly attribute boolean snapToLines;
  readonly attribute long linePosition;
  readonly attribute long textPosition;
  readonly attribute long size;
  readonly attribute DOMString alignment;
  readonly attribute DOMString voice;
};

The @track attribute links the cue to its TimedTrack.

The @id, @startTime, @endTime attributes expose a cue identifier and its associated time interval. The getCueAsSource() and getCueAsHTML() functions provide either an unparsed cue text content or a text content parsed into a HTML DOM subtree.

The @pauseOnExit attribute can be set to true/false and indicates whether at the end of the cue’s time interval the media playback should be paused and wait for user interaction to continue. This is particularly important as we are trying to support extended audio descriptions and extended captions.

The onenter and onexit functions are event handlers for the enter and exit events of the TimedTrackCue.

The @direction, @snapToLines, @linePosition, @textPosition, @size, @alignment and @voice attributes expose WebSRT positioning and semantic markup of the cue.

My only concerns with this part of the specification are:

The WebSRT-related attributes in the TimedTrackCue are in conflict with CSS attributes and really should not be introduced into HTML5, since they are WebSRT-specific. They will not exist in other types of in-band or out-of-band timed text tracks. As there is a mapping to do already, why not rely on already available CSS features.
There is no API to expose header-specific metadata from timed text tracks into JavaScript. This such as the copyright holder, the creation date and the usage rights of a timed text track would be useful to have available. I would propose to add a list of name-value metadata elements to the TimedTrack API.
In addition, I would propose to allow media fragment hyperlinks in a <video> @src attribute to point to the @id of a TimedTextCue, thus defining that the playback position should be moved to the time offset of that TimedTextCue. This is a useful feature and builds on bringing named media fragment URIs and TimedTracks together.

RENDERING TimedTracks

The third part of the timed track framework deals with how to render the timed text cues in a Web page. The rendering rules are explained in the HTML5 rendering section.

I’ve extracted the following rough steps from the rendering algorithm:

All timed tracks of a media resource that are in “showing” mode are rendered together to avoid overlapping text from multiple tracks.
The timed tracks cues that are to be rendered are collected from the active timed tracks and ordered by the timed track order first and by their start time second. Where there are identical start times, the cues are ordered by their end time, earliest first, or by their creation order if all else is identical.
Each cue gets its own CSS box.
The text in the CSS boxes is positioned and formated by interpreting the positioning and formatting instructions of WebSRT that are provided on the cues.
An anonymous inline CSS box is created into which all the cue CSS boxes are wrapped.
The wrapping CSS box gets the dimensions of the video viewport. The cue CSS boxes are positioned so they don’t overlap. The text inside the cue CSS boxes inside the wrapping CSS box is wrapped at the edges if necessary.

To overcome security concerns with this kind of direct rendering of a CSS box into the Web page where text comes potentially from a different and malicious Web site, it is required to have the cues come from the same origin as the Web page.

To allow application of a restricted set of CSS properties to the timed text cues, a set of pseudo-selectors was introduced. This is necessary since all the CSS boxes are anonymous and cannot be addressed from the Web page. The introduced pseudo-selectors are ::cue to address a complete cue CSS box, and ::cue-part to address a subpart of a cue CSS box based on a set of identifiers provided by WebSRT.

I have several issues with this approach:

I believe that it is not a good idea to only restrict rendering to same-origin files. This will disallow the use of external captioning services (or even just a separate caption server of the same company) to link to for providing the captions to a video. Henri Sivonen proposed a means to overcome this by parsing every cue basically as its own HTML document (well, the body of a document) and then rendering these in iFrame-manner into the Web page. This would overcome the same-origin restriction. It would also allow to do away with the new ::cue CSS selectors, thus simplifying the solution.
In general I am concerned about how tightly the rendering is tied to WebSRT. Step 4 should not be in the HTML5 specification, but only apply to WebSRT. Every external format should provide its own mapping to CSS. As it is specified right now, other formats, such as e.g. 3GPP in MPEG-4 or Kate in Ogg, are required to map their format and positioning information to WebSRT instructions. These are then converted again using the WebSRT to CSS mapping rules. That seems overkill.
I also find step 6 very limiting, since only the video viewport is regarded as a potential rendering area – this is also the reason why there is no rendering for audio elements. Instead, it would make a lot more sense if a CSS box was provided by the HTML page – the default being the video viewport, but it could be changed to any area on screen. This would allow to render music lyrics under or above an audio element, or render captions below a video element to avoid any overlap at all.

SUMMARY AND FURTHER NEEDS

We’ve made huge progress on accessibility features for HTML5 media elements with the specifications that Ian proposed. I think we can move it to a flexible and feature-rich framework as the improvements that Henri, myself and others have proposed are included.

This will meet most of the requirements that the W3C HTML Accessibility Task Force has collected for media elements where the requirements relate to accessibility functionality provided through alternative text resources.

However, we are not solving any of the accessibility needs that relate to alternative audio-visual tracks and resources. In particular there is no solution yet to deal with multi-track audio or video files that have e.g. sign language or audio description tracks in them – not to speak of the issues that can be introduced through dealing with separate media resources from several sites that need to be played back in sync. This latter may be a challenge for future versions of HTML5, since needs for such synchoronisation of multiple resources have to be explored further.

In a first instance, we will require an API to expose in-band tracks, a means to control their activation interactively in a UI, and a description of how they should be rendered. E.g. should a sign language track be rendered as pciture-in-picture? Clear audio and Sign translation are the two key accessibility needs that can be satisfied with such a multi-track solution.

Finally, another key requirement area for media accessibility is described in a section called “Content Navigation by Content Structure”. This describes the need for vision-impaired users to be able to navigate through a media resource based on semantic markup – think of it as similar to a navigation through a book by book chapters and paragraphs. The introduction of chapter markers goes some way towards satisfying this need, but chapter markers tend to address only big time intervals in a video and don’t let you navigate on a different level to subchapters and paragraphs. It is possible to provide that navigation through providing several chapter tracks at different resolution levels, but then they are not linked together and navigation cannot easily swap between resolution levels.

An alternative might be to include different resolution levels inside a single chapter track and somehow control the UI to manage them as different resolutions. This would only require an additional attribute on text cues and could be useful to other types of text tracks, too. For example, captions could be navigated based on scenes, shots, coversations, or individual captions. Some experimentation will be required here before we can introduce a sensible extension to the given media accessibility framework.

Flash for iPhone 4? Yes! Frash 0.02!

2010-08-06T09:58:22+00:00

DISCLAIMER: The compiled binaries are based upon ALPHA code by COMEX. Code from the github repository can change minutely creating a significantly more stable version of code. DO NOT expect this compiled code to work on every Flash object or even be the slightest bit stable.

So, jailbreak for the iPhone 4 is out and comex seems to have gotten around to updating the Frash source on github for both iOS and iPhone 4. Hooray! I set out on a mission to compile it from scratch. Here I’ll detail the steps you need to produce to compile it and provide a pre-compiled binary for the less so technical (or lazy). This is essentially Frash version 0.02, but as noted before, the github repository can be updated at any time fixing any number of bugs.

As far as I know this will only work with armv7 devices, so: iPhone 3GS, iPod Touch 3G, iPad, iPhone 4. iOS4 / iPad 3.2.x only.

HOW TO INSTALL

1. Download the precompiled binary (Frash-0.02.deb | Mirror) and SFTP it to /tmp on your iPhone/iPad

2. SSH into your phone as root and type:

dpkg -i /tmp/Frash-0.02.deb

NOTE: If you had Frash 0.01 installed on your iPad, remove it first (look for it in Cydia packages). Also, I haven’t yet tested this on iPad, but it certainly works on iPhone 4 with iOS 4.0.1.

3. Reboot your device if you want, but you shouldn’t need to.

HOW TO COMPILE (ADVANCED USERS)

What you’ll need:

Mac OS X 10.6 with iPhone SDK 4.0 / 4.0.1 installed
git-osx-installer
libflashplayer.so from an Android 2.2 (Froyo) image (no link for this due to copyright issues, Google it!)
DebMaker-osx
ldid

Instructions:

1. Install git (just run the downloaded DMG) and check out the Frash git repository

git clone http://github.com/comex/frash.git

libflashplayer.so

2. Patch Xcode to compile Dynamic Libraries for iOS

a) Edit /Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Specifications/iPhoneOSProductTypes.xcspec

Make sure to place the code snippet in between the first “(” and the first “{“

    // Dynamic library
    {   Type = ProductType;
        Identifier = com.apple.product-type.library.dynamic;
        Class = PBXDynamicLibraryProductType;
        Name = "Dynamic Library";
        Description = "Dynamic library";
        IconNamePrefix = "TargetPlugin";
        DefaultTargetName = "Dynamic Library";
        DefaultBuildProperties = {
            FULL_PRODUCT_NAME = "$(EXECUTABLE_NAME)";
            MACH_O_TYPE = "mh_dylib";
            REZ_EXECUTABLE = YES;
            EXECUTABLE_SUFFIX = ".$(EXECUTABLE_EXTENSION)";
            EXECUTABLE_EXTENSION = "dylib";
            PUBLIC_HEADERS_FOLDER_PATH = "/usr/local/include";
            PRIVATE_HEADERS_FOLDER_PATH = "/usr/local/include";
            INSTALL_PATH = "/usr/local/lib";
            DYLIB_INSTALL_NAME_BASE = "$(INSTALL_PATH)";
            LD_DYLIB_INSTALL_NAME = "$(DYLIB_INSTALL_NAME_BASE:standardizepath)/$(EXECUTABLE_PATH)";
            DYLIB_COMPATIBILITY_VERSION = "1";
            DYLIB_CURRENT_VERSION = "1";
            FRAMEWORK_FLAG_PREFIX = "-framework";
            LIBRARY_FLAG_PREFIX = "-l";
            LIBRARY_FLAG_NOSPACE = YES;
            STRIP_STYLE = "debugging";
            GCC_INLINES_ARE_PRIVATE_EXTERN = YES;
            CODE_SIGNING_ALLOWED = YES;
        };
        PackageTypes = (
            com.apple.package-type.mach-o-dylib   // default
        );
    },

b) Edit /Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Specifications/iPhoneOSPackageTypes.xcspec

Make sure to place the code snippet in between the first “(” and the first “{“

    // Mach-O dynamic library
    {   Type = PackageType;
        Identifier = com.apple.package-type.mach-o-dylib;
        Name = "Mach-O Dynamic Library";
        Description = "Mach-O dynamic library";
        DefaultBuildSettings = {
            EXECUTABLE_PREFIX = "";
            EXECUTABLE_SUFFIX = "";
            EXECUTABLE_NAME = "$(EXECUTABLE_PREFIX)$(PRODUCT_NAME)$(EXECUTABLE_VARIANT_SUFFIX)$(EXECUTABLE_SUFFIX)";
            EXECUTABLE_PATH = "$(EXECUTABLE_NAME)";
        };
        ProductReference = {
            FileType = compiled.mach-o.dylib;
            Name = "$(EXECUTABLE_NAME)";
            IsLaunchable = NO;
        };
    },

3. Patch Xcode to allow compilation without a Provisioning Profile (i.e. not paying the $99 Apple tax)

a) Open /Developer/Platforms/iPhoneOS.platform/Info.plist and replace all instances of “XCiPhoneOSCodeSignContext” with “XCCodeSignContext”

4. Create a self-signed certificate to sign the Frash libraries

a) Open Applications > Utilities > Keychain Access

b) From the Keychain Access menu choose Certificate Assistant > Create a Certificate

c) Choose any name, set the type to “Self Signed Root”, check “Let me override defaults” and make it a “Code Signing” certificate

d) Specify any serial number, set the expiry to whatever you want. Longer than a year if you want to compile other iPhone libraries in the future.

e) Fill in any details for Certificate Information

f) Skip to the end of the wizard, using the default settings

5. Install ldid

a) Copy the ldid file to /usr/bin and make it executable (as root)

sudo cp ldid /usr/bin/
sudo chmod +x /usr/bin/ldid

6. Install DebMaker

a) Extract the DebMaker zip and extract dpkg-dep for later use

sudo cp DebMaker.app/Contents/Resources/dpkg-deb /usr/bin/
sudo chmod +x /usr/bin/dpkg-deb

7. Edit the default Xcode configuration for Frash

a) Open Player2/Player2.xcodeproj in Xcode

b) Make sure to switch the current configuration to Device 3.2 | Release

c) Open up “Get Info” for the Device 3.2 | Release configuration and set the “Code Signing Identity” to the self-signed certificate made earlier and ensure the project is compiling for iPhone Device 3.2. Ignore warnings about setting to compile for armv6, keep it compiling for armv7.

d) Save the project and exit Xcode

8. Time to compile! Comex provides the build steps in the README for iPhone OS 3.2 but it needs to be changed slightly:

sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk /var/sdk
sudo export PATH=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:$PATH
sudo cp -a /System/Library/Frameworks/IOSurface.framework/Versions/A/Headers Player2/IOSurface
sudo cp -a /System/Library/Frameworks/IOSurface.framework/Versions/A/Headers food/IOSurface
sudo cp -a /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/CoreText.framework/Versions/A/Headers food/CoreText
sudo cp -a /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk/System/Library/Frameworks/IOKit.framework/Versions/A/Headers/ food/IOKit
sudo cp -a /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk/System/Library/Frameworks/IOKit.framework/Versions/A/Headers/ Player2/IOKit
sudo cp /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk/Entitlements.plist .
sudo make -C rpc
sudo make -C utils
sudo make -C libgcc
sudo make -C food
sudo xcodebuild -alltargets -project Player2/Player2.xcodeproj

9. Now package it up into a deb

a) Edit DEBIAN/control and add a conflicts line to the end of the file. This will conflict with the Frash 0.01 deb released previously. You should remove that before installing this.

Conflicts: net.ozzapoo.frash

b) Place your copy of libflashplayer.so in the frash directory.

c) Modify fakeinstall to include libflashplayer.so by adding a line before “chown -R 501 iroot/var/mobile”

cp -a libflashplayer.so iroot/var/mobile/frash/

d) Make fakeinstall executable and run it as root. You should end up with a frash.deb file.

chmod +x fakeinstall
sudo ./fakeinstal
mv frash.deb Frash-0.02.deb

10. Hooray! Now follow the novice install section using the deb you just compiled (instead of the precompiled version, duh) to install Frash!

Boston

2010-08-06T02:54:25+00:00

I will be in Boston next week to give a presentation on Network Bandwidth Control in Virtualized Environments at LinuxCon and attend the related mini-summits. Its been a while since I've attended an event in North America, I'm looking forward to it.

Off to the printer today!

2010-08-05T18:09:00+00:00

The files go to the printer today. All 1552 pages...

Burning Man shower test

2010-08-05T00:19:35+00:00

Yazz has been making an epic over engineered shower for camp.

Update on “A Tale Of a Bug”

2010-08-04T16:14:34+00:00

The bug I talked about a little while ago has now also had the fix I wrote committed to the mysql-trunk 5.5.6-m3 repository.

4 Aug 2010

2010-08-04T12:46:11+00:00

A while ago, I reluctantly took over maintaining the get_iplayer tool: http://www.infradead.org/get_iplayer/.

When you search for it by name, my page isn't very high up on the list of results. Hopefully a few syndicated links to it from various planets that carry this will help...

IRC Meet for LUGs

2010-08-04T03:25:15+00:00

Date:

Repeats every month on the last Tuesday until Tue Dec 28 2010 .

Tuesday, 31 August 2010 - 8:00pm - 8:30pm

Tuesday, 28 September 2010 - 8:00pm - 8:30pm

Tuesday, 26 October 2010 - 8:00pm - 8:30pm

Tuesday, 30 November 2010 - 8:00pm - 8:30pm

LUGComms IRC Meeting in #linux-aus-lugs on freenode.

A chance for organisers and members of Linux Groups around Australia to discuss issues, share ideas and lead their teams.

Customizing the RPMs from pgrpms.org

2010-08-04T02:00:23+00:00

To pick up where Devrim left off in customizing RPMs, here are some more tips for getting your very own RPMs built:

Create a VM with your favorite operating system (I’m using versions of CentOS). I need both 32-bit OS and 64-bit OS. This is much easier to manage with separate, local VMs.
Install spectool (available here), and SVN
The other dependancies were: gcc glibc-devel bison flex python-devel tcl-devel readline-devel zlib-devel openssl-devel krb5-devel e2fsprocs-devel libxml2-devel libxslt-devel pam-devel
Edit the postgresql-$VERSION.spec file to your liking: If you’re adding patches, you need to add them in TWO places – first in the Patch#: group, and then again below where the %patch# series starts. Finally, if you’re adding an entirely new package (say in 8.2, pg_standby in contrib), you’ll need to also add the binary (or library, or whatever) to the appropriate %files clause later in the spec file. It’s also a good idea to modify ‘Release’. Here’s a sample diff of my spec file:

--- postgresql-8.2.spec (revision 188) +++ postgresql-8.2.spec (working copy) @@ -74,7 +74,7 @@ Summary: PostgreSQL client programs and libraries Name: postgresql Version: 8.2.17 -Release: 1PGDG%{?dist} +Release: 1test%{?dist} License: BSD Group: Applications/Databases Url: http://www.postgresql.org/ @@ -95,7 +95,9 @@ Patch4: postgresql-test.patch Patch6: postgresql-perl-rpath.patch Patch8: postgresql-prefer-ncurses.patch +Patch7: postgresql-pgstat-dir.patch Patch9: postgresql-use-zoneinfo.patch +Patch10: pg_standby.patch

Buildrequires: perl glibc-devel bison flex
Requires: /sbin/ldconfig initscripts
@@ -282,7 +284,9 @@
%patch4 -p1
%patch6 -p1
%patch8 -p1
+%patch7 -p1
%patch9 -p1
+%patch10 -p1

pushd doc
tar -zcf postgres.tar.gz *.html stylesheet.css
@@ -604,6 +608,7 @@
%{_bindir}/pg_controldata
%{_bindir}/pg_ctl
%{_bindir}/pg_resetxlog
+%{_bindir}/pg_standby
%{_bindir}/postgres
%{_bindir}/postmaster
%{_mandir}/man1/initdb.*

How have you customized RPMs using this repo? Share your .spec files!

Complete cover

2010-08-03T22:06:48+00:00

The folk at No Starch have done a fantastic job with the cover! The text color on the front got tweaked a little, and now we have the spine and back cover design as well. Here's the complete cover.

Front cover photo: Rob Suisted
Back cover photo: Lynley Cook
Cover Design: Octopod Studios

HailDB 2.0.0 released!

2010-08-03T00:52:59+00:00

(Reposted from the HailDB Blog. See also the announcement on the Drizzle Blog.)
We’ve made our first HailDB release! We’ve decided to make this a very conservative release. Fixing some minor bugs, getting a lot of compiler warnings fixed and start to make the name change in the source from Embedded InnoDB to HailDB.

Migrating your software to use HailDB is really simple. In fact, for this release, it shouldn’t take more than 5 minutes.

Highlights of this release:

A lot of compiler warnings have been fixed.
The build system is now pandora-build.
some small bugs have been fixed
Header file is now haildb.h instead of innodb.h
We display “HailDB” instead of “Embedded InnoDB”
Library name is libhaildb instead of libinnodb
It is probably binary compatible with the last Embedded InnoDB release, but we don’t have explicit tests for that, so YMMV.

Check out the Launchpad page on 2.0.0 and you can download the tarball either from there or right here:

haildb-2.0.0.tar.gz
MD5: 183b81bfe2303aed435cdc8babf11d2b
SHA1: 065e6a2f2cb2949efd7b8f3ed664bc1ac655cd75

Last changes today

2010-08-02T22:57:00+00:00

The release of Linux 2.6.35 necessitated what is hopefully the last change I make to the book (in the preface). Files should go the printer this week!

unittest API, part 2

2010-08-02T12:15:15+00:00

In part 1 of this humble attempt to document the interfaces and contracts that unittest actually cares about, we talked about TestSuite and TestCase, how they both implement a common interface that's used for running tests, ITest and how they each implement their own interfaces, ITestSuite and ITestCase.

Now we're moving on to a much more complicated object, TestResult, to see how we can pick apart the ways it interacts with the rest of the system.

TestResult

A TestResult object is all about dealing with the results of tests, as you might expect. However, it doesn't generally represent a single test result. You could say it represents the results of a number of tests, but I don't think that's terribly helpful.

Better to think of a TestResult object as an event handler. A TestResult object receives events from a test run and then does something with them.

Just as TestCase has a two-faced nature, presenting one interface to the testing framework and another to test authors, so to TestResult can be thought of has having many interfaces:

Its interface to a TestCase. This can be thought of as the test event handling interface
A result querying interface, normally used by a test runner
An interface for events that come from the test runner, the runner event handling interface.
An execution control interface.

Note that the result querying interface and the runner event handling interface together make up the interface between the TestResult and test runner.

Let's start with the test event handling interface. The methods below are the interface between TestCase.run() and TestResult. (I guess TestCase.debug too, but no one cares about it).

startTest(test): Called when test commences running. Although not enforced, it's impolite to provide any results for test before calling this.
stopTest(test): Called when test is completely finished. Although not enforced, it's impolite to provide any more results for test after calling this, unless you call startTest(test) again first.
addSuccess(test): Called when test has been shown to be successful. The default implementation does nothing.
addError(test, err): Called when test raises an unexpected error. err is a tuple such as you might get from sys.exc_info(). Calling this method for the first time must change the result of wasSuccessful().
addFailure(test, err): Called when test has failed one of its assertions. err is a tuple such as you might get from sys.exc_info().

The above interface is tightly coupled to the implementation of TestCase.run(). In particular, if you wish to add more kinds of results to your testing framework ("skip" results are a fairly common addition), then you must change both TestCase.run() and the TestResult interface.

If you do something like that, I recommend making sure that your modified TestCase can handle TestResult objects that do not provide the extensions to the interface that you need. One common way of doing this is to have the TestCase fall back to the primitive result types, e.g. "skip" might become "success" for a TestResult that doesn't know what skipping means.

Importantly, the interface between TestCase and TestResult has been fattened in Python 2.7.

addSkip(test, reason): Called when test is skipped. reason is a string explaining why the test was skipped.
addExpectedFailure(test, err): Called when test failed in a way that was expected. err is a tuple such as the one returned by sys.exc_info().
addUnexpectedSuccess(test): Called when test was expected to fail, but didn't.

The following interface is a way of learning about test results after they have happened, the result querying interface, and is part of the contract between the test runner and the TestResult.

wasSuccessful(): If there have been no errors and no failures, return True. Return False otherwise.
testsRun: An integer that is the number of tests that have been run.
errors: A list of tuples of (test, error_message) for all of the tests with unexpected errors, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.
failures: A list of tuples of (test, error_message) for all of the failing tests, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.

And of course, Python 2.7 fattens this interface again to have the following:

skipped: A list of tuples of (test, reason) for all of the skipped tests, where test is an ITestCase and reason is a string suitable for display to humans, generally containing a traceback.
expectedFailures: A list of tuples of (test, error_message) for all of the tests that were expected to fail and failed in the manner they were expected to, where test is an ITestCase and error_message is a string suitable for display to humans, generally containing a traceback.
unexpectedSuccesses: A list of all of the tests that unexpectedly succeeded. Members of the list are ITestCases.

In Python 2.7, TestResult also extended its interface to the test runner beyond simple result querying and into allowing the test runner itself to send two very important events to the TestResult, behold the runner event handling interface:

startTestRun(): Called before any tests have been run. It is impolite to provide any test results before calling this.
stopTestRun(): Called after all the tests have finished running. It is impolite to provide any test results after calling this. A TestResult object is generally not expected to handle any events at all after this method has been called.

Some test runners rely on TestResults to use those events to display the results to the user. These runners frequently do not use the result querying part of the interface.

There is one more interface that TestResult implements: the execution control interface:

stop(): Signal that the execution of further tests should stop now. Sets shouldStop to True.
shouldStop: If True, then test execution should stop. TestSuite.run() should monitor this value and stop execution if ever it is True.

This interface is mostly used as a way of handling KeyboardInterrupts cleanly.

Summary

If you want your TestResult object to work with standard Python TestCase objects, or any TestCase objects that try to stick close to the standard, then you must provide the test event handling interface described above. If you are writing your own test framework or test runner, you care about this, because you want to run everyone's unit tests.

If you want your TestResult object to work with the standard Python test runner before Python 2.7, then you must provide the result querying interface. If you are using the standard Python test runner, you care about this. For Trial or testtools, you must provide the runner event handling interface. For anything else, I'm afraid you are on your own.

Always provide the execution control interface.

Comments

In this documentation, I've been trying to describe the various interfaces without inserting too much of my own opinion about their design. However, I think some commentary might actually help to make things easier to understand.

By providing a querying interface for TestResult to be used by a test runner, the original designers of unittest practically insisted that responsibility for displaying the results of a test run be split between two different classes. The TestResult takes care of displaying incremental feedback from the running tests and the test runner takes care of displaying the summary. You can see evidence of this design in Python 2.6's unittest.py, where there's a hidden _TextTestResult subclass which has extra methods that are called only by a special TextTestRunner.

The addition of startTestRun() and stopTestRun() mean that now a TestResult object can be fully in charge of displaying its results. As such, providing a query interface and exposing details like the list of test failures somewhat vestigial.

I'm less happy with this post than the previous one. As such your critique is even more welcome.

Still to come: the interface for test authors and just what is a test runner anyway?

Update: Remove ambiguity in expectedFailures description (see comments). Thanks Aaron.

Installing NCARG and NCL on a 64-bit AMD Opteron Cluster running CentOS 5 Linux

2010-08-02T05:22:38+00:00

NCAR is the (US) National Centre for Atmospheric Research, NCARG is a graphics library, maps database and applications for the same, and NCAR Command Language (NCL) is interpreted language for the above. Installing the software from source is not recommended: "We highly recommend that you use a precompiled NCL binary rather than attempting a build from source code....

Show and tell

2010-08-01T22:42:32+00:00

OSSAT (that is, Open Source Show and Tell) events have been happening in London on a 3-6 monthly basis since 2008. The events are a showcase for people working with open source, open standards, open data to tell people about what they’ve been up to and share what’s great about it.

Now it’s Wellington’s turn. Grant McLean has been busy setting up WOSSAT – Wellington open source show and tell – and he’s written a blog post all about it.

Get involved: follow @wossat on twitter or identi.ca; even better, if you’ve got a story that you’re itching to tell, get in touch with Grant.

Pinnacles Tramp

2010-08-01T11:10:25+00:00

Took Friday off and tramped up to the Pinnacles in the Kauaeranga valley in the Coromandel. The Pinnacles Hut is handy to Auckland being couple of of hours drive away. This was the fifth time I’ve been up there (though one time was just a day trip).

Tramped Tarawaere track which gets a bit rough and isn’t the usual way of getting to the hut, but is much more interesting. Having started late, I ended up tramping in the dark by headlamp for the last part along Billygoat track. It is interesting how the colour gradually fades and the shadows get deeper and deeper, but you hold off using a torch because once you do all that is visible is the spot it points at and that lacks depth (is that a hole or a rock?). At least that is how it is with my torch, maybe I need a better one.

Got to the hut about 7pm to find I was the only one staying there that night. And I had been worried about crowds. Slept in the dining area with the fire going.

Next day I climbed up the Pinnacles themselves. DoC have added new steps and ladders, it’s all a lot easier than the mud and rock handholds, I’m not complaining. Got a nice view of the inside of a cloud there. Was hoping it might clear, but I guess I’ve seen the views before.

Came back down via Webb Creek. So many stairs! But nice in a ferny mossy waterfall way. Then back to the car and drove home to soak tired legs.

More photos.

Life after Weta?

2010-07-31T17:11:32+00:00

I keep forgetting that we aren't all connected to the same hive mind and am surprised when friends don't know that I have left Weta (again). Everybody asks why I chose to leave, and the answer is simple. Despite the fact that I adore my job, doing it left me with very little emotional energy to do anything else. I realised that for most of my adult life, work really has been my life. For a long time that was fine because I loved my work and was happy to make it my life. However with each passing year that compromise has seemed less acceptable. With "Avatar" wrapping up, and especially now that I'm single again, it felt foolish to not take a punt and try something different.

Of course this leaves the question of what exactly is the next step, and honestly the answer is that I don't know. What I do know is I've been talking about travelling the world for longer then I can remember. As a kid, shuttling between California and New Zealand with my parents, I remember idolising the scruffy hippy kids that I'd see congregating in the corners of airports. Over the last <gulp> twenty years I've ended relationships and quit jobs to head off on this journey more times then I care to count. Yet I always allowed life to pull me back before the final plunge. The time has arrived to stare the dream square in the face and either plunge in … or get over it.

What I do know is that I'll be spending the next couple months in Wellington enjoying summer and the company of friends. Then around April I'll be heading off towards Southeast Asia. My destination? The cheapest flight that gets me to Southeast Asia, I'm guessing Bangkok but hope to be surprised. From there I intend to slowly make my way westward. Since I'm not in a hurry I intend to stop for as long as I like and enjoy everything which seems worth savouring. I'll be avoiding planes and will instead travel by foot, bicycle, motorbike, boat and bus as much as possible. After exploring Southeast Asia I hope to make my way into China and from there to Tibet, down through Nepal and into India. After that I really have no idea, I suspect my initial travel companions will have headed for home by then so I'll be properly on my own. Then depending on how comfortable I am in my new traveller shoes, Africa or Eastern Europe seem the most likely choices.

I've been saying that this is going to be a years trip but really I have no idea. More realistically it'll end when I: get sick of it, run out of money, find something I don't want to leave or end up back in New Zealand.

Between now and then I hope to keep the new travel section of this updated with what I learn and experience along the way.

Staying quit

2010-07-31T17:01:13+00:00

Not everybody smokes in Indonesia, but it seems like everybody does. Pinched between gnarled fingers or hanging carelessly from lips — stall owners, bikers and touts all peer out at the world through clouds of smoke. Within hours of landing the cloying smell of clove cigarettes has jolted me back back ten years, memories jostling for attention as they flicker before my eyes. Simultaneously my dormant addiction begins wriggling into consciousness, reaching out into my thoughts, looking for leverage to make "yes" the only answer.

My introduction to clove cigarettes was through Kaleigh and her civilised habit of smoking a single Djarum in the evenings. My ignorance was fascinated and I quickly discovered that lounging in the evening sun, a quiet buzz in hand, was the perfect antidote to whatever stresses the day had delivered. I had not yet learned about addiction, how it waits and yearns, how it lingers in the corners never quite allowing absolution. It is images from this honeymoon period, as my addiction and I become acquainted, which now fill my mind.

Driving with Kaleigh from Seattle to Anchorage over the Alcan Highway. We barely knew each other and were trapped in her jeep, desperately trying to keep the conversation alive through two and a half thousand miles of British Columbia's icy wasteland.

At the end of our relationship, standing on my porch and solemnly swearing off women for the next five years, only to meet Teresa less then a month later. Huddling with her on the back steps of Internet Alaska as she bummed cloves from me, the subzero temperatures providing the perfect excuse to sit close enough to kindle an office romance. I still didn't think of myself as a smoker and I remember my confusion when several weeks later she showed up on the steps with a pack of Camels. I discovered that she was an ex-smoker, and she discovered that cloves are three quarters tobacco.

Watching our friend Mark, as he watched Teresa and I, transition from smoking cloves to smoking Camels. No words were spoken but I understood what he had recognised, and why he no longer joined us for the occasional smoke.

Moving to Portland and discovering a city I could love, a city I still miss. Those endless hours with other volunteers from Personal Telco, sitting around and dreaming in bars and cafes about how we were going to build a free citywide wireless cloud with nothing but geek fervour. The collapse of the dotcom era, my friends and I falling in and out of employment as the economy heaved and twisted. Being trapped in a job I didn't yet realise I detested, and failing to adjust to the grim realities of an enterprise support role at a doomed startup. It still astounds me that James had the grace to allow me to work four ten hour days, so I could spend a day each week at home working to help Personal Telco's dreams become reality.

In the stinking hot week of Portland's summer, relaxing barefoot in the shade on our front steps. With sudden conviction I knew that I wanted to have kids, that I wanted to get married and that I'd really like for us to own our own home. Then the bitter realisation that despite a lifetime of privilege and opportunity, I had nothing but memories to show for it. In those moments I lost the sense that I had all the time in the world, I robbed myself of the freedom to let every experience unfurl in its own time. Years later I realise that there are many things right with having nothing but memories to show for your life.

The honeymoon ended the first time I tried to quit, and failed, but the memories don't stop.

My first extended travels — three months with Teresa chain smoking through England, France, Holland, Germany, New Zealand and Australia. My failed proposal attempt at the Eiffel Tower and a successful one in the Coromandel. Successfully quitting with Teresa at the end of our travels. Becoming a smoker again through the final months of our marriage and plunging into the abyss of divorce.

Modern etiquette won't let me say that that smoking was a good companion, but it has been a companion. As I write this I'm shocked at how much of my life has been spent as a smoker. How many memories I can conjure of me with a fag in one hand and a coffee or beer in the other.

I remember reading in Wired that a big tobacco executive thought that he was going to get rich creating genetically modified tobacco that didn't have any nicotine. Cigarettes that are just as dangerous but not addictive. I howled with laughter, reading it out loud through snorts of derision, at this gross misunderstanding. They had it completely backwards. What they need to do is create a cigarette which is still addictive, but that doesn't kill you. The addiction is the point.

Smoking provided a reason to get up from my desk and spend ten minutes away from the computer. A reason to meet Teresa on the front porch in the evenings where we would talk and dream about our lives. Rituals full of comfort in cars and airports, coffee shops and bars. A way to be occupied while being alone. And never underestimate the camaraderie of smokers in a world in which they are beset. Each shared cigarette is the beginning of a new friendship, and all smokers share.

For me it's not the quitting which is hard, it's the staying quit. As the challenge of abstinence fades, my vigilance wanes. It didn't take long, maybe a week? A couple beers in Nusa Lembongan and I was bumming smokes from Tania. The surfers in Medewi were more then happy to share their cloves. A week later I succumbed to the pleasures of my own pack, purchased at the Bromo Permai hotel.

Today I'm in Singapore, sitting at a table full of Nick's dive buddies, laughing my ass off as they joke and tease, teaching me swear words in Bahasa, Hokkien and Thai. Everybody at the table is smoking, drinking and laughing but only I know that I'm smoking my last pack. Tomorrow we get on the train for Malaysia, and in these final hours each one tastes sweeter.

Can Melting Update

2010-07-31T10:04:00+00:00

Using the bottom of a MAPP gas can, which is larger and sturdier than a coconut cream tin, I've managed to melt and cast 20 cans at a go - and I still have a lot of my $6 bag of BBQ charcoal left.

When you get enough in the can, the aluminium squeezes out of the dross easier and you can pour it directly into a mould.

30 Jul 2010

2010-07-30T15:03:02+00:00

Got bored of having to run 'make install' when hacking on Evolution, partly because libtool insanity makes it take too long — as for some reason it relinks everything as it installs it. Perhaps that was needed for FORTRAN77 programs on OSF/1, but it isn't needed on my modern Linux system. I hate libtool. But even without that, re-running 'make install' every time you change a line of code is a pain.

For a while I took to manually symlinking the libraries and executables I was working on, from my build directory into their installed locations. But I kept missing some out and that was a pain too.

My current solution, which excited mbarnes sufficiently that I felt I ought to share it more widely, is to re-run autogen.sh with the --enable-fast-install argument, then build it and run 'make INSTALL=install_symlink.sh install'. Then all files get installed as symlinks instead of being copied, and all I have to do is hack code, type 'make', and run evolution again.

The script is a dirty hack and there are much better ways to do it — some of which would even cope with filenames that have spaces in. But it works for me, and makes Evolution hacking a little easier.

Installing GCC on a cluster running Linux

2010-07-29T23:37:05+00:00

The GNU Compiler Collection (usually shortened to GCC) is a compiler system for C (gcc), C++ (g++), Java (gcj), Ada (GNAT), Objective-C (gobjc), Objective-C++ (gobjc++), and Fortran (gfortran). Circumstances will arise when you will want to use a particular version of GCC rather than the system version.

Set the environment variables manually for GMP, MPFR and MPC. Or use modules.

module load gmp/4.3.1 module load mpfr module load mpc

540W of LED rope for monorail track

2010-07-29T22:21:29+00:00

Testing how much power the rope will pull off the generators

Index

2010-07-29T20:02:00+00:00

We're getting close now. The index is nearly done. It has about 5900 entries, with about 9600 locators. (I've rounded the numbers because they'll probably change a little as we do some fine tuning.) We're typesetting the index small (8.5 point), but it's still going to run to 60 pages. Current target is for files to go to the printer in about a week, and for the printing to be finished near the end of August, which means available in September.

v0.21 released

2010-07-29T20:00:08+00:00

It’s been a while, but v0.21 is ready. Most of the work this time around has been on stability. There is one key new feature, however: RBD, the rados block device, which let you create a virtual disk backed by objects stored in the Ceph cluster. The images can be mapped natively by the ceph kernel module or via a driver in qemu/KVM. Although neither of those drivers is upstream yet, the server side functionality and admin tools are in place.

Changes since v0.20 include:

improved logging infrastructure
log rotate
mkfs improvements
rbd tool, and rados class
mds: return ENOTEMPTY when removing directory with snapshots
mds: lazy io support (experimental)
msgr: send messages directory to connection handles (more efficient)
faster atomic_t via libatomic-ops
mon: recovery improvements, fixes (e.g. when one mon is down for a long time)
mon: warn on monitor clock drift
osd: large object support
osd: heartbeat improvements, fixes
osd: journaling fixes, improvements (bugs, better use of direct io)
osd: snapshot rollback op (for rbd)
radosgw fixes, improvements
many memory leaks and other bugs fixed

The project roadmap has been updated and is available via the issue tracker.

Relevant URLs:

Direct download at http://ceph.newdream.net/download/ceph-0.21.tar.gz
For Debian packages see http://ceph.newdream.net/wiki/Debian

unittest API, part 1

2010-07-29T17:56:11+00:00

It's a little known fact, but unittest actually has an API.

This isn't the API that you deal with when you write tests, but rather an API that unittest itself uses when running tests. You could think of it as two interfaces: one for test frameworks and one for test authors. Both APIs are real, but both are poorly documented and often misunderstood or abused.

TestCase

An instance of TestCase represents a single test. What you think of as a single test is up to you, but most of the time it's a unit test.

A TestCase object must provide the following methods.

This first list of methods can be thought of as a single interface, which these blog posts will call ITest given the lack of any better name.

countTestCases(): A method that returns the number of test cases this represents. It should always return 1.
run(result=None): Calling this method actually runs the test. result is a TestResult object. run must call result.startTest(self) when it commences running the test and result.stopTest(self) when it is finished. Between these calls it must call a method on result to signal the result of the test. run must never raise an exception, and its return value is ignored. If result is not provided, the TestCase is obliged to make one.
__call__(result): Identical to run(result), provided for backwards compatibility.
debug(): Calling this method runs the test without collecting its results. It may raise exceptions. This method is rarely called by test frameworks.

The following methods are specific to individual test case objects. We call this interface ITestCase.

id(): Should return a string that uniquely identifies the test. For Python tests, the fully-qualified Python name works well. The uniqueness of the id is not enforced.
shortDescription(): Should return a string that describes the test. Many test frameworks use this value to display test results.
__str__: Should return a string that describes the test. Frequently the same as either shortDescription() or id(). Many test frameworks use this value to display test results.

There is also a second interface, one that matters to code that subclasses TestCase. We'll deal with that in a later post.

TestSuite

A TestSuite represents nothing more or less than a bunch of tests.

A TestSuite must provide the ITest interface described above, with the differences that you would expect from something that represents many tests: countTestCases returns the number of tests in the suite; run runs many tests and thus calls result.startTest and kin many times over; debug is the same and can explode anywhere.

One difference is that TestSuite.run must stop running tests as soon as it detects that result.shouldStop is true.

In addition, TestSuite implements the following interface, which I'm giving the completely arbitrary non-existent name of ITestSuite.

addTest(test): Takes an ITest and adds it to the suite.
addTests(tests): Takes an iterable of ITests and adds them to the suite. Normally equivalent to [suite.addTest(test) for test in tests].
__iter__: All test suites must be iterable. Iterating over a test suite yields ITests. These may differ from the ITests provided to addTest and addTests.

In later posts, I hope to document TestResult, the subclassing interface of TestCase and tell you exactly what I think about test loaders, test runners and the like.

I'm blogging this partly because I don't know where else to write this up, but mostly because I need your help to make sure that I'm being clear and correct. Please comment with questions and corrections, and let me know if you find this at all helpful.

Perdition 1.19-rc3 Released

2010-07-29T08:16:17+00:00

I have released version 1.19-rc3 of Perdition. This release fixes several problems. Perhaps most significantly, Thunderbird 3.1 can now re-connect to perdition SSL/TLS.

The code and related libraries are available as tarballs here. More information is available in the announcement email. More information about perdition here.

kexec-tools 2.0.2 Released

2010-07-29T05:01:27+00:00

I have released version 2.0.2 of kexec-tools. the user-space portion of kexec a soft-reboot and crash-dump facility of Linux and Xen.

There are changes since 2.0.2-rc1.

The code is available as a tarball here and in git here. More information is available in the announcement email.

kexec-tools recently moved, so the URLs above are new.

Moved

2010-07-29T03:23:52+00:00

My home page and blog have moved. The new locations are:

Home: horms.org/
Feed: horms.org/pleb_blossom/rss.xml

GTK+ Meeting @ GUADEC 2010 – update

2010-07-28T12:42:58+00:00

since I’m stupid and I scheduled the team meeting before I could actually be in The Hague, I had to move the meeting. and, again, since I’m stupid I managed to schedule it against important talks of the day — and my own.

so: the GTK+ team meeting has been moved to the common area after the last talk of Wednesday, July 28th. sorry for the inconvenience I caused, and please don’t hurt me when you see me.

28 Jul 2010

2010-07-28T08:08:19+00:00

Things I hate today include:

Symbian on my Nokia N97 — for spontaneously rebooting as soon as I got off the ferry.
Google Maps — for not caching the map tiles I'd carefully downloaded while I was on the free ferry wireless, showing my route to the hotel.
Mobile phone networks — for the insane amount of money it will have cost me to re-download the same map tiles again, as I was driving.

It's almost as if it's a conspiracy — especially between the latter two.

I really need to get myself an N900 and start using maemo-mapper again. Every time I try to use non-free software, it hurts.

Drupal Bookmarks Module

2010-07-28T02:10:41+00:00

When developing a website, and in particular an intranet, for an organisation there is often a debate on what should be on the frontpage.

Different departments and different individuals all seem to have "strong opinions" on the matter.

HOWTO screw up launching a free software project

2010-07-27T18:40:54+00:00

Josh Berkus gave a great talk at linux.conf.au 2010 (the CFP for linux.conf.au 2011 is open until August 7th) entitled “How to destroy your community” (lwn coverage). It was a simple, patented, 10 step program, finely homed over time to have maximum effect. Each step is simple and we can all name a dozen companies that have done at least three of them.

Simon Phipps this past week at OSCON talked about Open Source Continuity in practice – specifically mentioning some open source software projects that were at Sun but have since been abandoned by Oracle and different strategies you can put in place to ensure your software survives, and check lists for software you use to see if it will survive.

So what can you do to not destroy your community, but ensure you never get one to begin with?

Similar to destroying your community, you can just make it hard: “#1 is to make the project depend as much as possible on difficult tools.”

#1 A Contributor License Agreement and Copyright Assignment.

If you happen to be in the unfortunate situation of being employed, this means you get to talk to lawyers. While your employer may well have an excellent Open Source Contribution Policy that lets you hack on GPL software on nights and weekends without a problem – if you’re handing over all the rights to another company – there gets to be lawyer time.

Your 1hr of contribution has now just ballooned. You’re going to use up resources of your employer (hey, lawyers are not cheap), it’s going to suck up your work time talking to them, and if you can get away from this in under several hours over a few weeks, you’re doing amazingly well – especially if you work for a large company.

If you are the kind of person with strong moral convictions, this is a non-starter. It is completely valid to not want to waste your employers’ time and money for a weekend project.

People scratching their own itch, however small is how free software gets to be so awesome.

I think we got this almost right with OpenStack. If you compare the agreement to the Apache License, there’s so much common wording it ends up pretty much saying that you agree you are able to submit things to the project under the Apache license. This (of course) makes the entire thing pretty redundant as if people are going to be dishonest about submitting things under the Apache licnese there’s no reason they’re not going to be dishonest and sign this too.

You could also never make it about people – just make it about your company.

#2 Make it all about the company, and never about the project

People are not going to show up, do free work for you to make your company big, huge and yourself rich.

People are self serving. They see software they want only a few patches away, they see software that serves their company only a few patches away. They see software that is an excellent starting point for something totally different.

I’m not sure why this is down at number three… it’s possibly the biggest one for danger signs that you’re going to destroy something that doesn’t even yet exist…

#3 Open Core

This pretty much automatically means that you’re not going to accept certain patches for reasons of increasing your own company’s short term profit. i.e. software is no longer judged on technical merits, but rather political ones.

There is enough politics in free software as it is, creating more is not a feature.

So when people ask me about how I think the OpenStack launch went, I really want people to know how amazing it can be to just not fuck it up to begin with. Initial damage is very, very hard to ever undo. The number of Open Source software projects originally coming out of a company that are long running, have a wide variety of contributors and survive the original company are much smaller than you think.

PostgreSQL has survived many companies coming and going around it, and is stronger than ever. MySQL only has a developer community around it almost in spite of the companies that have shepherded the project. With Drizzle I think we’ve been doing okay – I think we need to work on some things, but they’re more generic to teams of people working on software in general rather than anything to do with a company.

Unexpected Downtime

2010-07-27T05:39:54+00:00

I seem to have picked up some sort of virus. I'm mostly offline while I wait for this to clear up. Please forgive any delays...

Python 3

2010-07-27T00:40:28+00:00

I would be much more sympathetic to the whole Python 3 endeavour if they had made a serious effort to keep the major 2.x releases mutually compatible.

Installing LAMMPS on a 64-bit AMD Opteron Cluster running CentOS 5 Linux (again)

2010-07-26T06:28:12+00:00

LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator) is used to model atoms or as a parallel particle simulator at the atomic, meso, or continuum scale. It has potentials for soft materials (biomolecules, polymers) and solid-state materials (metals, semiconductors) and coarse-grained or mesoscopic systems. You may remember a difficult install from the past.

In stillness …

2010-07-25T09:44:06+00:00

Bending into Padahastasana, the energy courses through my legs. In sympathetic unison the crickets count the seconds away.

Listening to the drone of my teachers voice, I feel a gecko scurry over my toes to hunt for moths in the glare of the lights.

Looking up from my book, I see ochre tinged cliffs melting into the jungle below. Fork-tailed swallows flit through the skies chasing insects and cuddle on the roof, sharing chirps of intimacy.

Drifting in the pool, I strain to float effortlessly. Above me, in a haze of yellow and black butterflies, swarms of red dragonflies zigzag across the water.

Playing on the beach, we are joined by two young children. They howl in Thai as we skim a frisbee over their heads and laugh with unspoken pride when they pry the disk from the sky.

Hanging out in an empty pub, we make aimless conversation with the owner. As we laugh and talk a party slowly forms around us, John and Boi on the guitar, Kris and I singing.

Struggling to shape my thoughts into words, I'm approached by a young Indian boy who stares longingly at my computer. To his astonishment I don't have any games installed, instead we entertain ourselves by taking silly pictures while his father looks on and laughs.

Watching children play in the distance with a broken surfboard in the sea. Skimming it across the wash they shriek with laughter each time one of them gets catapulted into the air by an incoming wave.

Sitting, facing north, I close my eyes. I can hear the wind in the trees and feel the beat of my heart as I begin to count my breath.

Cans to Metal

2010-07-24T04:14:00+00:00

I was always told that the first thing you need to know about melting aluminium cans is that you don't really want to melt aluminium cans; it's too hard. Find other chunkier scrap aluminium instead. But I'm a sucker so I did it anyway.

The big problem is that there is a lot of surface area on the cans, and that this encourages the aluminium to oxidise. Oxidised aluminium holds on to the liquid aluminium like a sponge, so unlike when you melt lead or pewter the metal does not all automagically drop to the bottom. It needs fluxes, chemicals, special furnaces and the like. Yeah, right.

So, take a load of heatproof bricks or pavers and in a stable location away from flammable stuff construct a sturdy furnace. You'll note that mine is built from ordinary pavers on a mud bank, surrounded by foliage, and looks like it'll collapse at any moment. The trick is to have a long tunnel going into the bottom of the furnace, with a little doorway on top to allow you to put things in and take them out. The chimney takes the sparks up and away from your face and hair, and needs to be narrower than the furnace bottom.

Down the tunnel, you direct a hair-dryer, blower, hot-air stripper (good idea Forrest!), powerful fan or similar. This encourages flames and sparks from ordinary BBQ charcoal at the bottom of the furnace. Do not place the fan up against the tunnel or it may get flamed. If I need to tell you to keep a bucket of water and hose at hand, and warn you not to wet the fan you really shouldn't be doing this.

Back to the cans. Flatten these as much as possible and pack the flattened remains hard into a steel can. This steel can lasts only one burn, so pack in as much as you can.

Now light the furnace. A couple of lit newspaper balls in the tunnel generally does the trick, or poke a blowtorch in there. Once you turn the fan on it's a matter of moments before the whole thing flares up. Once it's going turn off the fan, open the door and rearrange the charcoal so you can put the can inside. Put more charcoal against the can, close the door and turn the fan on again. At this point you realise the value of some peep-hole or crack through which you can watch the inferno.

When the can is orange hot the contents miraculously shrink. once everything in the can is a nice, bright orange, turn off the fan and take it out - I use BBQ tongs. Rattle the contents about a bit, then turn it upside-down on a wide fireproof surface. Squish the pile of dross and molten aluminium streams out. Try to round these up into big blobs so they don't oxidise much when you re-melt them.

Now you have reasonably-sized lumps of aluminium that you can do traditional backyard casting with.

Waiting in the rain

2010-07-22T21:58:51+00:00

By 8am this morning, a crowd of 13 people had gathered in the drizzle outside Wellington’s Magnummac store, eagerly awaiting the New Zealand launch of Apple’s iPad.

I decided to ask them why.

“This device is changing the rules of digital engagement”, I couldn’t wait to get my hands on one; I want to be the first person in New Zealand to get one.” The hopeful, but ill-informed, gentleman at the head of the queue told me.

“You know that people have been importing them for months?”

“This device is changing the rules of digital engagement” he replied, effectively ending the interview.

I moved along the queue, stopping in front of a lady who, I quickly noticed, had a tattoo of a heart with an apple in the middle of it on each hand and “A-P-L-E” spelled out across her knuckles. I moved along the queue.

A normal looking gentleman told me: “We want to stand at the intersection of computers and humanism”. I noticed that he was nervously fingering a small white book on which I could make out the title “The Quotations of Chairman Jobs”, and I fled.

As I passed the last person, he grabbed me and confided that, in his inebriated state, he’d thought it was the queue for the soup kitchen. I explained. We fled together along Vivian Street.

Travelling, Speaking, Scaling and Aegiring

2010-07-22T12:23:18+00:00

The next couple of months are going to be a crazy ride. I will be visiting at least 7 countries, speaking on 8 or more days in a 5 week period. The talks will be focused on Drupal and Aegir. My schedule is below.

Horizontally Scaling Drupal - Melbourne

On 7 August I'll be running a 1 day workshop around the theme of horizontally scaling Drupal. The content is built on the knowledge I developed building, deploying and managing around 2100 sites for a client. This event has very limited capacity and has almost sold out.

DrupalCon - Denmark

Denmark is hosting the European leg of DrupalCon this year. I will be attending the full conference. I won't be presenting, but I will be getting involved with some of the BoFs. I had a ball at DrupalCon San Francisco earlier in the year.

Efficiently Managing Many Drupal Sites - Slovakia

After spending a couple of days recovering from DrupalCon, I'll be teaming up with the crew at Sven Creative in Bratslavia, to run a 2 day intensive workshop on horizontally scaling Drupal and development workflows. For more information check out the workshop website.

Free Software Balkans - Albania

On the weekend of 11-12 September, the inaugural Free Software Balkans Conference will be held at the University of Vlore, Albania. I'll be there speaking about Drupal and Aegir. In addition to this I will be running half day build your first Drupal site workshops around the country. The dates and locations for the workshops are still being finalised.

OSI Days - India

On my way back to Australia I will be taking a side trip to Chennai, via Delhi, for OSI Days 2010, Asia's largest open source conference. I will be presenting sessions on Aegir and Drupal. This looks like it will be a huge event.

Other Events

I've launched a new site workshops.davehall.com.au to list my training and speaking engagements. As dates are locked in I'll be adding them to the site.

If you would like to meet with me while I'm on the road, add me to your tripit network, follow me on identi.ca or twitter or add me to your network on LinkedIn.

A tale of a bug…

2010-07-22T00:10:19+00:00

So I sometimes get asked if we funnel back bug reports or patches back to MySQL from Drizzle. Also, MariaDB adds some interest here as they are a lot closer (and indeed compatible with) to MySQL. With Drizzle, we have deviated really quite heavily from the MySQL codebase. There are still some common areas, but they’re getting rarer (especially to just directly apply a patch).

Back in June 2009, while working on Drizzle at Sun, I found a bug that I knew would affect both. The patch would even directly apply (well… close, but I made one anyway).

So the typical process of me filing a MySQL bug these days is:

Stewart files bug
In the next window of Sveta being awake, it’s verified.

This happened within a really short time.

Unfortunately, what happens next isn’t nearly as awesome.

Namely, nothing. For a year.

So a year later, I filed it in launchpad for MariaDB.

So, MariaDB is gearing up for a release, it’s a relatively low priority bug (but it does have a working, correct and obvious patch), within 2 months, Monty applied it and improved the error checking around it.

So MariaDB bug 588599 is Fix Committed (June 2nd 2010 – July 20th 2010), MySQL Bug 45377 is still Verified (July 20th 2009 – ….).

(and yes, this tends to be a general pattern I find)

But Mark says he gets things through… so yay for him.2

How we solve problems

2010-07-21T07:23:01+00:00

This description from Newsweek on what happens in our brains when we're creatively solving problems is wonderful. I don't think I could have come up with these words but they describe exactly what it feels like to me when I'm involved in difficult problem solving.

When you try to solve a problem, you begin by concentrating on obvious facts and familiar solutions, to see if the answer lies there. This is a mostly left-brain stage of attack. If the answer doesn’t come, the right and left hemispheres of the brain activate together. Neural networks on the right side scan remote memories that could be vaguely relevant. A wide range of distant information that is normally tuned out becomes available to the left hemisphere, which searches for unseen patterns, alternative meanings, and high-level abstractions.

Having glimpsed such a connection, the left brain must quickly lock in on it before it escapes. The attention system must radically reverse gears, going from defocused attention to extremely focused attention. In a flash, the brain pulls together these disparate shreds of thought and binds them into a new single idea that enters consciousness. This is the “aha!” moment of insight, often followed by a spark of pleasure as the brain recognizes the novelty of what it’s come up with.

Now the brain must evaluate the idea it just generated. Is it worth pursuing? Creativity requires constant shifting, blender pulses of both divergent thinking and convergent thinking, to combine new information with old and forgotten ideas. Highly creative people are very good at marshaling their brains into bilateral mode, and the more creative they are, the more they dual-activate.

Coding Style As A Failure Of Language Design

2010-07-21T04:41:17+00:00

Variance in coding style is a huge problem. Reading code where the style varies all over the place is painful. Moving code from one place to another and having to restyle it is awful. Constantly adjusting the style in which you're writing code to conform to the local style of the project, module, file, function or line you're modifying is miring.

Therefore projects adopt style rules to encourage and enforce a uniform style for the project's code. However, these rules still have to be learned, and adherence to them checked and corrected, usually by humans. This takes a lot of time and effort, and imperfect enforcement means code style consistency gradually decays over time. And even if it were not so, code moving between projects looks out of place because style rules are rarely identical between projects --- unless you reformat it all, in which case you damage the relationship with the original code.

I see this as a failure of language design. Languages already make rules about syntax that are somewhat arbitrary. Projects imposing additional syntax restrictions indicate that the language did not constrain the syntax enough; if the language syntax was sufficiently constrained, projects would not feel the need to do it. Syntax would be uniform within and across projects, and developers would not need to learn multiple variants of the same language. More syntactic restrictions would be checked and enforced by the compiler, reducing the need for human (or even tool-assisted) review. IDE assistance could be more precise.

Two major counter-arguments arise. People will argue that coding style is a personal preference and therefore diversity should be allowed. This is true if you only participate in particularly small projects, but if you work in a large project then --- unless you are exceptionally fortunate --- you will have to deal with a coding style that is not your preference, no matter what. (Maciej Stachowiak once said that willingness to subjugate one's personal preferences to a project's preferences is a useful barometer of character, and I agree!)

A more interesting counter-argument is that many coding style rules aren't sufficiently formalized so as to be machine-checkable, and might even be very difficult to formalize at all. This is true; for example, line-breaking rules or variable naming rules might be very difficult to formalize. So I relax my thesis to claim that at least those rules which can be formalized should be baked into the language.

(Figuring out exactly which rules can be formalized, and exploring alternative syntax designs that maximize automatic style checkability while still being nice syntax, sound like fun research! Programming language syntax is one of those areas that I think has been greatly under-researched, especially from the HCI point of view.)

pyflakes-doctest

2010-07-20T14:08:10+00:00

Someone at Canonical (I don't know who) wrote something cool once to get Pyflakes (the best Python linter) to run on doctests.

It has recently been deleted from the Launchpad tree, but since it's so useful I thought I'd make it available.

Download pyflakes-doctest whenever you'd like. If you can get it into Pyflakes trunk, then you'll become even more wonderfully, deliciously fabulous and creamy than you undoubtedly are already. Get rid of the Launchpad-specific stuff though.

At OSCON

2010-07-20T02:24:15+00:00

I’m at OSCON this week. Come say hi and talk Drizzle, Rackspace, cloud, photography, vegan food or brewing.

Flying

2010-07-20T00:55:18+00:00

Due to tyre troubles with my other two bikes I was "forced" to ride my Cannondale road bike to work this morning. The first time I've ridden it since well before Hikari was born last November. Wow, that bike is light and voom it flies. (And I put the brakes back together correctly after shipping it from Sydney!)

In semi-related news, its really warm in Tokyo at the moment.

Perks

2010-07-19T21:14:04+00:00

Working on plugin bugs sucks, but sometimes it's fun to watch a video clip over and over again.

Every Single (Awesome) Game I’ve Played

2010-07-19T17:12:04+00:00

Today, I spent a couple of hours thinking back to how awesome the good ol’ days of DOS games were and that now…I hardly play games. Mainly because I find them all pretty damn boring. I thought it’d be awesome to make a 1995 super-computer, say a couple of Pentium Pros and a kick-ass 3dfx Voodoo card (or two), just to play some retro games. Alas, finding rare parts, even on eBay, is rather difficult and it’d probably be a pretty expensive ordeal. Guess I’ll just have to stick with DOSbox and VMware.

On another note, I present you with the list of awesome game I’ve played. You’ll notice how it begins with a shit-ton of DOS games, ventures briefly into Windows, then heads to the PSX, then to WoW, and finally, the world of casual gaming.

The DOS Era (1991 – 1997)
Ancients
Battle Chess 4000
Bio Menace
Blake Stone
Blood
Blues Brothers – Jukebox Adventure
Bouncing Babies
Captain Comic
Carmageddon
Catacombs 3-D
CD-Man
Command & Conquer: Red Alert
Commander Keen 1, 2, 3, 4, 5, Dreams, 6
Companions of Xanth
Cosmo’s Cosmic Adventure
Crysal Caves
Dangerous Dave
Descent, 2
Doom 1, 2
Duke Nukem 1, 2, 3D (Atomic)
Dune
Earthworm Jim
ElectroMan
Epic Pinball
Fantastic Dizzy
Flashback
Freddy Pharkas – Frontier Pharmacist
Fuzzy’s World of Miniature Space Golf
Gertrude’s Secrets
Goldfields
Gorillas (QBASIC)
Grand Theft Auto 1, 2
Heretic, Shadow of the Serpent Riders
Hexen: Beyond Heretic, Deathkings of the Dark Citadel
Hocus Pocus
Hugo’s House of Horrors, Whodunit, Jungle of Doom
IndyCar Racing
Jazz Jackrabbit, 2
Jetpack
Ken’s Labyrinth
King’s Quest 1, 2, 3, 4, 5, 6, 7
Lamborghini: American Challenge
Lemmings
Links: The Challenge of Golf
Lotus III: The Ultimate Challenge
Mario Is Missing
Monster Bash
Need for Speed
Nibbles (QBASIC)
Oddworld: Abe’s Oddysee
One Must Fall: 2097
Prince of Persia
Quake
Raptor
Reader Rabbit
Rise of the Triad
Sango Fighter
Shadow Warrior
Sherlock Holmes: Consulting Detective
Sim City 2000
Sim City Classic
Sim Farm
Stunts
SWIV 3D
Terminal Velocity
The Incredible Machine
Theme Park
Tomb Raider
Turok
Wacky Wheels
Warcraft 1, 2
Wipeout
Wolfenstein 3D (Spear of Destiny)
WordMAX
Worms
Zool
The Windows Era (~1995 – 2004)
3D Movie Maker
3D Pinball for Windows
Age of Empires 1, 2, 3
Ballistics
Battlefield 1942
Catz
Chip’s Challenge
Counter Strike
Crazy Taxi
Daytona USA
Deer Hunter
Destruction Derby
DogDay
Doom 3
Doom 3, Resurrection of Evil
Dope Wars
Elasto Mania
Enter The Matrix
FIFA 97
Flight Simulator 2004
Fury3
Grim Fandango
Half Life
Halo
Heretic II
HeXen II, Portal of Praevus
Hitman: Codename 47, 2: Silent Assassin, Contracts, Blood Money
Hong Kong Mahjong
Hover!
I.G.I.
Jurassic Park: Chaos Island
Kid Pix
Kingpin: Life of Crime
Math Blaster
Math Circus
Max Payne, Max Payne II
MechWarrior 2
Metal Gear Solid 2
Micro Machines V3
Microsoft Golf
MindMaze (that game in Microsoft Encarta!)
Monster Truck Madness
Motocross Madness 1, 2
Myst, Riven, III, IV, V, realMYST
Need for Speed 2
Need for Speed: Underground, Underground 2
Postal, 2
Putt-Putt
Pyst
Quake II, The Reckoning, Ground Zero
Quake III Arena, Team Arena, Quake Live,
Rainbow Six: Rouge Spear
Rally Championship
Red Faction
Redneck Rampage
Sega Rally
Sim Park
SimTower
SkiFree
Soldier of Fortune
Sonic CD
Starcraft, Brood War
SWAT
The Land of Um
The Sims
Theme Hospital
Titanic: Adventure Out Of Time
ToCA Race Driver 2
Typing of the Dead
Unreal, Unreal Tournament, Unreal Tournament 2003, Unreal 2, Unreal Tournament 2004,
Virtua Cop
V-Rally
Where In The World Is Carmen Sandiego?
Yoot Tower
The PSX Era (1999 – 2002)
Ace Combat 1, 2
Colin McRae Rally 1, 2.0
Crash Bandicoot, 2:Cortex Strikes Back, 3:Warped
Croc 1, 2
Die Hard
Dino Crisis
Driver
Gex
Gran Turismo 1, 2
Metal Gear Solid: Integral
Point Blank
R4: Ridge Racer Type 4
Silent Hill
Siphon Filter
Spyro the Dragon
Tenchu
Time Crisis
Wargames: Defcon 1
WoW Era (2005 – 2009)
World of Warcraft, Burning Crusade, Wrath of the Lich King
Post-WoW Era (2008 – 2010)
Colin McRae: DiRT 1, DiRT 2
Counter-Strike: Source
Crysis
Knytt, Stories
Left 4 Dead 1, Left 4 Dead 2
Portal
Team Fortress 2
Unreal Tournament 3
Within A Deep Forest
iPhone Era (2009 – 2010)
Angry Birds
BubblePRO
Doodle Jump
Flight Control
Need for Speed Undercover
Paper Toss
Peggle
Plants vs. Zombies
Rock Band
Spikey’s Bounce Around
Super Monkey Ball 1, 2
Tap Tap Revenge
Other
Donkey Kong Country (SNES)
GoldenEye (N64)
Granny’s Garden (C64)
Guitar Hero (Xbox 360)
Pokemon Red, Blue, Green, Yellow (GB)
Sonic 1, 2, 3, & Knuckles (Mega Drive)
Super Mario World (SNES)
Wayne Gretzky’s 3D Hockey (N64)
Wii Sports (Wii)

I am sure to update this list as I can think of any DOS games that come to mind. I currently have one game racking my brain that I played a ton and just can’t remember the name, sigh.

Summit

2010-07-19T13:07:19+00:00

I'd better say something about the Mozilla summit at Whistler before it recedes into the past...

It was fabulous, of course. The setting, the food, the people, the talks, the work, the demos, the mission, all really good.

Whistler was lovely. It had been cold up to the week we arrived so while it was 30C in the village at times later in the week, 20 minutes in a chairlift brought you up into the snow. And on top of Whistler Mountain the snow banks were six metres deep in places.

I wisely arrived a couple of days early and on the Monday I spent almost the entire day hiking alone up to Singing Pass. Unfortunately I didn't reach Singing Pass --- around the snow line the track was completely blocked by fallen trees (see below). I climbed over a couple of dozen of them but with no end in sight, decided that was a losing proposition and turned back. Nevertheless it was a wonderful time and some solitude was excellent fortification for the subsequent social overload.

I'm not really a party person and working large groups of people is a learned skill for me. It's helpful when I have goals to accomplish and I know a lot of the people. In some ways it wasn't all that different from networking at an academic conference, except we're all on the same side.

The main downside of the summit was that I was completely exhausted during most of it because I was simply too excited to sleep. Talking to people, going to sessions, playing games, and hacking code are irresistible.

Next time we should plan some "work week" days immediately after the summit. This would give us more time to talk in depth --- there were so many people at the summit I wanted to talk to, I couldn't give lots of time to more than a few. It would give us more time to code. We should have done it this time but I wasn't on the ball.

As usual I met many contributors for the first time. That's one of the best things about the summits. A few I really wanted to meet didn't make it --- you know who you are!

I managed to get together with some Mozilla Christians for some prayer time. That was very encouraging.

I saw bears. Woohoo!

This is the track near the head of the valley leading up to Singing Pass. I'm looking back along the track at the trees I've already clambered over! Definitely more trees than track at this point. People told me it was probably the leftovers of an avalanche.

View over Whistler valley from the top of Whistler Peak.

Western Digital to fix Licensing?

2010-07-18T13:50:53+00:00

Over the last few months months I've been corresponding with Dennis Ulrich of Western Digital (WDC) about my concerns with the EULA for the My Book World Edition (MBWE) and their obligations under the GPL. To say it has been a drawn out process is an understatement.

It has taken some time to get WDC to understand the situation. There has been confusing messages about what the situation is with the EULA, the GPL and what license covers what pieces of code. The bottom line is that currently users must check the header of each file to ascertain which license applies to it, even though the downloads are marked as GPL.

Although WDC is moving slowly, they do seem to be commited to making the situation clearer in the next iteration of their MBWE product line. Based on a recent phone call with Dennis, the legal and engineering teams are working together to ensure various licenses complied with and their software engineers are aware of their obligations.

The next version of the MBWE is likely to ship with a revised EULA and properly inform the users of their rights under the GPL. This text is still being developed. At the same time it is still unclear if WDC will backport their changes to their existing NAS products. As it is a simple string change it shouldn't be too hard for them to dedicate a few resources to audit the code and update the strings.

It is unfortunate that WDC appears to have little interest in developing their MBWE product range as a hacker friendly FOSS product. It appears the licensing fix that will be implemented by WDC will more clearly delinate the FOSS and non free components of the MBWE firmware. Clarity is always an important legal consideration, but doesn't help foster a community. WDC seem to have little interest in fostering a hacker community around their products. This is an unfortunate decision by the company.

Many manufacturers of embedded devices only start releasing source for their firmware after being caught out for violating the GPL. WDC is to be commended for complying with the requirements of the GPL from the start. Altough there is no legal requirement for them to make the web gui code and other non free components available, WDC already does. It would be disappointing if they chose to take a backward step and stopped distributing parts of the firmware.

It is possible, and perfectly legal, for WDC to stop distributing the source for the proprietary components. At the same time it would not take much effort for them to release the whole platform as GPL or another FOSS friendly license. WDC is already required to do a code drop every time they release a new version of their hardware or firmware. I suspect it would be faster and easier to push all code to a public git repository than pick through and dump selected components as tarballs on a website.

WDC already have their support team dealing with customer bug reports. Maintaining a mailing list, a bug tracker, a wiki and maybe a public source code repository on somewhere like gitorious is likely to take less than 1 full time employee. The benefit for WDC would be great.

Not only is the hacker community likely to contribute bugs fixes and propose or even develop new features, they can help increase sales. I'm sure the good will generated by the switch to truly open approach to the MBWE product line would outweigh the cost of the additional resources required.

Let's hope Western Digital's fix is a FOSS friendly one. I will post more news as things progress.

PDXPUG Day 2010: 9.0 and all sorts of good stuff

2010-07-18T07:49:14+00:00

I’m nearly ready for PDXPUG Day 2010.

We’ve got some fun talks lined up from leaders in the Postgres community in town for OSCON. It’s free to attend. We’ll at the Oregon Convention Center, room D131.

Stop by sometime during the day if you’re in town!

Here’s our schedule:

10am: What’s new in 9.0 – Selena Deckelmann
11am: ORMs and Their Discontents – Christophe Pettus
1:30pm: Mining Your Logs For Fun and Profit – Josh Berkus
2:30pm: PORTAL – Dan Colish
3:30pm: An Introduction to Managing and Troubleshooting PostgreSQL on Windows – Tim Bruce
4:30pm: PostgreSQL in Brazilian Army and Air Force – Luis Dosso

6pm-10pm: Party at the Gotham Tavern!

16 Jul 2010

2010-07-16T14:29:43+00:00

I've just been working on Evolution's reply code, and have added a couple more of those annoying "nag pop-ups", including this one which I expect a lot of people will appreciate when they don't get the resulting mail:

It's currently set to trigger if you hit 'Reply to All' on a message with more than 15 recipients; unless it's a mailing list message. And of course you can see that it's trivial to turn it off if you never want to see it again.

I've also taken a moment to write down and post some thoughts on the 'Reply to All' vs. 'Reply to List' debate for mailing list messages.

Retained Layers

2010-07-16T04:02:20+00:00

Retained Layers has landed and seems to have stuck.

In a previous post I talked about our layers framework. Up until now we've constructed a new layer tree on every paint. With retained layers, we update our layer tree incrementally as content changes in the window. More importantly, we are able to cache the contents of layers. So for example if you have an element with partial opacity, we can cache the rendering of that element in a layer and every time you paint (perhaps with a different opacity each time), we can paint the cached contents without rerendering the element. This provides noticeable speedups for some kinds of fade effects. When we use layers to render CSS transforms, we'll get speedups there too.

One side effect is that we were able to totally reimplement scrolling. Our current scrolling code relies on shifting pixels around in the window for speed, and that often doesn't work well, for example on pages where scrolling content overlays stationary content or vice versa. It's also prone to visible tearing on some platforms because we can't scroll and repaint in a single atomic paint operation. But now, our retained layer tree effectively contains the entire window contents pre-rendered. We can scroll by just adjusting the offsets of some layers and recompositing the layer tree. (Well, almost ... it's slightly more complicated.) This is a relatively simple approach. It eliminates tearing. It lets us aggressively accelerate the scrolling of nasty scrolling pages with complex contents, because we're able to separate the moving content into different layers from the non-moving content and scrolling becomes a matter of simply repainting the scrolled-into-view strip of the moving content and then blending layers together. We're seeing significant improvements in scrolling many kinds of pages. Soon I hope to blog again with more about how this works, what it can do and what the current limitations are.

Immediate improvements are nice, but the most important benefit of retained layers is that it lays down infrastructure we will be exploiting in all kinds of ways. Our D3D and GL layer implementations benefit from reducing browser rendering and caching more rendered content in layer buffers, since compositing buffers is very cheap with GPUs. With those backends, because scrolling is fully layer-based, it will be accelerated by the GPU! Chris Jones is working on enabling layer composition to be in a dedicated process, at which point we'll be able to scroll without synchronizing with the Web content process --- in particular we'll be able to maintain a constant frame rate of smooth scrolling no matter what the Web content is doing, i.e. super-smooth. This will especially benefit Fennec. Fennec will also benefit because its current tile-caching implementation can be replaced with something layer-based; our layer-based scrolling permits the layer manager to cache prerendered content that's not (yet) visible.

Before we ship the next Firefox release there are a few regressions to work on and some performance and quality knobs to tweak. But I'm feeling quite relieved, since this was one of the long poles for that release and the one I've been most on the hook for.

linux.conf.au 2011 CFP Open!

2010-07-15T00:59:23+00:00

Head on over to http://lca2011.linux.org.au/ and check it out!

You’ve got until August 7th to put in a paper, miniconf, poster or tutorial.

Things I’d like to see come from my kinda world:

topics on running large numbers of machines
latest in large scale web infrastructure
latest going on in the IO space: (SSD, filesystems, SSD as L2 cache)
Applications of above technologies and what it means for application performance
Scalable and massive tcp daemons (i.e. Eric should come talk on scalestack)
exploration of pain points in current technologies and discussion on ways to fix them (from people really in the know)
A Hydra tutorial: starting with stock Ubuntu lucid, and exiting the tutorial with some analysis running on my project.
Something that completely takes me off guard and is awesome.

I’d love to see people from the MySQL, Drizzle and Rackspace worlds have a decent presence. For those who’ve never heard of/been to an LCA before: we reject at least another whole conference worth of papers. It’s the conference on the calendar that everything else moves around.

Proposals submitted...

2010-07-14T23:45:00+00:00

The Linux Conference Australia call for papers is now out, and I've submitted two papers - one for a talk and one for a tutorial. Now the waiting begins...

In 2009 I got accepted to give a talk on writing good user documentation. I'd submitted several papers before then but never got accepted; the chief reason was that I had submitted papers about stuff I was interesed in but was not actually a key contributor to. LCA is crazy hard to get to speak at, but is totally worth it because they really treat speakers well. And to me it's addictive - I loved it so much in 2009 I wanted to do awesome things just to get a place in 2010.

That didn't work out for me; mainly because I'm a neophile. I tend to be interested in a whole bunch of things but only shallowly - occasionally (such as when I decided to write the doco for LMMS) I dip in but I rarely seem to be able to sustain that involvement before the next thing comes along and lures me away. But I'm more hopeful I can get a speakership at 2011 because I'm putting forward two proposals for things that I'm actually really involved in and know about.

Ah well. Now for three months of anticipation. Better keep on working on my electric motorbike then...

Cat Stacking

2010-07-14T20:35:34+00:00

Cat stacking: The heroic practice of coordination or organising ("wrangling") obstreperous or otherwise uncooperative agents into a consistent, coherent pattern or unit e.g. wrestling those time sheets into a semblance of order was an exercise in cat stacking if ever there was one.

Just wanted to make sure that I claimed this term on behalf of myself and my colleagues who (we believe) coined it at Egressive where the term is in common use.

Stupid RCU Tricks: Holding Off RCU Read-Side Critical Sections

2010-07-14T05:34:01+00:00

RCU callbacks are registered via call_rcu(). After an RCU grace period elapses, the callback (which is a C-language function) is invoked. RCU's fundamental guarantee states that once an RCU grace period has elapsed, all RCU read-side critical sections that were executing when the grace period began will have completed. (An RCU read-side critical section is a fragment of code enclosed by rcu_read_lock() and rcu_read_unlock().)

Now, an RCU callback, being a C-language function, has a definite beginning and end. But what about synchronize_rcu(), which blocks until an RCU read-side critical section has elapsed? How does RCU know how long to hold off new RCU read-side critical sections once synchronize_rcu() returns?

“HTML5 Audio And Video Accessibility, Internationalisation And Usability” talk at Mozilla Summit

2010-07-13T19:35:48+00:00

For 2 months now, I have been quietly working along on a new Mozilla contract that I received to continue working on HTML5 media accessibility. Thanks Mozilla!

Lots has been happening – the W3C HTML5 accessibility task force published a requirements document, the Media Text Associations proposal made it into the HTML5 draft as a <track> element, and there are discussions about the advantages and disadvantages of the new WebSRT caption format that Ian Hickson created in the WHATWG HTML5 draft.

In attending the Mozilla Summit last week, I had a chance to present the current state of development of HTML5 media accessibility and some of the ongoing work. I focused on the following four current activities on the technical side of things, which are key to satisfying many of the collected media accessibility requirements:

Multitrack Video Support
External Text Tracks Markup in HTML5
External Text Track File Format
Direct Access to Media Fragments

The first three now already have first drafts in the HTML5 specification, though the details still need to be improved and an external text track file format agreed on. The last has had a major push ahead with the Media Fragments WG publishing a Last Call Working Draft. So, on the specification side of things, major progress has been made. On the implementation – even on the example implementation – side of things, we still fall down badly. This is where my focus will lie in the next few months.

Follow this link to read through my slides from the Mozilla 2010 summit.

13 Jul 2010

2010-07-13T09:29:12+00:00

Yay Brazil!. They're making it illegal to use DRM to prevent "fair dealing" with copyrighted works, or access to works which are in the public domain. It's also legal to "crack" DRM if you're only doing it for the purpose of "fair dealing".

So, for example, it would be legal for me to crack the DRM on the eBooks I buy, which is necessary just so that I can read them. Currently I have to break the law just to be able to buy and use eBooks.

UK citizens, go here and add your vote; it's very simple to register if you haven't already done so.