Cluster computing for Chicken & Ketchup

The most recent episode of my podcast was edited on my new Mac Pro, which is powered by two quad-core Intel Xeon processors. It should be released late tonight or early tomorrow. Since I had the extra computing power, I edited the episode in HDV, instead of DV like the previous episodes. While editing I noticed a huge performance increase: I could work with many more layers in real time, and this was with much higher quality video.

Beyond the lack of performance frustrations during editing, the other big advantage I was looking for in the Mac Pro was a decrease in compression time when I export my final product for the web. The Mac Pro certainly is many times faster than my PowerMac G4 during compression, but, being the status/progress bar addict that I am, I noticed something strange when I opened up the Activity Monitor. The 8 cores weren’t being used to their fullest potential:

So I did 30 seconds of research, and found an article from Bare Feats suggesting that I create virtual cluster on my machine to decrease compression time. Compressor has the ability to split tasks and spread them across a cluster of Compressor instances, which traditionally is made up of multiple computers on a network. But you can also create a cluster of multiple instances on one machine. This is called a “virtual cluster.”

Unfortunately, using this technique I can’t send my Final Cut Pro sequence directly to Compressor. I must first export a movie file, and then send that to Compressor. So, is it worth the extra step for some more efficient use of processor cycles? Let’s find out, with an unscientific test. It’s super-unscientific since I used different clips that weren’t even the same duration, and I don’t know if my sequence was fully rendered before I exported directly from FCP.

But here are the stats, for HDV to 1920×1080 H.264 video:

• Without virtual clustering, sequence with a duration of 8:32 direct from FCP: 50 minutes
• With a virtual cluster of 8 instances, movie file with duration of 7:52: 14 minutes (plus about 2 minutes to export the movie file)

Even with all of the variables I introduced, I believe the advantage of setting up a virtual cluster on a multi-core machine is pretty obvious. Check out the CPU monitor during the clustered processing:

As fascinating as this may be, all of this investigation is time-consuming and often frustrating. And I’m still trying to get this video exported in an HD format that will play on an Apple TV, which is one of the reasons why as of this writing, the episode has not yet been published. So that’s why I’m glad I decided not to become a full-time video editor.