Volume Number: 20 (2004)
Issue Number: 4
Column Tag: Programming

QuickTime Toolkit

by Tim Monroe

Snow Day

It's commonly believed that Apple's iPod portable music player uses a bare-bones operating system developed by a company called Pixo, Inc., and that the device itself is a closed system insofar as only Apple can add software modules -- such as the games or contacts list manager -- to the device. And, it's further believed, even if there were a way to load third-party applications onto the device, actually writing such applications requires access to development tools whose availability is tightly controlled and that are expensive to license. In effect, the iPod is even less open than the original Macintosh computer (which is to say, not very open at all).

In fact, none of these beliefs is true. Once we know what's going on, it's fairly simple to develop quite powerful applications for the iPod using nothing more than a text editor and a programming language that most of us had considered to be long since obsolete. In this article, I'll show just how easy it is to write iPod applications by developing an application that can open and display QuickTime movies on the iPod. In addition, I'll show how to support the standard editing operations (cut, copy, paste, and undo). The result will be an iPod-based version of our QuickTime sample application, QTShell. For reasons that will become clear soon, let's call this application "SnoEez". Figure 1 shows a frame of a movie being displayed by SnoEez. (You may recognize this as a cropped version of our standard penguin movie, with the X-ray video effect applied to the entire movie.)

We'll begin by figuring out which operating system really powers the iPod and which tools we need to use to develop custom applications for it. Then we'll jump into developing SnoEez.

It's easy enough to cast doubt on the idea that the iPod runs an operating system developed by Pixo (which is not, of course, to be confused with Pixar Animation Studios). Pixo is a small company that indeed makes an OS for portable devices like cell phones and PDAs. But it was recently acquired by Sun Microsystems, and it's difficult to imagine that Apple would allow a key piece of technology for one of its most popular and most lucrative products to fall into the hands of a competitor. As we'll see, the iPod does rely on some Pixo software, but that software is just a thin layer of imaging software, not the entire operating system. The Pixo software handles screen drawing and item selection, but not much more. In a nutshell, Pixo's software is to the iPod as QuickDraw was to the original Macintosh: a key element, perhaps, but not nearly the whole enchilada. And, more importantly, the Pixo software in the iPod is likely to be phased out in the future, just as QuickDraw is being phased out in favor of Mac OS X's Quartz imaging layer.

Let's begin by doing a little detective work to figure out what operating system really is powering the iPod. Connect your iPod to your Mac OS X computer and then, in a Terminal window, navigate to the iPod. For instance, if your iPod is named, say, "iDegger", navigate to /Volumes/iDegger. In this directory there is a folder called iPod_Control. List its contents; you'll see something like this:

[Kant:/Volumes/iDegger/iPod_Control] monroe% ls -l
total 8
drwxrwxrwx   4 monroe  unknown  136 Mar  4 13:55 Device
drwxrwxrwx  22 monroe  unknown  748 Feb 26  2002 Music
-rwxr-xr-x   1 monroe  unknown   38 Mar  6 14:23 iPodPrefs
drwxrwxrwx  12 monroe  unknown  408 Mar  6 14:23 iTunes

For present purposes, we want to look at the Device folder. Listing its contents gives us this:

[Kant:/Volumes/iDegger/iPod_Control/Device] monroe% ls -l
total 4
-rwxrwxrwx  1 monroe  unknown  2872 Mar  4 13:55 Prefs
-rwxrwxrwx  1 monroe  unknown   413 Mar  4 05:55 SysInfo

That's not very revealing. But if we list the contents of the Device folder using ls with the undocumented option af (for "all files"?), we get a far different picture:

[Kant:/Volumes/iDegger/iPod_Control/Device] monroe% ls -laf
total 41
-rwxrwxrwx  1 monroe  unknown  2872 Mar  4 13:55 Prefs
-rwxrwxrwx  1 monroe  unknown   413 Mar  4 05:55 SysInfo
drwxr-xr-x  5 root    wheel     190 Sep 22 08:33 bin
drwxrwxrwt  6 root    wheel     204 Jan 29  2003 cores
dr-xr-xr-x  2 root    wheel     512 Mar  6 13:38 dev
lrwxrwxr-t  1 root    admin      11 Mar  6 13:38 etc 
-r--r--r--  1 root    admin  709440 Mar  6 13:38 mach.sym
-rw-r--r--  1 root    wheel  744576 Apr  1 15:21 mach_krn

That's right, the iPod is running a slimmed-down version of Mac OS X! Why am I not surprised?

So far, so good. At this point it might seem like we're done, since all we need to do is compile our application using the standard development tools like Xcode and Interface Builder and then move it into some appropriate location in the iPod. After all, if the iPod is really running Mac OS X, then the executable format is well known (Mach-O).

But of course it's not that simple. The main problem is that the iPod does not have the graphics horsepower to support the Quartz imaging system. That's where the Pixo software comes into play. Pixo provides the imaging system for the iPod, which is a collection of screens of information. Some of these screens present lists, such as the "menu" screens or the lists of songs or playlists; some of these screens present linear controls managed with the iPod's scroll wheel; and some of these screens present raw data (as in Figure 1). So ultimately we need to figure out how to access the Pixo imaging primitives.

First, however, to continue our detective work, let's open one of the existing iPod applications with a hex editor. If we open, say, the Solitaire game, we'll see this stream of hex values:

37 4A 6F 79 21 70 65 66 66 70 77 70 63 60 5C 59 
56 61 4e 56 5f 52 0a 37 60 5b 5c 57 5c 4f 2d 63 
52 5f 60 56 5c 5b 2d 3e 3b 40 0a 37 35 50 36 2d
4e 5d 5d 59 52 2d 50 5c 5a 5d 62 61 52 5f 39 2d
56 5b 50 3b 0a 2d 2d 51 52 53 56 5b 52 35 34 56
5b 56 61 50 4e 5f 51 60 35 36 34 36 2d 47 56 5b
35 56 61 50 4e 5f 51 60 6c 52 5b 51 36 0a ...

In ASCII format, this looks like this:

7Joy!peffpwpc`\YVaNV_R.7`[\W\O-c
R_`V\[->;@.75P6-N]]YR-P\Z]baR_9-
V[P;.--QRSV[R54V[VaPN_Q`5646-G5V
[VaPN_Q`lR[Q6.

The "Joy!peff" at the beginning of the data stream might lead us to believe that the executable format is not Mach-O, but the earlier PEF format adopted from IBM when the PowerPC chips were first used in Macintosh computers. But, once again, we would be wrong. If perchance we are clever enough to subtract 13 from each ASCII value, we get this stream of bytes (splitting the lines whenever we encounter a newline character):

*=bl.cXYYcgcV SOLITAIRE
*SNOJOB VERSION 1.3
*(C) APPLE COMPUTER, INC.
  DEFINE('INITCARDS()') :(INITCARDS_END)

Lo and behold! We've cracked the riddle. The Solitaire game isn't a compiled application at all; rather, it's just a script of commands encoded using a version of the popular rot-13 algorithm. The script language is strongly reminiscent of a venerable programming language called SNOBOL. Judging by the second comment line (introduced with the "*" character), it looks like Apple is using a previously unknown variant of the language called SNOJOB. Clever indeed.

In the interest of saving time, I'll spare you any more tedious reverse engineering and just present the results of my investigations. As best I can tell, SNOJOB is simply classic SNOBOL with bindings to the Pixo imaging system. The SNOBOL language was first developed in the early 1960's by a team of researchers at Bell Laboratories led by Ralph Griswold. The name is an acronym of sorts for "String-Oriented Symbolic Language", which reflects the focus of the language on processing strings and matching patterns. In fact, SNOBOL is one of the few languages ever devised in which patterns (or regular expressions) are full-fledged objects in the language.

We don't have space for a complete investigation of the SNOBOL language, so let's focus on a few salient features. Perhaps the most striking feature of the language is its utter simplicity. It provides no control structures as we would understand them nowadays, and no type declarations. A SNOBOL script is a series of lines, and each line has this structure:

label   expression   branching-info

The label is any arbitrary string that uniquely identifies the line; this label provides a means for other lines in the script (or indeed the same line) to branch to that line. The expression is any legal SNOBOL expression, which always returns a value. The branching-info is interpreted on the basis of that value. If the branching information is of the form ":S(label)", then if the evaluation of the expression is successful, the program continues executing at the line with the specified label. If the branching information is of the form ":F(label)", then if the evaluation of the expression is unsuccessful, the program continues executing at the line with the specified label. Finally, if the branching information is of the form ":(label)", then the program continues executing at the line with the specified label whether the evaluation of the expression succeeds or fails.

Any of the three parts of a line of script can be omitted, and the branching information can include both :S and :F items. If a line contains no branching information, then the program continues executing at the next line. If present, a label must be the first item on the line, and the expression must be indented if no label is present.

The valid expressions include checks for equality like "EQ(N,10)" and calls to user-defined functions. They also include pattern matches, which are indicated by the space character " ". (Yes, white space is an operator in the SNOBOL language. No, I'm not making this up!) So the following expression branches to a line labeled "LN1" if the value of the variable OS_TYPE matches the pattern "Darwin":

   OS_TYPE "Darwin"            :S(LN1)

To match the pattern "Darwin", that value could be "Darwin" or "Darwinian" or the like.

Listing 1 shows a SNOBOL version of the basename function we've encountered in several previous articles. Given a full pathname, it returns the portion of the pathname following the rightmost path separator.

basename
      DEFINE('basename(fName)')                     :(basename_END)
basename
      seg = fName
LP      seg BREAK('/') LEN(1) REM . seg            :S(LP)
      basename = seg                                    :(RETURN)
basename_END

(It's perhaps useful to know that SNOBOL matches the smallest possible string.)

SNOBOL is distinctive among scripting languages in that the SNOBOL interpreter does not first parse a script and then process its input according to the parsed script (as is the case with more common scripting languages like Sed, Awk, or Perl.) Rather, the SNOBOL interpreter just executes each line in sequence, according to the evaluation and branching instructions specified in the script. There is, to repeat, no separate parsing phase. That's why, as you can see, we need to branch unconditionally to the basename_END label after executing the DEFINE command (which declares to the interpreter that "basename" is a user-defined function). Otherwise, the body of the function would be executed immediately, not just when the function is invoked by a line of script.

To read input, we need to make an explicit call to the INPUT command. Listing 2 shows our script's "main event loop", which reads input and processes it.

*      process commands from the standard input
INP   inputLine = INPUT                              :F(END)
*      match the input line 
      inputLine 'doOpen'                             :S(OPN)
      inputLine 'doNew'                              :S(NEW)
      inputLine 'doClose'                            :S(CLS)
      inputLine 'doSave'                             :S(SAV)
      inputLine 'doSaveAs'                           :S(SVA)
      inputLine 'doExit'                             :S(EXT)
      inputLine 'doUndo'                             :S(UND)
      inputLine 'doEdit ' REM . OPERATION            :S(EDT)
      inputLine 'doSelect'                           :S(SLT)
      inputLine 'doAbout'                            :S(ABT)
      inputLine 'openMovie ' REM . ARGS              :S(OPF)
      inputLine 'doButton ' REM . ARGS               :S(BTN)
      
*      nothing matched; go get another line....      
                                                     :(INP)
*      handle the commands matched just above
OPN   doOpen()                                       :(INP)
NEW   doNew()                                        :(INP)
CLS   doClose()                                      :(INP)
SAV   doSave()                                       :(INP)
SVA   doSaveAs()                                     :(INP)
EXT   doExit()                                       :(INP)
UND   doUndo()                                       :(INP)
EDT   doEdit(OPERATION)                              :(INP)
SLT   doSelect                                       :(INP)
ABT   doAbout()                                      :(INP)
OPF   ARGS BREAK(' ') . FILE LEN(1) REM . ISNEW
+      openMovie(FILE, ISNEW)                        :(INP)
BTN   ARGS BREAK(' ') . WINDOW LEN(1) REM . BUT
+      doButton(WINDOW, BUT)                         :(INP)
END

You'll notice that we read a line of input, match it against patterns we know how to process, and then branch unconditionally back to the INP label, which reads another line of input. Only when we encounter an error reading input or when some user-defined function branches to the END label does the script stop processing. (By the way, the "+" character as the first character on a line indicates a continuation line.)

But where does the input for SnoEez come from? As we'll see shortly, it's generated by SnoEez itself, when handling user actions. First however let's see how to construct our application's user interface.

An application running on the iPod consists of a series of screens. An application's main screen is the first screen displayed when the application is launched. Figure 2 shows the main screen of SnoEez.

This main screen is also a menu screen, meaning that it consists of a list of items that can be selected using the iPod's scroll wheel and selection button (the button in the middle of the scroll wheel). We can define a main screen using the Pixo function NEW_SCREEN like this:

      main = NEW_SCREEN(kMenuType, "SnoEez", kMainScreen)

We add items to a menu screen using the ADD_ITEM function. SnoEez' main screen is constructed using these lines of code:

      ADD_ITEM(main, "File", kNoFunction, file)
      ADD_ITEM(main, "Edit", kNoFunction, edit)
      ADD_ITEM(main, "About", kNoFunction, help)

Here the fourth parameters are identifiers for child screens (or subscreens), which must have been defined before we call ADD_ITEM. We can construct these subscreens like this:

      file = NEW_SCREEN(kMenuType, "SnoEez \u401 File", main)
      edit = NEW_SCREEN(kMenuType, "SnoEez \u401 Edit", main)
      help = NEW_SCREEN(kTextType, "About SnoEez", main)

As you might surmise, the fourth parameter here is the parent screen, which is the screen selected when the user presses the Menu button on the iPod.

Notice that the help screen is declared to be of type kTextType. This indicates that that screen consists solely of text. We can add text to the help screen with this code:

      ADD_TEXT(help, "A QuickTime\nmovie player and 
+ editor\nbuilt with SNOJOB\n\n\u169 2004 Tim Monroe")

This gives us the About screen shown in Figure 3. ADD_TEXT understands C-style escape sequences (such as "\n") and Unicode escape sequences (such as "\u169" for the copyright symbol "(c)").

It's easy enough to populate the File and Edit screens (Figures 4 and 5). The only new thing to learn is that we need to specify the name of the function to call, along with any parameters, when a screen item is selected. For instance, we can add a New item to the File screen like this:

      ADD_ITEM(file, "New", "doNew", kNoChildScreen)

When the user selects the New item in the File screen, SNOJOB interpreter passes the string "doNew" back to the application as part of the standard input. As we saw in Listing 2, our script matches that string and calls the doNew function.

Listing 3 shows the complete definition of the setUpScreens function.

setUpScreens
      DEFINE('setUpScreens()')                  :(setUpScreens_END)
setUpScreens
*      create a new main screen
      main = NEW_SCREEN(kMenuType, "SnoEez", kMainScreen)
      file = NEW_SCREEN(kMenuType, "SnoEez \u401 File", main)
      edit = NEW_SCREEN(kMenuType, "SnoEez \u401 Edit", main)
      help = NEW_SCREEN(kTextType, "About SnoEez", main)
      ADD_ITEM(main, "File", kNoFunction, file)
      ADD_ITEM(main, "Edit", kNoFunction, edit)
      ADD_ITEM(main, "About", kNoFunction, help)
* * * * File screen
      ADD_ITEM(file, "New", "doNew", kNoChildScreen)
      ADD_ITEM(file, "Open", "doOpen", kDeferredChild)
      ADD_ITEM(file, "Close", "doClose", kNoChildScreen)
      ADD_ITEM(file, "Save", "doSave", kNoChildScreen)
      ADD_ITEM(file, "Save As", "doSaveAs", kDeferredChild)
   
* * * * Edit screen
      ADD_ITEM(edit, "Undo", "doEdit undo", kNoChildScreen)
      ADD_ITEM(edit, "Cut", "doEdit cut", kNoChildScreen)
      ADD_ITEM(edit, "Copy", "doEdit copy", kNoChildScreen)
      ADD_ITEM(edit, "Paste", "doEdit paste", kNoChildScreen)
      ADD_ITEM(edit, "Select", "doSelect", kDeferredChild)
      
* * * * About screen
      ADD_TEXT(help, "A QuickTime\nmovie player and 
+ editor\nbuilt with SNOJOB\n\n\u169 2004 Tim Monroe")
                                                         :(RETURN)
setUpScreens_END

Several commands use the constant kDeferredChild as their fourth parameter. This allows us to pretend for the moment that the menu item has a child screen without having already created one explicitly. The function invoked by the item selection will need to create the child screen.

So how does QuickTime fit into this picture? SNOJOB has the following useful property: any call to a function that is not a built-in SNOBOL function (for instance, LEN), a Pixo user-interface function (for instance, ADD_ITEM), or a user-defined function (for instance, setUpScreens) is assumed to be contained in CarbonLib. The parameters to these external functions are specified in the script as strings but are coerced to the proper type by the SNOJOB interpreter. Opening and controlling QuickTime movies is therefore just a matter of resorting to the Carbon-based QuickTime APIs whenever necessary.

Let's start by seeing how to open a movie file. As we've seen (in Listing 3), selecting the Open item in the File screen results in our script's doOpen function being called. The first thing we need to do is display a list of the QuickTime movies stored on the iPod, as shown in Figure 6.

The Pixo user-interface software provides an extremely easy way to do this:

      fileName = NEW_LIST_SCREEN("Movies", "Movies", file)

Here the first parameter is the title of the new movie-selection screen. The second parameter is the name of the folder that contains the QuickTime movies we want to display. This folder is assumed to reside within the iPod_Control folder we encountered earlier. The third parameter is of course the parent screen.

If the user selects one of the movie files displayed in the movie list, then its name is returned and assigned to the fileName variable. If the user cancels the movie file selection by pressing the iPod's Menu button, an empty string is returned. We exit doOpen if fileName is the empty string with this code:

      IDENT(fileName)                           :S(RETURN)

(The IDENT function compares two or more strings for identity and succeeds if they are character-for-character identical; in any SNOBOL function, missing parameters are taken to be the empty string.)

We can open the selected file by executing the Carbon functions OpenMovieFile and NewMovieFromFile. The only "gotcha" is that OpenMovieFile called from a SNOJOB script is assumed to pass a pathname, not a file system specification.

      OpenMovieFile(fileName, refNum, fsRdWrPerm)
      NewMovieFromFile(movie, refNum, resID, 0, 
+                 newMovieActive)

Now we need to create a new screen within which the movie will be displayed. Once again we'll call NEW_SCREEN, this time like this:

      moov = NEW_SCREEN(kDataType, "", main)

This indicates that the moov screen is a data screen, where the entire contents of the screen are supplied by the application.

As always, the preferred way to manage playback of QuickTime movies is by attaching a movie controller to the movie, which we can do with these lines of code:

      SetRect(rect, 0, 0, ScreenWid(), ScreenHgt())
      mc = NewMovieController(movie, rect, mcTopLeftMovie)

The functions ScreenWid and ScreenHgt are built-in Pixo functions that return the width and height of the entire iPod screen. By using these functions instead of hardcoding values, we allow our application to run on all present and future iPods.

To allow the user to edit movies, we need to explicitly enable editing:

      MCEnableEditing(mc, 1)

Finally, we need to set the movie's graphics world to the data screen:

      SetMovieGWorld(movie, moov)

The Pixo user-interface layer takes care of moving pixels drawn by the movie controller into the data screen moov.

Since SnoEez can open multiple movies and (for instance) allow the user to cut and paste data from one movie to another, we need to keep track of the movie controller and other pieces of data associated with any particular movie. SNOBOL does not provide any mechanism for defining structured data types, but we can simulate such types with associative arrays (also known as dictionaries or hashes). So we'll maintain a global associative array called appData. In fact, we'll use the appData array for two purposes, to hold global data values and to hold data associated with a particular window. Listing 4 shows the definition of the initApp function, which allocates the global array and initializes its values.

initApp
      DEFINE('initApp()')                        :(initApp_END)
initApp
      
* * * * constants
*      constants identifying fields in the appData table
      DIRTY_FIELD            = 10
      FNAME_FIELD            = 11
      BNAME_FIELD            = 12
      MOVIE_FIELD            = 13
      CNTRL_FIELD            = 14
      CTYPE_FIELD            = 15
      OPRTN_FIELD            = 16
      
      NEWNO_FIELD            = 20
      WINNO_FIELD            = 21
      
*      values for the CTYPE_FIELD field
      CNTRL_LINEAR           = 0
      CNTRL_VR               = 1
      
*      some strings
      NEW_MOVIE_NAME         = "Untitled"
      APP_NAME               = "SnoEez"
      WIN_NAME               = "winRTM"
* * * * global variables
*      create the table we use to store global data
      appData = TABLE()
      
*      initialize global variables
      appData[NEWNO_FIELD]   = 1
      appData[WINNO_FIELD]   = 1
*      set up the screens
      setUpScreens()                              :(RETURN)
initApp_END

When we open a movie file, we execute this code to create a unique identifier for the movie:

      winName = WIN_NAME appData[WINNO_FIELD]
      appData[WINNO_FIELD] = appData[WINNO_FIELD] + 1

Then we store the relevant data into the appData array:

      appData[winName "," MOVIE_FIELD] = movie
      appData[winName "," FNAME_FIELD] = fileName
      appData[winName "," BNAME_FIELD] = basename(fileName)
      appData[winName "," CNTRL_FIELD] = mc
      appData[winName "," DIRTY_FIELD] = 0

Happily, the iPod's buttons, originally intended to control music playback, can be used to control movie playback while SnoEez is the active application. We can very easily bind the operation of the buttons and the scroll wheel to standard movie controller operations. For instance, to make a press on the Fast Forward button position the active movie at its end, we can execute this code:

      topMovie = topMovieWindow()
      BIND_BUTTON(kFastForward, GoToEndOfMovie(topMovie))

Here, topMovieWindow is a user-defined function that returns the topmost movie. The implementation of this function is left as an exercise for the reader.

Let's look briefly at editing movies. Listing 5 shows the doEdit function for handling the four basic edit operations.

doEdit
      DEFINE('doEdit(op)')                           :(doEdit_END)
doEdit
      winName = topMovieWindow()
      mc = [winName "," CNTRL_FIELD]
      IDENT(op, "undo") MCUndo(mc)                   :S(CH)
      IDENT(op, "cut") mov = MCCut(mc)               :S(CO)
      IDENT(op, "copy") mov = MCCopy(mc)             :S(CO)
      IDENT(op, "paste") MCPaste(mc)                 :S(CH) :F(RETURN)
CO      PutMovieOnScrap(mov, 0)
CH      IDENT(op, "copy")                               :S(RETURN)
      setWindowDirty(winName, 1)                     :(RETURN)
doEdit_END

The only difficult task involved in movie editing is setting a selection. The best solution I've been able to devise is to use the scroll wheel to select from the current movie time either forward or backward. Figure 7 shows a selection bar superimposed on the movie screen after the user selects the Select item.

The code required to display and manage this bar is fairly lengthy, and I've omitted it in the interests of saving space.

Once we've got our SNOJOB script completed, it's easy to install it and any QuickTime movies onto the iPod. The movies of course need to be copied into the Movies folder in the iPod_Control folder. The script needs to be encoded by adding 13 to the ASCII value of each character in the file. (This isn't strictly a rot-13 encoding, since classic rot-13 alters only alphabetic characters and leaves other symbols untouched.) Finally, move the encoded script into the Extras folder in the iPod. SnoEez now appears in the list of iPod extras, as shown in Figure 8.

Supporting playback and editing of QuickTime movies on the iPod is surprisingly straightforward, once we've untangled the knots and ignored the red herrings that Apple has thrown in our path. All it takes is a little sleuthing, a little knowledge of SNOBOL, and a little persistence. No fooling!

Tim Monroe is a member of the QuickTime engineering team at Apple. You can contact him at monroe@mactech.com. The views expressed here are not necessarily shared by his employer.