Jun 00 QTToolkit
Volume Number: 16 (2000)
Issue Number: 6
Column Tag: QuickTime Toolkit
Making Movies
by Tim Monroe
Creating QuickTime Movie Files
Introduction
So far in this series of articles on programming QuickTime, we've learned how to open movie files and allow the user to interact with the movies in those files. We've learned how to perform some basic operations on movies, such as editing them and exporting them under new formats. We've also learned how to open and display image files and to perform some simple transformations on those images. And, for good measure, we've managed to do everything in a fully cross-platform manner, so that our applications run both under the Macintosh operating system and under Windows, with exactly the same features.
Now it's time to take an important step forward in our journey through QuickTime: it's time to learn how to create QuickTime movie files. Even though QuickTime supports a vast array of types of media (audio, video, text, sprite animation, vector image, virtual reality, and so forth), it expects that all of the media data and the information describing that data will be stored in a specific format, called the QuickTime movie file format. This means that once we learn the basic sequence of operations required to create files that adhere to this format, we can fairly easily apply that knowledge to create files containing any of these types of media data. For the moment, we're going to restrict our attention to creating a QuickTime movie file with a single video track. But the techniques we learn here will come into play over and over again in the future when we want to create movie files that contain sound data, text, music, sprites, and other types of media data.
Happily, the Movie Toolbox provides a set of high-level programming interfaces that we can use to create movie files without having to know very much about the actual details of the QuickTime movie file format. We need to know some of those details, however, so we'll begin by surveying the structure of QuickTime movies and QuickTime movie files. Then we'll be ready to create movie files that exhibit that structure.
The Structure of QuickTime Movies
What exactly is a QuickTime movie? There are really two answers to this question. So far in this series, we've been concerned primarily with QuickTime movies as they exist in memory. That is to say, we've been working with items of type Movie that we can play, edit, attach movie controllers to, and so forth. We typically obtain these movies by opening QuickTime movie files, which exist on disk (or some other storage medium). The movie file contains the audio and video data that is presented to the user when a movie is being played, as well as other information (called metadata) that describes how that audio and video data is organized and synchronized. Of course, QuickTime isn't limited to playing audio and video data, so we'll need a more general term to refer to the movie data contained in the movie file; let's call it media data.
As a rough preliminary characterization, then, we can say that a QuickTime movie file is a file that contains the movie's media data and the associated metadata. (As we'll see later, a movie file might not actually contain the media data, but only references to the media data. For the moment, however, we'll ignore this possibility.) A QuickTime movie, on the other hand, is a structure that contains information about the media data. In effect, a QuickTime movie is just a bookkeeping device that contains the information necessary to retrieve the media data from the QuickTime movie file and present it to the user at the appropriate moment.
The exact structure of QuickTime movies is private. If you take a look into the header file Movies.h, you'll see that a variable of type Movie is a pointer to a MovieRecord structure, which is defined like this:
struct MovieRecord {
long data[1];
};
All we can tell about the structure of a MovieRecord is that it contains at least 4 bytes of data. We can operate on movies only by using the application programming interfaces provided by the Movie Toolbox.
A movie consists of one or more tracks, plus some optional movie user data. A track has a starting time and duration, and is associated with exactly one kind of media data. So, for instance, a track might hold audio data or video data, but not both. A movie can contain several tracks, each associated with a different kind of media data. Also, a movie can contain several tracks that are associated with the same kind of media data. For instance, a movie might have a video track and several audio tracks (perhaps in different languages).
The structure of a track is also private. A variable of type Track is a pointer to a TrackRecord structure, which is defined like this:
struct TrackRecord {
long data[1];
};
A track contains a single media structure (or, more briefly, a media), which references, but does not contain, the media data associated with the track. Once again, as you've probably guessed, the structure of a media is private. A variable of type Media is a pointer to a MediaRecord structure, which is defined like this:
struct MediaRecord{
long data[1];
};
Figure 1 shows a standard way of depicting the structure of a QuickTime movie. As you can see, this movie contains three tracks: two video tracks and one sound track. Associated with a movie is a movie time coordinate system, which provides a means to measure time in a movie. The basic unit of time measurement for a movie is the movie's time unit, and the number of time units that elapse per second is the movie's time scale. In the movie shown in Figure 1, the time scale is 600, so the time unit is 1/600 of a second. The sound track is scheduled to begin playing after 1 second has elapsed in the movie, and video track 2 is scheduled to begin playing after about 1.5 seconds have elapsed. QuickTime requires that all tracks start at time 0, but the track data can begin later in the movie; the empty space between the beginning of the movie and the beginning of the track data is called the track offset.
Figure 1. The structure of a QuickTime movie.
Keep in mind that Figure 1 is intended to illustrate the logical structure of a QuickTime movie, not the actual arrangement of bytes in memory. To repeat, the actual structure of a QuickTime movie in memory is private. The only way to operate on movies, tracks, or media is to use the APIs provided by the Movie Toolbox.
The Structure of QuickTime Movie Files
In contrast to the undocumented, private structure of QuickTime movies, the structure of QuickTime movie files is completely public. This means, among other things, that QuickTime movie files can be built on any operating system, whether or not the QuickTime software is available on that operating system. That's because a QuickTime movie file is just a sequence of bytes structured according to an openly documented specification. Note, however, that you don't actually have to know the details of that specification to build QuickTime movie files. The Movie Toolbox provides a set of high-level functions that we can use to build QuickTime movie files with a minimum of fuss. (Whew!) Nevertheless, it will be useful for us to become acquainted with at least some of those details, if only to understand better what it is we are building and why we need to do certain things when building QuickTime movie files. So let's take a quick look at the structure of QuickTime movie files.
It turns out that the basic structure of QuickTime movie files has evolved in the years since QuickTime was introduced. Files created back then will still play okay nowadays (at least on Macintosh computers), but we can make life easier for ourselves and our viewers if we build files that conform to the currently preferred format, which is single-fork, self-contained, Fast Start, interleaved movie files. Let's unravel that mouthful.
Double-fork and Single-fork Movie Files
When QuickTime was first released, back in 1991, it ran only on the Macintosh operating system. Accordingly, the default behavior of the Movie Toolbox was to build movie files that took advantage of the dual-fork nature of files in the Macintosh file system, where each file has both a resource fork and a data fork. The movie's metadata was stored in the resource fork and the movie's media data was stored in the data fork. This clean separation of metadata from media data meant that the Movie Toolbox could very easily find the metadata, parse it, and load it into memory as a QuickTime movie, without ever touching the media data. Figure 2 shows the original structure of QuickTime movie files as double-fork files.
Figure 2. A double-fork movie file.
Figure 2 also reveals a bit more about the internal structure of the data and resource forks of a double-fork QuickTime movie file. Consider first the data fork. The media data is contained in a structure called an atom. An atom is simply a collection of data preceded by an atom header, which consists of a four-byte length value and a four-byte type. (The length value includes the 8 bytes occupied by the atom header.) In this case, the atom is a movie data atom, whose type is 'mdat'. The data contained in a movie data atom is one or more media samples, which together comprise the movie's media data. A media sample is a single element of movie data. (You can think of a video frame as a single sample.)
Now let's turn our attention to the resource fork in Figure 2. As you can see, the resource fork contains both a resource header and a resource map (which are contained in every well-formed Macintosh resource file). The resource fork also contains a resource of type 'moov', which is called the movie resource. The movie resource contains the movie metadata, which — lo and behold — is structured as an atom of type 'moov'. This atom is called the movie atom.
The main problem with double-fork movie files is that they cannot easily be transported to operating systems that do not support resource forks. To facilitate cross-platform deployment of multimedia content, QuickTime also supports single-fork movie files. In this case, the movie media data and the movie metadata are stored in one file (which is the data fork on Macintosh systems). Typically, the movie atom is simply appended to the end of the movie data atom, as shown in Figure 3. This movie file can be opened and played back on any operating system that supports the QuickTime playback software. As a result, we shall always build single-fork movie files whenever we create QuickTime movie files.
Figure 3. A single-fork movie file.
Before we move on, let's take a moment to try to prevent some potential confusion. As we've just seen, both single- and double-fork movie files store the movie metadata in an atom of type 'moov', called the movie atom. In a double-fork movie file, the movie atom is contained in a resource of type 'moov', called the movie resource. Indeed, in a double-fork movie file, the movie atom and the movie resource are identical. This fact has encouraged some developers to use the terms "movie resource" and "movie atom" interchangeably. But this is unfortunate, since (as you can see) there is no movie resource in a single-fork movie file, but there is a movie atom.
The potential for confusion is compounded by the fact that the Movie Toolbox includes the function AddMovieResource, which we will need to call to add the movie atom to our single-fork movie files (see, for instance, Listing 1). Originally, QuickTime did not provide a way to create single-fork movie files directly. Instead, you had to create a double-fork movie file and then call the function FlattenMovieData to make a single-fork copy of that double-fork movie file. In QuickTime 3.0, AddMovieResource was revised to support direct creation of single-fork movie files. Nowadays, AddMovieAtom would perhaps be a better name than AddMovieResource.
Fast Start Movie Files
Notice that, according to the default structure of single-fork movie files (as shown in Figure 3), the movie atom is stored at the end of the file. The reason for this is quite simple. Typically, we build a new QuickTime movie file by creating an empty movie file, adding new tracks to the movie, adding a new media to the track, and then adding some media samples to the media. As this process continues, the Movie Toolbox keeps track of the gradually-changing metadata, which can be added to the movie file only after all the movie data has been added. The movie metadata is rather like a cargo manifest, which can be deemed accurate only after all the cargo has been loaded.
In some cases, this default location of the movie atom is unproblematic. When the movie file is local (that is, stored on a local hard disk, CD-ROM, or other random-access storage device), QuickTime can search through the file quickly enough that it can find the movie metadata — which it needs in order to start playing the movie — with no appreciable delay. But when the movie file is being sent across a local area network, or indeed across the Internet, it isn't generally possible to randomly access the file's data in this manner. This means that QuickTime cannot start playing remotely-stored movies until the entire movie has been downloaded from the remote storage device to the local playback machine.
To remedy this situation, QuickTime provides support for creating Fast Start movie files, in which the movie atom is stored as the first atom in a single-fork movie file. Figure 4 shows the structure of a Fast Start movie file. As you can see, the only difference between a Fast Start movie file and a default single-fork movie file is the location of the movie atom. This simple change, however, is enough to enable QuickTime to start playing movies served from the web via HTTP or FTP before the entire movie has downloaded.
Figure 4. A Fast Start movie file.
To create a Fast Start movie file, we need to create a single-fork movie file and then call FlattenMovieData with the flattenForceMovieResourceBeforeMovieData flag. We'll see how to do this later.
Reference and Self-contained Movie Files
Earlier I warned that a movie file might not actually contain the movie's media data, but only references to the media data, which is contained in some other file (called the media file). A file that has this characteristic is called a reference movie file. Figure 5 illustrates this possibility. Here, the movie file itself contains only the movie atom, which contains references to the media data. Reference movie files can be useful if you want to share some data among two or more QuickTime movies. By referencing the shared media data instead of containing copies of it, the associated QuickTime movie files can take up substantially less space.
Figure 5. A reference movie file and its target media file.
It's also possible for a reference movie file to refer to media data in more than one media file. Indeed, it's possible for a reference movie file to refer to media data in some media file as well as media data contained in the reference movie file itself. (The first kind of media reference is called an external reference, and the second kind is called an internal reference.) QuickTime really doesn't care where the media data is stored, as long as can resolve the references to that data at playback time.
The opposite of a reference movie is a self-contained movie file. In a self-contained movie file, all the media data needed for playing the movie is contained in the movie file itself. Or, put another way, a self-contained movie file does not depend on any other files. Or, put yet another way, all media references in a self-contained movie file are internal references. Figure 6 shows a self-contained movie file, in which the media references in the movie atom are resolved to media data in the movie file itself.
Figure 6. A self-contained movie file.
If you've got a reference movie file, you can create a self-contained movie file from it by calling FlattenMovieData. (Note, however, that it's possible to build a self-contained movie file directly, without having to create a reference movie file and then call FlattenMovieData to create another movie file.) FlattenMovieData resolves all media references and copies the referenced data into the movie file itself. For this reason, a self-contained movie file is sometimes also called a flattened movie file. But this terminology can be confusing, since FlattenMovieData also makes copies of any media data that is multiply referenced by internal media references. That is to say, if a self-contained movie file contains several internal references to the same media data, then the result of calling FlattenMovieData on that file will be a larger movie file that contains several copies of the referenced media data (each of which is referred to exactly once by the movie atom). So, if we reserve the term "flattened" for any movie files that might have been created by FlattenMovieData, it follows that not all self-contained movie files are flattened movie files.
To make matters worse, the term "flattening" is sometimes also used to refer to the process of converting a double-fork movie file into a single-fork movie file. So, in this parlance, a flattened movie file is simply any single-fork movie file, whether or not it contains any external media references. Perhaps the best course of action is simply to avoid using the term "flattened" altogether.
Interleaved and Non-Interleaved Movie Files
We've almost reached the end of our survey of QuickTime movie file organizations. There is only one final twist that we need to consider. Typically, when we build a movie file that contains more than one kind of media data, we add the media samples for one kind of media data (for example, video data) and then we add the media samples for another kind of media data (for example, audio data). The resulting file is a non-interleaved movie file, as shown in Figure 7.
Figure 7. A non-interleaved movie file.
Non-interleaved movie files do not provide optimal playback from slower storage devices (such as CD-ROMs) where seeking can be expensive, or in situations (such as network playback) where random access to the file data is not possible. Instead, it's better to reorganize the media data so that it's arranged in temporal order. That is to say, we'd like the audio and video data for the first second of the movie to be stored close to one another, followed by the audio and video data for the next second of the movie, and so forth. Figure 8 shows an interleaved movie file that has the desired structure.
Figure 8. An interleaved movie file.
To create an interleaved movie file, we typically create a single-fork movie file and then call FlattenMovieData. By default, FlattenMovieData interleaves the media data to ensure optimal movie playback. You can override this default behavior by passing the flattenDontInterleaveFlatten flag to FlattenMovieData; the result would be a self-contained, non-interleaved movie file.
Creating QuickTime Movie Files
Our ultimate goal, then, is to create a single-fork, self-contained, Fast Start, interleaved movie file. The resulting file will play on all platforms that support QuickTime movie playback and be suitable for distributing on CD-ROM and over the Internet. To simplify things, we're going to create a movie file that contains a single video track. We'll postpone adding other kinds of tracks to future articles.
To create a single-fork, self-contained QuickTime movie file, we typically need to perform eight operations, in this order:
- Create a new, empty movie file and a movie that references that file (CreateMovieFile).
- Add a new track to that movie (NewMovieTrack).
- Add a new media to that track (NewTrackMedia).
- Add media samples to the media.
- Insert a reference to the media segment into the track at the desired offset (InsertMediaIntoTrack).
- Add the movie atom to the movie file (AddMovieResource).
- Close the movie file (CloseMovieFile).
- Dispose of the movie (DisposeMovie).
If we want the final movie file to be either Fast Start or interleaved, we need to perform one additional operation:
- Place the movie atom as the first atom in the movie file, and interleave the media data (FlattenMovieData ).
As you can see, each of these steps — except for step 4 — can be accomplished with a single Movie Toolbox function. We're going to use these eight functions, in this order, over and over again as we build different kinds of QuickTime movie files, here and in future articles. So you should take a moment to burn these steps into your memory. The only real differences between building, say, a video movie file and a wired sprite movie file concern step 4, adding media samples to the media. Even though the movies in those two kinds of files have radically different playback characteristics, the fundamental structure of the QuickTime movie files is the same, and so the way we build those files will be essentially the same.
Let's take a preliminary look at how this all comes together in practice. Listing 1 defines the function QTMM_CreateVideoMovie, which is the routine used by our sample application QTMakeMovie for building a QuickTime movie file. QTMM_CreateVideoMovie begins by calling the framework function QTFrame_PutFile to elicit from the user a file name and location for the new movie file. Then QTMM_CreateVideoMovie calls the Movie Toolbox functions listed above to build a movie file. Step 4, adding media samples to the media, is handled by the application function QTMM_AddVideoSamplesToMedia, which is sandwiched between the two calls BeginMediaEdits and EndMediaEdits.
Listing 1: Creating a QuickTime movie.
QTMM_CreateVideoMovie
OSErr QTMM_CreateVideoMovie (void)
{
Movie myMovie = NULL;
Track myTrack = NULL;
Media myMedia = NULL;
FSSpec myFile;
Boolean myIsSelected = false;
Boolean myIsReplacing = false;
StringPtr myPrompt =
QTUtils_ConvertCToPascalString(kNewMoviePrompt);
StringPtr myFileName =
QTUtils_ConvertCToPascalString(kNewMovieFileName);
long myFlags = createMovieFileDeleteCurFile |
createMovieFileDontCreateResFile;
short myResRefNum = 0;
short myResID = movieInDataForkResID;
OSErr myErr = noErr;
// prompt the user for new file name
QTFrame_PutFile(myPrompt, myFileName, &myFile,
&myIsSelected, &myIsReplacing);
myErr = myIsSelected ? noErr : userCanceledErr;
if (myErr != noErr)
goto bail;
// create a movie file for the destination movie
myErr = CreateMovieFile(&myFile, sigMoviePlayer,
smCurrentScript, myFlags, &myResRefNum, &myMovie);
if (myErr != noErr)
goto bail;
// create the movie track and media
myTrack = NewMovieTrack(myMovie,
FixRatio(kVideoTrackWidth, 1),
FixRatio(kVideoTrackHeight, 1),
kNoVolume);
myErr = GetMoviesError();
if (myErr != noErr)
goto bail;
myMedia = NewTrackMedia(myTrack, VideoMediaType,
kVideoTimeScale, NULL, 0);
myErr = GetMoviesError();
if (myErr != noErr)
goto bail;
// create the media samples
myErr = BeginMediaEdits(myMedia);
if (myErr != noErr)
goto bail;
myErr = QTMM_AddVideoSamplesToMedia(myMedia,
kVideoTrackWidth, kVideoTrackHeight);
if (myErr != noErr)
goto bail;
myErr = EndMediaEdits(myMedia);
if (myErr != noErr)
goto bail;
// add the media to the track
myErr = InsertMediaIntoTrack(myTrack, 0, 0,
GetMediaDuration(myMedia), fixed1);
if (myErr != noErr)
goto bail;
// add the movie atom to the movie file
myErr = AddMovieResource(myMovie, myResRefNum, &myResID,
NULL);
bail:
if (myResRefNum != 0)
CloseMovieFile(myResRefNum);
if (myMovie != NULL)
DisposeMovie(myMovie);
free(myPrompt);
free(myFileName);
return(myErr);
}
In the following subsections, we'll gradually dissect Listing 1.
Creating a New Movie File
The first thing we need to do to create a QuickTime movie file is create a new, empty QuickTime movie file. We can do this by calling the CreateMovieFile function, like this:
myFlags = createMovieFileDeleteCurFile |
createMovieFileDontCreateResFile;
myErr = CreateMovieFile(&myFile, sigMoviePlayer,
smCurrentScript, myFlags, &myResRefNum, &myMovie);
Here, myFile is a file system specification that indicates the name and location of the new movie file. The second parameter specifies the creator type for the new file; in this case, we're passing the constant sigMoviePlayer (defined in the file FileTypesAndCreators.h) to indicate that we want the file's creator to be the QuickTime Player application (née MoviePlayer). Of course, you are free to use any other creator type that you wish. The third parameter specifies the script in which the movie should be created; here, as always when we need to specify a script system, we pass the constant smCurrentScript.
The fourth parameter to CreateMovieFile is a set of flags that modify the default behavior of CreateMovieFile. Generally, we will pass the two flags indicated, createMovieFileDeleteCurFile and createMovieFileDontCreateResFile, which tell CreateMovieFile to delete any file already having the name and location specified by the myFile parameter and to create a single-fork movie file.
The last two parameters allow CreateMovieFile to pass information back to us, if it successfully creates the specified movie file. In the fifth parameter, it returns to us the file reference number for the opened movie file. (Don't let the parameter name myResRefNum fool you; because we passed the createMovieFileDontCreateResFile flag, the opened file fork is a data fork, not a resource fork.) CreateMovieFile accepts a flag that tells it not to open the newly created file, but here we want it to open the file so that we can add data to it.
In the final parameter, CreateMovieFile passes back to us an identifier for a new, empty movie that references the file we just created. The parameter myMovie is of type Movie, which is precisely the sort of in-memory QuickTime movie that we've been using in previous articles. From here on in, we won't do anything directly to the movie file itself. Rather, we're going to build the movie (by adding tracks and media samples to it) and then later use the AddMovieResource function to write the movie atom into the movie file
Adding Tracks to a Movie
Once we've called CreateMovieFile to create a new movie file and a new, empty movie, we need to add a track to the movie. We do this by calling the NewMovieTrack function, like this:
myTrack = NewMovieTrack(myMovie,
FixRatio(kVideoTrackWidth, 1), FixRatio(kVideoTrackHeight, 1), kNoVolume);
NewMovieTrack needs to know which movie to add the new track to; here we pass the movie identifier that we obtained from the call to CreateMovieFile. NewMovieTrack also needs to know the dimensions of the new track. Since we're building a movie from scratch, we'll specify some predefined dimensions. The height and width are expected to be in units of type Fixed, so we'll use the FixRatio function to convert the predefined integer height and widths to that type. In the present case, the track is 202 pixels high and 152 pixels wide:
#define kVideoTrackHeight 202
#define kVideoTrackWidth 152
(These are the dimensions of a picture stored in the resource fork of the QTMakeMovie application.)
The final parameter to NewMovieTrack is a value that specifies the desired volume level of the new track. Because the track we are about to create is a video track, we pass the constant kNoVolume to indicate that the volume level should be 0.
Adding a Media to a Track
Recall that a track is associated with exactly one media structure (or media). The media structure contains information about the type and location of the actual media samples that comprise the QuickTime movie data. We create a new media and associate it with an existing track by calling the NewTrackMedia function, like this:
myMedia = NewTrackMedia(myTrack, VideoMediaType, kVideoTimeScale, NULL, 0);
The first parameter is the track with which the new media is to be associated; here of course we use the track that we just created by calling NewMovieTrack. The second parameter specifies the type of media we want to create. As you can see, we've used the constant VideoMediaType to specify that we want to create a new video media. The Movie Toolbox defines constants for all media types for which QuickTime supplies a media handler (that is, a component that knows how to interpret a specific kind of media data). Here are a few of the common media types defined in Movies.h:
enum {
VideoMediaType = FOUR_CHAR_CODE('vide'),
SoundMediaType = FOUR_CHAR_CODE('soun'),
TextMediaType = FOUR_CHAR_CODE('text'),
MusicMediaType = FOUR_CHAR_CODE('musi'),
TimeCodeMediaType = FOUR_CHAR_CODE('tmcd'),
SpriteMediaType = FOUR_CHAR_CODE('sprt'),
FlashMediaType = FOUR_CHAR_CODE('flsh'),
MovieMediaType = FOUR_CHAR_CODE('moov'),
TweenMediaType = FOUR_CHAR_CODE('twen')
};
Over time, we will turn our attention to each of these media types and see how to build a movie that contains media data of that type. For the moment, as you know, we're focusing on building a movie with video media data.
The third parameter to the NewTrackMedia function specifies the media time scale, which is the number of media time units that elapse per second. The media time unit defines the media time coordinate system. The media time coordinate system is independent of the movie time coordinate system, which we discussed briefly earlier. In addition to interpreting the media data, the media handler for a track's media also manages the task of mapping values from the movie time coordinate system to the media time coordinate system. For example, when the movie is ready to display a video frame, it passes the current movie time to the video media handler, which converts that time into the video media time coordinate system and then retrieves the video data at that media time.
The movie time coordinate system is independent of the media time coordinate system largely to allow each media handler to work with time units that are appropriate to the kind of media it is handling. Video media often uses a time scale of 600, which is evenly divisible into the most common frames-per-second values (8, 10, 12, 15, 24, 30, and 60). Sound media typically uses a time scale equal to the sample rate of the sound, so some common sound media time scales are 11,025 (11 kHz), 22,254.54 (22 kHz), and 44,100 (44 kHz).
The default QuickTime movie time scale is 600. This value was selected to facilitate the mapping between the movie time coordinate system and the video media time coordinate system. Our sample application QTMakeMovie uses 600 as the video media time scale:
#define kVideoTimeScale 600 // 600 units per second
The fourth parameter to NewTrackMedia specifies a media data reference, which identifies the file (or other media container) that is to hold the media data for the specified track. Here we pass the value NULL, to indicate that the media data is to be stored in the file associated with the movie (that is, the file created when we called CreateMovieFile). It would be possible to specify some other file to hold the media data for this track; however, since we want to create a self-contained file, we'll pass NULL here. For the same reason, we pass 0 in the fifth parameter. If we were passing a non-NULL media data reference in the fourth parameter, we would use the fifth parameter to specify the type of data reference.
Adding Samples to a Media
We've been occupied so far with creating the movie metadata — the "bookkeeping details" — that will end up inside the movie atom in the QuickTime movie file. What we need to do now is add some media data to the movie file. The media data is what the user will ultimately see or hear when the movie is played, and what we would normally think of as the meat-and-potatoes of the QuickTime movie file. As we saw earlier, the media data is comprised of one or more media samples.
To add samples to an existing media, we need to open a media-editing session. That is to say, we need to inform the Movie Toolbox that we want to change the media samples referenced by a track's media. We can do this by calling the BeginMediaEdits function, specifying the media we want to edit:
myErr = BeginMediaEdits(myMedia);
When we call BeginMediaEdits, the Movie Toolbox opens the media container, if it isn't already open, and makes sure that we can add samples to it. In the present case, the media container is the QuickTime movie file that we created when we called CreateMovieFile.
Finally, we've done all the groundwork that we need to do before we can add some media data to our movie file. We've created a movie, a track, and a media, and we've opened a media-editing session. All that remains is to call AddMediaSample one or more times to add some media samples to the media container. Unfortunately, we haven't created any media samples yet. For the moment, let's defer going into those details, so that we can spend the remainder of this section focusing on the overall movie-making process. Let's just call an application-defined function that adds the desired video media samples to the media:
myErr = QTMM_AddVideoSamplesToMedia(myMedia,
kVideoTrackWidth, kVideoTrackHeight);
We'll spend the entire next section, "Adding Media Samples", walking through the function QTMM_AddVideoSamplesToMedia.
When we are finished adding samples to the media, we close the media-editing session by calling the EndMediaEdits function, like this:
myErr = EndMediaEdits(myMedia);
Inserting the Media Segment
Once we've finished adding samples to the media and ended the media-editing session, we need to add a reference to some or all of that media to the track. In particular, we need to specify when in the track the media segment is to begin; we also need to specify the portion of the media samples to be inserted into the track. We can handle all of this with the InsertMediaIntoTrack function:
myErr = InsertMediaIntoTrack(myTrack,
0, // time in track
0, // beginning of media segment
GetMediaDuration(myMedia), // duration of media segment
fixed1);
We're not doing anything fancy here, but there are a few things to keep in mind. The second parameter indicates where in the specified track the media data is to be inserted. The value of that parameter is interpreted in the movie time scale. So, if we want the video samples to start playing 2 seconds after the movie begins, and if the movie time scale is 600, then we would pass the value 1200 in the second parameter. In the present case, we want the video samples to start at the beginning of the movie, so we pass 0.
The second and third parameters specify the beginning and duration of the media segment to which we want to insert a reference into the target track. Both of these values should be expressed in the media time scale. Since we want to insert a reference to the entire media data, we set the beginning time to 0 and the duration to the value returned by the GetMediaDuration function.
The last parameter to InsertMediaIntoTrack indicates the media rate, which is the rate at which the media samples are to be played. We want the samples to be played at their natural playback rate, so we pass the constant fixed1.
Adding the Movie Atom
Now that we've inserted a reference to the media data into the movie's single video track, we've finally collected all the movie metadata we need, so we can add the movie atom to the QuickTime movie file. We do this by calling the AddMovieResource function, like this:
myErr = AddMovieResource(myMovie, myResRefNum, &myResID, NULL);
AddMovieResource takes the information contained in the movie myMovie and creates a movie atom, which it then appends to the file specified by the myResRefNum parameter.
Originally, the third and fourth parameters specified the resource ID number and the resource name for the movie resource (which, you'll recall, was added to the file's resource fork as a resource of type 'moov'). Nowadays, we are no longer interested in creating double-fork movie files, so we do not need to pass a movie resource name. Moreover, the Movie Toolbox now supports the constant movieInDataForkResID for the resource ID parameter, which indicates that the movie atom should be added to the data fork, not to the resource fork. So before we call AddMovieResource, we'll set the desired resource ID, like this:
myResID = movieInDataForkResID;
Finishing Up
We are essentially finished constructing a QuickTime movie file. Only two minor housekeeping tasks remain: we need to close the movie file that the Movie Toolbox automatically opened for us when we called CreateMovieFile, and we need to dispose of the movie that CreateMovieFile returned to us (and to which we subsequently added a video track). The function QTMM_CreateVideoMovie ends with these lines of code:
if (myResRefNum != 0)
CloseMovieFile(myResRefNum);
if (myMovie != NULL)
DisposeMovie(myMovie);
Adding Media Samples
In the previous section, we gradually dissected the function QTMM_CreateVideoMovie (defined in Listing 1) to learn how to create a QuickTime movie file with a video track. We postponed, however, the all-important task of adding media samples to the movie file. Instead, QTMM_CreateVideoMovie just called the function QTMM_AddVideoSamplesToMedia, which adds some video samples to the video track's media. The reason we deferred our treatment of adding media samples is simply that a full consideration of that topic would have detracted from the main point of the previous section, which was to emphasize the general steps involved in creating any kind of QuickTime movie file. Now let's tie up the loose ends by seeing how to add some video samples to the media of the movie's track.
To add a sample to a media, we really need only two things. First, of course, we need the media sample data. For a video track, the sample data usually consists of a compressed image. For a sound track, the sample data usually consists of some digitized (and possibly compressed) audio data. For a text track, the sample data consists of some number of characters. And so forth, for all the other kinds of media supported by QuickTime.
Second, we need a concise description of the kind of data contained in the media sample. The media type (specified in the call to NewTrackMedia) is generally not sufficient to allow the media handler to know how to process the media samples. For instance, a video track can contain images compressed using any one of a large number of video codecs (compressors/decompressors), and the track can have virtually arbitrary dimensions. For the video media handler to be able to display the video media samples correctly, it needs to know which compressor was used, the bit depth of the images, the dimensions of the images, and a handful or so of other pieces of information.
The AddMediaSample function expects us to pass it this information in a structure called a sample description. The file Movies.h defines a sample description like this:
struct SampleDescription {
long descSize;
long dataFormat;
long resvd1;
short resvd2;
short dataRefIndex;
};
The only two fields that we usually need to fill in are descSize, which specifies the size of the sample description, and dataFormat, which specifies the codec that compressed the data. But except for some very simple media types, this structure isn't complete enough. Instead, for video media data, we'll fill in an image description, defined in ImageCompression.h like this:
struct ImageDescription {
long idSize;
CodecType cType;
long resvd1;
short resvd2;
short dataRefIndex;
short version;
short revisionLevel;
long vendor;
CodecQ temporalQuality;
CodecQ spatialQuality;
short width;
short height;
Fixed hRes;
Fixed vRes;
long dataSize;
short frameCount;
Str31 name;
short depth;
short clutID;
};
As you can see, the first five fields of an image description are of the same type as the first five fields of a sample description (at least if you know that CodecType is defined as long). In effect, an image description is a sample description that contains some extra fields to hold information about video media. So, when we call AddMediaSample to add a video media sample, we'll pass it an image description, suitably cast to a sample description. But we're not ready to call AddMediaSample quite yet. First we need to create some video media data, the compressed images that, when decompressed, become the frames of our movie. Let's draw some images and compress them.
Drawing Video Frames
Our video media data is going to consist of a series of individual compressed images. We are of course free to use any images we like. For fun, let's make the first image in the series a completely white rectangle, the last image in the series a copy of the QuickTime penguin picture, and the intermediate images a progressively greater blend of the first and last images. Figure 9 shows the first and last movie frames, together with three intermediate frames. (In technical jargon, we're going to create a cross-fade between the first and last images.)
Figure 9. Some frames of the final movie file.
Listing 2 defines the QTMM_DrawFrame function, which draws a frame of a specified height and width into a specified offscreen graphics world. QTMM_DrawFrame also takes as a parameter the sample number; it uses that number to determine how much blending of the first and last images to perform.
Listing 2: Drawing a frame of the movie.
QTMM_DrawFrame
static void QTMM_DrawFrame (short theTrackWidth, short theTrackHeight, long theNumSample, GWorldPtr theGWorld)
{
Handle myHandle = NULL;
char myData[kPICTFileHeaderSize];
static PicHandle myPicture = NULL;
static GWorldPtr myGWorld = NULL;
static GraphicsImportComponent myImporter = NULL;
Rect myRect;
RGBColor myColor;
ComponentResult myErr = noErr;
MacSetRect(&myRect, 0, 0, theTrackWidth, theTrackHeight);
if (myPicture == NULL) {
myErr = NewGWorld(&myGWorld, kPixelDepth, &myRect, NULL,
NULL, (GWorldFlags)0);
if (myErr != noErr)
goto bail;
// read a picture from our resource file
myPicture = GetPicture(kPictureID);
if (myPicture == NULL)
goto bail;
// use Munger to prepend a 512-byte header onto the picture data;
// this converts the PICT resource data into in-memory PICT file data
// (see Ice Floe 14 for an explanation of this)
myHandle = (Handle)myPicture;
Munger(myHandle, 0, NULL, 0, myData,
kPICTFileHeaderSize);
// get a graphics importer for the picture
myErr = OpenADefaultComponent(
GraphicsImporterComponentType,
kQTFileTypePicture,
&myImporter);
if (myErr != noErr)
goto bail;
// configure the graphics importer
myErr = GraphicsImportSetGWorld(myImporter, myGWorld,
NULL);
if (myErr != noErr)
goto bail;
myErr = GraphicsImportSetDataHandle(myImporter,
myHandle);
if (myErr != noErr)
goto bail;
myErr = GraphicsImportSetBoundsRect(myImporter, &myRect);
if (myErr != noErr)
goto bail;
// draw the picture into the source GWorld
myErr = GraphicsImportDraw(myImporter);
if (myErr != noErr)
goto bail;
}
// set the blend amount (0 = fully transparent; 0xffff = fully opaque)
myColor.red = (theNumSample - 1) *
(0xffff / kNumVideoFrames - 1);
myColor.green = (theNumSample - 1) *
(0xffff / kNumVideoFrames - 1);
myColor.blue = (theNumSample - 1) *
(0xffff / kNumVideoFrames - 1);
OpColor(&myColor);
// blend the picture (in the source GWorld) into the empty rectangle
// (in the destination GWorld)
CopyBits((BitMapPtr)*GetGWorldPixMap(myGWorld),
(BitMapPtr)*GetGWorldPixMap(theGWorld),
&myRect,
&myRect,
blend,
NULL);
if (theNumSample == kNumVideoFrames)
goto bail;
return;
bail:
if (myHandle != NULL)
DisposeHandle(myHandle);
if (myPicture != NULL)
ReleaseResource((Handle)myPicture);
if (myImporter != NULL)
CloseComponent(myImporter);
}
I won't say much more about QTMM_DrawFrame, because it's not directly concerned with building QuickTime movie files. Basically, it reads an image from a 'PICT' resource, attaches a graphics importer to that image, draws that image into an offscreen graphics world, and then blends that image with the all-white image in the offscreen graphics world passed to QTMM_DrawFrame. You can work through the function if you like, or you can just trust me that, when passed values from 1 to kNumVideoFrames, it generates a series of images that includes the five shown in Figure 9.
Compressing Video Frames
Now that we've got an image that we want to include as a media sample of our QuickTime movie file, we need to compress it so that it takes up less space on disk. If we didn't do any compression on the video frames, each 152-by-202 pixel frame would occupy about 122,000 bytes, so our entire 100-frame movie file would occupy about 12 megabytes. That's a bit large for a 10-second movie. By compressing the movie frames, we can reduce the size of the final movie file to around 470 kilobytes (using JPEG compression) with no appreciable loss of image quality.
The only downside to compressing the movie frames (aside from any image degradation introduced by the image compressor) is that it takes time to compress the frames while building the movie file and it also takes time to decompress the frames during movie playback. But decompressing compressed video media samples is almost always faster than reading larger chunks of uncompressed data from a storage device, and it's orders of magnitude faster than receiving larger chunks of data across a network connection. So compression not only reduces the amount of storage required by a movie file, it also enhances playback performance by shifting much of the work from the storage device or network connection to the CPU.
QuickTime provides a rich set of services for compressing and decompressing images and sequences of images, principally provided by the Image Compression Manager (or, more briefly, the ICM). Right now we are concerned only with compressing our images into video media samples, and we can handle this by using only two ICM functions, GetMaxCompressionSize and CompressImage. GetMaxCompressionSize tells us the maximum size of our compressed image, given parameters that specify the size of the image to be compressed, the compressor, and the desired level of compression. We will use that maximum size when allocating a data buffer, into which CompressImage will write the compressed image. As a bonus, CompressImage also returns to us an image description whose fields have been set to the correct values for the image it has compressed. As we've seen, we'll need that image description when we eventually call AddMediaSample.
The first thing we need to do, before we can draw or compress our image, is create a new offscreen graphics world, which will hold the image drawn by QTMM_DrawFrame. So QTMM_AddVideoSamplesToMedia begins with these lines of code:
MacSetRect(&myRect, 0, 0, theTrackWidth, theTrackHeight);
myErr = NewGWorld(&myGWorld, kPixelDepth, &myRect, NULL,
NULL, (GWorldFlags)0);
if (myErr != noErr)
goto bail;
myPixMap = GetGWorldPixMap(myGWorld);
if (myPixMap == NULL)
goto bail;
LockPixels(myPixMap);
There is nothing fancy here; we just set the rectangle myRect to the width and height of the movie frame and call NewGWorld to create an offscreen graphics world of that size. Then we retrieve the pixel map associated with that graphics world and lock the pixel map. Both GetMaxCompressionSize and CompressImage operate directly on pixel maps, not on offscreen graphics worlds.
Now we can call GetMaxCompressionSize, like this:
myErr = GetMaxCompressionSize( myPixMap,
&myRect,
0, // let ICM choose depth
codecNormalQuality,
myCodecType,
(CompressorComponent)anyCodec,
&myMaxComprSize);
As you can see, we pass GetMaxCompressionSize the pixel map, the image rectangle, an image bit depth, a compression quality specifier, the desired compression type, a codec specifier, and a pointer to a long integer. GetMaxCompressionSize will update that long integer with the maximum number of bytes required to hold an image having the specified characteristics compressed using the specified compressor.
Once we know the maximum size a compressed image will occupy, we can allocate a buffer to hold the compressed image data, by calling NewHandle:
myComprDataHdl = NewHandle(myMaxComprSize);
if (myComprDataHdl == NULL)
goto bail;
HLockHi(myComprDataHdl);
myComprDataPtr = *myComprDataHdl;
Notice that we lock the handle and then dereference it to obtain a pointer to the data buffer. That's because CompressImage wants us to pass it a pointer, not a handle.
We are almost ready to call CompressImage. We need only one more thing, namely a handle to an image description. As I mentioned earlier, CompressImage will fill in the fields of the image description we pass it; it will also resize the image description as necessary, so we'll just pass it a handle to a four-byte block of memory:
myImageDesc = (ImageDescriptionHandle)NewHandle(4);
After we've drawn an image into the offscreen graphics world myGWorld, we can compress that image by calling CompressImage, like this:
myErr = CompressImage( myPixMap,
&myRect,
codecNormalQuality,
myCodecType,
myImageDesc,
myComprDataPtr);
CompressImage takes as parameters the pixel map of the offscreen graphics world, a rectangle that specifies the portion of the image to compress, the same compression quality and compressor type that we previously passed to GetMaxCompressionSize, the image description, and the pointer to a buffer. If all goes well, myComprDataPtr will point to the compressed image data and myImageDesc will be resized and updated. The actual size of the data buffer can be found by inspecting the dataSize field of the image description.
Adding Video Frames to the Media
We can finish this all off with one more step. Finally we are ready to call AddMediaSample to add the compressed image data to the media as a video media sample:
myErr = AddMediaSample( theMedia,
myComprDataHdl,
0, // no offset in data
(**myImageDesc).dataSize,
kVideoFrameDuration, // frame duration
(SampleDescriptionHandle)myImageDesc,
1, // one sample
0, // self-contained samples
NULL);
The first two parameters here are the media to which we want to add the sample and the sample data itself. The third parameter specifies the byte offset into the sample data buffer at which AddMediaSample should start reading the sample data; since the entire buffer comprises the sample data, we specify 0 as the byte offset. The fourth parameter indicates the number of bytes to be copied into the movie file. As we saw just above, we can use the dataSize field of the image description to obtain that size.
The fifth parameter specifies the duration, in the media time scale, of the sample being added to the media. We are giving each frame of the movie the same duration, which is specified by the constant kVideoFrameDuration:
#define kVideoFrameDuration kVideoTimeScale/10
Each frame of the movie has a duration of 1/10 second. Since there are 100 frames in the movie, the entire movie will last 10 seconds.
The sixth parameter to AddMediaSample is a handle to a sample description. As predicted earlier, we can just cast our image description handle to a sample description handle. The seventh parameter indicates the number of samples contained in the data buffer. The eighth parameter is a set of flags, which we can leave set to the default value of 0. The last parameter is a pointer to a time value, in which AddMediaSample will return the media time at which the sample was added. We don't care about that information, so we pass NULL.
Listing 3 gives a complete definition of the QTMM_AddVideoSamplesToMedia function, which adds 100 samples to the specified media, thereby creating the movie data for our QuickTime movie file.
Listing 3: Adding samples to a media.
QTMM_AddVideoSamplesToMedia
OSErr QTMM_AddVideoSamplesToMedia (Media theMedia, short theTrackWidth, short theTrackHeight)
{
GWorldPtr myGWorld = NULL;
PixMapHandle myPixMap = NULL;
CodecType myCodecType = kJPEGCodecType;
long myNumSample;
long myMaxComprSize = 0L;
Handle myComprDataHdl = NULL;
Ptr myComprDataPtr = NULL;
ImageDescriptionHandle myImageDesc = NULL;
CGrafPtr mySavedPort = NULL;
GDHandle mySavedDevice = NULL;
Rect myRect;
OSErr myErr = noErr;
MacSetRect(&myRect, 0, 0, theTrackWidth, theTrackHeight);
myErr = NewGWorld(&myGWorld, kPixelDepth, &myRect, NULL,
NULL, (GWorldFlags)0);
if (myErr != noErr)
goto bail;
myPixMap = GetGWorldPixMap(myGWorld);
if (myPixMap == NULL)
goto bail;
LockPixels(myPixMap);
myErr = GetMaxCompressionSize( myPixMap,
&myRect,
0, // let ICM choose depth
codecNormalQuality,
myCodecType,
(CompressorComponent)anyCodec,
&myMaxComprSize);
if (myErr != noErr)
goto bail;
myComprDataHdl = NewHandle(myMaxComprSize);
if (myComprDataHdl == NULL)
goto bail;
HLockHi(myComprDataHdl);
myComprDataPtr = *myComprDataHdl;
myImageDesc = (ImageDescriptionHandle)NewHandle(4);
if (myImageDesc == NULL)
goto bail;
GetGWorld(&mySavedPort, &mySavedDevice);
SetGWorld(myGWorld, NULL);
for (myNumSample = 1; myNumSample <= kNumVideoFrames;
myNumSample++) {
EraseRect(&myRect);
QTMM_DrawFrame(theTrackWidth, theTrackHeight,
myNumSample, myGWorld);
myErr = CompressImage( myPixMap,
&myRect,
codecNormalQuality,
myCodecType,
myImageDesc,
myComprDataPtr);
if (myErr != noErr)
goto bail;
myErr = AddMediaSample( theMedia,
myComprDataHdl,
0, // no offset in data
(**myImageDesc).dataSize,
kVideoFrameDuration, // frame duration
(SampleDescriptionHandle)myImageDesc,
1, // one sample
0, // self-contained samples
NULL);
if (myErr != noErr)
goto bail;
}
bail:
SetGWorld(mySavedPort, mySavedDevice);
if (myImageDesc != NULL)
DisposeHandle((Handle)myImageDesc);
if (myComprDataHdl != NULL)
DisposeHandle(myComprDataHdl);
if (myGWorld != NULL)
DisposeGWorld(myGWorld);
return(myErr);
}
Saving a Movie
We've now seen how to create a single-fork, self-contained QuickTime movie file. If we also want to create a Fast Start or interleaved movie file, we need to take one more step and call FlattenMovieData on the new movie. Our sample application QTMakeMovie does not take this extra step, but the application framework that QTMakeMovie is built upon does provide a way for us to do this. We simply need to open the new movie file and select the "Save As..." menu item in the File menu. This will result in our application executing the QTFrame_SaveAsMovieFile function, which will prompt us for a new file name and location and then call FlattenMovieData to create a Fast Start, interleaved movie file. The call to FlattenMovieData looks like this:
myNewMovie = FlattenMovieData(
myMovie,
flattenAddMovieToDataFork |
flattenForceMovieResourceBeforeMovieData,
&myFile,
sigMoviePlayer,
smSystemScript,
createMovieFileDeleteCurFile |
createMovieFileDontCreateResFile);
By default, FlattenMovieData interleaves the movie data, so there is no need to do anything special to get an interleaved movie file. (Of course, since the movie we built in this article has only one type of media data, it's already interleaved.) We get a Fast Start movie file by specifying the constant flattenForceMovieResourceBeforeMovieData in the second parameter.
This Month's Code
The Code folder accompanying this article contains the project files, source code, and resource data of the sample application QTMakeMovie, for both Macintosh and Windows. QTMakeMovie is just like our basic QTShell application, except that the Test menu contains a single menu item, "Create New Movie". When the user selects that item, QTMakeMovie calls the QTMM_CreateVideoMovie function (defined in Listing 1) to create the "appearing-penguin" movie file.
In addition to showing how to create QuickTime movie files, this month's code takes yet another important step forward. To wit, this code is completely Carbonized, so that the compiled application runs both under the "classic" Macintosh operating systems that support Carbon (namely, Mac OS 8 and 9) and under Mac OS X. Figure 10 shows a frame of the movie created by QTMakeMovie when opened by the application running under Developer Preview 3 of Mac OS X. As you can see, our movie window automatically uses the stylish new "Aqua" interface provided by OS X.
Figure 10. A movie window in Mac OS X.
The code changes made in QTMakeMovie to support Carbon are not very great, since I've been gradually moving toward Carbon-compliant APIs in previous versions of the application framework. The biggest changes concerned the project files for the Macintosh version of the application. As you can see in Figure 11, all the Macintosh-specific libraries have been replaced by the single stub library CarbonLib. (The downside here is that QTMakeMovie will not run on "classic" Macintosh operating systems that do not have the CarbonLib system extension installed.)
Figure 11.The project window for the Carbonized QTMakeMovie.
You can download the Carbon SDK (which includes both the CarbonLib stub library and the CarbonLib system extension) from <http://developer.apple.com/sdk/>.
Tim Monroe has worked at Apple for over 10 years, first as a technical writer in the Inside Macintosh group and later as a software engineer in the QuickTime group. Currently he is developing sample code and utilities for the QuickTime software development kit. You can reach him at monroe@apple.com.
