Volume Number: 17 (2001)
Issue Number: 10
Column Tag: QuickTime Toolkit

F/X 2

by Tim Monroe

Using Video Effects with Movie Segments, Images, and Sprites

In the previous QuickTime Toolkit article ("F/X" in MacTech, September 2001), we investigated a few of the most basic ways to use the QuickTime video effects architecture, which allows us to apply video effects to tracks in movies and to images. We saw how to work with generators (zero-source effects) and how to apply a filter to a video track and a transition to a pair of video tracks. We also saw how to specify effects parameters and use the effects parameters dialog box to elicit an effect and some effects parameters from the user.

In this article, we're going to continue working with the QuickTime video effects architecture. We'll see how to apply an effect to part of a movie and how to use an effect as the image for a sprite. We're also going to see how to apply an effect to an image (that is, not to a track in a movie). This will lead us, for the first time, to work directly with image decompressors (since, as we saw last time, effects are rendered by image decompressor components). In fact, the techniques we learn for decompressing image sequences will be useful in the future (perhaps even in our very next article).

Our sample application once again is QTEffects (the same one as in the previous article); its Test menu is shown in Figure 1.

Figure 1: The Test menu of QTEffects.

In this article, we'll see how to handle the fourth menu item and the final two. Let's begin by seeing how to add an effect to a movie segment.

We saw in the previous article that it's fairly simple to add a video effect to an entire track. We just add an effects track that has the same track offset and duration as the source track, and we link the effects track to the source track by creating track references from the effects track to the source track and by setting the input map of the effects track appropriately. The media handler for the source track feeds all of its decompressed frames to the component specified in the effects track, which processes those frames further.

To apply a video effect to only part of a source track requires a bit more work. As we saw briefly last time, we can do this by creating a copy of the track segment that we want to apply the effect to; the effect then uses the track segment copy its source, as shown in Figure 2.

Figure 2: A filter applied to part of a video track

When we want to add a two-source effect to part of a movie, the ideas are fundamentally the same. Suppose we've got a movie with two video tracks that overlap for some part of the movie (as shown in Figure 3).

Figure 3: Two overlapping video tracks

We want to apply a transition during the time the two tracks overlap; to do this, we can make copies of the appropriate track segments and use them as sources for the effects track, as seen in Figure 4.

Figure 4: A transition applied to parts of two video tracks

Given what we learned in the previous article, all we really need to do now is learn how to create a new track that holds only part of the data of an existing track. But in fact we already know how to do that. When we were discussing data references ("Somewhere I'll Find You" in MacTech, October 2000), we saw how to use the InsertTrackSegment function to copy media data from one track to another. We can use that function here to create the video track segment copy, as shown in Listing 1. Notice that we also call CopyTrackSettings to copy the source track matrix, clipping region, graphics mode, and other properties into the destination track.

Listing 1: Creating a copy of a video track segment

mySrcTrack1 = NewMovieTrack(theMovie, myWidth, myHeight, 
            kNoVolume);
if (mySrcTrack1 == NULL)
   return(paramErr);

mySrcMedia1 = NewTrackMedia(mySrcTrack1, VideoMediaType, 
            myTimeScale, NULL, 0);
if (mySrcMedia1 == NULL)
   return(paramErr);

#if COPY_MOVIE_MEDIA
myErr = BeginMediaEdits(mySrcMedia1);
if (myErr != noErr)
   return(myErr);
#endif
myErr = CopyTrackSettings(myVidTrack1, mySrcTrack1);
myErr = InsertTrackSegment(myVidTrack1, mySrcTrack1, 
            theStartTime, theDuration, theStartTime);
if (myErr != noErr)
   return(myErr);

#if COPY_MOVIE_MEDIA
EndMediaEdits(mySrcMedia1);
#endif

The value of the compiler flag COPY_MOVIE_MEDIA determines whether the new track segment contains a copy of the media data in the original video track or the new track segment contains only references to that media data. In QTEffects, we set the value of that flag to 0, to minimize the resulting file size.

To make sure that the original video track is hidden behind the new effects track for the duration of the effect, we need to set the track layer of the effects track to be lower than the track layer of the video track. Toward the beginning of QTEffects_AddEffectToMovieSegment, we call the EffectsUtils_GetFrontmostTrackLayer function to retrieve the lowest layer of any video track in the movie, like this:

myLayer = EffectsUtils_GetFrontmostTrackLayer(theMovie, 
            VideoMediaType);

Once we've created the effects track, we then set its layer like this:

SetTrackLayer(myEffectTrack, myLayer - 1);

Listing 2 shows our definition of EffectsUtils_GetFrontmostTrackLayer.

Listing 2: Finding the lowest layer of a track of a certain kind

short EffectsUtils_GetFrontmostTrackLayer (Movie theMovie, 
            OSType theTrackType)
{
   short      myLayer = 0;
   short      myIndex = 1;
   Track      myTrack = NULL;

   // get the layer number of the first track of the specified kind;
   // if no track of that kind exists in the movie, return 0
   myTrack = GetMovieIndTrackType(theMovie, 1, theTrackType, 
            movieTrackMediaType | movieTrackEnabledOnly);
   if (myTrack == NULL)
      return(myLayer);

   myLayer = GetTrackLayer(myTrack);

   // see if any of the remaining tracks have lower layer numbers
   while (myTrack != NULL) {
      if (myLayer > GetTrackLayer(myTrack))
         myLayer = GetTrackLayer(myTrack);
      myIndex++;
      myTrack = GetMovieIndTrackType(theMovie, myIndex, 
            theTrackType, movieTrackMediaType | 
            movieTrackEnabledOnly);
   }

   return(myLayer);
}

See the file QTEffects.c for the complete definition of the QTEffects_AddEffectToMovieSegment function, which is called in response to the "Add Effect to Movie Segment" menu item.

Up to now, we've considered QuickTime video effects only as applied to movies. It's also possible to apply effects to still images. For instance, Figure 5 shows a still image that's had the emboss effect applied to it.

Figure 5: An image with the emboss effect

Figure 6 shows the same image, with the x-ray version of the color tint effect.

Figure 6: An image with the x-ray color tint effect

In this section, we'll see how to apply a filter to an image. We won't actually learn how to apply a transition to a pair of images, but we'll write our code in such a way that it will be easy for the motivated reader to extend it to do so. See the end of this article for pointers to code that does in fact implement transitions between pairs of images.

Decompressing Images

When we apply an effect to a track in a movie, the data describing the effect is stored in the movie itself (in an effects track, of course) and the effect is rendered automatically by QuickTime when the movie is played. Our job, as we've seen, is simply to create the effects track and link it to its source tracks. QuickTime takes care of the nitty-gritty details of retrieving the effect description, interpreting the effects track input map, and applying the effect to the source tracks.

When we want to apply an effect to an image, however, we're more or less on our own. Our application is going to have to keep track of the relevant effects data (that is, the effect description and the image description, along with the source data) and render the effect itself. Since QuickTime effects are implemented as image decompressor components, we need to open an image decompressor and apply it to the source data (the original image). We've previously worked with image compressors, to compress single images and sequences of images. (See "Honey, I Shrunk the Kids" in MacTech, February 2001.) Now it's time to tackle the other end of the compression/decompression process.

Let's begin by learning how to decompress a single image. Remember that we can compress an image by calling the Image Compression Manager (ICM) functions GetMaxCompressionSize and CompressImage. GetMaxCompressionSize tells us the maximum size of the buffer we'll need to hold a compressed image, and CompressImage actually compresses the image. The source data is stored as a pixel map, and the compressed data is written into a buffer. To decompress an image, we can call DecompressImage, which takes a buffer of data and expands it into a pixel map. DecompressImage is declared essentially like this:

OSErr DecompressImage (Ptr data, ImageDescriptionHandle desc,
            PixMapHandle dst, const Rect *srcRect, 
            const Rect *dstRect short mode, RgnHandle mask);

The data parameter points to the compressed data that we want to decompress, and the dst parameter is a handle to a pixel map into which the data will be decompressed. The desc parameter is a handle to an image description, which specifies (among other things) the format of the compressed data and the bounds of the image. The srcRect parameter specifies which part of the image rectangle we want to decompress. This rectangle must lie within the rectangle whose upper-left corner is (0,Ê0) and whose lower-right corner is ((**desc).width, (**desc).height). To specify the entire source rectangle, we can pass the value NULL for the srcRect parameter.

The dstRect parameter specifies the rectangle into which the image is to be decompressed. Typically we'll decompress into the entire destination pixel map, so we would pass the value (**dst).bounds. The mode parameter indicates the desired transfer mode, which is often srcCopy. Finally, the mask parameter is a handle to a region that specifies a drawing mask (or clipping region) for the destination pixel map; only pixels that lie within the mask are drawn into the destination pixel map. To draw into the entire pixel map, set this parameter to NULL.

Here's a typical call to DecompressImage:

myErr = DecompressImage(myData, myDesc, myPixMap, NULL, 
            (**myPixMap).bounds, srcCopy, NULL);

If this call completes successfully, then we could use myPixMap anywhere we'd use a pixel map; for instance, we could copy it into a window by calling the CopyBits function.

For greater control of decompression operations, we can use the FDecompressImage function. FDecompressImage takes all the parameters of the DecompressImage function, plus a handful of additional parameters that allow us to translate or scale the image during decompression, select a particular image quality, specify a progress function that displays a progress dialog box during lengthy decompressions, and so forth.

It turns out, however, that neither DecompressImage nor FDecompressImage allows us to handle QuickTime video effects. Their main limitation is that they provide no easy way to specify an effect's source or sources. To do that, we need to use ICM functions that decompress an image sequence.

Decompressing Image Sequences

In a previous article ("Honey, I Shrunk the Kids", cited earlier), we compressed a sequence of images using the three standard image compression dialog component functions SCCompressSequenceBegin, SCCompressSequenceFrame, and SCCompressSequenceEnd. The ICM also provides the more general functions CompressSequenceBegin, CompressSequenceFrame, and CDSequenceEnd for initiating and managing a compression sequence. To decompress a sequence of images, we'll use DecompressSequenceBeginS, DecompressSequenceFrameWhen, and CDSequenceEnd. (Notice that CDSequenceEnd can be used to end both an image compression sequence and an image decompression sequence.)

To begin a decompression sequence, we call DecompressSequenceBeginS, which is declared essentially like this:

OSErr DecompressSequenceBeginS (ImageSequence *seqID,
            ImageDescriptionHandle desc, Ptr data, long dataSize,
            CGrafPtr port, GDHandle gdh, const Rect *srcRect,
            MatrixRecordPtr matrix, short mode, RgnHandle mask,
            CodecFlags flags, CodecQ accuracy,
            DecompressorComponent codec);

Some of the parameters here are identical to the parameters of DecompressImage. As with DecompressImage, we pass in a buffer of data, a source rectangle, a transfer mode, and a drawing mask. As with FDecompressImage, we pass in a transformation matrix and a quality setting (in the accuracy parameter). The port and gdh parameters specify the graphics port and graphics device into which the decompressed data will be written. (We shall decompress our data into an offscreen graphics world, in which case we can set gdh to NULL.) The codec parameter specifies the image decompressor component that we want to be used for the decompression sequence; since the image description already indicates the relevant codec, we'll pass NULL in this parameter. Finally, the flags parameter is used to specify any special memory-allocation requirements for the decompressor component; we'll pass 0 to indicate no special requirements here.

DecompressSequenceBeginS uses the image data passed in the data parameter and the other information to preflight the decompression sequence. An instance of the specified decompressor component is opened and initialized, and any additional buffers are allocated. If DecompressSequenceBeginS completes successfully, it returns in the seqID parameter a sequence identifier, which we'll use in subsequent calls to manage the decompression sequence. A sequence identifier is of type ImageSequence, which is declared like this:

typedef long            ImageSequence;

Once we've set up a decompression sequence, we can decompress individual frames of the image sequence by calling DecompressSequenceFrameWhen. We pass in the sequence identifier, the data to be decompressed, and some information about the frame's time location in the sequence. For most filters (one-source effects), the notion of time is not really relevant. But for transitions, the effects components do need to know where in the complete image sequence a particular frame lies. So we need to attach some timing information to the image sequence. We'll do this by creating a time base.

Storing the Decompression Data

Let's see how we can tie this all into our sample application, QTEffects. As you know, the shell application upon which we've built QTEffects is able to open image files in a window, using a graphics importer to draw the image whenever necessary — namely, whenever the image window receives an update event (on Macintosh) or a WM_PAINT message (on Windows). By default, the graphics importer is configured (by a call to GraphicsImportSetGWorld) to draw directly into the image window. To support adding a filter to an image, we need to set the graphics importer to draw into some other location (an offscreen graphics world), which we then use as the source for the effect. When we call DecompressSequenceBeginS, we'll set the image window as the drawing destination. So the original image is first drawn into an offscreen graphics world and then "decompressed" (using an effects component) into the onscreen image window.

Each image window opened by QTEffects therefore needs to have some additional data associated with it. As usual, we store such additional window-specific data in an application data record, a handle to which is stored in the fAppData field of the window data record. Here's how we'll declare the ApplicationDataRecord structure for QTEffects:

typedef struct ApplicationDataRecord {
   OSType                           fEffectType;
   ImageDescriptionHandle   fSampleDescription;
   ImageSequence               fEffectSequenceID;
   QTAtomContainer               fEffectDescription;
   TimeBase                        fTimeBase;
   GWorldPtr                     fGW;
   ImageDescriptionHandle   fGWDesc;
} ApplicationDataRecord, *ApplicationDataPtr, 
            **ApplicationDataHdl;

The fEffectType field specifies the type of filter we want to apply to the image; in QTEffects, this is always kFilmNoiseImageFilterType. The fSampleDescription field is a sample description for the effect, and the fEffectDescription field is an effect description for the effect. The fEffectSequenceID field holds the sequence identifier returned by DecompressSequenceBeginS. The fGW field holds a pointer to the offscreen graphics world that serves as the effect source, and the fGWDesc field is a handle to a second image description, which describes the image in the offscreen graphics world. Finally, the fTimeBase field specifies the time base that we'll use for timing information. Once again, this field is largely nugatory for filters, but I've included it to make it easier to extend this code to support transitions.

When we open a new image window, we'll execute this line of code to create the application data record:

(**theWindowObject).fAppData = 
            QTEffects_InitWindowData(theWindowObject);

The QTEffects_InitWindowData function is defined in Listing 3.

Listing 3: Initializing the data for an image window

Handle QTEffects_InitWindowData 
            (WindowObject theWindowObject)
{
   ApplicationDataHdl         myAppData = NULL;

   // if we already have some window data, dump it
   myAppData = (ApplicationDataHdl)
            QTFrame_GetAppDataFromWindowObject(theWindowObject);
   if (myAppData != NULL)
      QTEffects_DumpWindowData(theWindowObject);

   // allocate and initialize our application data
   myAppData = (ApplicationDataHdl)
            NewHandleClear(sizeof(ApplicationDataRecord));

   return((Handle)myAppData);
}

As you can see, we clear out any existing data attached to the window (by calling QTEffects_DumpWindowData, discussed later) and then call NewHandleClear to allocate a new block of memory to hold an application data record. When the user selects the "Add Film Noise To Image" menu item, we call the QTEffects_AddFilmNoiseToImage function, defined in Listing 4.

Listing 4: Filling in the application data record

void QTEffects_AddFilmNoiseToImage 
            (WindowObject theWindowObject)
{
   ApplicationDataHdl            myAppData = NULL;
   GraphicsImportComponent      myImporter = NULL;
   Rect                                 myRect;

   if (theWindowObject == NULL)
      return;

   myAppData = 
            (ApplicationDataHdl)(**theWindowObject).fAppData;
   if (myAppData == NULL)
      return;

   myImporter = (**theWindowObject).fGraphicsImporter;
   if (myImporter == NULL)
      return;

   GraphicsImportGetBoundsRect(myImporter, &myRect);

   // set up the initial state
   (**myAppData).fSampleDescription = 
            EffectsUtils_MakeSampleDescription(kImageEffectType, 
            myRect.right - myRect.left,
            myRect.bottom - myRect.top);
   (**myAppData).fEffectDescription = 
            EffectsUtils_CreateEffectDescription
            (kImageEffectType, kSourceOneName, kSourceNoneName, 
            kSourceNoneName);
   (**myAppData).fEffectType               = kImageEffectType;
   (**myAppData).fEffectSequenceID      = 0L;
   (**myAppData).fTimeBase                  = NULL;

   QTEffects_SetUpEffectSequence(theWindowObject);
}

QTEffects_AddFilmNoiseToImage creates the sample description and the effect description, using utility functions defined in the file EffectsUtilities.c. It also sets the effect type to the film noise effect, using the constant kImageEffectType (which is defined as kFilmNoiseImageFilterType in the file QTEffects.h). Finally, QTEffects_AddFilmNoiseToImage calls the function QTEffects_SetUpEffectSequence to complete the effects set-up process.

Setting Up the Effect

So far we've managed only to allocate the storage we need to maintain the information about our decompression sequence and to create the sample description and the effect description for the film noise effect. We can go ahead and call DecompressSequenceBeginS, like this:

myErr = DecompressSequenceBeginS(
               &(**myAppData).fEffectSequenceID,
               (**myAppData).fSampleDescription,
               *(**myAppData).fEffectDescription,
               GetHandleSize((**myAppData).fEffectDescription),
               (CGrafPtr)QTFrame_GetPortFromWindowReference(
                     (**theWindowObject).fWindow),
               NULL, NULL, NULL, ditherCopy, NULL, 0,
               codecNormalQuality, NULL);

The first parameter is the location in which a sequence identifier will be returned to us. The next two parameters specify the sample description and effect description, which we created earlier in the QTEffects_AddFilmNoiseToImage function. Notice that the effect description is the buffer of data that is "decompressed" to render the effect. That's right: the effects component takes as its input data the effect description. The image data to which the effect is applied is specified as the source of the effect. (We'll see how to do that in a moment.) We tell the effects component to draw the rendered effect into the onscreen image window using this expression:

(CGrafPtr)QTFrame_GetPortFromWindowReference(
            (**theWindowObject).fWindow)

Now we need to allocate the offscreen graphics world into which the graphics importer will draw the image and from which the effects component will take its source data. We can do that like this:

GraphicsImportGetBoundsRect(myImporter, &myRect);
HLock((Handle)myAppData);

// allocate a new GWorld
myErr = QTNewGWorld(&(**myAppData).fGW, 32, &myRect, NULL, 
            NULL, kICMTempThenAppMemory);

The kICMTempThenAppMemory flag tells QuickTime to try to allocate the offscreen graphics world from any available memory that's not assigned to any running process; if there isn't enough of that memory, QuickTime allocates the graphics world from the application's heap.

Once we've successfully allocated the offscreen graphics world, we want to draw the original image into it. We can accomplish this with two easy graphics importer calls:

GraphicsImportSetGWorld(myImporter, (**myAppData).fGW, NULL);
GraphicsImportDraw(myImporter);

Next, we need to set the image in this offscreen graphics world to be the source data for the image sequence. We'll use the CDSequenceNewDataSource function to create a new data source and the CDSequenceSetSourceData function to install that source as the image sequence source. CDSequenceNewDataSource takes the sequence identifier and an image description for the source and returns a value of type ImageSequenceDataSource:

myErr = CDSequenceNewDataSource
            ((**myAppData).fEffectSequenceID, &mySrc, 
            kSourceOneName, 1, (Handle)(**myAppData).fGWDesc, 
            NULL, 0);

We can create the image description contained in (**myAppData).fGWDesc by calling the MakeImageDescriptionForPixMap function. Once we've got the new source identifier, we can call CDSequenceSetSourceData:

CDSequenceSetSourceData(mySrc, GetPixBaseAddr(mySrcPixMap), 
            (**(**myAppData).fGWDesc).dataSize);

We're almost done setting up the decompression sequence. All that remains is to create a time base and attach it to the decompression sequence.

(**myAppData).fTimeBase = NewTimeBase();

SetTimeBaseRate((**myAppData).fTimeBase, 0);
myErr = CDSequenceSetTimeBase
            ((**myAppData).fEffectSequenceID, 
            (**myAppData).fTimeBase);

Notice that we set the time base rate to 0, since the effect is going to be run outside of a QuickTime movie. We can't count on QuickTime to run the effect for us, so we're going to have to call DecompressSequenceFrameWhen ourselves. Before we get to that, however, let's take a look at the complete definition of QTEffects_SetUpEffectSequence (Listing 5).

Listing 5: Setting up the effect decompression sequence

static OSErr QTEffects_SetUpEffectSequence 
            (WindowObject theWindowObject)
{
   ApplicationDataHdl            myAppData = NULL;
   ImageSequenceDataSource      mySrc = 0;
   PixMapHandle                     mySrcPixMap = NULL;
   GraphicsImportComponent      myImporter = NULL;
   Rect                                 myRect;
   OSErr                              myErr = paramErr;

   myAppData = (ApplicationDataHdl)
            QTFrame_GetAppDataFromWindowObject(theWindowObject);
   if (myAppData == NULL)
      goto bail;

   // if an effect sequence is already set up, end it
   if ((**myAppData).fEffectSequenceID != 0L) {
      CDSequenceEnd((**myAppData).fEffectSequenceID);
      (**myAppData).fEffectSequenceID = 0L;
   }

   // if there is a timebase already set up, dispose of it
   if ((**myAppData).fTimeBase != NULL) {
      DisposeTimeBase((**myAppData).fTimeBase);
      (**myAppData).fTimeBase = NULL;
   }

   // make an effects sequence
   HLock((Handle)(**myAppData).fEffectDescription);

   // prepare the decompression sequence for playback
   myErr = DecompressSequenceBeginS(
               &(**myAppData).fEffectSequenceID,
               (**myAppData).fSampleDescription,
               *(**myAppData).fEffectDescription,
               GetHandleSize((**myAppData).fEffectDescription),
               (CGrafPtr)QTFrame_GetPortFromWindowReference(
                     (**theWindowObject).fWindow),
               NULL,
               NULL,
               NULL,
               ditherCopy,
               NULL,
               0,
               codecNormalQuality,
               NULL);

   HUnlock((Handle)(**myAppData).fEffectDescription);
   if (myErr != noErr)
      goto bail;

   // create the offscreen GWorld holding the original image data
   myImporter = (**theWindowObject).fGraphicsImporter;
   if (myImporter == NULL)
      goto bail;

   // set the size of the GWorld
   GraphicsImportGetBoundsRect(myImporter, &myRect);

   HLock((Handle)myAppData);

   // allocate a new GWorld
   myErr = QTNewGWorld(&(**myAppData).fGW, 32, &myRect, NULL, 
            NULL, kICMTempThenAppMemory);
   if (myErr != noErr)
      goto bail;

   // lock the pixmap
   LockPixels(GetGWorldPixMap((**myAppData).fGW));

   GraphicsImportSetGWorld(myImporter, (**myAppData).fGW, 
            NULL);
   GraphicsImportDraw(myImporter);

   // get the pixel maps for the GWorlds
   mySrcPixMap = GetGWorldPixMap((**myAppData).fGW);
   if (mySrcPixMap == NULL)
      goto bail;

   // make the effect source
   if ((**myAppData).fGW == NULL)
      goto bail;

   myErr = MakeImageDescriptionForPixMap(mySrcPixMap, 
            &(**myAppData).fGWDesc);
   if (myErr != noErr)
      goto bail;

   myErr = CDSequenceNewDataSource
            ((**myAppData).fEffectSequenceID, &mySrc, 
            kSourceOneName, 1, (Handle)(**myAppData).fGWDesc, 
            NULL, 0);
   if (myErr != noErr)
      goto bail;

   CDSequenceSetSourceData(mySrc, GetPixBaseAddr(mySrcPixMap), 
            (**(**myAppData).fGWDesc).dataSize);

   // create a new time base and associate it with the decompression sequence
   (**myAppData).fTimeBase = NewTimeBase();
   myErr = GetMoviesError();
   if (myErr != noErr)
      goto bail;

   SetTimeBaseRate((**myAppData).fTimeBase, 0);
   myErr = CDSequenceSetTimeBase
            ((**myAppData).fEffectSequenceID, 
            (**myAppData).fTimeBase);

bail:
   HUnlock((Handle)myAppData);

   return(myErr);
}

Running the Effect

The essential step that remains is to call DecompressSequenceFrameWhen to draw the image, with the film noise effect, into the onscreen image window. For most filters, applying an effect is a one-shot deal. That is to say, we really need to call DecompressSequenceFrameWhen only once to get the full visual effect (ignoring of course any redrawing that is required to handle update events and paint messages). But the film noise effect is an oddball here, since the hairs and scratches applied to the image change over time. The film noise effect isn't a transition, but it is sensitive to the passage of time. So we want to call DecompressSequenceFrameWhen repeatedly. Our standard way to do that is to add some code to the QTApp_Idle function; Listing 6 shows the lines we'll add to our idle-time handler.

Listing 6: Tasking the effect decompression sequence

if ((**myWindowObject).fGraphicsImporter != NULL) {
   ApplicationDataHdl         myAppData;

   myAppData = 
            (ApplicationDataHdl)(**myWindowObject).fAppData;
   if (myAppData != NULL)
      if ((**myAppData).fEffectSequenceID != 0L)
         QTEffects_RunEffect(myWindowObject, 0);
}

If myWindowObject picks out an image window that has an active effect, then we call the function QTEffects_RunEffect to run the effect.

QTEffects_RunEffect is fairly simple; indeed, it consists largely of a call to DecompressSequenceFrameWhen. The only complication is that we need to specify a time value when we call DecompressSequenceFrameWhen, passing in an ICM frame time record (of type ICMFrameTimeRecord). The ICM frame time record is declared like this:

struct ICMFrameTimeRecord {
   wide                  value;
   long                  scale;
   void                  *base;
   long                  duration;
   Fixed               rate;
   long                  recordSize;
   long                  frameNumber;
   long                  flags;
   wide                  virtualStartTime;
   long                  virtualDuration;
};

For rendering a filter, most of these fields can be set to 0 (except for recordSize, which should of course be sizeof(ICMFrameTimeRecord)). The QTEffects_RunEffect function, shown in Listing 7, sets them to values that are appropriate when running a transition.

Listing 7: Running the effect decompression sequence

OSErr QTEffects_RunEffect 
            (WindowObject theWindowObject, TimeValue theTime)
{
   ApplicationDataHdl         myAppData = NULL;
   ICMFrameTimeRecord         myFrameTime;
   OSErr                           myErr = paramErr;

   myAppData = (ApplicationDataHdl)
            QTFrame_GetAppDataFromWindowObject(theWindowObject);
   if (myAppData == NULL)
      goto bail;

   if (((**myAppData).fEffectDescription == NULL) || 
            ((**myAppData).fEffectSequenceID == 0L))
      goto bail;

   // set the timebase time to the step of the sequence to be rendered
   SetTimeBaseValue((**myAppData).fTimeBase, theTime, 
            gNumberOfSteps);

   myFrameTime.value.hi         = 0;
   myFrameTime.value.lo         = theTime;
   myFrameTime.scale               = gNumberOfSteps;
   myFrameTime.base               = 0;
   myFrameTime.duration         = gNumberOfSteps;
   myFrameTime.rate               = 0;
   myFrameTime.recordSize      = sizeof(myFrameTime);
   myFrameTime.frameNumber      = 1;
   myFrameTime.flags               = 
            icmFrameTimeHasVirtualStartTimeAndDuration;
   myFrameTime.virtualStartTime.lo      = 0;
   myFrameTime.virtualStartTime.hi      = 0;
   myFrameTime.virtualDuration            = gNumberOfSteps;

   HLock((Handle)(**myAppData).fEffectDescription);

   myErr = DecompressSequenceFrameWhen(
               (**myAppData).fEffectSequenceID,
               *((Handle)(**myAppData).fEffectDescription),
               GetHandleSize
                     ((Handle)(**myAppData).fEffectDescription),
               0, NULL, NULL, &myFrameTime);

   HUnlock((Handle)(**myAppData).fEffectDescription);

   if (myErr != noErr)
      goto bail;

bail:
   return(myErr);
}

Notice that we've passed NULL as the fifth parameter to DecompressSequenceFrameWhen. If instead we were to pass a pointer to a variable of type CodecFlags (which is an unsigned short integer), then DecompressSequenceFrameWhen would return in that location a set of decompression status flags that give us information about the just-completed decompression operation. As of QuickTime 5.0, these flags are defined:

enum {
   codecFlagOutUpdateOnNextIdle                  = (1L << 9),
   codecFlagOutUpdateOnDataSourceChange      = (1L << 10),
   codecFlagSequenceSensitive                     = (1L << 11),
   codecFlagOutUpdateOnTimeChange               = (1L << 12),
   codecFlagImageBufferNotSourceImage         = (1L << 13),
   codecFlagUsedNewImageBuffer                  = (1L << 14),
   codecFlagUsedImageBuffer                        = (1L << 15)
};

We can inspect the codecFlagOutUpdateOnTimeChange flag (which is, alas, currently undocumented) to see whether we should render the effect repeatedly, as time changes. If that flag is clear, then we need to call DecompressSequenceFrameWhen again only when the source image changes. I'll leave it as an exercise for the reader to modify QTEffects to avoid calling DecompressSequenceFrameWhen unnecessarily.

Finishing Up

So we've completed the work required to apply a video effect to an image. When the user closes the image window, we need to call CDSequenceEnd to end the decompression sequence and then dispose of any additional memory we allocated to run the sequence. Listing 8 shows our definition of QTEffects_DumpWindowData, which handles all this clean-up.

Listing 8: Disposing of the application data

void QTEffects_DumpWindowData (WindowObject theWindowObject)
{
   ApplicationDataHdl      myAppData = NULL;

   myAppData = (ApplicationDataHdl)
            QTFrame_GetAppDataFromWindowObject(theWindowObject);
   if (myAppData != NULL) {
      if ((**myAppData).fGWDesc != NULL)
         DisposeHandle((Handle)(**myAppData).fGWDesc);

      if ((**myAppData).fGW != NULL)
         DisposeGWorld((**myAppData).fGW);

      if ((**myAppData).fSampleDescription != NULL)
         DisposeHandle
            ((Handle)(**myAppData).fSampleDescription);

      if ((**myAppData).fEffectDescription != NULL)
         QTDisposeAtomContainer
            ((**myAppData).fEffectDescription);

      if ((**myAppData).fEffectSequenceID != 0L)
         CDSequenceEnd((**myAppData).fEffectSequenceID);

      if ((**myAppData).fTimeBase != NULL)
         DisposeTimeBase((**myAppData).fTimeBase);

      DisposeHandle((Handle)myAppData);
      (**theWindowObject).fAppData = NULL;
   }
}

Let's continue our investigation of the QuickTime video effects architecture by learning how to use a video effect as a sprite image. In a previous article ("An Extremely Goofy Movie" in MacTech, April 2001), we saw how to use a video track as a sprite image override, so that the sprite uses the frames in the video track as the source for its images. It's just as easy to use an effect as the source for a sprite's images, and this opens up the door to some truly impressive QuickTime movies. Figure 7 shows a simple example, where the image of a sprite is provided by the fire effect.

Figure 7: A sprite image overridden by the fire effect

Figure 8 shows another simple example, where the image of a sprite is provided by the ripple effect. The ripple effect makes it appear that the penguin is submerged in a pool of water that gently undulates. (This is rather difficult to see from a single screen shot, however.)

Figure 8: A sprite image overridden by the ripple effect

In this section, we'll see how to create a sprite movie that uses an effect to supply the images for one of the sprites in the movie. We'll also see how to pass user actions to an effects component, to take advantage of any special capabilities of that component.

Using Effects as Image Overrides

In the movie shown in Figure 8, the sprite track contains two sprites: (1) our standard penguin sprite and (2) a sprite whose bounding box fills the entire movie rectangle. We use the ripple effect as an image override for the second sprite, so that the entire movie, including the penguin, appears to be under water.

When the user selects the "Make Sprite Effect Movie..." menu item, QTEffects calls the QTEffects_MakeSpriteEffectMovie function. We won't consider this function in detail, as it's virtually identical to functions we've considered in the past (for instance, see QTSprites_CreateSpritesMovie in "A Goofy Movie" in MacTech, March 2001). The important step in QTEffects_MakeSpriteEffectMovie consists of a single line of code, which we use to add the appropriate samples to the sprite media:

QTEffects_AddPenguinMovieSamplesToMedia(myMedia);

Recall that a sprite track consists of one or more key frame samples, which contain the images for the sprites in the track and which also specify the initial properties of those sprites. (Sprite tracks can also contain override samples, to animate the sprite by specifying changes to the sprite properties; our current sprite movie does not contain any override samples.) The image data for the penguin sprite is stored in an atom of type kSpriteImageDataAtomType in the key frame sample and consists of an image description followed immediately by the sprite image data. We specify the penguin image data in our standard way, by calling the utility function SpriteUtils_AddPICTImageToKeyFrameSample. The image data for the ripple sprite is also stored in an atom of type kSpriteImageDataAtomType; in this case, however, the atom data consists of an image description followed immediately by the effect description for the desired effect. To add the image data for the ripple sprite, we call the function QTEffects_AddRippleEffectAsSpriteImage, defined in Listing 9.

Listing 9: Adding an effect as a sprite image

void QTEffects_AddRippleEffectAsSpriteImage 
            (QTAtomContainer theKeySample, QTAtomID theImageID)
{
   ImageDescriptionHandle      mySampleDesc = NULL;
   QTAtomContainer                  myEffectDesc = NULL;
   OSType                              myType = kWaterRippleCodecType;
   OSErr                              myErr = noErr;

   // create a sample description
   mySampleDesc = EffectsUtils_MakeSampleDescription(myType, 
            kPenguinTrackWidth, kPenguinTrackHeight);
   if (mySampleDesc == NULL)
      goto bail;

   // create an effect description
   myEffectDesc = EffectsUtils_CreateEffectDescription(myType, 
            kSourceNoneName, kSourceNoneName, kSourceNoneName);
   if (myEffectDesc == NULL)
      goto bail;

   SpriteUtils_AddCompressedImageToKeyFrameSample
            (theKeySample, mySampleDesc, 
            GetHandleSize(myEffectDesc), *myEffectDesc, 
            theImageID, NULL, NULL);

bail:
   if (mySampleDesc != NULL)
      DisposeHandle((Handle)mySampleDesc);

   if (myEffectDesc != NULL)
      QTDisposeAtomContainer(myEffectDesc);

   return;
}

This is entirely straightforward: create a sample description for the ripple effect, create an effect description with no sources, and then call SpriteUtils_AddCompressedImageToKeyFrameSample to add the sample description and the effect description as the ripple sprite's image data.

For this movie to work properly, the ripple sprite must be situated in front of the penguin sprite, since the ripple effect is applied only to the movie area that lies underneath the ripple sprite. We can accomplish this, of course, by appropriately setting the sprite layer. In QTEffects, we'll set the penguin sprite's layer to 0 and the ripple sprite's layer to –1 when we create the sprite key frame sample (in QTEffects_AddPenguinMovieSamplesToMedia, defined below).

Passing Clicks to an Effects Component

The ripple effect component has a very cool feature: if the user clicks on a sprite whose image is supplied by the ripple effect, then additional, concentric ripples are drawn to simulate a stone's having been dropped in the water at the point of the mouse click. Figure 9 shows a few frames of the penguin movie immediately after the user has clicked the mouse button.

Figure 9: New ripples from a user click

Remember that the user's clicks on the ripple sprite are intercepted by the movie controller and passed to the sprite media handler for processing. We can instruct the sprite media handler to send them to the ripple effect component by adding some wiring to the sprite, like this:

WiredUtils_AddQTEventAndActionAtoms(mySpriteData, 
            kParentAtomIsContainer, kQTEventMouseClick, 
            kActionSpritePassMouseToCodec, NULL);

The kActionSpritePassMouseToCodec action tells the sprite media handler to pass the current location of the cursor to whatever component is drawing the sprite's image. Not all components can do anything useful with that information; in fact, the ripple component is currently the only effects component that accepts mouse locations.

Listing 10 shows the complete definition of QTEffects_AddPenguinMovieSamplesToMedia.

Listing 10: Adding samples to the ripple penguin sprite track

static void QTEffects_AddPenguinMovieSamplesToMedia 
               (Media theMedia)
{
   QTAtomContainer         mySample = NULL;
   QTAtomContainer         mySpriteData = NULL;
   RGBColor                  myKeyColor;
   Point                     myLocation;
   short                     isVisible, myIndex, myLayer;
   OSErr                     myErr = noErr;

   // create a new, empty key frame sample
   myErr = QTNewAtomContainer(&mySample);
   if (myErr != noErr)
      goto bail;

   myKeyColor.red = myKeyColor.green = myKeyColor.blue = 
            0xffff;      // white

   // add images to the key frame sample
   SpriteUtils_AddPICTImageToKeyFrameSample(mySample, 
            kPenguinPictID, &myKeyColor, 1, NULL, NULL);
   QTEffects_AddRippleEffectAsSpriteImage(mySample, 2);

   myErr = QTNewAtomContainer(&mySpriteData);
   if (myErr != noErr)
      goto bail;

   // the penguin sprite
   myLocation.h      = 0;
   myLocation.v      = 0;
   isVisible         = true;
   myIndex            = 1;
   myLayer            = 0;

   SpriteUtils_SetSpriteData(mySpriteData, &myLocation, 
            &isVisible, &myLayer, &myIndex, NULL, NULL, NULL);
   SpriteUtils_AddSpriteToSample(mySample, mySpriteData, 1);

   QTDisposeAtomContainer(mySpriteData);

   myErr = QTNewAtomContainer(&mySpriteData);
   if (myErr != noErr)
      goto bail;

   // the ripple sprite
   myLocation.h      = 0;
   myLocation.v      = 0;
   isVisible         = true;
   myIndex            = 2;
   myLayer            = -1;

   SpriteUtils_SetSpriteData(mySpriteData, &myLocation, 
            &isVisible, &myLayer, &myIndex, NULL, NULL, NULL);
   WiredUtils_AddQTEventAndActionAtoms(mySpriteData, 
            kParentAtomIsContainer, kQTEventMouseClick, 
            kActionSpritePassMouseToCodec, NULL);
   SpriteUtils_AddSpriteToSample(mySample, mySpriteData, 2);

   SpriteUtils_AddSpriteSampleToMedia(theMedia, mySample, 
            kSpriteMediaFrameDurationPenguin, true, NULL);

bail:
   if (mySample != NULL)
      QTDisposeAtomContainer(mySample);

   if (mySpriteData != NULL)
      QTDisposeAtomContainer(mySpriteData);
}

For fun, you might try changing kQTEventMouseClick into kQTEventMouseMoved. In that case, the additional concentric ripples will occur every time you move the cursor when it's over the movie rectangle. Surfs up!

Before we leave the topic of video effects, it's worth mentioning that QuickTime provides a set of low-level APIs that we can use in certain cases where we need greater control over the standard effects parameters dialog box. These low-level calls begin with the prefix "ImageCodec" instead of "QT"; so, for instance, we can call ImageCodecIsStandardParameterDialogEvent in places we previously called QTIsStandardParameterDialogEvent. The parameter lists for these two functions are identical except that the low-level function adds a parameter for a component instance. This allows us to restrict the operation to a specific effects component. For example, consider the standard effects parameters dialog box, shown once again in Figure 10.

Figure 10: The standard effects parameters dialog box

As you can see, all the one-source effects are listed in the upper-left corner of the dialog box. For certain purposes, we might want to display only the parameters that are relevant to a single effect. In that case, a dialog box like the one shown in Figure 11 is preferable.

Figure 11: An effects parameters dialog box for a single effect

We can display the dialog box shown in Figure 11 by calling the ImageCodecCreateStandardParameterDialog function, whose first parameter is a component instance for the desired effects component. Listing 11 shows some code that we might use to do this.

Listing 11: Showing the parameters dialog box for a specific effects component

ComponentDescription      myCD;
Component                     myComponent = NULL;
ComponentInstance            myInstance = NULL;
QTAtomContainer               myParamDesc = NULL;
QTParameterDialog            myEffectsDialog = 0L;

// set up a component description
myCD.componentType               = decompressorComponentType;
myCD.componentSubType            = kBlurImageFilterType;
myCD.componentManufacturer   = 0;
myCD.componentFlags               = 0;
myCD.componentFlagsMask         = 0;

// find the required component
myComponent = FindNextComponent(myComponent, &myCD);
if (myComponent == NULL)
   return(paramErr);

// open the component
myInstance = OpenComponent(myComponent);

// get the list of parameters for the effect
myErr = ImageCodecGetParameterList(myInstance, &myParamDesc);

// display the dialog box
myErr = ImageCodecCreateStandardParameterDialog(myInstance,
               myParamDesc, myEffectDesc, 0, NULL, 0, 
               &myEffectsDialog);

The low-level APIs are also useful if we want to embed the effects parameter dialog items into a custom dialog box, as illustrated in Figure 12. To do this, we need to call ImageCodecCreateStandardParameterDialog, as in Listing 11, passing a pointer to an existing dialog box as the fifth parameter and the dialog item index of a user item as the sixth parameter. The user item is replaced by the controls from the standard parameters dialog box.

Figure 12: Effects parameters dialog items embedded in a custom dialog box

In general, you should use either the low-level interfaces or the high-level interfaces, but not both. So if we call ImageCodecCreateStandardParameterDialog to display the effects parameters dialog box or to embed some effects parameters controls into a custom dialog box, then we should also call ImageCodecStandardParameterDialogDoAction to process events in the dialog box and ImageCodecDismissStandardParameterDialog to close the dialog box. Certain high-level functions, however, can safely be intermixed with the low-level functions. A good example is QTGetEffectsList, for which there is no low-level equivalent.

The QuickTime video effects architecture provides an easy-to-use but extremely powerful set of tools for adding video effects to movies and images. We can use it to access over 100 different generators, filters, and transitions. In this article and the previous one, we've seen how to add effects to movies, images, and sprite tracks. We've also seen how to display the effects parameters dialog box to allow the user to fine-tune an effect. And we've briefly touched on the low-level effects functions that QuickTime provides. For a more complete example of using these low-level functions, see the QTShowEffect sample code package found at http://developer.apple.com/samplecode/Sample_Code/QuickTime/Effects.htm (along with a handful of other effects-related sample code packages). QTShowEffect also shows how to apply a transition to two images.

Thanks are due once again to Tom Dowdy for reviewing this article and suggesting some improvements.

Tim Monroe is a member of the QuickTime engineering team. You can contact him at monroe@apple.com.