QuickTime and Offscreen GWorlds

Manipulate QuickTime video frame by frame.

By Kent Sandvik, Apple Developer Technical Support

This article explains how to write QuickTime video frames (samples) to offscreen GWorlds, and blit these frames back to the main screen, while avoiding flickering and frame drops if possible. The techniques shown should provide you with the information needed to implement various applications that manipulate QuickTime video frames offscreen.

For instance, you may want to draw something on top of each QuickTime frame displayed, as quickly as possible. Or you want to modify a movie’s pixmaps before they are displayed. In all cases, we don’t want any flickering due to redrawing.

You may want to step through each video frame one at a time, modifying the frame’s pixmap, and you are not worried about frame drops or other time critical conditions. In other words, it is fine for you to browse and modify frames at your own pace because the QuickTime movie is not playing.

Maybe you want to be informed every time QuickTime has drawn a frame, and perform additional operations unrelated to the frame itself. In other words, you are not interested in the actual GWorld of the movie.

We will cover these three cases: how to draw to an offscreen pixmap (remember, QuickTime works only with 32 bit Color Quickdraw), modify the contents and blit this back to the main screen as quickly as possible; how to display and change frame information one frame at at time without playing the movie; and how to be notified when QuickTime has drawn a frame. The sample code also has functions that measure the slowdown that additional drawing imposes on playback, and measures how long the drawing operations take.

We assume that the reader has basic knowledge about GWorld and CopyBits techniques.

Fast Video Frame Drawing

When a QuickTime movie plays, it is of utmost importance that as little as possible is happening beside displaying movie frames on the screen. Time spent doing other activities will provide fewer CPU cycles for QuickTime to read, arrange and display media samples. If the application spends time doing additional activities, it will directly impact the QuickTime movie’s playback performance.

So, whatever we do when we annotate the video frames, we’d better do it quickly. The less we do, the better.

QuickTime will always use only one GWorld for drawing. Most importantly, it uses this single GWorld for decompression purposes, building pixmaps based on sample information in the video track. For instance, if we are dealing with a decompressor that understands temporal compression (where only the changes from frame to frame are recorded), the pixmaps are built using temporal information, and QuickTime will not redraw every pixel in the GWorld. This means that we can’t draw directly on top of existing QuickTime video frame information. If we do, our drawing information will be retained in the GWorld, not erased at all, and we get the dreaded “smearing” effect.

In order for us to control the drawing, or have access to the specific GWorld that QuickTime is using, we need to detour QuickTime to use a known GWorld. We can think in terms of layered drawing environments. We also want to be informed when the frame is drawn into this GWorld. When we’re informed by QuickTime, we can manipulate the GWorld contents, or blit the contents back to the main window port and then do our drawing operations on top of the copied image (this is a layer oriented approach).

To avoid flickering, we need to copy the original video track’s GWorld contents to another GWorld, do our drawing in this second GWorld, and then finally blit this GWorld pixmap to the main screen. This sounds like a lot of work, but you can run tests to measure if performance is suffering from two CopyBits calls.

QuickTime 1.6 introduced the a method to specifying which GWorld a track will be drawn into. The function SetTrackGWorld takes a specified GWorld we have created and tells the QuickTime toolbox to draw a specified video track into a specified GWorld. In addition, with SetTrackGWorld, we can specify that a specific callback function is called when a frame is transferred from a track into this GWorld. When this track transfer callback function is triggered, we can do additional graphics operations on the frame pixmap (or anything else).

As long as we don’t spend too much time inside a track transfer callback, copying from a specified GWorld back to the main screen is reasonably fast. However, we need to always worry about speed. The more work we do before the operation, the better. For instance, we should preallocate any buffers, handles and anything else that does not change dynamically during a track transfer callback. Memory allocation may take more time than expected, especially if the Memory Manager needs to compact the heaps in order to find memory blocks of required size.

Similarly, the less work we do inside the callback, the better. As we will use CopyBits at the end of the track transfer callback, it is important to know how to optimize the CopyBits function.

Preallocation, Precalculation

Before the track transfer callback we create the needed GWorlds and other buffers, and also try to precalculate any values that we know won’t change while we are inside the track transfer callback. To some degree smart compilers will also optimize away unnecessary operations, but it makes sense to do this ourselves as well.

For example, we want to calculate how far we are into the movie based on how many frames have been drawn. We need the total duration of the movie for the calculations; we can’t avoid the fact that we need to calculate the value inside the track transfer callback. However, we could store away the duration value of the movie before the track transfer callback is operating.

CopyBits Optimization

CopyBits optimization is very well documented in the Tech Note “QD21 Of Time and Space and _CopyBits”. Here’s a short summary, a comparison of the techniques covering our example of QuickTime video frame blitting:.

We should try to copy the smallest possible area; the less bytes CopyBits need to move, the better. Now, in the case of QuickTime movies with defined movie box dimensions we can’t avoid this, we need to copy the whole area from the GWorld to the final destination. However, if we know we want to copy parts of the movie, we could cheat and just copy the rectangular area we are interested in.

In most cases we use the srcCopy mode when copying video frames from the offscreen to the main port, and this is the fastest transfer mode. Using other modes, such as transparent or blend mode, will slow down the copy process considerably. Note that adding the dither mode will really slow down CopyBits. Various codecs, such as Cinepak, will dither anyway (from 24 bit mode down to 8 or 16 bit depth of the monitor GWorld).

It makes sense to specify the background and the foreground colors for the current port, so that the background color is white and the foreground color is black. This will provide a hint to CopyBits so that it won’t do colorization operations while copying information. We also assume that the GDevice pixel map’s color table has white in the first position and black in the last position (this is the normal setup of a Macintosh color table).

The alignment of pixels in the source pixel map is also very important. If CopyBits needs to realign the pixels it will slow down the copy process considerably. For instance, 32-bit CPU environments are at their fastest when they can transfer long words aligned on long-word boundaries in memory. This is the reason why AlignWindow is a very important QuickTime call; AlignWindow will move the specified window to the nearest optimal alignment position on the screen (or screen video memory position). This means that if this is done with the destination GWorld, then CopyBits will operate very fast. QuickTime movies are mostly also aligned for four-byte values (movie rectangle dimension are divided by four) so this will help as well. If the movie does not have this property (check this with for instance MoviePlayer), then it’s time to re-crop the movie so that we have such sizes. Sometimes cropping of the original movie might cause odd sized movie rectangle values.

We should also, if possible, use the smallest pixel depth. If we want to operate in 256 color mode we should stick to 8-bit pixel depths. Note that various QuickTime codecs are optimized for various pixel depths. For instance, Cinepak is optimized for 24-bit color mode, and it will dither to 16 and 8-bit screen depths. One issue is if we should stick to the original image depth of the movie and build a GWorld based on this pixel depth, and then later copy from this GWorld to the monitor bit depth GWorld. Or if we should create the offscreen GWorld with the same bit depth as the monitor bit depth. The sample code has both approaches so you could do more thorough tests to narrow down what case is the fastest one in your application.

We should also stick to the same source and destination rectangles so that CopyBits does not need to scale the copied image. In most cases we want to use the movie box rectangle sizes, as we want to blit the whole movie back to the monitor window.

Color QuickDraw expects a color table attached to every indexed pixel map (8 bit mode). Color tables specify what color each pixel value in the pixel map represents. Color QuickDraw often uses the ctSeed field of the color table data structure as a fast check for color table equality. If two color tables have the same ctSeed, then Color QuickDraw often assumes that their color table contents are equal. We could make use of this feature and copy the ctSeed value (long word) from the original color table to the destination color table, and this way CopyBits assumes that the two color tables are identical and will copy the pixels directly without any color mapping. This will improve CopyBits performance in 8-bit mode. We will copy the existing color table from the GDevice, and use it to create the two offscreen GWorlds.

It also makes sense to make an estimate about the slowest platform the application is aimed for. This way we will know if our offscreen operations will work properly for a targeted entry level system or not. One cardinal sin is to implement most of the application using a high performance PowerPC development system, and then do the system testing using a low performance CPU system. If possible test out the graphics drawing ideas with low end machines before you complete the project.

Setting Up the Track Transfer Proc

The basic code we use for setting up the test environment creates a window, gets a QuickTime movie from a file, adjusts the movie box (rectangle) values, and makes sure that our movie GWorld points to the window port, or:

 
SetMovieGWorld(gMovie, (GWorldPtr)gWindow, NULL);

After this we fetch the duration of the movie and find the first suitable video track. Most QuickTime movies have just one video track, so we assume that this is the case in this test application. We also fetch the pixel depth of the movie using a specific call from the QuickTime utilities function library (more about that at the end of this article). Or we could get the monitor bit depth from the monitor GDevice structure. Note that in to speed things up, we have left out the error testing code in the example:

 
gMovieDuration = GetMovieDuration(gMovie);
 
firstVideoTrack = GetMovieIndTrackType(gMovie, 1, VideoMediaType, 
 movieTrackMediaType); 
 
firstVideoMedia = GetTrackMedia(firstVideoTrack); 
mediaPixelDepth = QTUGetVideoMediaPixelDepth(firstVideoMedia, 1);
monitorPixelDepth = (**(**aSavedGDevice).gdPMap).pixelSize;

Then we create our GWorlds we will use for the offscreen handling (one for the video track redirection, and one for the final composition), use SetTrackGWorld to redirect the GWorld drawing to a specified GWorld, and also install the callback:

//note that monitorDept == 0 implies default monitor depth
anErr = NewGWorld(&gTrackGWorld, monitorPixelDepth, &gMovieRect, 
 colorTable, gTrackGDevice, 0);  
anErr = NewGWorld(&gComposeGWorld, monitorPixelDepth, &gMovieRect, 
 colorTable, gComposeGDevice, 0); 

SetTrackGWorld(firstVideoTrack, (CGrafPtr)gTrackGWorld,
 gTrackGDevice,
 NewTrackTransferProc(MyTrackTransferProc), 0L);

We could also specify NULL in place of the GWorld and GDevice parameters. This will trigger our track transfer callback, but will not do any redirection of the frame display to a specified GWorld. We could also use the SetMovieDrawingCompleteProc to install a similar callback for this same reason, to indicate when a frame has been drawn.

SetTrackGWorld(firstVideoTrack, NULL, NULL,                    
 NewTrackTransferProc(MyTrackTransferProc), 0L);

Track Transfer Callback

As mentioned earlier, the less we do inside a track transfer procedure, and the quicker we do what we need to do, the better. The example will show what the estimated frame per second rate is based on calculations from the movie itself, and after the movie has played it will also show how many frames per second were actually drawn.

In most cases, we will do the normal operations required for the CopyBits operation, such as locking down the pixmap handles before CopyBits, store away the current GWorld that we will restore later, and switch over to the new GWorld. Note that when we enter the track transfer callback, the valid GWorld is the correct destination, not the GWorld we specified for the track transfer operation:

offscreenPixMap = GetGWorldPixMap( (GWorldPtr)gTrackGWorld);  
if (!LockPixels(trackPixMap)) goto Closure;
 
offscreenPixMap2 = GetGWorldPixMap( (GWorldPtr)gComposeGWorld);  
if (!LockPixels(composePixMap)) goto Closure;

GetGWorld(&aSavedPort, &aSavedGDevice);  
SetGWorld( (CGrafPtr)gComposeGWorld, NULL);

Now we copy from the GWorld to the composite GWorld. Before that we make sure that the foreground and background colors are set to black and white:

ForeColor(blackColor); BackColor(whiteColor); 
CopyBits( (BitMap *) *trackPixMap, (BitMap *) *composePixMap,
 &gMovieRect, &gMovieRect, srcCopy, NULL );

After this we will do the needed drawing operations by drawing to the composite GWorld, for example:

PenSize(1,1); 
ForeColor(whiteColor); 
FrameRect(&gTimeDurationRect);
percentage = 100L * GetMovieTime(gMovie, NULL) / gMovieDuration;
MoveTo(5,5); 
ForeColor(yellowColor); 
PenSize(4,4); 
LineTo( percentage +5L, 5); 

This will produce the layer effect we wanted. In other words, QuickTime will draw to the offscreen GWorld.

Finally we blit this composite offscreen back to the main window, then restore the GWorlds and unlock the offscreen pixmap handle, and return.

SetGWorld( (CGrafPtr)gWindow, NULL );
ForeColor(blackColor); BackColor(whiteColor); 
 
CopyBits( (BitMap *) *composePixMap, 
 (BitMap *) &gWindow->portPixMap, 
 &gMovieRect, &gMovieRect, srcCopy, NULL );   

SetGWorld(aSavedPort, aSavedGDevice);
UnlockPixels(offscreenPixMap);  UnlockPixels(offscreenPixMap2); 

return anErr;

Frame by Frame Drawing to Offscreen GWorld.

This second technique shows how to draw individual frames to an offscreen GWorld and blit these back to the main screen. In this case we are not worried about drawing speed, so we can afford to write a more generalized, self-contained function that can be called, on demand, from many contexts. The function will both set up the GWorld environments, draw, and tear down the created offscreen GWorlds. We can also afford more extensive error testing to make sure that this function will signal any kind of odd behavior.

In this example, we create a transitional effect between two frames. The frames are specified by two time values. The effect is drawn in offscreen GWorlds.

Our function will begin by querying the movie’s GDevice for the bit depth and color table information. Then, using this information, we call NewGWorld twice to create two separate offscreen GWorlds. Then we redirect QuickTime to draw into one offscreen GWorld, set the movie to the desired time value, and force an update of the movie frame (sample). We do the same thing with the second GWorld using the second time value. After this we could do various other operations. In the example we will scroll in the first frame to the left and the second frame is scrolled in from the right. Inside this scrolling loop we initially blit to the second GWorld from the first to create the scrolling transition effect, then we blit the second GWorld to the main screen. Finally, before terminating, we dispose of any used GWorld heap space.

In this particular case we are not that worried about speed (even if speed should always be a concern). Instead we work on issues related to the what is happening when the frame is drawn offscreen, or what transition effects we could apply when we change the frames.

The CopyBits optimization guidelines, as always, are valid. If we want to create a really graphics intensive transition, and this transition must happen quickly, then we need to optimize the function. For the time being, I like this self-contained design, as it sets up all the needed memory allocations and releases these every time we call the function.

Here’s the code. First we do the normal housekeeping to store away the current GWorld and GDevice, and we also get the pixel sizes and color table for the GDevice that we need later when we create the GWorlds:

GetGWorld(&aSavedPort, &aSavedGDevice);
GetMovieGWorld(theMovie, &moviePort, &movieGDevice);
screenDepth = (**(**aSavedGDevice).gdPMap).pixelSize;
colorTable = (**(**aSavedGDevice).gdPMap).pmTable;

We create two equal GWorlds (including identical color tables) for the offscreen writing later, and also lock down the pixmaps.

anErr = NewGWorld(&frameGWorld1, screenDepth, &movieRect, 
 colorTable, NULL, 0); 
 
anErr = NewGWorld(&frameGWorld2, screenDepth, &movieRect,      
 colorTable, NULL, 0); 

pixMap1 = GetGWorldPixMap(frameGWorld1); 
if (!LockPixels(pixMap1)) goto Closure;
pixMap2 = GetGWorldPixMap(frameGWorld2); 
if ( !LockPixels(pixMap2)) goto Closure;

Now we will start drawing; we will set the movie GWorld so that it points at one of our offscreen GWorlds, set the time value of the movie to a specified time value (in other words go to the time value), update the movie and also call MoviesTask so that the movie frame is indeed drawn into the offscreen GWorld:

 
SetMovieGWorld(theMovie, frameGWorld1, GetGWorldDevice(frameGWorld1));
SetMovieTimeValue(theMovie, fromTimePoint);
UpdateMovie(theMovie); MoviesTask(theMovie, 0);

We will do the same thing with the second frame, but we are going to adjust the time value of the movie so that this other frame is drawn from that particular new time position:

SetMovieGWorld(theMovie, frameGWorld2, GetGWorldDevice(frameGWorld2));
SetMovieTimeValue(theMovie, toTimePoint); 
UpdateMovie(theMovie); MoviesTask(theMovie, 0);

We want to restrict the drawing by clipping to a particular region. In this case clipping to the movie rect makes sense. Before that we will create a place holder for the current clip region (NewRgn) and we will store away the current clip region:

clipRegion = NewRgn();  
GetClip(clipRegion); ClipRect(&movieRect);

The following code will create the scroll effect. What we want to do is to scroll the movie rect nSteps pixels to the left, set the rect again, and copybits first from the first GWorld to the second one, and then from the second one over to the main screen:

scrollRegion = NewRgn();  
screenSize = movieRect.right - movieRect.left;

for(nSteps = 10; nSteps <= screenSize; nSteps += 10)  {
 SetGWorld( frameGWorld1, NULL );
 ForeColor(blackColor); BackColor(whiteColor); 
 
 ScrollRect(&movieRect, -10, 0, scrollRegion);
 SetRect(&sourceRect, movieRect.left, movieRect.top, 
 movieRect.left + nSteps, movieRect.bottom);
 SetRect(&destinationRect, movieRect.right - nSteps,
 movieRect.top, movieRect.right, movieRect.bottom);
 
 CopyBits( (BitMap *) *pixMap2, (BitMap *) *pixMap1, &sourceRect, 
 &destinationRect, srcCopy, NULL );
 SetGWorld(aSavedPort, aSavedGDevice);
 ForeColor(blackColor); BackColor(whiteColor); 
 CopyBits(  (BitMap *) *pixMap1, 
 (BitMap *) &aSavedPort->portPixMap, 
 &movieRect, &movieRect, srcCopy, NULL );                }

Finally we will unlock the pixels, restore the clip region, restore the GWorld/GDevice environment, and destroy any handles we have created inside this function:

UnlockPixels(pixMap1); UnlockPixels(pixMap2);
SetClip(clipRegion);
 
if(frameGWorld1 != NULL)  DisposeGWorld(frameGWorld1);
if(frameGWorld2 != NULL)  DisposeGWorld(frameGWorld2);
if(scrollRegion != NULL)  DisposeRgn(scrollRegion);
if(clipRegion != NULL)    DisposeRgn(clipRegion);

SetMovieGWorld(theMovie, moviePort, movieGDevice);
SetGWorld(aSavedPort, aSavedGDevice);

return anErr;

We could use the function as shown below. ScrollToNextVideoSample is the name of the function we described earlier. This is our WaitNextEventLoop:

 
for(;;) {
 WaitNextEvent(everyEvent, &anEvent, 60, NULL);
 
 if(anEvent.what == mouseDown)
 break;
 if(anEvent.what == keyDown) {
 GetMovieNextInterestingTime(gMovie, nextTimeMediaSample,
 1, &mediaType, gCurrentTime, 0, &gNextTime, NULL);

 if(gNextTime == -1) break;
 
 SetGWorld(gWindow, NULL);
 anErr = ScrollToNextVideoSample(gMovie, gCurrentTime,         
 gNextTime);
 gCurrentTime = gNextTime;
 }
}

Every time we hit a key we will get one video frame at a time using GetMovieNextInterestingTime, get the starting point for the next value, and pass this one and the current value so that we will scroll inside the function from the current one to the next video sample. If GetMovieNextInterestingTime returns -1 it tells us that there are no more video samples. Finally we set the currentTime equal to nextTime so that GetMovieNextInterestingTime has a new starting point forward in time.

The third technique we will describe is how QuickTime informs you when it has drawn a frame. We can use the transfer proc for this; we can also have another callback called a MovieDrawingCompleteProc. If we install this callback, it will be triggered every time QuickTime has drawn a frame.

We will create a universal Proc pointer for the callback, and then install it using SetMovieDrawingCompleteProc. Note that the documentation for this function, available in the QT 04 QuickTime 1.6 Features tech note, is not complete. This function takes four parameters in the Universal Interfaces 2.0, and the second one is a flag stating how this callback is triggered. The old interface is still honored, but your compiler may complain about a problem with the function prototype.

We have the same implementation issues as in the earlier examples. We should spend as little time as possible inside the callback in order to avoid any playback performance degradation.

Here’s an example of how to install this callback:

MovieDrawingCompleteUPP gMovieProc;

gMovieProc = NewMovieDrawingCompleteProc(&MyQTMovieDrawingCompleteProc);

SetMovieDrawingCompleteProc(gMovie, 
 movieDrawingCallWhenChanged, gMovieProc, 0);

We do the exact same drawing as when we used the track transfer proc and the specific GWorld. The difference is that QuickTime will use the same GWorld so what we need to do is to erase rectangular areas so that our drawing will always draw on top of the underlying pixels.

This technique could also be used for simple playback performance measuring. This code is part of the sample provided. If you want to calculate how many frames are drawn per second, install a MovieDrawingCompleteProc, increase a global counter, and calculate the frame rate per defined time interval afterwards.

Debugging

While writing this code I encountered various situations where the code didn’t work (of course), or malfunctioned badly. I tried to document these cases so that others might learn from my mistakes, and avoid potential pitfalls. To speed development and debugging, I used a few techniques described below.

I used many rects in my application, and sometimes the difference between the rects are small, but there’s still a difference. I inserted temporary FrameRect calls here and there to see exactly where the rects are placed inside the PortRect.

Another concern is keeping track of which GWorld to blit from, and where to blit? Sometimes I wrote temporary CopyBits calls that copied pixels from a specific GWorld directly to my Window portRect. This way I knew what the GWorld contained at a particular point of time.

DebugStrs are handy if I want to know if a callback is really triggered or not. Also, the Metrowerks debugger is very good, and there were many times I stepped through the functions in order to know exactly what happened.

I do a lot of error testing in my code (left out from the examples published), and I have a specific assertion macro that will trigger a DebugStr if the assertion is wrong. If possible I try to wrap this macro around any OSErrs returned from the toolbox. In some cases the OSErrs don’t need to be tested in the final application, so I leave these out from the production code.

Here’s an example:

GetMovieNextInterestingTime(theMovie,
 nextTimeMediaSample+nextTimeEdgeOK, (TimeValue)1, 
 &media, 0,  fixed1, &startPoint, NULL);
anErr = GetMoviesError(); DebugAssert(anErr == noErr);
return anErr;

This would catch the problem immediately, DebugAssert will print the file name and line number in MacsBug. If we disable the DEBUG global flag DebugAssert the macro will expand the macro to a NULL statement, and a smart compiler will optimize that statement away.

QTUtilities Library

The sample code is using selected functions from a utility library with QuickTime functions, provided by Apple DTS. This collection of functions provides both the core set of needed QuickTime related operations, as well as examples of operations that you might tweak or modify in your liking. The DTSQTUtilities kit is available from developer CDs, eWorld, AppleLink or Internet:

http://www.info.apple.com/dev/devinfo/quicktime/quicktime.html

And also from Apple’s ftp server:

ftp://ftp.info.apple.com/dts/quicktime/

I hope these examples will inspire you to write more QuickTime based applications and multimedia titles using the techniques shown.

References

QT4 - QuickTime 1.6 Features Tech Note

QD13 - Principia Off-Screen Graphics Environments Tech Note

QD21 - Of Time and Space and _CopyBits Tech Note