Volume Number: 21 (2005)
Issue Number: 8
Column Tag: Programming
by Tim Monroe
In the previous three articles (in MacTech, May, June, and July 2005), we have investigated the QTKit, Apple's new framework for accessing QuickTime capabilities in Cocoa applications and tools. We have used this framework -- introduced in Tiger and also available in Panther with QuickTime 7 -- primarily to build a sample application that can open and display one or more QuickTime movie files (or other media files openable as QuickTime movies) and that supports the standard suite of movie editing and document management operations.
The sample application that we have been developing, KitEez, performs really quite well. Files open and display quickly, and cut-and-paste editing works as smoothly as could be expected. And, as mentioned in one of those previous articles, QTKit-based drag-and-drop performance in most instances greatly exceeds that provided by the standard movie controller component.
But, truth be told, we haven't really asked KitEez to handle any potentially slow operations. Currently it can open only files selectable in the file-opening dialog box, which means that we're not likely to see any real delay when opening a movie file. If KitEez were to support files specified by remote URLs, we'd definitely see some slowdown in movie opening. Also, some operations that we can perform using QuickTime functions -- like importing or exporting movies and large pictures -- can take a considerable amount of time. If KitEez were to support those sorts of operations, we'd want to make sure that the user was able to keep working while they are churning away. That generally means that we'd need to learn how to perform QTKit operations on a background thread.
In this article, we're going to take a look at threading and QTKit. Since QTKit is built on top of QuickTime, it inherits whatever limitations exist in QuickTime with regard to being able to perform operations on background threads. QuickTime has supported threaded operations since Mac OS X 10.3 and later, with QuickTime 6.4 and later. That is to say, it's now possible to move potentially time-consuming operations to a background thread, thereby freeing up the main thread for user interaction. Gone are the days when importing or exporting a large file invariably ties up your QuickTime application.
To achieve this, we'll need to use a few new QuickTime functions to set up the QuickTime environment on a background thread. We'll need to learn how to safely move QTKit objects between threads, and we'll need to be careful about which operations are performed on a background thread. But with these three issues covered, we should be able to provide a far better user experience than is possible in a single-threaded application.
Before we begin, a couple of provisos. It would be good if you were already familiar with application threading in general and with Cocoa's NSThread class in particular. If not, don't worry; you should be able to pick up what you need along the way. Just don't expect a detailed discussion of threading models here. Also, I will be focusing on invoking QTKit and other Cocoa methods on background threads; see the documents mentioned in the References section for more complete information about threading in QuickTime generally.
As mentioned above, our sample application KitEez currently can open only files selectable in the file-opening dialog box, which generally means that it will be opening only local files. In the previous article, we saw that we can also use the initWithURL:error: method to open a remote file specified by a URL. To elicit a URL from the user, we might display a dialog box like the one in Figure 1. (Adding this dialog box to KitEez is left as an exercise for the reader.)
Figure 1: A URL dialog
box
As you know, initWithURL:error: operates asynchronously, just like all the QTKit movie-opening methods; that is to say, it returns almost immediately, so that our application can continue processing while the movie data loads. So we might be tempted to immediately assign the QTMovie object to the movie view in our document window, like this:
if ([QTMovie canInitWithURL:url]) {
movie = [[QTMovie alloc] initWithURL:url error:nil];
if (movie)
[movieView setMovie:movie];
}
This would work fine, from the standpoint of having QuickTime download the movie data without further intervention from our application. Experience has shown, however, that it's best to defer the setMovie: call until a sufficient amount of movie data has been downloaded. To do that, we can install a notification handler for the QTMovieLoadStateDidChangeNotification notification, like this:
if ([QTMovie canInitWithURL:url]) {
movie = [[QTMovie alloc] initWithURL:url error:nil];
if (movie)
[[NSNotificationCenter defaultCenter] addObserver:self
selector:@selector(loadStateChanged:)
name:QTMovieLoadStateDidChangeNotification
object:movie];
}
Whenever the load state of the downloading movie changes, the loadStateChanged: method will be called. (See "Loaded" in MacTech, September 2002 for a complete discussion of movie load states.) One easy implementation of loadStateChanged: is shown in Listing 1.
Listing 1: Handling load state-changed notifications
- (void)loadStateChanged:(NSNotification *)notification
{
if ([[movie attributeForKey:QTMovieLoadStateAttribute]
longValue] >= kMovieLoadStatePlayable) {
[[NSNotificationCenter defaultCenter]
removeObserver:self
name:QTMovieLoadStateDidChangeNotification
object:movie];
[movieView setMovie:movie];
[movie release];
[[movieView movie] play];
}
}
As you can see, we wait until the load state of the movie reaches the kMovieLoadStatePlayable level, at which time we remove the notification listener for the specified notification, call setMovie:, release our QTMovie object (since the movie view will retain it), and then start the movie playing. The kMovieLoadStatePlayable constant is defined in the QuickTime header file Movies.h. Here is the complete set of load state constants:
enum {
kMovieLoadStateError = -1L,
kMovieLoadStateLoading = 1000,
kMovieLoadStateLoaded = 2000,
kMovieLoadStatePlayable = 10000,
kMovieLoadStatePlaythroughOK = 20000,
kMovieLoadStateComplete = 100000L
};
We need to defer the call to setMovie: until the movie is playable to work around a bug in the first release of QTKit. If we didn't do this, and instead just called setMovie: immediately after the call to initWithURL:error:, we would find that the movie view would not automatically redraw itself once any video data had arrived. The workaround is simple enough and indeed provides an easy way for us to start the movie playing at an appropriate time.
It's also useful to know how to get the load state of a QTMovie object when we want to add importing and exporting support to our applications. The reason is simple: whenever we want to export or flatten or save a movie, we need to have all of its movie and media data on hand. The QTMovie method that we use to export or flatten a movie, writeToFile:withAttributes:, internally checks the movie's load state and returns NO if it's not at least kMovieLoadStateComplete. But we might need to make this check ourselves, when adjusting some menu items. If not all the movie and media data is available, then for instance the Export... menu item should not be enabled. Listing 2 shows a segment of a document class' override of the validateMenuItem: method.
Listing 2: Enabling menu items
- (BOOL)validateMenuItem:(NSMenuItem *)menuItem
{
BOOL valid = NO;
SEL action;
action = [menuItem action];
if (action == @selector(doExport:))
valid = ([[movieView movie]
attributeForKey:QTMovieLoadStateAttribute] longValue]
>= kMovieLoadStateComplete);
// other lines omitted...
return valid;
}
Keep in mind that we have not yet reached a position where we need to move any processing out of the main thread and into a secondary or background thread. The periodic movie tasking that is required to keep a remote movie steadily downloading happens automatically on the main thread and does not generally consume so much processor time that the responsiveness of the main thread is adversely impacted. So, although opening a movie specified by a URL may take a significant amount of time, QuickTime (and hence QTKit) already knows how to do that asynchronously without blocking the main thread.
When we move into the realm of movie importing and exporting -- that is, converting potentially large amounts of data -- we cross an important threshold. Although it is certainly possible to export a movie by calling writeToFile:withAttributes: on the main thread, it isn't really advisable, since the call would execute synchronously. Listing 3 shows how not to define the doExport: method referenced in Listing 2.
Listing 3: Exporting a movie as 3GPP
- (IBAction)doExport:(id)sender
{
NSDictionary *dict = [NSDictionary
dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool:YES], QTMovieExport,
[NSNumber numberWithLong:kQTFileType3GPP],
QTMovieExportType, nil];
[[movieView movie] writeToFile:@"/tmp/sample.3gp"
withAttributes:dict];
}
If the movie is very large, this method could take quite a while to complete. During that time, the user would be unable to do anything with our application except move windows around. Not very exciting.
A slightly better solution involves using the movie:shouldContinueOperation:withPhase:atPercent:withAttributes: delegate method described briefly in the previous article. As I mentioned, this is a wrapper around QuickTime's movie progress function, which we have used in earlier articles to display a dialog box showing the progress of the export and to allow the user to cancel the operation. Figure 2 shows the sheet we'll display from within that delegate method.
Figure 2: A progress
sheet
We could implement this delegate method as shown in Listing 4.
Listing 4: Displaying a cancelable progress sheet
- (BOOL)movie:(QTMovie *)movie
shouldContinueOperation:(NSString *)op
withPhase:(QTMovieOperationPhase)phase
atPercent:(NSNumber *)percent
withAttributes:(NSDictionary *)attributes
{
OSErr err = noErr;
NSEvent *event;
double percentDone = [percent doubleValue] * 100.0;
switch (phase) {
case QTMovieOperationBeginPhase:
// set up the progress panel
[progressText setStringValue:op];
[progressBar setDoubleValue:0];
// show the progress sheet
[NSApp beginSheet:progressPanel
modalForWindow:[movieView window] modalDelegate:nil
didEndSelector:nil contextInfo:nil];
break;
case QTMovieOperationUpdatePercentPhase:
// update the percent done
[progressBar setDoubleValue:percentDone];
[progressBar display];
break;
case QTMovieOperationEndPhase:
[NSApp endSheet:progressPanel];
[progressPanel close];
break;
}
// cancel (if requested)
event = [progressPanel
nextEventMatchingMask:NSLeftMouseUpMask
untilDate:[NSDate distantPast]
inMode:NSDefaultRunLoopMode dequeue:YES];
if (event && NSPointInRect([event locationInWindow],
[cancelButton frame])) {
[cancelButton performClick:self];
err = userCanceledErr;
}
return (err == noErr);
}
This is certainly a better solution than having no sheet at all, but it's really not satisfactory. Just distracting the user with a progress bar is not going to make the export go any faster or be any less synchronous. And the manner in which we check for clicks on the Cancel button is not really very good, even if it is the best we can hope for in a non-threaded application.
So we really do need to move to a multithreaded application if we want to be able to provide acceptable responsiveness in our application's user interface while performing potentially lengthy operations like exporting a movie. As we'll see, spawning a thread to execute some code in a Cocoa application is as easy as calling the NSThread method detachNewThreadSelector:toTarget:withObject:. The complexities we shall encounter arise from the fact that QuickTime was not originally written to be thread-safe, and making it work in a threaded environment requires some assistance from the application developer. For some of the theory, consult the documents listed at the end of this article, particularly the Tech Note on threading QuickTime applications. For the moment, we will content ourselves with the practical implications of that theory. In summary, they are these:
(1)Before any QuickTime APIs (including QTKit methods) can be called on a background thread, the function EnterMoviesOnThread must be called on that thread.
(2) After all QuickTime APIs (including QTKit methods) have been called on a background thread, the function ExitMoviesOnThread must be called on that thread.
(3) A movie created on one thread that is to be accessed on some other thread must first be detached from the first thread (by calling DetachMovieFromCurrentThread) and attached to that other thread (by calling AttachMovieToCurrentThread).
(4)QuickTime APIs (including QTKit methods) executing on a background thread may internally attempt to instantiate components that are not thread-safe; when that happens, the result code componentNotThreadSafeErr (-2098) will be returned. In that case, you might want to retry the operation on the main thread.
Implication (4) has an important corollary. Recall from the first article on QTKit that a QTMovie object is a Cocoa representation of a QuickTime movie and a QuickTime movie controller. That is to say, a QTMovie object is associated with a Movie instance and a MovieController instance. Currently, no movie controller components are thread-safe. This means:
(5) All QTMovie objects must be created on the main thread.
In theory, creating a QTMovie object on the main thread and then migrating it to a background thread is no less dangerous than creating it on a background thread. Experience has shown, however, that it appears to be safe to call QTMovie methods on a QTMovie object that has been thus migrated. At any rate, this is the best we can do given the current state of the QTKit and the underlying movie controller components.
Transferring Movies Between Threads
To see how these implications play out in practice, let's walk through some code that exports a QuickTime movie on a background thread. If the application's Export... menu item is connected to the doExport: method, we can start the export process by calling detachNewThreadSelector:toTarget:withObject:, passing the selector of the application's doExportOnThread: method. The doExport: method is shown in Listing 5.
Listing 5: Starting a background export
- (IBAction)doExport:(id)sender
{
NSSavePanel *savePanel = [NSSavePanel savePanel];
Movie qtMovie = [[movieView movie] quickTimeMovie];
int result;
OSErr err = noErr;
result = [savePanel runModal];
if (result == NSOKButton) {
SEL sel = @selector(doExportOnThread:);
[[movieView movie] stop];
err = DetachMovieFromCurrentThread(qtMovie);
if (err)
return;
// show the progress sheet
[NSApp beginSheet:progressPanel
modalForWindow:[movieView window] modalDelegate:nil
didEndSelector:nil contextInfo:nil];
[NSThread detachNewThreadSelector:sel toTarget:self
withObject:[savePanel filename]];
}
}
There is nothing particularly noteworthy here except for the call to DetachMovieFromCurrentThread. The doExport: method is called on the main thread, so we need to explicitly detach the Movie associated with the QTMovie object from the main thread so that it can later be attached to a background thread. Notice also that we have moved the call to display the progress sheet out of the delegate method and into the doExport: method.
Accessing Movies on Background Threads
Now let's start building the doExportOnThread: method. Its declaration should look like this:
- (IBAction)doExportOnThread:(id)sender;
Here, the sender object is in fact the name of the file into which the exported movie is to be written (as you can see from Listing 5). Since doExportOnThread: is to be run on a background thread, it needs to create and release an autorelease pool, and it needs to call EnterMoviesOnThread and ExitMoviesOnThread. Listing 6 shows the basic skeleton of a method that is to support QuickTime calls on a background thread.
Listing 6: Exporting a movie in the background (skeleton version)
- (IBAction)doExportOnThread:(id)sender
{
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc]
init];
OSErr err = EnterMoviesOnThread(0);
if (err)
goto bail;
err = AttachMovieToCurrentThread([movie quickTimeMovie]);
if (err)
goto bail;
// QuickTime/QTKit calls can go in here....
DetachMovieFromCurrentThread([movie quickTimeMovie]);
ExitMoviesOnThread();
bail:
[pool release];
[NSThread exit];
}
The EnterMoviesOnThread function is declared like this:
OSErr EnterMoviesOnThread (UInt32 inFlags);
Currently only one bit in the inFlags parameter is defined, namely kQTEnterMoviesFlagDontSetComponentsThreadMode. Setting this flag forces no change to be made to the Component Manager threading mode. By default, EnterMoviesOnThread automatically sets the Component Manager threading mode to kCSAcceptThreadSafeComponentsOnlyMode, which indicates that only thread-safe components shall be allowed. Since this is the mode we desire, we'll pass 0 when we call EnterMoviesOnThread.
When we are finished using QuickTime API calls on a particular background thread, we need to call ExitMoviesOnThread, which takes no parameters. Each call to EnterMoviesOnThread must be balanced by a call to ExitMoviesOnThread.
The main thing we need to do is add some code that exports the specified movie into the filename indicated by the sender parameter. That code might look like this:
NSDictionary *dict = [NSDictionary
dictionaryWithObjectsAndKeys:
[NSNumber numberWithInt:1], QTMovieExport,
[NSNumber numberWithInt:kQTFileType3GPP],
QTMovieExportType, nil];
[movie writeToFile:sender withAttributes:dict];
Once the writeToFile:withAttributes: method completes, we need to make sure that the progress panel is removed and that the movie is transferred back to the main thread. We can do that by adding one more line of code, just before the bail label:
[self performSelectorOnMainThread:
@selector(finishedExporting) withObject:nil
waitUntilDone:YES];
Listing 7 shows our implementation of the finishedExporting method.
Listing 7: Cleaning up after an export operation
-(void)finishedExporting
{
[NSApp endSheet:progressPanel];
[progressPanel close];
AttachMovieToCurrentThread([movie quickTimeMovie]);
}
And so we are done building the doExportOnThread: method and the methods it calls.
Handling the Cancel Button
One final task awaits us, namely displaying the progress sheet, updating its progress bar, and handling clicks on the Cancel button. It turns out that we already have most of the code we need at hand, in the form of our movie:shouldContinueOperation:withPhase:atPercent:withAttributes: delegate method. The first thing we need to change is the cheesy way in which we detect clicks on the Cancel button. In the nib file, we configure that button to initiate the doCancel: action, implemented in Listing 8.
Listing 8: Handling clicks on the Cancel button
- (IBAction)doCancel:(id)sender
{
cancel = YES;
}
Then, in the delegate method, we toss all the code that looks for mouse-up events in the button and replace it with this easy test:
if (cancel) err = userCanceledErr;
Also, we cannot set the values of the text field and the progress bar from within the delegate method, because this delegate method wraps a movie progress function, which is called on the same thread as the export operation -- that is, on a background thread. In general, a background thread must never directly alter the application's user interface. What we need to do is have the delegate method use the NSObject method performSelectorOnMainThread:withObject:waitUntilDone:, as we did earlier. So we'll rework the various case blocks like this:
case QTMovieOperationUpdatePercentPhase: // update the percent done [self updateProgress:op toNumber:percent]; break;
Listing 9 shows our definition of updateProgress:toNumber:.
Listing 9: Sending UI updates to the main thread
- (void)updateProgress:(NSString*)msg
toNumber:(NSNumber*)value
{
NSDictionary* dict = [NSDictionary
dictionaryWithObjectsAndKeys:msg, @"msg",
value, @"value", nil];
[self performSelectorOnMainThread:
@selector(updateProgressInMainThread:)
withObject:dict waitUntilDone:NO];
return ;
}
Finally, Listing 10 shows the code we run on the main thread to update the items in the progress panel.
Listing 10: Updating the progress panel
- (void)updateProgressInMainThread:(NSDictionary*)dict
{
NSString* msg = [dict objectForKey:@"msg"];
double value = [[dict objectForKey:@"value"] doubleValue]
* 100.0;
[progressText setStringValue:msg];
[progressBar setDoubleValue:value];
}
For completeness, let's take a last look at the revised version of the delegate method we are using to drive the progress updating (Listing 11).
Listing 11: Displaying a cancelable progress sheet (revised)
- (BOOL)movie:(QTMovie *)movie
shouldContinueOperation:(NSString *)op
withPhase:(QTMovieOperationPhase)phase
atPercent:(NSNumber *)percent
withAttributes:(NSDictionary *)attributes
{
OSErr err = noErr;
switch (phase) {
case QTMovieOperationBeginPhase:
// set up the progress panel
[self updateProgress:op toNumber:
[NSNumber numberWithLong:0]];
break;
case QTMovieOperationUpdatePercentPhase:
// update the percent done
[self updateProgress:op toNumber:percent];
break;
case QTMovieOperationEndPhase:
break;
}
if (cancel)
err = userCanceledErr;
return (err == noErr);
}
In this article, we've taken a look at executing QTKit methods on secondary threads, in an effort to offload lengthy operations from the main thread and thus improve the responsiveness of our application. In particular, we've seen how to export a large movie without blocking the playback of other movies that we might have open and without preventing the user from opening other movies. The basic rules we need to adhere to are relatively simple: (1) make sure to properly initialize and deinitialize the QuickTime environment on secondary threads (by calling EnterMoviesOnThread and ExitMoviesOnThread); (2) make sure to detach a movie from one thread and attach it to another thread if you need to operate on it in multiple threads (by calling DetachMovieFromCurrentThread and AttachMovieToCurrentThread); and (3) make sure to perform any user interface processing on the main thread.
Credits and References
A few of the routines used here are based on code by Michael B. Johnson. A more exhaustive discussion of threading in QuickTime can be found in Technical Note TN2125, "Thread-safe programming in QuickTime", available at http://developer.apple.com/technotes/tn/ tn2125.html. You can also find a discussion of the QuickTime threading APIs in the QuickTime 6.4 API Reference.
Tim Monroe is a member of the QuickTime engineering team at Apple. You can contact him at monroe@mactech.com. The views expressed here are not necessarily shared by his employer.
