Volume Number: 25
Issue Number: 03
Column Tag: Road to Code

The Road to Code: Passing the Test

Automated Unit Testing

by Dave Dribin

This month's topic is one of my favorites. Once you've written an application, how do you know that it actually works? First, we have to define what we mean by "works." The word "works" means different things to different people. To the developer, "works" may mean "works as intended." To the end user, "works" often means "works as I expect it to work." In an ideal world, the developer's and end user's viewpoint are the same, but unfortunately that's not always the case.

In this article, we're going to concentrate on the developer's perspective. As a developer, how do you know that your application works as intended? The simple answer to this question is: run your application, do all the stuff you would expect a user to do, and see if it fails. Of course, this is very vague. What's "all the stuff you would expect a user to do?" Does this list of stuff come from your memory? Do you write it down, so you can repeat the test after you've made changes? If you do write it down, how detailed is the list? Also, how do you know if anything is failing? Is failure subjective to your mood, or are there hard and fast criteria that describes the failure cases?

The process of running your application to verify correct behavior is called testing. The individual items in the list of stuff that gets run are called tests. Wouldn't it be great if you could write tests that run and verify themselves? As it turns out, this kind of testing exists and is called automated software testing. Done properly, the tests are run automatically, they are verified, and the results are reported. This kind of testing is very powerful because they can be run automatically and on set schedules, from once a night to every time you build your code.

A lot has been written about software testing. Today, there are generally two kinds of automated testing: acceptance tests and unit tests. Acceptance tests are often called black box tests because you write your tests by only testing exposed functionality. You test an application, as a whole, without concern for how it is actually implemented. If you were testing an email application, you might test if you could actually send and receive email messages.

Unit tests are often called white box tests. The scope of unit tests is to test out individual components of an application, i.e. specific classes or libraries. They are called white box, because, as the developer, you can use your knowledge about the implementation to write more thorough tests.

This article is going to discuss unit tests in more depth. We're not going to discuss acceptance tests, automated or otherwise, because unit tests are easier to get started with and have a big payoff. We're going to discus how to write automated unit tests in Objective-C using the OCUnit testing framework and how to integrate them into Xcode.

Okay, so let's get to the code. Let's take our trusty old Rectangle class and add unit tests to ensure that it is implemented properly. The interface header of the code we are working with is shown in Listing 1.

#import <Foundation/Foundation.h>
@interface Rectangle : NSObject <NSCoding>
{
    float _leftX;
    float _bottomY;
    float _width;
    float _height;
}
@property float leftX;
@property float bottomY;
@property float width;
@property float height;
@property (readonly) float area;
@property (readonly) float perimeter;
-  (id)initWithLeftX:(float)leftX
            bottomY:(float)bottomY
             rightX:(float)rightX
               topY:(float)topY;
@end

Even without seeing the implementation file, we can see a few possible things to test. The area and perimeter are calculated, so we could test that those are calculated properly. But we can also see that the width and height are calculated. Our constructor takes two points, the lower-left and upper-right, but exposes the lower-left, width, and height as properties. We could test that the height and width are calculated properly. Finally, our class implements the NSCoding protocol. We could test that our archiving works and that we can unarchive previously archived objects.

Let's start simple and verify that our area calculation is working. How do we write code to do this? We can start by writing a simple command line application as shown in Listing 2.

#import <Foundation/Foundation.h>
#import "Rectangle.h"
int main(int argc, char * argv[])
{
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
    
    Rectangle * rectangle =
        [[Rectangle alloc]   initWithLeftX:0
                                 bottomY:0
                                  rightX:15
                                    topY:10];
    [rectangle autorelease];
    
    printf("area is: %.2f\n", rectangle.area);
    
    [pool drain];
    return 0;
}

All this does is create a rectangle with a width of 15 and height of 10 and print its area to the console. How do we know the area is correct? Well, being the geometry geniuses that we are, we know the area is the width times the height, thus it should be 150. We run the program and see if we get the correct output:

area is: 150.00

Yup, that's correct! The problem here is that our testing process requires a human to be involved. A person needs to run the test, look at the output, and verify that the output is correct. There's a lot of room for error. People can forget to run this test, and the person running the test may not know what the correct output should be, off the top of their head. So let's get the computer more involved here. Let's encode the correct output in the unit test by placing an if statement checking for the correct area:

    if (rectangle.area == 150.0f)
        printf("area test passed\n");
    else
        printf("area test failed\n");

Now, the human operator does not need to know the correct answer, and can just look at the output for "passed" vs. "failed." Of course, we still need a human to look at the output, which is less than ideal.

We 'd like to have a better way to indicate failure if the area is not correct. Fortunately, there is the concept of assertions. Assertions are conditions that must be true, in order for a computer program to continue execution. The C language has a function called assert that will terminate the program if the condition fails. We could replace our if statement with this:

    assert(rectangle.area == 150.0f);

This does nothing if the condition is true, but terminates the program if the condition is false. To show you what happens when an assertion fails, let me change that line of code to verify against 100, instead of 150. Running with a failed assertion will result in the following console ouput:

Assertion failed: (rectangle.area == 100.0f), function main, file unit_test.m, line 15.

I've truncated the output a bit, but you can see that it prints the failed condition, along with the filename and line number that the failure occurred on. A person running this test program just needs to ensure that the program runs successfully. If they see any assertion failures, they know that a test has failed, and they can investigate further.

At this point, we can beef up our unit tests a bit. Let's add an assertion for the perimeter, too. Our whole unit test application would now look like Listing 3.

#import <Foundation/Foundation.h>
#import "Rectangle.h"
int main(int argc, char * argv[])
{
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
    
    Rectangle * rectangle =
        [[Rectangle alloc] initWithLeftX:0
                                 bottomY:0
                                  rightX:15
                                    topY:10];
    [rectangle autorelease];
    
    assert(rectangle.area == 150.0f);
    assert(rectangle.perimeter == 50.0f);
    
    [pool drain];
    return 0;
}

We could continue further by adding more and more assertions in our main function, but this would soon get unwieldy. We'd be creating lots of unrelated objects and the code would get quite long. Sure, we could break up the parts of the test into individual functions, however it'd be even better to encapsulate our unit tests inside classes.

Let's say we create a RectangleTest class. We could then have methods to test for the different aspects of the Rectangle class:

- (void)testArea
{
    Rectangle * rectangle =
        [[Rectangle alloc] initWithLeftX:0
                                 bottomY:0
                                  rightX:15
                                    topY:10];
    [rectangle autorelease];
    assert(rectangle.area == 150.0f);
}
- (void)testPerimeter
{
    // ...
}
@end

Our main function could just instantiate an instance of this class and call each of the test methods:

    RectangleTest * rectangleTest = [[RectangleTest alloc] init];
    [rectangleTest testArea];
    [rectangleTest testPerimeter];
    [rectangleTest release];

This restructuring of our code is a nice improvement. However, our main function is still bound to get a little unwieldy. If we have tens or hundreds of test classes, each with many test methods, that's a lot of tedious code to write.

Wouldn't it be great if we could just automatically find every test class and automatically call each method that begins with the string "test"? Fortunately, we are not the first people to think this would be a good idea, and someone has already done this work for us. There is a project called OCUnit that does just this!

OCUnit is a framework designed to make writing and running unit tests in Objective-C easier. Frameworks, like OCUnit, that assist writing unit tests are called testing frameworks. Testing frameworks exist in many different languages for many different programming environments. Probably the most popular testing framework is called JUnit for writing unit tests in Java, written by Kent Beck and Erich Gamma. JUnit itself is a Java port of a framework called SUnit, which was written for Smalltalk also by Kent Beck. JUnit has spawned many testing frameworks that follow similar principals, and even share similar names, for example CPPUnit for C++ and NUnit for C#. As Objective-C users, we have OCUnit. These JUnit-style testing frameworks are collectively called xUnit testing frameworks.

Luckily for us, OCUnit is included as part of a standard Xcode installation. You don't need to download or install anything. There's even a file template for creating a new unit test class, as shown in Figure 1. OCUnit is also called SenTestingKit because the project started at a company called Sen:Te. You will see this heritage more as we dig into the code. That's why, for example, the file template includes the <SenTestingKit/SenTestingKit.h> header file.

Figure 1: New test case class template

Before creating some unit tests, we need to talk about our code organization a bit. Where should our unit test classes go? Do we need to create a separate GUI or command line application? If so, how do we run our tests? Xcode provides a special target for unit tests called a unit test bundle.

To create a unit test bundle, select the Project > New Target... and chose Unit Test Bundle from the Cocoa section, as shown in Figure 2. Click Next and use UnitTests as the target name.

Figure 2: New Unit Test Bundle target

So now that we have a unit test bundle, how do we run the tests? Bundles in Mac OS X are neither applications nor frameworks. They're just a module of loadable code, so they cannot be run directly. Xcode comes with an application that loads these test bundles, automatically runs all your tests, and prints the results. The Unit Test Bundle target template automatically sets this up for you.

We need to add some unit test classes to this target to fully demonstrate this, so let's create a new Objective-C test class case using the file template and add it to our UnitTests target. Name this class RectangleTest, since we want to test the Rectangle class, as shown in Figure 3. I always name my test classes the same as the class that they're testing plus a "Test" suffix. This indicates that these are indeed a test class, but also which class is being tested.

Figure 3: Creating RectangleTest class

Let's take a look at what this file template created for us. The header file is shown in Listing 4.

#import <SenTestingKit/SenTestingKit.h>
@interface RectangleTest : SenTestCase {
}
@end

In order for OCUnit to help us out, we do need to follow a few rules. The first rule is that all test classes must derive from SenTestCase instead of NSObject, as shown in the header file. Using SenTestCase allows for OCUnit to find every unit test class easily and provides some base functionality that we can build upon. The implementation file initially contains no methods. The second rule is that test methods must begin with the string "test". OCUnit will automatically call every one of these test methods for us.

Inside each test method, we can include assertions that verify that a test passed. However, we cannot use the standard C assert function. The final rule is that we must use OCUnit's special assert functions. Let's create a simple test method, just to demonstrate how this works.

- (void)testAssertions
{
    STAssertTrue(1 == 1, @"Testing value of 1");
}

The STAssertTrue assertion function is very similar to the standard C assert function except it has an extra argument for a message. To run this test and verify the assertion, all you have to do is build the UnitTests build target. We've never had an Xcode project that has had multiple build targets before, so let's go over how to choose a build target. The easiest way, in my opinion, is to use the Overview toolbar button. From this pulldown menu, chose UnitTests under Active Target, as in Figure 4. The active target is the target that gets built when you select the Build > Build menu or use Command-B. By changing the active target, we are telling Xcode to build the unit test bundle instead of our application.

Figure 4: Changing the Active Target

With the new active target selected, we can now build it. It should build successfully, and at first it doesn't look any different than building an application. If you open up the Build Results window (Command-Shift-B), and you show the build transcript, as in Figure 5, you'll see that it actually runs your tests, too.

Figure 5: Build transcript

You can see that it's running the testAssertions test case of RectangeTest and that it passed. You'll even get a summary of how many tests passed and failed and how long they took to run:

Test Suite '/tmp/build/Debug/UnitTests.octest(Tests)' started at 2009-01-22 12:47:28 -0600
Test Suite 'RectangleTest' started at 2009-01-22 12:47:28 -0600
Test Case '-[RectangleTest testAssertions]' passed (0.000 seconds).
Test Suite 'RectangleTest' finished at 2009-01-22 12:47:28 -0600.
Executed 1 test, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite '/tmp/build/Debug/UnitTests.octest(Tests)' finished at 2009-01-22 12:47:28 -0600.
Executed 1 test, with 0 failures (0 unexpected) in 0.000 (0.003) seconds

Normally, you don't need to look at the build transcript. You can just assume that a successful build means all your tests passed. But what happens when a test assertion fails? Let's try this out by causing our assertion to fail:

- (void)testAssertions
{
    STAssertTrue(1 == 42, @"Testing value of 1");
}

As 1 obviously isn't equal to 42, this assertion will fail. Now, build the test bundle, again. You'll notice that the build fails. The code actually compiles, but a failing assertion causes the build to fail. You'll see an error bubble and you'll also see an error in your Build Results window, as in Figure 6.

Figure 6: Failing build due to an assertion

The build error message looks just like a compiler error, but the message indicates it's a failed assertion:

error: -[RectangleTest testAssertions] : "1 == 42" should be true. Testing value of 1

This error message gives us the test method that failed, along with a handy message telling us what failed. It also includes the message we added to the assertion. You can click on the error message, and it will bring you right to the line of code where the assertion failed, as well.

Now that we've got our unit test bundle in place and we have Xcode setup to run our tests and verify our assertions, we can start writing some real unit tests. Change the RectangleTest implementation to match Listing 5.

#import "RectangleTest.h"
#import "Rectangle.h"
@implementation RectangleTest
- (void)testArea
{
    Rectangle * rectangle =
        [[Rectangle alloc]   initWithLeftX:0
                                 bottomY:0
                                  rightX:15
                                    topY:10];
    [rectangle autorelease];
    
    STAssertEqualsWithAccuracy(rectangle.area, 150.0f, 0.01, nil);
}
- (void)testPerimeter
{
    Rectangle * rectangle =
          [[Rectangle alloc] initWithLeftX:0
                                 bottomY:0
                                  rightX:15
                                    topY:10];
    [rectangle autorelease];
    
    STAssertEqualsWithAccuracy(rectangle.perimeter, 50.0f, 0.01, nil);
}
@end

This looks very similar to our earlier testArea method except that we use an OCUnit assertion function, STAssertEqualsWithAccuracy, instead of assert. One nice thing about OCUnit is that it comes with a bunch of different assertion functions, not just STAssertTrue. See the Xcode documentation for a full list of assertions functions, but some of the most common ones are:

STAssertEquals
STAssertEqualObjects
STAssertNil

This particular assertion function works a little differently than STAssertTrue. It takes four different arguments: an actual value, an expected value, an accuracy delta, and a message. It tests for equality of floating point numbers by comparing the actual and expected values. But because a computer cannot exactly represent floating point numbers, we need to include an accuracy delta that takes care of this slop. In our assertion, if rectangle.area returns 150 plus or minus 0.01, i.e. between 149.99 and 150.01, then the assertion will pass. We are not including an assertion message, hence the final nil.

Before building our test bundle again to watch our tests pass, we need to include the Rectangle class in our UnitTests target. You can do this by double clicking on the Rectange.m file in the Groups & Files section of Xcode. Switch the Targets tab, and check the checkbox next to the UnitTests bundle, as in Figure 7. This means that Rectangle.m will be compiled and linked into the UnitTests bundle.

Figure 7: Modifying Rectangle.m targets

Now, when you build the UnitTests target, it should build successfully. This means our code compiled and our unit tests pass. Just to verify that all is work, let's change the expected value of our area to be 140. Now when we build, you should get a build error:

error: -[RectangleTest testArea] : '150.000000' should be equal to '140.000000' + or - '0.01':

The nice thing about this error message is that it shows you what the actual and expected values are. This can help determine why a test is failing without using a debugger.

Switch the assertion back to 150 so we have passing tests and now revel in the fact that we have some unit tests for our Rectangle class.

One problem with our unit test bundle is that we must remember to change the active target, otherwise it will not get built. I really want to setup Xcode such that our unit tests get run, even when we building our main Rectangles application target. We can do this with target dependencies.

In Xcode, a target can be dependent on another target. This means one target will not get built until another target builds successfully. We're going to use this to make our Rectangles application target dependent on the UnitTests bundle target. Thus, our application will only get built if our unit tests pass.

Open up the Targets section of the Groups & Files section, and double click on the Rectangles application target. With the General tab selected, you should see a list called Direct Dependencies. This is initially empty, but click on the plus button to add a dependency. Select UnitTests and then click on Add Target button. The result is shown in Figure 8.

Figure 8: Adding a target dependency

Now change the active target in the Overview pulldown of the toolbar to be Rectangles, again. When you build this target, it will first build the UnitTests target. If any unit tests fail, it won't even try to build the application. This is a nice setup, so long as the unit tests run fast enough. And proper unit tests should run fast, so fast you won't even notice they're getting run. We can now add more unit test methods to our RectangleTest class to ensure our Rectangle class is working properly.

The Xcode documentation describes two kinds of unit test bundles: dependent and independent. What I described above is called independent test bundles by the documentation. The unit test bundle is completely independent from the main application. The Xcode documentation recommends the dependent variation, though. Having tried the dependent variation, I do not recommend it. Dependent tests run your application in a special "test" mode, where it launches your application, and then it loads and starts the unit tests, before the application is fully initialized. The problem I've found is that parts of your application end up running, and the program can get into some weird states. Also, dependent tests do not work for testing frameworks, command line applications, or static libraries.

The only downside of independent tests mentioned in the documentation is that tests must be run manually. As I showed above, you can work around this by setting up target dependencies. Another downside to independent tests is that code being tested needs to be compiled into both the application target as well as the unit test target. Using a static library for this common code can easily mitigate this problem. How to setup a static library is unfortunately beyond the scope of this article, but I can assure you it's not very difficult.

We've talked what unit testing is and how to write unit tests, and now I want to talk a little bit more about why you should write unit tests. As I mentioned earlier, unit tests typically target individual classes. If you verify that each individual class works as intended, there's a high probability that your application as a whole will work, at least from the developer's viewpoint. Of course, unit testing isn't perfect. Even if you verified that every class works in isolation, when you assemble individual classes into an application, things may not work so well. If you really want to test the final, assembled application, you'll need to work with acceptance tests. Unit tests, however, are easier to write and run than acceptance tests, and you can still test a vast majority of your application with them. But even if you can't perfectly test your application, you can get an enormous benefit by having unit tests alone.

One of the hidden benefits of writing unit tests is that it improves the quality of your code. In order to be tested in isolation, classes need to be written and designed for testability. The attributes that make your code more testable are the same attributes that make your code high quality and clean. Clean code is a very subjective term, but I think most engineers can agree what ugly code looks like. Large classes with tons of methods, highly coupled and interdependent classes than cannot be pulled out from your application individually, and classes that are hard to understand are all good examples of ugly. We're veering off into object-oriented design, but this is important stuff for real-world applications.

Unit testing is also essential for refactoring. Refactoring is the act of making small changes to your application in order to make improvements. The idea is that these small improvements can significantly increase the code quality of your application. Even though refactoring only makes small changes to your application, these changes can still have unintended side effects resulting in broken code. However, having automated unit tests can provide a safeguard. If your tests pass after you've completed your refactoring, you've gained confidence that you haven't broken anything.

Unit testing and testing as a whole is a large topic. Many books have been written on the topic. We've covered the basics of how to write and execute unit tests in Xcode with OCUnit. I'm sure we'll be seeing more of unit testing in future articles.

Dave Dribin has been writing professional software for over eleven years. After five years programming embedded C in the telecom industry and a brief stint riding the Internet bubble, he decided to venture out on his own. Since 2001, he has been providing independent consulting services, and in 2006, he founded Bit Maki, Inc. Find out more at http://www.bitmaki.com/ and http://www.dribin.org/dave/.