TweetFollow Us on Twitter

Automate your testing... with Eggplant

Volume Number: 19 (2003)
Issue Number: 9
Column Tag: Software Testing

Automate your testing... with Eggplant

Exploring a new tool for performing automatic tests

by Dave Kelly

Test Automation on the Mac - A Brief Overview

Your code has bugs in it. What? So, you don't like hearing that, eh? But it's not your fault. All software has bugs. Finding all the bugs in your software is a big job so you need all the help you can get, right?

Bug finding is an art that complements code writing. Somewhere along the way someone (probably an overworked quality engineer) figured out that time and effort could be saved by using automated testing. There are several automation tools available for testing Windows software including software from Rational, Mercury Interactive, Segue Software, and others. But, until recently there was nothing that come close for Mac OS X. In Mac OS X, the tester has had to rely only on AppleScript or Perl for test automation. It doesn't take too long to realize that although they are very powerful tools, neither AppleScript nor Perl are adequate for testing some things. GUIs and functionality that can only be accessed through the software being tested to name a few. Even the new GUI Scripting (System Events) beta software (see http://www.apple.com/applescript/GUI/) does not allow the GUI of some software to be tested. We're now going to take a look at Redstone Software's Eggplant software for testing Mac OS X software.

Redstone Software, Inc. is a subsidiary of Gresham Computing plc, which is based in Southampton, UK. Gresham Computing (GHT) is traded on the London Stock Exchange. In late 2001, Gresham Computing started work on a new product internally named Operation Screaming Eggplant. Their goal was to deliver a product with high ease-of-use for the neophyte yet still offer powerful capabilities for the experienced tester. The Mac OS X platform was chosen as the base OS for this product for two major reasons. First, they wanted to establish the product before competing in the PC marketplace where competition is much more fierce. Second, they believed that Mac OS X had the right features and extensibility to support the enterprise market.

Redstone Software, Inc. was created in early 2002 and in October of 2002 became the first company to deliver a cross-platform automation tool to the Mac OS X platform. They stuck with the code name and named their product Eggplant. Their belief is that automation products of this kind are traditionally too complex and require specialized development skills that are rarely found in testers. In addition, they also believe that very few automation projects have been successful, although there are cases where automation has worked. In these cases automation is performed for tests requiring special or repetitive computations, such as acceptance tests, where test execution and results are predictable.

It might be interesting to note that Redstone Software uses eXtreme Programming (XP). This is very apparent by their willingness to listen to and support their customers. Through this intense support of their customer's needs, they released several versions of Eggplant--the most recent on being version 1.3 in May 2003. The remainder of this article will focus on Eggplant v1.3.

The Eggplant software runs on your host Macintosh. This is the computer that controls the test. The software you are testing runs on a separate computer called the "System-Under-Test" (SUT). The SUT runs Virtual Network Computer (VNC) server software. VNC is a remote display system that allows you to view the desktop of the "server" computer on another computer anywhere on the Internet. The VNC server software could be running on another Macintosh or it could be running on a Unix or Windows computer. Eggplant is able to connect to the SUT using the VNC server software.

VNC originated from AT&T Laboratories Cambridge's development of very-thin-client ATM network computers called the Videotile (see http://www.uk.research.att.com/tile.html). The VNC viewer and server are basically software-only versions of this ATM Network Computer. VNC is free and is distributed under the terms of the GNU Public License. Eggplant incorporates VNC client software to give it full access to the SUT GUI, keyboard, and mouse using any VNC server. Any VNC client "viewer" can display the desktop of any VNC server; however, to use Eggplant you'll have to use the client that is part of Eggplant itself. VNC clients are available for every major OS including: Mac OS 9, OS X, windows 95/98/ME/2000/XP, UNIX (AIX, Solaris, HPUX), and LINUX (see http://www.uk.research.att.com/vnc/). The Redstone VNC server, OSXvnc 1.2 (see http://www.redstonesoftware.com/osxvnc/) is recommended for use with Eggplant because of fixes made that improve performance and increase stability.

The setup for running Eggplant consists of just starting up the 496KB OSXvnc application and starting the server by clicking the "Start Server" button (see figure 1). The OSXvnc application will need to remain running or the connection will be broken. You'll also want to hide the OSXvnc application before running your test.


Figure 1. OSXvnc Interface

Although the real testing happens on the SUT, the host Macintosh is where the action is. When you open Eggplant the first thing you'll want to do is connect to the SUT, if anything else just so you can see how it works. After entering your license you'll see a window that prompts you to establish a connection with a VNC server. You can connect with the IP address, port and password of the SUT. It is also possible to connect using IP over Firewire or through an SSH connection.


Figure 2. Eggplant connection window

Once the connection is established, you see the Eggplant monitor and can control the SUT. The live remote screen in Figure 3 shows the desktop of the SUT.


Figure 3. Eggplant Remote Screen viewer

At this point, the VNC viewer can only control the remote SUT and capture images. That might be useful for some things but you'll need to create a script in order to begin writing a test. You start by creating a suite. When a new suite is created, Eggplant creates a directory structure to store the scripts, images, and log files pertaining to that suite. The suite window in figure 4 is used to access all of the features of the suite.


Figure 4. Eggplant suite window

Now for the heart and soul of the Eggplant... capturing an image. Once you open a new script you're ready to start capturing. Building test scripts is dependent on being able to capture images through the viewer window. Without capturing images, there isn't much that you can do.

The viewer has two modes, Capture Mode and Live Mode. Eggplant starts out in live mode where you can see and control whatever goes on with the SUT. By selecting "Enter Capture Mode", you are able to select a rectangular section of the SUT's screen through the Eggplant viewer, along with a hot spot, represented by a red crosshair (see figure 5). The hot spot represents the point that will be clicked when the script is run. The rectangular section is the area that will be captured as a snapshot. After selecting the image and setting the crosshair to the point you want to click, you can choose between standard commands that will move the mouse and click or type text on the SUT's screen. The standard commands that can be used are Click, DoubleClick, MoveTo, Drag, Add Image, Wait, WaitFor, TypeText and TypeCommand. These pretty much cover anything you will need to do to test your software.


Figure 5. icon selected and ready to capture

After selecting your image, setting the click point, and selecting a command (example "DoubleClick"), Eggplant will add the image to the suite's set of images and add the command to the script. For this example, this line would be added to the current script window:

DoubleClick "image0001"

Meanwhile, the image is added to the suite's images as in figure 6.


Figure 6. suite images

After capturing a sequence of images/commands, you can go back and establish your initial setup conditions and run the script (or select a portion of the script to run). The viewer screen is compared to the saved image for each command and if it finds a match, the command is executed. If it fails, the failure is stored in the suite's Results log. A log is stored for each time you run the script. During execution you can also log your own results from the script.

It should be apparent that the key is having the saved image match the captured image when the script runs. Eggplant does a great job of finding the correct image on the SUT's screen, but there are certain circumstances you must be aware of. For example, if the desktop background looks like a solid background, but isn't really a solid background, then the image won't be found if it is not exactly where it was when the saved image was captured. The solution then is to make sure that the captured images are over a real "solid" background. Also, it is helpful to only capture the minimum that is required to identify the image being compared. For example, it may only be necessary to capture the text of an image, or maybe just the center portion of the image. It only needs to be enough to be identified as a unique image.

Eggplant is very forgiving when matching captured images. When images are captured, Eggplant attempts to automatically detect the right setting for color matching. In most cases it defaults to "Tolerant" which means that it will allow small variations in the color to be accepted. The "Precise" setting will look for a more exact match in the colors. Then there is the special "Pulsing" setting that allows color matching even when the color is a pulsing Mac OS X Aqua button. It's a good thing they included these color matching options so images, such as pulsing buttons, could be matched.

In Search of the "right" Automatic testing tool

It will help if you know what you're looking for in an automatic testing tool. Regrettably, there isn't much available for Mac OS X to compare with when looking for automation tools. It would still be worthwhile to know what you want the test environment to provide and what you're going to have to add to it to make it work for you. The following is a short list of "stuff" to look for in test automation software and comments about how well Eggplant does in each of the areas listed.

  • Will the test tool interfere with the software under test? The VNC server on the SUT is not very intrusive so you'll hardly even know it's there. You'll have to know your system under test well enough to know if a VNC server would have a material impact on it.

    Eggplant relies on an IP connection. If your software makes changes to network settings Eggplant's connection with the SUT could be broken. This behavior needs to be taken into account when writing scripts.

  • Can the test be controlled by scripts? Eggplant has a built in test script language called SenseTalk. SenseTalk is a HyperTalk-like language. The language is quite powerful so you should be able to do many of the things you need to do.

    The script is run on the host Macintosh (not the SUT) so all of the local commands refer to the host. For example, you can read/write to a local disk file (you might do this to record some test results or get data for a test from a file), but the file must be available to the host computer. If you need access to the SUT's files, there is an approach that may be acceptable. The workaround is to turn on file sharing on the SUT and mount the SUT's disk on the host computer.

  • Can scripts be captured or do I have to modify scripts after capturing them to make them work? Since images are an integral part of the commands it is necessary to capture the images that will be compared. You cannot just start typing without also capturing an image. Some automation tools will claim that all you have to do is capture your script and then run it and you're all set. Eggplant is really very different because you are not capturing your script as much as you are capturing images that the script will use. Even with Eggplant scripts don't write themselves (it is wishful thinking when any company claims their tools will write the scripts for you). With Eggplant, the process of writing the scripts includes creating the images that will be used to compare against when the script runs. Fortunately, this process makes it very easy to get the script up and running.

    Redstone Software makes the claims "Manual test-oriented commands allow the non-programmer to quickly create scripts. Eggplant also offers the flexible and robust SenseTalk scripting language and the ability to test multiple applications and platforms", "Eggplant interactive script development simplifies test automation" and SenseTalk scripting language provides high ease of use and powerful capabilities". Unlike other automatic software test applications, they make no claims that the script can be captured and run without modification. There will be a certain amount of script writing to get your tests up and running, but as they claim, Eggplant simplifies script development.

  • Can scripts be reused? Test scripts can call other test scripts. In this sense, "libraries" of scripts can be written to be used over and over. In addition, scripts belonging to other suites can be called by specifying them in the Helper Suites list.

  • Can scripts be scheduled to run at a certain time? The scheduler can setup a selected batch of scripts so that they will run on specific SUTs, but you are not able to set up a specific time for a script to run. You could set up a script that checked the time before it did anything though. Redstone recommends use of the cron function to schedule tests. The batch of scripts can also be "scheduled" to repeat however many times you specify.

  • Can scripts be easily printed/documented? Yes, scripts can be printed. However, images cannot be printed from the Eggplant application. The images are stored with the suite as .tiff files, and those can easily be printed if necessary. Additionally, scripts can contain comments to further document what is being done.

  • Is there a test infrastructure in place? Not completely. The test suite environment of Eggplant includes a script editor, image list (showing images included in the suite), results list, script scheduler, and Helper Suites list. Eggplant could be improved to include a better debugging environment, the ability to link test requirements to tests being run, and an expanded capability to keep track of test results.

  • Is there a mechanism in place to keep track of test status? The Results tab is actually a very nice system for keeping track of test status. Building a test logging mechanism can be a very time consuming task. Fortunately, with Eggplant you don't need to build anything, and the logging is built right into the script language. Unfortunately, there isn't a way to print the results from within the Eggplant application. But, fortunately, the file is stored as a plain text log file that can be easily printed.

  • Is there an extra cost per user? Can multiple tests be run concurrently? If you want to run multiple test streams against different SUTs at the same time, then that requires multiple licenses. The current pricing scheme for multiple licenses on a single host is as follows. The base license allows execution of one Eggplant test stream for $3400. A second test stream costs $1,700 and the third test stream adds $850. For $5,950, one system can execute three different Eggplant test streams.

    Four or more concurrent stream licenses are $425 each. A lower end Mac can easily handle three streams; higher end Macs can of course run more. What many users do is to use one Mac as an Eggplant execution system and have licenses on other machines for script development.

    The Eggplant pricing is based on the number of test streams being run with no additional charge for the number, location, or type of systems-under-test.

  • Is there any debugging capability built-in? Although there is no ability to add breakpoints, any portion of a script can be run at any time by selecting it and clicking on "Run Selection" in the script editor window.

Summary

Eggplant is a fine application. The really cool thing about it is that it can run tests on multiple platforms (all you need is the VNC server for the SUT you're testing on). Please don't think that Eggplant is for software testing only. Developers can automate the build process. System Administrators using Eggplant can automate server maintenance. Any repetitive task is a candidate for automation. This is the first really usable automatic software tool that has been available for Mac OS X. It is very depressing to hear other companies tell you "Sorry, we don't support the Mac". Well, this changes everything because now there's a tool that works with Mac OS X.

Bibliography and References

Kelly, Dave. "Software Automation and the Product Life Cycle". MacTech Magazine (formerly MacTutor) 13:10.

Cem Kaner, Jack Falk, Hung Quoc Nguyen. "Testing Computer Software". 2nd edn. New York 1999.


Dave Kelly is currently a Software Quality Engineer at EarthLink and has many years of software testing experience at EarthLink, Hewlett Packard, Symantec and Xerox. He has worked and played with the Macintosh since 1984 and was one of the founding editorial board members of MacTutor Magazine (now MacTech Magazine). You can reach him at dkelly@earthlink.net.

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

BusyCal 2.6.6 - Powerful calendar app wi...
BusyCal is an award-winning desktop calendar that combines personal productivity features for individuals with powerful calendar sharing capabilities for families and workgroups. BusyCal's unique... Read more
MegaSeg 5.9.6 - Professional MP3 DJ appl...
MegaSeg is a complete solution for pro audio/video DJ mixing, radio automation, and music scheduling with rock-solid performance and an easy-to-use design. Mix with visual waveforms and Magic... Read more
KeyCue 7.5 - Displays all menu shortcut...
Need more than one license? Get KeyCue Family for just $15.99. KeyCue lets you view available Command key shortcuts. Simply press and hold the Command key to view a list of all available keyboard... Read more
Apple Pro Video Formats 2.0.1 - Updates...
Apple Pro Video Formats brings updates to Apple's professional-level codes for Final Cut Pro X, Motion 5, and Compressor 4. Version 2.0.1: Support for the following professional video codecs Apple... Read more
Maya 2015 - Professional 3D modeling and...
Maya is an award-winning software and powerful, integrated 3D modeling, animation, visual effects, and rendering solution. Because Maya is based on an open architecture, all your work can be scripted... Read more
EtreCheck 2.2 - For troubleshooting your...
EtreCheck is a simple little app to display the important details of your system configuration and allow you to copy that information to the Clipboard. It is meant to be used with Apple Support... Read more
OmniOutliner Pro 4.2 - Pro version of th...
OmniOutliner Pro is a flexible program for creating, collecting, and organizing information. Give your creativity a kick start by using an application that's actually designed to help you think. It's... Read more
VLC Media Player 2.2.1 - Popular multime...
VLC Media Player is a highly portable multimedia player for various audio and video formats (MPEG-1, MPEG-2, MPEG-4, DivX, MP3, OGG, ...) as well as DVDs, VCDs, and various streaming protocols. It... Read more
Nisus Writer Pro 2.1.1 - Multilingual wo...
Nisus Writer Pro is a powerful multilingual word processor, similar to its entry level products, but brings new features such as table of contents, indexing, bookmarks, widow and orphan control,... Read more
Tinderbox 6.2.0 - Store and organize you...
Tinderbox is a personal content management assistant. It stores your notes, ideas, and plans. It can help you organize and understand them. And Tinderbox helps you share ideas through Web journals... Read more

Lifeline... (Games)
Lifeline... 1.1 Device: iOS Universal Category: Games Price: $2.99, Version: 1.1 (iTunes) Description: Lifeline is a playable, branching story of survival against all odds. Using your iPhone, iPad, or Apple Watch, you will help... | Read more »
Pandemic: The Board Game Has Gone Univer...
Don't let the virus win! Now you can download Pandemic: The Board Game, by F2Z Digital Media, for all of your iOS devices. The app is based on the fantastic board game by Z-man games. As employees of the CDC, you and your friends will have to work... | Read more »
Get Ready to Read Bloomberg Business on...
Fans of Bloomberg Business will soon be able to get all their news on the Apple Watch. The app lets you get the top headlines on your main screen and bookmark stories to read later. Using the motion detection in the Apple Watch, the headlines are... | Read more »
Watch This Homerun is Batting for the Ap...
Eyes Wide Games' Watch This Homerun is purportedly the first sports game coming to the Apple Watch, where you'll be up to bat as the pitcher tries to out-manuever you with fastballs, curveballs, and changeups. Using one-touch controls you can try to... | Read more »
Field Trip Can Take You on a Guided Tour...
Field Trip, by Google’s Niantic Labs, is an exploration app that gives you details about the awesome places you can discover wherever you find yourself. The app can show you local history, delicious restraunts, the best places to shop, and places to... | Read more »
Watch Your Six - SPY_WATCH is Infiltrati...
SPY_WATCH, by Bossa Studios, is a new game designed for the Apple Watch. Runmor has it your spy agency has fallen out of favor. To save it, you'll need to train-up a spy and send them on missions to earn you a stunningly suspicious reputation and... | Read more »
Both Halo: Spartan Assault and Halo: Spa...
Halo: Spartan Assault and Halo: Spartan Strike, by Microsoft, have officially landed on the App Store. Spartan Assault pits you against the Covenant with missions geared to tell the story of the origin of Spartan Ops. In Spartan Strike you'll delve... | Read more »
The Apple Watch Could Revolutionize the...
It’s not here yet but there’s that developing sneaky feeling that the Apple Watch, despite its price tag and low battery life, might yet change quite a lot about how we conduct our lives. While I don’t think it’s going to be an overnight... | Read more »
Mad Skills Motocross 2 Version 2.0 is He...
Mad Skills Motocross 2 fans got some good news this week as Turborilla has given the game its biggest update yet. Now you'll have access to Versus mode where you can compete against your friends in timed challenges. Turborilla has implemented a... | Read more »
Kids Can Practice Healthy Living With Gr...
Bobaka is releasing a new interactive book called Green Riding Hood  in May. The app teaches kids about yoga and organic style of life through mini-games and a fun take on the classic Little Red Riding Hood fairy tale. | Read more »

Price Scanner via MacPrices.net

Sale! 15-inch Retina MacBook Pros for up to $...
 MacMall has 15″ Retina MacBook Pros on sale for up to $255 off MSRP. Shipping is free: - 15″ 2.2GHz Retina MacBook Pro: $1794.99 save $205 - 15″ 2.5GHz Retina MacBook Pro: $2244.99 save $255 Adorama... Read more
New 2015 MacBook Airs on sale for up to $75 o...
Save up to $75 on the purchase of a new 2015 13″ or 11″ 1.6GHz MacBook Air at the following resellers. Shipping is free with each model: 11" 128GB MSRP $899 11" 256GB... Read more
Clearance 13-inch Retina MacBook Pros availab...
B&H Photo has leftover 2014 13″ Retina MacBook Pros on sale for up to $250 off original MSRP. Shipping is free, and B&H charges NY sales tax only: - 13″ 2.6GHz/128GB Retina MacBook Pro: $1129... Read more
Clearance 2014 MacBook Airs available startin...
B&H Photo has clearance 2014 MacBook Airs available for up to $200 off original MSRP. Shipping is free, and B&H charges NY sales tax only: - 11″ 128GB MacBook Air: $729 $170 off original MSRP... Read more
16GB iPad mini 3 on sale for $349, save $50
B&H Photo has the 16GB iPad mini 3 WiFi on sale for $349 including free shipping plus NY sales tax only. Their price is $50 off MSRP, and it’s the lowest price available for this model. Read more
Mac minis on sale for up to $75 off, starting...
MacMall has Mac minis on sale for up to $75 off MSRP including free shipping. Their prices are the lowest available for these models from any reseller: - 1.4GHz Mac mini: $459.99 $40 off - 2.6GHz Mac... Read more
Taichi Temple First Tai Chi Motion Sensor App...
Zhen Wu LLC has announced the official launch of Taichi Temple 1.0, the first motion sensor app for Tai Chi, offering a revolutionary new way to de-compress, relax and exercise all at the same time.... Read more
CleanExit – Erase your Hard Drive Quickly, Se...
CleanExit works on both Macs and PCs, securely and permanently deleting all files from any type of hard drive, flash-based drive or camera media card making the files permanently unrecoverable.... Read more
250 iPhone 6 Tips eBook Released for $1.99
Bournemouth, UK based iOS Guides has released 250 iPhone 6 Tips, a new eBook available in the iBookstore that reveals a wealth of tips and tutorials for iPhone 6 and iPhone 6 Plus. Priced at $1.99,... Read more
TigerText Introduces First Secure Enterprise...
TigerText, a provider of secure, real-time messaging for the enterprise, has announced the launch of TigerText for the Apple Watch. TigerText for the Apple Watch enables users to securely send and... Read more

Jobs Board

*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
*Apple* Solutions Consultant - Retail Sales...
**Job Summary** As an Apple Solutions Consultant (ASC) you are the link between our customers and our products. Your role is to drive the Apple business in a retail Read more
DevOps Software Engineer - *Apple* Pay, iOS...
**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring Read more
*Apple* Pay - Site Reliability Engineer - Ap...
**Job Summary** Imagine what you could do here. At Apple , great ideas have a way of becoming great products, services, and customer experiences very quickly. Bring Read more
Sr. Technical Services Consultant, *Apple*...
**Job Summary** Apple Professional Services (APS) has an opening for a senior technical position that contributes to Apple 's efforts for strategic and transactional Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.