TweetFollow Us on Twitter

Web Benchmarking 101

Volume Number: 22 (2006)
Issue Number: 4
Column Tag: Web Performance Testing

Web Benchmarking 101

A guide to stress testing your website with ApacheBench and JMeter

by Jin Lin and Emmanuel Stein

Introduction

Web performance testing can be a daunting subject for the uninitiated. With the sheer number of tools available and their various feature sets, it can be hard to know where to begin. In this article we will explore the use of two popular open source tools, ApacheBench (aka ab) and JMeter, to perform simple benchmarking tests. Rather than going through the extensive feature set offered by these applications, particularly JMeter, we will focus on measuring the response time and the effective throughput of your web application across different user load scenarios. Whether serving a few static pages or a fully dynamic site, the tools discussed will offer invaluable insight into the overall performance of your web applications, and serve as a basis for isolating system bottlenecks and conducting capacity planning for your web applications.

Benchmarking Considerations

Although, the tests and tools that we will use in this article represent integral components used in the testing process, it is equally important to consider the environment in which such tests are performed. If you are running a popular website with significant traffic, you may consider performing benchmarks during non-peak hours to obtain more accurate results and avoid rendering your server unusable to users. If possible, you should test your web server in a staged environment and before actual deployment. This will enable you to adjust server parameters, install cache mechanisms, and otherwise assess your site's performance without interrupting service.

ApacheBench

ApacheBench, a default component of the Apache Server distribution, is a simple command line tool, designed to measure the performance of Apache server. Preinstalled on Mac OS X, it is able to supply information such as elapsed time, requests per second, time per request, compression rate, transfer rate, and connection time under different concurrence scenarios.

Though not an exhaustive list (see the ab man page for more details), the following represents the most commonly used switches employed when testing typical usage scenarios with ab.

  • -n: Specifies the number of requests to perform for a given test. Use this option to measure the time required to process a given number of client requests. When used alone, without the -c option, ab does not process requests concurrently.

  • -t: Specifies a timeframe, in seconds, to spend benchmarking. This option is useful in determining the number of requests that your server is able to process in a given amount of time.

  • -c: Specifies number of simultaneous connections. Of use in testing how your server performs when multiple clients are hitting a given page at the same time. This switch may be used with either -n or -t, to measure the total elapsed time required to process a given number of client requests simultaneously, and the number of concurrent requests that may be processed over a set amount of time, respectively.

  • -g: Exports a TSV (Tab Separated Values) file, which can be imported into applications like Excel, GNUplot, and Mathematica for further analysis and/or graphic visualization.

  • -A: Used to supply username and password information for sites that require authentication. This enables you to conduct testing on password-protected sites.

Having covered some of the basic parameters of ab, we will now explore its use in the context of simple web benchmarking scenarios.

For our first example, we will be measuring the time it takes to process 1000 requests (i.e. -n 1000), from 10 simultaneous user connections (i.e. -c 10) against OS X's default user website. Make sure Apache is running by launching System Preferences and enabling personal web sharing from the sharing preference pane. Once Apache is running, launch the Terminal application and enter the following command: ab -n 1000 -c 10 http://localhost/~username to begin the benchmark. When the command is executed the terminal will display the progress of the benchmark run, as well as, a series of metrics (Figure 1).


Figure 1. Output of ab -n 1000 -c 10 http://localhost/~estein

Be aware that when you run the ab command against a domain or IP address the benchmark defaults to the site's home page; in this case, index.html. To specify a particular page, simply point to its full path (e.g. http://localhost/~estein/MyDynamicContent.php). This will allow you to test the performance of specific pages hosted by your site in terms of the amount of stress they each place on the server. This is particularly important when dealing with sites that have both static and dynamic content. When testing dynamic pages keep in mind that ab reports a failed request when it encounters content that changes over time (e.g. dynamic data). As such, when running ab against dynamic portions of your site, you can safely ignore the "failed requests" messages.

For our next example, we will be measuring the number of requests that can be completed within 60 seconds (-t 60) with 10 concurrent connections (-c 10). Enter the following command in the Terminal to execute this benchmark: ab -t 60 -c 10 http://localhost/~username The output of this command, seen in Figure 2, differs from the first example in that -n is not specified. When -n is not specified, ab sets the number of requests to a default value of 50,000. It is important to realize that, when using the -t switch, it is possible for the benchmark to complete, before the specified time period has elapsed, thus rendering the time variable irrelevant. Therefore, when using the -t and -n switches in tandem, it is important to consider whether your time variable exceeds the time it takes to process n requests.


Figure 2. Output of ab -t 60 -c 10 http://localhost/~estein

JMeter

JMeter is a pure Java application, developed under the purview of the Apache Jakarta project, which offers a wide array of tools for benchmarking all manner of client-serverbased applications. JMeter's modular architecture allows it to simulate a variety of load scenarios across multiple servers, networks, and objects. For the purposes of this article, however, we will only be using a subset of JMeter's capabilities to perform basic load testing. For information on using JMeter to perform detailed performance measures on more complex web applications you may refer to the user guide at http://jakarta.apache.org/jmeter/usermanual/index.html

Before we begin, it is instructive to go over the major architectural components that constitute JMeter.

  • Test Plan: Defines the test sequence that JMeter will carry out during benchmarking.

  • Thread Group: Used to delineate specific testing conditions, such as number of concurrent users, the Thread Group plays host to all other elements that comprise a test sequence.

  • Config Element: Responsible for defining configuration information for the Sampler and Logic Controller elements.

  • Sampler: A type of controller that designates both the protocol type and nature of requests within a given test plan.

  • Logic Controller: A controller element that determines the sequence logic used by JMeter to initiate protocol requests.

  • Timer: Used to manage the latency across queued user requests.

  • Listener: Responsible for recording and visualizing of test sequence data.

Getting Started

JMeter requires JDK 1.4 or higher to run properly. Type java -version in the Terminal application to discover which version of JDK is installed on your machine. Currently Apple supplies Java 1.3.1-1.4.2 Release 2 and J2SE 5.0 release 3 through regular software updates, as well as, at the following links:

http://www.apple.com/downloads/macosx/apple/java131and142release2.html

http://www.apple.com/downloads/macosx/apple/java2se50release3.html

In our tests, both JDK versions were able to run JMeter without a hitch. However, you will likely notice some performance improvement when using the J2SE 5.0 release 3. To specify this version as your system default or to change the precedence settings for your installed Java Virtual Machines (JVMs), use the Java Preferences utility located in /Applications/Utilities/Java/Java2SE 5.0.

The latest version of JMeter is available as a free download at http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi Depending on your preferences, you can download the application as source code or binary as either .zip or .tgz archives. The easiest option for most OS X users will be to download the binary in zip format, which can be easily uncompressed in the GUI by double clicking the jakarta-jmeter-2.1.x.zip archive. The resulting jakarta-jmeter-2.1.1 folder can be placed in your Applications folder or other location within the filesystem. You can launch JMeter by either double clicking the ApacheJMeter.jar file, located in the bin directory of the Jakarta-jmeter-2.1.1 folder, or by executing the jmeter shell script located in the same directory. Please note, that if you wish to run jmeter from the command line you will need to make the script executable using the chmod command. For command line interactive use, please refer to the "Getting Started" section of the JMeter user manual: http://jakarta.apache.org/jmeter/usermanual/get-started.html


Figure 3. JMeter's main window

JMeter in Action

In this example, we will create a test plan that simulates 10 users initiating simultaneous requests to 2 different URLs and which repeats the test 100 times. Begin with the addition of a new Thread Group by right clicking (or control clicking) the Test Plan element in the left pane and selecting Add >Thread Group. Next, select the Thread Group element to display its configuration window and set the Number of Threads to 10 (representing users), the Ramp-Up Period to 0 (to initiate all users at once), and the Loop Count to 100.


Figure 4. Configuring Thread Group element

To add the next element of our test plan, HTTP Request Defaults, right click on the Thread Group element, and select Add > Config Element > HTTP Request Defaults (Figure 5).


Figure 5. Add HTTP Request Defaults

Select the HTTP Request Defaults icon under Thread Group and enter a server name (e.g. localhost or FQDN) in the configuration window (Figure 6).


Figure 6. Configuring HTTP Request Defaults element

Right click on the Thread Group element and then select Add>Sampler>HTTP Request twice, to create two HTTP Request elements, one for each web page. The Name and Path fields for each of the HTTP Request elements should be set to appropriate values. For the purpose of this example, we will name the first HTTP Request element home, and set its path to ~username/index.html and name the second HTTP Request another and set its path to ~username/another.html (figure 7).


Figure 7. Configuring the "another" HTTP Request element

Now we need to add a timer to pause each request for a random period of time. Right click on the Thread Group and then select Add>Timer>Gaussian Random Time.

For the last element, we will add listeners by right clicking on the Thread Group element, then selecting Add>Listener>Aggregate Report. This particular listener lists, in tabular form, the response time, request count, min, max, average, error rate, and approximate number of request processes in seconds (e.g. Throughput), as well as, throughput in terms of Kilobytes per second. You may add additional listeners, such as Graph Results, View Results in Table, and View Results Tree, depending on your requirements.

To begin the test plan, simply select Run>Start from within the JMeter menu bar. You will be prompted to save the test plan, if you haven't already done so. During a successful run of your test plan, you may click on the Aggregate Report listener element (figure 8), or any other listener types you may have added, to view your dynamically updated test results. Figure 9, for example, demonstrates some of the graphing functions of JMeter's Graph Results listener element.


Figure 8. Aggregate Report listener view


Figure 9. Graph Results listener view

Conclusion

In this article, we employed both ab and JMeter to "stress test" a locally hosted site via simple load scenarios. The amount of time a server takes to respond to clients' requests across different load levels, is one of the most important metrics for evaluating web performance. Through the manipulation of load test variables, such as the number of concurrent requests, and with reference to tool-specific data visualization processes, we are better able to ascertain the extent to which these variables impact a server's maximum sustainable load. Such information is critical for proper website management, planning and eventual performance tuning.


Jin Lin and Emmanuel Stein are partners in the consulting firm MacVerse, Corp, which offers implementation, system administration, and development services geared towards the enterprise market. You may reach them at info@macverse.com

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

SyncTwoFolders 2.0.9b1 - Syncs two user-...
SyncTwoFolders simply synchronizes two folders. It supports synchronization across mounted network drives and it is a possibility to run a simulation showing in a log what will be done. Please visit... Read more
VueScan 9.5.09 - Scanner software with a...
VueScan is a scanning program that works with most high-quality flatbed and film scanners to produce scans that have excellent color fidelity and color balance. VueScan is easy to use, and has... Read more
Airmail 2.1 - Powerful, minimal email cl...
Airmail 2 is a new mail client designed for OS X 10.10 Yosemite with fast performance and intuitive interaction. Support for iCloud, MS Exchange, Gmail, Google Apps, IMAP, POP3, Yahoo!, AOL, Outlook.... Read more
Logitech Control Center 3.9.2 - Keyboard...
Logitech Control Center (LCC) is designed to support OS X and allows you to take full advantage of your Logitech keyboard, mouse, or trackball. With the LCC you can: Browse the Internet using... Read more
Adobe Acrobat Pro 15.007.20033 - Powerfu...
Acrobat Pro DC is available only as a part of Adobe Creative Cloud, and can only be installed and/or updated through Adobe's Creative Cloud app. Adobe Acrobat Pro DC with Adobe Document Cloud... Read more
CleanMyMac 3.0.1 - Delete files that was...
CleanMyMac makes space for the things you love. Sporting a range of ingenious new features, CleanMyMac lets you safely and intelligently scan and clean your entire system, delete large, unused files... Read more
Evernote 6.0.10 - Create searchable note...
Evernote allows you to easily capture information in any environment using whatever device or platform you find most convenient, and makes this information accessible and searchable at anytime, from... Read more
CleanApp 5.0.1 - Application deinstaller...
CleanApp is an application deinstaller and archiver.... Your hard drive gets fuller day by day, but do you know why? CleanApp 5 provides you with insights how to reclaim disk space. There are... Read more
Quicken 2015 2.5.0 - Complete personal f...
Quicken 2015 helps you manage all your personal finances in one place, so you can see where you're spending and where you can save. Quicken automatically categorizes your financial transactions,... Read more
Tonality Pro 1.1.4 - Professional-grade...
Tonality Pro gives you the power to create stunning and dramatic black & white images. This is a complete monochrome image editor with more than 150 one-click style presets, totally unique... Read more

Grim Fandango Remastered (Games)
Grim Fandango Remastered 1.0.0 Device: iOS iPhone Category: Games Price: $9.99, Version: 1.0.0 (iTunes) Description: | Read more »
This Week at 148Apps: April 27-May 1, 20...
May Merriment at 148Apps How do you know what apps are worth your time and money? Just look to the review team at 148Apps. We sort through the chaos and find the apps you're looking for. The ones we love become Editor’s Choice, standing out above... | Read more »
The Ensign, Prequel to A Dark Room, is C...
Amirali Rajan's The Ensign is the prequel to the text-based adventure A Dark Room, and it's gone free for a couple of days. [Read more] | Read more »
Streamline - calendar (Productivity)
Streamline - calendar 1.0 Device: iOS iPhone Category: Productivity Price: $3.99, Version: 1.0 (iTunes) Description: Do you dare to break conventions? Streamline is the first real evolution from the paper diary. It is a patented and... | Read more »
Plants vs. Zombies Celebrates its Sixth...
It's pretty wild to think that the  Plants vs. Zombies  series has been around for six years already. To celebrate this mildly unconventional milestone, Popcap is giving players some free stuff in Plants vs. Zombies 2. Because they got the whole "... | Read more »
Dropbox Adds MS Office Documents and Mor...
Dropbox, the popular file storage solution, has announced an update that adds three new features for you to enjoy. [Read more] | Read more »
The All-Consuming Snake is Back and Hung...
Taneli Armanto, the designer of Nokia 6110 Snake, is back to work. Together with Rumilus Design, he's working on a new Snake game titled Snake Rewind. [Read more] | Read more »
WAVE Personal Calendar Now Connects You...
WAVE Calendar, by e-sites, has added an interesting new feature: real life personal assistants. The e-sites webpage has been redesigned so personal assistants can apply to work directly with WAVE. [Read more] | Read more »
Find Your Vacation Home With Dwellable
Dwellable is a new travel app that helps you to find rental homes for your vacation, allowing you to swipe your way through listings from over 15,000 destinations worldwide. [Read more] | Read more »
Play Endless Golf in Flick Golf Free...
Full Fat has released a new update for Flick Golf! Free. The ‘Eliminator’ update brings a whole new way to play with its endless round. [Read more] | Read more »

Price Scanner via MacPrices.net

Millennials’ Social Networking Habits Pose T...
Research indicates young people may be threat to corporate information-security, says tech expert Karl Volkmann, Chief Technical Officer of Chicago-based SRV Network, Inc. From Facebook to Twitter,... Read more
BuyVia Releases Apple Watch Shopping App
San Mateo, California-based BuyVia (http://www.buyvia.com), the first website and mobile app that combines in-store and online shopping across smart phones, tablets and laptops to help shoppers get... Read more
Apple Certified Refurbished iMacs available f...
The Apple Store has Apple Certified Refurbished iMacs available for up to $380 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 27″ 3.5GHz 5K iMac – $2119 $... Read more
13-inch 2.5GHz MacBook Pro (refurbished) avai...
The Apple Store has Apple Certified Refurbished 13″ 2.5GHz MacBook Pros available for $829, or $270 off the cost of new models. Apple’s one-year warranty is standard, and shipping is free: - 13″ 2.... Read more
New 2015 MacBook Airs on sale for up to $80 o...
Save up to $80 on the purchase of a new 2015 13″ or 11″ 1.6GHz MacBook Air at the following resellers. Shipping is free with each model: 11" 128GB MSRP $899 11" 256GB... Read more
College Student Deals: Additional $50 off Mac...
Take an additional $50 off all MacBooks and iMacs at Best Buy Online with their College Students Deals Savings, valid through May 9, 2015. Anyone with a valid .EDU email address can take advantage of... Read more
Save up to $300 on a new Mac, $30 on an iPad,...
Purchase a new Mac or iPad at The Apple Store for Education and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free,... Read more
iPad-Based iLynn Program Provides A Private U...
Lynn University of Boca Raton, Florida has announced that its innovative iPad-powered online bachelor’s degree program will be called iLynn. Beginning in fall 2015, Lynn U. says the new program will... Read more
iPad mini 4 Rumors Begin To Gel
First, I’ve been really hoping there will even be an iPad mini 4 at all. The smallest iPad has reportedly been hit pretty hard by iPhone 6 Plus cannibalization, along with getting caught in the... Read more
Mac Pros on sale for up to $260 off MSRP
B&H Photo has Mac Pros on sale for up to $260 off MSRP. Shipping is free, and B&H charges sales tax in NY only: - 3.7GHz 4-core Mac Pro: $2799, $200 off MSRP - 3.5GHz 6-core Mac Pro: $3719.99... Read more

Jobs Board

*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
*Apple* Retail - Multiple Positions (US) - A...
Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, you're also the Read more
Senior Identity Architect - *Apple* Pay - A...
Changing the world is all in a day039s work at Apple . If you love innovation, here039s your chance to make a career of it. You039ll work hard. But the job comes with Read more
Hardware Systems Integration Engineer - *App...
Changing the world is all in a day039s work at Apple . If you love innovation, here039s your chance to make a career of it. You039ll work hard. But the job comes with Read more
Senior Identity Architect - *Apple* Pay - A...
Changing the world is all in a day039s work at Apple . If you love innovation, here039s your chance to make a career of it. You039ll work hard. But the job comes with Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.