TweetFollow Us on Twitter

Web Benchmarking 101

Volume Number: 22 (2006)
Issue Number: 4
Column Tag: Web Performance Testing

Web Benchmarking 101

A guide to stress testing your website with ApacheBench and JMeter

by Jin Lin and Emmanuel Stein

Introduction

Web performance testing can be a daunting subject for the uninitiated. With the sheer number of tools available and their various feature sets, it can be hard to know where to begin. In this article we will explore the use of two popular open source tools, ApacheBench (aka ab) and JMeter, to perform simple benchmarking tests. Rather than going through the extensive feature set offered by these applications, particularly JMeter, we will focus on measuring the response time and the effective throughput of your web application across different user load scenarios. Whether serving a few static pages or a fully dynamic site, the tools discussed will offer invaluable insight into the overall performance of your web applications, and serve as a basis for isolating system bottlenecks and conducting capacity planning for your web applications.

Benchmarking Considerations

Although, the tests and tools that we will use in this article represent integral components used in the testing process, it is equally important to consider the environment in which such tests are performed. If you are running a popular website with significant traffic, you may consider performing benchmarks during non-peak hours to obtain more accurate results and avoid rendering your server unusable to users. If possible, you should test your web server in a staged environment and before actual deployment. This will enable you to adjust server parameters, install cache mechanisms, and otherwise assess your site's performance without interrupting service.

ApacheBench

ApacheBench, a default component of the Apache Server distribution, is a simple command line tool, designed to measure the performance of Apache server. Preinstalled on Mac OS X, it is able to supply information such as elapsed time, requests per second, time per request, compression rate, transfer rate, and connection time under different concurrence scenarios.

Though not an exhaustive list (see the ab man page for more details), the following represents the most commonly used switches employed when testing typical usage scenarios with ab.

  • -n: Specifies the number of requests to perform for a given test. Use this option to measure the time required to process a given number of client requests. When used alone, without the -c option, ab does not process requests concurrently.

  • -t: Specifies a timeframe, in seconds, to spend benchmarking. This option is useful in determining the number of requests that your server is able to process in a given amount of time.

  • -c: Specifies number of simultaneous connections. Of use in testing how your server performs when multiple clients are hitting a given page at the same time. This switch may be used with either -n or -t, to measure the total elapsed time required to process a given number of client requests simultaneously, and the number of concurrent requests that may be processed over a set amount of time, respectively.

  • -g: Exports a TSV (Tab Separated Values) file, which can be imported into applications like Excel, GNUplot, and Mathematica for further analysis and/or graphic visualization.

  • -A: Used to supply username and password information for sites that require authentication. This enables you to conduct testing on password-protected sites.

Having covered some of the basic parameters of ab, we will now explore its use in the context of simple web benchmarking scenarios.

For our first example, we will be measuring the time it takes to process 1000 requests (i.e. -n 1000), from 10 simultaneous user connections (i.e. -c 10) against OS X's default user website. Make sure Apache is running by launching System Preferences and enabling personal web sharing from the sharing preference pane. Once Apache is running, launch the Terminal application and enter the following command: ab -n 1000 -c 10 http://localhost/~username to begin the benchmark. When the command is executed the terminal will display the progress of the benchmark run, as well as, a series of metrics (Figure 1).


Figure 1. Output of ab -n 1000 -c 10 http://localhost/~estein

Be aware that when you run the ab command against a domain or IP address the benchmark defaults to the site's home page; in this case, index.html. To specify a particular page, simply point to its full path (e.g. http://localhost/~estein/MyDynamicContent.php). This will allow you to test the performance of specific pages hosted by your site in terms of the amount of stress they each place on the server. This is particularly important when dealing with sites that have both static and dynamic content. When testing dynamic pages keep in mind that ab reports a failed request when it encounters content that changes over time (e.g. dynamic data). As such, when running ab against dynamic portions of your site, you can safely ignore the "failed requests" messages.

For our next example, we will be measuring the number of requests that can be completed within 60 seconds (-t 60) with 10 concurrent connections (-c 10). Enter the following command in the Terminal to execute this benchmark: ab -t 60 -c 10 http://localhost/~username The output of this command, seen in Figure 2, differs from the first example in that -n is not specified. When -n is not specified, ab sets the number of requests to a default value of 50,000. It is important to realize that, when using the -t switch, it is possible for the benchmark to complete, before the specified time period has elapsed, thus rendering the time variable irrelevant. Therefore, when using the -t and -n switches in tandem, it is important to consider whether your time variable exceeds the time it takes to process n requests.


Figure 2. Output of ab -t 60 -c 10 http://localhost/~estein

JMeter

JMeter is a pure Java application, developed under the purview of the Apache Jakarta project, which offers a wide array of tools for benchmarking all manner of client-serverbased applications. JMeter's modular architecture allows it to simulate a variety of load scenarios across multiple servers, networks, and objects. For the purposes of this article, however, we will only be using a subset of JMeter's capabilities to perform basic load testing. For information on using JMeter to perform detailed performance measures on more complex web applications you may refer to the user guide at http://jakarta.apache.org/jmeter/usermanual/index.html

Before we begin, it is instructive to go over the major architectural components that constitute JMeter.

  • Test Plan: Defines the test sequence that JMeter will carry out during benchmarking.

  • Thread Group: Used to delineate specific testing conditions, such as number of concurrent users, the Thread Group plays host to all other elements that comprise a test sequence.

  • Config Element: Responsible for defining configuration information for the Sampler and Logic Controller elements.

  • Sampler: A type of controller that designates both the protocol type and nature of requests within a given test plan.

  • Logic Controller: A controller element that determines the sequence logic used by JMeter to initiate protocol requests.

  • Timer: Used to manage the latency across queued user requests.

  • Listener: Responsible for recording and visualizing of test sequence data.

Getting Started

JMeter requires JDK 1.4 or higher to run properly. Type java -version in the Terminal application to discover which version of JDK is installed on your machine. Currently Apple supplies Java 1.3.1-1.4.2 Release 2 and J2SE 5.0 release 3 through regular software updates, as well as, at the following links:

http://www.apple.com/downloads/macosx/apple/java131and142release2.html

http://www.apple.com/downloads/macosx/apple/java2se50release3.html

In our tests, both JDK versions were able to run JMeter without a hitch. However, you will likely notice some performance improvement when using the J2SE 5.0 release 3. To specify this version as your system default or to change the precedence settings for your installed Java Virtual Machines (JVMs), use the Java Preferences utility located in /Applications/Utilities/Java/Java2SE 5.0.

The latest version of JMeter is available as a free download at http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi Depending on your preferences, you can download the application as source code or binary as either .zip or .tgz archives. The easiest option for most OS X users will be to download the binary in zip format, which can be easily uncompressed in the GUI by double clicking the jakarta-jmeter-2.1.x.zip archive. The resulting jakarta-jmeter-2.1.1 folder can be placed in your Applications folder or other location within the filesystem. You can launch JMeter by either double clicking the ApacheJMeter.jar file, located in the bin directory of the Jakarta-jmeter-2.1.1 folder, or by executing the jmeter shell script located in the same directory. Please note, that if you wish to run jmeter from the command line you will need to make the script executable using the chmod command. For command line interactive use, please refer to the "Getting Started" section of the JMeter user manual: http://jakarta.apache.org/jmeter/usermanual/get-started.html


Figure 3. JMeter's main window

JMeter in Action

In this example, we will create a test plan that simulates 10 users initiating simultaneous requests to 2 different URLs and which repeats the test 100 times. Begin with the addition of a new Thread Group by right clicking (or control clicking) the Test Plan element in the left pane and selecting Add >Thread Group. Next, select the Thread Group element to display its configuration window and set the Number of Threads to 10 (representing users), the Ramp-Up Period to 0 (to initiate all users at once), and the Loop Count to 100.


Figure 4. Configuring Thread Group element

To add the next element of our test plan, HTTP Request Defaults, right click on the Thread Group element, and select Add > Config Element > HTTP Request Defaults (Figure 5).


Figure 5. Add HTTP Request Defaults

Select the HTTP Request Defaults icon under Thread Group and enter a server name (e.g. localhost or FQDN) in the configuration window (Figure 6).


Figure 6. Configuring HTTP Request Defaults element

Right click on the Thread Group element and then select Add>Sampler>HTTP Request twice, to create two HTTP Request elements, one for each web page. The Name and Path fields for each of the HTTP Request elements should be set to appropriate values. For the purpose of this example, we will name the first HTTP Request element home, and set its path to ~username/index.html and name the second HTTP Request another and set its path to ~username/another.html (figure 7).


Figure 7. Configuring the "another" HTTP Request element

Now we need to add a timer to pause each request for a random period of time. Right click on the Thread Group and then select Add>Timer>Gaussian Random Time.

For the last element, we will add listeners by right clicking on the Thread Group element, then selecting Add>Listener>Aggregate Report. This particular listener lists, in tabular form, the response time, request count, min, max, average, error rate, and approximate number of request processes in seconds (e.g. Throughput), as well as, throughput in terms of Kilobytes per second. You may add additional listeners, such as Graph Results, View Results in Table, and View Results Tree, depending on your requirements.

To begin the test plan, simply select Run>Start from within the JMeter menu bar. You will be prompted to save the test plan, if you haven't already done so. During a successful run of your test plan, you may click on the Aggregate Report listener element (figure 8), or any other listener types you may have added, to view your dynamically updated test results. Figure 9, for example, demonstrates some of the graphing functions of JMeter's Graph Results listener element.


Figure 8. Aggregate Report listener view


Figure 9. Graph Results listener view

Conclusion

In this article, we employed both ab and JMeter to "stress test" a locally hosted site via simple load scenarios. The amount of time a server takes to respond to clients' requests across different load levels, is one of the most important metrics for evaluating web performance. Through the manipulation of load test variables, such as the number of concurrent requests, and with reference to tool-specific data visualization processes, we are better able to ascertain the extent to which these variables impact a server's maximum sustainable load. Such information is critical for proper website management, planning and eventual performance tuning.


Jin Lin and Emmanuel Stein are partners in the consulting firm MacVerse, Corp, which offers implementation, system administration, and development services geared towards the enterprise market. You may reach them at info@macverse.com

 
AAPL
$104.48
Apple Inc.
+1.49
MSFT
$44.81
Microsoft Corpora
+0.43
GOOG
$542.21
Google Inc.
+9.50

MacTech Search:
Community Search:

Software Updates via MacUpdate

jAlbum Pro 12.2.4 - Organize your digita...
jAlbum Pro has all the features you love in jAlbum, but comes with a commercial license. With jAlbum, you can create gorgeous custom photo galleries for the Web without writing a line of code!... Read more
jAlbum 12.2.4 - Create custom photo gall...
With jAlbum, you can create gorgeous custom photo galleries for the Web without writing a line of code! Beginner-friendly, with pro results Simply drag and drop photos into groups, choose a design... Read more
ExpanDrive 4.1.7 - Access remote files o...
ExpanDrive builds cloud storage in every application, acts just like a USB drive plugged into your Mac. With ExpanDrive, you can securely access any remote file server directly from the Finder or... Read more
OmniOutliner Pro 4.1.3 - Pro version of...
OmniOutliner Pro is a flexible program for creating, collecting, and organizing information. Give your creativity a kick start by using an application that's actually designed to help you think. It'... Read more
Evernote 5.6.2 - Create searchable notes...
Evernote allows you to easily capture information in any environment using whatever device or platform you find most convenient, and makes this information accessible and searchable at anytime, from... Read more
OmniOutliner 4.1.3 - Organize your ideas...
OmniOutliner is a flexible program for creating, collecting, and organizing information. Give your creativity a kick start by using an application that's actually designed to help you think. It's... Read more
BBEdit 11.0 - Powerful text and HTML edi...
BBEdit is the leading professional HTML and text editor for the Mac. Specifically crafted in response to the needs of Web authors and software developers, this award-winning product provides a... Read more
Apple Security Update 2014-005 - For OS...
Apple Security Update is recommended for all users and improves the security of Mac OS X. For information on the security content of this update, please visit this website: http://support.apple.com/... Read more
EyeTV 3.6.6 - Watch and record TV on you...
EyeTV brings a rich TV experience to your Mac. Watch live TV on your Mac. Pause, rewind, and record whenever you want. EyeTV gives you powerful control over what you watch and how you watch it. Put... Read more
RapidWeaver 6.0 - Create template-based...
RapidWeaver is a next-generation Web design application to help you easily create professional-looking Web sites in minutes. No knowledge of complex code is required, RapidWeaver will take care of... Read more

Latest Forum Discussions

See All

Jam Messenger Review
Jam Messenger Review By Jennifer Allen on October 23rd, 2014 Our Rating: :: SIMPLE MESSAGINGiPhone App - Designed for the iPhone, compatible with the iPad Want a very quick way to send push-based messages? Jam Messenger is basic... | Read more »
Felllice (Games)
Felllice 1.0 Device: iOS Universal Category: Games Price: $.99, Version: 1.0 (iTunes) Description: EXCLUSIVE PRICE DROP! 50% OFF FOR A LIMITED TIME! EAT EAT EAT AND GROW ! | Read more »
The Arrow Game: by Grazie Media (Games)
The Arrow Game: by Grazie Media 1.0 Device: iOS Universal Category: Games Price: $.99, Version: 1.0 (iTunes) Description: Guide a flying arrow through skyscrapers and city streets to hit a distant target. Experience an endless... | Read more »
Worldly (Games)
Worldly 1.2 Device: iOS Universal Category: Games Price: $2.99, Version: 1.2 (iTunes) Description: | Read more »
Money Pro - EASY! Bills, Budgets and Acc...
Money Pro - EASY! Bills, Budgets and Accounts w/ Sync 1.0 Device: iOS Universal Category: Finance Price: $4.99, Version: 1.0 (iTunes) Description: Manage money like a pro. Money Pro is the next generation of Money app (over 2 million... | Read more »
Pro Strategy Football 2014 (Games)
Pro Strategy Football 2014 2014.141001 Device: iOS Universal Category: Games Price: $4.99, Version: 2014.141001 (iTunes) Description: Take the proven strategy of the PSF franchise and add in Casual Play, improved graphics and... | Read more »
Super Glyph Quest (Games)
Super Glyph Quest 1.01 Device: iOS Universal Category: Games Price: $2.99, Version: 1.01 (iTunes) Description: Adventure is back Questers! Combine elemental glyphs together to cast powerful spells and vanquish adorable monsters in... | Read more »
Fighting Fantasy: Caverns of the Snow Wi...
Fighting Fantasy: Caverns of the Snow Witch 1.0 Device: iOS Universal Category: Games Price: $5.99, Version: 1.0 (iTunes) Description: Travel to Northern Allansia’s perilous Icefinger Mountains to defeat the wicked Snow Witch in this... | Read more »
Star Warfare 2: Payback Review
Star Warfare 2: Payback Review By Blake Grundman on October 22nd, 2014 Our Rating: :: ONE-TRICK PONYUniversal App - Designed for iPhone and iPad Unfortunately, it doesn’t take long for Star Warfare 2’s free-firing fun to turn into... | Read more »
TinType by Hipstamatic (Photography)
TinType by Hipstamatic 1.0 Device: iOS iPhone Category: Photography Price: $.99, Version: 1.0 (iTunes) Description: Create hauntingly beautiful, soul capturing portraits with TinType by Hipstamatic. Inspired by daguerreotypes,... | Read more »

Price Scanner via MacPrices.net

WD My Passport Pro Bus-Powered Thunderbolt RA...
WD’s My Passport Pro RAID solution is powered by an integrated Thunderbolt cable for true portability and speeds as high as 233 MB/s. HighlightsOverviewSpecifications Transfer, Back Up And Edit In... Read more
Save with Best Buy’s College Student Deals
Take an additional $50 off all MacBooks and iMacs at Best Buy Online with their College Students Deals Savings, valid through November 1st. Anyone with a valid .EDU email address can take advantage... Read more
iPad Air 2 & iPad mini 3 Best Tablets Yet...
The new iPads turned out to be pretty much everything I’d been hoping for and more than I’d expected.”More” particularly in terms of a drinking-from-a-firehose choice of models and configurations,... Read more
Drafts 4 Reinvents iOS Productivity App
N Richland Hills, Texas based Agile Tortoise has announced the release of Drafts 4 for iPhone and iPad. Drafts is a quick capture note taking app with flexible output actions. Drafts 4 scales from... Read more
AT&T accepting preorders for new iPads fo...
AT&T Wireless is accepting preorders for the new iPad Air 2 and iPad mini 3, cellular models, for $100 off MSRP with a 2-year service agreement: - 16GB iPad Air 2 WiFi + Cellular: $529.99 - 64GB... Read more
Apple offering refurbished Mac Pros for up to...
The Apple Store is offering Apple Certified Refurbished 2013 Mac Pros for up to $600 off the cost of new models. An Apple one-year warranty is included with each Mac Pro, and shipping is free. The... Read more
Select MacBook Airs $100 off MSRP, free shipp...
B&H Photo has 2014 a couple of MacBook Airs on sale for $100 off MSRP. Shipping is free, and B&H charges NY sales tax only. They also include free copies of Parallels Desktop and LoJack for... Read more
13-inch 2.5GHz MacBook Pro on sale for $100 o...
B&H Photo has the 13″ 2.5GHz MacBook Pro on sale for $999.99 including free shipping plus NY sales tax only. Their price is $100 off MSRP. Read more
Strong iPhone, Mac And App Store Sales Drive...
Apple on Monday announced financial results for its fiscal 2014 fourth quarter ended September 27, 2014. The Company posted quarterly revenue of $42.1 billion and quarterly net profit of $8.5 billion... Read more
Apple Posts How-To For OS X Recovery
OS X 10.7 Lion and later include OS X Recovery. This feature includes all of the tools you need to reinstall OS X, repair your disk, and even restore from a Time Machine backup. OS X Recovery... Read more

Jobs Board

Senior Event Manager, *Apple* Retail Market...
…This senior level position is responsible for leading and imagining the Apple Retail Team's global event strategy. Delivering an overarching brand story; in-store, Read more
*Apple* Solutions Consultant (ASC) - Apple (...
**Job Summary** The ASC is an Apple employee who serves as an Apple brand ambassador and influencer in a Reseller's store. The ASC's role is to grow Apple Read more
Project Manager / Business Analyst, WW *Appl...
…a senior project manager / business analyst to work within our Worldwide Apple Fulfillment Operations and the Business Process Re-engineering team. This role will work Read more
*Apple* Retail - Multiple Positions (US) - A...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Position Opening at *Apple* - Apple (United...
…customers purchase our products, you're the one who helps them get more out of their new Apple technology. Your day in the Apple Store is filled with a range of Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.