TweetFollow Us on Twitter

Web Benchmarking 101

Volume Number: 22 (2006)
Issue Number: 4
Column Tag: Web Performance Testing

Web Benchmarking 101

A guide to stress testing your website with ApacheBench and JMeter

by Jin Lin and Emmanuel Stein

Introduction

Web performance testing can be a daunting subject for the uninitiated. With the sheer number of tools available and their various feature sets, it can be hard to know where to begin. In this article we will explore the use of two popular open source tools, ApacheBench (aka ab) and JMeter, to perform simple benchmarking tests. Rather than going through the extensive feature set offered by these applications, particularly JMeter, we will focus on measuring the response time and the effective throughput of your web application across different user load scenarios. Whether serving a few static pages or a fully dynamic site, the tools discussed will offer invaluable insight into the overall performance of your web applications, and serve as a basis for isolating system bottlenecks and conducting capacity planning for your web applications.

Benchmarking Considerations

Although, the tests and tools that we will use in this article represent integral components used in the testing process, it is equally important to consider the environment in which such tests are performed. If you are running a popular website with significant traffic, you may consider performing benchmarks during non-peak hours to obtain more accurate results and avoid rendering your server unusable to users. If possible, you should test your web server in a staged environment and before actual deployment. This will enable you to adjust server parameters, install cache mechanisms, and otherwise assess your site's performance without interrupting service.

ApacheBench

ApacheBench, a default component of the Apache Server distribution, is a simple command line tool, designed to measure the performance of Apache server. Preinstalled on Mac OS X, it is able to supply information such as elapsed time, requests per second, time per request, compression rate, transfer rate, and connection time under different concurrence scenarios.

Though not an exhaustive list (see the ab man page for more details), the following represents the most commonly used switches employed when testing typical usage scenarios with ab.

  • -n: Specifies the number of requests to perform for a given test. Use this option to measure the time required to process a given number of client requests. When used alone, without the -c option, ab does not process requests concurrently.

  • -t: Specifies a timeframe, in seconds, to spend benchmarking. This option is useful in determining the number of requests that your server is able to process in a given amount of time.

  • -c: Specifies number of simultaneous connections. Of use in testing how your server performs when multiple clients are hitting a given page at the same time. This switch may be used with either -n or -t, to measure the total elapsed time required to process a given number of client requests simultaneously, and the number of concurrent requests that may be processed over a set amount of time, respectively.

  • -g: Exports a TSV (Tab Separated Values) file, which can be imported into applications like Excel, GNUplot, and Mathematica for further analysis and/or graphic visualization.

  • -A: Used to supply username and password information for sites that require authentication. This enables you to conduct testing on password-protected sites.

Having covered some of the basic parameters of ab, we will now explore its use in the context of simple web benchmarking scenarios.

For our first example, we will be measuring the time it takes to process 1000 requests (i.e. -n 1000), from 10 simultaneous user connections (i.e. -c 10) against OS X's default user website. Make sure Apache is running by launching System Preferences and enabling personal web sharing from the sharing preference pane. Once Apache is running, launch the Terminal application and enter the following command: ab -n 1000 -c 10 http://localhost/~username to begin the benchmark. When the command is executed the terminal will display the progress of the benchmark run, as well as, a series of metrics (Figure 1).


Figure 1. Output of ab -n 1000 -c 10 http://localhost/~estein

Be aware that when you run the ab command against a domain or IP address the benchmark defaults to the site's home page; in this case, index.html. To specify a particular page, simply point to its full path (e.g. http://localhost/~estein/MyDynamicContent.php). This will allow you to test the performance of specific pages hosted by your site in terms of the amount of stress they each place on the server. This is particularly important when dealing with sites that have both static and dynamic content. When testing dynamic pages keep in mind that ab reports a failed request when it encounters content that changes over time (e.g. dynamic data). As such, when running ab against dynamic portions of your site, you can safely ignore the "failed requests" messages.

For our next example, we will be measuring the number of requests that can be completed within 60 seconds (-t 60) with 10 concurrent connections (-c 10). Enter the following command in the Terminal to execute this benchmark: ab -t 60 -c 10 http://localhost/~username The output of this command, seen in Figure 2, differs from the first example in that -n is not specified. When -n is not specified, ab sets the number of requests to a default value of 50,000. It is important to realize that, when using the -t switch, it is possible for the benchmark to complete, before the specified time period has elapsed, thus rendering the time variable irrelevant. Therefore, when using the -t and -n switches in tandem, it is important to consider whether your time variable exceeds the time it takes to process n requests.


Figure 2. Output of ab -t 60 -c 10 http://localhost/~estein

JMeter

JMeter is a pure Java application, developed under the purview of the Apache Jakarta project, which offers a wide array of tools for benchmarking all manner of client-serverbased applications. JMeter's modular architecture allows it to simulate a variety of load scenarios across multiple servers, networks, and objects. For the purposes of this article, however, we will only be using a subset of JMeter's capabilities to perform basic load testing. For information on using JMeter to perform detailed performance measures on more complex web applications you may refer to the user guide at http://jakarta.apache.org/jmeter/usermanual/index.html

Before we begin, it is instructive to go over the major architectural components that constitute JMeter.

  • Test Plan: Defines the test sequence that JMeter will carry out during benchmarking.

  • Thread Group: Used to delineate specific testing conditions, such as number of concurrent users, the Thread Group plays host to all other elements that comprise a test sequence.

  • Config Element: Responsible for defining configuration information for the Sampler and Logic Controller elements.

  • Sampler: A type of controller that designates both the protocol type and nature of requests within a given test plan.

  • Logic Controller: A controller element that determines the sequence logic used by JMeter to initiate protocol requests.

  • Timer: Used to manage the latency across queued user requests.

  • Listener: Responsible for recording and visualizing of test sequence data.

Getting Started

JMeter requires JDK 1.4 or higher to run properly. Type java -version in the Terminal application to discover which version of JDK is installed on your machine. Currently Apple supplies Java 1.3.1-1.4.2 Release 2 and J2SE 5.0 release 3 through regular software updates, as well as, at the following links:

http://www.apple.com/downloads/macosx/apple/java131and142release2.html

http://www.apple.com/downloads/macosx/apple/java2se50release3.html

In our tests, both JDK versions were able to run JMeter without a hitch. However, you will likely notice some performance improvement when using the J2SE 5.0 release 3. To specify this version as your system default or to change the precedence settings for your installed Java Virtual Machines (JVMs), use the Java Preferences utility located in /Applications/Utilities/Java/Java2SE 5.0.

The latest version of JMeter is available as a free download at http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi Depending on your preferences, you can download the application as source code or binary as either .zip or .tgz archives. The easiest option for most OS X users will be to download the binary in zip format, which can be easily uncompressed in the GUI by double clicking the jakarta-jmeter-2.1.x.zip archive. The resulting jakarta-jmeter-2.1.1 folder can be placed in your Applications folder or other location within the filesystem. You can launch JMeter by either double clicking the ApacheJMeter.jar file, located in the bin directory of the Jakarta-jmeter-2.1.1 folder, or by executing the jmeter shell script located in the same directory. Please note, that if you wish to run jmeter from the command line you will need to make the script executable using the chmod command. For command line interactive use, please refer to the "Getting Started" section of the JMeter user manual: http://jakarta.apache.org/jmeter/usermanual/get-started.html


Figure 3. JMeter's main window

JMeter in Action

In this example, we will create a test plan that simulates 10 users initiating simultaneous requests to 2 different URLs and which repeats the test 100 times. Begin with the addition of a new Thread Group by right clicking (or control clicking) the Test Plan element in the left pane and selecting Add >Thread Group. Next, select the Thread Group element to display its configuration window and set the Number of Threads to 10 (representing users), the Ramp-Up Period to 0 (to initiate all users at once), and the Loop Count to 100.


Figure 4. Configuring Thread Group element

To add the next element of our test plan, HTTP Request Defaults, right click on the Thread Group element, and select Add > Config Element > HTTP Request Defaults (Figure 5).


Figure 5. Add HTTP Request Defaults

Select the HTTP Request Defaults icon under Thread Group and enter a server name (e.g. localhost or FQDN) in the configuration window (Figure 6).


Figure 6. Configuring HTTP Request Defaults element

Right click on the Thread Group element and then select Add>Sampler>HTTP Request twice, to create two HTTP Request elements, one for each web page. The Name and Path fields for each of the HTTP Request elements should be set to appropriate values. For the purpose of this example, we will name the first HTTP Request element home, and set its path to ~username/index.html and name the second HTTP Request another and set its path to ~username/another.html (figure 7).


Figure 7. Configuring the "another" HTTP Request element

Now we need to add a timer to pause each request for a random period of time. Right click on the Thread Group and then select Add>Timer>Gaussian Random Time.

For the last element, we will add listeners by right clicking on the Thread Group element, then selecting Add>Listener>Aggregate Report. This particular listener lists, in tabular form, the response time, request count, min, max, average, error rate, and approximate number of request processes in seconds (e.g. Throughput), as well as, throughput in terms of Kilobytes per second. You may add additional listeners, such as Graph Results, View Results in Table, and View Results Tree, depending on your requirements.

To begin the test plan, simply select Run>Start from within the JMeter menu bar. You will be prompted to save the test plan, if you haven't already done so. During a successful run of your test plan, you may click on the Aggregate Report listener element (figure 8), or any other listener types you may have added, to view your dynamically updated test results. Figure 9, for example, demonstrates some of the graphing functions of JMeter's Graph Results listener element.


Figure 8. Aggregate Report listener view


Figure 9. Graph Results listener view

Conclusion

In this article, we employed both ab and JMeter to "stress test" a locally hosted site via simple load scenarios. The amount of time a server takes to respond to clients' requests across different load levels, is one of the most important metrics for evaluating web performance. Through the manipulation of load test variables, such as the number of concurrent requests, and with reference to tool-specific data visualization processes, we are better able to ascertain the extent to which these variables impact a server's maximum sustainable load. Such information is critical for proper website management, planning and eventual performance tuning.


Jin Lin and Emmanuel Stein are partners in the consulting firm MacVerse, Corp, which offers implementation, system administration, and development services geared towards the enterprise market. You may reach them at info@macverse.com

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Google Earth 7.1.8.3036 - View and contr...
Google Earth gives you a wealth of imagery and geographic information. Explore destinations like Maui and Paris, or browse content from Wikipedia, National Geographic, and more. Google Earth combines... Read more
QuickBooks 16.1.11.1556 R12 - Financial...
QuickBooks helps you manage your business easily and efficiently. Organize your finances all in one place, track money going in and out of your business, and spot areas where you can save. Built for... Read more
FileZilla 3.24.0 - Fast and reliable FTP...
FileZilla (ported from Windows) is a fast and reliable FTP client and server with lots of useful features and an intuitive interface. Version 3.24.0: New The context menu for remote file search... Read more
Bookends 12.7.8 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more
Duplicate Annihilator 5.8.3 - Find and d...
Duplicate Annihilator takes on the time-consuming task of comparing the images in your iPhoto library using effective algorithms to make sure that no duplicate escapes. Duplicate Annihilator detects... Read more
BusyContacts 1.1.6 - Fast, efficient con...
BusyContacts is a contact manager for OS X that makes creating, finding, and managing contacts faster and more efficient. It brings to contact management the same power, flexibility, and sharing... Read more
MarsEdit 3.7.10 - Quick and convenient b...
MarsEdit is a blog editor for OS X that makes editing your blog like writing email, with spell-checking, drafts, multiple windows, and even AppleScript support. It works with with most blog services... Read more
BusyCal 3.1.4 - Powerful calendar app wi...
BusyCal is an award-winning desktop calendar that combines personal productivity features for individuals with powerful calendar sharing capabilities for families and workgroups. Its unique features... Read more
VirtualBox 5.1.14 - x86 virtualization s...
VirtualBox is a family of powerful x86 virtualization products for enterprise as well as home use. Not only is VirtualBox an extremely feature rich, high performance product for enterprise customers... Read more
Bookends 12.7.8 - Reference management a...
Bookends is a full-featured bibliography/reference and information-management system for students and professionals. Bookends uses the cloud to sync reference libraries on all the Macs you use.... Read more

Super Mario Run dashes onto Android in M...
Super Mario Run was one of the biggest mobile launches in 2016 before it was met with a lukewarm response by many. While the game itself plays a treat, it's pretty hard to swallow the steep price for the full game. With that said, Android users... | Read more »
WarFriends Beginner's Guide: How to...
Chillingo's new game, WarFriends, is finally available world wide, and so far it's a refreshing change from common mobile game trends. The game's a mix of tower defense, third person shooter, and collectible card game. There's a lot to unpack here... | Read more »
Super Gridland (Entertainment)
Super Gridland 1.0 Device: iOS Universal Category: Entertainment Price: $1.99, Version: 1.0 (iTunes) Description: Match. Build. Survive. "exquisitely tuned" - Rock Paper Shotgun No in-app purches, and no ads! | Read more »
Red's Kingdom (Games)
Red's Kingdom 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: Mad King Mac has kidnapped your father and stolen your golden nut! Solve puzzles and battle goons as you explore and battle your... | Read more »
Turbo League Guide: How to tame the cont...
| Read more »
Fire Emblem: Heroes coming to Google Pla...
Nintendo gave us our first look at Fire Emblem: Heroes, the upcoming mobile Fire Emblem game the company hinted at last year. Revealed at the Fire Emblem Direct event held today, the game will condense the series' tactical RPG combat into bite-... | Read more »
ReSlice (Music)
ReSlice 1.0 Device: iOS Universal Category: Music Price: $9.99, Version: 1.0 (iTunes) Description: Audio Slice Machine Slice your audio samples with ReSlice and create flexible musical atoms which can be triggered by MIDI notes or... | Read more »
Stickman Surfer rides in with the tide t...
Stickson is back and this time he's taken up yet another extreme sport - surfing. Stickman Surfer is out this Thursday on both iOS and Android, so if you've been following the other Stickman adventures, you might be interested in picking this one... | Read more »
Z-Exemplar (Games)
Z-Exemplar 1.4 Device: iOS Universal Category: Games Price: $3.99, Version: 1.4 (iTunes) Description: | Read more »
5 dastardly difficult roguelikes like th...
Edmund McMillen's popular roguelike creation The Binding of Isaac: Rebirth has finally crawled onto mobile devices. It's a grotesque dual-stick shooter that tosses you into an endless, procedurally generated basement as you, the pitiable Isaac,... | Read more »

Price Scanner via MacPrices.net

Twelve South Releases RelaxedLeather Cases fo...
Inspired by the laid-back luxury of burnished leather boots and crafted in rich tones of taupe, herb and marsala, RelaxedLeather cases deliver smart, easy protection for the iPhone 7. Each genuine... Read more
Week’s Best Deal: New 2016 13-inch 2.0GHz Mac...
Amazon has the new 2016 13″ 2.0GHz non-Touch Bar MacBook Pros on sale for a limited time for $225 off MSRP including free shipping: - 13″ 2.0GHz MacBook Pro, Space Gray (MLL42LL/A): $1274.99 $225 off... Read more
Back in stock: Apple refurbished Mac minis fr...
Apple has Certified Refurbished Mac minis available starting at $419. Apple’s one-year warranty is included with each mini, and shipping is free: - 1.4GHz Mac mini: $419 $80 off MSRP - 2.6GHz Mac... Read more
Apple Ranked ‘Most Intimate Brand’
The top ranked ‘”intimate” brands continued to outperform the S&P and Fortune 500 indices in revenue and profit over the past 10 years, according to MBLM’s Brand Intimacy 2017 Report, the largest... Read more
B-Eng introduces SSD Health Check for Mac OS
Fehraltorf, Switzerland based independant Swiss company- B-Eng has announced the release and immediate availability of SSD Health Check 1.0, the company’s new hard drive utility for Mac OS X. As the... Read more
Apple’s Education discount saves up to $300 o...
Purchase a new Mac or iPad using Apple’s Education Store and take up to $300 off MSRP. All teachers, students, and staff of any educational institution qualify for the discount. Shipping is free: -... Read more
4-core 3.7GHz Mac Pro on sale for $2290, save...
Guitar Center has the 3.7GHz 4-core Mac Pro (MD253LL/A) on sale for $2289.97 including free shipping or free local store pickup (if available). Their price is a $710 savings over standard MSRP for... Read more
128GB Apple iPad Air 2, refurbished, availabl...
Apple has Certified Refurbished 128GB iPad Air 2s WiFis available for $419 including free shipping. That’s an $80 savings over standard MSRP for this model. A standard Apple one-year warranty is... Read more
13-inch 2.7GHz Retina MacBook Pro on sale for...
B&H Photo has the 2015 13″ 2.7GHz/128GB Retina Apple MacBook Pro on sale for $100 off MSRP. Shipping is free, and B&H charges NY tax only: - 13″ 2.7GHz/128GB Retina MacBook Pro (MF839LL/A): $... Read more
Laptop Market – Flight To Quality? – The ‘Boo...
Preliminary quarterly PC shipments data released by Gartner Inc. last week reveal an interesting disparity between sales performance of major name PC vendors as opposed to that of less well-known... Read more

Jobs Board

*Apple* macOS Systems Integration Administra...
…most exceptional support available in the industry. SCI is seeking an Junior Apple macOS systems integration administrator that will be responsible for providing Read more
*Apple* Retail - Multiple Positions- Deer Pa...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
*Apple* Technician - nfrastructure (United S...
Let’s Work Together Apple Technician This position is based in Portland, ME Life at nfrastructure At nfrastructure, we understand that our success results from our Read more
*Apple* Mobile Master - Best Buy (United Sta...
**467692BR** **Job Title:** Apple Mobile Master **Location Number:** 000602-Columbia MO-Store **Job Description:** **What does a Best Buy Apple Mobile Master Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.