TweetFollow Us on Twitter

Web Benchmarking 101

Volume Number: 22 (2006)
Issue Number: 4
Column Tag: Web Performance Testing

Web Benchmarking 101

A guide to stress testing your website with ApacheBench and JMeter

by Jin Lin and Emmanuel Stein

Introduction

Web performance testing can be a daunting subject for the uninitiated. With the sheer number of tools available and their various feature sets, it can be hard to know where to begin. In this article we will explore the use of two popular open source tools, ApacheBench (aka ab) and JMeter, to perform simple benchmarking tests. Rather than going through the extensive feature set offered by these applications, particularly JMeter, we will focus on measuring the response time and the effective throughput of your web application across different user load scenarios. Whether serving a few static pages or a fully dynamic site, the tools discussed will offer invaluable insight into the overall performance of your web applications, and serve as a basis for isolating system bottlenecks and conducting capacity planning for your web applications.

Benchmarking Considerations

Although, the tests and tools that we will use in this article represent integral components used in the testing process, it is equally important to consider the environment in which such tests are performed. If you are running a popular website with significant traffic, you may consider performing benchmarks during non-peak hours to obtain more accurate results and avoid rendering your server unusable to users. If possible, you should test your web server in a staged environment and before actual deployment. This will enable you to adjust server parameters, install cache mechanisms, and otherwise assess your site's performance without interrupting service.

ApacheBench

ApacheBench, a default component of the Apache Server distribution, is a simple command line tool, designed to measure the performance of Apache server. Preinstalled on Mac OS X, it is able to supply information such as elapsed time, requests per second, time per request, compression rate, transfer rate, and connection time under different concurrence scenarios.

Though not an exhaustive list (see the ab man page for more details), the following represents the most commonly used switches employed when testing typical usage scenarios with ab.

  • -n: Specifies the number of requests to perform for a given test. Use this option to measure the time required to process a given number of client requests. When used alone, without the -c option, ab does not process requests concurrently.

  • -t: Specifies a timeframe, in seconds, to spend benchmarking. This option is useful in determining the number of requests that your server is able to process in a given amount of time.

  • -c: Specifies number of simultaneous connections. Of use in testing how your server performs when multiple clients are hitting a given page at the same time. This switch may be used with either -n or -t, to measure the total elapsed time required to process a given number of client requests simultaneously, and the number of concurrent requests that may be processed over a set amount of time, respectively.

  • -g: Exports a TSV (Tab Separated Values) file, which can be imported into applications like Excel, GNUplot, and Mathematica for further analysis and/or graphic visualization.

  • -A: Used to supply username and password information for sites that require authentication. This enables you to conduct testing on password-protected sites.

Having covered some of the basic parameters of ab, we will now explore its use in the context of simple web benchmarking scenarios.

For our first example, we will be measuring the time it takes to process 1000 requests (i.e. -n 1000), from 10 simultaneous user connections (i.e. -c 10) against OS X's default user website. Make sure Apache is running by launching System Preferences and enabling personal web sharing from the sharing preference pane. Once Apache is running, launch the Terminal application and enter the following command: ab -n 1000 -c 10 http://localhost/~username to begin the benchmark. When the command is executed the terminal will display the progress of the benchmark run, as well as, a series of metrics (Figure 1).


Figure 1. Output of ab -n 1000 -c 10 http://localhost/~estein

Be aware that when you run the ab command against a domain or IP address the benchmark defaults to the site's home page; in this case, index.html. To specify a particular page, simply point to its full path (e.g. http://localhost/~estein/MyDynamicContent.php). This will allow you to test the performance of specific pages hosted by your site in terms of the amount of stress they each place on the server. This is particularly important when dealing with sites that have both static and dynamic content. When testing dynamic pages keep in mind that ab reports a failed request when it encounters content that changes over time (e.g. dynamic data). As such, when running ab against dynamic portions of your site, you can safely ignore the "failed requests" messages.

For our next example, we will be measuring the number of requests that can be completed within 60 seconds (-t 60) with 10 concurrent connections (-c 10). Enter the following command in the Terminal to execute this benchmark: ab -t 60 -c 10 http://localhost/~username The output of this command, seen in Figure 2, differs from the first example in that -n is not specified. When -n is not specified, ab sets the number of requests to a default value of 50,000. It is important to realize that, when using the -t switch, it is possible for the benchmark to complete, before the specified time period has elapsed, thus rendering the time variable irrelevant. Therefore, when using the -t and -n switches in tandem, it is important to consider whether your time variable exceeds the time it takes to process n requests.


Figure 2. Output of ab -t 60 -c 10 http://localhost/~estein

JMeter

JMeter is a pure Java application, developed under the purview of the Apache Jakarta project, which offers a wide array of tools for benchmarking all manner of client-serverbased applications. JMeter's modular architecture allows it to simulate a variety of load scenarios across multiple servers, networks, and objects. For the purposes of this article, however, we will only be using a subset of JMeter's capabilities to perform basic load testing. For information on using JMeter to perform detailed performance measures on more complex web applications you may refer to the user guide at http://jakarta.apache.org/jmeter/usermanual/index.html

Before we begin, it is instructive to go over the major architectural components that constitute JMeter.

  • Test Plan: Defines the test sequence that JMeter will carry out during benchmarking.

  • Thread Group: Used to delineate specific testing conditions, such as number of concurrent users, the Thread Group plays host to all other elements that comprise a test sequence.

  • Config Element: Responsible for defining configuration information for the Sampler and Logic Controller elements.

  • Sampler: A type of controller that designates both the protocol type and nature of requests within a given test plan.

  • Logic Controller: A controller element that determines the sequence logic used by JMeter to initiate protocol requests.

  • Timer: Used to manage the latency across queued user requests.

  • Listener: Responsible for recording and visualizing of test sequence data.

Getting Started

JMeter requires JDK 1.4 or higher to run properly. Type java -version in the Terminal application to discover which version of JDK is installed on your machine. Currently Apple supplies Java 1.3.1-1.4.2 Release 2 and J2SE 5.0 release 3 through regular software updates, as well as, at the following links:

http://www.apple.com/downloads/macosx/apple/java131and142release2.html

http://www.apple.com/downloads/macosx/apple/java2se50release3.html

In our tests, both JDK versions were able to run JMeter without a hitch. However, you will likely notice some performance improvement when using the J2SE 5.0 release 3. To specify this version as your system default or to change the precedence settings for your installed Java Virtual Machines (JVMs), use the Java Preferences utility located in /Applications/Utilities/Java/Java2SE 5.0.

The latest version of JMeter is available as a free download at http://jakarta.apache.org/site/downloads/downloads_jmeter.cgi Depending on your preferences, you can download the application as source code or binary as either .zip or .tgz archives. The easiest option for most OS X users will be to download the binary in zip format, which can be easily uncompressed in the GUI by double clicking the jakarta-jmeter-2.1.x.zip archive. The resulting jakarta-jmeter-2.1.1 folder can be placed in your Applications folder or other location within the filesystem. You can launch JMeter by either double clicking the ApacheJMeter.jar file, located in the bin directory of the Jakarta-jmeter-2.1.1 folder, or by executing the jmeter shell script located in the same directory. Please note, that if you wish to run jmeter from the command line you will need to make the script executable using the chmod command. For command line interactive use, please refer to the "Getting Started" section of the JMeter user manual: http://jakarta.apache.org/jmeter/usermanual/get-started.html


Figure 3. JMeter's main window

JMeter in Action

In this example, we will create a test plan that simulates 10 users initiating simultaneous requests to 2 different URLs and which repeats the test 100 times. Begin with the addition of a new Thread Group by right clicking (or control clicking) the Test Plan element in the left pane and selecting Add >Thread Group. Next, select the Thread Group element to display its configuration window and set the Number of Threads to 10 (representing users), the Ramp-Up Period to 0 (to initiate all users at once), and the Loop Count to 100.


Figure 4. Configuring Thread Group element

To add the next element of our test plan, HTTP Request Defaults, right click on the Thread Group element, and select Add > Config Element > HTTP Request Defaults (Figure 5).


Figure 5. Add HTTP Request Defaults

Select the HTTP Request Defaults icon under Thread Group and enter a server name (e.g. localhost or FQDN) in the configuration window (Figure 6).


Figure 6. Configuring HTTP Request Defaults element

Right click on the Thread Group element and then select Add>Sampler>HTTP Request twice, to create two HTTP Request elements, one for each web page. The Name and Path fields for each of the HTTP Request elements should be set to appropriate values. For the purpose of this example, we will name the first HTTP Request element home, and set its path to ~username/index.html and name the second HTTP Request another and set its path to ~username/another.html (figure 7).


Figure 7. Configuring the "another" HTTP Request element

Now we need to add a timer to pause each request for a random period of time. Right click on the Thread Group and then select Add>Timer>Gaussian Random Time.

For the last element, we will add listeners by right clicking on the Thread Group element, then selecting Add>Listener>Aggregate Report. This particular listener lists, in tabular form, the response time, request count, min, max, average, error rate, and approximate number of request processes in seconds (e.g. Throughput), as well as, throughput in terms of Kilobytes per second. You may add additional listeners, such as Graph Results, View Results in Table, and View Results Tree, depending on your requirements.

To begin the test plan, simply select Run>Start from within the JMeter menu bar. You will be prompted to save the test plan, if you haven't already done so. During a successful run of your test plan, you may click on the Aggregate Report listener element (figure 8), or any other listener types you may have added, to view your dynamically updated test results. Figure 9, for example, demonstrates some of the graphing functions of JMeter's Graph Results listener element.


Figure 8. Aggregate Report listener view


Figure 9. Graph Results listener view

Conclusion

In this article, we employed both ab and JMeter to "stress test" a locally hosted site via simple load scenarios. The amount of time a server takes to respond to clients' requests across different load levels, is one of the most important metrics for evaluating web performance. Through the manipulation of load test variables, such as the number of concurrent requests, and with reference to tool-specific data visualization processes, we are better able to ascertain the extent to which these variables impact a server's maximum sustainable load. Such information is critical for proper website management, planning and eventual performance tuning.


Jin Lin and Emmanuel Stein are partners in the consulting firm MacVerse, Corp, which offers implementation, system administration, and development services geared towards the enterprise market. You may reach them at info@macverse.com

 

Community Search:
MacTech Search:

Software Updates via MacUpdate

Sound Studio 4.8.11 - Robust audio recor...
Sound Studio lets you easily record and professionally edit audio on your Mac. Easily rip vinyls and digitize cassette tapes, or record lectures and voice memos. Prepare for live shows with live... Read more
BetterTouchTool 2.291 - Customize Multi-...
BetterTouchTool adds many new, fully customizable gestures to the Magic Mouse, Multi-Touch MacBook trackpad, and Magic Trackpad. These gestures are customizable: Magic Mouse: Pinch in / out (zoom... Read more
Carbon Copy Cloner 4.1.18 - Easy-to-use...
Carbon Copy Cloner backups are better than ordinary backups. Suppose the unthinkable happens while you're under deadline to finish a project: your Mac is unresponsive and all you hear is an ominous,... Read more
Hopper Disassembler 4.2.14- - Binary dis...
Hopper Disassembler is a binary disassembler, decompiler, and debugger for 32- and 64-bit executables. It will let you disassemble any binary you want, and provide you all the information about its... Read more
VOX 2.8.30 - Music player that supports...
VOX just sounds better! The beauty is in its simplicity, yet behind the minimal exterior lies a powerful music player with a ton of features and support for all audio formats you should ever need.... Read more
Default Folder X 5.1.6b3 - Enhances Open...
Default Folder X attaches a toolbar to the right side of the Open and Save dialogs in any OS X-native application. The toolbar gives you fast access to various folders and commands. You just click on... Read more
CleanMyMac 3.8.6 - $39.95
CleanMyMac makes space for the things you love. Sporting a range of ingenious new features, CleanMyMac lets you safely and intelligently scan and clean your entire system, delete large, unused files... Read more
Postbox 5.0.17 - Powerful and flexible e...
Postbox is a new email application that helps you organize your work life and get stuff done. It has all the elegance and simplicity of Apple Mail, but with more power and flexibility to manage even... Read more
Amazon Chime 4.6.5852 - Amazon-based com...
Amazon Chime is a communications service that transforms online meetings with a secure, easy-to-use application that you can trust. Amazon Chime works seamlessly across your devices so that you can... Read more
coconutBattery 3.6.3 - Displays info abo...
With coconutBattery you're always aware of your current battery health. It shows you live information about your battery such as how often it was charged and how is the current maximum capacity in... Read more

The best 2v2 card combos in Clash Royale
2v2 is making it's grand return toClash Royalequite soon. 2v2 has quickly become one of the game's most popular gameplay modes, though they still have yet to make it a permanent fixture in the game. 2v2 is exciting and adds some new flavor to... | Read more »
The best games we played this week - Aug...
Another busy week has come to a close. We played a lot of excellent games this week and now it's time to look back and reflect on some our favorites. Here are our picks for the week of August 18. [Read more] | Read more »
War Wings beginner's guide - how to...
War Wings is the newest project from well-established game maker Miniclip. It's a World War II aerial dogfighting game with loads of different airplane models to unlock and battle. The game offers plenty of single player and multiplayer action. We... | Read more »
How to win every 2v2 battle in Clash Roy...
2v2 is coming back to Clash Royale in a big way. Although it's only been available for temporary periods of time, 2v2 has seen a hugely positive fan response, with players clamoring for more team-based gameplay. Soon we'll get yet another taste of... | Read more »
Roll to Win with Game of Dice’s new upda...
Joycity’s hit Game of Dice gets a big new update this week, introducing new maps, mechanics, and even costumes. The update sets players loose on an exciting new map, The Cursed Tower, that allows folks to use special Runes mid-match. If you feel... | Read more »
Bottom of the 9th (Games)
Bottom of the 9th 1.0.1 Device: iOS iPhone Category: Games Price: $4.99, Version: 1.0.1 (iTunes) Description: Play the most exciting moment of baseball in this fast-paced dice and card game! | Read more »
The best apps for viewing the solar ecli...
If you somehow missed the news, many parts of the United States will be witness to a total solar eclipse on August 21 for the first time in over 90 years. It'll be possible to see the eclipse in at least some capacity throughout the continental U... | Read more »
The 5 best mobile survival games
Games like ARK: Survival Evolved and Conan Exiles have taken the world of gaming by storm. The market is now flooded with hardcore survival games that send players off into the game's world with nothing but maybe the clothes on their back. Never... | Read more »
Portal Walk (Games)
Portal Walk 1.0 Device: iOS Universal Category: Games Price: $1.99, Version: 1.0 (iTunes) Description: Portal Walk is adventure and relaxing platform game about Eugene. Eugene stuck between worlds and trying to find way back home.... | Read more »
Technobabylon (Games)
Technobabylon 1.0 Device: iOS Universal Category: Games Price: $4.99, Version: 1.0 (iTunes) Description: City of Newton, 2087. Genetic engineering is the norm, the addictive Trance has replaced almost any need for human interaction,... | Read more »

Price Scanner via MacPrices.net

Weekend sale: 13-inch MacBook Pros for up to...
Amazon has new 2017 13″ MacBook Pros on sale today for up to $200 off MSRP, each including free shipping: – 13″ 3.1GHz/256GB Space Gray MacBook Pro (MPXV2LL/A): $1599.99 $200 off MSRP – 13″ 3.1GHz/... Read more
Back To School With The Edge Desk All-in-one...
Back to school is just around the corner, and the ergonomically correct Edge Desk all-in-one portable kneeling desk is ideal for students living in dorms and small apartments, Edge Desk features:... Read more
Norton Core Secure Wi-Fi Router Now Available...
First introduced at the 2017 Consumer Electronics Show (CES), Norton Core, a secure, high-performance Wi-Fi router, fundamentally changed the concept of Wi-Fi routers by making security the primary... Read more
ViewSonic Adds New 27-inch 4K UHD Monitor to...
ViewSonic Corp. has introduced the VP2785-4K, a 27-inch 4K UHD (3840×2160) monitor that delivers precise and consistent color representation and performance to ensure incredible image quality. Built... Read more
Apple now offering Certified Refurbished 2017...
Apple is now offering Certified Refurbished 2017 27″ iMacs for up to $350 off original MSRP. Apple’s one-year warranty is standard, and shipping is free. The following models are available: – 27″ 3.... Read more
13-inch 2.3GHz MacBook Pros on sale for $100...
Amazon has the new 2017 13″ 2.3GHz MacBook Pros on sale today for $100 off MSRP, each including free shipping: – 13″ 2.3GHz/128GB Space Gray MacBook Pro (MPXQ2LL/A): $1199.99 $100 off MSRP – 13″ 2.... Read more
Clearance 2016 13-inch MacBook Airs available...
B&H Photo has clearance 2016 13″ MacBook Airs available for up to $200 off original MSRP. Shipping is free, and B&H charges NY & NJ sales tax only: – 13″ 1.6GHz/128GB MacBook Air (MMGF2LL... Read more
Clearance 21-inch and 27-inch iMacs available...
B&H Photo has clearance 21″ and 27″ Apple iMacs available for up to $500 off original MSRP, each including free shipping plus NY & NJ sales tax only: – 27″ 3.3GHz iMac 5K: $1799 $500 off... Read more
New iOS 11 Productivity Features Welcome But...
The iOS community is in late summer holding mode awaiting the September arrival of the iPhone 8 and iOS 11. iOS 11 public betas have been available for months — number six was released this week —... Read more
Samsung Electronics Launches New Portable SSD...
Samsung Electronics America, Inc. has announced the launch of Samsung Portable SSD T5 – its newest portable solid state drive (PSSD) that raises the bar for the performance of external memory... Read more

Jobs Board

Map Quality Analyst Lead, *Apple* Maps Eval...
…a team of subject matter experts measuring the quality of a range of Apple Maps algorithms, such as Map search, recommendations, and routing. Our group works with Read more
Business Development Manager, *Apple* Pay -...
Job Summary Apple Pay is seeking an experienced Business Development professional to join the Apple Pay team to develop partnerships and strategic alliances with Read more
Information Systems Engineer, *Apple* Retai...
Job Summary Do you have a passion for the technology and systems that make Apple amazing? Are you interested in joining IS&T Retail on what they do best? This is an Read more
*Apple* Retail - Multiple Positions - Apple,...
Job Description: Sales Specialist - Retail Customer Service and Sales Transform Apple Store visitors into loyal Apple customers. When customers enter the store, Read more
Development Operations and Site Reliability E...
Development Operations and Site Reliability Engineer, Apple Payment Gateway Job Number: 57572631 Santa Clara Valley, California, United States Posted: Jul. 27, 2017 Read more
All contents are Copyright 1984-2011 by Xplain Corporation. All rights reserved. Theme designed by Icreon.