Load testing
Encyclopedia
Load testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation. When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing
. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test.
There is little agreement on what the specific goals of load testing are. The term is often used synonymously with software performance testing, reliability testing, and volume testing
. Load testing is a type of non-functional testing.
community. Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program concurrently. As such, this testing is most relevant for multi-user systems; often one built using a client/server model, such as web servers. However, other types of software systems can also be load tested. For example, a word processor or graphics editor can be forced to read an extremely large document; or a financial package can be forced to generate a report based on several years' worth of data. The most accurate load testing simulates actual use, as opposed to testing using theoretical or analytical modeling.
Load testing lets you measure your website's QOS performance based on actual customer behavior. Nearly all the load testing tools and frame-works follow the classical load testing paradigm, which is listed in Figure 1. When
customers visit your web site, a script recorder records the communication and then creates related interaction scripts. A load generator tries to replay the recorded scripts, which could possibly be modified with different test parameters before replay. In the replay procedure, both the hardware and software statistics will be monitored and collected by the conductor, these statistics include the CPU, memory, disk IO
of the physical servers and the response time, throughput of the System Under Test (short as SUT), etc. And at last, all these statistics will be analyzed and a load testing report will
be generated.
Load and performance testing analyzes software intended for a multi-user audience by subjecting the software to different amounts of virtual and live users while monitoring performance measurements under these different loads. Load and performance testing is usually conducted in a test environment identical to the production environment before the software system is permitted to go live.
As an example, a web site with shopping cart
capability is required to support 100 concurrent users broken out into following activities:
A test analyst can use various load testing tools to create these VUsers and their activities. Once the test has started and reached a steady state, the application is being tested at the 100 VUser load as described above. The application’s performance can then be monitored and captured.
The specifics of a load test plan
or script
will generally vary across organizations. For example, in the bulleted list above, the first item could represent 25 VUsers browsing unique items, random items, or a selected set of items depending upon the test plan or script developed. However, all load test plans attempt to simulate system performance across a range of anticipated peak workflows and volumes. The criteria for passing or failing a load test (pass/fail criteria) are generally different across organizations as well. There are no standards specifying acceptable load testing performance metrics.
A common misconception is that load testing software provides record and playback capabilities like regression testing
tools. Load testing tools analyze the entire OSI protocol stack
whereas most regression testing tools focus on GUI
performance. For example, a regression testing tool will record and playback a mouse click on a button on a web browser, but a load testing tool will send out hypertext the web browser sends after the user clicks the button. In a multiple-user environment, load testing tools can send out hypertext for multiple users with each user having a unique login ID, password, etc.
The popular load testing tools available also provide insight into the causes for slow performance. There are numerous possible causes for slow system performance, including, but not limited to, the following:
Load testing is especially important if the application, system or service will be subject to a service level agreement
or SLA.
Many performance testers are running this test, but they call it different names. This name was selected by the Panelists and many Performance Testers in 2011 Online Performance Summit by STP.
There are already many tools and frameworks available to do the load testing from both commercial and open source.
The Supply of Machinery (Safety) Regulation 1992 UK state that load testing is undertaken before the equipment is put into service for the first time.
Load testing can be either Performance,Static or Dynamic.
Performance testing is when the stated safe working load (SWL) for a configuration is used to determine that the item performs to the manufactures specification. If an item fails this test then any further tests are pointless.
Static testing is when a load at a factor above the SWL is applied. The item is not operated through all configurations as it is not a requirement of this test.
Dynamic testing is when a load at a factor above the SWL is applied. The item is then operated fully through all configurations and motions. Care must be taken during this test as there is a great risk of catastrophic failure if incorrectly carried out.
The design criteria, relevant legislation or the Competent Person will dictate what test is required.
Under the Lifting Operations and Lifting Equipment Regulations 1998 UK load testing after the initial test is required if a major component is replaced, if the item is moved from one location to another or as dictated by the Competent Person
The loads required for a test are stipulated by the item under test, but here are a few to be aware off.
Powered lifting equipment Static test to 1.25 SWL and dynamic test to 1.1 SWL.
Manual lifting equipment Static test to 1.5 SWL
For lifting accessories.
2 SWL for items up to 30 tonne capacity.
1.5 SWL for items above 30 tonne capacity.
1 SWL for items above 100 tonnes.
. The tester consists of a large resistor that has a resistance similar to a car's starter motor and a meter to read the battery's output voltage both in the unloaded and loaded state. When the tester is used, the battery's open circuit voltage is checked first. If the open circuit voltage is below spec (12.6 volts for a fully charged battery), the battery is charged first. After reading the battery's open circuit voltage, the load is applied. When applied, it draws approximately the same current the car's starter motor would draw during cranking. Based on the specified cold cranking amperes of the battery, if the voltage under load falls below a certain point, the battery is bad. Load tests are also used on running cars to check the output of the car's alternator
.
Stress testing
Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results...
. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test.
There is little agreement on what the specific goals of load testing are. The term is often used synonymously with software performance testing, reliability testing, and volume testing
Volume testing
Volume Testing belongs to the group of non-functional tests, which are often misunderstood and/or used interchangeably. Volume testing refers to testing a software application with a certain amount of data. This amount can, in generic terms, be the database size or it could also be the size of an...
. Load testing is a type of non-functional testing.
Software load testing
The term load testing is used in different ways in the professional software testingSoftware testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
community. Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program concurrently. As such, this testing is most relevant for multi-user systems; often one built using a client/server model, such as web servers. However, other types of software systems can also be load tested. For example, a word processor or graphics editor can be forced to read an extremely large document; or a financial package can be forced to generate a report based on several years' worth of data. The most accurate load testing simulates actual use, as opposed to testing using theoretical or analytical modeling.
Load testing lets you measure your website's QOS performance based on actual customer behavior. Nearly all the load testing tools and frame-works follow the classical load testing paradigm, which is listed in Figure 1. When
customers visit your web site, a script recorder records the communication and then creates related interaction scripts. A load generator tries to replay the recorded scripts, which could possibly be modified with different test parameters before replay. In the replay procedure, both the hardware and software statistics will be monitored and collected by the conductor, these statistics include the CPU, memory, disk IO
of the physical servers and the response time, throughput of the System Under Test (short as SUT), etc. And at last, all these statistics will be analyzed and a load testing report will
be generated.
Load and performance testing analyzes software intended for a multi-user audience by subjecting the software to different amounts of virtual and live users while monitoring performance measurements under these different loads. Load and performance testing is usually conducted in a test environment identical to the production environment before the software system is permitted to go live.
As an example, a web site with shopping cart
Shopping cart software
Shopping cart software is software used in e-commerce to assist people making purchases online, analogous to the American English term 'shopping cart'...
capability is required to support 100 concurrent users broken out into following activities:
- 25 Virtual Users (VUsers) log in, browse through items and then log off
- 25 VUsers log in, add items to their shopping cart, check out and then log off
- 25 VUsers log in, return items previously purchased and then log off
- 25 VUsers just log in without any subsequent activity
A test analyst can use various load testing tools to create these VUsers and their activities. Once the test has started and reached a steady state, the application is being tested at the 100 VUser load as described above. The application’s performance can then be monitored and captured.
The specifics of a load test plan
Test plan
A test plan is a document detailing a systematic approach to testing a system such as a machine or software. The plan typically contains a detailed understanding of what the eventual workflow will be.-Test plans:...
or script
Test script
A test script in software testing is a set of instructions that will be performed on the system under test to test that the system functions as expected.There are various means for executing test scripts.* Manual testing...
will generally vary across organizations. For example, in the bulleted list above, the first item could represent 25 VUsers browsing unique items, random items, or a selected set of items depending upon the test plan or script developed. However, all load test plans attempt to simulate system performance across a range of anticipated peak workflows and volumes. The criteria for passing or failing a load test (pass/fail criteria) are generally different across organizations as well. There are no standards specifying acceptable load testing performance metrics.
A common misconception is that load testing software provides record and playback capabilities like regression testing
Regression testing
Regression testing is any type of software testing that seeks to uncover new errors, or regressions, in existing functionality after changes have been made to a system, such as functional enhancements, patches or configuration changes....
tools. Load testing tools analyze the entire OSI protocol stack
OSI model
The Open Systems Interconnection model is a product of the Open Systems Interconnection effort at the International Organization for Standardization. It is a prescription of characterizing and standardizing the functions of a communications system in terms of abstraction layers. Similar...
whereas most regression testing tools focus on GUI
Gui
Gui or guee is a generic term to refer to grilled dishes in Korean cuisine. These most commonly have meat or fish as their primary ingredient, but may in some cases also comprise grilled vegetables or other vegetarian ingredients. The term derives from the verb, "gupda" in Korean, which literally...
performance. For example, a regression testing tool will record and playback a mouse click on a button on a web browser, but a load testing tool will send out hypertext the web browser sends after the user clicks the button. In a multiple-user environment, load testing tools can send out hypertext for multiple users with each user having a unique login ID, password, etc.
The popular load testing tools available also provide insight into the causes for slow performance. There are numerous possible causes for slow system performance, including, but not limited to, the following:
- Application server(s) or software
- Database server(s)
- Network – latencyLatency (engineering)Latency is a measure of time delay experienced in a system, the precise definition of which depends on the system and the time being measured. Latencies may have different meaning in different contexts.-Packet-switched networks:...
, congestionNetwork congestionIn data networking and queueing theory, network congestion occurs when a link or node is carrying so much data that its quality of service deteriorates. Typical effects include queueing delay, packet loss or the blocking of new connections...
, etc. - Client-side processing
- Load balancing between multiple servers
Load testing is especially important if the application, system or service will be subject to a service level agreement
Service Level Agreement
A service-level agreement is a part of a service contract where the level of service is formally defined. In practice, the term SLA is sometimes used to refer to the contracted delivery time or performance...
or SLA.
User Experience Under Load test
In the example above, while the device under test (DUT) is under production load - 100 VUsers, run the target application. The performance of the target application here would be the User Experience Under Load. It describe how fast or slow the DUT responds, and how satisfied or how the user actually perceives performance.Many performance testers are running this test, but they call it different names. This name was selected by the Panelists and many Performance Testers in 2011 Online Performance Summit by STP.
There are already many tools and frameworks available to do the load testing from both commercial and open source.
Load testing tools
| Tool Name | Company Name | Notes |
|---|---|---|
| AppLoader AppLoader NRG Global’s AppLoader is a performance and load testing automation solution for analyzing all system’s tiers behavior while simulating the load. AppLoader can emulate number of users simultaneously from the GUI level to produce real life user loads. It monitors front end and backend system... |
NRG Global | Load and Performance testing Solution. Automates tests on the GUI level of the application. Can be used for unit, integration, and regression testing as well. Licensed. |
| IBM Rational Performance Tester IBM Rational Performance Tester IBM Rational Performance Tester is a tool for automated performance testing of web and server based applications from the Rational Software division of IBM. It allows users to create tests that mimic user transactions between an application client and server. During test execution, these... |
IBM IBM International Business Machines Corporation or IBM is an American multinational technology and consulting corporation headquartered in Armonk, New York, United States. IBM manufactures and sells computer hardware and software, and it offers infrastructure, hosting and consulting services in areas... |
Eclipse based large scale performance testing tool primarily used for executing large volume performance tests to measure system response time for server based applications. Licensed. |
| Apache JMeter | An Apache Jakarta open source project | Java desktop application for load testing and performance measurement. |
| Load Test (included with Soatest Soatest Parasoft SOAtest is "a comprehensive testing and analysis tool suite for tailored to the unique testing and validation needs of Service Oriented Architectures." It helps testers test the message layer functionality of their services by automating their testing and supports numerous transport... |
Parasoft Parasoft Parasoft is an independent software vendor with headquarters in Monrovia, California. It was founded in 1987 by five graduates of the California Institute of Technology who had been working on Caltech Cosmic Cube.... |
Performance testing tool that verifies functionality and performance under load. Supports SOAtest tests, JUnits, lightweight socket-based components. Detects concurrency issues. |
| LoadRunner LoadRunner HP LoadRunner software is an automated performance and testing product from Hewlett-Packard for examining system behaviour and performance, while generating actual load. HP acquired LoadRunner as part of its acquisition of Mercury Interactive in November 2006... |
HP | Performance testing tool primarily used for executing large numbers of tests (or a large number or virtual users) concurrently. Can be used for unit and integration testing as well. Licensed. |
| OpenSTA OpenSTA OpenSTA is a feature-rich GUI-based web server benchmarking utility that can perform scripted HTTP and HTTPS heavy load tests with performance measurements. It is freely available and distributable under the open source GNU General Public License. OpenSTA currently only runs on Microsoft... |
Open System Testing Architecture | Open source web load/stress testing application, licensed under the Gnu GPL. Utilizes a distributed software architecture based on CORBA. OpenSTA binaries available for Windows. |
| SilkPerformer Borland Borland Software Corporation is a software company first headquartered in Scotts Valley, California, Cupertino, California and finally Austin, Texas. It is now a Micro Focus subsidiary. It was founded in 1983 by Niels Jensen, Ole Henriksen, Mogens Glad and Philippe Kahn.-The 1980s:... |
Micro Focus | Performance testing in an open and sharable model which allows realistic load tests for thousands of users running business scenarios across a broad range of enterprise application environments. |
| SLAMD SLAMD The SLAMD Distributed Load Generation Engine is a Java-based application designed for stress testing and performance analysis of network-based applications. Initial development of SLAMD was performed at Sun Microsystems, with recent development sponsored primarily by .SLAMD is available under the... |
Open source, 100% Java web application, scriptable, distributed with Tomcat Tomcat Tomcat may refer to:* A male cat, known as a tom or tommiecat* Tomcats , a 2001 comedy film* Apache Tomcat, an open-source Java Servlet Container developed by the Apache Software Foundation* Beretta Tomcat, a small .32 caliber pistol... . |
|
| Visual Studio Load Test | Microsoft Microsoft Microsoft Corporation is an American public multinational corporation headquartered in Redmond, Washington, USA that develops, manufactures, licenses, and supports a wide range of products and services predominantly related to computing through its various product divisions... |
Visual Studio includes a load test tool which enables a developer to execute a variety of tests (web, unit etc...) with a combination of configurations to simulate real user load. |
Mechanical load testing
The purpose of a mechanical load test is to verify that all the component parts of a structure including materials, base-fixings are fit for task and loading it is designed for.The Supply of Machinery (Safety) Regulation 1992 UK state that load testing is undertaken before the equipment is put into service for the first time.
Load testing can be either Performance,Static or Dynamic.
Performance testing is when the stated safe working load (SWL) for a configuration is used to determine that the item performs to the manufactures specification. If an item fails this test then any further tests are pointless.
Static testing is when a load at a factor above the SWL is applied. The item is not operated through all configurations as it is not a requirement of this test.
Dynamic testing is when a load at a factor above the SWL is applied. The item is then operated fully through all configurations and motions. Care must be taken during this test as there is a great risk of catastrophic failure if incorrectly carried out.
The design criteria, relevant legislation or the Competent Person will dictate what test is required.
Under the Lifting Operations and Lifting Equipment Regulations 1998 UK load testing after the initial test is required if a major component is replaced, if the item is moved from one location to another or as dictated by the Competent Person
The loads required for a test are stipulated by the item under test, but here are a few to be aware off.
Powered lifting equipment Static test to 1.25 SWL and dynamic test to 1.1 SWL.
Manual lifting equipment Static test to 1.5 SWL
For lifting accessories.
2 SWL for items up to 30 tonne capacity.
1.5 SWL for items above 30 tonne capacity.
1 SWL for items above 100 tonnes.
Car charging system
A load test can be used to evaluate the health of a car's batteryBattery (electricity)
An electrical battery is one or more electrochemical cells that convert stored chemical energy into electrical energy. Since the invention of the first battery in 1800 by Alessandro Volta and especially since the technically improved Daniell cell in 1836, batteries have become a common power...
. The tester consists of a large resistor that has a resistance similar to a car's starter motor and a meter to read the battery's output voltage both in the unloaded and loaded state. When the tester is used, the battery's open circuit voltage is checked first. If the open circuit voltage is below spec (12.6 volts for a fully charged battery), the battery is charged first. After reading the battery's open circuit voltage, the load is applied. When applied, it draws approximately the same current the car's starter motor would draw during cranking. Based on the specified cold cranking amperes of the battery, if the voltage under load falls below a certain point, the battery is bad. Load tests are also used on running cars to check the output of the car's alternator
Alternator
An alternator is an electromechanical device that converts mechanical energy to electrical energy in the form of alternating current.Most alternators use a rotating magnetic field but linear alternators are occasionally used...
.
See also
- soapUISoapuiSoapUI is an open source web service testing tool for service-oriented architectures . Its functionality covers web service inspection, invoking, development, simulation and mocking, functional testing, load and compliance testing. Eviware also released a Pro version as a commercial product which...
- Software testingSoftware testingSoftware testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
- Web server benchmarkingWeb server benchmarkingWeb server benchmarking is the process of estimating a web server performance in order to find if the server can serve sufficiently high workload.- Key parameters :The performance is usually measured in terms of:...
- Web load testing
External links
- Modeling the Real World for Load Testing Web Sites by Steven Splaine