Stress testing
Encyclopedia
Stress testing is a form of testing that is used to determine the stability of a given system or entity. It involves testing beyond normal operational capacity, often to a breaking point, in order to observe the results. Stress testing may have a more specific meaning in certain industries, such as fatigue testing
for materials.
, a system stress test refers to tests that put a greater emphasis on robustness, availability
, and error handling under a heavy load, rather than on what would be considered correct behavior under normal circumstances. In particular, the goals of such tests may be to ensure the software does not crash
in conditions of insufficient computational resources (such as memory or disk space
), unusually high concurrency
, or denial of service attacks.
Examples:
Stress testing may be contrasted with load testing:
s often test items under expected stress or even under accelerated stress. The goal is to determine the operating life of the item or to determine modes of failure.
Stress testing, in general, should put the hardware under exaggerated levels of stress in order to ensure stability when used in a normal environment.
, such as in overclocking
, underclocking
, overvolting, and undervolting, it may be necessary to verify if the new parameters (usually CPU core voltage
and frequency
) are suitable for heavy CPU load
s. This is done by running a CPU-intensive program (usually Prime95
) for extended periods of time, to test whether the computer hangs
or crashes
. CPU stress testing is also referred to as torture testing. Software that is suitable for torture testing should typically run instructions that utilise the entire chip rather than only a few of its units.
Stress testing a CPU over the course of 24 hours at 100% load is, in most cases, sufficient enough to determine that the CPU will function correctly in normal usage scenarios, where CPU usage fluctuates at low levels (50% and under), such as on a desktop computer.
. They may test the instrument under, for example, the following stresses:
This type of analysis has become increasingly widespread, and has been taken up by various governmental bodies (such as the FSA
in the UK) as a regulatory requirement on certain financial institutions to ensure adequate capital allocation levels to cover potential losses incurred during extreme, but plausible, events. This emphasis on adequate, risk adjusted determination of capital has been further enhanced by modifications to banking regulations such as Basel II
. Stress testing models typically allow not only the testing of individual stressors, but also combinations of different events. There is also usually the ability to test the current exposure to a known historical scenario (such as the Russian debt default in 1998 or 9/11 attacks) to ensure the liquidity of the institution.
Stress testing reveals how well a portfolio is positioned in the event forecasts prove true. Stress testing also lends insight into a portfolio's vulnerabilities. Though extreme events are never certain, studying their performance implications strengthens understanding.
Defining stress tests
Stress testing defines a scenario and uses a specific algorithm to determine the expected impact on a portfolio's return should such a scenario occur. There are three types of scenarios:
In an exponentially weighted stress test, historical periods more like the defined scenario receive a more significant weighting in the predicted outcome. The defined decay rate lets the tester manipulate the relative importance of the most similar periods. In the standard stress test, each period is equally weighted.
Stress tests in payment and settlement systems
Another form of financial stress testing is the stress testing of financial infrastructure. As part of Central Banks' market infrastructure oversight functions, stress tests have been applied to payment and securities settlement systems. Since ultimately, the Banks need to meet their obligations in Central Bank money held in payment systems that are commonly operated or closely supervised by central banks
(e.g. CHAPS, FedWire, Target2, which are also referred to as large value payment systems)[, it is of great interest to monitor these systems' participants' (mainly banks) liquidity positions.
The amount of liquidity held by banks on their accounts can be a lot less (and usually is) than the total value of transferred payments during a day. The total amount of liquidity needed by banks to settle a given set of payments is dependent on the balancedness of the circulation of money from account to account (reciprocity of payments), the timing of payments and the netting procedures used. The inability of some participants to send payments can cause severe falls in settlement ratios of payments. The failure of one participant to send payments can have negative contagion effects on other participants' liquidity positions and their potential to send payments.
By using stress tests it is possible to evaluate the short term effects of events such as bank failures or technical communication breakdowns that lead to the inability of chosen participants to send payments. These effects can be viewed as direct effects on the participant, but also as systemic contagion effects. How hard the other participants will be hit by a chosen failure scenario will be dependent on the available collateral and initial liquidity of participants, and their potential to bring in more liquidity. Stress test conducted on payment systems help to evaluate the short term adequacy and sufficiency of the prevailing liquidity levels and buffers of banks, and the contingency measures of the studied payment systems.
After WENRA released the first proposal of a stress test, there has been criticism that the stress test was not going to be strict enough.
Fatigue (material)
'In materials science, fatigue is the progressive and localized structural damage that occurs when a material is subjected to cyclic loading. The nominal maximum stress values are less than the ultimate tensile stress limit, and may be below the yield stress limit of the material.Fatigue occurs...
for materials.
Computer software
In software testingSoftware testing
Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
, a system stress test refers to tests that put a greater emphasis on robustness, availability
Availability
In telecommunications and reliability theory, the term availability has the following meanings:* The degree to which a system, subsystem, or equipment is in a specified operable and committable state at the start of a mission, when the mission is called for at an unknown, i.e., a random, time...
, and error handling under a heavy load, rather than on what would be considered correct behavior under normal circumstances. In particular, the goals of such tests may be to ensure the software does not crash
Crash (computing)
A crash in computing is a condition where a computer or a program, either an application or part of the operating system, ceases to function properly, often exiting after encountering errors. Often the offending program may appear to freeze or hang until a crash reporting service documents...
in conditions of insufficient computational resources (such as memory or disk space
Disk storage
Disk storage or disc storage is a general category of storage mechanisms, in which data are digitally recorded by various electronic, magnetic, optical, or mechanical methods on a surface layer deposited of one or more planar, round and rotating disks...
), unusually high concurrency
Concurrency (computer science)
In computer science, concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other...
, or denial of service attacks.
Examples:
- A web serverWeb serverWeb server can refer to either the hardware or the software that helps to deliver content that can be accessed through the Internet....
may be stress tested using scriptsShell scriptA shell script is a script written for the shell, or command line interpreter, of an operating system. It is often considered a simple domain-specific programming language...
, botsInternet botInternet bots, also known as web robots, WWW robots or simply bots, are software applications that run automated tasks over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human alone...
, and various denial of service tools to observe the performance of a web site during peak loads.
Stress testing may be contrasted with load testing:
- Load testing examines the entire environment and database, while measuring the response time, whereas stress testing focuses on identified transactions, pushing to a level so as to break transactions or systems.
- During stress testing, if transactions are selectively stressed, the database may not experience much load, but the transactions are heavily stressed. On the other hand, during load testing the database experiences a heavy load, while some transactions may not be stressed.
- System stress testing, also known as stress testing, is loading the concurrent users over and beyond the level that the system can handle, so it breaks at the weakest link within the entire system.
Hardware
Reliability engineerReliability engineering
Reliability engineering is an engineering field, that deals with the study, evaluation, and life-cycle management of reliability: the ability of a system or component to perform its required functions under stated conditions for a specified period of time. It is often measured as a probability of...
s often test items under expected stress or even under accelerated stress. The goal is to determine the operating life of the item or to determine modes of failure.
Stress testing, in general, should put the hardware under exaggerated levels of stress in order to ensure stability when used in a normal environment.
Computer processors
When modifying the operating parameters of a CPUCentral processing unit
The central processing unit is the portion of a computer system that carries out the instructions of a computer program, to perform the basic arithmetical, logical, and input/output operations of the system. The CPU plays a role somewhat analogous to the brain in the computer. The term has been in...
, such as in overclocking
Overclocking
Overclocking is the process of operating a computer component at a higher clock rate than it was designed for or was specified by the manufacturer, but some manufacturers purposely underclock their components to improve battery life. Many people just overclock or 'rightclock' their hardware to...
, underclocking
Underclocking
Underclocking, also known as downclocking, is the practice of modifying a synchronous circuit's timing settings to run at a lower clock rate than it was specified to operate at. It may be said to be the computer equivalent of driving a car below the speed limit...
, overvolting, and undervolting, it may be necessary to verify if the new parameters (usually CPU core voltage
CPU core voltage
The CPU core voltage is the power supply voltage supplied to the CPU , GPU, or other device containing a processing core...
and frequency
Clock rate
The clock rate typically refers to the frequency that a CPU is running at.For example, a crystal oscillator frequency reference typically is synonymous with a fixed sinusoidal waveform, a clock rate is that frequency reference translated by electronic circuitry into a corresponding square wave...
) are suitable for heavy CPU load
Load (computing)
In UNIX computing, the system load is a measure of the amount of work that a computer system performs. The load average represents the average system load over a period of time...
s. This is done by running a CPU-intensive program (usually Prime95
Prime95
Prime95 is the name of the Microsoft Windows-based software application written by George Woltman that is used by GIMPS, a distributed computing project dedicated to finding new Mersenne prime numbers....
) for extended periods of time, to test whether the computer hangs
Hang (computing)
In computing, a hang or freeze occurs when either a single computer program, or the whole system ceases to respond to inputs. In the most commonly encountered scenario, a workstation with a graphical user interface, all windows belonging to the frozen program become static, and though the mouse...
or crashes
Crash (computing)
A crash in computing is a condition where a computer or a program, either an application or part of the operating system, ceases to function properly, often exiting after encountering errors. Often the offending program may appear to freeze or hang until a crash reporting service documents...
. CPU stress testing is also referred to as torture testing. Software that is suitable for torture testing should typically run instructions that utilise the entire chip rather than only a few of its units.
Stress testing a CPU over the course of 24 hours at 100% load is, in most cases, sufficient enough to determine that the CPU will function correctly in normal usage scenarios, where CPU usage fluctuates at low levels (50% and under), such as on a desktop computer.
Financial sector
Instead of doing financial projection on a "best estimate" basis, a company may do stress testing where they look at how robust a financial instrument is in certain crashes, a form of scenario analysisScenario analysis
Scenario analysis is a process of analyzing possible future events by considering alternative possible outcomes . Thus, the scenario analysis, which is a main method of projections, does not try to show one exact picture of the future. Instead, it presents consciously several alternative future...
. They may test the instrument under, for example, the following stresses:
- What happens if equity markets crash by more than x% this year?
- What happens if interest rates go up by at least y%?
- What if half the instruments in the portfolio terminate their contracts in the fifth year?
- What happens if oil prices rise by 200%?
This type of analysis has become increasingly widespread, and has been taken up by various governmental bodies (such as the FSA
Financial Services Authority
The Financial Services Authority is a quasi-judicial body responsible for the regulation of the financial services industry in the United Kingdom. Its board is appointed by the Treasury and the organisation is structured as a company limited by guarantee and owned by the UK government. Its main...
in the UK) as a regulatory requirement on certain financial institutions to ensure adequate capital allocation levels to cover potential losses incurred during extreme, but plausible, events. This emphasis on adequate, risk adjusted determination of capital has been further enhanced by modifications to banking regulations such as Basel II
Basel II
Basel II is the second of the Basel Accords, which are recommendations on banking laws and regulations issued by the Basel Committee on Banking Supervision...
. Stress testing models typically allow not only the testing of individual stressors, but also combinations of different events. There is also usually the ability to test the current exposure to a known historical scenario (such as the Russian debt default in 1998 or 9/11 attacks) to ensure the liquidity of the institution.
Stress testing reveals how well a portfolio is positioned in the event forecasts prove true. Stress testing also lends insight into a portfolio's vulnerabilities. Though extreme events are never certain, studying their performance implications strengthens understanding.
Defining stress tests
Stress testing defines a scenario and uses a specific algorithm to determine the expected impact on a portfolio's return should such a scenario occur. There are three types of scenarios:
- Extreme event: hypothesize the portfolio's return given the recurrence of a historical event. Current positions and risk exposures are combined with the historical factor returns.
- Risk factor shock: shock any factor in the chosen risk model by a user-specified amount. The factor exposures remain unchanged, while the covariance matrix is used to adjust the factor returns based on their correlation with the shocked factor.
- External factor shock: instead of a risk factor, shock any index, macro-economic series (e.g., oil prices), or custom series (e.g., exchange rates). Using regression analysis, new factor returns are estimated as a result of the shock.
In an exponentially weighted stress test, historical periods more like the defined scenario receive a more significant weighting in the predicted outcome. The defined decay rate lets the tester manipulate the relative importance of the most similar periods. In the standard stress test, each period is equally weighted.
Stress tests in payment and settlement systems
Another form of financial stress testing is the stress testing of financial infrastructure. As part of Central Banks' market infrastructure oversight functions, stress tests have been applied to payment and securities settlement systems. Since ultimately, the Banks need to meet their obligations in Central Bank money held in payment systems that are commonly operated or closely supervised by central banks
(e.g. CHAPS, FedWire, Target2, which are also referred to as large value payment systems)[, it is of great interest to monitor these systems' participants' (mainly banks) liquidity positions.
The amount of liquidity held by banks on their accounts can be a lot less (and usually is) than the total value of transferred payments during a day. The total amount of liquidity needed by banks to settle a given set of payments is dependent on the balancedness of the circulation of money from account to account (reciprocity of payments), the timing of payments and the netting procedures used. The inability of some participants to send payments can cause severe falls in settlement ratios of payments. The failure of one participant to send payments can have negative contagion effects on other participants' liquidity positions and their potential to send payments.
By using stress tests it is possible to evaluate the short term effects of events such as bank failures or technical communication breakdowns that lead to the inability of chosen participants to send payments. These effects can be viewed as direct effects on the participant, but also as systemic contagion effects. How hard the other participants will be hit by a chosen failure scenario will be dependent on the available collateral and initial liquidity of participants, and their potential to bring in more liquidity. Stress test conducted on payment systems help to evaluate the short term adequacy and sufficiency of the prevailing liquidity levels and buffers of banks, and the contingency measures of the studied payment systems.
Nuclear power plants
After the 2011 earthquake in Japan the European Commission decided that all nuclear power plants in Europe have to undergo a stress test to verify they still comply with the highest safety standards.After WENRA released the first proposal of a stress test, there has been criticism that the stress test was not going to be strict enough.
See also
- Burn-in
- Destructive testingDestructive testingIn destructive testing, tests are carried out to the specimen's failure, in order to understand a specimen's structural performance or material behaviour under different loads...
- Load and performance test tools
- Black box testingBlack box testingBlack-box testing is a method of software testing that tests the functionality of an application as opposed to its internal structures or workings . Specific knowledge of the application's code/internal structure and programming knowledge in general is not required...
- Load testingLoad testingLoad testing is the process of putting demand on a system or device and measuring its response. Load testing is performed to determine a system’s behavior under both normal and anticipated peak load conditions. It helps to identify the maximum operating capacity of an application as well as any...
- Performance testingPerformance testingIn software engineering, performance testing is in general testing performed to determine how a system performs in terms of responsiveness and stability under a particular workload...
- Scenario analysisScenario analysisScenario analysis is a process of analyzing possible future events by considering alternative possible outcomes . Thus, the scenario analysis, which is a main method of projections, does not try to show one exact picture of the future. Instead, it presents consciously several alternative future...
- SimulationSimulationSimulation is the imitation of some real thing available, state of affairs, or process. The act of simulating something generally entails representing certain key characteristics or behaviours of a selected physical or abstract system....
- Software testingSoftware testingSoftware testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. Software testing can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software...
- White box testingWhite box testingWhite-box testing is a method of testing software that tests internal structures or workings of an application, as opposed to its functionality...
- Technischer ÜberwachungsvereinTechnischer ÜberwachungsvereinTÜVs are German organizations that work to validate the safety of products of all kinds to protect humans and the environment against hazards. As independent consultants, they examine plants, motor vehicles, energy installations, devices and products TÜVs (short for Technischer...
(TÜV) - product testing and certification - Concurrency testing using the CHESS model checkerCHESS model checkerCHESS is a software model checker for finding errors/heisenbugs in multithreaded software by systematic exploration of thread schedules. It finds errors, such as data-races, deadlocks, livelocks, and data-corruption induced access violations, that are extremely hard to find with current testing tools...
- JinxJinx DebuggerJinx is a concurrency debugger that deterministically controls the interleaving of workloads across processor cores, focusing on shared memory interactions. Using this deterministic approach, Jinx is able to increase the frequency of occurrence of elusive shared memory bugs, sometimes called...
automates stress testing by automatically exploring unlikely execution scenarios. - Highly Accelerated Life TestHighly Accelerated Life TestA highly accelerated life test , is a stress testing methodology for accelerating product reliability during the engineering development process. It is commonly applied to electronic equipment and is performed to identify and thus help resolve design weaknesses in newly-developed equipment...