Sensitivity analysis - AbsoluteAstronomy.com

Sensitivity analysis is the study of how the variation (uncertainty) in the output of a statistical model

Statistical model

A statistical model is a formalization of relationships between variables in the form of mathematical equations. A statistical model describes how one or more random variables are related to one or more random variables. The model is statistical as the variables are not deterministically but...

can be attributed to different variations in the inputs of the model. Put another way, it is a technique for systematically changing variables

Variable (mathematics)

In mathematics, a variable is a value that may change within the scope of a given problem or set of operations. In contrast, a constant is a value that remains unchanged, though often unknown or undetermined. The concepts of constants and variables are fundamental to many areas of mathematics and...

in a model to determine the effects of such changes.

In any budgeting process there are always variables that are uncertain. Future tax rates, interest rates, inflation rates, headcount, operating expenses and other variables may not be known with great precision. Sensitivity analysis answers the question, "if these variables deviate from expectations, what will the effect be (on the business, model, system, or whatever is being analyzed)?"

In more general terms uncertainty and sensitivity analysis investigate the robustness of a study when the study includes some form of statistical modelling. Sensitivity analysis can be useful to computer modelers for a range of purposes, including:

Support decision making or the development of recommendations for decision makers (e.g. testing the robustness of a result);
Enhancing communication from modellers to decision makers (e.g. by making recommendations more credible, understandable, compelling or persuasive);
Increased understanding or quantification of the system (e.g. understanding relationships between input and output variables); and
Model development (e.g. searching for errors in the model).

Overview

Statistical problems met in social, economic or natural sciences may entail the use of statistical models, which generally do not lend themselves to a straightforward understanding of the relationship between input factors (what goes into the model) and output (the model’s dependent variables). Such an appreciation, i.e. the understanding of how the model behaves in response to changes in its inputs, is of fundamental importance to ensure a correct use of the models.

A statistical model

Statistical model

is defined by a series of equations, input factors, parameters, and variables aimed at characterizing the process being investigated.

Input is subject to many sources of uncertainty including errors of measurement

Measurement

Measurement is the process or the result of determining the ratio of a physical quantity, such as a length, time, temperature etc., to a unit of measurement, such as the metre, second or degree Celsius...

, absence of information and poor or partial understanding of the driving forces and mechanisms. This uncertainty imposes a limit on our confidence

Confidence

Confidence is generally described as a state of being certain either that a hypothesis or prediction is correct or that a chosen course of action is the best or most effective. Self-confidence is having confidence in oneself. Arrogance or hubris in this comparison, is having unmerited...

in the response or output of the model. Further, models may have to cope with the natural intrinsic variability of the system, such as the occurrence of stochastic

Stochastic

Stochastic refers to systems whose behaviour is intrinsically non-deterministic. A stochastic process is one whose behavior is non-deterministic, in that a system's subsequent state is determined both by the process's predictable actions and by a random element. However, according to M. Kac and E...

events.

Good modeling practice requires that the modeler provides an evaluation of the confidence in the model, possibly assessing the uncertainties associated with the modeling process and with the outcome of the model itself. Uncertainty

Uncertainty

Uncertainty is a term used in subtly different ways in a number of fields, including physics, philosophy, statistics, economics, finance, insurance, psychology, sociology, engineering, and information science...

and Sensitivity Analysis offer valid tools for characterizing the uncertainty associated with a model. Uncertainty analysis

Uncertainty analysis

Calibrated parameter does not necessarily represents reality, as reality is much more complex. Any any prediction has its own complexities of reality that cannot be represented uniquely in the calibrated model; tehrefore, there is a potential error. Such error must be accounted for when making...

(UA) quantifies the uncertainty in the outcome of a model. Sensitivity Analysis has the complementary role of ordering by importance the strength and relevance of the inputs in determining the variation in the output.

In models involving many input variables sensitivity analysis is an essential ingredient of model building and quality assurance. National and international agencies involved in impact assessment studies have included sections devoted to sensitivity analysis in their guidelines. Examples are the European Commission

European Commission

The European Commission is the executive body of the European Union. The body is responsible for proposing legislation, implementing decisions, upholding the Union's treaties and the general day-to-day running of the Union....

, the White House Office of Management and Budget, the Intergovernmental Panel on Climate Change

Intergovernmental Panel on Climate Change

The Intergovernmental Panel on Climate Change is a scientific intergovernmental body which provides comprehensive assessments of current scientific, technical and socio-economic information worldwide about the risk of climate change caused by human activity, its potential environmental and...

and US Environmental Protection Agency.

Sometimes a sensitivity analysis may reveal surprising insights about the subject of interest. For instance, the field of multi-criteria decision making (MCDM) studies (among other topics) the problem of how to select the best alternative among a number of competing alternatives. This is an important task in decision making

Decision making

Decision making can be regarded as the mental processes resulting in the selection of a course of action among several alternative scenarios. Every decision making process produces a final choice. The output can be an action or an opinion of choice.- Overview :Human performance in decision terms...

. In such a setting each alternative is described in terms of a set of evaluative criteria. These criteria are associated with weights of importance. Intuitively, one may think that the larger the weight for a criterion is, the more critical that criterion should be. However, this may not be the case. It is important to distinguish here the notion of criticality with that of importance. By critical, we mean that a criterion with small change (as a percentage) in its weight, may cause a significant change of the final solution. It is possible criteria with rather small weights of importance (i.e., ones that are not so important in that respect) to be much more critical in a given situation than ones with larger weights. That is, a sensitivity analysis may shed light into issues not anticipated at the beginning of a study. This, in turn, may dramatically improve the effectiveness of the initial study and assist in the successful implementation of the final solution.

Methodology

There are several possible procedures to perform uncertainty (UA) and sensitivity analysis (SA). Important classes of methods are:

Local methods, such as the simple derivative of the output with respect to an input factor :,

where the subscript

indicates that the derivative is taken at some fixed point in the space of the input (hence the 'local' in the name of the class). Adjoint modelling and Automated Differentiation are methods in this class.

A sampling
Sampling (statistics)
In statistics and survey methodology, sampling is concerned with the selection of a subset of individuals from within a population to estimate characteristics of the whole population....

-based sensitivity is one in which the model is executed repeatedly for combinations of values sampled from the distribution
Probability distribution
In probability theory, a probability mass, probability density, or probability distribution is a function that describes the probability of a random variable taking certain values....

(assumed known) of the input factors. Once the sample is generated, several strategies (including simple input-output scatterplots) can be used to derive sensitivity measures for the factors.
Methods based on emulators (e.g. Bayesian). With these methods the value of the output , or directly the value of the sensitivity measure of a factor , is treated as a stochastic process and estimated from the available computer-generated data points. This is useful when the computer program which describes the model is expensive to run.
Screening methods. This is a particular instance of sampling based methods. The objective here is to estimate a few active factors in models with many factors. One of the most commonly used screening method is the elementary effect method
Elementary effects method
The elementary effects method is the most used screening method in sensitivity analysis. It is applied to identify non-influential inputs for a computationally costly mathematical model or for a model with a large number of inputs, where the costs of estimating other sensitivity analysis measures...

.
Variance based methods. Here the unconditional variance of is decomposed into terms due to individual factors plus terms due to interaction among factors. Full variance decompositions are only meaningful when the input factors are independent from one another.
High Dimensional Model Representations (HDMR). The term is due to H. Rabitz and include as a particular case the variance based methods. In HDMR the output is expressed as a linear combination of terms of increasing dimensionality.
Methods based on Monte Carlo filtering. These are also sampling-based and the objective here is to identify regions in the space of the input factors corresponding to particular values (e.g. high or low) of the output.

Often (e.g. in sampling-based methods) UA and SA are performed jointly by executing the model repeatedly for combination of factor values sampled with some probability distribution

Probability distribution

In probability theory, a probability mass, probability density, or probability distribution is a function that describes the probability of a random variable taking certain values....

. The following steps can be listed:

Specify the target function
Function (mathematics)
In mathematics, a function associates one quantity, the argument of the function, also known as the input, with another quantity, the value of the function, also known as the output. A function assigns exactly one output to each input. The argument and the value may be real numbers, but they can...

of interest.
- It is easier to communicate the results of a sensitivity analysis when the target of interest has a direct relation to the problem tackled by the model.
Assign a probability density function
Probability density function
In probability theory, a probability density function , or density of a continuous random variable is a function that describes the relative likelihood for this random variable to occur at a given point. The probability for the random variable to fall within a particular region is given by the...

to the selected factors.
- When this involves eliciting experts' opinion this is the most expensive and time consuming part of the analysis.
Generate a matrix
Matrix (mathematics)
In mathematics, a matrix is a rectangular array of numbers, symbols, or expressions. The individual items in a matrix are called its elements or entries. An example of a matrix with six elements isMatrices of the same size can be added or subtracted element by element...

of inputs with that distribution(s) through an appropriate design.
- As in experimental design, a good design for numerical experiments should give a maximum of effects with a minimum of computed points.
Evaluate the model and compute the distribution of the target function.
- This is the computer-time intensive step.
Select a method for assessing the influence or relative importance of each input factor on the target function.
- This depends upon the purpose of the analysis, e.g. model simplification, factor prioritization, uncertainty reduction, etc.

Assumptions vs. inferences

In uncertainty and sensitivity analysis there is a crucial trade off between how scrupulous an analyst is in exploring the input assumptions and how wide the resulting inference

Inference

Inference is the act or process of deriving logical conclusions from premises known or assumed to be true. The conclusion drawn is also called an idiomatic. The laws of valid inference are studied in the field of logic.Human inference Inference is the act or process of deriving logical conclusions...

may be. The point is well illustrated by the econometrician Edward E. Leamer (1990) :

I have proposed a form of organized sensitivity analysis that I call ‘global sensitivity analysis’ in which a neighborhood of alternative assumptions is selected and the corresponding interval of inferences is identified. Conclusions are judged to be sturdy only if the neighborhood of assumptions is wide enough to be credible and the corresponding interval of inferences is narrow enough to be useful.

Note Leamer’s emphasis is on the need for 'credibility' in the selection of assumptions. The easiest way to invalidate a model is to demonstrate that it is fragile with respect to the uncertainty in the assumptions or to show that its assumptions have not been taken 'wide enough'. The same concept is expressed by Jerome R. Ravetz, for whom bad modeling is when uncertainties in inputs must be suppressed lest outputs become indeterminate.

Errors

In a sensitivity analysis, a Type I error is assessing as important a non-important factor and a Type II error is assessing as non-important an important factor. A Type III error corresponds to analysing the wrong problem, e.g. via an incorrect specification of the input uncertainties. Possible pitfalls in a sensitivity analysis are:

Unclear purpose of the analysis. Different statistical tests and measures are applied to the problem and different factors rankings are obtained. The test should instead be tailored to the purpose of the analysis, e.g. one uses Monte Carlo filtering if one is interested in which factors are most responsible for generating high/low values of the output.

Too many model outputs are considered. This may be acceptable for quality assurance of sub-models but should be avoided when presenting the results of the overall analysis.

Piecewise sensitivity. This is when one performs sensitivity analysis on one sub-model at a time. This approach is non conservative as it might overlook interactions among factors in different sub-models (Type II error).

The OAT paradox

In sensitivity analysis a common approach is that of changing one-factor-at-a-time (OAT), to see what effect this produces on the output. OAT customarily involves:

Moving one factor at a time and

Going back to the central/baseline point after each movement.

This appears a logical approach as any change observed in the output will unambiguously be due to the single factor changed. Furthermore by changing one factor at a time one can keep all other factors fixed to their central or baseline value. This increases the comparability of the results (all ‘effects’ are computed with reference to the same central point in space) and minimizes the chances of computer programme crashes, more likely when several input factors are changed simultaneously. The later occurrence is particularly annoying to modellers as in this case one does not know which factor's variation caused the model to crash.

The paradox is that this approach, apparently sound, is non-explorative, with exploration decreasing rapidly with the number of factors. With two factors, and hence in two dimensions, the OAT explores (partially) a circle instead of the full square (see figure). In this case one step along the abscissa moving from the origin, followed by a similar step along the ordinate—always moving from the origin, will leave us inside the circle and will never take us to the gray corners.
In k dimensions, the volume of the hyper-sphere included in (and tangent to) the unitary hyper-cube divided by that of the hyper-cube itself, goes rapidly to zero (e.g. it is less than 1% already for k = 10, see Figure). Note also that all OAT points are at most a distance one from the origin by design. Given that the diagonal of the hypercube is

dimensions, if the points are distributed randomly there will be points (in the corners) which are distant from the origin

. In ten dimensions there are

corners.

Latin hypercube sampling

Latin hypercube sampling

Latin hypercube sampling is a statistical method for generating a distribution of plausible collections of parameter values from a multidimensional distribution. The sampling method is often applied in uncertainty analysis....

is often used in contexts where researchers feel the assumption of independent-uncertainty is too strong and it is desirable to explore corners of the factor-space.

Related concepts

While uncertainty analysis studies the overall uncertainty

Uncertainty

in the conclusions of the study, sensitivity analysis tries to identify what source of uncertainty weights more on the study's conclusions. For example, several guidelines for modelling (see e.g. one from the US EPA) or for impact assessment

Impact assessment

Impact assessment is "a process aimed at structuring and supporting the development of policies. It identifies and assesses the problem at stake and the objectives pursued. It identifies the main options for achieving the objective and analyses their likely impacts in the economic, environmental...

(see one from the European Commission) prescribe sensitivity analysis as a tool to ensure the quality of the modelling/assessment.

The problem setting in sensitivity analysis has strong similarities with design of experiments

Design of experiments

In general usage, design of experiments or experimental design is the design of any information-gathering exercises where variation is present, whether under the full control of the experimenter or not. However, in statistics, these terms are usually used for controlled experiments...

. In design of experiments one studies the effect of some process or intervention (the 'treatment') on some objects (the 'experimental units'). In sensitivity analysis one looks at the effect of varying the inputs of a mathematical model on the output of the model itself. In both disciplines one strives to obtain information from the system with a minimum of physical or numerical experiments.

Applications

Sensitivity analysis can be used

To simplify models

To investigate the robustness of the model predictions

To play what-if analysis exploring the impact of varying input assumptions and scenarios

As an element of quality assurance (unexpected factors sensitivities may be associated to coding errors or misspecifications).

It provides as well information on:

Factors that mostly contribute to the output
Output
Output is the term denoting either an exit or changes which exit a system and which activate/modify a process. It is an abstract concept, used in the modeling, system design and system exploitation.-In control theory:...

variability

The region in the space of input factors for which the model output is either maximum or minimum or within pre-defined bounds (see Monte Carlo filtering above)

Optimal
Optimization (mathematics)
In mathematics, computational science, or management science, mathematical optimization refers to the selection of a best element from some set of available alternatives....

— or instability — regions within the space of factors for use in a subsequent calibration
Calibration
Calibration is a comparison between measurements – one of known magnitude or correctness made or set with one device and another measurement made in as similar a way as possible with a second device....

study

Interaction
Interaction (statistics)
In statistics, an interaction may arise when considering the relationship among three or more variables, and describes a situation in which the simultaneous influence of two variables on a third is not additive...

between factors

Sensitivity Analysis is common in physics and chemistry, in financial

FINANCIAL

FINANCIAL is the weekly English-language newspaper with offices in Tbilisi, Georgia and Kiev, Ukraine. Published by Intelligence Group LLC, FINANCIAL is focused on opinion leaders and top business decision-makers; It's about world’s largest companies, investing, careers, and small business. It is...

applications, risk analysis, signal processing

Signal processing

Signal processing is an area of systems engineering, electrical engineering and applied mathematics that deals with operations on or analysis of signals, in either discrete or continuous time...

, neural networks

Neural Networks

Neural Networks is the official journal of the three oldest societies dedicated to research in neural networks: International Neural Network Society, European Neural Network Society and Japanese Neural Network Society, published by Elsevier...

and any area where models are developed. Sensitivity analysis can also be used in model-based policy assessment studies . Sensitivity analysis can be used to assess the robustness of composite indicators , also known as indices, such as the Environmental Performance Index.

Environmental

Computer environmental models are increasingly used in a wide variety of studies and applications.
For example global climate model

Global climate model

A General Circulation Model is a mathematical model of the general circulation of a planetary atmosphere or ocean and based on the Navier–Stokes equations on a rotating sphere with thermodynamic terms for various energy sources . These equations are the basis for complex computer programs commonly...

are used for both short term weather forecasts and long term climate change

Climate change

Climate change is a significant and lasting change in the statistical distribution of weather patterns over periods ranging from decades to millions of years. It may be a change in average weather conditions or the distribution of events around that average...

.

Moreover, computer models are increasingly used for environmental decision making at a local scale, for example for assessing the impact of a waste water treatment plant on a river flow, or for assessing the behavior and life length of bio-filters for contaminated waste water.

In both cases sensitivity analysis may help understanding the contribution of the various sources of uncertainty to the model output uncertainty and system performance in general.
In these cases, depending on model complexity, different sampling strategies may be advisable and traditional sensitivity indexes have to be generalized to cover multivariate sensitivity analysis, heteroskedastic effects and correlated inputs.

Business

In a decision problem, the analyst may want to identify cost drivers as well as other quantities for which we need to acquire better knowledge in order to make an informed decision. On the other hand, some quantities have no influence on the predictions, so that we can save resources at no loss in accuracy by relaxing some of the conditions. See Corporate finance: Quantifying uncertainty.
Sensitivity analysis can help in a variety of other circumstances which can be handled by the settings illustrated below:

to identify critical assumptions or compare alternative model structures
guide future data collections
detect important criteria
optimize the tolerance of manufactured parts in terms of the uncertainty in the parameters
optimize resources allocation
model simplification or model lumping, etc.

However there are also some problems associated with sensitivity analysis in the business context:

Variables are often interdependent, which makes examining them each individually unrealistic, e.g.: changing one factor such as sales volume, will most likely affect other factors such as the selling price.
Often the assumptions upon which the analysis is based are made by using past experience/data which may not hold in the future.
Assigning a maximum and minimum (or optimistic and pessimistic) value is open to subjective interpretation. For instance one persons 'optimistic' forecast may be more conservative than that of another person performing a different part of the analysis. This sort of subjectivity can adversely affect the accuracy and overall objectivity of the analysis.

In modern econometrics the use of sensitivity analysis to anticipate criticism is the subject of one of the ten commandments of applied econometrics (from Kennedy, 2007 ):

Thou shall confess in the presence of sensitivity. Corollary: Thou shall anticipate criticism [···] When reporting a sensitivity analysis, researchers should explain fully their specification search so that the readers can judge for themselves how the results may have been affected. This is basically an ‘honesty is the best policy’ approach, advocated by Leamer, (1978).

Chemical kinetics

With the accumulation of knowledge about kinetic mechanisms under investigation and with the advance of power of modern computing technologies, detailed complex kinetic models are increasingly used as predictive tools and as aids for understanding the underlying phenomena. A kinetic model is usually described by a set of differential equations representing the concentration-time relationship. Sensitivity analysis has been proven to be a powerful tool to investigate a complex kinetic model.

Kinetic parameters are frequently determined from experimental data via nonlinear estimation. Sensitivity analysis can be used for optimal experimental design, e.g. determining initial conditions, measurement positions, and sampling time, to generate informative data which are critical to estimation accuracy. A great number of parameters in a complex model can be candidates for estimation but not all are estimable. Sensitivity analysis can be used to identify the influential parameters which can be determined from available data while screening out the unimportant ones. Sensitivity analysis can also be used to identify the redundant species and reactions allowing model reduction.

In meta-analysis

In a meta analysis, a sensitivity analysis tests if the results are sensitive to restrictions on the data included. Common examples are large trials only, higher quality trials only, and more recent trials only. If results are consistent it provides stronger evidence of an effect and of generalizability.

External links

Sixth Summer School on Sensitivity Analysis, Villa La Stella, Fiesole, Florence (ITALY) 14–17 September 2010
Sixth International Conference on Sensitivity Analysis of Model Output, Bocconi University, Milan (ITALY), 19–22 July 2010
Sensitivity analysis at the Joint Research Centre of the European Commission
- SimLab, the free software for global sensitivity analysis of the Joint Research Centre

The source of this article is wikipedia, the free encyclopedia. The text of this article is licensed under the GFDL.