Stein's lemma
Encyclopedia
Stein's lemma, named in honor of Charles Stein, is a theorem
of probability theory
that is of interest primarily because of its applications to statistical inference
— in particular, to James–Stein estimation and empirical Bayes method
s — and its applications to portfolio choice theory
.
with expectation
μ and variance
σ2. Further suppose g is a function for which the two expectations E(g(X) (X − μ) ) and E( g ′(X) ) both exist (the existence of the expectation of any random variable is equivalent to the finiteness of the expectation of its absolute value
). Then
In general, suppose X and Y are jointly normally distributed. Then
In order to prove the univariate version of this lemma, recall that the probability density function
for the normal distribution with expectation 0 and variance 1 is
and that for a normal distribution with expectation μ and variance σ2 is
Then use integration by parts
.
, that is, X has the density
Suppose either this density has support
, or it has support 
and 
as 
.
Then if g is any differentiable function such that
,
Extensions to elliptically-contoured distributions also exist.
Theorem
In mathematics, a theorem is a statement that has been proven on the basis of previously established statements, such as other theorems, and previously accepted statements, such as axioms...
of probability theory
Probability theory
Probability theory is the branch of mathematics concerned with analysis of random phenomena. The central objects of probability theory are random variables, stochastic processes, and events: mathematical abstractions of non-deterministic events or measured quantities that may either be single...
that is of interest primarily because of its applications to statistical inference
Statistical inference
In statistics, statistical inference is the process of drawing conclusions from data that are subject to random variation, for example, observational errors or sampling variation...
— in particular, to James–Stein estimation and empirical Bayes method
Empirical Bayes method
Empirical Bayes methods are procedures for statistical inference in which the prior distribution is estimated from the data. This approach stands in contrast to standardBayesian methods, for which the prior distribution is fixed before any data are observed...
s — and its applications to portfolio choice theory
Modern portfolio theory
Modern portfolio theory is a theory of investment which attempts to maximize portfolio expected return for a given amount of portfolio risk, or equivalently minimize risk for a given level of expected return, by carefully choosing the proportions of various assets...
.
Statement of the lemma
Suppose X is a normally distributed random variableRandom variable
In probability and statistics, a random variable or stochastic variable is, roughly speaking, a variable whose value results from a measurement on some type of random process. Formally, it is a function from a probability space, typically to the real numbers, which is measurable functionmeasurable...
with expectation
Expected value
In probability theory, the expected value of a random variable is the weighted average of all possible values that this random variable can take on...
μ and variance
Variance
In probability theory and statistics, the variance is a measure of how far a set of numbers is spread out. It is one of several descriptors of a probability distribution, describing how far the numbers lie from the mean . In particular, the variance is one of the moments of a distribution...
σ2. Further suppose g is a function for which the two expectations E(g(X) (X − μ) ) and E( g ′(X) ) both exist (the existence of the expectation of any random variable is equivalent to the finiteness of the expectation of its absolute value
Absolute value
In mathematics, the absolute value |a| of a real number a is the numerical value of a without regard to its sign. So, for example, the absolute value of 3 is 3, and the absolute value of -3 is also 3...
). Then
In general, suppose X and Y are jointly normally distributed. Then
In order to prove the univariate version of this lemma, recall that the probability density function
Probability density function
In probability theory, a probability density function , or density of a continuous random variable is a function that describes the relative likelihood for this random variable to occur at a given point. The probability for the random variable to fall within a particular region is given by the...
for the normal distribution with expectation 0 and variance 1 is
and that for a normal distribution with expectation μ and variance σ2 is
Then use integration by parts
Integration by parts
In calculus, and more generally in mathematical analysis, integration by parts is a rule that transforms the integral of products of functions into other integrals...
.
More general statement
Suppose X is in an exponential familyExponential family
In probability and statistics, an exponential family is an important class of probability distributions sharing a certain form, specified below. This special form is chosen for mathematical convenience, on account of some useful algebraic properties, as well as for generality, as exponential...
, that is, X has the density
Suppose either this density has support
, or it has support and as .Then if g is any differentiable function such that
,Extensions to elliptically-contoured distributions also exist.