Generalized p-value
Encyclopedia
In statistics
, a generalized p-value is an extended version of the classical p-value
, which except in a limited number of applications, provide only approximate solutions.
Conventional statistical methods do not provide exact solutions to many statistical problems such as those arise in mixed model
s and MANOVA, especially when the problem involves many nuisance parameters. As a result, practitioners often resort to approximate statistical methods or asymptotic statistical methods
that are valid only with large samples. With small samples, such methods often have poor performance . Use of approximate and asymptotic methods may lead to misleading conclusions or may fail to detect truly significant
results from experiment
s.
Tests based on generalized p-values are exact statistical methods in that they are based on exact probability statements. While conventional statistical methods do not provide exact solutions to such problems as testing variance components or ANOVA under unequal variances, exact tests for such problems can be obtained based on generalized p-values.
In order to overcome the shortcomings of the classical p-values, Tsui and Weerahandi extended the classical definition so that one can obtain exact solutions for such problems as the Behrens–Fisher problem and testing variance components. This is accomplished by allowing test variables to depend on observable random vectors as well as their observed values, as in the Bayesian treatment of the problem, but without having to treat constant parameters as random variables.
, and variance 
, suppose 
and 
are the sample mean and the sample variance. Inferences on all unknown parameters can be based on the distributional results
and
Now suppose we need to test the coefficient of variation,
. While the problem is not trivial with conventional p-values, the task can be easily accomplished based on the generalized test variable
where
is the observed value of 
and 
is the observed value of 
. Note that the distribution of 
and its observed value are both free of nuisance parameters. Therefore, a test of a hypothesis with a one-sided alternative such as 
can be based on the generalized p-value 
, a quantity that can be evaluated via Monte Carlo simulation or using the non-central t-distribution.
Statistics
Statistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
, a generalized p-value is an extended version of the classical p-value
P-value
In statistical significance testing, the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. One often "rejects the null hypothesis" when the p-value is less than the significance level α ,...
, which except in a limited number of applications, provide only approximate solutions.
Conventional statistical methods do not provide exact solutions to many statistical problems such as those arise in mixed model
Mixed model
A mixed model is a statistical model containing both fixed effects and random effects, that is mixed effects. These models are useful in a wide variety of disciplines in the physical, biological and social sciences....
s and MANOVA, especially when the problem involves many nuisance parameters. As a result, practitioners often resort to approximate statistical methods or asymptotic statistical methods
Asymptotic theory (statistics)
In statistics, asymptotic theory, or large sample theory, is a generic framework for assessment of properties of estimators and statistical tests...
that are valid only with large samples. With small samples, such methods often have poor performance . Use of approximate and asymptotic methods may lead to misleading conclusions or may fail to detect truly significant
Statistical significance
In statistics, a result is called statistically significant if it is unlikely to have occurred by chance. The phrase test of significance was coined by Ronald Fisher....
results from experiment
Experiment
An experiment is a methodical procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. Experiments vary greatly in their goal and scale, but always rely on repeatable procedure and logical analysis of the results...
s.
Tests based on generalized p-values are exact statistical methods in that they are based on exact probability statements. While conventional statistical methods do not provide exact solutions to such problems as testing variance components or ANOVA under unequal variances, exact tests for such problems can be obtained based on generalized p-values.
In order to overcome the shortcomings of the classical p-values, Tsui and Weerahandi extended the classical definition so that one can obtain exact solutions for such problems as the Behrens–Fisher problem and testing variance components. This is accomplished by allowing test variables to depend on observable random vectors as well as their observed values, as in the Bayesian treatment of the problem, but without having to treat constant parameters as random variables.
A simple case
To describe the idea of generalized p-values in a simple example, consider a situation of sampling from a normal population with mean, and variance , suppose and are the sample mean and the sample variance. Inferences on all unknown parameters can be based on the distributional resultsand
Now suppose we need to test the coefficient of variation,
. While the problem is not trivial with conventional p-values, the task can be easily accomplished based on the generalized test variablewhere
is the observed value of and is the observed value of . Note that the distribution of and its observed value are both free of nuisance parameters. Therefore, a test of a hypothesis with a one-sided alternative such as can be based on the generalized p-value , a quantity that can be evaluated via Monte Carlo simulation or using the non-central t-distribution.