Publication bias
Encyclopedia
Publication bias is the tendency of research
ers, editors, and pharmaceutical companies to handle the reporting of experimental results that are positive (i.e. showing a significant
finding) differently from results that are negative
(i.e. supporting the null hypothesis
) or inconclusive, leading to bias in the overall published literature. Such bias occurs despite the fact that studies with significant results do not appear to be superior to studies with a null result with respect to quality of design. It has been found that statistically significant results are three times more likely to be published than papers affirming a null result. It also has been found that the most common reason for non-publication is an investigator's declining to submit results for publication
(because of the investigator's loss of interest in the topic, the investigator's anticipation that others will not be interested in null results, etc.), underlining researchers' role in publication bias phenomena.
In an effort to decrease this problem, some prominent medical journal
s require registration of a trial before it commences so that unfavorable results are not withheld from publication. Several such registries
exist, but researchers are often unaware of them. In addition, attempts to identify unpublished studies have proved very difficult and often unsatisfactory. Another strategy suggested by a meta-analysis is caution in the use of small and non-randomised clinical trials because of their demonstrated high susceptibility to error and bias.
Outcome reporting bias occurs when several outcomes within a trial are measured but these are reported selectively depending on the strength and direction of those results. A related term that has been coined is HARKing (Hypothesizing After the Results are Known).
may be conducted but never reported, and those that are not reported may on average report different results from those that are reported. An extreme scenario is that a given null hypothesis
of interest is in fact true, i.e. the association being studied does not exist, but the 5% of studies that by chance show a statistically significant
result are published, while the remaining 95% where the null hypothesis was not rejected languish in researchers' file drawers. Even a small number of studies lost "in the file drawer" can result in a significant bias.
The term was coined by the psychologist Robert Rosenthal
in 1979.
may distort meta-analyses
and systematic review
s of large numbers of studies—on which evidence-based medicine
, for example, increasingly relies. The problem may be particularly significant when the research is sponsored by entities that may have a financial or ideological interest in achieving favorable results.
Those undertaking meta-analyses and systematic reviews need to take account of publication bias in the methods they use for identifying the studies to include in the review. Among other techniques to minimize the effects of publication bias, they may need to perform a thorough search for unpublished studies, and to use such analytical tools as a Begg's funnel plot
or Egger's plot to quantify the potential presence of publication bias. Tests for publications bias rely on the underlying theory that small studies with small sample size (and large variance) would be more prone to publication bias, while large-scale studies would be less likely to escape public knowledge and more likely to be published regardless of significance of findings. Thus, when overall estimates are plotted against the variance (sample size), a symmetrical funnel is usually formed in the absence of publication bias, while a skewed asymmetrical funnel is observed in presence of potential publication bias.
Extending the funnel plot, the "Trim and Fill" method has also been suggested as a method to infer the existence of unpublished hidden studies, as determined from a funnel plot, and subsequently correct the meta-analysis by imputing the presence of missing studies to yield an unbiased pooled estimate.
, negative papers are most likely to be suppressed:
Ioannidis further asserts that "claimed research findings may often be simply accurate measures of the prevailing bias".
, The Lancet
, Annals of Internal Medicine
, and JAMA
) announced that they would no longer publish results of drug research sponsored by pharmaceutical companies unless that research was registered in a public database from the start. Furthermore, some journals, e.g. Trials
, encourage publication of study protocols
in their journals.
Research
Research can be defined as the scientific search for knowledge, or as any systematic investigation, to establish novel facts, solve new or existing problems, prove new ideas, or develop new theories, usually using a scientific method...
ers, editors, and pharmaceutical companies to handle the reporting of experimental results that are positive (i.e. showing a significant
Statistical significance
In statistics, a result is called statistically significant if it is unlikely to have occurred by chance. The phrase test of significance was coined by Ronald Fisher....
finding) differently from results that are negative
Null result
In science, a null result is a result without the expected content: that is, the proposed result is absent. It is an experimental outcome which does not show an otherwise expected effect. This does not imply a result of zero or nothing, simply a result that does not support the hypothesis...
(i.e. supporting the null hypothesis
Null hypothesis
The practice of science involves formulating and testing hypotheses, assertions that are capable of being proven false using a test of observed data. The null hypothesis typically corresponds to a general or default position...
) or inconclusive, leading to bias in the overall published literature. Such bias occurs despite the fact that studies with significant results do not appear to be superior to studies with a null result with respect to quality of design. It has been found that statistically significant results are three times more likely to be published than papers affirming a null result. It also has been found that the most common reason for non-publication is an investigator's declining to submit results for publication
Non-response bias
Non-response bias occurs in statistical surveys if the answers of respondents differ from the potential answers of those who did not answer.- Example :...
(because of the investigator's loss of interest in the topic, the investigator's anticipation that others will not be interested in null results, etc.), underlining researchers' role in publication bias phenomena.
In an effort to decrease this problem, some prominent medical journal
Medical journal
A public health journal is a scientific journal devoted to the field of public health, including epidemiology, biostatistics, and health care . Public health journals, like most scientific journals, are peer-reviewed...
s require registration of a trial before it commences so that unfavorable results are not withheld from publication. Several such registries
Clinical trials registry
A clinical trials registry, sometimes abbreviated as a CTR, is an official platform and catalog for registering a clinical trial. Some countries require clinical trials being conducted in that country to be registered, other do not require it, but often strongly encourage it...
exist, but researchers are often unaware of them. In addition, attempts to identify unpublished studies have proved very difficult and often unsatisfactory. Another strategy suggested by a meta-analysis is caution in the use of small and non-randomised clinical trials because of their demonstrated high susceptibility to error and bias.
Definition
Positive results bias, a type of publication bias, occurs when authors are more likely to submit, or editors accept, positive than null (negative or inconclusive) results. A related term, "the file drawer problem", refers to the tendency for negative or inconclusive results to remain unpublished by their authors.Outcome reporting bias occurs when several outcomes within a trial are measured but these are reported selectively depending on the strength and direction of those results. A related term that has been coined is HARKing (Hypothesizing After the Results are Known).
The file drawer effect
The file drawer effect, or file drawer problem, is that many studies in a given area of researchResearch
Research can be defined as the scientific search for knowledge, or as any systematic investigation, to establish novel facts, solve new or existing problems, prove new ideas, or develop new theories, usually using a scientific method...
may be conducted but never reported, and those that are not reported may on average report different results from those that are reported. An extreme scenario is that a given null hypothesis
Null hypothesis
The practice of science involves formulating and testing hypotheses, assertions that are capable of being proven false using a test of observed data. The null hypothesis typically corresponds to a general or default position...
of interest is in fact true, i.e. the association being studied does not exist, but the 5% of studies that by chance show a statistically significant
Statistical significance
In statistics, a result is called statistically significant if it is unlikely to have occurred by chance. The phrase test of significance was coined by Ronald Fisher....
result are published, while the remaining 95% where the null hypothesis was not rejected languish in researchers' file drawers. Even a small number of studies lost "in the file drawer" can result in a significant bias.
The term was coined by the psychologist Robert Rosenthal
Robert Rosenthal (psychologist)
Robert Rosenthal is Distinguished Professor of Psychology at the University of California, Riverside. His interests include self-fulfilling prophecies, which he explored in a well-known study of the Pygmalion Effect: the effect of teachers' expectations on students.Rosenthal was born in Giessen,...
in 1979.
Effect on meta-analysis
The effect of this is that published studies may not be truly representative of all valid studies undertaken, and this biasBias
Bias is an inclination to present or hold a partial perspective at the expense of alternatives. Bias can come in many forms.-In judgement and decision making:...
may distort meta-analyses
Meta-analysis
In statistics, a meta-analysis combines the results of several studies that address a set of related research hypotheses. In its simplest form, this is normally by identification of a common measure of effect size, for which a weighted average might be the output of a meta-analyses. Here the...
and systematic review
Systematic review
A systematic review is a literature review focused on a research question that tries to identify, appraise, select and synthesize all high quality research evidence relevant to that question. Systematic reviews of high-quality randomized controlled trials are crucial to evidence-based medicine...
s of large numbers of studies—on which evidence-based medicine
Evidence-based medicine
Evidence-based medicine or evidence-based practice aims to apply the best available evidence gained from the scientific method to clinical decision making. It seeks to assess the strength of evidence of the risks and benefits of treatments and diagnostic tests...
, for example, increasingly relies. The problem may be particularly significant when the research is sponsored by entities that may have a financial or ideological interest in achieving favorable results.
Those undertaking meta-analyses and systematic reviews need to take account of publication bias in the methods they use for identifying the studies to include in the review. Among other techniques to minimize the effects of publication bias, they may need to perform a thorough search for unpublished studies, and to use such analytical tools as a Begg's funnel plot
Funnel plot
A funnel plot is a useful graph designed to check the existence of publication bias in systematic reviews and meta-analyses. It assumes that the largest studies will be near the average, and small studies will be spread on both sides of the average...
or Egger's plot to quantify the potential presence of publication bias. Tests for publications bias rely on the underlying theory that small studies with small sample size (and large variance) would be more prone to publication bias, while large-scale studies would be less likely to escape public knowledge and more likely to be published regardless of significance of findings. Thus, when overall estimates are plotted against the variance (sample size), a symmetrical funnel is usually formed in the absence of publication bias, while a skewed asymmetrical funnel is observed in presence of potential publication bias.
Extending the funnel plot, the "Trim and Fill" method has also been suggested as a method to infer the existence of unpublished hidden studies, as determined from a funnel plot, and subsequently correct the meta-analysis by imputing the presence of missing studies to yield an unbiased pooled estimate.
Examples of publication bias
One study compared Chinese and non-Chinese studies of gene-disease associations and found that "Chinese studies in general reported a stronger gene-disease association and more frequently a statistically significant result". One possible interpretation of this result is selective publication (publication bias).Risks
According to John IoannidisJohn P. A. Ioannidis
John P. A. Ioannidis is a professor and chairman at the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine as well as tenured adjunct professor at Tufts University School of Medicine and Professor of Medicine and Director of the Stanford Prevention Research Center at...
, negative papers are most likely to be suppressed:
- when the studies conducted in a field are smaller
- when effect sizes are smaller
- when there is a greater number and lesser preselection of tested relationships
- where there is greater flexibility in designs, definitions, outcomes, and analytical modes
- when there is greater financial and other interest and prejudice
- when more teams are involved in a scientific field in chase of statistical significance.
Ioannidis further asserts that "claimed research findings may often be simply accurate measures of the prevailing bias".
Remedies
Ioannidis' remedies include:- Better powered studies
- Low-bias meta-analysisMeta-analysisIn statistics, a meta-analysis combines the results of several studies that address a set of related research hypotheses. In its simplest form, this is normally by identification of a common measure of effect size, for which a weighted average might be the output of a meta-analyses. Here the...
- Large studies where they can be expected to give very definitive results or test major, general concepts
- Low-bias meta-analysis
- Enhanced research standards including
- Pre-registration of protocols (as for randomized trials)
- Registration or networking of data collections within fields (as in fields where researchers are expected to generate hypotheses after collecting data)
- Adopting from randomized controlled trials the principles of developing and adhering to a protocol.
- Considering, before running an experiment, what they believe the chances are that they are testing a true or non-true relationship.
- Properly assessing the false positive report probability based on the statistical power of the test
- Reconfirming (whenever ethically acceptable) established findings of "classic" studies, using large studies designed with minimal bias
Study registration
In September 2004, editors of several prominent medical journals (including the New England Journal of MedicineNew England Journal of Medicine
The New England Journal of Medicine is an English-language peer-reviewed medical journal published by the Massachusetts Medical Society. It describes itself as the oldest continuously published medical journal in the world.-History:...
, The Lancet
The Lancet
The Lancet is a weekly peer-reviewed general medical journal. It is one of the world's best known, oldest, and most respected general medical journals...
, Annals of Internal Medicine
Annals of Internal Medicine
Annals of Internal Medicine is an academic medical journal published by the American College of Physicians . It publishes research articles and reviews in the area of internal medicine. Its current editor is Christine Laine...
, and JAMA
Journal of the American Medical Association
The Journal of the American Medical Association is a weekly, peer-reviewed, medical journal, published by the American Medical Association. Beginning in July 2011, the editor in chief will be Howard C. Bauchner, vice chairman of pediatrics at Boston University’s School of Medicine, replacing ...
) announced that they would no longer publish results of drug research sponsored by pharmaceutical companies unless that research was registered in a public database from the start. Furthermore, some journals, e.g. Trials
Trials (journal)
Trials is an open access, peer reviewed online journal regarding performance and outcomes of randomized controlled trials. The journal is published by BioMed Central whose editors in chief are Doug Altman Curt Furberg, Jeremy Grimshaw, and Peter Rothwell...
, encourage publication of study protocols
Clinical trial protocol
A clinical trial protocol is a document that describes the objective, design, methodology, statistical considerations, and organization of a clinical trial...
in their journals.
External links
- The Truth Wears Off: Is there something wrong with the scientific method? -- Jonah Lehrer
- Register of clinical trials conducted in the US and around the world, maintained by the National Library of Medicine, Bethesda
- Skeptic's Dictionary: positive outcome bias.
- Skeptic's Dictionary: file-drawer effect.
- Journal of Negative Results in Biomedicine
- The All Results Journals
- Journal of Articles in Support of the Null Hypothesis
- Interesting article on 'the decline effect' and the role of publication bias in that.