Survivorship bias
Encyclopedia
Survivorship bias is the logical error of concentrating on the people or things that "survived" some process and inadvertently overlooking those that didn't because of their lack of visibility. This can lead to false conclusions in several different ways. The survivors may literally be people, as in a medical study, or could be companies or research subjects or applicants for a job, or anything that must make it past some selection process to be considered further.
Survivorship bias can lead to overly optimistic beliefs because failures are ignored, such as when companies that no longer exist are excluded from analyses of financial performance. It can also lead to the false belief that the successes in a group have some special property, rather than being just lucky. For example, if the three of the five students with the best college grades went to the same high school, that can lead one to believe that the high school must offer an excellent education. This could be true, but the question cannot be answered without looking at the grades of all the other students from that high school, not just the ones who "survived" the top-five selection process.
Survivorship bias is a type of selection bias
.
For example, a mutual fund
company's selection of funds today will include only those that are successful now. Many losing funds are closed and merged into other funds to hide poor performance. In theory, 90% of extant funds could truthfully claim to have performance in the first quartile of their peers if the peer group includes funds that have closed.
In 1996 Elton, Gruber, & Blake showed that survivorship bias is larger in the small-fund sector than in large mutual funds (presumably because small funds have a high probability of folding). They estimate the size of the bias across the U.S. mutual fund industry as 0.9% per annum, where the bias is defined and measured as:
Additionally, in quantitative back-testing of market performance or other characteristics, survivorship bias is the use of a current index membership set rather than using the actual constituent changes over time. Consider a backtest to 1990 to find the average performance (total return) of S&P 500 members who have paid dividends within the previous year. To use the current 500 members only and create an historical equity line of the total return of the companies that met the criteria, would be adding survivorship bias to the results. S&P maintains an index of healthy companies, removing companies that no longer meet their criteria as a representative of the large-cap U.S. stock market. Companies that had healthy growth on their way to inclusion in the S&P 500, would be counted as if they were in the index during that growth period, when they were not. Instead there may have been another company in the index that was losing market capitalization and was destined for the S&P 600 Small-cap Index, that was later removed and would not be counted in the results. Using the actual membership of the index, applying entry and exit dates to gain the appropriate return during inclusion in the index, would allow for a bias-free output.
in applications outside finance
, where studies on the remaining population are fallaciously compared with the historic average despite the survivors having unusual properties. Mostly, the unusual property in question is a track record of success (like the successful funds).
For example, the parapsychology
researcher Joseph Banks Rhine
believed he had identified the few individuals from hundreds of potential subjects who had powers of ESP
. His calculations were based on the improbability of these few subjects guessing the Zener cards shown to a partner by chance.
A major criticism which surfaced against his calculations was the possibility of unconscious survivor bias in subject selections. He was accused of failing to take into account the large effective size of his sample (all the people he didn't choose as 'strong telepaths' because they failed at an earlier testing stage). Had he done this he might have seen that from the large sample, one or two individuals would probably achieve the track record of success he had found purely by chance. (Similarly, many investor
s believe that chance is the main reason that most successful fund managers have the track records they do.)
Writing about the Rhine case
, Martin Gardner
explained that he didn't think the experimenters had made such obvious mistakes out of statistical naiveté, but as a result of subtly disregarding some poor subjects. He said that without trickery of any kind, there would always be some people who had improbable success, if a large enough sample were taken. To illustrate this, he speculates about what would happen if one hundred professors of psychology
read Rhine's work and decided to make their own tests; he said that survivor bias would winnow out the typical failed experiments, but encourage the lucky successes to continue testing. He thought that the common null hypothesis
(of no result) wouldn't be reported, but:
He concludes:
If enough scientist
s study a phenomenon, some will find statistically significant results by chance, and these are the experiments submitted for publication. To combat this, some editors now call for the submission of 'negative' scientific findings, where "nothing happened."
Survivorship bias is one of the issues discussed in the provocative 2005 paper "Why Most Published Research Findings Are False."
Survivorship bias can lead to overly optimistic beliefs because failures are ignored, such as when companies that no longer exist are excluded from analyses of financial performance. It can also lead to the false belief that the successes in a group have some special property, rather than being just lucky. For example, if the three of the five students with the best college grades went to the same high school, that can lead one to believe that the high school must offer an excellent education. This could be true, but the question cannot be answered without looking at the grades of all the other students from that high school, not just the ones who "survived" the top-five selection process.
Survivorship bias is a type of selection bias
Selection bias
Selection bias is a statistical bias in which there is an error in choosing the individuals or groups to take part in a scientific study. It is sometimes referred to as the selection effect. The term "selection bias" most often refers to the distortion of a statistical analysis, resulting from the...
.
In finance
In finance, survivorship bias is the tendency for failed companies to be excluded from performance studies because they no longer exist. It often causes the results of studies to skew higher because only companies which were successful enough to survive until the end of the period are included.For example, a mutual fund
Mutual fund
A mutual fund is a professionally managed type of collective investment scheme that pools money from many investors to buy stocks, bonds, short-term money market instruments, and/or other securities.- Overview :...
company's selection of funds today will include only those that are successful now. Many losing funds are closed and merged into other funds to hide poor performance. In theory, 90% of extant funds could truthfully claim to have performance in the first quartile of their peers if the peer group includes funds that have closed.
In 1996 Elton, Gruber, & Blake showed that survivorship bias is larger in the small-fund sector than in large mutual funds (presumably because small funds have a high probability of folding). They estimate the size of the bias across the U.S. mutual fund industry as 0.9% per annum, where the bias is defined and measured as:
- "Bias is defined as average a for surviving funds minus average
for all funds" - (Where aAlpha (investment)Alpha is a risk-adjusted measure of the so-called active return on an investment. It is the return in excess of the compensation for the risk borne, and thus commonly used to assess active managers' performances...
is the risk-adjusted return over the S&P 500S&P 500The S&P 500 is a free-float capitalization-weighted index published since 1957 of the prices of 500 large-cap common stocks actively traded in the United States. The stocks included in the S&P 500 are those of large publicly held companies that trade on either of the two largest American stock...
. This is the standard measure of mutual fund out-performance).
Additionally, in quantitative back-testing of market performance or other characteristics, survivorship bias is the use of a current index membership set rather than using the actual constituent changes over time. Consider a backtest to 1990 to find the average performance (total return) of S&P 500 members who have paid dividends within the previous year. To use the current 500 members only and create an historical equity line of the total return of the companies that met the criteria, would be adding survivorship bias to the results. S&P maintains an index of healthy companies, removing companies that no longer meet their criteria as a representative of the large-cap U.S. stock market. Companies that had healthy growth on their way to inclusion in the S&P 500, would be counted as if they were in the index during that growth period, when they were not. Instead there may have been another company in the index that was losing market capitalization and was destined for the S&P 600 Small-cap Index, that was later removed and would not be counted in the results. Using the actual membership of the index, applying entry and exit dates to gain the appropriate return during inclusion in the index, would allow for a bias-free output.
As a general experimental flaw
Survivorship bias (or survivor bias) is a statistical artifactArtifact (error)
In natural science and signal processing, an artifact is any error in the perception or representation of any visual or aural information introduced by the involved equipment or technique....
in applications outside finance
Finance
"Finance" is often defined simply as the management of money or “funds” management Modern finance, however, is a family of business activity that includes the origination, marketing, and management of cash and money surrogates through a variety of capital accounts, instruments, and markets created...
, where studies on the remaining population are fallaciously compared with the historic average despite the survivors having unusual properties. Mostly, the unusual property in question is a track record of success (like the successful funds).
For example, the parapsychology
Parapsychology
The term parapsychology was coined in or around 1889 by philosopher Max Dessoir, and originates from para meaning "alongside", and psychology. The term was adopted by J.B. Rhine in the 1930s as a replacement for the term psychical research...
researcher Joseph Banks Rhine
Joseph Banks Rhine
Joseph Banks Rhine was a botanist who later developed an interest in parapsychology and psychology. Rhine founded the parapsychology lab at Duke University, the Journal of Parapsychology, and the Foundation for Research on the Nature of Man...
believed he had identified the few individuals from hundreds of potential subjects who had powers of ESP
Extra-sensory perception
Extrasensory perception involves reception of information not gained through the recognized physical senses but sensed with the mind. The term was coined by Frederic Myers, and adopted by Duke University psychologist J. B. Rhine to denote psychic abilities such as telepathy, clairaudience, and...
. His calculations were based on the improbability of these few subjects guessing the Zener cards shown to a partner by chance.
A major criticism which surfaced against his calculations was the possibility of unconscious survivor bias in subject selections. He was accused of failing to take into account the large effective size of his sample (all the people he didn't choose as 'strong telepaths' because they failed at an earlier testing stage). Had he done this he might have seen that from the large sample, one or two individuals would probably achieve the track record of success he had found purely by chance. (Similarly, many investor
Investor
An investor is a party that makes an investment into one or more categories of assets --- equity, debt securities, real estate, currency, commodity, derivatives such as put and call options, etc...
s believe that chance is the main reason that most successful fund managers have the track records they do.)
Writing about the Rhine case
Fads and Fallacies in the Name of Science
Fads and Fallacies in the Name of Science, also known just as In the Name of Science, was Martin Gardner's second book, and has become a classic in the literature of entertaining scientific skepticism...
, Martin Gardner
Martin Gardner
Martin Gardner was an American mathematics and science writer specializing in recreational mathematics, but with interests encompassing micromagic, stage magic, literature , philosophy, scientific skepticism, and religion...
explained that he didn't think the experimenters had made such obvious mistakes out of statistical naiveté, but as a result of subtly disregarding some poor subjects. He said that without trickery of any kind, there would always be some people who had improbable success, if a large enough sample were taken. To illustrate this, he speculates about what would happen if one hundred professors of psychology
Psychology
Psychology is the study of the mind and behavior. Its immediate goal is to understand individuals and groups by both establishing general principles and researching specific cases. For many, the ultimate goal of psychology is to benefit society...
read Rhine's work and decided to make their own tests; he said that survivor bias would winnow out the typical failed experiments, but encourage the lucky successes to continue testing. He thought that the common null hypothesis
Null hypothesis
The practice of science involves formulating and testing hypotheses, assertions that are capable of being proven false using a test of observed data. The null hypothesis typically corresponds to a general or default position...
(of no result) wouldn't be reported, but:
- "Eventually, one experimenter remains whose subject has made high scores for six or seven successive sessions. Neither experimenter nor subject is aware of the other ninety-nine projects, and so both have a strong delusion that ESP is operating."
He concludes:
- "The experimenter writes an enthusiastic paper, sends it to Rhine who publishes it in his magazine, and the readers are greatly impressed".
If enough scientist
Scientist
A scientist in a broad sense is one engaging in a systematic activity to acquire knowledge. In a more restricted sense, a scientist is an individual who uses the scientific method. The person may be an expert in one or more areas of science. This article focuses on the more restricted use of the word...
s study a phenomenon, some will find statistically significant results by chance, and these are the experiments submitted for publication. To combat this, some editors now call for the submission of 'negative' scientific findings, where "nothing happened."
Survivorship bias is one of the issues discussed in the provocative 2005 paper "Why Most Published Research Findings Are False."
See also
- EconometricsEconometricsEconometrics has been defined as "the application of mathematics and statistical methods to economic data" and described as the branch of economics "that aims to give empirical content to economic relations." More precisely, it is "the quantitative analysis of actual economic phenomena based on...
- Meta-analysisMeta-analysisIn statistics, a meta-analysis combines the results of several studies that address a set of related research hypotheses. In its simplest form, this is normally by identification of a common measure of effect size, for which a weighted average might be the output of a meta-analyses. Here the...
- Selection biasSelection biasSelection bias is a statistical bias in which there is an error in choosing the individuals or groups to take part in a scientific study. It is sometimes referred to as the selection effect. The term "selection bias" most often refers to the distortion of a statistical analysis, resulting from the...
- Texas sharpshooter fallacyTexas sharpshooter fallacyThe Texas sharpshooter fallacy is a logical fallacy in which pieces of information that have no relationship to one another are called out for their similarities, and that similarity is used for claiming the existence of a pattern. This fallacy is the philosophical/rhetorical application of the...
- Derren Brown’s “The System”
- Fooled by RandomnessFooled by RandomnessFooled by Randomness: The Hidden Role of Chance in Life and in the Markets is a book written by Nassim Nicholas Taleb about the fallibility of human knowledge.-Reaction:The book was selected by Fortune as one of the 75 "Smartest Books of All Time."...
by Nassim Nicholas Taleb