Manipulation checks
Encyclopedia
Manipulation check is a term in experimental research in the social sciences which refers to certain kinds of secondary evaluations of an experiment.
subjects to different levels of the manipulation ("experimental conditions"). The experimenter then observes whether variation in the manipulated variables cause differences in the dependent variable
.
Manipulation checks are separate measured variables that show what the manipulated variables concurrently affect (besides the dependent variable of interest). Manipulations are NOT intended to verify that the manipulated factor caused variation in the dependent variable. Random assignment, manipulation before measurement of the dependent variable, and statistical tests
of effect of the manipulated variable on the dependent variable verify this. Thus, a failed manipulation check does not refute that the manipulation caused variation in the dependent variable.
In contrast, a successful manipulation check can help an experimenter rule out reasons why a manipulation may have failed to influence an independent variable. When a manipulation creates significant differences between experimental conditions in both (1) the dependent variable and (2) the measured manipulation check variable, the interpretation is that (1) the manipulation "causes" variation in the dependent variable (the "effect") and (2) the manipulation also explains variation in some other, more theoretically obvious measured variable that it is expected to concurrently influence, which assists in interpreting the "cause" (i.e., it only help interpret the "cause"; it is not necessary to affirm that the "cause" causes an effect).
Overview
In experiments, an experimenter manipulates some aspect of a process or task and randomly assignsRandom assignment
Random assignment or random placement is an experimental technique for assigning subjects to different treatments . The thinking behind random assignment is that by randomizing treatment assignment, then the group attributes for the different treatments will be roughly equivalent and therefore any...
subjects to different levels of the manipulation ("experimental conditions"). The experimenter then observes whether variation in the manipulated variables cause differences in the dependent variable
Dependent and independent variables
The terms "dependent variable" and "independent variable" are used in similar but subtly different ways in mathematics and statistics as part of the standard terminology in those subjects...
.
Manipulation checks are separate measured variables that show what the manipulated variables concurrently affect (besides the dependent variable of interest). Manipulations are NOT intended to verify that the manipulated factor caused variation in the dependent variable. Random assignment, manipulation before measurement of the dependent variable, and statistical tests
Statistical hypothesis testing
A statistical hypothesis test is a method of making decisions using data, whether from a controlled experiment or an observational study . In statistics, a result is called statistically significant if it is unlikely to have occurred by chance alone, according to a pre-determined threshold...
of effect of the manipulated variable on the dependent variable verify this. Thus, a failed manipulation check does not refute that the manipulation caused variation in the dependent variable.
In contrast, a successful manipulation check can help an experimenter rule out reasons why a manipulation may have failed to influence an independent variable. When a manipulation creates significant differences between experimental conditions in both (1) the dependent variable and (2) the measured manipulation check variable, the interpretation is that (1) the manipulation "causes" variation in the dependent variable (the "effect") and (2) the manipulation also explains variation in some other, more theoretically obvious measured variable that it is expected to concurrently influence, which assists in interpreting the "cause" (i.e., it only help interpret the "cause"; it is not necessary to affirm that the "cause" causes an effect).