Law of comparative judgment
Encyclopedia
The law of comparative judgment was conceived by L. L. Thurstone. In modern day terminology, it is more aptly described as a model that is used to obtain measurements from any process of pairwise comparison
. Examples of such processes are the comparison of perceived intensity of physical stimuli
, such as the weights of objects, and comparisons of the extremity of an attitude expressed within statements, such as statements about capital punishment. The measurements represent how we perceive objects, rather than being measurements of actual physical properties. This kind of measurement is the focus of psychometrics
and psychophysics
.
In somewhat more technical terms, the law of comparative judgment is a mathematical representation of a discriminal process, which is any process in which a comparison
is made between pairs of a collection of entities with respect to magnitudes of an attribute, trait, attitude
, and so on. The theoretical basis for the model is closely related to item response theory
and the theory underlying the Rasch model
, which are used in Psychology and Education to analyse data from questionnaires and tests.
for a particular 'project in measurement
' involving the comparison between a series of stimuli, such as weights and handwriting specimens, in pairs. He soon extended the domain of application of the law of comparative judgment to things that have no obvious physical counterpart, such as attitudes and values (Thurstone, 1929). For example, in one experiment, people compared statements about capital punishment to judge which of each pair expressed a stronger positive (or negative) attitude.
The essential idea behind Thurstone's process and model is that it can be used to scale a collection of stimuli based on simple comparisons between stimuli two at a time: that is, based on a series of pairwise comparisons
. For example, suppose that someone wishes to measure the perceived weights of a series of five objects of varying masses. By having people compare the weights of the objects in pairs, data can be obtained and the law of comparative judgment applied to estimate scale values of the perceived weights. This is the perceptual counterpart to the physical weight of the objects. That is, the scale represents how heavy people perceive the objects to be based on the comparisons.
Although Thurstone referred to it as a law, as stated above, in terms of modern psychometric theory the 'law' of comparative judgment is more aptly described as a measurement model. It represents a general theoretical model which, applied in a particular empirical context, constitutes a scientific hypothesis regarding the outcomes of comparisons between some collection of objects. If data agree with the model, it is possible to produce a scale from the data.
Weber's law essentially states that how much people perceive physical stimuli to change depends on how big a stimulus is. For example, if someone compares a light object of 1 kg with one slightly heavier, they can notice a relatively small difference, perhaps when the second object is 1.2 kg. On the other hand, if someone compares a heavy object of 30 kg with a second, the second must be quite a bit larger for a person to notice the difference, perhaps when the second object is 36 kg. People tend to perceive differences that are proportional to the size rather than always noticing a specific difference irrespective of the size. The same applies to brightness, pressure, warmth, loudness and so on.
Thurstone stated Weber's law as follows: "The stimulus increase which is correctly discriminated in any specified proportion of attempts (except 0 and 100 per cent) is a constant fraction of the stimulus magnitude" (Thurstone, 1959, p. 61). He considered that Weber's law said nothing directly about sensation intensities at all. In terms of Thurstone's conceptual framework, the association posited between perceived stimulus intensity and the physical magnitude of the stimulus in the Weber-Fechner law will only hold when Weber's law holds and the just noticeable difference
(JND) is treated as a unit
of measurement. Importantly, this is not simply given a priori (Michell, 1997, p. 355), as is implied by purely mathematical derivations of the one law from the other. It is, rather, an empirical
question whether measurements have been obtained; one which requires justification through the process of stating and testing a well-defined hypothesis
in order to ascertain whether specific theoretical criteria for measurement have been satisfied. Some of the relevant criteria were articulated by Thurstone, in a preliminary fashion, including what he termed the additivity criterion. Accordingly, from the point of view of Thurstone's approach, treating the JND as a unit is justifiable provided only that the discriminal dispersions are uniform for all stimuli considered in a given experimental context. Similar issues are associated with Stevens' power law
.
In addition, Thurstone employed the approach to clarify other similarities and differences between Weber's law, the Weber-Fechner law, and the LCJ. An important clarification is that the LCJ does not necessarily involve a physical stimulus, whereas the other 'laws' do. Another key difference is that Weber's law and the LCJ involve proportions of comparisons in which one stimulus is judged greater than another whereas the so-called Weber-Fechner law does not.
in which:
The discriminal dispersion of a stimulus i is the dispersion of fluctuations of the discriminal process for a uniform repeated stimulus, denoted
, where 
represents the mode of such values. Thurstone (1959, p. 20) used the term discriminal process to refer to the "psychological values of psychophysics"; that is, the values on a psychological continuum associated with a given stimulus.
where
In this case of the model, the difference
can be inferred directly from the proportion of instances in which j is judged greater than i if it is hypothesised that 
is distributed according to some density function, such as the normal distribution or logistic function. In order to do so, it is necessary to let 
, which is in effect an arbitrary choice of the unit
of measurement. Letting
be the proportion of occasions on which i is judged greater than j, if, for example, 
and it is hypothesised that 
is normally distributed, then it would be inferred that 
.
When a simple logistic function is employed instead of the normal density function, then the model has the structure of the Bradley-Terry-Luce model (BTL model) (Bradley & Terry, 1952; Luce, 1959). In turn, the Rasch model
for dichotomous data (Rasch, 1960/1980) is identical to the BTL model after the person parameter
of the Rasch model has been eliminated, as is achieved through statistical conditioning during the process of Conditional Maximum Likelihood
estimation. With this in mind, the specification of uniform discriminal dispersions is equivalent to the requirement of parallel Item Characteristic Curves (ICCs) in the Rasch model. Accordingly, as shown by Andrich (1978), the Rasch model should, in principle, yield essentially the same results as those obtained from a Thurstone scale
. Like the Rasch model, when applied in a given empirical context, Case 5 of the LCJ constitutes a mathematized hypothesis which embodies theoretical criteria for measurement.
, a structured technique for helping people deal with complex decisions. It uses pairwise comparisons of tangible and intangible factors to construct ratio scales that are useful in making important decisions.
Pairwise comparison
Pairwise comparison generally refers to any process of comparing entities in pairs to judge which of each entity is preferred, or has a greater amount of some quantitative property. The method of pairwise comparison is used in the scientific study of preferences, attitudes, voting systems, social...
. Examples of such processes are the comparison of perceived intensity of physical stimuli
Stimulus (physiology)
In physiology, a stimulus is a detectable change in the internal or external environment. The ability of an organism or organ to respond to external stimuli is called sensitivity....
, such as the weights of objects, and comparisons of the extremity of an attitude expressed within statements, such as statements about capital punishment. The measurements represent how we perceive objects, rather than being measurements of actual physical properties. This kind of measurement is the focus of psychometrics
Psychometrics
Psychometrics is the field of study concerned with the theory and technique of psychological measurement, which includes the measurement of knowledge, abilities, attitudes, personality traits, and educational measurement...
and psychophysics
Psychophysics
Psychophysics quantitatively investigates the relationship between physical stimuli and the sensations and perceptions they effect. Psychophysics has been described as "the scientific study of the relation between stimulus and sensation" or, more completely, as "the analysis of perceptual...
.
In somewhat more technical terms, the law of comparative judgment is a mathematical representation of a discriminal process, which is any process in which a comparison
Comparison
Comparison may refer to:-Language:* Comparison , a feature of many languages* Degree of comparison, an English language grammatical feature* Mass comparison, a test for the relatedness of languages-Mathematics:...
is made between pairs of a collection of entities with respect to magnitudes of an attribute, trait, attitude
Attitude (psychology)
An attitude is a hypothetical construct that represents an individual's degree of like or dislike for something. Attitudes are generally positive or negative views of a person, place, thing, or event— this is often referred to as the attitude object...
, and so on. The theoretical basis for the model is closely related to item response theory
Item response theory
In psychometrics, item response theory also known as latent trait theory, strong true score theory, or modern mental test theory, is a paradigm for the design, analysis, and scoring of tests, questionnaires, and similar instruments measuring abilities, attitudes, or other variables. It is based...
and the theory underlying the Rasch model
Rasch model
Rasch models are used for analysing data from assessments to measure variables such as abilities, attitudes, and personality traits. For example, they may be used to estimate a student's reading ability from answers to questions on a reading assessment, or the extremity of a person's attitude to...
, which are used in Psychology and Education to analyse data from questionnaires and tests.
Background
Thurstone published a paper on the law of comparative judgment in 1927. In this paper he introduced the underlying concept of a psychological continuumContinuum (theory)
Continuum theories or models explain variation as involving a gradual quantitative transition without abrupt changes or discontinuities. It can be contrasted with 'categorical' models which propose qualitatively different states.-In physics:...
for a particular 'project in measurement
Measurement
Measurement is the process or the result of determining the ratio of a physical quantity, such as a length, time, temperature etc., to a unit of measurement, such as the metre, second or degree Celsius...
' involving the comparison between a series of stimuli, such as weights and handwriting specimens, in pairs. He soon extended the domain of application of the law of comparative judgment to things that have no obvious physical counterpart, such as attitudes and values (Thurstone, 1929). For example, in one experiment, people compared statements about capital punishment to judge which of each pair expressed a stronger positive (or negative) attitude.
The essential idea behind Thurstone's process and model is that it can be used to scale a collection of stimuli based on simple comparisons between stimuli two at a time: that is, based on a series of pairwise comparisons
Pairwise comparison
Pairwise comparison generally refers to any process of comparing entities in pairs to judge which of each entity is preferred, or has a greater amount of some quantitative property. The method of pairwise comparison is used in the scientific study of preferences, attitudes, voting systems, social...
. For example, suppose that someone wishes to measure the perceived weights of a series of five objects of varying masses. By having people compare the weights of the objects in pairs, data can be obtained and the law of comparative judgment applied to estimate scale values of the perceived weights. This is the perceptual counterpart to the physical weight of the objects. That is, the scale represents how heavy people perceive the objects to be based on the comparisons.
Although Thurstone referred to it as a law, as stated above, in terms of modern psychometric theory the 'law' of comparative judgment is more aptly described as a measurement model. It represents a general theoretical model which, applied in a particular empirical context, constitutes a scientific hypothesis regarding the outcomes of comparisons between some collection of objects. If data agree with the model, it is possible to produce a scale from the data.
Relationships to pre-existing psychophysical theory
Thurstone showed that in terms of his conceptual framework, Weber's law and the so-called Weber-Fechner law, which are generally regarded as one and the same, are independent, in the sense that one may be applicable but not the other to a given collection of experimental data. In particular, Thurstone showed that if Fechner's law applies and the discriminal dispersions associated with stimuli are constant (as in Case 5 of the LCJ outlined below), then Weber's law will also be verified. He considered that the Weber-Fechner law and the LCJ both involve a linear measurement on a psychological continuum whereas Weber's law does not.Weber's law essentially states that how much people perceive physical stimuli to change depends on how big a stimulus is. For example, if someone compares a light object of 1 kg with one slightly heavier, they can notice a relatively small difference, perhaps when the second object is 1.2 kg. On the other hand, if someone compares a heavy object of 30 kg with a second, the second must be quite a bit larger for a person to notice the difference, perhaps when the second object is 36 kg. People tend to perceive differences that are proportional to the size rather than always noticing a specific difference irrespective of the size. The same applies to brightness, pressure, warmth, loudness and so on.
Thurstone stated Weber's law as follows: "The stimulus increase which is correctly discriminated in any specified proportion of attempts (except 0 and 100 per cent) is a constant fraction of the stimulus magnitude" (Thurstone, 1959, p. 61). He considered that Weber's law said nothing directly about sensation intensities at all. In terms of Thurstone's conceptual framework, the association posited between perceived stimulus intensity and the physical magnitude of the stimulus in the Weber-Fechner law will only hold when Weber's law holds and the just noticeable difference
Just noticeable difference
In psychophysics, a just noticeable difference, customarily abbreviated with lowercase letters as jnd, is the smallest detectable difference between a starting and secondary level of a particular sensory stimulus...
(JND) is treated as a unit
Units of measurement
A unit of measurement is a definite magnitude of a physical quantity, defined and adopted by convention and/or by law, that is used as a standard for measurement of the same physical quantity. Any other value of the physical quantity can be expressed as a simple multiple of the unit of...
of measurement. Importantly, this is not simply given a priori (Michell, 1997, p. 355), as is implied by purely mathematical derivations of the one law from the other. It is, rather, an empirical
Empirical
The word empirical denotes information gained by means of observation or experimentation. Empirical data are data produced by an experiment or observation....
question whether measurements have been obtained; one which requires justification through the process of stating and testing a well-defined hypothesis
Hypothesis
A hypothesis is a proposed explanation for a phenomenon. The term derives from the Greek, ὑποτιθέναι – hypotithenai meaning "to put under" or "to suppose". For a hypothesis to be put forward as a scientific hypothesis, the scientific method requires that one can test it...
in order to ascertain whether specific theoretical criteria for measurement have been satisfied. Some of the relevant criteria were articulated by Thurstone, in a preliminary fashion, including what he termed the additivity criterion. Accordingly, from the point of view of Thurstone's approach, treating the JND as a unit is justifiable provided only that the discriminal dispersions are uniform for all stimuli considered in a given experimental context. Similar issues are associated with Stevens' power law
Stevens' power law
Stevens' power law is a proposed relationship between the magnitude of a physical stimulus and its perceived intensity or strength. It is often considered to supersede the Weber–Fechner law on the basis that it describes a wider range of sensations, although critics argue that the validity of the...
.
In addition, Thurstone employed the approach to clarify other similarities and differences between Weber's law, the Weber-Fechner law, and the LCJ. An important clarification is that the LCJ does not necessarily involve a physical stimulus, whereas the other 'laws' do. Another key difference is that Weber's law and the LCJ involve proportions of comparisons in which one stimulus is judged greater than another whereas the so-called Weber-Fechner law does not.
The general form of the law of comparative judgment
The most general form of the LCJ isin which:
is the psychological scale value of stimuli i
is the sigma corresponding with the proportion of occasions on which the magnitude of stimulus i is judged to exceed the magnitude of stimulus j
is the discriminal dispersion of a stimulus 
is the correlation between the discriminal deviations of stimuli i and j
The discriminal dispersion of a stimulus i is the dispersion of fluctuations of the discriminal process for a uniform repeated stimulus, denoted
, where represents the mode of such values. Thurstone (1959, p. 20) used the term discriminal process to refer to the "psychological values of psychophysics"; that is, the values on a psychological continuum associated with a given stimulus.Case 5 of the law of comparative judgment
Thurstone specified five particular cases of the 'law', or measurement model. An important case of the model is Case 5, in which the discriminal dispersions are specified to be uniform and uncorrelated. This form of the model can be represented as follows:where
In this case of the model, the difference
can be inferred directly from the proportion of instances in which j is judged greater than i if it is hypothesised that is distributed according to some density function, such as the normal distribution or logistic function. In order to do so, it is necessary to let , which is in effect an arbitrary choice of the unitUnits of measurement
A unit of measurement is a definite magnitude of a physical quantity, defined and adopted by convention and/or by law, that is used as a standard for measurement of the same physical quantity. Any other value of the physical quantity can be expressed as a simple multiple of the unit of...
of measurement. Letting
be the proportion of occasions on which i is judged greater than j, if, for example, and it is hypothesised that is normally distributed, then it would be inferred that .When a simple logistic function is employed instead of the normal density function, then the model has the structure of the Bradley-Terry-Luce model (BTL model) (Bradley & Terry, 1952; Luce, 1959). In turn, the Rasch model
Rasch model
Rasch models are used for analysing data from assessments to measure variables such as abilities, attitudes, and personality traits. For example, they may be used to estimate a student's reading ability from answers to questions on a reading assessment, or the extremity of a person's attitude to...
for dichotomous data (Rasch, 1960/1980) is identical to the BTL model after the person parameter
Parameter
Parameter from Ancient Greek παρά also “para” meaning “beside, subsidiary” and μέτρον also “metron” meaning “measure”, can be interpreted in mathematics, logic, linguistics, environmental science and other disciplines....
of the Rasch model has been eliminated, as is achieved through statistical conditioning during the process of Conditional Maximum Likelihood
Maximum likelihood
In statistics, maximum-likelihood estimation is a method of estimating the parameters of a statistical model. When applied to a data set and given a statistical model, maximum-likelihood estimation provides estimates for the model's parameters....
estimation. With this in mind, the specification of uniform discriminal dispersions is equivalent to the requirement of parallel Item Characteristic Curves (ICCs) in the Rasch model. Accordingly, as shown by Andrich (1978), the Rasch model should, in principle, yield essentially the same results as those obtained from a Thurstone scale
Thurstone scale
In psychology, the Thurstone scale was the first formal technique for measuring an attitude. It was developed by Louis Leon Thurstone in 1928, as a means of measuring attitudes towards religion. It is made up of statements about a particular issue, and each statement has a numerical value...
. Like the Rasch model, when applied in a given empirical context, Case 5 of the LCJ constitutes a mathematized hypothesis which embodies theoretical criteria for measurement.
Applications
One important application involving the law of comparative judgment is the widely-used Analytic Hierarchy ProcessAnalytic Hierarchy Process
The Analytic Hierarchy Process is a structured technique for organizing and analyzing complex decisions. Based on mathematics and psychology, it was developed by Thomas L...
, a structured technique for helping people deal with complex decisions. It uses pairwise comparisons of tangible and intangible factors to construct ratio scales that are useful in making important decisions.
External links
- Adaptive comparative judgementAdaptive comparative judgementAdaptive Comparative Judgement is a technique borrowed from psychophysics which is able to generate reliable results for educational assessment - as such it is an alternative to traditional exam script marking. In the approach judges are presented with pairs of student work and are then asked to...
- "The Measurement of Psychological Value." by L.L. Thurstone
- L.L. Thurstone psychometric laboratory