Absolute probability judgement
Encyclopedia
Absolute probability judgement is a technique used in the field of human reliability
assessment (HRA), for the purposes of evaluating the probability of a human error occurring throughout the completion of a specific task. From such analyses measures can then be taken to reduce the likelihood of errors occurring within a system and therefore lead to an improvement in the overall levels of safety. There exist three primary reasons for conducting an HRA; error identification, error quantification and error reduction. As there exist a number of techniques used for such purposes, they can be split into one of two classifications; first generation techniques and second generation techniques. First generation techniques work on the basis of the simple dichotomy of ‘fits/doesn’t fit’ in the matching of the error situation in context with related error identification and quantification and second generation techniques are more theory based in their assessment and quantification of errors. ‘HRA techniques have been utilised in a range of industries including healthcare, engineering
, nuclear, transportation and business sector; each technique has varying uses within different disciplines.
Absolute probability judgement, which is also known as direct numerical estimation , is based on the quantification
of human error probabilities (HEPs). It is grounded on the premise that people cannot recall or are unable to estimate with certainty, the probability of a given event occurring. Expert judgement is typically desirable for utilisation in the technique when there is little or no data with which to calculate HEPs, or when the data is unsuitable or difficult to understand. In theory, qualitative knowledge built through the experts’ experience can be translated into quantitative data
such as HEPs.
Required of the experts is a good level of both substantive experience (i.e. the expert must have a suitable level of knowledge of the problem domain) and normative experience (i.e. it must be possible for the expert, perhaps with the aid of a facilitator
, to translate this knowledge explicitly into probabilities). If experts possess the required substantive knowledge but lack knowledge which is normative in nature, the experts may be trained or assisted in ensuring that the knowledge and expertise requiring to be captured is translated into the correct probabilities i.e. to ensure that it is an accurate representation of the experts’ judgements.
judgement is an expert judgement-based approach which involves using the beliefs of experts (e.g. front-line staff, process engineers etc.) to estimate HEPs. There are two primary forms of the technique; Group Methods and Single Expert Methods i.e. it can be done either as a group or as an individual exercise. Group methods tend to be the more popular and widely used as they are more robust and are less subject to bias
. Moreover, within the context of use, it is unusual for a single individual to possess all the required information and expertise to be able to solely estimate, in an accurate manner, the human reliability in question. In the group approach, the outcome of aggregating individual knowledge and opinions is more reliable.
method is very similar to the Aggregated Individual Method in that experts make their initial estimates in isolation. However following this stage, the experts are then shown the outcome that all other participants have arrived at and are then able to re-consider the estimates which they initially made. The re-estimates are then aggregated using the geometric mean
. This allows for some information sharing
, whilst avoiding most group-led biases; however there still remains the problem of a lack of discussion.
and the exchange of ideas and also promotes equal opportunity to participate in discussion. However, it can also prove to be logistically awkward to co-ordinate as it requires that all experts be together in the same location in order for the discussion to take place. Due to this technicality, personalities and other biasing mechanisms such as overconfidence, recent availability and anchoring may become a factor, thus increasing the potential for the results to be skewed. If the circumstance arises in which there is a deadlock
or breakdown in group dynamics
, it then becomes necessary to revert to one of the other group absolute probability judgement methods.
The chosen experts must have a good working knowledge of the tasks which require to be assessed. The correct number of experts is dependent upon what seems most practicable, while considering any constraints such as spatial and financial availability. However, it should be noted that the larger the group the more likely problems are to arise.
2. Prepare task statement
Task statements are a necessary component of the method; tasks are specified in detail. The more fuller the explanation of the task within the statement, the less likely it will be that the experts will resort to making individual guesses about the tasks. The statement should also ensure that any assumptions are clearly stated in an interpretable format for all experts to understand. The optimal level of detail will be governed by the nature of the task under consideration and the required use of the final HEP estimation.
3. Prepare response booklet
These booklets detail the task statement and design of the scale to use in assessing error probability and by which experts can indicate their judgements.. The scale must be one which allows differences to be made apparent. The booklet also includes instructions, assumptions and sample items.
4. Develop instructions for subjects
Instructions are required to specify to the experts the reasons for the session, otherwise they may guess such reasons which may cause bias in the resultant estimates of human reliability
.
5. Obtain judgements
Experts are required to reveal their judgements on each of the tasks; this can be done in a group or individually. If done by the former means, a facilitator is often used to prevent any bias and help overcome any problems.
6. Calculate inter-judge consistency
This is a method by which the differences in the HEP estimates of individual experts can be compared; a statistical formulation is used for such purposes.
7. Aggregate individual estimates
Where group consensus methods are not used, it is necessary to compute an aggregate for each of the individual estimates for each HEP.
8. Uncertainty bound estimation
Calculated by using statistical approaches involving confidence ranges.
Paris
, using a group consensus methodology.
During the duration of the session it was revealed that the ease with which the experts were able to arrive at a consensus was low with regards to the differing estimates of the various HEP values. Discussions often changed individuals’ thinking e.g. in the light of new information or interpretations, but this did not ease reaching an agreement. Due to this difficulty, it was therefore necessary to aggregate the individual estimates in order to calculate a geometric mean of these.
The following table displays a sample of the results obtained.
Table: Pilot absolute probability judgement Session–extract of results
In various cases, the range of figures separating the maximum and minimum values proved to be too large to allow to aggregated value to be accepted with confidence
These values are the events in the risk model which require to be quantified. There are 3 primary errors in the model that may occur:
There were various reasons which can explain the reasons why there was such a large difference in the estimates provided by the group: the group of experts was largely diverse and the experience of the individuals differed. Experience with Ground Based Augmentation System (GBAS) also showed differences. This process was a new experience for all of the experts participating in the process and there was only a single day, in which the session was taking place, to become familiar with its use and use it correctly. Of most significance was the fact that the detail of the assessments was very fine, which the staff were not used to. Experts also became confused about the way in which the assessment took place; errors were not considered on their own and were analysed as a group. This meant that the values estimated represented a contribution of the error on a system failure as opposed to a single contribution to system failure.
Human reliability
Human reliability is related to the field of human factors engineering and ergonomics, and refers to the reliability of humans in fields such as manufacturing, transportation, the military, or medicine...
assessment (HRA), for the purposes of evaluating the probability of a human error occurring throughout the completion of a specific task. From such analyses measures can then be taken to reduce the likelihood of errors occurring within a system and therefore lead to an improvement in the overall levels of safety. There exist three primary reasons for conducting an HRA; error identification, error quantification and error reduction. As there exist a number of techniques used for such purposes, they can be split into one of two classifications; first generation techniques and second generation techniques. First generation techniques work on the basis of the simple dichotomy of ‘fits/doesn’t fit’ in the matching of the error situation in context with related error identification and quantification and second generation techniques are more theory based in their assessment and quantification of errors. ‘HRA techniques have been utilised in a range of industries including healthcare, engineering
Engineering
Engineering is the discipline, art, skill and profession of acquiring and applying scientific, mathematical, economic, social, and practical knowledge, in order to design and build structures, machines, devices, systems, materials and processes that safely realize improvements to the lives of...
, nuclear, transportation and business sector; each technique has varying uses within different disciplines.
Absolute probability judgement, which is also known as direct numerical estimation , is based on the quantification
Quantification
Quantification has several distinct senses. In mathematics and empirical science, it is the act of counting and measuring that maps human sense observations and experiences into members of some set of numbers. Quantification in this sense is fundamental to the scientific method.In logic,...
of human error probabilities (HEPs). It is grounded on the premise that people cannot recall or are unable to estimate with certainty, the probability of a given event occurring. Expert judgement is typically desirable for utilisation in the technique when there is little or no data with which to calculate HEPs, or when the data is unsuitable or difficult to understand. In theory, qualitative knowledge built through the experts’ experience can be translated into quantitative data
Numerical data
Numerical data is data measured or identified on a numerical scale. Numerical data can be analyzed using statistical methods, and results can be displayed using tables, charts, histograms and graphs. For example, a researcher will ask a questions to a participant that include words how often, how...
such as HEPs.
Required of the experts is a good level of both substantive experience (i.e. the expert must have a suitable level of knowledge of the problem domain) and normative experience (i.e. it must be possible for the expert, perhaps with the aid of a facilitator
Facilitator
A facilitator is someone who helps a group of people understand their common objectives and assists them to plan to achieve them without taking a particular position in the discussion...
, to translate this knowledge explicitly into probabilities). If experts possess the required substantive knowledge but lack knowledge which is normative in nature, the experts may be trained or assisted in ensuring that the knowledge and expertise requiring to be captured is translated into the correct probabilities i.e. to ensure that it is an accurate representation of the experts’ judgements.
Background
Absolute probabilityProbability
Probability is ordinarily used to describe an attitude of mind towards some proposition of whose truth we arenot certain. The proposition of interest is usually of the form "Will a specific event occur?" The attitude of mind is of the form "How certain are we that the event will occur?" The...
judgement is an expert judgement-based approach which involves using the beliefs of experts (e.g. front-line staff, process engineers etc.) to estimate HEPs. There are two primary forms of the technique; Group Methods and Single Expert Methods i.e. it can be done either as a group or as an individual exercise. Group methods tend to be the more popular and widely used as they are more robust and are less subject to bias
Bias
Bias is an inclination to present or hold a partial perspective at the expense of alternatives. Bias can come in many forms.-In judgement and decision making:...
. Moreover, within the context of use, it is unusual for a single individual to possess all the required information and expertise to be able to solely estimate, in an accurate manner, the human reliability in question. In the group approach, the outcome of aggregating individual knowledge and opinions is more reliable.
Absolute probability judgement methodologies
There are 4 main group methods by which absolute probability judgement can be conducted.Aggregated individual method
Utilising this method, experts make their estimates individually without actually meeting or discussing the task. The estimates are then aggregated by taking the geometric mean of the individual experts’ estimates for each task. The major drawback to this method is that there is no shared expertise through the group; however a positive of this is that due to the individuality of the process, any conflict such as dominating personalities or conflicting personalities is avoided and the results are therefore free of any bias.Delphi method
Developed by Dalkey , the Delphi methodDelphi method
The Delphi method is a structured communication technique, originally developed as a systematic, interactive forecasting method which relies on a panel of experts.In the standard version, the experts answer questionnaires in two or more rounds...
method is very similar to the Aggregated Individual Method in that experts make their initial estimates in isolation. However following this stage, the experts are then shown the outcome that all other participants have arrived at and are then able to re-consider the estimates which they initially made. The re-estimates are then aggregated using the geometric mean
Geometric mean
The geometric mean, in mathematics, is a type of mean or average, which indicates the central tendency or typical value of a set of numbers. It is similar to the arithmetic mean, except that the numbers are multiplied and then the nth root of the resulting product is taken.For instance, the...
. This allows for some information sharing
Information sharing
The term "information sharing" gained popularity as a result of the 9/11 Commission Hearings and its report of the United States government's lack of response to information known about the planned terrorist attack on the New York City World Trade Center prior to the event...
, whilst avoiding most group-led biases; however there still remains the problem of a lack of discussion.
Nominal group technique (NGT)
This technique takes the Delphi method and introduces limited discussion/consultation between the experts. By this means, information-sharing is superior, and group domination is mitigated by having the experts separately come to their own conclusion before aggregating the HEP scores.Consensus group method
This is the most group-centred approach and requires that the group must come to a consensus on the HEP estimates through discussion and mutual agreement. This method maximises knowledge sharingKnowledge sharing
Knowledge sharing is an activity through which knowledge is exchanged among people, friends, or members of a family, a community or an organization....
and the exchange of ideas and also promotes equal opportunity to participate in discussion. However, it can also prove to be logistically awkward to co-ordinate as it requires that all experts be together in the same location in order for the discussion to take place. Due to this technicality, personalities and other biasing mechanisms such as overconfidence, recent availability and anchoring may become a factor, thus increasing the potential for the results to be skewed. If the circumstance arises in which there is a deadlock
Deadlock
A deadlock is a situation where in two or more competing actions are each waiting for the other to finish, and thus neither ever does. It is often seen in a paradox like the "chicken or the egg"...
or breakdown in group dynamics
Group dynamics
Group dynamics refers to a system of behaviors and psychological processes that occur within a social group , or between social groups...
, it then becomes necessary to revert to one of the other group absolute probability judgement methods.
Absolute probability judgement procedure
1. Select subject matter expertsThe chosen experts must have a good working knowledge of the tasks which require to be assessed. The correct number of experts is dependent upon what seems most practicable, while considering any constraints such as spatial and financial availability. However, it should be noted that the larger the group the more likely problems are to arise.
2. Prepare task statement
Task statements are a necessary component of the method; tasks are specified in detail. The more fuller the explanation of the task within the statement, the less likely it will be that the experts will resort to making individual guesses about the tasks. The statement should also ensure that any assumptions are clearly stated in an interpretable format for all experts to understand. The optimal level of detail will be governed by the nature of the task under consideration and the required use of the final HEP estimation.
3. Prepare response booklet
These booklets detail the task statement and design of the scale to use in assessing error probability and by which experts can indicate their judgements.. The scale must be one which allows differences to be made apparent. The booklet also includes instructions, assumptions and sample items.
4. Develop instructions for subjects
Instructions are required to specify to the experts the reasons for the session, otherwise they may guess such reasons which may cause bias in the resultant estimates of human reliability
Human reliability
Human reliability is related to the field of human factors engineering and ergonomics, and refers to the reliability of humans in fields such as manufacturing, transportation, the military, or medicine...
.
5. Obtain judgements
Experts are required to reveal their judgements on each of the tasks; this can be done in a group or individually. If done by the former means, a facilitator is often used to prevent any bias and help overcome any problems.
6. Calculate inter-judge consistency
This is a method by which the differences in the HEP estimates of individual experts can be compared; a statistical formulation is used for such purposes.
7. Aggregate individual estimates
Where group consensus methods are not used, it is necessary to compute an aggregate for each of the individual estimates for each HEP.
8. Uncertainty bound estimation
Calculated by using statistical approaches involving confidence ranges.
Context
In this example, absolute probability judgement was utilised by Eurocontrol, at the experimental centre in BretignyBrétigny
Brétigny or Bretigny may refer to the following places:*in France:**Bretigny, Côte-d'Or, in the Côte-d'Or département**Brétigny, Eure, in the Eure département**Brétigny, Oise, in the Oise département...
Paris
Paris
Paris is the capital and largest city in France, situated on the river Seine, in northern France, at the heart of the Île-de-France region...
, using a group consensus methodology.
Required inputs
Each of the grades of staff included in the session took turns to provide estimates of the error probabilities, including ground staff, pilots and controllers. Prior to the beginning of the session, an introductory exercise was conducted to allow the participants to feel more comfortable with use of the technique; this involved an explanation to the background of the method and provided an overview of what the session would entail of. To increase familiarity of the method, exemplary templates were used to show how errors are estimated.Method
- Initial task statements of the project were created leaving space for individual opinion of task estimates and additional assumptions the group may have collectively foregone.
- A session was held in which the individual scenarios and tasks were accurately detailed to the experts
- Experts, with this knowledge, were then able to enter individual estimations for all tasks under consideration
- Discussion followed in which all participants were provided with the opportunity to express their opinion to the rest of the group
- Facilitation was then used in order to reach a group consensus on the estimate values. Further discussion and amendment took place when necessary.
During the duration of the session it was revealed that the ease with which the experts were able to arrive at a consensus was low with regards to the differing estimates of the various HEP values. Discussions often changed individuals’ thinking e.g. in the light of new information or interpretations, but this did not ease reaching an agreement. Due to this difficulty, it was therefore necessary to aggregate the individual estimates in order to calculate a geometric mean of these.
The following table displays a sample of the results obtained.
Table: Pilot absolute probability judgement Session–extract of results
| Potential Error (Code in Risk Model) | Maximum | Minimum | Range | Geometric Mean |
|---|---|---|---|---|
| C1a | 1.1E-03 | 2.0E-05 | 55 | 2.1E-04 |
| C1b | 2.5E-04 | 1.0E-05 | 25 | 3.5E-05 |
| D1 | 1.0E-03 | 1.0E-04 | 10 | 4.3E-04 |
| F1a | 4.0E-04 | 1.0E-05 | 40 | 6.9E-05 |
| F1b | 1.0E-03 | 1.0E-04 | 10 | 4.0E-04 |
| F1c | 1.0E-03 | 1.0E-04 | 10 | 4.6E-04 |
In various cases, the range of figures separating the maximum and minimum values proved to be too large to allow to aggregated value to be accepted with confidence
These values are the events in the risk model which require to be quantified. There are 3 primary errors in the model that may occur:
- C1: Capturing false information about final approach path
- D1: Failure to maintain a/c on final approach path
- F1: Selecting wrong runway
There were various reasons which can explain the reasons why there was such a large difference in the estimates provided by the group: the group of experts was largely diverse and the experience of the individuals differed. Experience with Ground Based Augmentation System (GBAS) also showed differences. This process was a new experience for all of the experts participating in the process and there was only a single day, in which the session was taking place, to become familiar with its use and use it correctly. Of most significance was the fact that the detail of the assessments was very fine, which the staff were not used to. Experts also became confused about the way in which the assessment took place; errors were not considered on their own and were analysed as a group. This meant that the values estimated represented a contribution of the error on a system failure as opposed to a single contribution to system failure.
Results/outcomes
- Controllers and pilots provided good estimates for the errors and these have been used in some safety cases
- Participants highlighted their understanding of the importance of their participation in the process to provide expertise, as opposed to using external safety analysts instead i.e. their understood their role in carrying out a Human Reliability Assessment of the system
- The experts were provided with a realistic representation of human performance within the system and therefore further safety requirements required to improve the safety and reduce the likelihood of the identified errors. This is particularly beneficial; for the future GBAS.
Lessons from the study
- Time is required to familiarise with the methodology and to understand what is needed to be done in the given context
- Experts are required to understand the circumstances in which HEPs are conditional
- There is a need for true experts to be included in the process and in significant number to allow for the necessary information to be gathered.
- The use of existing information in the process is always helpful for the purposes of standardisation
Advantages of absolute probability judgement
- The method is relatively quick and straightforward to employ. With a greater degree of group discussion in use of the technique, there is more qualitative data that is produced; this can be considered as a useful by-product of the assessment.
- Absolute probability judgement is not restricted to or specialised for use in a particular field; it is easily applicable to an HRA on any industrial sector thus making it a generic technique for use in a wide range of potential applications.[5]
- Useful suggestions may result from discussion as to ways in which a reduction in errors can be achieved
Disadvantages of absolute probability judgement
- Absolute probability judgement is prone to certain biases and group conflicts or problems. Selection of the correct group methodology or high-quality group facilitation may decrease the effect of these biases and increase the validity of the results.
- Locating suitable experts for the absolute probability judgement exercise is a difficult stage of the process, more so due to the ambiguity with which the term ‘expert’ can be defined
- Because there may be little or no empirical and/or quantitative reasoning underpinning the experts’ estimates, it is difficult to be certain of the validity of the final HEPs i.e. there is no means by which guesses can be validated