User experience evaluation
Encyclopedia
User experience evaluation means investigating how a person feels about using a system (product, service, non-commercial item, or a combination of them). It is non-trivial to evaluate user experience
and come up with solid results, since user experience
is subjective, context-dependent and dynamic over time. Laboratory experiments may work well for studying a specific aspect of user experience
, but holistic user experience
is optimally studied over a longer period of time with real users in a natural environment.
Detailed guidance for user experience evaluation is hard to give, since there are many different dimensions to consider when choosing the evaluation approach:
In all cases, however, there are certain aspects of user experience
that evaluators are interested in (measures), and certain procedures and techniques used for collecting the data (methods). When investigating user experience evaluation methods, we can identify methods for emotion assessment and overall UX assessment. The measures and methods for these two evaluation types are described below. Episodic UX can be evaluated with either approach, depending on the case.
and arousal
. Objective emotion data can be collected by psychophysiological measurements or by observing expressed emotion
s. Subjective emotional data can be collected by using self-report methods, which can be verbal or non-verbal.
Examples of emotion evaluation methods:
There is no agreement on the exact measures for evaluating the overall UX with a system, largely because different products aim at different kinds of experiences. However, there are some high-level constructs of user experience
that can be used as the basis for defining the user experience measures, for example:
Since the importance of the above user experience constructs is different to different people, it is an interesting option to define the overall UX measures together with each study participant. Another option to evaluate overall UX is to use simply a scale from positive to negative, without further consideration of the user experience constructs.
Overall UX assessment is methodologically different from objective emotion assessment, but similar to subjective emotion assessment. Generic subjective user experience evaluation methods include interviews, questionnaires, story-telling, and often, a combination of these. An individual method can collect data about a set of specific constructs of user experience, such as usability testing
is to collect data about usability construct.
Examples of overall UX evaluation methods (excluding traditional usability methods):
research. An increasing amount of companies including some of the world's biggest publishers have begun outsourcing UX evaluation or opening their own in-house labs
. Researchers use a variety of HCI
and psychological techniques to examine the effectiveness of the User Experience
of the games during the design process. There are also some companies starting to use biometrics
to scientifically measure the relationship between in-game events and the player's emotions and feelings (the UX), such as Vertical Slice and Serco ExperienceLab in the UK, and Valve Software, Electronic Arts, BoltPeters, and VMC Labs in the USA and Canada. The interest in this area comes from both academia and industry, sometimes enabling collaborative work.
Game UX work has been featured at professional venues, such as the Game Developers Conference
(GDC)
User experience
User experience is the way a person feels about using a product, system or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership, but it also includes a person’s perceptions of the practical aspects such...
and come up with solid results, since user experience
User experience
User experience is the way a person feels about using a product, system or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership, but it also includes a person’s perceptions of the practical aspects such...
is subjective, context-dependent and dynamic over time. Laboratory experiments may work well for studying a specific aspect of user experience
User experience
User experience is the way a person feels about using a product, system or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership, but it also includes a person’s perceptions of the practical aspects such...
, but holistic user experience
User experience
User experience is the way a person feels about using a product, system or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership, but it also includes a person’s perceptions of the practical aspects such...
is optimally studied over a longer period of time with real users in a natural environment.
Detailed guidance for user experience evaluation is hard to give, since there are many different dimensions to consider when choosing the evaluation approach:
- Goal: SummativeSummative AssessmentSummative assessment refers to the assessment of the learning and summarizes the development of learners at a particular time. After a period of work, e.g. a unit for two weeks, the learner sits for a test and then the teacher marks the test and assigns a score. The test aims to summarize learning...
(score) or FormativeFormative assessmentFormative assessment is a range of formal and informal assessment procedures employed by teachers during the learning process in order to modify teaching and learning activities to improve student attainment. It typically involves qualitative feedback for both student and teacher that focuses on...
(areas for improvement) - Approach: ObjectiveObjectivity (science)Objectivity in science is a value that informs how science is practiced and how scientific truths are created. It is the idea that scientists, in attempting to uncover truths about the natural world, must aspire to eliminate personal biases, a priori commitments, emotional involvement, etc...
or SubjectiveSubjectivitySubjectivity refers to the subject and his or her perspective, feelings, beliefs, and desires. In philosophy, the term is usually contrasted with objectivity.-Qualia:... - Data: QuantitativeQuantitative researchIn the social sciences, quantitative research refers to the systematic empirical investigation of social phenomena via statistical, mathematical or computational techniques. The objective of quantitative research is to develop and employ mathematical models, theories and/or hypotheses pertaining to...
or QualitativeQualitative researchQualitative research is a method of inquiry employed in many different academic disciplines, traditionally in the social sciences, but also in market research and further contexts. Qualitative researchers aim to gather an in-depth understanding of human behavior and the reasons that govern such... - Granularity: Momentary, episodic, or overall UX
- Setup: Lab or field
In all cases, however, there are certain aspects of user experience
User experience
User experience is the way a person feels about using a product, system or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership, but it also includes a person’s perceptions of the practical aspects such...
that evaluators are interested in (measures), and certain procedures and techniques used for collecting the data (methods). When investigating user experience evaluation methods, we can identify methods for emotion assessment and overall UX assessment. The measures and methods for these two evaluation types are described below. Episodic UX can be evaluated with either approach, depending on the case.
Emotion evaluation
When investigating momentary user experiences, we can evaluate the level of positive affect, negative affect, joy, surprise, frustration, etc. The measures for emotions are bound to the methods used for emotion assessment, but typical emotion measures are e.g. valenceValence (psychology)
Valence, as used in psychology, especially in discussing emotions, means the intrinsic attractiveness or aversiveness of an event, object, or situation. However, the term is also used to characterize and categorize specific emotions. For example, the emotions popularly referred to as "negative",...
and arousal
Arousal
Arousal is a physiological and psychological state of being awake or reactive to stimuli. It involves the activation of the reticular activating system in the brain stem, the autonomic nervous system and the endocrine system, leading to increased heart rate and blood pressure and a condition of...
. Objective emotion data can be collected by psychophysiological measurements or by observing expressed emotion
Expressed emotion
Expressed emotion , is a qualitative measure of the 'amount' of emotion displayed, typically in the family setting, usually by a family or care takers....
s. Subjective emotional data can be collected by using self-report methods, which can be verbal or non-verbal.
Examples of emotion evaluation methods:
- Psychophysiological emotion measurements aim to identify emotions from physiological changes in muscles (e.g. face), pupils, skin, heart, brains, etc.
- Expression observers monitor person's facial and other gestures or the tone of voice to identify emotions manually
- Think aloud protocolThink aloud protocolThink-aloud protocol is a method used to gather data in usability testing in product design and development, in psychology and a range of social sciences...
can be used for reporting emotions (real-time verbal self-report) - PANAS (retrospective verbal self-report)
- Geneva emotion wheel (retrospective verbal self-report)
- Emotion Slider (continuous non-verbal self-report)
- Sensual Evaluation Instrument (snapshot non-verbal self-report)
- PrEmo, a new version of EmoCards for assessing emotion (snapshot non-verbal self-report)
Overall UX evaluation
In contrast to identifying a momentary emotion, overall UX evaluation investigates how a person feels about a system as a whole, typically after using it for a while. Many of the overall UX evaluation methods are suitable also for evaluating episodic UX, i.e., assessing how a person feels about a specific interaction episode or after executing a task.There is no agreement on the exact measures for evaluating the overall UX with a system, largely because different products aim at different kinds of experiences. However, there are some high-level constructs of user experience
User experience
User experience is the way a person feels about using a product, system or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership, but it also includes a person’s perceptions of the practical aspects such...
that can be used as the basis for defining the user experience measures, for example:
- Utility: Does the user perceive the functions in the system as useful and fit for the purpose?
- Usability: Does the user feel that it is easy and efficient to get things done with the system?
- Aesthetics: Does the user see the system as visually attractive? Does it feel pleasurable in hand?
- Identification: Can I identify myself with the product? Do I look good when using it?
- Stimulation: Does the system give me inspiration? Or wow experiences?
- Value: Is the system important to me? What is its value for me?
Since the importance of the above user experience constructs is different to different people, it is an interesting option to define the overall UX measures together with each study participant. Another option to evaluate overall UX is to use simply a scale from positive to negative, without further consideration of the user experience constructs.
Overall UX assessment is methodologically different from objective emotion assessment, but similar to subjective emotion assessment. Generic subjective user experience evaluation methods include interviews, questionnaires, story-telling, and often, a combination of these. An individual method can collect data about a set of specific constructs of user experience, such as usability testing
Usability testing
Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system...
is to collect data about usability construct.
Examples of overall UX evaluation methods (excluding traditional usability methods):
- Diary methods for self-reporting experiences during field studies
- Experience Sampling MethodExperience sampling methodThis research methodology, developed by Larson and Csikszentmihalyi asks participants to stop at certain times and make notes of their experience in real time. The point is for them to record temporal things like feelings while in the moment . They can be given a journal with many identical pages...
(ESM) for self-reporting during field studies - Day Reconstruction Method (DRM) – story-telling to reveal the meaningful experiences during field studies
- AttrakDiff questionnaire for overall UX evaluation
- Ladder interviewLadder interviewA Ladder interview is an interviewing technique where a seemingly simple response to a question is pushed by the interviewer in order to find subconscious motives.-Example:...
s e.g. to find out attitudes or values behind behaviour or experience
UX in video games
A relatively new pursuit in video game playtesting is UX and UsabilityUsability
Usability is the ease of use and learnability of a human-made object. The object of use can be a software application, website, book, tool, machine, process, or anything a human interacts with. A usability study may be conducted as a primary job function by a usability analyst or as a secondary job...
research. An increasing amount of companies including some of the world's biggest publishers have begun outsourcing UX evaluation or opening their own in-house labs
. Researchers use a variety of HCI
Human–computer interaction
Human–computer Interaction is the study, planning, and design of the interaction between people and computers. It is often regarded as the intersection of computer science, behavioral sciences, design and several other fields of study...
and psychological techniques to examine the effectiveness of the User Experience
User experience
User experience is the way a person feels about using a product, system or service. User experience highlights the experiential, affective, meaningful and valuable aspects of human-computer interaction and product ownership, but it also includes a person’s perceptions of the practical aspects such...
of the games during the design process. There are also some companies starting to use biometrics
Biometrics
Biometrics As Jain & Ross point out, "the term biometric authentication is perhaps more appropriate than biometrics since the latter has been historically used in the field of statistics to refer to the analysis of biological data [36]" . consists of methods...
to scientifically measure the relationship between in-game events and the player's emotions and feelings (the UX), such as Vertical Slice and Serco ExperienceLab in the UK, and Valve Software, Electronic Arts, BoltPeters, and VMC Labs in the USA and Canada. The interest in this area comes from both academia and industry, sometimes enabling collaborative work.
Game UX work has been featured at professional venues, such as the Game Developers Conference
Game Developers Conference
The Game Developers Conference is the largest annual gathering of professional video game developers, focusing on learning, inspiration, and networking...
(GDC)