McGurk effect
Encyclopedia
The McGurk effect is a perceptual
phenomenon which demonstrates an interaction between hearing
and vision
in speech perception
. "It is a compelling illusion in which humans perceive mismatched audiovisual speech as a completely different syllable". The visual information a person gets from seeing a person speak changes the way they hear the sound. People who are used to watching dubbed movies may be among people who are not susceptible to the McGurk effect because they have, to some extent, learned to ignore the information they are getting from the mouths of the "speakers". If a person is not getting good auditory information, but good visual information, they may be more likely to experience the McGurk effect. Integration abilities for audio and visual information may also influence whether a person will experience the effect. People that are better at this have shown to be more susceptible. Many people are affected differently by the McGurk effect based on many factors, brain damages or disorders.
and John MacDonald
in 1976. This effect was discovered by accident when McGurk and his research assistant, MacDonald, asked a technician to dub a video with a different phoneme
other than the one spoken while conducting a study on how infants perceive language at different developmental stages. When the video was played back, both researchers heard a third phoneme rather than the one spoken or mouthed in the video.
's production is dubbed with a sound-recording of a different phoneme being spoken. Often, the perceived phoneme is a third, intermediate phoneme. As an example, the syllable /ba-ba/ is spoken over the lip movements of /ga-ga/, and the perception is of /da-da/. McGurk and McDonald originally believed that this resulted from the common phonetic
and visual properties of /b/ and /g/. Two types of illusion in response to incongruent audiovisual stimuli have been observed: fusions ('ba' auditory and 'ga' visual produce 'da') and combinations ('ga' auditory and 'ba' visual produce 'bga').This is your brains effort to provide your consciousness with its best guess about what the incoming information is trying to tell it. The information coming from the eyes and ears is contradictory and in this instance, the eyes (visual information) has had a greater effect on the brain and thus comes the fusion and combination responses.
, which means that it involves information from more than one sensory modality, in particular, audition
and vision
. The McGurk effect arises during phonetic processing because the integration of audio and visual information happens early in speech perception. The McGurk effect is very robust; that is, knowledge about it seems to have little effect on one's perception of it. This is different from certain optical illusion
s, which break down once one 'sees through' them. Some people, including those that have been researching the phenomenon for more than twenty years, experience the effect even when they are aware that it is taking place. With the exception of people who can identify most of what is being said from speech-reading
alone, most people are quite limited in their ability to identify speech from visual-only signals. A more extensive phenomenon is the ability of visual speech to increase the intelligibility of heard speech in a noisy environment. Visible speech can also alter the perception of perfectly audible speech sounds when the visual speech stimuli are mismatched with the auditory speech (as in the McGurk effect). Normally, speech perception is thought to be an auditory process, however, our use of information is immediate, automatic, and, to a large degree, unconsciousand therefore, despite what is widely accepted as true, speech isn’t something we hear. Speech is perceived by all of the senses working together (seeing, touching, and listening to a face move). The brain is often unaware of the separate sensory contributions of what it perceives. Therefore, when it comes to recognizing speech your brain cannot differentiate whether it is seeing or hearing the incoming information.
programs by making use of a video camera
and lip reading
software. It has also been examined in relation to witness testimony. Wareham & Wright's 2005 study showed that inconsistent visual information can change the perception of spoken utterances, suggesting that the McGurk effect may have many influences in everyday perception. Not limited to syllables, the effect can occur in whole words and have an effect on daily interactions that people are unaware of. Research into this area can provide information on not only theoretical questions, but also it can provide therapeutic and diagnostic relevance for those with disorders relating to audio and visual integration of speech cues.
of the brain make a contribution to the McGurk effect. They work together to integrate speech information that is received through the auditory and visual senses. A McGurk response is more likely to occur in right handed individuals when the face has privileged access to the right hemisphere and words to the left hemisphere. In people that have had callostomies done, the McGurk effect is still present but significantly slower.
to the left hemisphere of the brain, visual features often play a critical role in speech and language therapy. People with lesions in the left hemisphere of the brain show a greater McGurk effect than normal controls. Visual information strongly influences speech perception in these people. There is a lack of susceptibility to the McGurk illusion if left hemisphere damage resulted in a deficit to visual segmental speech perception.
individuals exhibit a smaller McGurk effect than normal readers of the same chronological
age, but they showed the same effect as reading-level age-matched readers. Dyslexics particularly differed for combination responses, not fusion responses. The smaller McGurk effect may be due to the difficulties dyslexics have in perceiving and producing consonant clusters
.
show a significantly lower McGurk effect than normal children. They use less visual information in speech perception, or have a reduced attention to articulatory gestures
, but have no trouble perceiving auditory-only cues.
(ASD) showed a significantly reduced McGurk effect than children without. However, if the stimuli was nonhuman (for example bouncing a tennis ball to the sound of a bouncing beach ball) then they scored similarly to children without ASD. Younger children with ASD show a very reduced McGurk effect, however, this diminishes with age. As the individuals grow up, the effect they show becomes closer to those that did not have ASD.
produces a hemisphere disconnection process. Less influence on visual stimulus is seen in patients with AD, which is a a reason for the lowered McGurk effect.
individuals as in normal individuals, however, it is not significantly different. Schizophrenia slows down the development of audiovisual integration and does not allow it to reach its developmental peak, however, no degradation is observed. Schizophrenics are more likely to rely on auditory cues than visual cues in speech perception.
show impaired perception of speech in all conditions (visual-only, auditory-only, and audio-visual), and therefore exhibited a small McGurk effect. The greatest difficulty for aphasics is in the visual-only condition showing that they use more auditory stimuli in speech perception. .
significance is not a necessary condition for a McGurk effect to occur, however, it does increase the strength of the effect.
congruency
had a significant impact on the McGurk illusion. The effect is experienced more often and rated as clearer in the semantically congruent condition relative to the incongruent condition. When a person was expecting a certain visual or auditory appearance based on the semantic information leading up to it, the McGurk effect was greatly increased.
. While looking at oneself in the mirror and articulating visual stimuli while listening to another auditory stimulus, a strong McGurk effect can be observed. In the other condition, where the listener speaks auditory stimuli softly while watching another person articulate the conflicting visual gestures, a McGurk effect can still be seen, although it is weaker.
synchrony
is not necessary for the McGurk effect to be present. Subjects are still strongly influenced by auditory stimuli even when it lagged the visual stimuli by 180 milliseconds (point at which McGurk effect begins to weaken). . There was less tolerance for the lack of synchrony if the auditory stimuli preceded the visual stimuli. In order to produce a significant weakening of the McGurk effect, the auditory stimuli had to precede the visual stimuli by 60 milliseconds, or lag by 240 milliseconds.
task (touching something). Touch is a sensory perception like vision and audition, therefore increasing attention to touch decreases the attention to auditory and visual senses.
in order to integrate audio and visual information in speech perception. There was no difference in the McGurk effect when the listener was focusing anywhere on the speakers face. The effect does not appear if the listener focuses beyond the speakers face. In order for the McGurk effect to become insignificant, the listeners gaze must deviate from the speakers mouth by at least 60 degrees.
, but the intensity of the McGurk effect can change between languages. Dutch, English, Spanish, German and Italian language listeners experience a robust McGurk effect, while it is weaker for Japanese and Chinese listeners. Most research on the McGurk effect between languages has been conducted between English and Japanese. There is a smaller McGurk effect in Japanese listeners than in English listeners. The cultural practice of face avoidance in Japanese people may have an effect on the McGurk effect, as well as tone
and syllabic structures of the language. This could also be why Chinese listeners are less susceptible to visual cues, and similar to Japanese, produce a smaller effect than English listeners. Studies have also shown that Japanese listeners do not show a developmental increase in visual influence after the age of six, as English children do. Japanese listeners are more able to identify an incompatibility between the visual and auditory stimulus than English listeners are. This result could be in relation to the fact that in Japanese, consonant clusters
do not exist. In noisy environments where speech is unintelligible, however, people of all languages resort to using visual stimuli and are then equally subject to the McGurk effect.McGurk effect works with speech perceivers of every language for which it has been tested
. These individuals tend to weigh visual information from speech more heavily than auditory information. In comparison to normal hearing individuals, this is not different unless there is more than one syllable, such as a word. Regarding the McGurk experiment, responses from cochlear implanted users produced the same responses as normal hearing individuals when an auditory bilabial
stimulus is dubbed onto a visual velar
stimulus. However, when an auditory dental stimulus is dubbed onto a visual bilabial
stimulus, the responses are quite different. The McGurk effect is still present in individuals with impaired hearing or using cochlear implants, although it is quite different in some aspects.
stimuli, a response that is consistent with the McGurk effect can be recorded. From just minutes to a couple of days old, infants can imitate adult facial movements, and within weeks of birth, infants can recognize lip movements and speech sounds. At this point, the integration of audio and visual information can happen, but not at a proficient level. The first evidence of the McGurk effect can be seen at four months of age, however, more evidence is found for 5 month olds. Through the process of habituating
an infant to a certain stimulus and then changing the stimulus (or part of it, such as ba-voiced/va-visual to da-voiced/va-visual), a response that simulates the McGurk effect becomes apparent. The strength of the McGurk effect displays a developmental pattern and increases throughout childhood into extends into adulthood.
Perception
Perception is the process of attaining awareness or understanding of the environment by organizing and interpreting sensory information. All perception involves signals in the nervous system, which in turn result from physical stimulation of the sense organs...
phenomenon which demonstrates an interaction between hearing
Hearing (sense)
Hearing is the ability to perceive sound by detecting vibrations through an organ such as the ear. It is one of the traditional five senses...
and vision
Visual perception
Visual perception is the ability to interpret information and surroundings from the effects of visible light reaching the eye. The resulting perception is also known as eyesight, sight, or vision...
in speech perception
Speech perception
Speech perception is the process by which the sounds of language are heard, interpreted and understood. The study of speech perception is closely linked to the fields of phonetics and phonology in linguistics and cognitive psychology and perception in psychology...
. "It is a compelling illusion in which humans perceive mismatched audiovisual speech as a completely different syllable". The visual information a person gets from seeing a person speak changes the way they hear the sound. People who are used to watching dubbed movies may be among people who are not susceptible to the McGurk effect because they have, to some extent, learned to ignore the information they are getting from the mouths of the "speakers". If a person is not getting good auditory information, but good visual information, they may be more likely to experience the McGurk effect. Integration abilities for audio and visual information may also influence whether a person will experience the effect. People that are better at this have shown to be more susceptible. Many people are affected differently by the McGurk effect based on many factors, brain damages or disorders.
Discovery
The McGurk effect is sometimes called the McGurk-MacDonald effect. It was first described in a paper by Harry McGurkHarry McGurk
Harry McGurk was a Scottish cognitive psychologist. He is known for his discovery of the McGurk effect, described in a 1976 paper with his research assistant John McDonald, while he was a senior developmental psychologist at the University of Surrey..He was born in Hillington, Glasgow on 23...
and John MacDonald
John Macdonald
John Macdonald may refer to:*John S. MacDonald, co-founder of MacDonald Dettwiler*John Macdonald , New Zealand forensic psychiatrist who coined the MacDonald triad of sociopathy- Government :...
in 1976. This effect was discovered by accident when McGurk and his research assistant, MacDonald, asked a technician to dub a video with a different phoneme
Phoneme
In a language or dialect, a phoneme is the smallest segmental unit of sound employed to form meaningful contrasts between utterances....
other than the one spoken while conducting a study on how infants perceive language at different developmental stages. When the video was played back, both researchers heard a third phoneme rather than the one spoken or mouthed in the video.
Experiment
This effect may be experienced when a video of one phonemePhoneme
In a language or dialect, a phoneme is the smallest segmental unit of sound employed to form meaningful contrasts between utterances....
's production is dubbed with a sound-recording of a different phoneme being spoken. Often, the perceived phoneme is a third, intermediate phoneme. As an example, the syllable /ba-ba/ is spoken over the lip movements of /ga-ga/, and the perception is of /da-da/. McGurk and McDonald originally believed that this resulted from the common phonetic
Phonetics
Phonetics is a branch of linguistics that comprises the study of the sounds of human speech, or—in the case of sign languages—the equivalent aspects of sign. It is concerned with the physical properties of speech sounds or signs : their physiological production, acoustic properties, auditory...
and visual properties of /b/ and /g/. Two types of illusion in response to incongruent audiovisual stimuli have been observed: fusions ('ba' auditory and 'ga' visual produce 'da') and combinations ('ga' auditory and 'ba' visual produce 'bga').This is your brains effort to provide your consciousness with its best guess about what the incoming information is trying to tell it. The information coming from the eyes and ears is contradictory and in this instance, the eyes (visual information) has had a greater effect on the brain and thus comes the fusion and combination responses.
History
Vision is the primary sense for humans, but speech perception is multimodalMultimodal interaction
Multimodal interaction provides the user with multiple modes of interfacing with a system. A multimodal interface provides several distinct tools for input and output of data.- Multimodal input :...
, which means that it involves information from more than one sensory modality, in particular, audition
Hearing (sense)
Hearing is the ability to perceive sound by detecting vibrations through an organ such as the ear. It is one of the traditional five senses...
and vision
Visual perception
Visual perception is the ability to interpret information and surroundings from the effects of visible light reaching the eye. The resulting perception is also known as eyesight, sight, or vision...
. The McGurk effect arises during phonetic processing because the integration of audio and visual information happens early in speech perception. The McGurk effect is very robust; that is, knowledge about it seems to have little effect on one's perception of it. This is different from certain optical illusion
Optical illusion
An optical illusion is characterized by visually perceived images that differ from objective reality. The information gathered by the eye is processed in the brain to give a perception that does not tally with a physical measurement of the stimulus source...
s, which break down once one 'sees through' them. Some people, including those that have been researching the phenomenon for more than twenty years, experience the effect even when they are aware that it is taking place. With the exception of people who can identify most of what is being said from speech-reading
Lip reading
Lip reading, also known as lipreading or speechreading, is a technique of understanding speech by visually interpreting the movements of the lips, face and tongue with information provided by the context, language, and any residual hearing....
alone, most people are quite limited in their ability to identify speech from visual-only signals. A more extensive phenomenon is the ability of visual speech to increase the intelligibility of heard speech in a noisy environment. Visible speech can also alter the perception of perfectly audible speech sounds when the visual speech stimuli are mismatched with the auditory speech (as in the McGurk effect). Normally, speech perception is thought to be an auditory process, however, our use of information is immediate, automatic, and, to a large degree, unconsciousand therefore, despite what is widely accepted as true, speech isn’t something we hear. Speech is perceived by all of the senses working together (seeing, touching, and listening to a face move). The brain is often unaware of the separate sensory contributions of what it perceives. Therefore, when it comes to recognizing speech your brain cannot differentiate whether it is seeing or hearing the incoming information.
Why is it important?
The McGurk effect is being used to produce more accurate speech recognitionSpeech recognition
Speech recognition converts spoken words to text. The term "voice recognition" is sometimes used to refer to recognition systems that must be trained to a particular speaker—as is the case for most desktop recognition software...
programs by making use of a video camera
Video camera
A video camera is a camera used for electronic motion picture acquisition, initially developed by the television industry but now common in other applications as well. The earliest video cameras were those of John Logie Baird, based on the electromechanical Nipkow disk and used by the BBC in...
and lip reading
Lip reading
Lip reading, also known as lipreading or speechreading, is a technique of understanding speech by visually interpreting the movements of the lips, face and tongue with information provided by the context, language, and any residual hearing....
software. It has also been examined in relation to witness testimony. Wareham & Wright's 2005 study showed that inconsistent visual information can change the perception of spoken utterances, suggesting that the McGurk effect may have many influences in everyday perception. Not limited to syllables, the effect can occur in whole words and have an effect on daily interactions that people are unaware of. Research into this area can provide information on not only theoretical questions, but also it can provide therapeutic and diagnostic relevance for those with disorders relating to audio and visual integration of speech cues.
Brain hemispheres
Both hemispheresCerebral hemisphere
A cerebral hemisphere is one of the two regions of the eutherian brain that are delineated by the median plane, . The brain can thus be described as being divided into left and right cerebral hemispheres. Each of these hemispheres has an outer layer of grey matter called the cerebral cortex that is...
of the brain make a contribution to the McGurk effect. They work together to integrate speech information that is received through the auditory and visual senses. A McGurk response is more likely to occur in right handed individuals when the face has privileged access to the right hemisphere and words to the left hemisphere. In people that have had callostomies done, the McGurk effect is still present but significantly slower.
Left hemisphere lesions
In people with lesionsLesion
A lesion is any abnormality in the tissue of an organism , usually caused by disease or trauma. Lesion is derived from the Latin word laesio which means injury.- Types :...
to the left hemisphere of the brain, visual features often play a critical role in speech and language therapy. People with lesions in the left hemisphere of the brain show a greater McGurk effect than normal controls. Visual information strongly influences speech perception in these people. There is a lack of susceptibility to the McGurk illusion if left hemisphere damage resulted in a deficit to visual segmental speech perception.
Right hemisphere damage
In people with right hemisphere damage, impairment on both visual-only and audio-visual integration tasks is exhibited, although they are still able to integrate the information to produce a McGurk effect. Integration only appears if the auditory information is audible and visual stimuli is used to improve performance when the auditory signal is impoverished. Therefore, there is a McGurk effect exhibited in people with damage to the right hemisphere of the brain but the effect is not as strong as a normal group.Dyslexia
DyslexicDyslexia
Dyslexia is a very broad term defining a learning disability that impairs a person's fluency or comprehension accuracy in being able to read, and which can manifest itself as a difficulty with phonological awareness, phonological decoding, orthographic coding, auditory short-term memory, or rapid...
individuals exhibit a smaller McGurk effect than normal readers of the same chronological
Chronology
Chronology is the science of arranging events in their order of occurrence in time, such as the use of a timeline or sequence of events. It is also "the determination of the actual temporal sequence of past events".Chronology is part of periodization...
age, but they showed the same effect as reading-level age-matched readers. Dyslexics particularly differed for combination responses, not fusion responses. The smaller McGurk effect may be due to the difficulties dyslexics have in perceiving and producing consonant clusters
Consonant cluster
In linguistics, a consonant cluster is a group of consonants which have no intervening vowel. In English, for example, the groups and are consonant clusters in the word splits....
.
Specific language impairment
Children with specific language impairmentSpecific language impairment
Specific language impairment is diagnosed when a child's language does not develop normally and the difficulties cannot be accounted for by generally slow development , physical abnormality of the speech apparatus, autistic disorder, acquired brain damage or hearing loss.-Overview:Specific...
show a significantly lower McGurk effect than normal children. They use less visual information in speech perception, or have a reduced attention to articulatory gestures
Articulatory gestures
Articulatory gestures are the actions necessary to enunciate language. Examples of articulatory gestures are the hand movements necessary to enunciate sign language and the mouth movements of speech...
, but have no trouble perceiving auditory-only cues.
Autism spectrum disorders
Children with autism spectrum disordersAutism spectrum
The term "autism spectrum" is often used to describe disorders that are currently classified as pervasive developmental disorders. Pervasive developmental disorders include autism, Asperger syndrome, Childhood disintegrative disorder, Rett syndrome and Pervasive Developmental Disorder Not Otherwise...
(ASD) showed a significantly reduced McGurk effect than children without. However, if the stimuli was nonhuman (for example bouncing a tennis ball to the sound of a bouncing beach ball) then they scored similarly to children without ASD. Younger children with ASD show a very reduced McGurk effect, however, this diminishes with age. As the individuals grow up, the effect they show becomes closer to those that did not have ASD.
Language-learning disabilities
Adults with language-learning disabilities exhibit a much smaller McGurk effect than normal aduts. These people are not as influenced by visual input as normal people. Therefore, people with poor language skills will produce a smaller McGurk effect. A reason for the smaller effect in this population is that there may be uncoupled activity between anterior and posterior regions of the brain, or left and right hemishperes.Alzheimer’s disease
In patients with Alzheimer’s disease (AD), there is a smaller McGurk effect exhibited than in normals. Often a reduced size of the corpus callosumCorpus callosum
The corpus callosum , also known as the colossal commissure, is a wide, flat bundle of neural fibers beneath the cortex in the eutherian brain at the longitudinal fissure. It connects the left and right cerebral hemispheres and facilitates interhemispheric communication...
produces a hemisphere disconnection process. Less influence on visual stimulus is seen in patients with AD, which is a a reason for the lowered McGurk effect.
Schizophrenia
The McGurk effect is not as pronounced in schizophrenicSchizophrenia
Schizophrenia is a mental disorder characterized by a disintegration of thought processes and of emotional responsiveness. It most commonly manifests itself as auditory hallucinations, paranoid or bizarre delusions, or disorganized speech and thinking, and it is accompanied by significant social...
individuals as in normal individuals, however, it is not significantly different. Schizophrenia slows down the development of audiovisual integration and does not allow it to reach its developmental peak, however, no degradation is observed. Schizophrenics are more likely to rely on auditory cues than visual cues in speech perception.
Aphasia
People with aphasiaAphasia
Aphasia is an impairment of language ability. This class of language disorder ranges from having difficulty remembering words to being completely unable to speak, read, or write....
show impaired perception of speech in all conditions (visual-only, auditory-only, and audio-visual), and therefore exhibited a small McGurk effect. The greatest difficulty for aphasics is in the visual-only condition showing that they use more auditory stimuli in speech perception. .
Cross-dubbing
Discrepancy in vowel category significantly reduced the magnitude of the McGurk effect for fusion responses. Auditory /a/ tokens dubbed onto visual /i/ articulations were more compatible than the reverse. This could be because /a/ has a wide range of articulatory configurations whereas /i/ is more limited, which makes it much easier for subjects to detect discrepancies in the stimuli. /i/ vowel contexts produce the strongest effect, while /a/ produces a moderate effect, and /u/ has almost no effect.Mouth visibility
The McGurk effect is stronger when the right side of the mouth is visible. People tend to get more visual information from the right side of a speakers mouth than the left or even the whole mouth. This relates to the hemispheric attention factors discussed in the brain hemispheres section above.Visual distractors
The McGurk effect is weaker when there is a visual distractor present that the listener is attending to. Visual attention modulates audiovisual speech perception. Another form of distraction is movement of the speaker. A stronger McGurk effect is elicited if the speakers face/head is motionless, rather than moving.Syllable structure
A strong McGurk effect can be seen for click-vowel syllables compared to weak effects for isolated clicks. This shows that the McGurk effect can happen in a non-speech environment. PhonologicalPhonology
Phonology is, broadly speaking, the subdiscipline of linguistics concerned with the sounds of language. That is, it is the systematic use of sound to encode meaning in any spoken human language, or the field of linguistics studying this use...
significance is not a necessary condition for a McGurk effect to occur, however, it does increase the strength of the effect.
Gender
Females show a stronger McGurk effect than males. Women show significantly greater visual influence on auditory speech than men did for brief visual stimuli, but no difference is apparent for full stimuli. Another aspect regarding gender is the issue of male faces and voices as stimuli in comparison to female faces and voices as stimuli. Although, there is no difference in the strength of the McGurk effect for either situation. If a male face is dubbed with a female voice, or vice versa, there is still no difference in strength of the McGurk effect. Knowing that the voice you hear is different from the face you see - even if different genders – doesn’t eliminate the McGurk effect.Familiarity
Subjects who are familiar with the faces of the speakers are less susceptible to the McGurk effect than those who are unfamiliar with the faces of the speakers. On the other hand, there was no difference regarding voice familiarity.Expectation
SemanticSemantics
Semantics is the study of meaning. It focuses on the relation between signifiers, such as words, phrases, signs and symbols, and what they stand for, their denotata....
congruency
Congruence
Congruence is the state achieved by coming together, the state of agreement. The Latin congruō meaning “I meet together, I agree”. As an abstract term, congruence means similarity between objects...
had a significant impact on the McGurk illusion. The effect is experienced more often and rated as clearer in the semantically congruent condition relative to the incongruent condition. When a person was expecting a certain visual or auditory appearance based on the semantic information leading up to it, the McGurk effect was greatly increased.
Self influence
The McGurk effect can be observed when the listener is also the speaker or articulatorManner of articulation
In linguistics, manner of articulation describes how the tongue, lips, jaw, and other speech organs are involved in making a sound. Often the concept is only used for the production of consonants, even though the movement of the articulars will also greatly alter the resonant properties of the...
. While looking at oneself in the mirror and articulating visual stimuli while listening to another auditory stimulus, a strong McGurk effect can be observed. In the other condition, where the listener speaks auditory stimuli softly while watching another person articulate the conflicting visual gestures, a McGurk effect can still be seen, although it is weaker.
Temporal Synchrony
TemporalTime
Time is a part of the measuring system used to sequence events, to compare the durations of events and the intervals between them, and to quantify rates of change such as the motions of objects....
synchrony
Synchronization
Synchronization is timekeeping which requires the coordination of events to operate a system in unison. The familiar conductor of an orchestra serves to keep the orchestra in time....
is not necessary for the McGurk effect to be present. Subjects are still strongly influenced by auditory stimuli even when it lagged the visual stimuli by 180 milliseconds (point at which McGurk effect begins to weaken). . There was less tolerance for the lack of synchrony if the auditory stimuli preceded the visual stimuli. In order to produce a significant weakening of the McGurk effect, the auditory stimuli had to precede the visual stimuli by 60 milliseconds, or lag by 240 milliseconds.
Physical task diversion
The McGurk effect was greatly reduced when attention was diverted to a tactileSomatosensory system
The somatosensory system is a diverse sensory system composed of the receptors and processing centres to produce the sensory modalities such as touch, temperature, proprioception , and nociception . The sensory receptors cover the skin and epithelia, skeletal muscles, bones and joints, internal...
task (touching something). Touch is a sensory perception like vision and audition, therefore increasing attention to touch decreases the attention to auditory and visual senses.
Gaze
Your eyes do not need to fixateFixation (visual)
Fixation or visual fixation is the maintaining of the visual gaze on a single location. Humans typically alternate saccades and visual fixations, the notable exception being in smooth pursuit, controlled by a different neural substrate that appear to have developed for hunting prey...
in order to integrate audio and visual information in speech perception. There was no difference in the McGurk effect when the listener was focusing anywhere on the speakers face. The effect does not appear if the listener focuses beyond the speakers face. In order for the McGurk effect to become insignificant, the listeners gaze must deviate from the speakers mouth by at least 60 degrees.
Other Languages
People of all languages rely to some extent on visual information in speech perceptionSpeech perception
Speech perception is the process by which the sounds of language are heard, interpreted and understood. The study of speech perception is closely linked to the fields of phonetics and phonology in linguistics and cognitive psychology and perception in psychology...
, but the intensity of the McGurk effect can change between languages. Dutch, English, Spanish, German and Italian language listeners experience a robust McGurk effect, while it is weaker for Japanese and Chinese listeners. Most research on the McGurk effect between languages has been conducted between English and Japanese. There is a smaller McGurk effect in Japanese listeners than in English listeners. The cultural practice of face avoidance in Japanese people may have an effect on the McGurk effect, as well as tone
Tone (linguistics)
Tone is the use of pitch in language to distinguish lexical or grammatical meaning—that is, to distinguish or inflect words. All verbal languages use pitch to express emotional and other paralinguistic information, and to convey emphasis, contrast, and other such features in what is called...
and syllabic structures of the language. This could also be why Chinese listeners are less susceptible to visual cues, and similar to Japanese, produce a smaller effect than English listeners. Studies have also shown that Japanese listeners do not show a developmental increase in visual influence after the age of six, as English children do. Japanese listeners are more able to identify an incompatibility between the visual and auditory stimulus than English listeners are. This result could be in relation to the fact that in Japanese, consonant clusters
Consonant cluster
In linguistics, a consonant cluster is a group of consonants which have no intervening vowel. In English, for example, the groups and are consonant clusters in the word splits....
do not exist. In noisy environments where speech is unintelligible, however, people of all languages resort to using visual stimuli and are then equally subject to the McGurk effect.McGurk effect works with speech perceivers of every language for which it has been tested
Hearing Impairment
Experiments have been conducted involving hearing impaired individuals as well as individuals that have had cochlear implantsCochlear implant
A cochlear implant is a surgically implanted electronic device that provides a sense of sound to a person who is profoundly deaf or severely hard of hearing...
. These individuals tend to weigh visual information from speech more heavily than auditory information. In comparison to normal hearing individuals, this is not different unless there is more than one syllable, such as a word. Regarding the McGurk experiment, responses from cochlear implanted users produced the same responses as normal hearing individuals when an auditory bilabial
Bilabial consonant
In phonetics, a bilabial consonant is a consonant articulated with both lips. The bilabial consonants identified by the International Phonetic Alphabet are:...
stimulus is dubbed onto a visual velar
Velar consonant
Velars are consonants articulated with the back part of the tongue against the soft palate, the back part of the roof of the mouth, known also as the velum)....
stimulus. However, when an auditory dental stimulus is dubbed onto a visual bilabial
Bilabial consonant
In phonetics, a bilabial consonant is a consonant articulated with both lips. The bilabial consonants identified by the International Phonetic Alphabet are:...
stimulus, the responses are quite different. The McGurk effect is still present in individuals with impaired hearing or using cochlear implants, although it is quite different in some aspects.
Infants
By measuring an infants attention to certain audiovisualAudio-visual
The term Audio-Visual may refer to works with both a sound and a visual component, the production or use of such works, or to equipment used to create and present such works...
stimuli, a response that is consistent with the McGurk effect can be recorded. From just minutes to a couple of days old, infants can imitate adult facial movements, and within weeks of birth, infants can recognize lip movements and speech sounds. At this point, the integration of audio and visual information can happen, but not at a proficient level. The first evidence of the McGurk effect can be seen at four months of age, however, more evidence is found for 5 month olds. Through the process of habituating
Habituation
Habituation can be defined as a process or as a procedure. As a process it is defined as a decrease in an elicited behavior resulting from the repeated presentation of an eliciting stimulus...
an infant to a certain stimulus and then changing the stimulus (or part of it, such as ba-voiced/va-visual to da-voiced/va-visual), a response that simulates the McGurk effect becomes apparent. The strength of the McGurk effect displays a developmental pattern and increases throughout childhood into extends into adulthood.
See also
- Duplex perceptionDuplex perceptionDuplex perception refers to the linguistic phenomenon whereby "part of the acoustic signal is used for both a speech and a nonspeech percept." A listener is presented with two simultaneous, dichotic stimuli. One ear receives an isolated third-formant transition that sounds like a nonspeech chirp....
- Lip readingLip readingLip reading, also known as lipreading or speechreading, is a technique of understanding speech by visually interpreting the movements of the lips, face and tongue with information provided by the context, language, and any residual hearing....
- Motor theory of speech perceptionMotor theory of speech perceptionthumb|250px|right|When we hear [[speech|spoken words]] we sense that they are made of auditory [[sound]]s. The motor theory of speech perception argues that behind the sounds we hear are the intended movements of the [[vocal tract]] that [[pronunciation|pronounces]] them.The motor theory of speech...
- Speech perceptionSpeech perceptionSpeech perception is the process by which the sounds of language are heard, interpreted and understood. The study of speech perception is closely linked to the fields of phonetics and phonology in linguistics and cognitive psychology and perception in psychology...
- VisemeVisemeA viseme is a representational unit used to classify speech sounds in the visual domain. The term viseme was introduced based on the interpretation of the phoneme as a basic unit of speech in the acoustic/auditory domain,...
External links
- A constraint-based explanation of the McGurk effect a write up of the McGurk effect by Paul Boersma of University of Amsterdam. PDF available from The Rutgers Optimality Archive.
- Try The McGurk Effect! - Horizon: Is Seeing Believing? - BBC Two