Dörte de Kok

PhD Project

Audiovisual Processing in Aphasic and Non-Brain-Damaged Listeners: The Whole is More than the Sum of its Parts

Period: September 2007-September 2011

In the first experiment, we studied the influence of phonetic dimensions on aphasic perception. Larger distinctions (in place of articulation, manner of articulation and voicing) were detected more easily than distinctions in only one dimension. Voicing was the dimension that was most difficult to process. The aphasic listeners benefited from additional speechreading (i.e. seeing the speaker instead of only listening). This beneficial influence was not limited to a specific dimension, but occurred for all three dimensions.

The second experiment was a McGurk-type study, in which the participants were confronted with non-matching auditory and visual information (the lip movement of /ka/ dubbed onto auditory /pa/). The most interesting finding was a dissociation in patterns between non-brain-damaged and aphasic listeners. The first group showed an increased reaction time for the McGurk stimuli compared to ‘regular’ audiovisual stimuli, while the aphasic listeners showed no differences in reaction time. We explained this dissociation within the TRACE model of language processing, stating that the conflicting information inhibited the selection in non-brain-damaged listeners. Due to a problem in inhibition this delay was not found for the aphasic participants.

In the next study we examined the brain correlates of auditory processing and audiovisual integration. We tested a group of non-brain-damaged participants in an ERP-paradigm. We found that the recognition of smaller differences between auditory stimuli demanded more attention, which matches the results from our first study. A comparison of auditory and audiovisual items revealed that the amplitude of the attention-related ERP component P3 was smaller in reaction to the auditory part of the audiovisual stimuli than to the purely auditory syllables. This shows that the processing of audiovisual stimuli is eased compared to auditory stimuli and requires fewer brain resources.

In a final study we tried to carry out the ERP research with aphasic participants. We implemented a control condition, based on pure tones instead of language material. In that control condition we found deviating ERP patters, although the behavioral performance was identical to that of the control participants. Therefore, we could not investigate the target phonemic processing capacities. We concluded that the brain lesion disrupted the generation of a regular ERP pattern. Therefore, we advice to be careful when investigat- ing the ERP of participants with brain damage and to always include a control condition that detects abnormal wave generation in general.

Hessler, D., Jonkers, R., Stowe, L. & Bastiaanse, R. (2013). The whole is more than the sum of its parts - audiovisual processing of phonemes investigated with ERPs. Brain and Language 124 (3), 213-224. Download preprint

Hessler, D., Jonkers, R. & Bastiaanse, R. (2012). Processing of audiovisual stimuli in aphasic and non-brain-damaged listeners. Aphasiology 26, 83-102.

Hessler, D. (2011). Audiovisual processing in aphasic and non-brain-damaged listeners. The whole is more than the sum of its parts. PhD Thesis.

Hessler, D. (2011). Audiovisuelle Verarbeitung von Phonemen bei Aphasie. In Hanne, S., Fritzsche, T., Ott, S., Adelt, A. (eds.) Spektrum Patholinguistik (4). Potsdam: Universitätsverlag Potsdam.

Hessler, D., Jonkers, R. & Bastiaanse, R. (2010). The influence of phonetic dimensions on aphasic speech perception. Clinical Linguistics and Phonetics 24, 980-996.

Participants

Dörte de Kok (Hessler)
Dr. Roel Jonkers
Prof. Dr. Roelien Bastiaanse

Funding

This project was funded by an Ubbo Emmius Fellowship