The Effects of Iconic Gestures and Babble Language on Sentence Intelligibility
No Thumbnail Available
Previous studies on gestural enhancement have examined word intelligibility in noise vocoded speech, or in one linguistic masker (e.g., Drijvers & Özyürek, 2017; Schubotz et al., 2020), and speech-in-speech research has typically performed unimodal studies (e.g., Brouwer et al., 2012; Van Engen, 2010). The current study combines these two fields to investigate multimodal communication in a more ecologically valid manner. In particular, this thesis investigated to what extent iconic co-speech gestures help sentence intelligibility in two different linguistic maskers. Thirty-two native Dutch participants performed a Dutch sentence recognition task in which they were presented with videos in which an actress uttered short Dutch sentences (e.g., Ze begint te openen ‘She starts to open’). Participants were presented with a total of six audiovisual conditions: two gesture conditions (gestures absent vs. gestures present) and three masker conditions (clear vs. French 2-talker babble vs. Dutch 2-talker babble), and were asked to type down what was said by the Dutch actress. The focus of the study was on the accurate identification of the action verbs at the end of the target sentences. The results demonstrated that performance on the task was better in the gesture compared to the non-gesture conditions. In addition, sentence intelligibility was better in French than in Dutch babble. The results and implications of this project may be valuable to everyone who engages in multimodal communication, and especially to a public who often works in public places where competing speech is present in the background.
Faculteit der Letteren