Cross-modal integration of language and face cues in German and French infants – CROSSMOD

This project aims to investigate when and how infants are able to bind auditory and facial speech cues into a coherent and meaningful representation. The very few studies that have addressed this question found some type of binding between voice characteristics and facial movements in infants, but so far failed to identify any relevant matching cues. We aim to determine when this binding emerges, how specific it is to a particular language and whether it narrows down to the native language toward the end of the first year. To address those questions we will use two languages, French and German, that greatly differ in terms of phonetic repertoire and prosody (intonation and rhythm). On the contrary, German and French faces are rather similar, which will avoid any “other-race effect”. In 4 experiments, we will systematically investigate the proposed binding between auditory and visual face cues and the role of phonetic and prosodic information. Using language which belongs to the same or different rhythmic classes as German or French, we also intend to investigate the generalization of the audio-visual binding. We will study whether the binding can be generalized to languages belonging to the same rhythmic class, such as English (like German) and Spanish (like French). In a last experiment, we will determine, with an eye tracking system, how infants visually explore the faces showing their native or a non native language. We are expecting to find difference in the attention toward the mouth region depending on the language used. The cross-linguistic aspect of the project requires a French-German collaboration. This makes it possible to recruit and study French and German learning infants from monolingual families. Most previous cross-linguistic studies focused on very different topics and only tested participants from one language with different language stimuli. The intended project, however, will take advantage of a cross-over design and use the unique possibility of a French-German collaboration to study participants with different native languages by using the same language stimuli in a joint project. Our proposed experiments are designed to chart the development of the binding between visual and auditory cues in speech. By conducting these tests at different ages we will provide a developmental profile of how experience shapes the multimodal representation of language. The two labs involved in this project are internationally renowned for their research on infants and the two PIs are used to work in collaboration onto developmental projects.