Using eye-tracking to study auditory comprehension in codeswitching: Evidence for the link between comprehension and production

Open Access
Valdes Kroff, Jorge Rodrigo
Graduate Program:
Doctor of Philosophy
Document Type:
Date of Defense:
April 30, 2012
Committee Members:
  • Paola Eulalia Dussias, Dissertation Advisor
  • Henry J Gerfen, Dissertation Advisor
  • Rena Torres Cacoullos, Committee Member
  • Judith Fran Kroll, Committee Member
  • Daniel J Weiss, Committee Member
  • codeswitching
  • grammatical gender
  • sentence processing
  • visual world paradigm
  • bilingualism
Bilinguals in the presence of other known bilinguals engage in codeswitching, broadly defined as the fluid alternation between languages in bilingual discourse (Poplack, 1980). Although the specific factors that influence codeswitching are varied, bilingual members of a community of codeswitchers are more likely to engage in intra-sentential codeswitching (e.g. El niño caught his friend a punto de romper el blender, "The boy caught his friend about to break the blender") in contrast to bilinguals who maintain a functional separation between the two languages. Codeswitching has been studied extensively in production, particularly from structural and social perspectives. In contrast, few studies have examined the comprehension of codeswitched speech from an experimental perspective. This dissertation attempts to address this gap by employing an innovative eye-tracking methodology known as the visual world paradigm (Cooper1974; Tanenhaus et al., 1995) to examine the role of grammatical gender in the processing of Spanish-English codeswitched speech. We focus on grammatical gender because researchers have observed a production asymmetry in its use in Spanish-English codeswitching. Specifically, article -- noun constructions (Mixed NPs) frequently appear as Spanish masculine articles followed by English nouns regardless of the gender of the Spanish translation equivalent, e.g. el juice, Sp. el [masc] jugo [masc] and el cookie, Sp. la [fem] galleta [fem]. Alternatively, Mixed NPs with Spanish feminine articles are infrequent and restrictively appear with English nouns that have feminine Spanish translation equivalents, e.g. la cookie, Sp. la [fem] galleta [fem] but *la juice, Sp. el [masc] jugo [masc]. This observation leads us to propose that codeswitching is an ideal test case to investigate the direct link between production and comprehension. Working within the Production-Distribution-Comprehension (PDC) framework (Gennari & MacDonald, 2009), we hypothesize that the gender production asymmetry reported in the production of codeswitched speech should be reflected in comprehension. To investigate this hypothesis, we first quantified the distribution of Mixed NPs using a bilingual spoken language corpus (Deuchar et al., 2012) as a means to confirm the production asymmetry described by other researchers (e.g. Jake et al, 2002; Otheguy & Lapidus, 2003; Clegg, 2006). Validating previous observations, we found that masculine marked Mixed NPs constitute the overwhelming majority of Mixed NPs (92%). In contrast, feminine marked Mixed NPs were rare in our corpus (3%). To examine the impact of this production asymmetry in comprehension, we recruited 2 groups of Spanish-English bilinguals from City College of New York (CCNY) categorized by their place of birth (U.S. born, N = 21, v. Latin born, N = 25). Both groups of bilinguals participated in three separate eyetracking experiments within one experimental session: a Spanish unilingual block, a lexical-level codeswitching block, and a sentence-level codeswitching block. In all three blocks, participants were shown a simple 2-picture display of concrete objects. While listening to recorded stimuli in Spanish (Spanish unilingual block) or in codeswitching, a target picture was named, and participants were instructed to click on the target picture. Concurrently, participants' eye movements were recorded with a desk-mounted eyetracker at a sampling rate of 1000 Hz. We capitalized on two previous findings in the eye-tracking literature. First, Spanish gender has been shown to be used as a facilitatory morpho-syntactic cue in informative contexts (Lew-Williams & Fernald, 2007). In contrast, phonological competition (i.e. overlapping phonology in potential target candidates) is shown to delay spoken language recognition (e.g. Allopenna et al., 1998). Therefore, our experimental items in the codeswitching blocks introduced a phonological manipulations such that all paired items overlapped in initial phonology but had Spanish translation equivalents that differed in gender, e.g. candy, Sp. caramelo [masc] and candle, Sp. vela [fem]. Furthermore, in both the Spanish unilingual and lexical-level codeswitching block, target items were embedded phrase-finally in a simple carrier phrase. In the sentence-level block, target items were embedded in sentence-medial position in variable sentential contexts. In order to encourage listening for comprehension, we introduced a plausibility judgment at the end of each trial. Moreover, half of the sentences began in English and the other half in Spanish. Each member of an experimental pair was a target and was combined with both Spanish articles resulting in four experimental conditions: feminine match trials, feminine mismatch trials, masculine match trials, and masculine mismatch trials. If the predictions of the PDC framework are correct, then bilinguals should not show any facilitatory processing for masculine conditions given its documented preference as the default article in Spanish-English codeswitching. In contrast, the feminine mismatch trial should result in increased delayed processing in comparison to feminine match trials. In our analysis, we compared the proportion of fixations to target items and distractor items for each condition in each experimental block. We conducted paired-t tests on the difference between the mean proportion of target and distractor fixations in 100 msec time regions from article onset. The Spanish unilingual block served as a baseline to examine gender processing in Spanish. To further support the validity of our materials and to interpret the bilingual data, we included a control group of Spanish monolinguals (N = 24). Replicating a previous study, the Spanish monolinguals revealed the online use of grammatical gender in informative contexts as evidenced by significantly higher mean proportion of fixations to target items in different gender trials at earlier time regions than for same gender trials. In contrast, the U.S. born bilingual group showed no facilitation due to gender in spoken language processing, exhibited by a similar timecourse for both different and same gender trials. The Latin born group showed facilitatory effects only for feminine conditions. The lexical level codeswitching block revealed that U.S. born bilinguals can use grammatical gender to strongly facilitate target identification in masculine conditions. Differences were also found for feminine conditions but were mainly attributable to increased difficulty in integration of feminine mismatch targets. In contrast, the Latin born group continued to exhibit strong facilitatory effects for feminine conditions. For masculine conditions, differences were found but were mainly attributable to increased difficulty in integration of masculine mismatch targets. For the sentence-level codeswitching block, the U.S. born bilinguals showed little modulation attributable to grammatical gender except for a strong facilitatory effect for feminine mismatch targets in Spanish-first codeswitching trials. In contrast, the Latin born group did reveal differences based on language manipulation and gender. Specifically, in English-first codeswitching trials, the Latin born bilinguals exhibited a weak facilitatory effect for feminine conditions that was neutralized in Spanish-first codeswitching trials. For masculine conditions, the masculine match trials revealed a similar timecourse of processing in both English-first and Spanish-first codeswitching trials, but masculine mismatch trials were more difficult to integrate in Spanish-first codeswitching trials. We discuss the findings in terms of gender processing in bilingualism and implications for the PDC model.