Control Stimuli in Experimental Code-Switching Research

Round 1

Reviewer 1 Report

See attached document

Comments for author File: Comments.pdf

Author Response

Overall, the contribution addresses the important question of methodology issues in research. In particular, acceptability judgements in code-switching data are analyzed here, claiming that a baseline comparison of acceptability is needed in order to detect variation.

I believe the paper needs to complement and discuss some important issues in the study of code- switching, as I will address below. The test setting is constructed very neatly and seems to be very accurate. However, the main limit of the study is the composition of the participant group, which is way too heterogeneous in my view, so that the results lose credibility.

The paper thus needs major revisions.

Context-related comments

Page 2, line 76 ff: Do you only consider studies on English/Spanish? Otherwise, for adjective placement, you might also refer to Cantone & MacSwan (2009)

Cantone, K.F. & MacSwan, J. 2009. Adjectives and Word Order – A focus on Italian-German Codeswitching, In L. Isurin, D. Winford & K. de Bot (Hgg.), Multidisciplinary Approaches to Codeswitching. Amsterdam: Benjamins, 243-277.

Response: No, this was not meant to be language-specific. This reference has been added. Thank you.

Page 4, examples: Why don’t you consider d. His friends have estudiado espanol? Is it linked to your discussion in Page 7? See below

Response: I’m not sure what the reviewer means here. The hypothetical example (1d) they present is equivalent to the sentence already provided in (1a). That is to say, I do consider it.

Page 5, second paragraph: Did you consider that participants might consider something different as unacceptable than the switching point you address and thus judge some different linguistic phenomenon than the one you mean?

Response: Yes, this is why monolingual equivalents of all the structures are also tested (after completing the code-switched versions). This study follows Ebert and Koronkiewicz (2018) in that regard. No unexpected unacceptability appeared in the monolingual ratings, indicating that the acceptability should be based on the switch point. I have added more explicit information about this to clarify.

Page 5, section 3.1: In my view, 20 participants with so many differences is a way too heterogeneous group. I consider this a great weakness of the study.

Response: I agree that the group is definitely heterogeneous, and it can be considered a limitation of the study. I have added a specific paragraph about this to the discussion. I have also included analyses regarding their language profile, as requested later in this review.

 

Page 6, section 3.2: Did you ask whether participants use to code-switch themselves? What is the concrete definition of acceptable here? It sounds strange, it is possible? I believe all these different explanations are very confusing for the participants.

Response: Yes, participants were asked about their language mixing habits in the background questionnaire. I have included this information explicitly in the description of the participants. As for the definition of acceptable, there was no concrete definition provided to participants. It is true that the use of varied language to describe the task of identifying acceptable sentences may seem confusing when reading the description out of context, but such explanations were always accompanied with examples, and there was a nuanced narrative throughout the training that guided participants through understanding how to understand acceptability. I’ve included the full task training text as an appendix to help clarify. Importantly, no participants indicated they were confused about how to complete the task.

Page 7, examples 3a and b: Are you saying these examples MUST be ungrammatical or are you saying they are not possible? Is this in line with MacSwan’s model? Line 297: please cite studies on this debate.

Response: I am saying they are ungrammatical, and yes, this is in line with MacSwan’s approach to CS. I’ve added more explicit wording to clarify. Also, the references have been added to the debate regarding auxiliary verb switches.

Page 8, lines 306-307: Again, please quote WHO considers these switches as unacceptable.

Response: These references have been added.

Page 8, results: Why aren’t there any analyses with respect to the language profile? Please analyze whether the groups you make differ w.r.t. proficiency in the languages, age of acquisition, self-rated proficiency and exposure. These aspects could matter a lot!

Response: I have included the requested analyses. Most comparisons came out as non-significant, but there was a difference with regard to age of acquisition for English and Spanish proficiency score.

Comments on the format/typos:
Page 1, line 41: delete second 'that' Restructure References alphabetically

Response: This has been fixed. Thank you.

Reviewer 2 Report

Please consider my comments in the .pdf attached below.

Comments for author File: Comments.pdf

Author Response

Summary:

This paper is a methods paper that expands on previous research by González-Vilbazo et al. (2013) on methodological aspects of code-switching (CS) research, especially with the use of acceptability judgment tasks (AJTs) to examine CS speech. The paper focuses, in particular, on control stimuli, that is, the stimuli used to compare a structure whose (un)grammaticality is under investigation. The paper has three main contributions:

The research design includes three Spanish-English CS that are present in natural-speech corpora and that are largely regarded as grammatical in previous research (complex- sentence switches, subject-predicate switches, and direct-object switches) and two Spanish-English CS that are largely considered ungrammatical in previous research (pronoun switches, present perfect switches) for an initial analysis to establish if these two large categories (grammatical vs ungrammatical switches) are significantly different from each other and whether differences are found within each category (complex- sentences switches vs. subject-predicate switches). With data from 20 Spanish heritage speakers, the paper reports significant differences between grammatical and ungrammatical stimuli but no differences within grammatical or within ungrammatical stimuli. Establishing these two comparison points is an important contribution to test the grammaticality of a switch for three reasons: (i) research in the past has only compared to a grammatical sentence, thus, possibly missing a mid-tier category, (ii) some switches do not have a natural comparison, and (iii) having a standard of grammatical vs. ungrammatical sentences across studies can facilitate comparisons across studies. Additionally, this design shows that the three types of grammatical switches and the two types of ungrammatical switches were treated similarly within each category and, therefore, they can and should be used as control stimuli to have more variety in the stimuli. The paper examines individual results and uses the control stimuli to group participants into those that distinguish between grammatical and ungrammatical stimuli and those that do not. Although they do not explore further why some participants do not distinguish, this analysis provides the basis for exclusion of participants who do not respond to the format of testing of AJTs.

Broad comments:

Background and motivation: This methods paper contributes to the literature in CS in a significant way. It is well-placed in the literature as a continuation of the methods paper by González-Vilbazo et al. (2013). In particular, it identifies a gap on the development of stimuli in AJTs in CS research. The gap is clearly identified (a need for methodological solutions to address heterogeneity among the participant results and/or how to operationalize (un)acceptability) and the paper addresses the gap successfully. Nonetheless, a few references are needed to support some of the claims. In particular, in page 7, lines 297-299, the sentence can be further supported by reference to work on auxiliary verb switches more generally and switching have/haber in Spanish-English CS. In the lines that follow, claims about present- progressive switches also need references added (page 8).

Response: These references have been added. Thank you.

Methodology: The paper is methodologically sound, as it tests the relevant conditions for establishing clearly grammatical and clearly ungrammatical sentences and the benefits for having these. The design is conscientious. Using real sentences from the Bangor Miami corpus, for instance, is a strength in the design. There are few methodological issues that can be addressed as limitations, as they are minor.

For the ungrammatical sentences, it would be useful to include that these were not present in the Bangor Miami corpus, to make it comparable to the grammatical sentences, which were extracted from this corpus. The author can examine the corpus and, if there are no examples, indicate this in the description of the materials. If there are examples, a brief explanation can be included in a footnote. Response: Thank you. No instances were found in the corpus, and that information has been added explicitly. For the pronoun stimuli, since the materials were presented visually, how did the author control for prosodic stress? A clarifying footnote should be enough to explain how they assured that the pronouns lacked the defining characteristics of strong pronouns (specially for Spanish pronouns). Response: The reviewer is right in pointing out that since it is a written AJT, prosodic stress was not able to be controlled. I have added a footnote citing Koronkiewicz and Ebert (2018) who provide evidence that the modality of presentation does not affect acceptability judgments, including pronouns. Is there any reason why or any concern about all filler stimuli being subject-predicate switches? Response: This was not intentional. Thank you for pointing that out. I have added a footnote. The authors used a 7-point scale and did not include a ‘not sure’ option. Usually with uneven-numbered scales it is a good idea to have a ‘not sure’ option so that they use the middle option to indicate that it is somewhere in between completely acceptable and completely unacceptable and not that they are not sure how to rate the sentence. Response: There was a ‘not sure’ option, however, I did not say so explicitly in the text. I have added that information. Also, by adding the task training as an appendix, it should also be clearer that the participants were able to use the middle option in such a manner. There is no item analysis and a low number of items per condition (at least considering other subdisciplines within linguistics) so there may be some variability that has not been examined. For the present-progressive switches, for instance, for the distinguishing group, could some items have received higher ratings than other and the results given be the result of certain items being rated high and others low? This can fall within the purview of future research recommendations. Response: I have added item analyses. The reviewer was right in that there seemed to be one lexicalization in particular that was bringing down the ratings slightly. Thank you. Related to this idea, too, is the use of ANOVAs instead of mixed-effects models, which have become the standard in the field. The type of analysis performed in mixed-effects models addresses directly differences across individuals. A lot of the problems derived from heterogeneity can be addressed statistically with mixed-effect models but, in this case, since it is the focus of the methods paper it does not seem necessary. In the case of variation between items, though, since it is not directly examined in the paper, the author may consider using a mixed-effects model with item as a random factor. Alternatively, this issue can just be noted as a limitation. Response: Thank you for bringing up this point. As mentioned, because the focus of the paper is heterogeneity in and of itself, I have decided to include the analysis used as a limitation. Similarly, the number of participants and the heterogeneity of the group (ages and places of origin) can also be briefly acknowledged in the discussion of limitations. Response: I have added some analyses regarding language profile to help resolve this issue, but I have also added it to the limitations.

Discussion and directions for future research: With respect to present-progressive switches, more discussion could be added as to how to proceed when the three groups of participants are identified: those who accept, those who reject, and those in between. This is especially important in light of the discussion about acceptability being categorical. Additionally, some of the methodological aspects mentioned in the previous section could be discussed in this section, especially those related to limitations and directions for future research.

Response: I have added more details about this in the discussion.

Overall, the paper addresses the three aims it is set out to explore:

Explore the differences between grammatical and ungrammatical stimuli Participant heterogeneity and exclusion of participants who do not use the scale Establish a baseline comparison of (un)grammaticality that allows for a middle tier because, if no ungrammatical items had been included, the middle group would have resembled the group that rated them as ungrammatical

Although there are some minor methodological recommendations, the paper offers important design suggestions that other researchers are likely to adopt.

Response: Thank you!

Specific comments:
Text should be checked for typos. I only found a couple:

On page 2, line 78 there is a double parenthesis )) On page 11, line 419 the second ‘unacceptable’ should be ‘acceptable’ instead. Response: These have been fixed. Thank you.

Round 2

Reviewer 2 Report

The paper has improved by the additions made.

Please, have a final check on typos (see line 495, "is" is missing).