Can Google Translate Rewire Your L2 English Processing?

Abstract

In this article, we address the question of whether exposure to the translated output of MT systems could result in changes in the cognitive processing of English as a second language (L2 English). To answer this question, we first conducted a survey with 90 Brazilian Portuguese L2 English speakers with the aim of understanding how and for what purposes they use web-based MT systems. To investigate whether MT systems are capable of influencing L2 English cognitive processing, we carried out a syntactic priming experiment with 32 Brazilian Portuguese speakers. We wanted to test whether speakers re-use in their subsequent speech in English the same syntactic alternative previously seen in the MT output, when using the popular Google Translate system to translate sentences from Portuguese into English. The results of the survey show that Brazilian Portuguese L2 English speakers use Google Translate as a tool supporting their speech in English as well as a source of English vocabulary learning. The results of the syntactic priming experiment show that exposure to an English syntactic alternative through GT can lead to the re-use of the same syntactic alternative in subsequent speech even if it is not the speaker’s preferred syntactic alternative in English. These findings suggest that GT is being used as a tool for language learning purposes and so is indeed capable of rewiring the processing of L2 English syntax.

1. Introduction

Until the 1940s, translation represented the main methodological approach adopted for second language teaching [1,2]. Nevertheless, over the last few decades, the translation-oriented approach has been considered outdated and so was gradually replaced by the up-to-date direct method, in which a second language is taught without any association whatsoever to the student’s first language. The reason for this replacement, according to the direct view, is that teaching through translation would end up in false notions of equivalence as the translated language could sound artificial in the second language and would reinforce the second language processing via the first language [2,3].

Despite the criticism involving the use of translation as a means for language teaching and learning and the widespread adoption of the direct method, teachers and language instructors of the 21st century were challenged by the emergency and popularization of translation technologies such as Machine Translation systems (MT), especially with the free availability of popular tools such as Google Translate system (GT). Although the quality of the output of these systems was considered poor in their earlier stages of development [4], advances in the field of Artificial Intelligence have brought about significant improvements to the MT output leading to an increase in MT users especially in the past five years. Way [5] notes with respect to MT usage on Facebook that “with 2 billion translations being provided on a daily basis, and almost 1 billion users seeing these translations each month, the numbers are truly staggering”. According to Way [6], GT translated an average of 143 billion words a day in 2016 (20 words/day for every person on the planet), just for a single (albeit the largest) MT service provider—across 100 language combinations, a doubling in translation volume in just 4 years. Hence, in contrast to the direct method claims, the availability of good quality web-based MT has made it possible for students to use translation tools as a support for language activities involving writing, reading and (possibly) speaking. Clifford [7] emphasizes that the latest advances in this technology have the potential to influence the way students deal with second language learning, especially English learning due to its emergence as a lingua franca and the language of science and business (See [8] for a view that English will be the last lingua franca: https://www.taus.net/think-tank/articles/translate-articles/mt-the-new-lingua-franca, accessed on 13 December 2020).

Even though language teachers and instructors restrict the use of MT in the classroom, the tool is often accessed at home by students to carry out their school work [7]. Other than that, as the system is available in the form of mobile applications, it facilitates user access to translation in a communicative context. If English students, for instance, want to form a sentence in English but are uncertain as how to precisely structure the sentence, they can quickly translate the sentence using an MT application in their mobile device and be more confident in speaking. Thus, what we observe is a dilemma in which teachers, language instructors and academic institutions – based on the direct method claims—do not approve the use of translation as a means for language teaching and learning, while in contrast we see a widespread use of web-based translation tools that stimulate learning through translation.

Considering this dilemma, a question that deserves to be investigated is whether language learning is really compromized by the popularization and widespread use of MT tools. Although MT systems instantaneously produce a translation that can be rapidly processed by users, it is possible that MT users (un)consciously learn from that very MT output. According to several second language processing models, the bilingual mind has to operate simultaneously in two languages that are in competition during the process of language comprehension and production. Thus, this conflict must be resolved for successful communication to take place [9]. We question here whether the use of MT systems is facilitating the resolution of this conflict, more specifically, facilitating the processing and acquisition of English syntactic structures.

We will focus in the present study on the English syntactic processing as the generation of syntactic structures in the second language is, as already previously outlined, challenging to speakers because it is a process that involves two languages in competition and, in many cases, syntactic structures that vary between a speaker’s first and second language. Moreover, in order to communicate a message either in one’s first or second language, more than one competing syntactic alternative could be used by speakers in a communicative context. For example, a speaker can deliver a message in English using both a passive structure (e.g., the boy has been kissed by the girl) or an active structure (e.g., the girl has kissed the boy). Another example would be verb phrases formulated with double objects (e.g., the boy gave the girl a book) versus prepositional object (e.g., the boy gave a book to the girl). Although these syntactic alternatives vary in terms of word order, they convey roughly the same meaning. In the second language, however, this process is exacerbated by the fact that some syntactic alternatives in the second language are not available in the first language. Thus, in these cases, during the processing of the second language, two conflicts must be resolved: (i) the competition between languages; and (ii) the competition between syntactic alternatives. At higher levels of language proficiency, adequacy and fluency of the message conveyed depend on the speakers’ language proficiency, while the syntactic alternative chosen depends not only on the speakers’ language proficiency but also on the context. The context involves the syntactic alternatives to which the speaker has been previously exposed to. Therefore, we believe it is worth investigating whether the interaction with web-based MT systems facilitates the processing of English syntax, especially the processing of syntactic alternatives that vary across a speaker’s first and second languages, thus posing to L2 English speakers additional processing challenges. We will examine how exposure to the MT preferred syntactic alternatives can influence a speaker’s syntactic choices while, at the same time, triggering syntactic learning if the syntactic alternative represents a challenging structure to be processed.

Some studies [10,11] show that, in the initial stages of second language acquisition, learners rely more on the lexical memory of single words, so they find difficulties and less automaticity in language tasks that involve proper combination of words in a sentence. Accordingly, it is at this stage and for that purpose that MT systems represent a source of support. Garcia and Pena [12] showed that MT systems help beginners to communicate more in a foreign language and that, the lower the L2 level, the greater the difference between the number of words composed using MT and the number of words written directly in the L2. Therefore, this research will help us to understand whether MT systems are facilitating the acquisition of a fundamental ability for language learning: the ability to combine words in a sentence to properly convey ideas in a second language. Moreover, we aim at demystifying the assumption that MT systems are preventing students from progressing towards autonomously mastering English as a second language [2]. We hypothesise that the MT output functions as a template for the re-use of syntactic structures, triggering in the subsequent utterance a syntactic learning process of that structure, especially those that differ between a speakers’ first and second language.

To test our general hypothesis, we examined the linguistic behaviour of Brazilian Portuguese speakers of English as a second language (L2 English) before and after performing a translation task involving sentences containing Prepositional Noun Phrases (e.g., The cover of the book is red) and the alternative Non-Prepositional Noun Phrase (e.g., The book cover is red) using an MT system. We chose to examine this language combination due to the differences in this syntactic construction between English and Portuguese. We address the following research questions:

• Do L2 English speakers adapt their language behaviour to mirror the syntactic alternative of the MT when speaking in English?
• If so, what are the factors that influence such behaviour?

To answer our research questions, we first carried out a survey with the purpose of (i) understanding if and how MT systems were being used as a tool to support language learning; and (ii) if so, which web-based engines were the ones most frequently used. After conducting this survey, we carried out a syntactic priming study using a pre-test, priming, post-test design which allowed us to observe speakers’ linguistic behaviour before and after using an MT system. With this design, it was also possible to examine if any learning trends took place with continuous processing of the challenging syntactic structure in English, and if this learning was of an implicit (unconscious) or explicit (conscious) nature. In addition, we aimed at investigating whether any learning trends observed varied as a function of our participants’ English proficiency levels.

Before presenting our methodology in detail and the results of the two studies, we describe in the next section how the syntactic priming methodological approach has been used by researchers to investigate language syntactic processing both in the first and second language as well as in the interaction between humans and intelligent systems.

2. Syntactic Priming

Syntactic priming (also known as “structural priming” or “structural alignment”) can be defined as the facilitation of syntactic processing as a consequence of repetition of syntactic structures across consecutive sentences [13,14]. In language production experiments, the syntactic priming effect has been shown to manifest itself behaviourally as an increased tendency to re-use syntactic structures that have been produced either by the speaker or an interlocutor. This tendency to re-use the same syntactic structure previously processed has been demonstrated experimently for different structures [15,16,17,18], in different languages [19,20] and using different priming modalities [21,22]. Several studies also showed syntactic priming evidence between human interlocutors in L2 interactions [23,24,25]. The results of these studies have revealed that less frequent syntactic structures prime more than more frequent structures do [26]. For example, passive structures which are less frequently used by English and Dutch speakers prime more than active structures that are more frequently used [15]. Some researchers call this effect the “inverse preference effect” and they claim that the most uncommon structures drive the priming effect [27].

The first laboratory study that investigated the effect of repetition of the syntactic structure was conducted by Bock [15]. Bock questioned whether the phenomenon was related to the maintenance of the syntactic structure representation in short-term memory or to implicit learning processes typical of a procedural mechanism. To address this issue, an experiment was conducted in which subjects listened to a sentence and had to describe an event represented in an image. The sentences contained four types of syntactic structure, namely: sentences with transitive verbs in the active voice (e.g., One of the fans punched the referee) and passive (e.g., The referee was punched by one of the fans); sentences with dative constructions containing double objects with (e.g., A rock climber sold some cocaine to an undercover agent) and without a preposition (e.g., A rock climber sold an undercover agent some cocaine). The images presented to the subjects for description varied in relation to whether the agent and patient of the events were animate or inanimate [±human]. The results revealed a tendency of English speakers to use the same syntactic structures of the phrases heard, i.e., active voice when listening to active voice, passive when exposed to passive etc., regardless of the order of the words in the sentence, the relationship between the event presented in the figure, the sentence heard or semantic factors such as the gender of the agent and the patient of the event. Based on these results, Bock concludes that the effect of repetition did not seem to be related to the temporary storage of the representation of the elements of a structure in working memory. In contrast, the effect of repetition seemed to be associated with more abstract processes typical of an independent mechanism of implicit nature responsible for syntactic processing. This suggests that the maintenance of the syntactic representation in short-term memory is not sufficient to explain the duration of the repetition phenomenon in long intervals of time.

Despite the differences in the methodological approach used in the studies mentioned as well as differences between the situations in which the phenomenon of repetition occurs, what these studies have in common is the influence exerted by the previous structure in the processing of the next structure. Given the presentation of a given syntactic structure, the chances that speakers produce sentences using that same structure increase. Traditionally, in laboratory studies that address the effect of repetition on syntactic processing, the sentence preceding the production event is called the prime, and the sentence to be produced is called a target. Therefore, if the production of the target is influenced by the processing of the prime structure, then the effect of syntactic priming is apparent. The syntactic priming effect refers thus to the facilitation of the processing of a syntactic structure as a result of the previous presentation of the same structure [28,29]. This facilitating effect of the structure presented above (prime), on a later structure (target) has been attributed to the reduction of the resources recruited for the processing of the target sentence [15].

The syntactic priming effect has also been shown to manifest itself between humans and computers [30]. Using a game in which a participant and their computer partners describe and match pictures, Cowan and colleagues [31] found syntactic alignment in human-computer speech-based interactions for both dative structures (e.g., give the waitress an apple vs. give the apple to the waitress) and noun phrase structures (e.g., a purple circle vs. a circle that is purple). This demonstrates that a computer system can also influence a speaker’s grammatical choices in speech-based interactions. Virtual reality studies [32] have also demonstrated syntactic alignment between humans and computer avatars for both passives and active structures.

Suzuki and Katagiri [33] have found prosodic alignment between humans and computers. In their experiment, people exhibited alignment of loudness and response latency in their speech in response to computer-generated speech. Oviatt and colleagues [34] found that children talking to computer partners spontaneously adapt several basic acoustic and prosodic features of their speech by 10–50%, with the largest adaptations involving utterance pause structure and amplitude. The study by Stoyanchev and Stent [35] observed both lexical and syntactic alignment between humans and the Let’s Go! system, which is a telephone-based spoken dialog that provides information about bus routes in Pittsburgh.

From the findings presented above, it is possible to conclude that, in an interactive context, speakers tend to syntactically align with their interlocutors both in first and second languages as well as with computer partners. Importantly, these studies show that syntactic priming plays a central role in successful communication since it can promote mutual understanding through the semantic and structural representations shared by interlocutors [36].

The effect of syntactic priming has also been extensively observed not only in speech-based interactions, but also in the context of sentence comprehension. In general, these experiments deal with the reading of pairs or sequences of sentences that share the same syntactic structure. In such sequences, the first sentence is the prime and the second (or other sentences) the target. The purpose of these experiments is to investigate whether the cost of processing the target sentence is reduced in the face of the prime presentation. This reduction in the cost of processing was revealed by some behavioural studies whose results showed that target sentences are read faster than sentences that were not preceded by a prime. Traxler and Tooley [37], for example, in two experiments using an eye tracker and two experiments using the self-paced reading paradigm, investigated the processing of sentences containing reduced subordinate sentences in English. The results showed that the processing of subordinate sentences is facilitated (as reflected by reduced fixation time and higher reading speed) when sentences are preceded by others with the same language structure. This reduction in the processing costs of sentences preceded by a prime has also been observed in electrophysiological studies. The study by Ledoux and colleagues [38] indicates that the effect of prime on the comprehension of sentences is manifested in the attenuation of the amplitude of event-related potentials components associated with the processing of target sentences. In particular, it is also often associated with processes of reanalysis and repair of the syntactic structure [28,39].

It is noteworthy that the priming effect, both in speech-based interaction and in contexts of sentence comprehension, is magnified when the main verb of the prime sentence is repeated in the target sentence. In sets of sentences sharing the same syntactic structure, the reading time or fixation time of reading decreases if they contain the same structure and the same verb. This phenomenon is known as lexical boost [29]. In contexts of sentence comprehension, the effect of priming can only be observed if the main verb is repeated between the prime and the target(s) as the overlap of elements between prime and targets reinforces the activation of structural patterns present in prime processing, leading to a reduction in the cost of processing of the target sentences [40].

Within the literature addressing syntactic priming, a question that has been frequently discussed is: what does the reduction in the cost of processing as a consequence of syntactic priming reflect? Most researchers assume that the effect of syntactic priming reflects learning automaticity [15,17,27]. They explain that, when processing a syntactic structure, speakers of a language implicitly learn this structure, that is, they learn a type of abstract knowledge that is acquired unconsciously as an incidental consequence from the execution of some task. Thus, for this processing perspective, the prime would be responsible for activating changes in the system responsible for implicit learning processes. When reading, listening to or even uttering a sentence with a particular structure, the speaker activates and shapes this cognitive system.

If the effect of syntactic priming is the result of an implicit learning process and this process is established not only in production, but also in the reading comprehension of sentences, it is worthwhile reflecting on the following questions: (1) When reading the MT output, can a learning process emerge from the output comprehension? (2) If so, is this learning of an implicit (unconscious) or an explicit (conscious) nature? Investigating this question will bring insights into the role of intelligent systems has on the learning process of a second language.

Another important issue that should be considered in our investigation concerns the dependence on the effect of syntactic priming of the frequency of occurrence of the structure in the language. For example, in English and Dutch, active sentences are more frequent than sentences in the passive voice. In English, 88% of sentences occur in the active voice. In Dutch, this proportion is even higher, around 92% [41]. Several experiments have revealed that the effect of syntactic priming is more evident for the less preferred structure of these languages, such as passive sentences [15,32,42,43]. If a given structure is less frequent in the language, then the effect of syntactic priming associated with that structure increases. Jaeger and Snider [44] explain that the inverse effect of syntactic priming observed in the choice of a structure for production is the result of unexpected changes in the context of communication that shape the system responsible for implicit knowledge. In contrast, some production studies [29,41] showed that speakers are faster in producing the preferred structures than non-preferred structures. These findings suggest that, in fact, linguistic knowledge is obtained through a cognitive mechanism responsible for implicit learning, and that this mechanism can be modelled both by the linguistic experience acquired throughout life as well as by recent experiences. As shown, although the inverse preference effect was often observed in the examination of syntactic priming, most studies focused on English, Dutch and (to a lesser extent) German. To the best of our knowledge, there are no studies to date that have used the syntactic priming methodology as a vehicle for investigating the processing and representation of Brazilian Portuguese syntax in the context of a human-artificial system interaction. Therefore, it is also necessary to investigate whether the inverse preference effect widely observed in languages such as English, Dutch and German also occurs in the syntactic processing of Portuguese, especially in the processing of the comprehension of sentences in Brazilian Portuguese.

Before describing our experimental design and methodological approach, we present the results of a survey carried out with Brazilian Portuguese MT users which sought to understand if they have used or are using MT systems as a tool supporting their language learning process. We also investigate in this survey how students are interacting with the MT systems, what their preferred systems are and why they are using MT.

3. Materials and Method

3.1. Survey

The survey was conducted with 90 (62 women) Brazilian Portuguese L2 English speakers at all English proficiency levels living either in Brazil (45 respondents), Ireland (38 respondents) or other countries (7 respondents). Half of the respondents (45) reported to be attending English classes at the time of the survey completion and the other half reported to have studied English in the past. All respondents reported to use or have previously used MT systems for translation tasks involving English—Portuguese language pair. The survey was carried out using the Typeform (https://www.typeform.com/, accessed on 1 March 2021) online questionnaire. Respondents were recruited through posts on social media, e-mails, posters at English Schools in Dublin and word of mouth. Participation in the study was voluntary. The first set of questions was created to collect participants’ demographic data and English proficiency levels. Below we present the results of the survey.

Results

Table 1. Percentage of respondents’ English proficiency levels.

Advanced	Beginner	Intermediate	Not Sure
35	13	33	9
38.9%	14.4%	36.7%	10%

presents respondents’ English proficiency levels based on their self-evaluation.

Most of our respondents were women at the advanced and intermediate English proficiency levels and reported to have studied English at English Schools (78.9%). As shown in Figure 1, when we asked these MT users which system they use frequently or had previously used, 94.4% (85 respondents) reported to use or have used GT. Only one person reported to have used Translate.com, while 4 respondents reported to have used DeepL (deepl.com, accessed on 1 March 2021) or confused MT systems with dictionaries and aligned corpora such as Linguee (linguee.com, accessed on 1 March 2021).

We asked respondents to rate the frequency of usage of the MT tool based on a 5-point Likert scale, with 1 indicating “very rarely” (about 3 times in a year), 2 indicating “Rarely” (between 5 and 10 times in a year), 3 indicating “Sometimes” (one to three times in a month), 4 indicating “Frequently” (one or three times in a week) and 5 “very frequently” (every day). Most of the respondents (63.3%) reported to use MT systems frequently or very frequently.

Table 2. Frequency of MT usage (Google Translate).

Responses	Very Rarely	Rarely	Sometimes	Frequently	Very Frequently
Count	5	8	20	36	21
Percentage	5.6%	8.9%	22.2%	40%	23.3%

displays the percentages of frequency of MT usage.

When we compared the frequency of usage across respondents’ proficiency levels, we observed that MT tools are more frequently used at intermediate and advanced proficiency levels.

Table 3. Frequency of MT usage per English proficiency levels.

	Very Rarely	Rarely	Sometimes	Frequently	Very Frequently
Advanced	0%	5.6%	12.2%	14.4%	6.7%
Beginner	1.1%	1.1%	0.0%	7.8%	4.4%
Intermediate	2.2%	1.1%	7.8%	14.4%	1.1%
Not sure	2.2%	1.1%	2.2%	3.3%	1.1%

illustrates this trend.

Regarding the purposes of use, we asked participants if they used MT systems more frequently to translate sentences or words.

Table 4. Frequency MT usage per task.

Task	Very Rarely	Rarely	Occasionally	Frequently	Very Frequently
Translate sentences	16.70%	13.90%	33.30%	21.10%	15%
Translate words	9.50%	13.40%	30%	28.90%	18.40%

shows that they use the systems for both tasks, but more frequently to translate words, suggesting that language learners also use the systems as a bilingual dictionary.

We also investigated whether the systems are being used as a tool supporting communication in English. To this end, we asked participants if they use or have used MT systems to help them to speak or talk to someone. As displayed in

Table 5. Responses to the question: Have you ever used a translation system to help you speak/talk to someone in English?

	Count	Percentage
Yes	50	55.6%
No	40	44.4%

, 50 respondents (55.6%) out of 90, answered Yes to this question.

Among these respondents, we asked how the system helped them to speak or talk to someone, (Why have you used translation systems to help you talk/speak to someone in English?), most of the respondents (56%) answered that they had used the systems because there are words that they did not know in English and 52% reported to have used the systems to confirm if the sentences that they wanted to speak were correct, while 38% answered that they have used the systems to help them to construct a sentence in English. In addition, when we asked how MT systems helped them to speak in English, most respondents (76%) answered that MT systems helped to confirm whether the sentence they wanted to use was correct, and to translate words that they did not know.

Table 6. Responses to the question: Why have you used translation systems to help you talk/speak to someone in English?

Options	Percentage	Count
Because there are English words that I do not know	56%	28
Because I am not sure if the sentence I want to speak is correct	52%	26
Because I do not know how to form a sentence in English	38%	19
Other	8%	4

shows count and percentage of participants’ responses to the question.

From the results reported above, we can conclude that MT systems are no longer being used as a tool for gisting purposes only, but mainly as tool that helps users to communicate in a second language through tasks involving lexical knowledge (vocabulary) and syntactic knowledge (e.g., tasks involving sentence construction) in the second language. Importantly, as 76% answered that the systems helped to confirm sentence structure in English to speak to someone, we can assume that Brazilian L2 English speakers trust GT in these tasks. Thus, it is worth exploring whether the interaction between L2 English speakers and the popular GT can result in changes of the mental processing of English such as learning of syntactic alternatives that are challenging to process due to differences in word order between the source and target languages. It is worth noting that we recruited participants that were representative of the population that completed the survey. For instance, we recruited participants at the intermediate and advanced English levels following the results of the survey which revealed that GT is more frequently used by English students at these proficiency levels. Moreover, we included in our final dataset participants that answered to use GT to help them to speak to someone as well as for translation tasks involving English syntax (e.g., to confirm whether the sentence they want to speak or write is well formed in English).

In the next section, we present the methodology and results of the syntactic priming experiment carried out to investigate the role of MT systems in the cognitive processing of L2 English.

3.2. Syntactic Priming Study

Using a traditional syntactic priming experimental design [25], we investigate whether using GT for tasks involving translation of sentences from Portuguese into English could result in changes in the mental processing of L2 English, as reflected by adaptation of speakers’ preferred syntactic alternative to mirror the GT preferred syntactic alternative. We also looked for learning trends in the data by examining whether participants adapted their behaviour during the course of the experiment and whether this adaptation was short-lived or a long-lasting effect.

The experiment was designed to be conducted in three phases: pre-test, priming, and post-test phases. The pre-test phase was considered the baseline as in this phase participants were not influenced by the MT output when translating sentences from Portuguese into English. The priming phase involved a task in which participants were requested to translate sentences using GT from Portuguese into English and, immediately after this task, they were requested to describe images in English.

The rationale behind our experiment was the following: if participants described the images in English using the same structure previously seen in the output of GT more frequently than the structures used in the pre-test phase (which does not involve any interaction with GT), then our results would suggest that GT is capable of influencing participants’ grammatical choices. If we observe that the use of the challenging structure increases in the later trials with continuous exposure to the same structure leading participants to adapt to that structure in the course of the experiment, this would suggest that an implicit learning process took place in the course of the experiment as every instance of syntactic structure updates the speaker’s knowledge of that structure. According to some researchers [15,44], learning occurs because speakers adapt to the context with the aim of reducing errors, and use all the information available to them for this purpose. The post-test phase was carried out in the day following the pre-test and priming phases. It was included in the design aiming at examining whether any learning of the structures is a short-lived or long-lasting effect. If we observe a long-lasting effect, then we can conclude that a learning processing occurred during the priming phase and that this learning if of implicit nature.

Based on the results reported in the literature as well as on the results of our preliminary experiments [45], we hypothesise that will we see a priming effect emerging in the course of the experiment. If we observe such an effect, then we can assume that GT is playing a role in the cognitive processing of English syntactic structures.

3.2.1. Participants

We analyzed data from 40 Brazilian Portuguese L2 English-speaking volunteers recruited through posts on social media or through posters distributed in English schools in Dublin, Ireland. All participants recruited were native speakers of Portuguese and were living in Dublin at the time of data collection. In return for taking part in the experiment, all participants received a 10 $ϵ$ voucher.

We removed data from 8 participants as they reported to have had difficulties in the execution of the experimental tasks as they got sleepy or too tired during the session. Thus, our final sample included data from 32 participants (11 men, mean age = 33.8—sd = 5.8), all of whom were requested to read a plain language statement and to sign the informed consent form in order to take part in the experiment. All participants included in our final sample reported to use or have used GT as a tool supporting spoken English and were at the intermediate or advanced English levels according to the online English test (https://www.cambridgeenglish.org/test-your-english/general-english/, accessed on 1 March 2021) with maximum score 25 and minimum score 1 (mean = 13.7; SD = 4.4). The English proficieny test was taken by all participants on the second day of the study after completion of the post-test phase. In addition, on the second day, all participants were requested to complete the survey presented in Section 3.1. As the survey was an optional task, the survey questionnaire was filled by 30 out of 40 participants recruited after completion of the post-test phase.

3.2.2. Materials

Baseline pre-test. For the pre-test phase, we selected 26 images from an online image repository (https://elements.envato.com/, accessed on 1 March 2020) depicting 26 Portuguese sentences. 6 images and 6 sentences (30% of the materials) were used as filler trials, i.e., trials that were not of interest to the experimenter but prevented participants from guessing what the experiment was measuring. Thus, the actual trials of the pre-test phase were constructed using 20 images depicting 20 Portuguese sentences. The 20 sentences of the experimental trials expressed a relation of possession between nouns and they were composed of a noun phrase + “to be” + complement (e.g., a janela do escritório está quebrada) which could be translated by participants into English using either a Prepositional Noun Phrase Structure (PNP) such as The window of the office is broken or a Noun Phrase Structure (NP) such as The office window is broken. The 6 filler trials were constructed using images depicting sentences composed of subject + to be verb + complement (e.g., A porta está trancada “the door is locked”). Therefore, compared to the experimental trials, filler trials did not contain a noun phrase as subject that could be translated from Portuguese into English using an NP or PNP structure.

Both experimental and filler trials were presented to participants in random order. Figure 2 illustrates the trials presented to participants in the baseline pre-test phase.

Priming test. From the same image repository, we selected 78 images depicting 78 Portuguese sentences. These 78 items were used to construct 26 prime-target triplets. As in the pre-test phase, 20 out of the 26 trials were experimental trials while the remaining 6 trials were filler trials (30% of the experiment trials). Figure 3 illustrates the 20 experimental trials presented to participants in the priming phase, and Figure 4 illustrates the 6 filler trials presented in the priming phase. Both filler trials and experimental trials were presented to participants in random order in the priming test phase.

Prime-target triplets. The prime-target triplets were composed of two prime items preceding a target item. To create the 20 prime-target triplets, we used a total of 60 sentences and 60 images depicting those sentences. In addition, we used 18 sentences and 18 images depicting those sentences to create the 6 filler prime-target triplets.

The two prime items preceding the target of the 20 experiment trials consisted of a sentence in Portuguese composed of a noun phrase expressing a relation of possession between nouns + ‘to be’ + complement (e.g., A capa do livro é vermelha—“The book cover is red”) which were presented to participants to translate from Portuguese into English using GT on their own mobile device. The target items consisted of an image and three English words (randomized across the target trials) presented above the images. The two prime items preceding the targets of the remaining 6 filler trials consisted of sentences in Portuguese composed by a subject + an intransitive verb (e.g., The man is writing) presented to participants to translate from Portuguese into English using GT on their own mobile device. The target consisted of an image for description in English and an intransitive verb (e.g., knitting) appearing on top of the image. Prime-target triplets and filler prime-target triplets were intertwined in the priming test phase and randomized across participants.

Post-test phase. Materials used in the post-test phase were the same as those used in the pre-test phase.

3.2.3. Procedures

The stimuli of all phases were presented on a computer screen using Psychopy software 3.0. (https://www.psychopy.org/, accessed on 1 March 2021) During the pretest and priming test phases, all verbal responses were recorded on Quick Player voice recorder.

Baseline pre-test phase. The baseline pre-test was presented to participants before the priming test phase. Experimental trials and filler trials were randomized across participants. In this phase, participants were instructed to speak out loud their translation version of the sentences from Portuguese into English using the English words presented at the bottom of the screen which were introduced in order to avoid lexical retrieval issues in L2 during the task. All sentences could be translated from Portuguese into English using either a PNP structure or a NP structure.

Priming test phase. In the two prime items preceding the target item, participants were requested to translate the prime sentences appearing above the images using GT on their own mobile device, and to read the translation out loud in order to trigger the syntactic priming effect. Immediately after machine-translating the prime sentences, participants were presented with the target item and were instructed to describe the image on the screen with a simple sentence using the three words appearing on top of the image (experiment trials) or the intransitive verb (filler trials). Participants were also instructed to avoid including words that were not on the computer screen, or to avoid describing the images using prepositions of location (such as in, on, at, etc). All images could be described using either an NP structure or a PNP structure.

Post-test phase. The post-test phase was carried out for all participants on the day following the pre-test and priming test-phases (24 hours after the start of the pre-test phase). The procedures of the post-test phase were the same as those used in the pre-test phase. As in the pre-test phase, experimental trials and filler trials were randomized across participants. After completion of the post-test phase, all participants were thoroughly debriefed by the experimenter who explained the objectives of the study, the experimental design and thanked them for taking part in the research. In addition, participants were requested to complete the Online English proficiency test. They were also asked if they could complete the survey questionnaire.

Training session. Two prime-target triplets in the two conditions (experiment and filler condition), not included in the main experiment, were used for training participants prior to the start of the priming test phase.

Table 7. Overview of the experimental design.

Day	Phases	Task
1	Pre-test	Translate sentences from Portuguese into English
	Priming test	Translate sentences via GT and describe images in English
2	Post-test	Translate sentences from Portuguese into English
	English proficiency test	Complete English proficiency test with 25 questions
	Survey (optional)	Complete survey

presents an overview of the tasks completed by participants in all experimental phases:

3.2.4. Hypothesis Based on GT Output

All sentences used in the experiment were previously machine-translated using GT on different days and at different times. We observed that the system translated all the sentences using an NP structure. We were aware, however, that GT output could vary across experiment trials during sessions, but as GT’s preferred structure is a challenging structure for L1 Portuguese speakers to process in English, we wanted to test whether GT output would be capable of eliciting priming effects even if in a few cases the system outputs an alternative structure.

Based on the results of our preliminary experiments [45], we hypothesise that in the pre-test phase, participants will translate the Portuguese sentences into English using a PNP structure as it resembles Portuguese in terms of word order, but they will mirror the GT syntactic alternative after a translation task using the system, thus manifesting syntactic priming effects. Furthermore, we hypothesise that this change in language behaviour will occur in the course of the experiment with continuous exposure to the same syntactic alternative, indicating that a learning process took place throughout the experiment session. In addition, we hypothesise that in the post-test phase, participants will use more NP structures than in the pre-test phase, indicating a persistence of the priming effect.

3.2.5. Results

First of all, it is worth recalling that we are testing whether GT output is capable of eliciting a syntactic priming effect manifested as the re-use of the same syntactic structure seen in the GT output when describing images in English. We are also investigating whether any priming effect observed would result in the learning of the primed structures via GT output. To search for learning trends in our data, we tested whether the priming effect would last until the day following the priming session.

Confirming our hypotheses, we observed that PNP represents the participants’ most preferred structure to describe a relation of possession between nouns in English. Without any influence of GT output, in the baseline pre-test phase, participants translated Portuguese sentences into English using more frequently PNP structures than alternative structures. Nevertheless, after exposure to the NP structure through GT output (priming phase), the amount of PNP produced by participants drops (from 33.1% in the baseline pre-test to 11.1% in the priming test) while the amount of NP structures increases (from 25.9% in the baseline pre-test to 51.1% in the priming phase). In the post-test phase, it is also possible to observe a similar pattern when compared to the baseline pre-test, i.e., decrease in the production of PNP structures and increase in the production of NP structures. This long-lasting priming effect suggests that syntactic priming in L2 English learners results in the implicit learning of structural representations.

It is also noteworthy that participants also produced frequently NP structures, but using the Portuguese word order which is incorrect in English (e.g., the cover book is red). We observed that this structure was produced by participants in an attempt to produce the NP structure seen in the GT output in the correct order. This behaviour suggests that participants were primed by the GT output (Although we have coded this data as “Other” (not primed)), but did not produced the NP in the correct order due to the lack of automaticity in the English syntax.

Table 8. Percentages of all structures produced by participants distributed by experimental phases. Noun Phrase Structure (NP), Prepositional Noun Phrase Structure (PNP), Genitive case (S), any other structure produced (Other).

	Pre-Test	Priming	Post-Test
NP	25.9%	51.1%	45.8%
PNP	33.1%	11.1%	20.8%
S	19.4%	9.5%	11.7%
Other	21.6%	28.3%	21.7%

displays the percentages of all structures produced by participants and their distribution across the experimental phases.

Another interesting result is that, as predicted, the average number of NP structures produced occurred in the course of the experiment trials, indicating that participants adapted to the structure as the amount of exposure to that structure via GT output increases over time as the experiment proceeds. Figure 5 illustrates this trend.

Our data also shows that participants with higher English test grades (higher English proficiency) were more primed by the GT output, i.e., produced more NP structure in target items than participants at lower levels of English proficiency which produced more PNP or other structures. Figure 6 illustrates this trend.

Interestingly, as we can see from Figure 6, the production of NP structures in the baseline phase was associated with higher English test grades (grades greater than 15). However, in the priming and post-test phases, the grades decreased (English test grades around 15), indicating that participants at intermediate proficiency English levels started to learn from the prime sentences and produced more NP structures. In contrast, participants with English test grades below 15 produced more PNP structures in the three phases of the test, suggesting they were less prone to being influenced by the GT output.

3.3. Data Coding

Following Cowan et al. [31], during experimentas sessions, the experimenter noted the syntactic structure (NP, PNP, S or Other) used by participants to describe the target images in English. A description was noted as a “PNP” if the image was described using the preposition of (e.g., the table of the office); as “NP” if participants used a noun phrase structure in English without the preposition in the correct word order (e.g., The office table) mirroring the GT output, and "other" if they used a noun phase structure in the incorrect order (e.g., the table office), ’S to express a relation of possession between nouns (e.g., the office’s table) or if they came up with any other structure (e.g., the table in the office) that differed from GT preferred syntactic alternative. These data were then considered the categorical dependent variable used for data analyses.

All experimental sessions were audio recorded. The recordings of the sessions allowed a sanity check carried out by an independent blind rater to make sure that all data was correctly coded by the experimenter.

In the baseline pre-test phase, we coded sentences translated with noun phrases as “1” and sentences translated using PNP or any other structure as “0”. Based on the observation that GT translated the sentences using a PNP structures in only 8.5% of the prime trials (105 trials out of 1240 trials), we considered that participants were primed by the GT if they produced NP structures and not primed if they produced PNP structures or other. Thus, we coded as “1” if participants re-used the noun phrase structures when describing images in English, indicating that they had been primed by the GT preferred syntactic alternative. We coded as “0” if participants produced a PNP structure or produced any other structure, indicating that they had not been primed by the GT output. As in the baseline pre-test, in the post-test phase sentences translated by participants using a PNP or any other structure were coded as “0”, and sentences translated using a NP structure were coded as “1”.

This design allowed us to examine whether participants changed their linguistic behaviour as reflected by the re-use of GT’s preferred syntactic alternative from the pre-test phase to the priming phase as well as from the pre-test and priming phases to the post-test phase.

Statistical Analysis

We analyzed 60 data points per participant (20 data points collected from each test phase), totalling 1920 data points (60 × 32 = 1920). Data points from filler trials were not included in the analysis as they were introduced in the experiment only for distraction purposes.

The analysis was carried out in the R language [46] package lme4 [47]. We used a logit mixed effects maximal model including as random effects participants and items. The fixed effects included in the model to explain the dependent binary variable Prime were: English Test Grades (continuous), Items number (continuous), Items identity (factorial) with levels from 1 to 20, Test (factorial) with three levels: baseline (pre-test), priming test and post-test. To analyze whether the GT output that diverged from its preferred syntactic alternative (NP) (e.g., output sentences in English translated from Portuguese using a PNP structure with "of" or any other preposition) influenced participants’ syntactic choices, we included two more factorial variables: Prime 1 and Prime 2 with levels NP and PNP. All factorial predictors were dummy coded (all means compared to the baseline group) and all continuous factors (English Test Grade and Items number) were centered.

To select the model that best fits our data, we adopted a backwards procedure starting with a minimum model. Then we included all possible interactions as well as all possible random slopes one by one. Model comparisons were made using a likelihood ratio test to verify if each of the random slopes and each of the interactions added to the minimum model were justified [48]. As a result, we removed random intercepts for items as this factor did not explain any variance in the participants’ responses. Following Bar et al. [49], we chose the maximal model that best fitted our data, as shown in

Table 9. Mixed-effects logit model that best explains our dataset. <0.05 * <0.01 ** <0.001 ***.

Random Effects:
Groups Name	Variance	Std.Dev.	Corr
Participants (Intercept)	15.887	12.604
Testposttest	30.652	17.508	0.18
Testpriming	0.4144	0.6437	−0.55–0.10
Fixed Effects:
	Estimate	Std. Error	z value	Pr (>|z|)
(Intercept)	−142.839	0.25633	−5.572	2.51 × 10${}^{-8}$ ***
EnglishGrade	106.742	0.26386	4.045	5.22 × 10${}^{-5}$ ***
Post-test	111.068	0.36157	3.072	0.00213 **
Priming test	139.187	0.48106	2.893	0.00381 **
Items	0.15073	0.05689	2.649	0.00806 **
Prime 1	0.03179	0.31701	0.100	0.92012
Prime2	0.12538	0.34094	0.368	0.71306
EnglishGrade:Posttest	0.06076	0.38903	0.156	0.87588
EnglishGrade:Priming	−0.50047	0.20684	−2.420	0.01554 *

.

We observe that participants’ syntactic choices in the priming test and post-test differed significantly from the baseline pre-test phase (p < 0.001). The negative estimate for the intercept indicates that, in the baseline condition, responses using PNP structures were more frequent than in the priming and post-test phases. Thus, after the prime items in which participants were exposed to NP structures via GT output, participants produced more NP structures than in the baseline pre-test with no influence of GT output. Factor Items was also significant (p < 0.01), confirming thus the effect observed in Figure 5 which shows that, at later trials, the amount of NP structures produced by participants increases.

The model also shows a significant effect of participants English test grade (p < 0.001) and a significant interaction (p < 0.05) between English test grade with Priming test. This interaction showed that participants at higher English proficiency levels (as reflected by higher English test grades) were more primed by the GT output than participants at lower levels of English proficiency. Importantly, the same significant interaction, however, could not be observed between factors English Test Grade and Post-test. This lack of interaction suggests that, on the following day, participants of all English proficiency levels were subject to a long-lasting priming effect.

The model did not reveal any significant effect of factors Prime 1 and Prime 2. This lack of significance indicates that the few GT output translations with structures diverging from the NP structures (PNP structures) did not influence the priming effect triggered by the NP syntactic alternative frequently appearing on GT output.

4. General Discussion and Conclusions

Our aim in this study was to investigate whether popular web-based MT systems can influence L2 English syntactic processing and elicit changes in the L2 English speakers’ linguistic behaviour. The general questions addressed were: (1) Do L2 English speakers adapt their language behaviour to mirror the MT system output when speaking in English? (2) If so, what are the factors that influence such changes?

To answer these questions, we first carried out a survey in which we investigated how Brazilian Portuguese L2 English students and speakers use popular web-based MT systems. The results of this survey showed that GT is the most popular system and that they have been using this MT service as a tool supporting tasks involving English vocabulary as well as English syntax. The survey also showed that the systems are more frequently used by respondents at intermediate and advanced English proficiency levels. The results of this survey were imperative to determine participants’ profiles as well as the MT system we should use when designing the syntactic priming experiment.

For the behavioural experiment, we tested 32 Brazilian Portuguese L2 English students at intermediate and advanced English proficiency with the aim of investigating whether after machine-translating sentences from Portuguese into English using GT, we would observe changes in their linguistic behaviour as a consequence of priming effects elicited by the MT output. The most important finding of our research is that a robust long-lasting priming effect was observed between the GT output and the participants as reflected by re-use of the GT preferred syntactic alternative in their subsequent English speech. Thus, this result indicates that L2 English speakers adapt their language behaviour to mirror the MT system output when speaking in English.

The results presented here replicated our previous findings [45] as well as results from several studies in the field of Psycholinguistics (e.g., [29,32,41,43,50,51]) and HCI literature (e.g., [31,32]). The so-called inverse preference effect, widely observed in psycholinguistic studies, was observed in the present study as participants re-used their least preferred syntactic alternative (the NP structures) when describing images in English after exposure to this syntactic alternative through GT output. Even though only around 8% of the MT output for the prime sentences was a PNP structure which is the participants’ most preferred structure and the easiest for them to process, it was GT’s NP structure that elicited the priming effect. Thus, the present study replicated the inverse preference effect reported in the literature, as the most challenging structure (in this case NP structures) became more frequent in the participants’ subsequent speech.

Our study has also shown a robust priming effect between a computer system (GT) and humans. Participants trusted the GT output enough to change their linguistic behaviour in order to mirror the system’s choices. This finding replicates results in the field of HCI indicating that priming effects can be elicited by computer systems or non-humans. In addition, our results show that a priming effect can be elicited not only between human-human interactions, but also between human-computer interactions.

In addition to the priming effect, we observed a learning effect that emerged throughout the experiment sessions as the amount of NP structures produced by participants increased as the experiment progressed. The learning effect was then confirmed by the post-test results showing that even after 24 h’ exposure, participants were significantly (p < 0.001) more likely to produce an NP structure when tested on the same sentences than in the baseline pre-test. Both the learning effect emerging throughout the experiment session and the significant increase in the amount of NP structures produced in the post-test as compared to the baseline pre-test suggest that the learning effect observed in the present study was of an implicit nature. We validate this assumption based on the argument that if the effect had significantly diminished in the post-test, we could assume that the effect was short-lived and that participants re-used the structures due to simply storing them in their short-term memory. However, we observed that the priming effect was capable of reorganizing participants’ L2 syntactic processing mechanisms by changing their least preferred syntactic alternative to describe a relation of possession as their preferred one, even without the influence of the GT output. Another interesting finding is that the lack of interaction between post-test results and participants’ proficiency levels indicates that differences in proficiency levels did not influence in the persistence of the priming effect in the day following the priming session.

Based on the results described here, we conclude that the novelty of the present study lies in the observation that an MT system is capable of eliciting a long-lasting priming effect which results in implicit language learning of a challenging-to-process structure and reorganization of syntactic processing in the second language. The most important finding is that, contrary to the existing taboo involving translation as a method for language learning, our study shows that learning of L2 syntax can implicitly emerge from translation tasks and that knowledge can emerge from the interaction with a web-based MT system. The knowledge generated from this research is, therefore, of pedagogical relevance for the second language learning and teaching field. This study reveals that language instructors can assume a more optimistic perspective in relation to the use of MT systems with students, as our results show that (even unconsciously) students can learn from the MT output and such learning can be generalized by participants from a task involving comprehension (reading the translation) to a task involving production (speaking).

Additionally, we conclude that the syntactic priming paradigm represents an ecologically valid method to study MT-human interaction. Our study has obtained reliable results since they replicate findings of a number of previous studies in the fields of HCI Psycholinguistics. Thus, the methodology employed in the present study can be used by other researchers in different fields of research to address questions involving MT-human interaction.

It is worth mentioning that, although our results provide evidence as to whether influence of MT on English syntactic processing, it does not provide evidence whether GT can negatively influence English learning. We wonder whether MT pitfalls can be learnt and generalized by users when speaking or writing in English. Another limitation is that we tested a popular system; in future research, we aim to investigate whether a syntactic priming effect could be observed in the interaction between a less popular MT system and language learners. In addition, we aim at investigating whether it is possible to build an MT system constructed to train users’ language behaviour, and expose the participants to the output from this system, so this too remains an avenue for future work.