Double Consistency Regularization for Transformer Networks

Abstract

The large-scale and deep-layer deep neural network based on the Transformer model is very powerful in sequence tasks, but it is prone to overfitting for small-scale training data. Moreover, the prediction result of the model with a small disturbance input is significantly lower than that without disturbance. In this work, we propose a double consistency regularization (DOCR) method for the end-to-end model structure, which separately constrains the output of the encoder and decoder during the training process to alleviate the above problems. Specifically, on the basis of the cross-entropy loss function, we build the mean model by integrating the model parameters of the previous rounds and measure the consistency between the models by calculating the KL divergence between the features of the encoder output and the probability distribution of the decoder output of the mean model and the base model so as to impose regularization constraints on the solution space of the model. We conducted extensive experiments on machine translation tasks, and the results show that the BLEU score increased by 2.60 on average, demonstrating the effectiveness of DOCR in improving model performance and its complementary impacts with other regularization techniques.

1. Introduction

End-to-end models [1,2,3] with different deep neural network topologies have excelled in sequential tasks like neural machine translation (NMT). On the strength of its impressive performance in recent years, Transformer [4], the most popular end-to-end model, has evolved into the most fundamental model in NMT. Even so, the scale of state-of-the-art Transformer neural networks is still linear in the number of training examples. In low-resource task research, the amount of model parameters is much greater than the amount of data. Therefore, a key point for research in low-resource tasks is how to solve Transformer’s tendency to overfit small-scale labeled data [5,6,7].

Some academics employ semi-supervised learning to address the issue of less labeled data and add a significant proportion of unlabeled data to the model training process to address these issues [8]. This approach does not require specific labels and is frequently based on the prediction vector produced by the model. By generating an unsupervised regularization loss term between the disturbed prediction result and the normal prediction result on the unlabeled data, this approach increases the generalization ability of the model [9,10,11].

On the basis of supervised cross-entropy loss, the $\pi$ model [9] adds unlabeled samples in each round to different noises, propagates forward twice, and calculates the consistency loss of the two prediction results. The final training target is the sum of the cross-entropy loss and consistency loss. The temporal ensembling [9] model replaces the result of the second forward propagation of the unlabeled data with the prediction results obtained in the past epoch to calculate the consistency loss based on the $\pi$ model. Mean teacher [10] uses two models, in which the teacher model is updated by the exponential moving average [11] (EMA) of the student model parameters so as to update the model weight online at each step and obtain the consistency loss by calculating the error between the prediction results of the teacher model and the student model. Many variants were also proposed later, such as dual students [12], Fast-SWA [13], virtual adversarial training [14], interpolation consistency training [15], unsupervised data augmentation [16], etc.

These methods raise the training overhead due to the increase in the amount of data. And the presence of some data in unlabeled data is harmful to the model. In this paper, our strategy to solve the overfitting problem under the condition of scarce labeled data is to introduce more regularization methods. We propose a consistent regularization method for end-to-end model frameworks that can be used for supervised learning under few-shot conditions called double consistency regularization (DOCR). DOCR encourages the model to produce high-confidence and high-consistency feature extraction and prediction outputs for inputs with similar distances in the feature space. In particular, data points with different labels are separated in low-density regions based on the smoothing assumption and the clustering assumption, and comparable data points have similar outputs. Therefore, if the input of the model is slightly perturbed, then the encoder output and decoder prediction results should not change significantly, and the output should be consistent, thereby constraining the model. In contrast to the previous work that only imposed constraints on the decoder’s output space, we impose double constraints on the solution space of the model by measuring the consistency between the features extracted by the two model encoders and the predicted probabilities of the decoder, and then we realize the regularization model’s internal parameters.

We conduct tests on small-sample machine translation tasks to test the efficacy of our method, and the results demonstrate that DOCR may successfully enhance model performance, decrease overfitting, and provide complementing effects with other regularization techniques. Compared with the base Transformer model, DOCR improves the scores on the IWSLT’14 German-English and IWSLT’15 English-Vietnamese datasets by 2.51 BLEU and 1.87 BLEU points, respectively. When used in combination with other regularizations, model performance improves by about 5.00 BLEU points. In addition, we conducted experiments on the state-of-the-art kNN-KD model [17] in the field of knowledge distillation and verified that our regularization method also has a certain improvement effect on the strong baseline model.

In the field of deep learning, the low-resource problem is an urgent challenge. There exists a large amount of low-sample data in tasks such as image recognition, machine translation, and speech recognition. However, deep models perform poorly in the face of these few-sample tasks due to the need to use a large amount of supervised data for training. One of the difficulties faced is the overfitting problem. From the above experimental results, our method can effectively mitigate the model overfitting phenomenon. Therefore, when using large-scale deep learning models to train small-sample tasks, our method should be used to alleviate the model overfitting phenomenon and thus improve model performance.

Our main contributions are summarized below:

• We propose DOCR, which adopts a two-model framework and utilizes consistent regularization to alleviate the overfitting problem of large-scale models that are prone to overfitting in the face of small-sample tasks.
• Extensive experiments on machine translation tasks verify the effectiveness of our approach, and the model performance improves significantly after using DOCR, which effectively mitigates the overfitting problem by constraining the encoder and decoder.
• We conduct a large number of ablation and analytical experiments to further analyze the reasons for the improvement of the model.

2. Method

Our method introduces the mean model based on cross-entropy and constrains the model solution space by regularizing the consistency loss between the mean model and the base model [18]. The model framework is shown in Figure 1. Algorithm 1 illustrates the entire algorithm. The following three aspects are described in detail: model construction, the noise model, and model training.

2.1. Model Construction

We used the basic model of the previous step to build the mean model. The structure of the two models is identical. We used the exponential moving average to set the parameter update rule of the mean model as follows:

$${\theta }_m\leftarrow \lambda {\theta }_m+(1-\lambda ){\theta }_b,$$

(1)

where ${\theta }_b$ is the parameter of the base model, which is updated through backpropagation, and ${\theta }_m$ is the parameter of mean model. If the starting model is given too much weight, then the training effect will be quite weak due to the initial model’s low performance. To balance the weight of the model parameters in the various training stages, we will therefore adhere to the cosine schedule from 0.996 to 1 during the training period [19].

2.2. Noise Model

Before the model was trained, the input statements needed to be noised using a noise model $N\left(x\right)$. The noise model contained two different types of noise. The first type of noise was words randomly discarded in the input sentences with a probability $p_w$. The second type of noise was a randomized arrangement of the input sentences following the condition

$$\forall i\in \{1,n\},\left|\sigma \right(i)-i|\le k,$$

where n is the sentence length, k is an adjustable parameter, and $\sigma$ is the randomized sentence. It has been proven that $p_w=0.1$, $k=2$ are the parameter that enables the model to achieve the best performance.

2.3. Model Training

We input the given input x with different perturbations to the basic model and the mean model, denoted as $x_b$ and $x_m$, respectively, the feature outputs of the encoders of the two models are denoted as $f_b\left(x_b\right)$ and $f_m\left(x_m\right)$, respectively, and the predicted outputs of the decoder are denoted as $p_b\left(y_i\mid {\mathbf{x}}_b,{\mathbf{y}}_{<i}\right)$ and $p_m\left(y_i\mid {\mathbf{x}}_m,{\mathbf{y}}_{<i}\right)$. Then, $f_b\left(x_b\right)$ and $f_m\left(x_m\right)$ are transformed into probability distributions $p_b\left(x_b\right)$ and $p_m\left(x_m\right)$ by the softmax layer, respectively. For the similar input, the model encoder and decoder should have consistent outputs. Algorithm 1 illustrates the entire training process. We used the KL divergence [20] to measure the distance between the encoder output and decoder output distributions of the two models and define the consistency loss:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{con}=& \sum _i{D}_{KL}\left(p_{m,i}\left(x_m\right)\parallel p_{b,i}\left(x_b\right)\right)\hfill \\ \hfill +& \sum _{y_i\in \mathcal{V}}{D}_{KL}\left(p_m\left(y_i\mid {\mathbf{x}}_m,{\mathbf{y}}_{<i}\right)\parallel p_b\left(y_i\mid {\mathbf{x}}_b,{\mathbf{y}}_{<i}\right)\right).\hfill \end{array}$$

(2)

Algorithm 1: DOCR

Input: Batch data S, parallel corpus D, mean model M, base model B. for each S in D do         Calculate the cross-entropy loss ${\mathcal{L}}_{ce}$.         Calculate the loss of consistency ${\mathcal{L}}_{con}$.         Calculate the total loss $\mathcal{L}$.         Update the base model B via gradient backpropagation.         Update the mean model $M\leftarrow \lambda M+(1-\lambda )B$. end

After our tests, the KL divergence was the more suitable objective function for our framework. In addition to that, we also tested using the MSE loss, MAE loss, or JS divergence as the objective function for consistency training in our experiments, the results of which are in Section 4.2.1. When the MSE loss and MAE loss were used as the objective functions for consistency training, we had

$$\begin{array}{cc}\hfill {\mathcal{L}}_{MSE}=& \frac{1}{n}\sum _i^n{∥p_{m,i}\left(x_m\right)-p_{b,i}\left(x_b\right)∥}_2^2+\frac{1}{n}\sum _i^n{∥p_m\left(y_i\mid {\mathbf{x}}_m,{\mathbf{y}}_{<i}\right)-p_b\left(y_i\mid {\mathbf{x}}_b,{\mathbf{y}}_{<i}\right)∥}_2^2,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{MAE}=& \frac{1}{n}\sum _i^n{∥p_{m,i}\left(x_m\right)-p_{b,i}\left(x_b\right)∥}_2+\frac{1}{n}\sum _i^n{∥p_m\left(y_i\mid {\mathbf{x}}_m,{\mathbf{y}}_{<i}\right)-p_b\left(y_i\mid {\mathbf{x}}_b,{\mathbf{y}}_{<i}\right)∥}_2,\hfill \end{array}$$

(4)

where n is the number of samples in each batch. When the JS divergence was used as the objective function for consistency training, we had

$$\begin{array}{cc}\hfill {\mathcal{L}}_{JS}=& \sum _i{D}_{JS}\left(p_{m,i}\left(x_m\right)\parallel p_{b,i}\left(x_b\right)\right)\hfill \\ \hfill +& \sum _{y_i\in \mathcal{V}}{D}_{JS}\left(p_m\left(y_i\mid {\mathbf{x}}_m,{\mathbf{y}}_{<i}\right)\parallel p_b\left(y_i\mid {\mathbf{x}}_b,{\mathbf{y}}_{<i}\right)\right).\hfill \end{array}$$

(5)

The final model training objective is defined as

$$\mathcal{L}={\mathcal{L}}_{ce}+\alpha {\mathcal{L}}_{con},$$

(6)

where $\alpha$ is a weight to balance the two loss values. In order to better balance the proportion of consistency loss during training, we defined $\alpha$ with the function sigmoid [21] and made a simple transformation of the function to make the previous function as small as possible. Thus, the model pays more attention to the standard supervision training in the early stage of training and pays more attention to the consistency training in the later stage. Here, $\alpha$ is defined as

$$\alpha =\mu \frac{1}{1+exp(-e-\gamma /2)},$$

(7)

where $\mu$ and $\gamma$ are hyperparameters and e is the epoch number. Because the teacher model is greatly affected by the performance of the previous training model, the definition of $\alpha$ and the choice of hyperparameters greatly affected the results of the experiment. In addition, we also tested three other functions that met our requirements to define $\alpha$ (Figure 2): cosine [10], linear, and Equation (8). In Section 4.2, we give a comparison of their effects:

$$\alpha =\left\{\begin{array}{ll}0.2\mu ,\hfill & \mathrm{if}e<\gamma \\ \mu ,\hfill & \mathrm{otherwise}\end{array}\right.$$

(8)

3. Experiments

3.1. Datasets

In order to prove the effectiveness and generalization of DOCR, we conducted experiments on the machine translation small-sample task [2]. We conducted experiments on the IWSLT’14 German-English (De-En) and IWSLT’15 English-Vietnamese (En-Vi) datasets. In the experiment on the De-En dataset, the train set, vaild set, and test set sizes were 160,000, 7300, and 6500, respectively, and in the experiment on the En-Vi dataset, the train set, vaild set, and test set sizes were 133,000, 1500, and 1300 respectively.

3.2. Preprocessing and Evaluation

We used the preprocessing steps provided by fairseq [22] to split the data for the IWSLT’14 German-English dataset. The preprocessed dataset supplied by [23] was used for the IWSLT’15 English-Vietnamese dataset, with tst2012 serving as the validation set and tst2013 serving as the test set. Using the scripts that Moses provided [24], sentences were tokenized and truecased. We used BPE [25] to extract shared subword units from the combination of the source and target content. The size of the vocabulary in the IWSLT De-En dataset was 10,000, and the size of the vocabulary in the IWSLT En-Vi dataset was 17,000.

As is customary, the entire work was evaluated using the case-insensitive BLEU score [26] determined by the multi-bleu.perl script [17].

3.3. Competitive Models

Powerful and extensively utilized regularization techniques, including both classical and semi-supervised methods, served as our primary baseline. We compared DOCR to dropout [5], label smoothing [27], mean teacher [10], and unsupervised data augmentation (UDA) [16]. On the dataset used in this research, we re-experimented with these algorithms, and the settings were consistent with DOCR. We also verified the validity of DOCR on a strong baseline: kNN-KD [17].

3.4. Training Settings

Our algorithms were all implemented using the fairseq toolkit [22]. In the kNN-KD task, we followed the settings of [17] and adopted faiss [28] to replicate it. The models in the article all use the Transformer structure. The configuration we used, called transformer_iwslt_de_en, includes six encoder layers, six decoder layers, an embedding size of 512, a feedforward size of 1024, and 4 attention heads. We used a beam search with a beam size of five. The hyperparameters of $\mu$ and $\gamma$ were 0.5 and 40, respectively. On four NVIDIA V100 GPUs, we trained our models using the Adam optimizer [29]. An overview of the experimental set-up is shown in

Table 1. Hyperparameter settings.

Hyperparameters	Value
Max tokens	8192
Learning rate	5 × 10${}^{-4}$
LR scheduler	Inverse sqrt
Minimal LR	1 × 10${}^{-9}$
Warm-up LR	1 × 10${}^{-7}$
Warm-up steps	4000
Gradient clipping	0.0
Weight decay	0.0
Optimizer	Adam
-${\beta }_1$	0.9
-${\beta }_2$	0.98
$\mu$	0.5
$\gamma$	40

. Unless otherwise stated, when comparing other algorithms, we similarly followed the hyperparameter settings in Table 1, which are the most basic settings for model training and did not affect the results.

4. Results of the Experiments

4.1. Main Results

We begin by confirming the DOCR’s efficacy and contrasting it with other widely used regularization methods, and we also show that our method showed complementary effects with other regularization methods. The experimental results are shown in

Table 2. Experiments on IWSLT’14 German-English (De-En) and IWSLT’15 English-Vietnamese (En-Vi) datasets. “↑” indicate that higher BLEU values indicate better model performance.

Models	De-En (BLEU↑)	En-Vi (BLEU↑)
Transformer (base)	31.01	27.06
+Dropout (0.3)	33.37	29.66
+Label smoothing (0.1)	32.89	28.97
+Mean teacher	33.03	29.32
+UDA	33.56	29.65
+DOCR	33.58	29.69
+Dropout (0.3)	36.13	32.22
+Label smoothing (0.1)	35.92	32.04
+Mean teacher	36.12	32.10
+UDA	36.39	32.43

.

4.1.1. Comparison with Mean Teacher and UDA

We now compare DOCR with two related semi-supervised regularization methods. The training data in the mean teacher and UDA methods are a mixed dataset of bilingual data and monolingual data. The experimental results show that the model performance improved by 2.02 and 2.26 BLEU points in the two translation tasks using the mean teacher algorithm, 2.55 and 2.59 BLEU points using the UDA algorithm, and 2.57 and 2.63 BLEU points using the DOCR algorithm. It can be seen that they all improved the performance of the model, with DOCR having the best results. We believe that this was due to the fact that the data processed by mean teacher and UDA contain additional monolingual data, and some utterances in these monolingual data were harmful to the model. DOCR, on the other hand, deals only with supervised data and produces regularization effects by restricting the model output and thus constraining the model solution space. The DOCR approach did not harm the model as much as the semi-supervised regularization approach.

4.1.2. Comparison with Dropout and Label Smoothing

We also compared DOCR with the dropout and label smoothing methods, which are currently widely used regularization methods applicable to any domain. The dropout methods achieve regularization by temporarily dropping neural network units from the network with a certain probability during the training process of a deep learning network. Dropout methods are also similar to a data augmentation approach. Label smoothing generates soft labels by applying a weighted average between the uniform distribution and hard labels to transform the hard labels into soft labels for smoother network optimization.

We compared DOCR with these two methods. The experimental results show that the model performance improved by 2.36 and 2.60 BLEU points in the two translation tasks using the dropout algorithm, 1.88 and 1.91 BLEU points using the label smoothing algorithm, and 2.57 and 2.63 BLEU points using the DOCR algorithm. It can be seen that all three methods can enhance the model. Compared with dropout and label smoothing, the process of regularizing the solution space can improve the performance of the model better or equally. This illustrates that regularizing the model solution space imposes more constraints on the model than adding noise. Therefore, it is proven that DOCR has some regularization effect.

4.1.3. Integration with Other Regularization Techniques

In addition to this, we further conducted experiments to verify that DOCR is complementary to other regularization techniques. We used DOCR together with other regularization techniques, and the results are shown in Table 2. The experimental results show that the model performance was greatly improved by using the dropout algorithm, label smoothing algorithm, mean teacher algorithm, and UDA algorithm on the basis of the DOCR algorithm. It can be seen that combining DOCR with other regularization techniques can improve the model performance to a greater or lesser extent, which suggests that the method of constraining the model solution space is complementary to the current popular regularization methods. The model performance was stronger when multiple regularization methods were used simultaneously, and no single method could completely solve the overfitting problem.

4.1.4. Results for the Base Model and kNN-KD

DOCR was used to solve the problem that the model is prone to overfitting in the bilingual corpus-constrained situation, and kNN-KD is a nonparametric knowledge distillation method proposed in [17] in 2022 which can significantly improve the model performance without additional data, and it is highly suitable for the bilingual corpus-constrained situation, which is in line with the application scenario of our method. The current state-of-the-art (SOTA) model for supervised training in machine translation is kNN-KD. Therefore, in addition to the basic Transformer model, we further tested the effectiveness of our method on kNN-KD, which is the best-performing supervised training algorithm at present.

We performed experiments on the basic Transformer model as well as on the strongly baseline kNN-KD model. Following the literature [17], we used dropout (0.3) and label smoothing (0.1) when conducting experiments on the kNN-KD model. The experiment results are shown in

Table 3. Results for the base model and kNN-KD. “↑” indicate that higher BLEU values indicate better model performance.

Models	De-En (BLEU↑)	En-Vi (BLEU↑)
Transformer (base)	31.01	27.06
+DOCR	33.58	29.69
kNN-MT	36.17	32.08
kNN-KD	36.30	32.27
+Mean teacher	36.42	32.41
+UDA	36.76	32.80
+DOCR	36.79	32.82

. In the table, kNN-MT [2] is the baseline model of the kNN-KD method (i.e., the kNN-KD method was improved with the kNN-MT method). The most primitive Transformer model was highly improved after using the DOCR method; kNN-KD-supervised training of the SOTA model in the field of machine translation improved the BLEU scores by 0.13 and 0.19 compared with the once SOTA model kNN-MT for both translation tasks. Our method improved the BLEU scores by 0.49 and 0.55 over kNN-KD, which is a relatively successful improvement.

In addition to this, we conducted experiments to evaluate the impact that other consistency regularization methods have on kNN-KD. We tested the change in model performance after using the mean teacher and UDA consistency regularization methods on top of kNN-KD. The experimental results show that in the two translation tasks, the model performance improved by 0.12 and 0.14 BLEU points using the mean teacher algorithm, 0.46 and 0.53 BLEU points using the UDA algorithm, and 0.49 and 0.54 BLEU points using the DOCR algorithm. It can be seen that our method performed equally as well as the best among the three consistency regularization methods, which suggests that constraining the encoder and decoder can be more effective in improving the model performance and mitigating the overfitting phenomenon.

4.2. Ablation Studies

In this section, dropout and label smoothing are used in the experiments.

4.2.1. Effect of Loss Used to Calculate the Consistency

Referring to the work of [10,18,30], we further investigated the performance of the model when using the MAE loss, MSE loss, JS divergence, and KL divergence as the consistency loss. We conducted experiments on the De-En dataset. The results are shown in

Table 4. BLEU scores of model when using different loss values as consistency loss.

Loss	De-En
MAE Loss	35.65
MSE Loss	35.84
JS Divergence	36.36
KL Divergence	36.41

, and they show that DOCR performed best when using the KL divergence as the consistency loss while fixing our hyperparameters.

4.2.2. Effect of the Hyperparameters

The hyperparameters $\alpha$ of DOCR determine the proportion of consistency loss in the total training objective. We analyzed its definition and its hyperparameters $\mu$ and $\gamma$. In terms of their definition, we fixed the hyperparameters $\mu$ and $\gamma$ to be 0.5 and 40, respectively, and investigated four functions: sigmoid, cosine, linear, and Equation (8). As shown in

Table 5. BLEU scores when using different functions as weight $\beta$.

Functions	BLEU
Eq	35.90
linear	35.88
cosine	36.29
sigmoid	36.41

, the effect of using the sigmoid function was obviously better than those of the other functions, and it was more suitable for the teacher model update rules in this experiment. In terms of hyperparameter selection, we studied the two parameters separately. We fixed $\mu$ to 0.5 to discuss how to choose $\gamma$ and $\gamma$ and 40 to discuss how to choose $\mu$. As shown in Figure 3, the results show that the BLEU score first increased with the increase in $\mu$ and $\gamma$ and reached the maximum when $\mu$ was 0.5 and $\gamma$ was 40. And then, with the increase in the parameters, the performance decreased, and the phenomenon of non-convergence occurred. This shows that the model can achieve its best performance when $\mu$ is 0.5 and $\gamma$ is 40.

4.2.3. Effect on Different Parts

In this section, we examine how consistency regularization affects various model components. We applied consistency regularization to the encoder, decoder, and the entire model. “Encoder” indicates that we only trained the encoder for consistency, and the encoder did not make any changes compared with standard end-to-end training, and “Decoder” and “All” are similarly defined. The results are shown in

Table 6. BLEU scores of models when using consistency regularization on different parts.

Models	Loss
	Encoder	Decoder	All
Transformer (base)	35.52	35.87	36.41
kNN-KD	36.41	36.56	36.79

, where it is clear that the strong regularization effect made the proposed method more effective when applied to the whole model. In addition to this, there was some improvement in the model performance when consistent regularization was applied to either the encoder or decoder alone. Each component in DOCR had a positive effect on improving the performance of the model, and the removal of any of the components negatively affected the model.

5. Analysis

5.1. Using Logit Is Better Than Using Features

We considered using the features output by the last layer of the encoder and decoder instead of the probability distribution (logit) as the item for calculating the consistency loss [30], which can regularize the parameters of the model more finely. The consistency loss is defined as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{con}=& \sum _i{D}_{KL}\left(f_{m,i}\left(x_m\right)\parallel f_{b,i}\left(x_b\right)\right)\hfill \\ \hfill +& \sum _{y_i\in \mathcal{V}}{D}_{KL}\left(f_m\left(y_i\mid {\mathbf{x}}_m,{\mathbf{y}}_{<i}\right)\parallel f_b\left(y_i\mid {\mathbf{x}}_b,{\mathbf{y}}_{<i}\right)\right),\hfill \end{array}$$

(9)

where $f_b\left(y_i\mid {\mathbf{x}}_b,{\mathbf{y}}_{<i}\right)$ and $f_m\left(y_i\mid {\mathbf{x}}_m,{\mathbf{y}}_{<i}\right)$ are the outputs of the last layer of the decoder.

We conducted the same experiment as in Section 4.1, and the results are shown in

Table 7. BLEU scores of models when using features as terms for computing consistency loss.

Models	De-En	En-Vi
Transformer (base)	31.01	27.06
+double consistency	36.41	32.48
+double feature consistency	35.65	31.83

. For both datasets, feature-based consistent regularization gave poorer results compared with probability distribution-based consistent regularization. This is due to the coarse-grained nature of the text, and too much regularization will harm the NMT model, according to our analysis.

5.2. Hallucinations

The model’s attention mechanism might not accurately reflect the model’s actual attention. The authors of [31] proposed the concept of hallucinations to further understand the NMT model. If modest input changes cause rapid changes in the output, then the model is hallucinating and is not really paying attention to the input. In order to verify that the model is more robust, we followed the algorithm in [32], used the 50 and 100 most common subwords as perturbations, and tested the model’s performance under input perturbations.

Table 8. Number of distinct sentences which caused hallucinations in the baseline and MFSD models.

Models	Hallucinations
	50 Subwords	100 Subwords
Transformer	24	47
Basic Model	16	33
Mean Model	13	29

shows the number of hallucinations of the model on the De-En test set in the baseline and DOCR. In DOCR training, tests were performed using both the basic and mean models. The number of hallucinations dropped on average by 30% in the basic model and 40% in the mean model compared with the supervised MT. This indicates that the model in our approach is more robust to interference and more focused on the input content. The results show that there were fewer hallucinations in the mean model than in the basic model, proving that it is a more stable model overall. This further confirms the implementability of the internal logic of our method.

5.3. Computation Overhead

Our approach employs a teacher-student framework to further strengthen the machine translation model, which increases the training complexity to some extent. On the IWSLT De-En standard dataset, we compared the computational overhead of the DOCR algorithm with the comparison algorithm. The experiments were run on four NVIDIA V100 GPUs. The comparison models included a supervised model without any algorithm, a model obtained by training with the mean teacher algorithm, and a model obtained by training with the UDA algorithm. The experimental results show that it took a total of 20 h to train the model using the simple supervised training method. The training of the model using the mean teacher algorithm took 26 h, which was 30% more than the supervised training took. The model using the UDA algorithm took 27 h to train, which was 35% more than supervised training. Our method increased the training time from 20 h to 24 h, which was 20% more than supervised training. Our method is more computationally expensive than supervised training because DOCR uses two models and needs to generate two generations of models. However, our method takes less time and improves performance more significantly than other consistency regularization methods. This is due to the fact that on top of the teacher-student framework, the mean teacher and UDA algorithms perform consistency regularization by adding monolingual data, which undoubtedly increases the amount of computation by a very large amount. DOCR does not increase the amount of data by consistency regularizing the outputs of the model encoder and decoder.

6. Conclusions

In this paper, we proposed a dual-consistency regularization method to address the problem of model overfitting under small-sample conditions so as to make more effective use of the few labeled data in low-resource situations. Based on the dual-model training framework, consistency regularization constraints were applied to the encoder and decoder parts separately, resulting in a robust improvement in the model performance. By conducting experiments on standard low-sample machine translation tasks and comparisons with classical regularization methods, it was demonstrated that this method can effectively improve the model performance and can complement other regularization methods to effectively alleviate the overfitting phenomenon and improve the model generalization ability. The analysis experiments further validate that our method can make the model more focused on the inputs, and our method can produce better results with less training overhead than other regularization methods.