Improving the Robustness of the Theil-Sen Estimator Using a Simple Heuristic-Based Modification

Abstract

One of the most widely used robust regression methods for solving simple linear regression problems is the Theil-Sen (TS) estimator. This estimator has some notable advantages; however, it does not belong to the most robust estimation methods (called high-breakdown estimators) and is prone to outliers whose distribution is highly asymmetric with respect to the correct data points. This paper presents a modification of the TS estimator, the Robustified Theil-Sen (RTS) estimator. The new method uses a heuristic-based selection procedure to reduce the number of initial estimates of the regression function parameters computed with at least one outlier, thereby improving the regression results. The use of this heuristic procedure only slightly increases the computational time required for using the RTS estimator compared to the TS estimator. Preliminary results of two numerical experiments presented in the paper show that the RTS estimator outperforms other comparable estimators, i.e., the TS estimator and the repeated median estimator, in terms of robustness. The results presented also suggest that the breakpoint value (which is a measure of the robustness of estimators) of the RTS estimator is higher than the breakpoint value of the TS estimator and equal to the breakpoint value of the high-breakpoint estimators.

1. Introduction

Regression analysis of a data set is a critical step in many applications [1,2]. One of the important problems that occurs during this analysis is the presence of outliers—that is, atypical values in the context of this data set—in the analyzed data sets and their negative influence on the regression results [3,4,5,6,7]. For real data, it should always be assumed that the analyzed data set can contain a certain number of outliers. It is important to note that the usage of standard and very commonly used regression methods based on the idea of Least Squares (LS) minimization (e.g., the Ordinary Least Squares (OLS) method) can lead to obtaining wrong results even when there is a single outlier in the analyzed data set. Hence, there is a need for regression methods that are not as sensitive to outliers as the LS-based methods—in the literature, such methods are referred to as robust regression methods [4,6].

One of the most widely used robust regression methods [8] for solving probably the most common regression problem, simple linear regression (SLR) in the presence of outliers, is the Theil-Sen (TS) estimator. The method was introduced by Henri Theil in 1950 [9] and later extended by Pranab K. Sen in 1968 [10]. Although more than 50 years have passed since its development, the TS estimator is still used in many areas, such as image processing and machine vision [11,12], object positioning using 5G and 6G networks [13], and data preprocessing for machine learning applications [14], to name a few. It should be noted that the TS estimator, along with the Mann–Kendall test [15,16], is widely used for trend analysis in earth, environmental, and physical geography research. These two methods are used, for example, in the analysis of air pollution [17,18], long-term changes in wind power [19], streamflow [20], long-term trends in meteorological and hydrological drought [21], changes in rainfall [22], rainfall and temperature trends [23,24], annual rainfall and meteorological drought [25], groundwater storage [26], vegetation trends [27] or changes in oceanic or sea wave height [28,29,30,31].

The popularity of the TS estimator is due to its advantages, including but not limited to its high asymptotic efficiency, its ability to estimate skewed or heteroskedastic data, and its simplicity and computational efficiency [32,33]. Disadvantages of the TS estimator include its limited robustness (which makes it not among the most robust estimators) and relatively large errors in estimation results when there are outliers whose distribution is highly asymmetric with respect to the correct data points.

The robustness of an estimator T can be evaluated using, among others, the asymptotic breakdown point measure

$${\epsilon }^{*}\left(T\right)=\underset{n\to \infty }{lim}\theta ,$$

(1)

where $\theta =o/n$, o is the number of outliers and n is the number of all data points in the analyzed data set $\mathbf{D}$. The measure ${\epsilon }^{*}\left(T\right)$ informs about the smallest ratio of o to n, for which the results obtained using T can be are arbitrary bad or meaningless [34,35,36]. In the case of the TS estimator [4]

$${\epsilon }^{*}\left(\mathrm{TS}\right)=1-\frac{1}{\sqrt{2}}\approx 0.293,$$

(2)

where the most robust estimators, known as high-breakdown estimators or resistant estimators, have ${\epsilon }^{*}\left(T\right)=0.5$. It should be noted that ${\epsilon }^{*}\left(T\right)=0.5$ is the maximum possible value of this parameter, and the value ${\epsilon }^{*}\left(T\right)>0.5$ will mean that in the analyzed data set, the number of outliers is larger than the number of correctly measured values [36].

This paper focuses on improving two previously mentioned drawbacks of the TS estimator, namely its limited robustness and its sensitivity to asymmetrically distributed outliers. The proposed solution, in the form of a Robustified TS (RTS) estimator, uses an additional simple heuristic-based procedure that leads to a significant improvement in the properties of the new estimator compared to the original TS estimator, at the cost of only a slight increase in the required computational effort.

The rest of this article is organized as follows. Section 2 briefly describes the TS estimator algorithm. Section 3 discusses the main reasons for the limited robustness of the TS estimator and presents the RTS estimator. Section 4 presents the results of two numerical experiments conducted to compare the RTS estimator with the TS and Repeated Median estimators. The last section, Section 5, concludes the article.

2. The Theil-Sen Estimator

The TS estimator can be used, among other problems, for solving the SLR problem, i.e., estimation of the regression model parameters calculated for a data set $\mathbf{D}=\{d_1,d_2,\dots ,d_i,\dots d_n\}$ of n bivariate data points $d_i=(x_i,y_i)$ representing observations (measurements), where $x_i$ is an explanatory (independent) variable and $y_i$ is an explained (dependent) variable, and the linear function

$$y_i=\alpha x_i+\beta ,$$

(3)

is used as a regression model. The purpose of SLR is to estimate the values of parameters $\alpha$ and $\beta$, that is, respectively, $\hat{\alpha }$ and $\hat{\beta }$, based on the data points $d_i\in \mathbf{D}$. Note that the number of outliers in the analyzed data set $\mathbf{D}$ should satisfy the condition

$$0\le o\le \frac{n}{2},$$

(4)

which is a common assumption when using robust estimators [36].

In the case of SLR, the TS estimator [9,10] works in two simple steps. In the first step, the estimation of the line slope $\alpha$ is calculated according to the formula

$$\hat{\alpha }=\underset{i<j}{median}\left({\tilde{\alpha }}_{ij}\right),$$

(5)

where

$${\tilde{\alpha }}_{ij}=\frac{y_j-y_i}{x_j-x_i}$$

(6)

and $i,j\in \{1,\dots ,n\}$. The second step is the estimation of $\hat{\beta }$, that is, the estimation of line interception with the Y axis—this step can be realized in one of two ways:

• Directly—in this case the value $\hat{\beta }$ is calculated based only on the values of data points

  $$\hat{\beta }=\underset{i<j}{median}\left({\tilde{\beta }}_{ij}\right),$$

  (7)

where

$${\tilde{\beta }}_{ij}=\frac{x_j{y}_i-x_i{y}_j}{x_j-x_i},$$

(8)

• Hierarchically—using the early calculated value, using (5), of $\hat{\alpha }$, that is,

  $$\hat{\beta }=\underset{i}{median}\left(y_i-\hat{\alpha }{x}_i\right).$$

  (9)

From now on, TSd and TSh will denote, respectively, the two above-mentioned versions of the TS estimator.

3. The Robustified Theil-Sen Estimator

3.1. Causes of the Limited Robustness of the TS Estimator

To understand the causes of the limited robustness of the TS estimator, it is important to analyze how the TSd estimator works. Let us assume that there is a bivariate data set $\mathbf{D}$, for which its cardinality $card\left(\mathbf{D}\right)=n$ and o of its data points are outliers, it means that $c=n-o$ of data points in $\mathbf{D}$ are correct data points. Taking this into account, it can be seen that only

$$\stackrel{̑}{d}=\frac{(n-o)(n-o-1)}{2}=\frac{c(c-1)}{2},$$

(10)

values of ${\tilde{\alpha }}_{ij}$ and ${\tilde{\beta }}_{ij}$ are calculated using data points $d_i$ and $d_j$, where each of these points is a correct data point, that is, none of them is an outlier. From now on, such estimates ${\tilde{\alpha }}_{ij}$ and ${\tilde{\beta }}_{ij}$ are referred to as correct estimates, and denoted, respectively, as ${\stackrel{̑}{\alpha }}_{ij}$ and ${\stackrel{̑}{\beta }}_{ij}$. Hence, $\stackrel{̑}{d}$ shows the number of correct estimates for each of the estimated parameters.

The number of all calculated estimators for $\alpha$ and $\beta$ is equal

$$t=card\left(\left\{{\tilde{\alpha }}_{ij}|i<j\right\}\right)=card\left(\left\{{\tilde{\beta }}_{ij}|i<j\right\}\right)=\frac{n(n-1)}{2}.$$

(11)

The relation between $\stackrel{̑}{d}$ and t is best represented by the ratio

$$\tau (n,c)=\frac{\stackrel{̑}{d}}{t}=\frac{c(c-1)}{n(n-1)}.$$

(12)

Let $\rho$ be a fraction of the correct data points in the analyzed data set, that is, $\rho =c/n$. Thus, substituting c by $\rho n$

$$\tau (n,\rho )=\frac{\rho n(\rho n-1)}{n(n-1)}=\frac{{\rho }^2}{1-\frac{1}{n}}-\frac{\rho }{n-1}.$$

(13)

Therefore, for $n\to \infty$,

$$\tau \left(\rho \right)=\underset{n\to \infty }{lim}\tau (n,\rho )={\rho }^2,$$

(14)

which means that for large n and $\rho =0.5$ only $25\%$ of all estimates are correct estimates; for smaller n, this fraction can be even smaller than $25\%$ (see Appendix A).

In the case of the TSd estimator, the median function is used to calculate the $\hat{\alpha }$ and $\hat{\beta }$ values, (5) and (7), respectively. Since the breakpoint value of the median function is ${\epsilon }^{*}\left(\mathrm{median}\right)=0.5$, whenever $\tau \left(n,\rho \right)\le 0.5$, the TSd estimator may return incorrect estimates. In the case of the TSh estimator, the above analysis applies only to the estimation of $\hat{\alpha }$. However, the wrong value of $\hat{\alpha }$ affects the estimation of $\hat{\beta }$, (8). This means that even in the case of the TSh estimator, the condition $\tau \left(n,\rho \right)>0.5$ must be satisfied to reduce the possibility of obtaining meaningless results.

3.2. General Idea of the RTS Estimator

The presented analysis shows that the quality of the estimation results obtained by the TS estimator can be improved by introducing two additional steps into this method, namely

• A selection of subsets of correct estimates $\stackrel{̑}{\mathbf{A}}=\left\{{\stackrel{̑}{\alpha }}_{ij}\right\}$ and $\stackrel{̑}{\mathbf{B}}=\left\{{\stackrel{̑}{\beta }}_{ij}\right\}$ from sets of all initial estimates, $\tilde{\mathbf{A}}=\left\{{\tilde{\alpha }}_{ij}\right\}$ and $\tilde{\mathbf{B}}=\left\{{\tilde{\beta }}_{ij}\right\}$, respectively,
• Searches for values of optimal estimates $\hat{\alpha }$ and $\hat{\beta }$ only among sets of correct estimators $\stackrel{̑}{\mathbf{A}}$ and $\stackrel{̑}{\mathbf{B}}$, respectively.

The use of these additional steps is the founding idea of the developed RTS estimator.

The search for optimal estimates $\hat{\alpha }$ and $\hat{\beta }$ from sets $\stackrel{̑}{\mathbf{A}}$ and $\stackrel{̑}{\mathbf{B}}$ can be carried out easily in the same way as in the TS estimator, that is, using the median function. Nonetheless, selecting the subsets of correct estimates, i.e., $\stackrel{̑}{\mathbf{A}}$ from $\tilde{\mathbf{A}}$ and $\stackrel{̑}{\mathbf{B}}$ from $\tilde{\mathbf{B}}$, is a much more difficult problem. However, it should be noted that the median function is a high-breakdown estimator (${\epsilon }^{*}(median)=0.5$), which means that the occurrence of a limited number of non-correct estimators in the $\stackrel{̑}{\mathbf{A}}$ or $\stackrel{̑}{\mathbf{B}}$ sets should only have a limited influence on the final estimation result obtained from the RTS estimator. Taking this into account, it is not necessary that the selection procedure be perfectly accurate—it is sufficient if only the majority of the selected estimates are correct estimates. The consequence of this observation is the possibility of using the heuristic approach described below as a selection procedure.

The heuristic selection procedure implemented in the RTS estimator is based on the observation that a difference

$${\Delta }_S^{\tilde{\mathbf{E}}}\left(i\right)=e_{i+s-1}-e_i,$$

(15)

where $e_i$ is the first and $e_{i+s-1}$ is the last element of the s-elementary subset of consecutive, sorted in ascending order, elements of the estimate set $\tilde{\mathbf{E}}$ (in the case of the RTS estimator, i.e., $\tilde{\mathbf{A}}$ and $\tilde{\mathbf{B}}$), is usually smaller in the cases when only correct estimates belong to this subset than in cases where this subset contains both correct and non-correct estimates or contains only non-correct estimates. Furthermore, the values of such defined differences of s-nearest elements usually increase with the increasing number of non-correct estimates belonging to these s-elementary subsets. These observations have been made for data sets $\mathbf{D}$ for which the dispersion of observation (measurement) error for the correct data points in $\mathbf{D}$ is smaller than the dispersion of outliers belonging to $\mathbf{D}$. This heuristic approach made it possible to reduce the task of selecting the correct estimates to a simple procedure of finding the minimum value in the one-dimensional list

$$\left({\Delta }_s^{\tilde{\mathbf{E}}}\left(1\right),{\Delta }_s^{\tilde{\mathbf{E}}}\left(2\right),\dots ,{\Delta }_s^{\tilde{\mathbf{E}}}(t-s+1)\right),$$

(16)

where s is the parameter of the procedure and represents the desired number of correct estimates in the set $\tilde{\mathbf{E}}$.

3.3. Description of the RTS Estimator Algorithm

The ideas presented in the previous section have been implemented in two versions of the RTS estimator, i.e., the directed (RTSd) and hierarchical (RTSh) versions. Similarly to the TSd and TSh estimators, the only difference between the RTSd and RTSh estimators is the method of calculation of the estimation of line interception $\hat{\beta }$. The algorithm for the calculation of the line slope estimate $\hat{\alpha }$ is common for both variants of the RTS estimator.

3.3.1. Calculation of $\hat{\alpha }$ in RTSd and RTSh Estimators

The first step of both versions of the RTS estimator is the calculation of all estimates ${\tilde{\alpha }}_{ij}$ of the line slope $\alpha$, that is,

$$\forall i<j:{\tilde{\alpha }}_{ij}=\frac{y_j-y_i}{x_j-x_i},$$

(17)

where all estimates ${\tilde{\alpha }}_{ij}$ form a set $\tilde{\mathbf{A}}$, i.e., $\tilde{\mathbf{A}}=\left\{{\tilde{\alpha }}_{ij}\right\}$. The next stage is the selection of a subset of correct estimates $\stackrel{̑}{\mathbf{A}}$ from $\tilde{\mathbf{A}}$. For this purpose, all elements of $\tilde{\mathbf{A}}$ have to be sorted in ascending order, after which

$$\tilde{\mathbf{A}}:=sort\left(\tilde{\mathbf{A}}\right)=sort\left(\left\{{\tilde{\alpha }}_{ij}\right\}\right)=\left({\tilde{\alpha }}_{\left(k\right)}\right),$$

(18)

where index $\left(k\right)$ is the ordering index, $sort(·)$ is an ordering operator, and $:=$ is an assignment operator. Hence,

$${\tilde{\alpha }}_{\left(1\right)}\le {\tilde{\alpha }}_{\left(2\right)}\le \dots \le {\tilde{\alpha }}_{\left(k\right)}\le \dots \le {\tilde{\alpha }}_{\left(t\right)},$$

(19)

where t is calculated using (11). The next step is the search for an optimal s-elementary subset of $\tilde{\mathbf{A}}$, i.e.,

$$\hat{k}=arg\underset{k}{min}\left({\Delta }_s^{\tilde{\mathbf{A}}}\left(k\right)\right)=arg\underset{k}{min}\left({\tilde{\alpha }}_{\left(l\right)}-{\tilde{\alpha }}_{\left(k\right)}\right),$$

(20)

where ${\Delta }_s^{\tilde{\mathbf{A}}}\left(k\right)$ is the k-th s-nearest elements difference for $\tilde{\mathbf{A}}$, the idea of which was introduced in Section 3.2, and

$$\begin{array}{ccc}k\hfill & \in & \{1,\dots ,t-s+1\},\hfill \\ l\hfill & =& k+s-1.\hfill \end{array}$$

(21)

The value of s can be easily calculated based on the estimated number of outliers $\tilde{o}$ in the analyzed data set $\mathbf{D}$. The relationship between $\tilde{o}$ and s is analogous to the relationship (10) between o and $\stackrel{̑}{d}$, so s is calculated as follows

$$s=\frac{(n-\tilde{o})(n-\tilde{o}-1)}{2},$$

(22)

where

$$\tilde{o}=⌊\tilde{\theta }·n⌋;$$

(23)

$\lfloor v\rfloor$ denotes a floor function, i.e., a function that returns the largest integer less than or equal to a given real number v. The $\tilde{\theta }$ is the only parameter of the RTS estimator that must be provided by the user, and it is the assumed ratio of the number of outliers to the number of all data points in the analyzed set $\mathbf{D}$, hence $0\le \tilde{\theta }\le 0.5$. For $\tilde{\theta }=0$, the RTS estimator will give the same results as the standard TS estimator. As the number of outliers o in the data set $\mathbf{D}$ increases, the value of the parameter $\tilde{\theta }$ should also be increased. Choosing $\tilde{\theta }=0.5$ means that the worst case is assumed, that is, that $50\%$ of the points in the set $\mathbf{D}$ are outliers. However, $\tilde{\theta }=0.5$ can be used as a default parameter—as a result, the RTS estimator can be considered a parameterless estimator.

Based on $\hat{k}$, the set $\stackrel{̑}{\mathbf{A}}$ is determined according to the formula

$$\stackrel{̑}{\mathbf{A}}=\left({\tilde{\alpha }}_{\left(\hat{k}\right)},{\tilde{\alpha }}_{\left(\hat{k}+1\right)},\dots ,{\tilde{\alpha }}_{\left(\hat{l}\right)}\right),$$

(24)

where $\hat{l}=\hat{k}+s-1$. The last stage of estimation of the slope of the line $\alpha$ is the calculation of $\hat{\alpha }$, which is defined as

$$\hat{\alpha }=median\left(\stackrel{̑}{\mathbf{A}}\right).$$

(25)

3.3.2. Calculation of $\hat{\beta }$ Using the RTSd Estimator

In the case of the RTSd estimator, the algorithm for $\hat{\beta }$ calculation is analogous to the algorithm presented above for calculation $\hat{\alpha }$. Hence, as a first set, $\tilde{\mathbf{B}}=\left\{{\tilde{\beta }}_{ij}\right\}$ is created. The estimates ${\tilde{\beta }}_{ij}$ are calculated as follows

$$\forall i<j:{\tilde{\beta }}_{i,j}=\frac{x_j{y}_i-x_i{y}_j}{x_j-x_i}.$$

(26)

Then, the selection of the subset $\stackrel{̑}{\mathbf{B}}$ of the correct estimates from the set $\tilde{\mathbf{B}}$ is realized. This procedure begins with a sorting of $\tilde{\mathbf{B}}$ in ascending order, that is

$$\tilde{\mathbf{B}}:=sort\left(\tilde{\mathbf{B}}\right)=sort\left(\left\{{\tilde{\beta }}_{ij}\right\}\right)=\left({\tilde{\beta }}_{\left(k\right)}\right),$$

(27)

which means that

$${\tilde{\beta }}_{\left(1\right)}\le {\tilde{\beta }}_{\left(2\right)}\le \dots \le {\tilde{\beta }}_{\left(k\right)}\le \dots \le {\tilde{\beta }}_{\left(t\right)}.$$

(28)

Then, the search for an optimal s-elementary subset of $\tilde{\mathbf{B}}$ is performed as follows

$$\hat{k}=arg\underset{k}{min}\left({\Delta }_s^{\tilde{\mathbf{B}}}\left(k\right)\right)=arg\underset{k}{min}\left({\tilde{\beta }}_{\left(l\right)}-{\tilde{\beta }}_{\left(k\right)}\right),$$

(29)

where k and l are given by (21), and its result is

$$\stackrel{̑}{\mathbf{B}}=\left({\tilde{\beta }}_{\left(\hat{k}\right)},\dots ,{\tilde{\beta }}_{\left(\hat{l}\right)}\right).$$

(30)

The last step of the RTSd estimator algorithm is the calculation of the final estimate of $\beta$ according to the formula

$$\hat{\beta }=median\left(\stackrel{̑}{\mathbf{B}}\right).$$

(31)

3.3.3. Calculation of $\hat{\beta }$ Using the RTSh Estimator

The calculation of $\hat{\beta }$ in the RTSh version of the RTS estimator is based on an early calculated value of $\hat{\alpha }$. This procedure is started by creating a set $\tilde{\mathbf{B}}=\left\{{\tilde{\beta }}_i\right\}$, where

$$\forall i:{\tilde{\beta }}_i=y_i-\hat{\alpha }{x}_i.$$

(32)

The next step is the procedure of selecting the subset $\stackrel{̑}{\mathbf{B}}$ from the set $\tilde{\mathbf{B}}$, which begins with the ascending sorting of $\tilde{\mathbf{B}}$, i.e.,

$$\tilde{\mathbf{B}}:=sort\left(\tilde{\mathbf{B}}\right)=sort\left(\left\{{\tilde{\beta }}_i\right\}\right)=\left({\tilde{\beta }}_{\left(k\right)}\right),$$

(33)

thereby

$${\tilde{\beta }}_{\left(1\right)}\le {\tilde{\beta }}_{\left(2\right)}\le \dots \le {\tilde{\beta }}_{\left(k\right)}\le \dots {\tilde{\beta }}_{\left(n\right)}.$$

(34)

Then, the optimal value of k is calculated as follows

$$\hat{k}=arg\underset{k}{min}\left({\Delta }_s^{\tilde{\mathbf{B}}}\left(k\right)\right)=arg\underset{k}{min}\left({\tilde{\beta }}_{\left(l\right)}-{\tilde{\beta }}_{\left(k\right)}\right),$$

(35)

but because the estimates ${\tilde{\beta }}_i$ are calculated based only on one data point $d_i$, the values k and l have to be defined using $\tilde{o}$ and n instead of using s and t values like in (21), thus

$$\begin{array}{ccc}k\hfill & \in & \{1,\dots ,n-\tilde{o}+1\},\hfill \\ l\hfill & =& k+\tilde{o}-1.\hfill \end{array}$$

(36)

The value $\tilde{o}$ is calculated according to (23). Then, using $\hat{k}$, subset $\stackrel{̑}{\mathbf{B}}$ is created as follows

$$\stackrel{̑}{\mathbf{B}}=\left({\tilde{\beta }}_{\left(\hat{k}\right)},\dots ,{\tilde{\beta }}_{\left(\hat{l}\right)}\right),$$

(37)

where, with respect to (36), $\hat{l}=\hat{k}+q-1$. Finally, the last step of the RTSh estimator algorithm is the calculation of the final estimate of $\beta$, that is,

$$\hat{\beta }=median\left(\stackrel{̑}{\mathbf{B}}\right).$$

(38)

4. Numerical Experiments

4.1. General Assumptions

The purpose of the experiments was to compare the newly developed RTS estimator with its direct predecessor, the TS estimator, and the modification of the TS estimator, the Repeated Median (RM) estimator [37,38]. The comparison was made in terms of

• Exp-A—evaluation of the dispersion of the estimation results,
• Exp-B—preliminary determination of the breakdown point value of the RTS estimator based on the comparison of the RTS estimation results with the results of the TS and RM estimators for data sets $\mathbf{D}$ containing a different number o of outliers.

In both experiments, the normal distribution (but with different parameters) is used for both the simulation correct observations and the outliers. In the experiments, the outliers are highly asymmetrically distributed with respect to the correct data points. In the experiments, the estimation results of $\hat{\alpha }$ and $\hat{\beta }$ were compared using both the directed and the hierarchical versions for the analyzed estimators. Using the RTS estimator requires specifying the estimated ratio $\tilde{\theta }$ of outliers in the analyzed set $\mathbf{D}$—the default value of $\tilde{\theta }=0.5$ was used in both experiments. Both experiments were carried out using specially developed software. The MATLAB R2023b Update 7 environment for the MS Win 10 operating system was used in the research.

4.2. Experiment A

The comparison was performed for the simulated SLR problem, with data sets $\mathbf{D}$ containing $n=100$ data points, and $o=40$ of them are outliers (hence, the outlier ratio in $\mathbf{D}$ is equal to $\theta =o/n=0.4$). Set $\mathbf{D}$ was created in four steps:

• Drawing of the values of $x_i\in [-2,10]$ coordinates according to a uniform distribution, $i=\{1,\dots ,100\}$,
• Draw, according to a uniform distribution, a list of $o=40$ indices i of data points $d_i$ that will be outliers,
• For correctly measured (correct, non-outlier) data points, their $y_i$ values are simulated with a normally distributed $\mathcal{N}(\mu ,\sigma )$ measurement error as follows

  $$y_i=\alpha x_i+\beta +\mathcal{N}(\mu ,\sigma ),$$

  (39)

  where $\alpha =1$, $\beta =5$, $\mu =0$, and $\sigma =1$,
• Outlier values are calculated according to the formula

  $$y_i=2\alpha x_i+\beta -\alpha x_{out}+{\Delta }_{out}+\mathcal{N}(0,{\sigma }_{out}),$$

  (40)

  where

  $${\Delta }_{out}=\left\{\begin{array}{ccc}\sigma ({\delta }_{out}+{\sigma }_{out})& \mathrm{for}& x<x_{out}\\ -\sigma ({\delta }_{out}+{\sigma }_{out})& \mathrm{for}& x\ge x_{out}\end{array}\right.$$

  (41)

  and ${\delta }_{out}=7.5$, ${\sigma }_{out}=5$, and $x_{out}=5$.

The exemplary data set $\mathbf{D}$ and the estimation results obtained for this set using the compared estimators are presented in Figure 1. In addition, this figure also presents the results obtained using the OLS estimator, as well as the results obtained using the OLS estimator but only for the correct data points contained in $\mathbf{D}$ (these results are denoted as OLSc). It should be noted that the results obtained with the OLSc estimator are only a theoretically realizable idealization intended to show how accurate the estimation can be for the data sets used in the experiment.

In order to obtain more reliable results for the comparison, the whole process of drawing the data sets $\mathbf{D}$ and calculating the estimates $\hat{\alpha }$ and $\hat{\beta }$ based on them was repeated 1000 times. The box plots of the obtained sets of the $\hat{\alpha }$ and $\hat{\beta }$ values are presented in Figure 2.

4.3. Experiment B

The purpose of the experiment was to test how replacing correct points with outliers affects the estimation results. To limit the influence of other factors, 100 specially created data sets $\mathcal{D}$ were used for the study. Each set $\mathcal{D}=\left\{{\mathbf{D}}_o\right\}$ contains data sets ${\mathbf{D}}_o$, where o is the number of outliers in that particular data set; in the experiment, the sets ${\mathbf{D}}_o\in \mathcal{D}$ are data sets of a simulated SLR problem, each ${\mathbf{D}}_o$ contains $n=100$ data points, and $o\in \{0,1,\dots ,55\}$ were used. For each $\mathcal{D}$, the sets ${\mathbf{D}}_o\in \mathcal{D}$ and ${\mathbf{D}}_{o+1}\in \mathcal{D}$ differ only in the value $y_{n-(o+1)}$ assigned to the data point $d_{n-(o+1)}=\left(x_{n-(o+1)},y_{n-(o+1)}\right)$, where the value of $x_{n-(o+1)}$ is the same in both data sets. Furthermore, in the set ${\mathbf{D}}_{\mathbf{o}}$, the data point $d_{n-(o+1)}$ is a correct data point, while in the set ${\mathbf{D}}_{o+1}$, this data point is already an outlier; other data points in both data sets are identical. For each data set ${\mathbf{D}}_o$, replacing the correct data point with an outlier is achieved by replacing the correct data point with the largest x in the data set. The above-described process of replacing data points can be observed in Figure 3.

Creating $\mathcal{D}$ sets is completed in three steps:

• The creation of each $\mathcal{D}$ set begins by drawing x coordinates for $n=100$ points in the range $[-2,10]$ using a uniform distribution. The resulting vector is sorted in ascending order so that the returned vector $\mathbf{X}=[x_1,x_2,\dots ,x_i,\dots ,x_n]$ satisfies the condition ${\forall }_{i<j}:x_i<x_j$. Then, the vector $\mathbf{Y}$ of y coordinates of the corrected data points was created according to the formula

  $$y_i=x_i+\beta +\mathcal{N}(\mu ,\sigma ),$$

  (42)

  where $\alpha =-1$, $\beta =10$, $\mu =0$, $\sigma =1$.
• Next, the vector ${\mathbf{Y}}_{out}=[y_1^{out},y_2^{out},\dots ,y_i^{out},\dots ,y_{o_{max}}^{out}]$ of y coordinates of $o_{max}=max\left(o\right)=55$ outliers were drawn using the normal distribution $\mathcal{N}({\mu }_{out},{\sigma }_{out})$, where ${\mu }_{out}=50$, ${\sigma }_{out}=10$.
• The final step is to create the data set ${\mathbf{D}}_o=\left\{d_i\right\}$ for each analyzed value of o; this is achieved as follows

  $$d_i=\left\{\begin{array}{ccc}(x_i,y_i)& \mathrm{for}& i\le n-o\left(\mathrm{correct}\mathrm{data}\mathrm{point}\right)\\ \left(x_i,y_{i-o_{max}}^{out}\right)& \mathrm{for}& i>n-o\left(\mathrm{outlier}\right)\end{array}.\right.$$

  (43)

The previously mentioned Figure 3 shows, in addition to the process of replacing data points with outliers, the effect of this process on changing the estimation results obtained with the tested robust estimators, as well as the OLS and OLSc estimators. Figure 4 shows the change, as a function of $\theta \in [0,0.55]$, in the median of the $\hat{\alpha }$ and $\hat{\beta }$ estimates obtained for all 100 simulated $\mathcal{D}$ data sets using the tested estimators.

4.4. Discussion of the Results

The presented results show that for the conditions of Exp-A, the RTS estimator outperforms other tested estimators in terms of accuracy and precision of the obtained results. It is also worth noting that the RTS estimator gives comparable results to those obtained by the OLSc estimator, which for a data set containing correct data points with homoscedastic error and outliers (data sets $\mathbf{D}$ used in Exp-A fulfill these conditions) is the optimal estimator in the class of linear unbiased estimators—this statement follows from the Gauss–Markov theorem.

The results of Exp-B show that the RTS estimator for an entire practically used range of $\theta \in [0,0.5)$ gives more accurate estimation results than the results obtained by the TS and RM estimators. Analyzing the effect of replacing subsequent correct data points with outliers, it is easy to see that among the tested estimators, the TS estimator is the most sensitive to these changes. The RM estimator is slightly less sensitive than the TS estimator, and the RTS estimator is virtually unaffected by these changes. In the range of $\theta =[0,0.47]$, the changes in the values of $\hat{\alpha }$ are hardly noticeable; see Figure 4a. In the case of the $\hat{\beta }$ estimator, this range is even larger; see Figure 4b.

A comparison of the results of Exp-A and especially Exp-B with the values of the breakdown points of the TS and RM estimators—${\epsilon }^{*}\left(\mathrm{TS}\right)\approx 0.293$ and ${\epsilon }^{*}\left(\mathrm{RM}\right)=0.5$ [39], respectively—together with knowledge of how the RTS estimator works—provides a basis for concluding that

• The breakdown point value for the RTS estimator is higher than the breakdown point value of the TS estimator, formally

  $${\epsilon }^{*}\left(\mathrm{RTS}\right)>{\epsilon }^{*}\left(\mathrm{TS}\right),$$

  (44)

and probably that

• The breakdown point value for the RTS estimator is equal to the breakdown point value of the RM estimator, hence formally

  $${\epsilon }^{*}\left(\mathrm{RTS}\right)={\epsilon }^{*}\left(\mathrm{RM}\right)=0.5.$$

  (45)

This last statement means that the RTS estimator is actually a high-breakdown estimator.

The conclusions presented above are based on the obtained results and the analysis of the algorithms of the compared estimators. Therefore, these results should be treated as preliminary hypotheses that require further research for their possible confirmation.

5. Conclusions

The preliminary results obtained with the RTS estimator presented in the article are promising. The improvement in robustness of the RTS estimator compared to the TS is significant. The RTS estimator also shows better robustness and stability than the RM estimator. It is worth noting that these quality improvements were achieved at the expense of only a slight increase in computational complexity (see Appendix B).

The preliminary nature of the presented research also means that more research is needed to better understand the properties of the new estimator, such as consistency and efficiency. It is also important to answer the question of how the number and distribution of data points and outliers, selection of the $\tilde{\theta }$ parameter value, as well as the properties of the observation (measurement) error, affect the estimation results obtained by the RTS estimator. Attempts to answer these questions will be the goal of future research.

No less important than the above-mentioned research on the properties of the RTS estimator is its comparison with robust estimators other than those used in this paper, including, among others, the Least Absolute Deviations, Least Trimmed Squares [40], and Least Median of Squares [40] estimators. Such a comparison is planned for both simulated data and data derived from real-world problems.