ASIDS: A Robust Data Synthesis Method for Generating Optimal Synthetic Samples

Abstract

Most existing data synthesis methods are designed to tackle problems with dataset imbalance, data anonymization, and an insufficient sample size. There is a lack of effective synthesis methods in cases where the actual datasets have a limited number of data points but a large number of features and unknown noise. Thus, in this paper we propose a data synthesis method named Adaptive Subspace Interpolation for Data Synthesis (ASIDS). The idea is to divide the original data feature space into several subspaces with an equal number of data points, and then perform interpolation on the data points in the adjacent subspaces. This method can adaptively adjust the sample size of the synthetic dataset that contains unknown noise, and the generated sample data typically contain minimal errors. Moreover, it adjusts the feature composition of the data points, which can significantly reduce the proportion of the data points with large fitting errors. Furthermore, the hyperparameters of this method have an intuitive interpretation and usually require little calibration. Analysis results obtained using simulated original data and benchmark original datasets demonstrate that ASIDS is a robust and stable method for data synthesis.

1. Introduction

Synthetic data present an effective solution to the challenges of inadequate or low-quality samples, particularly in the era of big data. The use of data synthesis methods to generate synthetic data provides a cost-effective and efficient alternative to collecting and labeling large amounts of real-world data. Furthermore, these methods can address privacy concerns associated with real-world data, making it safer to share and analyze [1]. Recently, there has been a surge in the use of synthetic data in machine learning, with various synthetic methods being developed [2,3].

Representative data synthesis methods can be classified into three categories. The first category entails techniques such as interpolation, extrapolation, and other methods [4] that generate additional data points representative of the underlying distribution. These data synthesis methods can extract more information from the dataset, thereby enhancing model generalization. For image data, deep-learning-based methods such as VAE and GAN can also be used to generate new data [5,6]. These methods are useful for improving the quality of the dataset and model generalization. The second category is meant to address the issue of dataset imbalance, such as with SMOTE [7] and some of its enhanced versions [8,9,10]. These methods can synthesize minority class samples to balance the dataset and improve the model’s performance. Finally, data sharing and research require sensitive data privacy protection. Synthetic data can help protect sensitive information while still enabling data sharing and research. Normally, we can add random noise to protect original data, like in differential privacy methods [11], generating synthetic data for sharing and research purposes. Overall, these methods help optimize the representation and use of data in various applications.

For some actual tasks, original data often contain a multitude of features and unknown noise [12]. Since the synthetic method involves generating new data based on existing data, the quality of the new points depends on the quality and quantity of the original data. If the quality of the original data is poor or the quantity is insufficient, then the synthetic data may have limitations in terms of quality [13]. However, there is a lack of effective synthetic methods for datasets with a restricted size and complex noise that can expand the size of the dataset.

Based on the purpose of improving the quality and quantity of the dataset, and motivated by piecewise linear interpolation and spline interpolation, we propose a robust and stable data synthesis method named Adaptive Subspace Interpolation for Data Synthesis (ASIDS), which aims to adaptively adjust the sample size and structure of the original dataset containing unknown noise. The idea is to divide the original feature space into several subspaces with an equal number of samples, and then perform linear interpolation for the samples in the adjacent subspaces. This method achieves sample optimization via two aspects. First, it can adaptively adjust the size of the dataset, and the expanded data typically contain minimal errors. Second, it adjusts the structure of the samples, which can significantly reduce the proportion of samples with large errors, thereby minimizing the impact of the noise in model generalization. Compared with other methods, ASIDS is particularly suitable for data containing unknown noise. Its main purpose is to expand the sample size and optimize the sample structure to uncover more hidden information in the data. Also, the samples synthesized by ASIDS tend to have smaller errors.

The rest of this paper is organized as follows: The existing interpolation research is reviewed in Section 2. Section 3 details the concept of the proposed ASIDS method and provides proof of the effectiveness of this method. The experimental results are presented and analyzed in Section 4. Finally, we conclude this paper in Section 5.

2. Related Work

Traditional interpolation methods are based on the function values of known data points for extrapolation and prediction. For a given dataset, there are generally two cases. In the case of two known data points and interpolation in between, the interpolation method can be selected based on distance, such as in nearest neighbor interpolation [14]. If it is assumed that the unknown point between these two data points is consistent with the straight line between them, a linear function, such as linear interpolation or piecewise linear interpolation [15,16], can be used to fit and interpolate. For interpolation among multiple given points, a commonly used method is to predict the value of the unknown point by constructing a high-degree polynomial based on the given points, such as is performed with Lagrange interpolation or Newton interpolation [17,18]. However, with an increase in the number of data points and the degree of the polynomial, there is a risk of overfitting and numerical instability (Runge’s phenomenon) [19]. Another solution is to construct a global smooth function by fitting a low-degree polynomial in a local region. In comparison to high-degree polynomial interpolation methods, this method has better smoothness and numerical stability, such as is seen in spline interpolation [20]. Nevertheless, as it is necessary to fit multiple local low-degree polynomials, this method can lead to relatively high computational complexity.

When expanding the sample size using interpolation methods, the selection of node positions and quantities has a significant impact on the accuracy and stability of the interpolation results. Typically, equidistant nodes [21] are used for interpolation position selection, which means the nodes are equally spaced within the interpolation interval. Chebyshev nodes [22] are selected within the interpolation interval to satisfy certain conditions for better fitting of the function. In addition, the choice of the number of interpolations can lead to instability of the data and insufficient validation accuracy. One way to determine the appropriate number of interpolations is by comparing the fitting degree of the model trained under different interpolation numbers with the original data [23], but this may lead to overfitting and the model having a poor generalization ability.

3. Proposed Method

In this section, we will explain the proposed ASIDS method in detail. ASIDS consists of two algorithms: K-Space and K-Match. To clearly illustrate the proposed method, we will first provide an overview of ASIDS and then introduce the specific algorithms involved.

3.1. Overview

ASIDS is mainly based on linear interpolation to increase the size and improve the quality of the original dataset. The idea is to divide the original feature space into several subspaces with an equal number of samples, and then perform linear interpolation for the samples in the adjacent subspaces. This method requires two hyperparameters (k and $\eta$) in advance. Parameter k is the number of samples existing in each feature subspace, while $\eta$ is the number of equidistant nodes interpolated per unit distance in the linear interpolation of the samples. This proposed method is illustrated in Figure 1.

The dataset $D={\{{\mathit{x}}_i,y_i\}}_{i=1}^n$ is given and is assumed to be contaminated with unknown noise, where ${\mathit{x}}_i\in \mathcal{X}=R^p$ and $y_i\in \mathcal{Y}=R$. Assuming ${\dot{y}}_i=f\left({\dot{\mathit{x}}}_i\right)$, where ${\dot{\mathit{x}}}_i$ is the actual value of ${\mathit{x}}_i$, ${\dot{y}}_i$ is the actual value of $y_i$, and $f(·)$ is a continuous function, then it can be taken to represent the real relationship between ${\mathit{x}}_i$ and y. Consider the model:

$${\dot{y}}_i+{ϵ}_{i,y}=f({\dot{\mathit{x}}}_i+{ϵ}_{i,x})+{ϵ}_i,$$

(1)

where ${ϵ}_{i,y}$ is the noise in ${\dot{y}}_i$, ${ϵ}_{i,x}$ is the noise in ${\dot{\mathit{x}}}_i$, and ${ϵ}_i$ represents the error term. Expression (1) can be rewritten as

$$y_i=f\left({\mathit{x}}_i\right)+{ϵ}_i.$$

(2)

Let ${\mathit{x}}_0=\{x_0^1,\dots ,x_0^p\}$, where $x_0^j=inf{\left\{x_i^j\right\}}_{i=1}^n$ for $\forall j=1,\dots ,p$, and call ${\mathit{x}}_0$ the sample minimum point.

Given the hyperparameter k, we provide an unsupervised clustering method called K-Space. As is shown in Figure 1b, the space can be partitioned into $n/k$ subspaces, each containing k samples; i.e., $\mathcal{X}={\cup }_{s=1}^{n/k}{\mathcal{X}}_s$, ${\mathcal{X}}_i\cap {\mathcal{X}}_j=\varnothing ,i,j=1,2,\dots ,\frac{n}{k}$, and $i\ne j$. The datasets corresponding to different subspaces are defined with $D={\cup }_{s=1}^{n/k}{D}_s,$$D_s={\left\{({\mathit{x}}_i^s,y_i^s)\right\}}_{i=1}^k$, and ${\mathit{x}}_i^s\in {\mathcal{X}}_s$. For two adjacent subspaces, since $f(·)$ is a continuous function, we assume that it can be approximated as a linear function $g(·)$, and then Equation (2) can be transformed into

$$y_i=g\left({\mathit{x}}_i\right)+{ϵ}_i+{ϵ}_i^{\prime },$$

(3)

where ${ϵ}_i^{\prime }$ is the linear fitting error term. When the distance between two adjacent subspaces approaches zero and the measurements of the subspaces tend to be zero, then ${ϵ}_i^{\prime }\to 0$ is obtained. Next, we will perform sample interpolation between adjacent subspaces.

We need to calculate the centers of each cluster, as follows:

$${\overline{\mathit{x}}}^s={\displaystyle \frac{1}{k}}\sum _{{\mathit{x}}_i^s\in D_s}{\mathit{x}}_i^s.$$

(4)

To achieve ${ϵ}_i^{\prime }\to 0$, we need to ensure that the interpolation is performed between clusters that are as close in distance as possible. For ${\left\{D_s\right\}}_{s=1}^{n/k}$, we define $D_{\left(1\right)}$, whose cluster center has the minimum distance to the sample minimum point ${\mathit{x}}_0$, and define $D_{\left(d\right)}$, whose cluster center has the minimum distance to the center of $D_{(d-1)},D_{(d-1)}\ne D_{\left(1\right)},\dots ,D_{(d-1)}$, and $d>1$.

$$D_{\left(1\right)}=\underset{D_s\in D}{argmin}dist({\mathit{x}}_0,{\overline{\mathit{x}}}^s),$$

(5)

$$D_{\left(d\right)}=\underset{\{D_s\in D\},D_s\ne D_{\left(1\right)},\dots ,D_{(d-1)}}{argmin}dist({\overline{\mathit{x}}}_0^{(d-1)},{\overline{\mathit{x}}}^s),$$

(6)

where ${\overline{\mathit{x}}}^{(d-1)}$ is the center of $D_{(d-1)}$. Interpolation is performed on ${\left\{D_{\left(d\right)}\right\}}_{d=1}^{n/k}$ sequentially according to the order of the d values, and interpolation is carried out only between adjacent subspaces (i.e., interpolate between $D_{\left(1\right)}$ and $D_{\left(2\right)}$, between $D_{\left(2\right)}$ and $D_{\left(3\right)}$, and so on).

When performing linear interpolation between adjacent subspaces, we should pair the k samples from the first subspace with an equal number of samples from the second subspace. The interpolation rules between adjacent subspaces are as follows:

• Linear interpolation can only be performed between two samples belonging to different adjacent subspace sets.
• Interpolation must be performed for each sample.
• Participation of each sample point is restricted to a single interpolation instance.

The number of matching schemes is $k!$. As is shown in Figure 1c, we provide a matching method called K-Match. Supposing ${ϵ}_i^{\prime }\to 0$, then this method can select a good-performing matching scheme of ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$ from $k!$.

Assuming $\mathit{x}$ and y are continuous variables, and given another hyperparameter $\eta$, then the number of samples inserted using the linear interpolation method between $D_{\left(d\right)}$ and $D_{(d+1)}$ is ${\displaystyle \sum _{i=1}^k}\lfloor \eta ·dist({\mathit{x}}_i^{\left(d\right)},{\mathit{x}}_i^{(d+1)})\rfloor$. Taking $({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)})\in D_{\left(d\right)}$ and $({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\in D_{(d+1)}$ as an example, then $\left\{\right({\mathit{x}}_{(d,d+1)}^{(m,i)},$$y_{(d,d+1)}^{(m,i)}{\left)\right\}}_{m=1}^{\lfloor \eta ·dist({\mathit{x}}_i^{\left(d\right)},{\mathit{x}}_i^{(d+1)})\rfloor }$ is the set of inserted samples, and the linear interpolation formula is defined as

$${\mathit{x}}_{(d,d+1)}^{(m,i)}={\mathit{x}}_i^{\left(d\right)}+m·{\displaystyle \frac{{\mathit{x}}_i^{(d+1)}-{\mathit{x}}_i^{\left(d\right)}}{\lfloor \eta ·dist({\mathit{x}}_i^{\left(d\right)},{\mathit{x}}_i^{(d+1)})\rfloor +1}},$$

(7)

$$y_{(d,d+1)}^{(m,i)}=y_i^{\left(d\right)}+m·{\displaystyle \frac{y_i^{(d+1)}-y_i^{\left(d\right)}}{\lfloor \eta ·dist(y_i^{\left(d\right)},y_i^{(d+1)})\rfloor +1}}.$$

(8)

After ASIDS processing, the original dataset will be optimized. The main steps of the ASIDS algorithm are summarized in Algorithm 1.

Algorithm 1: ASIDS.

The assumptions of ASIDS are as follows:

• $f(·)$ is a continuous function.
• The linear fitting error is ${ϵ}_i^{\prime }\to 0$.
• $\mathit{x}$ and y are continuous variables.

3.2. K-Space

The implementation of ASIDS requires an unsupervised clustering method to partition the feature space into multiple subspaces, each containing k samples. Based on this, we propose the K-Space clustering method. The clustering method has the following performance parameters:

• Each subspace contains an equal number of samples, i.e., $D={\cup }_{s=1}^{n/k}{D}_s,D_s={\left\{({\mathit{x}}_i^s,y_i^s)\right\}}_{i=1}^k$;
• Each sample belongs to only one subset, i.e., $D_i\cap D_j=\varnothing ,i\ne j$.

Maintaining continuity and similarity between adjacent subspaces is essential for synthesizing data via multiple linear interpolations in ASIDS. Our objective is to minimize the linear fitting error ${ϵ}_i^{\prime }$, which helps to satisfy ASIDS assumption 2 as much as possible.

To determine the sample set $D_s$ for subspace ${\mathcal{X}}_s$, it is necessary to determine the first sample ${\mathit{x}}_1^s$ in $D_s$.

$${\mathit{x}}_1^s=\underset{\mathit{x}:\mathit{x}\in D,\mathit{x}\notin D_1,\dots ,D_{s-1}}{argmin}dist(\mathit{x},{\overline{\mathit{x}}}^{s-1}),$$

(9)

where $s=1,\dots ,\frac{n}{k}$, ${\overline{\mathit{x}}}^{s-1}$ is the cluster center of $D_{s-1}$, and ${\overline{\mathit{x}}}^0={\mathit{x}}_0$. We define $D_s=\left\{x_1^s\right\}$ and determine ${\mathit{x}}_d^s$ as follows:

$${\mathit{x}}_d^s=\underset{\mathit{x}:\mathit{x}\in D,\mathit{x}\notin D_1,\dots ,D_s}{argmin}dist(\mathit{x},{\overline{\mathit{x}}}^s),$$

(10)

where $d=2,\dots ,k$. ${\mathit{x}}_d^s$ is obtained and $D_s\leftarrow D_s\cup \left\{{\mathit{x}}_d^s\right\}$ is updated.

The main steps of the K-Space algorithm are summarized in Algorithm 2.

Algorithm 2: K-Space.

3.3. K-Match

We can calculate the total error of the matching scheme to measure the quality of the scheme; for the sake of simplicity, let $\mathcal{X}=R$, and we calculate it as follows:

$$\sum _{i=1}^{k}S(x_i^{\left(d\right)},x_i^{(d+1)})=\sum _{i=1}^k{\int }_{x_i^{\left(d\right)}}^{x_i^{(d+1)}}|f\left(x\right)-L_i\left(x\right)|dx,$$

(11)

where $L_i\left(x\right)$ is the linear expression passing through the points $(x_i^{\left(d\right)},y_i^{\left(d\right)})$ and $(x_i^{(d+1)},y_i^{(d+1)})$.

Theorem 1.

Let ${\mathcal{X}}_{\left(d\right)}$ and ${\mathcal{X}}_{(d+1)}$ be two adjacent subspaces; the datasets corresponding to different subspaces are $D_{\left(d\right)},D_{(d+1)}$, and $(x_i^{\left(d\right)},y_i^{\left(d\right)})\in D_{\left(d\right)},(x_i^{(d+1)},y_i^{(d+1)})\in D_{(d+1)}$. Consider the model $y_i=f\left(x_i\right)+{ϵ}_i$, and let ${ϵ}_i^{\left(d\right)}=y_i^{\left(d\right)}-f\left(x_i^{\left(d\right)}\right)$. For $\forall i=1,2,\dots ,k$, suppose that ${ϵ}_i^{\prime }\to 0$; then, $E\left({\displaystyle \frac{S(x_i^{\left(d\right)},x_i^{(d+1)})}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\right)<E\left({\displaystyle \frac{|{ϵ}_i^{\left(d\right)}|+|{ϵ}_i^{(d+1)}|}{2}}\right)$.

Proof of Theorem 1.

Since ${ϵ}_i^{\prime }\to 0$, according to Equation (3) the model can be transformed into

$$y_i=g\left(x_i\right)+{ϵ}_i,$$

where $g(·)$ is a linear function. According to Equation (11), it follows that

$$S(x_i^{\left(d\right)},x_i^{(d+1)})={\int }_{x_i^{\left(d\right)}}^{x_i^{(d+1)}}|g\left(x\right)-L_i\left(x\right)|dx.$$

When ${ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0$, let $(x^{\prime },y^{\prime })$ be the intersection point between $y=L_i\left(x\right)$ and $y=g\left(x\right)$. We can simplify $S(x_i^{\left(d\right)},x_i^{(d+1)})$ using basic geometric area calculations, and according to the law of iterated expectations (LIE),   

$$\begin{array}{cc}\hfill E\left({\displaystyle \frac{S(x_i^{\left(d\right)},x_i^{(d+1)})}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\right)& ={\displaystyle \frac{E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0)}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\hfill \\ \hfill & +{\displaystyle \frac{E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0)}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\hfill \\ \hfill & ={\displaystyle \frac{E\left(|{ϵ}_i^{\left(d\right)}|+|{ϵ}_i^{(d+1)}|\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0)}{2}}\hfill \\ \hfill & +{\displaystyle \frac{\left(h_1·E|{ϵ}_i^{\left(d\right)}|+h_2·E\left|{ϵ}_i^{(d+1)}\right|\right)P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0)}{2}},\hfill \end{array}$$

where $h_1={\displaystyle \frac{|x^{\prime }-x_i^{\left(d\right)}|}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\mathrm{and}{h}_2={\displaystyle \frac{|x^{\prime }-x_i^{(d+1)}|}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}$. Since $P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0)+P({ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0)=1$, and $h_1+h_2=1$, it follows that $E\left({\displaystyle \frac{S(x_i^{\left(d\right)},x_i^{(d+1)})}{|x_i^{\left(d\right)}-x_i^{(d+1)}|}}\right)<E\left({\displaystyle \frac{|{ϵ}_i^{\left(d\right)}|+|{ϵ}_i^{(d+1)}|}{2}}\right)$. □

If our approach is to randomly select a matching scheme, the validity of this method can be proved by Theorem 1. However, randomly selecting a matching scheme does not guarantee the uniqueness of the results, and it also does not guarantee that we will necessarily select a good-performing matching scheme. We found that for $x_i^{\left(d\right)}$ and $x_i^{(d+1)}$, if ${ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0$, there will be a better interpolation effect.

Theorem 2.

Let $y_i^{\left(d\right)}=f\left(x_i^{\left(d\right)}\right)+{ϵ}_i^{\left(d\right)},y_i^{(d+1)}=f\left(x_i^{(d+1)}\right)+{ϵ}_i^{(d+1)}$. Suppose that ${ϵ}_i^{\prime }\to 0$; then, we can obtain $E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}<0\right)<E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}·{ϵ}_i^{(d+1)}\ge 0\right)$.

Proof of Theorem 2.

Since ${ϵ}_i^{\prime }\to 0$, as based on the proof of Theorem 1, we can obtain

$$\begin{array}{cc}\hfill & E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}\ge 0\right)\hfill \\ \hfill & ={\displaystyle \frac{|x_i^{\left(d\right)}-x_i^{(d+1)}|·E|{ϵ}_i^{\left(d\right)}|+|x_i^{\left(d\right)}-x_i^{(d+1)}|·E|{ϵ}_i^{(d+1)}|}{2}},\hfill \end{array}$$

$$\begin{array}{cc}\hfill & E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}<0\right)\hfill \\ \hfill & ={\displaystyle \frac{|x^{\prime }-x_i^{\left(d\right)}|·E|{ϵ}_i^{\left(d\right)}|+|x^{\prime }-x_i^{(d+1)}|·E|{ϵ}_i^{(d+1)}|}{2}}.\hfill \end{array}$$

It follows that

$$\begin{array}{c}\hfill E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}<0\right)<E\left(S(x_i^{\left(d\right)},x_i^{(d+1)})|{ϵ}_i^{\left(d\right)}{ϵ}_i^{(d+1)}\ge 0\right).\end{array}$$

□

According to Theorem 2, we can match samples with the opposite signs of ${ϵ}_i$ to achieve a good data synthesis effect. Therefore, the core idea of K-Match is to judge whether the sign of ${ϵ}_i$ is positive or negative for each sample, and then interpolate the samples with opposite signs as much as possible.

In K-Match, we need to choose an appropriate linear regression method to fit the dataset $D_{\left(d\right)}\cup D_{(d+1)}$ based on the performance of the noise. For example, lasso regression, locally weighted linear regression (LWLR) [24], and other methods can be used [25,26]. In our experiments, we used the OLS or SVR method to fit and obtain $\widehat{g}(·)$. Notably, the kernel function is linear in SVR. According to Equation (3), and supposing that the linear fitting error is ${ϵ}_i^{\prime }\to 0$ for dataset $D_{\left(d\right)}\cup D_{(d+1)}$, we can obtain

$${ϵ}_i=y_i-\widehat{g}\left({\mathit{x}}_i\right).$$

(12)

Then, we sort the samples in dataset $D_{\left(d\right)}$ in ascending order according to the value of ${ϵ}_i$, and obtain ${\left\{({\mathit{x}}_i^{\left(d\right)},y_i^{(d+1)})\right\}}_{i=1}^k$, whereas we sort the samples in dataset $D_{(d+1)}$ in descending order and obtain ${\left\{({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\right\}}_{i=1}^k$. As is shown in Figure 1d, the sorted datasets $D_{\left(d\right)}$ and $D_{(d+1)}$ are combined into the matching scheme ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$, the main steps of the K-Space algorithm are summarized in Algorithm 3.

Algorithm 3: K-Match.

Input: Subset $D_{\left(d\right)}={\left\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)})\right\}}_{i=1}^k,D_{(d+1)}=\{{({\mathit{x}}_i^{(d+1)},y_i^{(d+1)}\}}_{i=1}^k$    Output: Matching scheme ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$ 1  Fit the dataset $D_{\left(d\right)}\cup D_{(d+1)}$ and obtain $\widehat{g}(·)$ 2  Obtain $\left\{{ϵ}_i\right\}$ using Equation (10) 3  Sort the samples in $D_{\left(d\right)}$ and $D_{(d+1)}$ according to the value of ${ϵ}_i$, and obtain $D_{\left(d\right)}^{\prime }$ and $D_{(d+1)}^{\prime }$ 4  Combine $D_{\left(d\right)}^{\prime }$ and $D_{(d+1)}^{\prime }$ into ${\{({\mathit{x}}_i^{\left(d\right)},y_i^{\left(d\right)}),({\mathit{x}}_i^{(d+1)},y_i^{(d+1)})\}}_{i=1}^k$

3.4. Supplementary Notes

The proposed method can effectively expand the size of the dataset and adjust the dataset structure, reducing the proportion of samples that deviate significantly from the actual distribution, and thereby improve model generalization performance (see Figure 2).

The supplements to ASIDS are given below:

• The choice of the hyperparameter k is crucial, as different datasets require different values of k. Conversely, the hyperparameter $\eta$ tends to exhibit a better performance as its value increases, which will be illustrated in the experimental results in the following section.
• It is necessary to normalize the data if there is a significant difference in the dimensional scale between the features of the data. This avoids the issue of generating an excessive number of samples.
• In most cases, $n/k$ is not an integer, and for the excess samples, we usually have two solutions for handling this. The first one is to use the LOF algorithm [27] to filter out the excess samples that are not needed for ASIDS, as is shown in Figure 2c. Another solution is to treat the excess samples as a dataset $D^{\prime }$ of a subspace $D^{\prime }={\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^{nmodk}$. When interpolating between $D^{\prime }$ and other subspaces $D^{″}$, we choose an appropriate linear regression method to fit the dataset $D^{\prime }\cup D^{″}$ and obtain $\widehat{g}(·)$. Then, we use the same method to sort $D^{\prime }$ and $D^{″}$. Only $nmodk$ interpolations are performed, with each sample in $D^{\prime }$ being interpolated, while for $D^{″}$, only $nmodk$ samples are interpolated. Moreover, we interpolate the samples with the opposite signs of ${ϵ}_i$ as much as possible, as is shown in Figure 3.

4. Verification of the Performances of ASIDS

For the artificial data, which are also known as $f(·)$, we investigated the hyperparameter selection and optimization performance after processing with ASIDS. For the benchmark data, we studied the prediction performance using this method.

4.1. Simulated Datasets

Some verification indicators, including the proportion of samples with ${ϵ}_i$ greater than $\alpha$ and the mean square error (MSE), can be selected to test the optimization effect of ASIDS on the original sample after processing as follows:

$$p\left(\alpha \right)={\displaystyle \frac{1}{n}}\sum _{i=1}^{n}I(|f\left({\mathit{x}}_i\right)-y_i|>\alpha ),$$

(13)

$$MSE={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(f\left({\mathit{x}}_i\right)-y_i)}^2.$$

(14)

For the simulated datasets, ${\left\{{\mathit{x}}_i\right\}}_{i=1}^n$ is generated using $N_p(\mathbf{0},\mathit{E})$. Let ${\mathit{W}}_1\in {\mathbb{R}}_{(p\times p_1)}$ and ${\mathit{W}}_2\in {\mathbb{R}}_{(p_1\times 1)}$, where all elements of both ${\mathit{W}}_1$ and ${\mathit{W}}_2$ are independently and identically distributed as $N(0,1)$. Consider that $y_i=f\left({\mathit{x}}_i\right)+{ϵ}_i=tanh\left({\mathit{x}}_i^{\prime }{\mathit{W}}_1\right){\mathit{W}}_2+{ϵ}_i$.

$$tanh\left(x\right)={\displaystyle \frac{e^x-e^{-x}}{e^x+e^{-x}}}.$$

For a given dataset, we typically cannot ascertain the distribution of noise. Hence, we constructed datasets that contain unknown noises [12]. The unknown noises are simulated by a mixture of noises that contain uniform noises and Gaussian noises. The generation of ${ϵ}_i$ is explained in [28]. We fixed the random effects of the generated datasets; the experiments were repeated 100 times for each simulated dataset.

Table 1. Simulated datasets.

Simulated Datasets	${\mathit{ϵ}}_{\mathit{i}}$ Distribution	Sample Size	($\mathit{P},{\mathit{P}}_1$)
$D_1$	20%-N(0, 64) 30%-U(−8,8) 50%-N(0, 0.04)	500	(5, 3)
$D_2$	20%-N(0, 64) 30%-U(−8,8) 50%-N(0, 0.04)	200	(5, 3)
$D_3$	20%-N(0, 64) 30%-U(−8,8) 50%-N(0, 0.04)	1500	(5, 3)
$D_4$	20%-N(0, 64) 30%-U(−8,8) 50%-N(0, 0.04)	500	(1, 3)
$D_5$	20%-N(0, 64) 30%-U(−8,8) 50%-N(0, 0.04)	500	(20, 10)
$D_6$	40%-N(0, 64) 45%-U(−8,8) 15%-N(0, 0.04)	500	(5, 3)

shows six simulated datasets. Specifically, P is the number of features in each sample of the simulated datasets. $P_1$ is the dimension of the intermediate layer used to generate the y values.

4.1.1. Hyperparameter Selection

First, we investigated the determination of parameter k. To ensure that the number of feature subspaces was sufficient, we set the hyperparameter $\eta =1$, letting $k=1,2,\dots ,\lfloor \frac{n}{2}\rfloor$. To calculate the changes in the MSE of the datasets at different values of k, we obtained $k^{\prime }=\underset{k:k=1,2,\dots ,\lfloor \frac{n}{2}\rfloor ,\eta =1}{argmin}MSE$. The change trend in the MSE index before and after the ASISO processing is shown in Figure 4. Then, by letting $\eta =1,2,\dots ,30,k=k^{\prime }$, we calculated the changes in the MSE of the datasets (Figure 5), and obtained ${\eta }^{\prime }=\underset{\eta :\eta =1,2,\dots ,30,k=k^{\prime }}{argmin}MSE$. At last, we let $k=k^{\prime },\eta =1$, and ${\eta }^{\prime }$, and we calculated the changes in $p\left(\alpha \right)$ of the simulated datasets (Figure 6).

As can be seen from Figure 4, ASIDS demonstrates good optimization performance for datasets with varying sample sizes or feature dimensions. The experimental results also show that the performance of ASIDS does not decline dramatically as the content of noise with large variances increases. Hence, it can deal with unknown noise well and has good robustness. In addition, it was shown by the experimental results that ASIDS exhibits good optimization performance for all datasets; thus, it is also a stable data synthesis method. As can be seen from Figure 5, the hyperparameter $\eta$ has a generally monotonically decreasing relationship with the MSE, and the larger the value of $\eta$, the more significant the effect. From Figure 6, it can be observed that ASIDS can adaptively adjust the sample structure, which reduces the proportion of samples with large errors.

4.1.2. Comparison of Optimization Performance

To verify the optimization performance of ASIDS, we compared it with three other methods: piecewise linear interpolation, linear extrapolation, and nearest neighbor interpolation. Specifically, we let $k=k^{\prime }$ and $\eta ={\eta }^{\prime }$ for ASIDS. Based on the given dataset, we used the samples generated by ASIDS as interpolation points to calculate the output values of linear extrapolation and nearest neighbor interpolation. Moreover, by letting $k=1$ and $\eta ={\eta }^{\prime }$ for ASIDS, we can regard it as piecewise linear interpolation. The experimental results show that ASIDS has the smallest MSE among all methods for each dataset (Figure 7).

4.2. Benchmark Datasets

To verify the prediction performance of ASIDS, four benchmark datasets were used [29,30]. We partitioned the dataset into training and testing sets using a 7:3 ratio and normalized the data using the min–max normalization method. We preprocessed the training data using ASIDS, trained multiple machine learning models on the training set, and evaluated the prediction performance of the models on the testing set. The evaluation metric we used was the mean absolute error (MAE).

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n|y_i-{\widehat{y}}_i|.$$

(15)

In addition, we removed features that cannot be directly used, such as “Datetime” for bike sharing demand, “Month” for forest fires, and so on. We chose the K-nearest neighbor (KNN), random forest (RF), gradient boosting decision tree (GBDT), multilayer perceptron (MLP), and support vector regression (SVR) as the machine learning prediction models. Specifically, the kernel function is the radial basis function (RBF) in SVR. Moreover, we set the number of hidden layers to 3 for the MLP and used different numbers of neurons based on the input dimensionality and sample size. Prior to and subsequent to the application of ASIDS, the MLP retains an identical network architecture. Additionally, there are negligible variations in the count of base learners within ensemble methodologies, such as RF and GBDT. On a holistic level, these variations do not impart significant changes to the model’s complexity.

The experimental results of the five models are shown in

Table 2. Experimental results for benchmark datasets.

Datasets	Processing	Hyperparameter	Testing MAE (${10}^{-2}$)
			KNN	RF	MLP	SVR	GBDT
Bike Sharing	-	-	$2.20$	0.12	$0.54$	$4.26$	$0.23$
	ASIDS	$k=150,\eta =10$	$2.00$	0.06	$0.25$	$4.12$	$0.19$
Facebook	-	-	$6.60$	$2.25$	$8.19$	$7.34$	1.33
	ASIDS	$k=2,\eta =10$	$6.41$	$1.59$	$2.02$	$7.36$	$1.12$
Air Quality	-	-	$2.99$	$2.57$	$4.80$	$3.87$	$2.58$
	ASIDS	$k=20,\eta =100$	$2.89$	$2.55$	$2.08$	$3.84$	$2.77$
Forest Fires	-	-	$4.06$	$4.93$	$10.58$	$7.36$	$4.49$
	ASIDS	$k=10,\eta =100$	$3.89$	$4.32$	$4.80$	$10.05$	$4.14$

(the parameters for both the models and algorithms across diverse datasets were optimized using grid search and cross-validation). It is evident that ASIDS performs well on each benchmark dataset. This method is highly applicable to all five models, and in most cases, it can improve the predictive performance. It is worth mentioning that the dataset contains many categorical features which are not continuous variables. Moreover, for sparse samples (Facebook Metrics and Forest Fires), it is difficult to guarantee that the linear fitting error is ${ϵ}_i^{\prime }\to 0$ when interpolating between subspaces. This indicates that even if there are violations of the ASIDS assumptions in practical applications, this method may still achieve good optimization results. This further demonstrates that this method is robust and stable.

5. Conclusions

In this paper, we proposed a data synthesis method, ASIDS, which can adaptively adjust the size of the dataset, and the generated synthetic data typically contain minimal errors. Moreover, it can adjust the structure of the samples, which can significantly reduce the proportion of samples with large errors. The experimental results from the simulated datasets demonstrate that ASIDS can optimize the samples, and compared to other methods, the data generated using this method had smaller errors. ASIDS can deal with unknown noise better and has good robustness. The results from the benchmark datasets show that the proposed method is applicable for many machine models, and in most cases, it can improve the model generalization. ASIDS has some deficiencies and limitations. In practical applications, it should be considered whether the real scenario data can satisfy the assumptions of ASIDS. From the experimental results, it can be seen that the choice of hyperparameters has a great influence on the results. Future work may focus on practical applications and its integration with advanced machine learning techniques, and study of how to automatically or effectively select hyperparameters.