Orthogonal Matrix-Autoencoder-Based Encoding Method for Unordered Multi-Categorical Variables with Application to Neural Network Target Prediction Problems

Abstract

Neural network models, such as BP, LSTM, etc., support only numerical inputs, so data preprocessing needs to be carried out on the categorical variables to convert them into numerical data. For unordered multi-categorical variables, existing encoding methods may produce dimensional catastrophes and may also introduce additional order misrepresentation and distance bias in neural network computation. To solve the above problems, this paper proposes an unordered multi-categorical variable encoding method O-AE using orthogonal matrix for encoding and encoding representation learning and dimensionality reduction via an autoencoder. Bayesian optimization is used for hyperparameter optimization of the autoencoder. Finally, seven experiments were designed with the basic O-AE, Bayesian optimization of the hyperparameters of the autoencoder for O-AE, and other encoding methods to encode unordered multi-categorical variables in five datasets, and they were input into a BP neural network to carry out target prediction experiments. The results show that the experiments using O-AE and O-AE-b have better prediction results, proving that the method proposed in this paper is highly feasible and applicable and can be an optional method for the data processing of unordered multi-categorical variables.

1. Introduction

Target prediction is the prediction of a target associated with an input based on existing data by some method. Target prediction is increasingly important in finance [1,2], healthcare [3], manufacturing [4], power [5,6], weather [7], transportation [8], etc. Excellent prediction results can identify potential risks, provide credible prediction data, and improve users’ risk management ability; it can develop executable future plans and provide decision-making support for enterprises.

Neural networks, decision trees, linear regression, polynomial regression, etc., are methods to achieve target prediction. Kim [9] used multiple target prediction methods, conducted experiments under different numbers of features, feature types, and numbers of samples and concluded that the performance of artificial neural networks with target prediction is better under multi-categorical variable input conditions.

Overall, neural networks are the dominant model for solving target prediction problems at this stage. It is important to note that neural network models, such as BP, LSTM, etc., support only numerical data inputs; however, a large portion of the dataset contains non-numerical variables, i.e., categorical variables. For example, the binary categorical variable “drug reaction” includes two categories of “positive” and “negative”, and the “assessment result” is also a binary categorical variable, including “pass” and “fail”; another example is the multi-categorical variable “occupation”, which includes “teacher, firefighter, construction worker, doctor…” and other categories. Numerical variables are easier to interpret, but when categorical variables dominate the dataset, it is not easy to view data trends and make predictions [10]. Therefore, a method must be taken to map categorical variables to numerical values before inputting them into the neural network [11].

Categorical variable coding methods include one-hot encoding, label encoding, target encoding, embedding, etc. In this, one-hot encoding is applicable to unordered categorical variables, which generates N-dimensional binary vectors using 0, 1 for N categories. Label encoding applies to ordered categorical variables, and it maps each category to an integer starting from 0. Target encoding uses the target mean corresponding to the category variables instead of the categorical variables. Embedding, on the other hand, randomly generates weight vectors according to the specified embedding dimensions and maps the text into word vectors according to the indexes of the categorical variables.

Unordered multi-categorical variables are categorical variables that contain several (usually more than three) categories with no differences in order or distance between categories.

Some existing encoding methods are not entirely suitable for unordered multi-categorical variables; for example, when the number of categories is large, applying one-hot encoding or applying embedding with a large embedding dimension creates the risk of dimensionality explosion, and the data are too sparse after one-hot encoding. Embedding and label encoding also introduce additional order and distance to an originally unordered categorical variable. Target encoding, on the other hand, affects the ability of the neural network model to extract information with the risk of overfitting.

In the current research on the target prediction problem, scholars are more concerned about how to design a better model structure to make better prediction results, and when carrying out the input data processing, the data either do not include the categorical variables [12,13,14] or use other traditional coding methods. For example, Bu et al. [15] used label encoding for both ordered and unordered categorical variables for data preprocessing when adopting selective integrated learning for ship painting man-hour prediction. Hur et al. [16] predicted ship construction man-hours using deployable data at different times during the manufacturing process and converted the categorical variables into dummy variables when processing them. Sasan et al. [17] converted binary categorical variables into 1 and 2 labeled coding. Wang et al. [18] set working time and non-working time as 0 and 1 coding, respectively. Carrizosa et al. [19] processed all the categorical variables using one-hot encoding in a binary categorical problem in the presence of categorical variables. All of the above studies adopted the existing conventional encoding methods to carry out the codes for unordered multi-categorical variables but ignored the fact that the encoded values did not maintain the characteristics of unordered multi-categorical variables and also additionally introduced the misleading order and distance bias.

In his research on encoding methods for categorical variables, Sebastian [20] applied five encoding methods to three regression models in his experiments and proved that the regression results varied according to the encoding methods of categorical variables; Hien et al. [21] also verified that the three encoding methods of categorical variables, namely label encoding, one-hot encoding, and embedding, had different effects on the performance of a deep dense neural network and long- and short-term memory neural network models. Li et al. [22] proposed an ordered log-linear model for ordered categorical variables that can convert ordered categorical data into model-describable multi-categorical lists; Meulemeester et al. [23] used unsupervised embedding to convert categorical variables into word vectors. Dahouda et al. [24] extended the word-embedding approach by proposing a deep learning method to codify categorical variables. All of the above methods are more suitable for ordered categorical variables or for categorical variables with dependencies. Namgil et al. [25] devised a Bayesian network based method for converting categorical variables into data variables, and Jung et al. [26] proposed a method for updating data points in the kernel space of a continuous variable by using SVMs to reflect the effect of each categorical variable; all of the above methods are only suitable for the target value of a binary categorical variable.

In general, there is no encoding method for transforming unordered multi-categorical variables into numerical codes with the characteristics of unordered variables and small dimensions. Based on this need, this paper carries out a research on encoding methods for unordered multi-categorical variables. This paper also extends the problem of predicting the working hours of cruise ship production design tasks by analyzing the relationship between task workload and task working hours through the previous working hours data and the design attribute data in the ship area, such as the task type, the number of model structures, the planned working hours, the feedback working hours, etc., so as to provide credible and standards-based working hours solutions for the planning and scheduling of the design tasks when designing and constructing a new ship in the future, which is also the key link to promote the standardization of ship design and construction. In the ship production design task working hour prediction problem, ship area and task type are categorical variables containing dozens of unordered categories, so the unordered multi-categorical variables cannot be ignored.

Autoencoder, as a classical tool for feature extraction and data dimensionality reduction, has continued to show its unique value in the field of data processing in recent years. For example, the research of Yang et al. [27] is a profound exploration, in which they cleverly integrate a deep autoencoder network into the Orthogonal Nonnegative Matrix Factorization (ONMF) framework, which achieves an accurate capture and hierarchical parsing of the intrinsic structure of complex data. This innovation not only highlights the ability of the autoencoder to automatically extract high-level features from data in an unsupervised learning environment but also effectively reduces the data dimensionality while preserving key information through its unique network structure.

Based on this core feature of autoencoder, this paper proposes a new encoding method, O-AE, for unordered multi-categorical variables, which first uses an orthogonal matrix to numerically encode the unordered multi-categorical variables, ensuring that the codes are independent of each other in terms of position, are equal distance, and are the same size; second, it uses autoencoder to carry out representation learning on the numerical codes and then performs the dimensionality reduction to ensure that the data inputted to the neural network has learned the relevant characteristics of the orthogonal matrix at the same time low dimensionality. This paper is structured as follows. Section 2 introduces four traditional methods for encoding multi-categorical variables; Section 3 describes the encoding mechanism of O-AE proposed in this paper; Section 4 introduces the Bayesian optimization of the hyperparameters of autoencoder for O-AE; Section 5 designs seven encoding experiments for six encoding methods (including O-AE), designs the Bayesian optimization of the hyperparameters of the autoencoder for O-AE, designs seven encoding experiments, introduces the specific information of five datasets, and proposes the validation metrics; and Section 6 carries out the example validation and demonstrates and discusses the experimental results.

2. Encoding of Categorical Variables

This section presents the theory of several types of encoding methods that are used frequently. First, the parametric representation of unordered multi-categorical variables is presented along with the parametric representation after encoding.

Definition 1. 

Unordered multi-categorical variables are non-numerical variables with no size or positional differences between categories.

Definition 2. 

The sample size of unordered multi-categorical variables $\mathcal{C}\mathcal{V}$ is set as $\tilde{N}$, and the variable contains $N$ categories $c_i,i=\mathrm{1,2},\cdots N,N\ge 3,N\le \tilde{N}$. There are multiple data points of the same category in $\mathcal{C}\mathcal{V}$, among which there are $m$ data points in category $c_i$, $c_{i1},c_{i2},\cdots c_{im},m\ge 1.$

$$\mathcal{C}\mathcal{V}={\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_{i1}\\ c_{i2}\\ \cdots \\ c_N\end{array}\right]}_{\tilde{N}\times 1}\to \mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1},c_i=\left(c_{i1},c_{i2},\cdots c_{im}\right),m\ge 1$$

(1)

Using the encoding method $f$, the category $c_i$ is mapped to the numerical value $n_i$ such that the unordered multi-categorical variables $\mathcal{C}\mathcal{V}$ are mapped to the numerical variables $\mathcal{N}\mathcal{V}$.

$$\mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1}\stackrel{f}{\to }\mathcal{N}\mathcal{V}:{\left[\begin{array}{c}{n}_1\\ n_2\\ \cdots \\ n_i\\ \cdots \\ n_N\end{array}\right]}_{N\times 1}$$

(2)

The categorical data of the same category have the same numerical data under the same encoding method mapping.

The $m$ identical categories $c_i$ in $\mathcal{C}\mathcal{V}$, under $f$ mapping, $c_{i1},c_{i2},\cdots c_{im}$, are transformed into numerical values $n_{i1},n_{i2},\cdots n_{im}$, respectively, and $n_{i1}=n_{i2}=\cdots =n_{im}$.

$$\left\{\begin{array}{l}CV:\left[c_i\right]\stackrel{f}{\to }NV:\left[n_i\right]\\ c_i=\left(c_{i1},c_{i2},\cdots c_{im}\right),n_i=\left(n_{i1},n_{i2},\cdots n_{im}\right),m\ge 1,n_{i1}=n_{i2}=\cdots =n_{im}\end{array}\right.$$

(3)

2.1. Label Encoding

The label encoding $f_L$ maps the category $c_i$ to integers from 0 to $N-1$.

$$\mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1}\stackrel{f_L}{\to }\mathcal{N}\mathcal{V}:{\left[\begin{array}{c}0\\ 1\\ \cdots \\ i-1\\ \cdots \\ N-1\end{array}\right]}_{N\times 1}$$

(4)

2.2. Target Encoding

The target encoding $f_T$ maps the category $c_i$ to the mean of the target value corresponding to that category.

Assuming that the predicted target is $y$, the target value corresponding to category $c_i$ is $y_i$. When $c_i$ contains multiple similar data $c_{i1},c_{i2},\cdots c_{im}$ within $c_i$, the corresponding target values are $y_{i1},y_{i2},\cdots y_{im}$.

$$\mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1}\stackrel{f_T}{\to }\mathcal{N}\mathcal{V}:{\left[\begin{array}{c}{n}_1\\ n_2\\ \cdots \\ n_i\\ \cdots \\ n_N\end{array}\right]}_{N\times 1},n_i={\displaystyle \sum _{j=1}^m}{\displaystyle \raisebox{1ex}{$y_{ij}$}\!\left/ \!\raisebox{-1ex}{$m$}\right.}$$

(5)

2.3. One-Hot Encoding

One-hot encoding $f_O$ maps the $N$ categories $c_i$ into $N$ binary row vectors with $N$-dimensional size, where each binary vector has only one valid digit 1 and the rest of the positions are 0, and the positions of the valid digits are different between different vectors.

$$\mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1}\stackrel{f_O}{\to }\mathcal{N}\mathcal{V}:{\left[\begin{array}{c}{\mathit{n}}_{\mathbf{1}}\\ {\mathit{n}}_{\mathbf{2}}\\ \cdots \\ {\mathit{n}}_{\mathit{i}}\\ \cdots \\ {\mathit{n}}_{\mathit{N}}\end{array}\right]}_{N\times N}={\left[\begin{array}{c}\begin{array}{cccccc}1& 0& \cdots & 0& \cdots & 0\end{array}\\ \begin{array}{cccccc}0& 1& \cdots & 0& \cdots & 0\end{array}\\ \cdots \\ \begin{array}{cccccc}0& 0& \cdots & 1& \cdots & 0\end{array}\\ \cdots \\ \begin{array}{cccccc}0& 0& \cdots & 0& \cdots & 1\end{array}\end{array}\right]}_{N\times N}$$

(6)

For example, the categorical variable “Occupation” contains three categories “Teacher, Firefighter, Construction Worker”, which are mapped to binary vectors [1,0,0], [0,1,0], [0,0,1] using one-hot encoding $f_O$.

From this encoding method, it is easy to see that each vector is a unit vector and the vectors are orthogonal to each other, i.e., the vectors are equal in distance and independent of each other, which is fully consistent with the characteristics of unordered categorical variables.

However, one-hot encoding also has some problems: the vectors are only composed of 0 and 1, and when the categorical variables are of large categories, the encoding obtained from the mapping will be sparse and of huge dimensions, which will make the computational cost large.

2.4. Embedding

The embedding encoding $f_E$ maps each category to a point in a vector space, and vectors with similar categories are closer together in the vector space.

Setting embedding converts the categorical variable $\mathcal{C}\mathcal{V}$ into a numerical variable $\mathcal{N}\mathcal{V}$ with a specified embedding dimension ${\mathcal{d}}_E$. The embedding dimension ${\mathcal{d}}_E$ denotes the dimension of the mapped $n_i$, i.e., $c_i\stackrel{f_E}{\to }{\mathit{n}}_{\mathit{i}}=\left[\begin{array}{cccc}{n}_i^1& n_i^2& \cdots & n_i^{{\mathcal{d}}_E}\end{array}\right]$.

Embedding is roughly divided into four steps.

(1) Predefine the vector space, determine the number $N$ of categories $c_i$ contained within the categorical variables, specify the embedding dimension ${\mathcal{d}}_E$, and construct a weight matrix $\mathcal{W}$ of $N$ rows and ${\mathcal{d}}_E$ columns. The weight matrix is randomly generated by default from a standard normal distribution with mean 0 and standard deviation 1. That is, $\mathcal{W}~N\left(\mathrm{0,1}\right)$.

(2) Assign an integer index starting from 0 to each category in the categorical variables.

(3) Map the integer indexes into a predefined vector space by means of a lookup table (this table is the weight matrix $\mathcal{W}$), where each index is associated with a vector corresponding to the number of the weight matrix.

(4) Return the mapped numeric variables for the categorical variables.

$$\mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1}\stackrel{f_E}{\to }\mathcal{N}\mathcal{V}:{\left[\begin{array}{c}{\mathit{n}}_{\mathbf{1}}\\ {\mathit{n}}_{\mathbf{2}}\\ \cdots \\ {\mathit{n}}_{\mathit{i}}\\ \cdots \\ {\mathit{n}}_{\mathit{N}}\end{array}\right]}_{N\times {\mathcal{d}}_E}\to \mathcal{W}=\left[\begin{array}{c}\begin{array}{cccc}{n}_1^1& n_1^2& \cdots & n_1^{{\mathcal{d}}_E}\end{array}\\ \begin{array}{cccc}{n}_2^1& n_2^2& \cdots & n_2^{{\mathcal{d}}_E}\end{array}\\ \cdots \\ \begin{array}{cccc}{n}_i^1& n_i^2& \cdots & n_i^{{\mathcal{d}}_E}\end{array}\\ \cdots \\ \begin{array}{cccc}{n}_N^1& n_N^2& \cdots & n_N^{{\mathcal{d}}_E}\end{array}\end{array}\right]$$

(7)

3. Orthogonal Matrix-Autoencoder-Based Encoding Method for Unordered Multi-Categorical Variables

Among the existing methods for encoding categorical variables, part of the methods is not applicable to unordered multi-categorical variables with no size, order, or distance requirements between categories, and the other part of the methods have large dimensionality after encoding. Therefore, this paper proposes the method O-AE based on orthogonal matrix-autoencoder for encoding and dimension reduction of unordered multi-categorical variables. O-AE for unordered multi-categorical variables encoding and dimensionality reduction process is shown in Figure 1.

$$\mathcal{C}\mathcal{V}:\left[c_i\right]\stackrel{f_{O-AE}}{\to }\mathit{Z}\in \mathbb{Z}\iff 〈\mathcal{C}\mathcal{V}\to \mathit{A}\stackrel{QR}{\to }\mathit{Q}\to \mathit{X}\in \mathbb{X},\mathit{X}\in \mathbb{X}\stackrel{encoder}{\to }\mathit{Z}\in \mathbb{Z}\stackrel{decoder}{\to }\widehat{\mathit{X}}\in \mathbb{X}〉$$

(8)

The model is mainly divided into two parts, the original code generation part and the code reduction part.

The original code generation part is $\mathcal{C}\mathcal{V}\to \mathit{A}\stackrel{\mathrm{Q}\mathrm{R}}{\to }\mathit{Q}\to \mathit{X}\in \mathbb{X}$. The code dimensionality reduction part is $\mathit{X}\in \mathbb{X}\stackrel{encoder}{\to }\mathit{Z}\in \mathbb{Z}\stackrel{decoder}{\to }\widehat{\mathit{X}}\in \mathbb{X}$.

First, according to the number of categories $N$ in the unordered multi-categorical variables, the $N$-order orthogonal matrix $\mathit{Q}$ is constructed, and the orthogonal matrix is mapped one by one with the original unordered multi-categorical variables according to the positional indexes into an orthogonal numerical coding matrix $\mathit{X}$ with the same data size, independent positions, and equal distances.

Second, the orthogonal matrix of numerical code matrix $\mathit{X}$ is input to the autoencoder, and the autoencoder is trained step by step by compressed reconstruction coding through the encoder and decoder in the autoencoder; finally, the learned feature $\mathit{Z}$ is obtained.

3.1. Code Generation

The $N$-order square matrix is set as $\mathit{A}\in {\mathbb{R}}^{N\times N}$, $\mathit{A}=\left[{\mathit{a}}_{\mathbf{1}},{\mathit{a}}_{\mathbf{2}},\cdots ,{\mathit{a}}_{\mathit{i}},\cdots ,{\mathit{a}}_{\mathit{N}}\right]$, where ${\mathit{a}}_{\mathit{i}}$ is the column vector of $\mathit{A}$ and ${\mathit{a}}_{\mathit{i}}={\left[a_i^1,a_i^2,\cdots ,a_i^N\right]}^T$.

There exists a series of transformations that decompose $\mathit{A}$ into an orthogonal matrix $\mathit{Q}$ and an upper triangular matrix $\mathit{R}$, i.e., $\mathit{A}=\mathit{Q}\mathit{R}$. These transformations are also called QR decomposition of $\mathit{A}$.

The orthogonal matrix $\mathit{Q}$ is set to consist of $N$ column vectors, $\mathit{Q}=\left[{\mathit{q}}_{\mathbf{1}},{\mathit{q}}_{\mathbf{2}},\cdots ,{\mathit{q}}_{\mathit{i}},\cdots ,{\mathit{q}}_{\mathit{N}}\right]$, where ${\mathit{q}}_{\mathit{i}}={\left[q_i^1,q_i^2,\cdots ,q_i^N\right]}^T$.

The row and column vectors of an orthogonal matrix are two-by-two orthogonal unit vectors.

$$\mathit{Q}{\mathit{Q}}^{\mathit{T}}={\mathit{Q}}^{\mathit{T}}\mathit{Q}=\mathit{E}$$

(9)

$${{\mathit{q}}_{\mathit{i}}}^{\mathit{T}}{\mathit{q}}_{\mathit{j}}=\left\{\begin{array}{l}\mathbf{0},\mathit{i}\mathbf{\ne }\mathit{j}\\ \mathbf{1},\mathit{i}=\mathit{j}\end{array}\right.$$

(10)

$${‖{\mathit{q}}_{\mathit{i}}‖}_2=\mathbf{1}$$

(11)

Therefore, mapping unordered multi-categorical variables into orthogonal numerical codes can ensure that the different categories are independent of each other in terms of location, are equal distances, and are the same size, and at the same time increase the data richness to avoid underfitting in the subsequent prediction process brought about by data impoverishment.

The steps for encoding unordered multi-categorical variables using an orthogonal matrix are as follows.

(1) Construct $N$-order square matrix $\mathit{A}\in {\mathbb{R}}^{N\times N}$ based on the number of categories $N$ in the unordered multi-categorical variables.

(2) Perform the QR decomposition of $\mathit{A}$ using the Gram–Schmidt orthogonal transformation to obtain the orthogonal matrix $\mathit{Q}\in {\mathbb{R}}^{N\times N}$.

The Gram–Schmidt orthogonal transformation is able to convert non-orthogonal bases into orthogonal bases by first orthogonalizing the non-orthogonal bases and, second, unitizing them.

$$\left\{\begin{array}{l}{\tilde{\mathit{q}}}_{\mathbf{1}}={\mathit{a}}_{\mathbf{1}}\\ {\tilde{\mathit{q}}}_{\mathbf{2}}={\mathit{a}}_{\mathbf{2}}-{\displaystyle \frac{〈{\mathit{a}}_{\mathbf{2}},{\tilde{\mathit{q}}}_{\mathbf{1}}〉}{〈{\tilde{\mathit{q}}}_{\mathbf{1}},{\tilde{\mathit{q}}}_{\mathbf{1}}〉}}{\tilde{\mathit{q}}}_{\mathbf{1}}\\ \cdots \\ {\tilde{\mathit{q}}}_{\mathit{N}}={\mathit{a}}_{\mathit{N}}-{\displaystyle \frac{〈{\mathit{a}}_{\mathit{N}},{\tilde{\mathit{q}}}_{\mathbf{1}}〉}{〈{\tilde{\mathit{q}}}_{\mathbf{1}},{\tilde{\mathit{q}}}_{\mathbf{1}}〉}}{\tilde{\mathit{q}}}_{\mathbf{1}}-{\displaystyle \frac{〈{\mathit{a}}_{\mathit{N}},{\tilde{\mathit{q}}}_{\mathbf{2}}〉}{〈{\tilde{\mathit{q}}}_{\mathbf{2}},{\tilde{\mathit{q}}}_{\mathbf{2}}〉}}{\tilde{\mathit{q}}}_{\mathbf{2}}\cdots -{\displaystyle \frac{〈{\mathit{a}}_{\mathit{N}},{\tilde{\mathit{q}}}_{\mathit{N}-\mathbf{1}}〉}{〈{\tilde{\mathit{q}}}_{\mathit{N}-\mathbf{1}},{\tilde{\mathit{q}}}_{\mathit{N}-\mathbf{1}}〉}}{\tilde{\mathit{q}}}_{\mathit{N}-\mathbf{1}}\end{array}\right.$$

(12)

$$\left\{\begin{array}{l}{\mathit{q}}_{\mathbf{1}}={\displaystyle \frac{{\tilde{\mathit{q}}}_{\mathbf{1}}}{‖{\tilde{\mathit{q}}}_{\mathbf{1}}‖}}\\ {\mathit{q}}_{\mathbf{2}}={\displaystyle \frac{{\tilde{\mathit{q}}}_{\mathbf{2}}}{‖{\tilde{\mathit{q}}}_{\mathbf{2}}‖}}\\ \cdots \\ {\mathit{q}}_{\mathit{N}}={\displaystyle \frac{{\tilde{\mathit{q}}}_{\mathit{N}}}{‖{\tilde{\mathit{q}}}_{\mathit{N}}‖}}\end{array}\right.$$

(13)

$$\mathit{Q}=\left[{\mathit{q}}_{\mathbf{1}},{\mathit{q}}_{\mathbf{2}},\cdots ,{\mathit{q}}_{\mathit{i}},\cdots ,{\mathit{q}}_{\mathit{N}}\right]$$

(14)

(3) Construct the $N$-order identity matrix $\mathit{E}$. Using matrix multiplication, map the orthogonal matrix to the original unordered multi-categorical variables one by one according to the position indexes to obtain the orthogonal numerical code matrix $\mathit{X}\in {\mathbb{R}}^{N\times N}$, which consists of row vectors.

$$\mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1}\to \mathit{X}:{\left[\begin{array}{c}{\mathit{x}}_{\mathbf{1}}\\ {\mathit{x}}_{\mathbf{2}}\\ \cdots \\ {\mathit{x}}_{\mathit{i}}\\ \cdots \\ {\mathit{x}}_N\end{array}\right]}_{N\times N}={\mathit{E}}_{N\times N}{{\mathit{Q}}^T}_{N\times N}={\left[\begin{array}{c}{{\mathit{q}}_{\mathbf{1}}}^{\mathit{T}}\\ {{\mathit{q}}_{\mathbf{2}}}^{\mathit{T}}\\ \cdots \\ {{\mathit{q}}_{\mathit{i}}}^{\mathit{T}}\\ \cdots \\ {{\mathit{q}}_{\mathit{N}}}^{\mathit{T}}\end{array}\right]}_{N\times N}={\left[\begin{array}{c}\begin{array}{cccc}{q}_1^1& q_1^2& \cdots & q_1^N\end{array}\\ \begin{array}{cccc}{q}_2^1& q_2^2& \cdots & q_2^N\end{array}\\ \cdots \\ \begin{array}{cccc}{q}_i^1& q_i^2& \cdots & q_i^N\end{array}\\ \cdots \\ \begin{array}{cccc}{q}_N^1& q_N^2& \cdots & q_N^N\end{array}\end{array}\right]}_{N\times N}$$

(15)

where ${\mathit{x}}_{\mathit{i}}={{\mathit{q}}_{\mathit{i}}}^T$ and ${\mathit{x}}_{\mathit{i}}$ is the row vector of $\mathit{X}$.

After the encoding of unordered multi-categorical variables, input the orthogonal matrix of numerical codes $\mathit{X}$ into the autoencoder for representation learning and dimension reduction.

3.2. Code Dimensionality Reduction

An autoencoder (AE) [28] is an unsupervised neural network model, which mainly consists of two parts, an encoder and decoder, and has a symmetric structure. Through neural network training, an autoencoder can achieve the tasks of data denoising, feature learning, or data dimensionality reduction [29].

An example of the structure of the autoencoder and the working principle of the autoencoder is shown in Figure 2. The autoencoder includes an input layer, a hidden layer, latent space, and an output layer, each with a different number of nodes. In this case, the structure from the input layer to the latent space is referred to as the encoder, and the structure from the latent space to the output layer is referred to as the decoder. The number of nodes in the input layer is the size of the original dimension of the data, the number of nodes in the latent space is the dimension in which the data needs to be compressed, and the number of nodes in the hidden layer is in between the original dimension and the compressed dimension.

The working principle of the autoencoder is as follows: the encoder compresses the input data into a predefined low-dimensional encoding through a series of transformations $f$, and the decoder tries to reconstruct the low-dimensional encoding into the original input data through a series of transformations $g$. Iterative training of the autoencoder is carried out by minimizing the reconstruction error between the input and the output, and ultimately, it is hoped that the autoencoder learns the abstract feature representation of the samples $\mathit{z}$.

The working principle of the autoencoder to carry out data dimensionality reduction is shown in Equation (16).

$$\begin{array}{l}f:\mathbb{R}\to \mathbb{Z}\\ g:\mathbb{Z}\to \mathbb{R}\\ \mathit{z}=f\left({\mathit{W}}_{\mathbf{1}}\mathit{x}+{\mathit{b}}_{\mathbf{1}}\right)\\ \widehat{\mathit{x}}=g\left({\mathit{W}}_{\mathbf{2}}\mathit{z}+{\mathit{b}}_{\mathbf{2}}\right)\\ f,g=arg\underset{f,g}{\mathit{min}}{‖\mathit{x}-g\left(f\left(\mathit{x}\right)\right)‖}_2\end{array}$$

(16)

In this paper, multiple symmetric fully connected layers are designed in the encoder and decoder of the autoencoder to increase the complexity of the model and feature extraction capability by stacking multiple connected layers.

ReLU is used as the activation function; ReLU is simpler to compute and has relatively stable performance. ReLU can avoid the gradient disappearing during model training, increase the nonlinear fitting ability of the model, and have good performance on multi-class tasks and datasets. The ReLU formula is $f\left(\mathit{x}\right)=max\left(0,\mathit{x}\right)$.

Taking the autoencoder model containing four hidden layers and one-dimensional data compression dimension as an example in Figure 3, ${\mathit{X}}_{N\times N}$ denotes the orthogonal numerical code matrix, and ${\widehat{\mathit{X}}}_{N\times N}$ denotes the numerical matrix after the reconstruction of the autoencoder. Let ${\mathit{X}}^{\left(1\right)}\in {\mathbb{R}}^{N\times a},{\mathit{X}}^{\left(2\right)}\in {\mathbb{R}}^{N\times b},a,b<N$ denote the process data compressed by the encoding layer. Let ${\widehat{\mathit{X}}}^{\left(2\right)}\in {\mathbb{R}}^{N\times b},{\widehat{\mathit{X}}}^{\left(1\right)}\in {\mathbb{R}}^{N\times a},a,b<N$ denote the process data that has been reconstructed by the decoding layer, and let $\mathit{Z}\in {\mathbb{Z}}^{N\times 1}$ denote the coded data that has undergone dimensionality reduction.

$\mathit{X},{\mathit{X}}^{\left(1\right)},{\mathit{X}}^{\left(2\right)},{\widehat{\mathit{X}}}^{\left(2\right)},{\widehat{\mathit{X}}}^{\left(1\right)},\widehat{\mathit{X}}$ all consist of row vectors.

The change in data dimensions during the encoding and decoding of the orthogonal numerical code matrix by AE is shown in Equation (17).

$$\begin{array}{ll}{\mathit{X}}_{N\times N}={\left[\begin{array}{c}{\mathit{x}}_{\mathbf{1}}\\ {\mathit{x}}_{\mathbf{2}}\\ \cdots \\ {\mathit{x}}_{\mathit{i}}\\ \cdots \\ {\mathit{x}}_N\end{array}\right]}_{N\times N}& \stackrel{encoder}{\to }{{\mathit{X}}^{\left(\mathbf{1}\right)}}_{N\times a}={\left[\begin{array}{c}{{\mathit{x}}^{\left(\mathbf{1}\right)}}_{\mathbf{1}}\\ {{\mathit{x}}^{\left(\mathbf{1}\right)}}_{\mathbf{2}}\\ \cdots \\ {{\mathit{x}}^{\left(\mathbf{1}\right)}}_{\mathit{i}}\\ \cdots \\ {{\mathit{x}}^{\left(\mathbf{1}\right)}}_{\mathit{N}}\end{array}\right]}_{N\times a}\stackrel{encoder}{\to }{{\mathit{X}}^{\left(\mathbf{2}\right)}}_{N\times b}={\left[\begin{array}{c}{{\mathit{x}}^{\left(\mathbf{2}\right)}}_{\mathbf{1}}\\ {{\mathit{x}}^{\left(\mathbf{2}\right)}}_{\mathbf{2}}\\ \cdots \\ {{\mathit{x}}^{\left(\mathbf{2}\right)}}_{\mathit{i}}\\ \cdots \\ {{\mathit{x}}^{\left(\mathbf{2}\right)}}_{\mathit{N}}\end{array}\right]}_{N\times b}\stackrel{encoder}{\to }{\mathit{Z}}_{N\times 1}\\ & ={\left[\begin{array}{c}{\mathcal{z}}_1\\ {\mathcal{z}}_2\\ \cdots \\ {\mathcal{z}}_i\\ \cdots \\ {\mathcal{z}}_N\end{array}\right]}_{N\times 1}\stackrel{decoder}{\to }{{\widehat{\mathit{X}}}^{\left(\mathbf{2}\right)}}_{N\times b}={\left[\begin{array}{c}{{\widehat{\mathit{x}}}^{\left(\mathbf{2}\right)}}_{\mathbf{1}}\\ {{\widehat{\mathit{x}}}^{\left(\mathbf{2}\right)}}_{\mathbf{2}}\\ \cdots \\ {{\widehat{\mathit{x}}}^{\left(\mathbf{2}\right)}}_{\mathit{i}}\\ \cdots \\ {{\widehat{\mathit{x}}}^{\left(\mathbf{2}\right)}}_{\mathit{N}}\end{array}\right]}_{N\times b}\stackrel{decoder}{\to }{{\widehat{\mathit{X}}}^{\left(\mathbf{1}\right)}}_{N\times a}={\left[\begin{array}{c}{{\widehat{\mathit{x}}}^{\left(\mathbf{1}\right)}}_{\mathbf{1}}\\ {{\widehat{\mathit{x}}}^{\left(\mathbf{1}\right)}}_{\mathbf{2}}\\ \cdots \\ {{\widehat{\mathit{x}}}^{\left(\mathbf{1}\right)}}_{\mathit{i}}\\ \cdots \\ {{\widehat{\mathit{x}}}^{\left(\mathbf{1}\right)}}_{\mathit{N}}\end{array}\right]}_{N\times a}\stackrel{decoder}{\to }{\widehat{\mathit{X}}}_{N\times N}={\left[\begin{array}{c}{\widehat{\mathit{x}}}_{\mathbf{1}}\\ {\widehat{\mathit{x}}}_{\mathbf{2}}\\ \cdots \\ {\widehat{\mathit{x}}}_{\mathit{i}}\\ \cdots \\ {\widehat{\mathit{x}}}_N\end{array}\right]}_{N\times N}\end{array}$$

(17)

More specifically, the steps for the dimensionality reduction of the orthogonal vector ${\mathit{x}}_{\mathit{i}}$ in the orthogonal numerical coding matrix are as follows.

(1) By means of the encoder, the orthogonal vector ${\mathit{x}}_{\mathit{i}}$ is mapped from a high-dimensional space to a low-dimensional space to achieve data dimensionality reduction.

$$\begin{array}{l}{{\mathit{x}}^{\left(\mathbf{1}\right)}}_{\mathit{i}}=max\left(0,{\mathit{W}}_{\mathbf{1}}^{\left(\mathbf{1}\right)}{\mathit{x}}_{\mathit{i}}+{\mathit{b}}_{\mathbf{1}}^{\left(\mathbf{1}\right)}\right)\\ {{\mathit{x}}^{\left(\mathbf{2}\right)}}_{\mathit{i}}=max\left(0,{\mathit{W}}_{\mathbf{1}}^{\left(\mathbf{2}\right)}{{\mathit{x}}^{\left(\mathbf{1}\right)}}_{\mathit{i}}+{\mathit{b}}_{\mathbf{1}}^{\left(\mathbf{2}\right)}\right)\end{array}$$

(18)

The coded data ${\mathcal{z}}_i$ after feature extraction of the model is obtained by linearly transforming the output of the previous layer to avoid introducing additional nonlinear relationships to enhance the complexity of the coded data, as in Equation (19).

$${\mathcal{z}}_i={\mathit{W}}_{\mathbf{1}}^{\left(\mathbf{3}\right)}{{\mathit{x}}^{\left(\mathbf{2}\right)}}_{\mathit{i}}+{\mathit{b}}_{\mathbf{1}}^{\left(\mathbf{3}\right)}$$

(19)

(2) The original data ${\mathit{x}}_{\mathit{i}}$ is reconstructed by mapping the low-dimensional space back to the high-dimensional space through the decoder.

$$\begin{array}{l}{{\widehat{\mathit{x}}}^{\left(\mathbf{2}\right)}}_{\mathit{i}}=max\left(0,{\mathit{W}}_{\mathbf{2}}^{\left(\mathbf{1}\right)}{\mathcal{z}}_i+{\mathit{b}}_{\mathbf{2}}^{\left(\mathbf{1}\right)}\right)\\ {{\widehat{\mathit{x}}}^{\left(\mathbf{1}\right)}}_{\mathit{i}}=max\left(0,{\mathit{W}}_{\mathbf{2}}^{\left(\mathbf{2}\right)}{{\widehat{\mathit{x}}}^{\left(\mathbf{2}\right)}}_{\mathit{i}}+{\mathit{b}}_{\mathbf{2}}^{\left(\mathbf{2}\right)}\right)\end{array}$$

(20)

The final output of the decoding layer, $\widehat{\mathit{X}}$, is obtained by linearly transforming the output of the previous layer to satisfy the purpose of output data reconstruction, as in Equation (21).

$${\widehat{\mathit{x}}}_{\mathit{i}}={\mathit{W}}_{\mathbf{2}}^{\left(\mathbf{3}\right)}{{\widehat{\mathit{x}}}^{\left(\mathbf{1}\right)}}_{\mathit{i}}+{\mathit{b}}_{\mathbf{2}}^{\left(\mathbf{3}\right)}$$

(21)

(3) The error is calculated between the reconstructed data ${\widehat{\mathit{X}}}_{\mathit{N}\times \mathit{N}}$ and the original data ${\mathit{X}}_{N\times N}$, generally using the mean-square error as a loss function $\mathcal{L}$.

$$\mathcal{L}={‖{\mathit{x}}_{\mathit{i}}-{\widehat{\mathit{x}}}_{\mathit{i}}‖}_2$$

(22)

(4) With the goal of minimizing the reconstruction error, the weights $\mathit{W}$ and bias $\mathit{b}$ in the encoder and decoder are optimised by error back propagation using a gradient-based method for the purpose of training the autoencoder.

$${f,g}_{\left(\mathit{W},\mathit{b}\right)}=arg\underset{f,g}{\mathit{min}}\mathcal{L}\left({\mathit{x}}_{\mathit{i}},{\widehat{\mathit{x}}}_{\mathit{i}}\right)$$

(23)

(5) After autoencoder training, the autoencoder learns the feature ${\mathcal{z}}_i$ of the data.

$${\mathit{x}}_{\mathit{i}}\stackrel{f_{O-AE}}{\to }{\mathcal{z}}_i$$

(24)

The same is true for the other orthogonal vectors, which ultimately results in the reduced dimensionality of the orthogonal numerical code matrix with numerical code $\mathit{Z}\in {\mathbb{R}}^{N\times 1}$.

$${\mathit{X}}_{N\times N}\stackrel{f_{O-AE}}{\to }{\mathit{Z}}_{N\times 1}={\left[\begin{array}{c}{\mathcal{z}}_1\\ {\mathcal{z}}_2\\ \cdots \\ {\mathcal{z}}_i\\ \cdots \\ {\mathcal{z}}_N\end{array}\right]}_{N\times 1}$$

(25)

From Equation (3), the values are the same after encoding in the same category. Therefore, the coded values after dimensionality reduction are expanded to the original sample size $\mathit{Z}\in {\mathbb{R}}^{\tilde{N}\times 1}$ to obtain the coded expressions for all unordered multi-categorical variables.

$$\mathcal{C}\mathcal{V}:{\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_i\\ \cdots \\ c_N\end{array}\right]}_{N\times 1}\to \mathcal{C}\mathcal{V}={\left[\begin{array}{c}{c}_1\\ c_2\\ \cdots \\ c_{i1}\\ c_{i2}\\ \cdots \\ c_N\end{array}\right]}_{\tilde{N}\times 1}\stackrel{f_{O-AE}}{\to }{\mathit{Z}}_{\tilde{N}\times 1}={\left[\begin{array}{c}{\mathcal{z}}_1\\ {\mathcal{z}}_2\\ \cdots \\ {\mathcal{z}}_{i1}\\ {\mathcal{z}}_{i2}\\ \cdots \\ {\mathcal{z}}_N\end{array}\right]}_{\tilde{N}\times 1}$$

(26)

4. Bayesian Optimization of Autoencoder Hyperparameters

The structure of the autoencoder model is not fixed. Since a dataset may include multiple unordered multi-categorical variables with varying numbers of categories, the number of hidden layers and nodes in the autoencoder adapts according to the dataset’s size and the required level of data compression. Therefore, parameters such as the number of layers and nodes in the hidden layer of the autoencoder are the key influencing factors that affect the performance of the autoencoder and the effect of data dimensionality reduction.

Bayesian optimization is suitable for black-box optimization problems where the target function is complex and has no analytical expression and the computational cost is high. The core idea is to use a probabilistic agent model (e.g., Gaussian process or random forest) to approximate the objective function and use this model to select the next evaluation point so that the objective function can converge to the optimal value as soon as possible, which is a kind of optimization method based on a priori information. There are two core components in this process: the probabilistic model and the acquisition strategy.

Bayes’ theorem is shown in Equation (27).

$$p\left(y|x\right)={\displaystyle \frac{p\left(x|y\right)p\left(y\right)}{p\left(x\right)}}$$

(27)

This paper uses the TPE (Tree-structured Parzen Estimator) algorithm in the Optuna framework to specify the implementation of the Bayesian optimization technique. The TPE is a tree-structured Bayesian optimization method. In the TPE, two density functions are used to define $p\left(x|y\right)$.

$$p\left(x|y\right)=\left\{\begin{array}{c}l\left(x\right),y<y^{*}\\ g\left(x\right),y\ge y^{*}\end{array}\right.$$

(28)

The TPE uses Kernel Density Estimation (KDE) to model the probability distribution of objective function values. Specifically, the TPE divides the objective function value into two components: a probability distribution $l\left(x\right)$ for the better objective value and a probability distribution $g\left(x\right)$ for the worse objective value.

The TPE uses Expected Improvement (EI) as a collection function.

$${EI}_{y^{*}}\left(x\right)={\int }_{-\infty }^{+\infty }max\left(y^{*}-y,0\right)p_M\left(y|x\right)dy={\displaystyle \frac{{\int }_{-\infty }^{y^{*}}\left(y^{*}-y,0\right)p\left(y\right)dy}{\gamma +\left(1-\gamma \right)\frac{g\left(x\right)}{l\left(x\right)}}}$$

(29)

This leads to Equation (30).

$${EI}_{y^{*}}\left(x\right)\propto {\left(\gamma +\left(1-\gamma \right){\displaystyle \frac{g\left(x\right)}{l\left(x\right)}}\right)}^{-1}$$

(30)

The value of EI is proportional to ${\left(\gamma +\left(1-\gamma \right){\displaystyle \frac{g\left(x\right)}{l\left(x\right)}}\right)}^{-1}$ and depends on the ratio of the two probabilities, so it is necessary to find the $x$ that makes the ratio ${\displaystyle \frac{l\left(x\right)}{g\left(x\right)}}$ maximal and, in each iteration, return the maximum EI.

5. Experimental Design

5.1. Dataset

In order to validate the feasibility and effectiveness of the proposed method, this section carried out example validation on five datasets and compared the results with those of one-hot encoding, embedding, label encoding, and target encoding.

In this paper, the working-hour dataset of two typical majors in the cruise ship production design process, namely piping majors and hull structure majors, were selected from the major cruise shipbuilding yards in domestic market, and three public datasets containing unordered multi-categorical variables from UCI and Kaggle were also selected. The sample sizes of the five datasets as well as the number of categories in the unordered multi-categorical variables differed from each other, which made the method of this paper feasible to validate the universality of the method.

In particular, the sample size in the Gait dataset was so large that we randomly selected 1000 pieces of data to form a new dataset for the experiment.

The sample size of the data set, the number of variables, the number of categorical variables, the number of categories in the categorical variables, and the targets used for prediction are shown in the following table,

Table 1. Specific information on the dataset.

Dataset	Sample Size ($\tilde{\mathit{N}}$)	Total Number of Variables	Number of Continuous Variables	Categorical Variable			Target
				Number of Categorical Variables	Name	Number of Categories (N)
Piping	163	8	6	2	area	16	Feedback working hours
					task type	19
Hull structure	144	4	2	2	area	67	Feedback working hours
					task type	4
Gait [30]	1000	6	2	4	subject	10	angle
					condition	3
					joint	3
					leg	2
Restaurant [31]	1000	7	5	2	Cuisine_Type	4	Monthly_Revenue
					Promotions	2
Fish [32]	159	6	5	1	Species	7	Weight

.

5.2. Detailed Experimental Design

The dimension of the values obtained by different encoding methods is not the same. In this paper, we set the dimension of autoencoder compression to be one-dimensional.

Therefore, O-AE, label encoding, and target encoding received one-dimensional codes, one-hot encoding received the same dimensionality as the number of categories in the categorical variables, and embedding methods were free to set the dimensionality size.

We were concerned with the effect of different encoding methods on the input dimensions of the dataset as well as on the target prediction results of the neural network, so we designed 2 experiments with different embedding dimensions for embedding and for O-AE, one-hot encoding, embedding, label encoding, target encoding, and Bayesian optimization of the hyperparameters of the autoencoder for the O-AE-b method, respectively, in 1 experiment.

The encoded dataset was input to BP neural network to carry out target prediction.

In this paper, only the multi-categorical variables “subject” in the Gait dataset were encoded; the categorical variables “condition” and “joint” in the Gait dataset contained a small number of categories, and “leg” was a binary categorical variable, so the experiments were conducted using the labeled data that came with the original dataset.

The multi-categorical variables “Cuisine_Type” in the Restaurant dataset were encoded and the binary categorical variables “Promotions” were experimented with using the labeled data that came with the original data.

Continuous variables in the dataset were mapped to a standard normal distribution with a mean of 0 and standard deviation of 1 by Z-Score standardization.

The specific methods of the seven experiments and the total dimensions of the five datasets input to the BP neural network under each of the seven experimental codes are shown in

Table 2. The experimental design and the total input dimensions after encoding.

No.	Encoding Method	Code Dimension	Total Input Dimensions					Experiments Abbreviation
			Piping	Hull Structure	Gait	Restaurant	Fish
1	Orthogonal matrix-autoencoder-based encoding method	1	8	4	6	7	6	O-AE
2	Embedding	1	8	4	6	7	6	1-EE
3		5	16	12	10	11	10	5-EE
4	One-hot encoding	Number of categories (N)	31	73	15	10	12	OHE
5	Label encoding	1	8	4	6	7	6	LE
6	Target encoding	1	8	4	6	7	6	TE
7	Bayesian optimization of the hyperparameters of the autoencoder for O-AE	1	8	4	6	7	6	O-AE-b

.

5.3. Evaluation Metrics

In this paper, three classical evaluation metrics were used to assess the performance of different encoding methods in the BP neural network target prediction task, including Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and coefficient of determination ($R^2$).

$$MAE={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\left|y_i-y_i\right|, MAE\in \left[0, +\infty \right]$$

(31)

$$RMSE=\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}, RMSE\in \left[0,+\infty \right]$$

(32)

$$R^2=1-{\displaystyle \frac{{\sum }_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}{{\sum }_{i=1}^n{\left(y_i-\overline{y}\right)}^2}}, \overline{y}={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{y}_i, R^2\in \left[0,1\right]$$

(33)

where $n$ is the number of samples, $y_i$ is the actual value, ${\widehat{y}}_i$ is the predicted value, and $\overline{y}$ is the mean of the actual value.

MAE is the mean of the absolute errors, and RMSE is the square root of the mean of the sum of squares of the errors. MAE and RMSE are both measurements of the deviation of the predicted values from the actual values, with smaller values indicating better final predictions. $R^2$ is used to assess the linear relationship between the actual values and the predicted values, with values closer to 1 indicating a better fit of the model.

6. Example Validation

In this section, we analyze the performance of seven experiments. We implemented all the algorithms in Python and conducted all the experiments on a computer with an i7-11390H processor, 3.40 GHz CPU, and 16.0 GB RAM.

6.1. Parameter Setting

The relevant parameters of the autoencoder when carrying out Experiment 1 and Experiment 7 are shown in

Table 3. The parameter setting of the autoencoder in O-AE and O-AE-b.

Experiment	Loss Function	Activation Function	Optimizer	Learning Rate	Epoch	Batch Size	Patience
O-AE	MSE	Relu	Adam	0.001	2000	$\mathrm{Sample}\mathrm{size}(\tilde{N}$)	-
O-AE-b	MSE	Relu	Adam	0.001	2000	$\mathrm{Sample}\mathrm{size}(\tilde{N}$)	100

below. The batch size is set as the sample size of the dataset due to the small sample size of the dataset for the experimental design.

In particular, in Experiment 7, the unordered multi-categorical variables are divided into training and validation sets according to 7:3. The early-stopping strategy is introduced, and the patience parameter for early stopping is set to 100. The number of AE hidden layers and the number of nodes per layer are obtained with Bayesian optimization. The number of encoder hidden layers is set to be no more than five, and the decoder and encoder structures are symmetric.

The number of hidden layers of the autoencoder and the number of nodes in each layer in Experiment 1 are manually and dynamically adjusted according to different datasets. The epoch of Experiment 1 was kept the same as Experiment 7. Also, since there is no additional autoencoder hyperparameter optimization procedure for Experiment 1 and in order to compare with other experiments to demonstrate the applicability of Experiment 1, the early-stopping strategy is not used.

To carry out target prediction using a BP neural network, the dataset is divided into a training set, validation set, and test set in the ratio of 7:2:1.

The relevant parameters are shown in

Table 4. The parameter setting of the BP neural network.

Loss Function	Activation Function	Optimizer	Learning Rate	Epoch
MSE	Relu	Adam	0.001	1000
L2 Regularization	Batch size	Patience	Optional hidden layers	Optional nodes
0.0001	$\mathrm{Sample}\mathrm{size}(\tilde{N})$	100	10, 20, 30	1, 2, 3

. The batch size is set as the sample size of the dataset due to the small sample size of the dataset for the experimental design. In order to quickly conduct experiments and obtain a better prediction result, the hyperparameter optimization method of a 5-fold cross-validation grid search is used, and the number of nodes in each hidden layer is set to be the same; the number of optional hidden layers and the number of optional nodes are shown in Table 4. In order to improve the model performance and avoid overfitting, we introduce L2 Regularization and early-stopping strategy and set the patience parameter of early stopping to 100.

Meanwhile, the original code generation step randomly generates $N$-order square matrix $\mathit{A}\in {\mathbb{R}}^{N\times N}$ based on the number of categories $N$ in the unordered multi-categorical variables and fixes the random seed to ensure that the experimental data are the same.

6.2. Target Prediction Results

Under the above parameter settings, several calculations are carried out to obtain the results of the evaluation metrics for the five datasets in seven experiments. The number of nodes per layer of the encoding layer of the autoencoder (the decoding layer is symmetrical to the structure of the encoding layer, so the description is omitted) and the number of layers and nodes in the hidden layer of the BP neural network after hyperparameter optimization are shown in

Table 5. Optimal hyperparameter results for autoencoder and BP neural network.

Dataset	Hyperparameter		O-AE	1-EE	5-EE	OHE	LE	TE	O-AE-b
Piping	AE	area	16-6-1	-					16-13-11-9-5-1
		task type	19-6-1						19-18-16-13-12-1
	BP	layers	3	3	3	3	3	3	3
		nodes	30	10	20	30	30	20	30
Hull structure	AE	area	67-30-6-1	-					67-26-17-1
		task type	4-2-1						4-2-1
	BP	layers	3	3	3	3	3	3	3
		nodes	30	20	30	30	20	30	30
Gait	AE	subject	10-4-1	-					10-9-8-5-1
	BP	layers	3	3	3	3	3	3	3
		nodes	30	30	30	30	30	30	30
Restaurant	AE	Cuisine_Type	4-3-1	-					4-2-1
	BP	layers	3	3	3	3	3	3	3
		nodes	30	30	10	20	30	20	30
Fish	AE	Species	7-3-1	-					7-6-5-1
	BP	layers	3	3	3	3	3	3	3
		nodes	30	30	30	30	30	30	30

.

In this paper, we denote the model structure of the autoencoder by unordered multi-categorical variables’ original dimension − intermediate compression dimension − 1.

The results of the evaluation metrics for the five datasets in seven experiments are shown in

Table 6. The results of the evaluation metrics.

		O-AE	1-EE	5-EE	OHE	LE	TE	O-AE-b
Piping	MAE	4.3681	6.8860	6.1809	4.8380	5.6480	8.1285	4.5841
	RMSE	5.8296	9.8875	9.9672	7.9449	9.2587	11.1007	5.7244
	$R^2$	0.9718	0.9189	0.9175	0.9476	0.9288	0.8977	0.9728
Hull structure	MAE	8.4987	12.3219	10.6569	6.4534	9.9077	14.3765	5.8390
	RMSE	10.2222	17.5980	15.5944	12.5951	15.5421	19.2587	7.6556
	$R^2$	0.9628	0.8896	0.9133	0.9435	0.9139	0.8678	0.9791
Gait	MAE	3.4295	3.4297	3.1289	3.6434	3.8893	4.2187	3.3678
	RMSE	4.5314	4.7570	4.6198	5.1868	5.1943	5.7029	4.2098
	$R^2$	0.9319	0.9250	0.9292	0.9108	0.9105	0.8921	0.9412
Restaurant	MAE	40.8338	42.3132	42.3016	41.4832	41.1913	41.6640	40.9350
	RMSE	52.6461	54.2528	53.0498	52.8340	53.3646	52.6894	51.9348
	$R^2$	0.7559	0.7407	0.7521	0.7541	0.7492	0.7555	0.7624
Fish	MAE	21.7449	24.5180	22.9351	20.7503	23.8476	56.8344	20.8339
	RMSE	32.1668	34.4302	34.8801	33.4807	33.7599	72.4618	31.8064
	$R^2$	0.9867	0.9848	0.9844	0.9856	0.9854	0.9327	0.9870

Bold indicates the best and worst values of the evaluation metrics.

.

This paper plots the results of the target prediction experiments for each dataset, including the fitted regression plots between the predicted and actual values in the test set, as shown in Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8, and the line charts between the predicted and actual values in the test set, as shown in Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13.

6.3. Analysis of Results

In this paper, MAE, RMSE and $R^2$ are chosen as the evaluation metrics of target prediction results, and the smaller MAE and RMSE, and the closer $R^2$ is to 1 means the target prediction quality is better.

(1) Comparing the metrics results of target prediction.

In the Piping dataset, the MAE results obtained by O-AE are the best, the RMSE and $R^2$ results obtained by O-AE-b are the best, and the MAE results of O-AE-b are the best among the seven experiments except for O-AE. The RMSE and $R^2$ results of O-AE are also the best among the seven experiments except for O-AE-b. The results of all three metrics for TE are worse in the other experiments.

In the Hull structure dataset, the MAE, RMSE, and $R^2$ results obtained by O-AE-b are the best, and the RMSE and $R^2$ results obtained by O-AE are also the best among the seven experiments except for O-AE-b. The results of all three metrics for TE are worse in the other experiments.

In the Gait dataset, the MAE results obtained by 5-EE are the best, the RMSE and $R^2$ results obtained by O-AE-b are the best, and the MAE results of O-AE-b are the best among the seven experiments except for 5-EE. The RMSE and $R^2$ results of O-AE are also the best among the seven experiments except for O-AE-b. The results of all three metrics for TE are worse in the other experiments.

In the Fish dataset, the MAE results obtained by OHE are the best, the RMSE and $R^2$ results obtained by O-AE-b are the best, and the MAE results of O-AE-b are the best among the seven experiments except for OHE. The RMSE and $R^2$ results of O-AE are also the best among the seven experiments except for O-AE-b. The results of all three metrics for TE are worse in the other experiments.

In particular, the prediction results of the seven experiments on the Restaurant dataset are not very satisfactory, with the highest $R^2$ of only 0.7624, which is analyzed in this paper as a possible reason for the fact that the chosen BP neural network model is not applicable to the Restaurant dataset. However, in the Restaurant dataset, the MAE results obtained by O-AE are the best, the RMSE and $R^2$ results obtained by O-AE-b are the best, and the MAE results of O-AE-b are the best among the seven experiments except for O-AE. The RMSE and $R^2$ results of O-AE are also the best among the seven experiments except for O-AE-b. The results of all three metrics for 1-EE are worse in the other experiments.

Overall, the Bayesian optimization of the hyperparameters of the autoencoder for O-AE as well as the basic O-AE have consistently excellent target prediction results when experiments are carried out on different datasets. Analyzing the metric results, the basic O-AE can then meet the encoding needs of unordered multi-categorical variables, while the MAE results of O-AE are better than the MAE results of O-AE-b in some datasets.

(2) Comparing the experimental performance of different methods.

In other experiments, 1-EE and 5-EE are embedding methods with one and five embedding dimensions, respectively, and the metric results of 1-EE and 5-EE are in the middle of the range on the five datasets, with little difference in experimental performance. The metric results of OHE on the five datasets are also more stable, and the experimental performance is a little bit poorer than that of O-AE and O-AE-b, but OHE has the problems of unordered multi-categorical variables with larger dimensions and sparse data after encoding. The experimental performance of LE and TE on the five datasets is general mainly because the use of LE introduces additional order misclassification and distance bias in addition to the originally unordered features, and TE affects the fitting ability of the neural network model.

Overall, the experimental results of O-AE-b and O-AE are superior compared to other classical encoding methods. The unordered multi-categorical variables processed by O-AE-b and O-AE were able to satisfy the data input requirements of the subsequent neural network model and facilitated the ability of neural network data learning to improve the neural network target prediction results.

7. Conclusions

The objective of this paper is to find an encoding method for unordered multi-categorical variables that results in the lower dimensionality of the encoded data and better target prediction results when fed into a neural network.

After analyzing the characteristics of unordered multi-categorical variables and comparing them with other encoding methods, this paper proposes a method for the encoding and dimensionality reduction of unordered multi-categorical variables using an orthogonal matrix-autoencoder. Seven experiments are designed for validation using the basic O-AE, the Bayesian optimization of the hyperparameters of the autoencoder for O-AE, and several other classical encoding methods. The experimental results show that O-AE and O-AE-b have more stable and excellent evaluate metrics, indicating that the method proposed in this paper is highly feasible and applicable and can be an optional method for data processing of unordered multi-categorical variables.

Through the experimental analysis, the basic O-AE can meet the encoding requirements of unordered multi-categorical variables, but O-AE manually adjusts the number of layers and nodes in the hidden layer of the autoencoder and needs to adjust the parameters several times to achieve the optimal results of the metrics. O-AE-b uses Bayesian optimization to find the optimal number of layers and nodes in the hidden layer, but the Bayesian optimization of hyperparameters consumes more time than that of the basic O-AE. Therefore, the encoding method can be selected according to the characteristics of the problem and the demand of experimental resources in the specific use.

Although the research in this paper has achieved preliminary results, it still contains vast room for deepening. Future work will focus on optimizing the model performance and try to introduce activation functions such as PReLU or other regularization strategies such as L1-L2 with a view to further improve the model performance. Meanwhile, we will also work on improving the efficiency of Bayesian optimization in the O-AE-b model and exploring more new ways to efficiently optimize the hyperparameters of the autoencoder in order to promote the continuous progress of research in this area.