Determining Optimal Assembly Condition for Lens Module Production by Combining Genetic Algorithm and C-BLSTM

Abstract

Mobile camera modules are manufactured by aligning and assembling multiple differently shaped part lenses. Therefore, selecting the part lenses to assemble from candidates (called cavities) and determining the directional angle of each part lens for assembly have been important issues to maximize production yield. Currently, this process is manually conducted by experts at the manufacturing site, and the manual assembly condition optimization carries the risk of reduced production yield and increased failure cost as it largely depends on one’s expertise. Herein, we propose an AI framework that determines the optimal assembly condition including the combination of part lens cavities and the directional angles of part lenses. To achieve this, we combine the genetic algorithm with convolutional bidirectional long-term short-term memory (C-BLSTM). To the best of our knowledge, this is the first study on lens module production finding the optimal combination of part lens cavities and directional angles at the same time using machine learning methods. Based on experimental results using real-world datasets collected by lens module manufacturers, the proposed framework outperformed existing algorithms with an F1 score of 0.89. Moreover, the proposed method (S2S-AE) for predicting the directional angles exhibited the best performance compared to existing algorithms with an accuracy of 78.19%.

1. Introduction

Over recent years, data-driven artificial intelligence (AI) has proved to be a promising approach in various manufacturing fields. Moreover, several related topics have been addressed, including fault detection [1,2,3], fault diagnosis [4,5], fault inspection [6,7,8], facility failure pattern extraction [9], cost estimation [10,11], and demand forecasting [12]. However, fewer studies have addressed the problem of applying data-based AI to find the optimal conditions or conditions of parts in the manufacturing processes.

In the part lens assembly process for mobile camera modules (MCMs), the high failure costs of disposed part lenses and the long time taken to find the optimal assembly condition are the most urgent problems. As displayed in Figure 1, each lens module is manufactured by assembling a number of part lenses and spacers in a barrel. Considering that the part lenses are selected from dozens of cavities and each part lens is assembled in various directional angles, this is classified as a heuristic problem without a suitable model. Currently, experts at the manufacturing sites manually verify the optimal assembly conditions through testing, then input these assembly conditions into an automatic assembly machine to produce high-quality lens modules. Therefore, experts involved in the MCM lens assembly process experience difficulties, such as having to find assembly conditions through numerous trials and errors.

When manufacturing part lenses, multi-cavity molds are adopted, enabling multiple part lenses to be produced from each mold. The physical values of the produced part lenses deviate due to the inevitable errors in the physical properties of cavities in a mold. Experts name the part lenses by cavity number and determine the cavity number combination in terms of part lens types. For instance, for a lens module with seven types of part lenses aligned in a single barrel, the number of possible combinations of part lenses is $m^n$, where m is the number of cavities and n is the number of lens types. Meanwhile, the physical value of the part lens can be obtained by precisely measuring its outer shape. Then, experts select the directional angle of each part lens to align the optical axes of the lenses in a row. In an automatic assembly machine, the directional angle from 0° to 360° is adjusted by rotating it at intervals of 5°, so a number of combination sets of 360°/5° to the power of seven lenses can be obtained.

Herein, we solve the problem of determining two optimal assembly conditions for producing more productive and efficient lens modules through AI algorithms. The problem is divided into two sub-problems: First, a process of determining sets of cavity combinations of part lenses is performed. In this process, we also optimize in a way that minimizes the number of lenses that are discarded due to not being adopted for assembly. Second, the directional angle at which the optimal resolution is expected for the selected component lens according to the first step is predicted.

Several studies have attempted to solve the previous problems by using population-based metaheuristic methods, since those have the capability to solve problems with a high-dimensional search space [13]. A real-coded genetic algorithm (GA) was used in a multi-objective part lens optimization problem to apply it to a three-element optical lens module design [14]. Another study used the GA to find the optimum part lens sets for the combination of anomalous dispersion glass and materials inside a liquid lens by utilizing equations on their respective chromatic aberration and root mean square spot diameter [15]. Li et al. [16] suggested a GA and deep neural network (DNN)-based surrogate model for lens matching in lens module assembly to replace expensive optical simulators. Our lens module assembly optimization method, which selects both cavity combinations and orientation angle combinations, has previously been submitted for patent in Korea, the United States, and China [17].

Moreover, LSTM-based prediction methods can solve problems with a reflection of sequentiality [18,19]. Li et al. [20] presented an anomaly identification method in mechanical equipment based on an LSTM. Similarly, Essien and Giannetti [21] proposed a deep convolutional LSTM architecture to predict machine speed sequentially.

To this end, we propose a framework based on GA and C-BLSTM to determine the optimal assembly conditions for lens module production. First, the optimal set of cavity number combinations is determined by utilizing a unique design of a GA framework. Then, a sequence-to-sequence autoencoder (S2S-AE) and an architecture of a C-BLSTM are used to extract sequential representations and predict the directional angle combinations. Finally, we discuss our model’s generalization ability, which allows it to adapt appropriately to a new lens module product by exploiting feature engineering and transfer learning.

The remainder of this paper is organized as follows. Section 2 presents the proposed framework. Specifically, after briefly explaining an overview of the framework and its GUI, each process is described in detail sequentially. The effectiveness of the proposed framework is presented in Section 4, and the paper is concluded in Section 6.

2. Proposed Framework

2.1. Overview and Graphical User Interface (GUI)

The proposed framework is divided into two steps, as shown in Figure 2, mirroring the real lens module assembly process: determining the optimal set of cavity number combinations and the optimal directional angle combinations. In the initial phase, to simplify the problem’s complexity, the set of cavity combinations for part lenses is decided first based on the unique GA design, followed by the selection of the barrel cavity combination. In this context, the performance of each cavity number combination is evaluated by a pre-trained defect predictor. The input features for the predictor include the physical attributes of part lenses or a barrel, and the output indicates whether the given cavity number combination yields a defective product or not. Moving on to the subsequent step, given the optimal set of cavity number combinations, we ascertain the optimal set of directional angle combinations. To capture the sequential nature of the manual angle decision process used by experts, C-BLSTM is employed to predict directional angles for each individual lens.

We suggest the GUI and its algorithm for implementing the proposed framework in real-world manufacturing sites. As depicted in Figure 3, the system’s GUI shows a conventional and intuitive design with four sections: the physical attributes import section, the model parameter input section, the operation button section, and the output display section. The system requires two types of experts: machine learning engineers and assembly experts. Machine learning engineers are responsible for determining the parameters and setting them in advance at the GUI. At the manufacturing site, assembly experts import the physical attributes of part lenses and a barrel, executing the system to obtain optimal results. We recommend installing the system on a desktop adjacent to automatic assembly machines for seamless integration into the manufacturing process. This setup allows experts to operate the system directly. The detailed operation procedure is presented in Algorithm 1.

Algorithm 1 The operation procedure of the inference system GUI

Optimal assembly conditions for producing a lens module 1-1. Load features from an SQL table by selecting data such as part lens mold, part lens production date, barrel mold, and barrel production date. Except unusable part lenses by selecting cavity numbers. 1-2. Apply the dataset, which is imported instantaneously. 2-1. Determine main parameters such as population size, generation size, selection rate, mutation rate, and the dimension of MTF in the Parameter 1 section. Additionally, import CSV files of rule, past combination, and feature maximum spec. 2-2. Since the Parameter 2 section contains the core parameters of the algorithm, it is unable to change without determination by a developer. 2-3. Save setting all parameters. 3. Execute the framework of GA. 4. Finally, the optimal assembly conditions of part lens cavities after one-to-one matching with barrel cavities, and their predicted performances are exhibited as outputs.

2.2. Determining the Optimal Set of Cavity Number Combinations

This process consists of two steps. First, the optimal set of part lens cavity number combinations is determined based on the GA, which is a randomized search and optimization technique that is guided by the theory of Darwinian evolution and natural genetics [22]. Then, the appropriate barrel cavity numbers for non-defective combinations in a set are selected. During the process, defect predictors are used that are trained by production logs collected from the manufacturing site.

2.2.1. Determining a Set of Part Lens Cavity Number Combinations

Basic Principles of GA

The GA is appropriate for determining the optimal set of combinations because it performs a search in complex and large landscapes and provides near-optimal solutions for the objective of a given problem [23]. In the GA, parameters of the search space are encoded in the form of chromosomes, and a collection of such chromosomes is termed a population. According to evolutionary theory, the most remarkable chromosomes in a population are likely to survive and generate offspring. Thus, their biological heredity can be transmitted to new generations through biologically inspired operators, such as crossover and mutation [24]. In this evolutionary process, a fitness value is used to measure the degree of goodness of a chromosome, and the process continues for a fixed number of generations or until a specified performance is reached [25].

Chromosome Representation

Next, we define newly chromosomes and genes for exploring the optimal set of part lens cavity number combinations, where each chromosome represents a candidate solution to the problem [26]. Here, a gene is a unit of strings, and a chromosome is a set of multiple genes. Each element of a gene encodes high-dimensional parameters that are associated with part lens attributes (such as diameter and decenter) as a cavity number.

We denote a set of part lens cavity number combinations, which is equal to a chromosome, by a matrix $C={\left[c_{ij}\right]}^{I\times J}$, where each row of a matrix indicates a candidate combination. $c_{ij}$ indicates the cavity number of the j-th part lens in the i-th combination, where I and J represent the number of cavities in a multi-cavity mold and the number of part lenses in a lens module, respectively. Figure 4 depicts a chromosome with 7 part lenses and 16 cavities. In the figure, each gene element can be an integer value ranging from 1 to 16, since 16 part lenses are obtained once injection molding is performed. The cavity number cannot appear more than once in a column of the chromosome.

Initial Population Generation

The expert’s method of finding the optimal combination in a manufacturing site are adopted for generating the initial population. Originally, it took a long time to determine the optimal combination starting from random cavity number combinations. Thus, experts acquired know-how in the field to quickly find the optimal result. The know-hows are used to design the initial population strategies to provide a certain clue for solving constraint satisfaction problems (such as the one presented), which then converge rapidly to generate good results.

First, a chromosome is generated based on past combinations that caused non-defective lens modules, as depicted in Figure 5a. When determining the optimal set of cavity number combinations at a manufacturing site, experts use the cavity number combinations that have produced non-defective lens modules in the past. Therefore, our framework initially forms genes first based on this information from the past and randomly assigns unused cavity numbers to the remainder of the genes.

The second strategy is the regular combination-based generation, where a regular combination indicates a combination with all elements having the same value. When a new lens module product is produced at a manufacturing site, there is no previous information for non-defective lens modules. Thus, experts have previously assembled part lenses with the same cavity numbers. The chromosome generated based on this strategy is illustrated in Figure 5b.

It is possible to compose a chromosome based on rules controlling lens module quality in terms of physical attributes, such as shape and size factors. Bad cavities are listed and collected at only a few genes. This method is defined as rule-based generation, which is illustrated in Figure 5c. The final method is a random generation, where cavity numbers in a chromosome are randomly assigned without any rule.

Fitness Evaluation

Non-defective and recall scores are used as two fitness values, where a fitness value is a metric used to evaluate whether each chromosome is suitable to be for the new generation. The non-defective score means the number of genes causing a non-defective lens module, and it is denoted by $f_1$. The recall score is denoted by $f_2$ and represents the number of genes equal to the previous combinations that produced non-defective lens modules.

A deep learning-based defect predictor (DP) is employed for the evaluation of whether each gene consisting of a chromosome causes either a defective or non-defective lens module. The goal of the defect predictor is to classify the condition of a lens module precisely. As displayed in Figure 6, the DP consists of two parts: a defect predictor training part and a lens module defect prediction part. The predictor is trained using the physical attributes of part lenses (such as diameter and thickness) and a binary result of the corresponding lens module to capture the representation features. Then, an assembled lens module, according to the given cavity number combination, is classified as to whether a lens module is defective.

We address the data imbalance problem in production logs with a different form of loss function, the mean false error (MFE) [27]. The MFE is inspired by the concept of “false error” composed of “false positive error (FPE)” and “false negative error (FNE)” in a confusion matrix. This renders the MFE more sensitive to errors from the minority class by calculating the FPE and FNE, and the MFE is calculated as Equation (1).

$$\begin{array}{cc}\hfill MFE& =FPE+FNE,\hfill \\ \hfill FPE& =\frac{1}{P}\times \sum _{i=1}^{P}loss(y_i,\widehat{y_i}),\hfill \\ \hfill FNE& =\frac{1}{N}\times \sum _{i=1}^{N}loss(y_i,\widehat{y_i}),\hfill \end{array}$$

(1)

where P and N, respectively, indicate the number of instances in the majority and minority classes, and $y_n$ and $\widehat{y_n}$ are the real and predicted lens module conditions, respectively.

Selection

A selection strategy to reproduce an improved next generation is mainly governed by the two different fitness values: non-defective score and recall score, and the strategy is illustrated in Figure 7. The initial population (P) and parent candidates ($P_1$ and $P_2$) are generated as following Equation (2).

$$\begin{array}{c}\hfill P=\{C_1,C_2,\cdots ,C_n\},\\ \hfill \mathrm{Sorting}\mathrm{by}{f}_1\mathrm{score}:C_1^1,C_2^1,\cdots ,C_n^1,\\ \hfill \mathrm{Sorting}\mathrm{by}{f}_2\mathrm{score}:C_1^2,C_2^2,\cdots ,C_n^2,\\ \hfill P_1=\{C_1^1,C_2^1,\cdots ,C_m^1\},(m<n),\\ \hfill P_2=C_1^2,\end{array}$$

(2)

where $P_1$ is a set containing m chromosomes selected after sorting in descending order by the non-defective score ($f_1$), which equals the number of non-defective genes in a chromosome. Crossover operation on m chromosomes from $P_1$, and the first ordered chromosome with the highest recall score ($f_2$) from $P_2$ by m:1 matching will eventually produce an offspring set.

Crossover

New offspring are created from parent chromosomes based on two types of crossover operators: single-point crossover and gene-mix and swap crossover. The former is a method of easily maintaining feasibility, while the latter is a method for maintaining the good characteristics of both parents. When we run the model, we should set a threshold in range from 0 to 1, and this value controls the number of times that each operator is executed. For example, the system generates a random number, and if the number is smaller than the threshold value, the second crossover operator is executed, and otherwise, the first crossover operator is implemented. In the GUI, the threshold can be entered in ‘Crossover Type Rate’ as shown in Figure 3.

In the single-point crossover, one cut-point is selected, and the left part from the cut-point of Parent 1 and the right part from the cut-point of Parent 2 are combined to generate an offspring, as depicted in Figure 8a. With respect to the presented problem, it is challenging to maintain feasibility in which the cavity number per column should appear only once. The single-point crossover is easy to prevent broken feasibility that more than one cavity number exists in one column.

The concept of gene-mix and swap crossover, the concept is to preserve the good characteristics of a parent and replace any bad characteristics with the remarkable properties of the other parent. These processes are illustrated in Figure 8b in detail. Specifically, a chromosome of Parent 2 is copied, and the worst performance of the gene is replaced with the best performance of the gene from Parent 1. However, there is then the possibility of broken feasibility, since the worst gene in Parent 1 is not identical to the best gene in Parent 2. Therefore, a duplicated cavity number is swapped with a cavity number from the other columns.

Mutation

In the GA, the mutation is mainly performed by a simple swap operator, in which only a small random change is permitted with a rate of ∼0.2 to escape from a local minimum problem. For a selected chromosome, a random integer i in the range from 1 to the number of cavities I is chosen, which is termed the mutation point. Then, the i-th column of a chromosome is selected, and cavity numbers in the column are swapped, as described in Figure 9. This mutation operator can be implemented simply without affecting the feasibility of a chromosome.

2.2.2. Selecting Barrel Cavity Number

In this section, the process of selecting a potential barrel cavity number with part lens cavity number combinations obtained in the previous step is described. The most appropriate barrel cavity with the highest non-defective probability is selected for each part lens cavity number combination. DP is utilized in the same way as in the previous step, and it is trained with physical measurements of the barrels in addition to the part lenses.

2.3. Determining the Optimal Directional Angle Combinations

The process for determining the optimal directional angle combinations consists of two steps: feature extraction and directional angle prediction, as depicted in Figure 10. First, the representation features of physical attributes are extracted. The features are obtained by performing S2S-AE [28] which learns a sequentially meaningful representation of the input. Second, the directional angle for each part lens is predicted based on the previously extracted features. This step employes C-BLSTM, which is a method that combines deep convolutional neural networks (DCNN) [29] and bidirectional LSTM (BLSTM) [30]. C-BLSTM performs directional angle combination prediction by reflecting the sequentiality of the lens module assembly process.

2.3.1. Feature Extraction

In this step, the process of extracting the representation containing the sequential characteristics of multiple part lenses based on S2S-AE is described in detail. An S2S-AE consists of two recurrent neural networks that acts as a pair of functions $(e,d)$ and is designed to encode the input data and decode the encoded data [28]. It takes an input vector $\mathbf{x}\in {\mathbb{R}}^d$ and maps the input vector to a latent vector space $\mathbf{c}\in {\mathbb{R}}^{d^{\prime }}$ through an encoder e. Then, the latent representation $\mathbf{c}$ is mapped back to a reconstructed vector $\mathbf{z}\in {\mathbb{R}}^d$ through a decoder d. The objective function of the S2S-AE is defined as Equation (3), and the S2S-AE tries to minimize Equation (3).

$$L(\mathbf{x},\mathbf{z})={\parallel \mathbf{x}-\mathbf{z}\parallel }^2,$$

(3)

where the encoding and decoding functions are parameterized using neural networks.

Figure 11 presents a graphical depiction of the S2S-AE, where $x_{N_a}$ and $z_{N_a}$ represent an input vector and a reconstructed vector of a $N_a$-th lens, respectively, and $\mathbf{c}$ and $N_a$, respectively, indicate the latent representation of multiple lenses and the number of physical attributes comprising the input data. It is evident from the figure that the encoder sequentially reads each physical attribute of a lens sequence $\mathbf{x}$. After reaching the end of the lens sequence, the whole input sequence is summarized as $\mathbf{c}$. The decoder is trained to generate the output physical attribute sequence by predicting the symbol of the t-th state, $z_t$, given the hidden state and the summary, $\mathbf{c}$. Once the S2S-AE is trained, a summarized input sequence is generated, which is then used to predict the directional angles in the next step, is generated given an input sequence.

2.3.2. Directional Angle Prediction

The directional angle of each part lens is predicted based on C-BLSTM, which is a unique architecture derived from connecting the convolutional layer, and the BLSTM, which is a representation extracted from the physical attributes of the part lenses. A sequence of feature vectors of the lenses is obtained through the convolutional layer, which is transferred into sequential representations to translate the characteristics of the lens module assembly effectively. The directional angle of a part lens influences deciding the directional angles of other part lenses. Thus, we utilize a bidirectional LSTM (i.e., BLSTM [31]), which processes the input both in the standard order and reversed order during every time step [32]. The detailed structure of the C-BLSTM, including the activation function for each layer, is shown in Figure 12.

The BLSTM architecture is composed of blocks that perform as memory cells, and each memory cell consists of three multiplicative gates, termed the input, output, and forget gates [33]. The gates of each cell learn to open and close, gaining access to the memory blocks and allowing the cell to remember information until the activation of the forget gate. Many-to-many recurrent units are relevant for classifying the directional angles of lenses into 72 classes, each with the highest probability.

3. Discussion on Generalization

3.1. Need for Generalization

Despite the high performance of the proposed framework for producing existing lens module models, it is difficult to apply this model to new products. As with other studies on AI algorithms, model generation that naturally addresses generalization is a concern. Usually, lens module product changes occur every three or four months according to line-up of camera lens modules. Moreover, there are cases where the input space and data distribution are different [34], and obtaining data is difficult [35]. Feature engineering and the transfer learning method have emerged as a solution to this problem, which facilitates an easy domain shift [36].

3.2. Possible Approaches

We attempted to explore feature sets, which positively impacted the retention of performance, even when a lens module product was changed. As mentioned in the previous section, there are a number of variables as input features associated with outputs obtained from the physical attributes of a part lens. Finding the optimal feature set is also important for reducing the high dimensionality and avoiding the over-fitting. Therefore, we exploited k-based feature selection methods which was depicted in Figure 13. We set multiple subsets and used them to build defect predictors in parallel. After that, by comparing the performances of those, we could figure out which subset is the most effective to build a model.

We also employed a domain adversarial neural network (DANN) as one of the transfer learning methods with a set of invariant features in the source and target domains. We placed the current production lens modules (‘Domain 1’ and ‘Domain 2’) into the source domain, and a new product (‘Domain 3’) was set as a target domain. Ganin et al. [37] proposed DANN to address the problem of the difference between the distribution of the source and target domains. This employs a domain discriminator and a feature generator to complement the differences in the feature distributions between the source and target domains. The goal of DANN is to generate domain-transferable data features that are discriminative for the classification task and in-discriminative for the change between the source and target domains.

4. Experiments

4.1. Data Illustration

For the experiments, production logs collected from the assembly process of a real manufacturing site were used to evaluate the superiority of the proposed framework, and its statistics are described in

Table 1. Statistics of the number of production logs. Deduplicated production logs were utilized to evaluate the performances of the proposed defect predictors.

Environment	Category
	Original	Deduplicated
Manual test production	46,615	5532
Automatic mass production	29,451	2510
Total	76,066	8042

. Each production log contains a cavity number combination, a directional angle combination, physical attributes, and the final condition of a lens module. The levels of directional angle were 72 classes of every 5°. There are two production lines in the manufacturing site: a manual test production line operated by experts and an automatic mass production line operated by machines. The number of production logs from each line was 46,615 and 29,451, respectively. The imbalance ratio of the production logs of non-defective lens modules to the production logs of defective lens modules was 8:1.

We deduplicated the logs to avoid any bias during model training. There were logs with the same physical attributes yet different conditions. Therefore, we utilized a majority voting method to decide the final condition of a lens module during the deduplication (defective or non-defective). In addition, the final directional angle was determined through majority voting. Finally, 5532 and 2510 instances remained from the manual test and automatic mass production, respectively. Moreover, only non-defective lens modules’ production logs are utilized for the second process of the framework. In this case, the label features were defined by a directional angle combination. For the generalization, the number of logs from source domain and target domain were 7539 and 503, respectively.

4.2. Experimental Settings

First, we configured the experiment to determine the optimal set of cavity number combinations. For executing the GA, the parameters were meticulously fine-tuned to optimize both the convergence speed and the count of non-defective genes within the best chromosome of each generation. The population size was fixed at 500 individuals, and the generation size was set to 1000. The selection rate and mutation rate were established at 0.3 and 0.7, respectively. The initialization rate for regular combination-based, rule-based, and non-defective combination-based approaches were set at 0.3, 0.3, and 0.2, respectively. Finally, the crossover type rate was determined to be 0.3.

We compared the performances of the proposed DP with the compared methods. To validate the effectiveness of MFE as a loss function, we employed two categories of the compared methods, baseline methods and methods for alleviating the data imbalance problem. The former methods contain decision tree (DT), support vector classifier (SVC), multi-layer perceptron (MLP), and RF. The latter methods were the balanced bagging classifier (BBC), balanced random forest classifier (BRF), RUSBoost classifier (RUSBoost), and easy ensemble classifier (EEC). The compared methods are summarized in

Table 2. Compared methods used to investigate the proposed classification based defect predictor. The methods are divided into two categories, baseline methods and imbalance-processed methods.

Method		Description
Baseline	SVC	Hyperplane or set of hyperplanes in a high-dimensional space are constructed, and these are used to classify data-points
	DT	Type of decision support tool that maps decision rules and results into a tree structure
	RF	An ensemble learning method that outputs class or mean predictions from multiple decision trees
	DNN	Artificial Neural Network consisting of multiple hidden layers between the input and output layers
Imbalance-processed	BBC	An ensemble of meta-estimator that fits base classifiers each on randomly created subsets of the dataset and aggregate the individual predictions
	BRF	The method randomly under-samples each bootstrap sample to balance the dataset
	RUSBoost	Random under-sampling integrating in the learning of an AdaBoost classifier
	EEC	An ensemble of AdaBoost learners trained on different balanced bootstrap samples

.

We compared the performances of our directional angle prediction process that can capture the sequentiality of the lens module assembly process with two categories of existing prediction methods; shallow learning (SL)-based methods and deep learning (DL)-based methods. The SL methods consisted of RF, support vector machine (SVM), DT, MLP, and multivariate random forest (MRF). The DL-based methods were LSTM and C-BLSTM. Additionally, in terms of the autoencoder, single and deep autoencoders [38] were employed to validate the effectiveness of the S2S-AE, which could extract sequential characteristics inherent in input data.

We also set two directional angle prediction methods: step-wise and all-in-one. Originally, we predict the directional angles one-by-one by using the respective features of a part lens. For example, we use only part lens 1’s features for predicting its directional angle. However, in reality, there are cases where experts refer to the previous part lens’s characteristics to decide the next part lens’s directional angle, and experts also make the whole set at a time based on the physical measurements of part lenses. The step-wise method feeds the previous part lens’ predicted directional angle as an input feature of the prediction model for the next part lens. For the all-in-one method, we feed all features of every part lens to the prediction model and obtain the set of directional angles at once.

The F1 score was adopted to evaluate the performances of the optimal set of cavity number combination determining process and the generalization process. This metric is appropriate to measure classification performances more accurately for the imbalance dataset, since it is the harmonic mean of precision and recall [39] as followed Equation (4).

$$F1score=\frac{2\times Precision\times Recall}{Precision+Recall},$$

(4)

where $Precision$ and $Recall$ are represented as Equations (5) and (6), respectively.

$$Precision=\frac{TP}{TP+FP},$$

(5)

$$Recall=\frac{TP}{TP+FN}.$$

(6)

Here, $TP$ and $TN$, respectively, represent the number of logs that are correctly predicted as non-defective and defective, whereas $FP$ and $FN$ correspond to the number of logs that are non-defective yet predicted as defective and defective yet predicted as non-defective, respectively.

As a performance evaluation measure for the directional angle prediction, we utilized the accuracy defined as the ratio of the number of correctly-predicted production logs to the total number of production logs, which was calculated from Equation (7).

$$Accuracy=\frac{1}{T}\sum _{t=1}^T\delta (\widehat{{\mathbf{r}}_t}={\mathbf{r}}_t),$$

(7)

where $\widehat{{\mathbf{r}}_t}$ and ${\mathbf{r}}_t$, respectively, indicate the predicted directional angle combination and the real directional angle combination, and T represents the number of production logs belonging to the test set. $\delta (\widehat{{\mathbf{r}}_t}={\mathbf{r}}_t)$ returns a value of one if $\widehat{{\mathbf{r}}_t}$ is equal to ${\mathbf{r}}_t$ and returns a value of zero otherwise. For example, when the predicted directional angle combination was given as $[{45}^{\circ },{45}^{\circ },{45}^{\circ },{15}^{\circ },{90}^{\circ },{135}^{\circ },{90}^{\circ }]$ and the actual directional angle combination was $[{45}^{\circ },{45}^{\circ },{45}^{\circ },{15}^{\circ },{90}^{\circ },{135}^{\circ },{90}^{\circ }]$, $\delta (·)$ was 1. Furthermore, when the predicted combination was given as $[{15}^{\circ },{135}^{\circ },{45}^{\circ },{45}^{\circ },0^{\circ },{270}^{\circ },{135}^{\circ }]$ and the actual combination was $[{15}^{\circ },{135}^{\circ },0^{\circ },{15}^{\circ },{90}^{\circ },{45}^{\circ },{45}^{\circ },0^{\circ }]$, $\delta (·)$ was 0.

We set a unique validation approach to calculate the different performances fairly. Specifically, for all the performance comparison experiments, the unique production logs, where the cavity number combination only appeared once, extracted from the total number of logs were used as the test data. In contrast, the remaining logs were used to train the predictors. For example, when the cavity number combination of a specific production log was $[3,4,1,10,15,16,4]$, and it only appeared in the dataset once, the production log belonged to the test set. It should be noted that over-fitting could occur as our problem deals with an industrial dataset with significant information duplication due to automatic mass production.

4.3. Experimental Results

4.3.1. Performances of Determining the Optimal Set of Cavity Number Combinations

In this section, the effectiveness of the proposed framework is presented. First, the trend of the average number of non-defective genes in a chromosome according to the generation number was investigated to validate the effectiveness of the GA. Figure 14 displays the trends of fitness value against the reproduction, where the x-axis and y-axis indicate the generation number and the maximum number of non-defective genes per chromosome in a population, respectively. According to the results, the maximum number of non-defective genes tended to increase as the generation number increased. It was evident that the proposed GA design played a significant role in improving the superiority of the population.

Next, the defect prediction performances of the proposed DP and the compared methods are shown in Figure 15. The performances of the five baseline methods, four methods alleviating data imbalance problem, and the proposed method are represented according to the bar color: vacant, light grey-colored, and dark grey-colored, respectively. Between the baseline methods and the methods alleviating the imbalance problem, the latter performed much better than the former with a minimum of a 3.7% increase in the F1 score. Mainly, the F1 score of DNN, which utilizes the normal F1 score as the loss function, was 0.78, while the F1 score of RUSBoost, which exploited random under-sampling of major classes in learning an AdaBoost classifier, was 0.81. This result demonstrated that the approaches for dealing with the data imbalance problem positively affected the defect prediction performances with the imbalanced lens module production logs.

4.3.2. Performances of Determining the Optimal Set of Directional Angle Combinations

Three experiments were performed as follows to evaluate the effectiveness and efficiency of the proposed directional angle prediction method. First, the directional angle prediction performances of the employed methods were evaluated. Figure 16 presents a comparison of the proposed and compared methods according to the sequence regime in terms of accuracy. The best accuracy achieved was 78.19 when using the proposed method, while the worst was 0.95 when a simple regime was utilized and SVM was employed.

In terms of the method of prediction, for the three methods except for SVM, the best accuracy was obtained when a step-wise prediction model was employed. RF performed the best in the step-wise prediction model, where the accuracies of the RF, SVM, DT, and MLP methods were 59.31, 4.25, 22.84, and 12.56, respectively. The performances of the one-to-one prediction model were relatively lower than the step-wise and all-in-one prediction models. This result implied that the directional angle combination was positively correlated with the physical attributes of multiple lenses. The performance of C-BLSTM outperformed that of BLSTM, with the accuracies of 71.20 and 48.74, respectively. This result implied that there could be a positive correlation between latent feature extraction and the performance.

Finally, the prediction performances were compared according to the employed autoencoder method to validate the effectiveness of the proposed method. The comparison results are represented in Figure 17. The performances of the three autoencoders, simple, deep, and the proposed method (S2S-AE), are presented in terms of accuracy, and the C-BLSTM based prediction performance, which does not use any autoencoder, is included for comparison. The best accuracy was 78.19 when the proposed method (S2S-AE) was used, while the worst accuracy was 74.91 when the simple autoencoder was employed. These results implied that the feature extraction considering the sequential characteristic of the lens module assembly had a positive relationship with the performance. Moreover, it was noticeable that the performance of the deep AE-based method was superior to the single AE-based method. This result could be explained by the fact that multiple layers are more effective than a single layer for learning the discriminative features underlying the input physical attributes that affect the directional angle prediction.

4.3.3. Performances of Generalization

Lastly, we investigated the importance of generalization by comparing the performances between ours and the originals. The results on the selected feature subsets by k-based feature selection using three different product data are presented in Figure 18. The F1 scores confirmed that the ‘Set 1’ feature subset achieved the best performance (0.87), which improved by almost 20% for a new product model (Domain 3). Hence, feature selection is extremely important in deep learning because it serves as the most efficient way to improve model performance.

The performance of the DNN model trained with source domains (‘Domain 1’ and ‘Domain 2’) gradually reduced when applying this model to the target domain, ‘Domain 3’. However, when using transfer learning, the performance on ‘Domain 3’ was considerably improved, as displayed in Figure 19. In summary, the F1 score increased to 0.82, and MAPE lowered to 4.11 % for a target, ‘Domain 3’, using a set of selected lens features. This result proved the effect on the transfer learning by reducing overfitting for the source domain.

5. Assembly Test and Result

In this part, we selected new 14 combinations of ‘Domain 1’ lens products recommended by the AI system in order to make assembly tests on the combination optimization. We assembled part lenses into a barrel by using an automatic assembly machine with the predicted cavity number and directional angle combinations for each part lens. The judgment of these tests was the same mass production requirement, which is the yield of 100 samples assembled at the same combinations. Actually, the number of parts and time have been wasted for the ‘matrix test’, which means a ‘pre-test’ with 100 samples as a trial of the engineer’s knowledge or experience to find the correct combinations prior to mass production.

For the 14 combinations, the pass rate was 42.8% as shown in

Table 3. Quality evaluation of assembled samples.

Combination	Yield (%)	Pass (>75%)
1	100	∨
2	25
3	50
4	100	∨
5	75	∨
6	75	∨
7	75	∨
8	100	∨
9	25
10	25
11	25
12	25
13	0
14	0
Rate (%)		42.8

∨: the combination passed the test.

. Passed combinations can go straight to mass production without additional optimization by engineers. In particular, the fact that we were able to obtain three combinations with 100% yield is a signal that supports the need for automation through AI models at the manufacturing site. Even though yields for combination numbers 2, 3, 9, 10, 11, and 12 were not acceptable, these predicted angles can be also used as initial angles on their way to finding the best angles. As mentioned, we are installing, testing, and optimizing an AI inference system on automatic assembly process lines in the factory. It will help engineers performing the ‘matrix test’ to find the optimal cavity combination and the directional angles of part lenses.

6. Conclusions

In this paper, we introduced a novel framework for determining the optimal assembly conditions for lens module production, leveraging the combination of GA and C-BLSTM. The proposed framework consists of two distinct steps: determining the optimal set of cavity number combinations and directional angle combinations. To the best of our knowledge, this work represents the first attempt to address a complex problem for exploring the optimal assembly condition. We conducted experiments using a real-world dataset collected from a manufacturing company to assess the framework’s capabilities. Furthermore, we implemented the system in an actual manufacturing site, affirming and validating its superior performance.

Our future research directions are outlined as follows. It is imperative to integrate the two steps involved in determining the optimal set of cavity number combinations for both part lenses and a barrel. Although assembly experts traditionally identify cavity number combinations for part lenses and a barrel simultaneously, we chose to split this process into two steps to mitigate the extremely high complexity of the problem. In our next phase, we intend to transform the current directional angle classification into a regression problem. Given that the directional angle has a continuous value, we initially framed the prediction as a classification problem to simplify the complexity. Additionally, we plan to investigate effective strategies for handling frequent changes in lens module products by incorporating state-of-the-art transfer learning methods.