A NEAT Based Two Stage Neural Network Approach to Generate a Control Algorithm for a Pultrusion System

Abstract

Controlling complex systems by traditional control systems can sometimes lead to sub-optimal results since mathematical models do often not completely describe physical processes. An alternative approach is the use of a neural network based control algorithm. Neural Networks can approximate any function and as such are able to control even the most complex system. One challenge of this approach is the necessity of a high speed training loop to facilitate enough training rounds in a reasonable time frame to generate a viable control network. This paper overcomes this problem by employing a second neural network to approximate the output of a relatively slow 3D-FE-Pultrusion-Model. This approximation is by orders of magnitude faster than the original model with only minor deviations from the original models behaviour. This new model is then employed in a training loop to successfully train a NEAT based genetic control algorithm.

1. Introduction—Neural Networks in Engineering. Just a Modern Buzzword?

Artificial Neural Networks (ANN) are increasingly used in a wide range of engineering applications [1,2,3,4,5,6,7]. ANN have proven their excellent pattern recognition capability [8], whether optical [9], acoustical [10] or in other multi-feature datasets [11]. Even though they require high computational power and large high-speed memory for effective training, as well as a lot of, often counter-intuitive, hyperparameters adjustments, they can fit any function if they have a reasonable network complexity. The classic use case as a high-level classification algorithm is but one of their many uses.

The still rapidly increasing computing power enables the use of ANN tremendously. ANNs are already present in smart phones and are considered capable of managing highly complex environments such as modern computer games [12] with their complex control schemes in real time. This capability opens up an extremely wide range of applications.

One application is the control of real engineering systems. The authors of [13] show that neural networks can be used to implement near-optimal control algorithms for difficult-to-control systems with constraints.

One approach for such a control model is the use of reinforcement learning with a genetic algorithm such as the NeuroEvolution of Augmenting Topologies (NEAT) algorithm [14]. The NEAT algorithm enables genetic evolution of a neural network to optimize the behavior for a given problem. The authors of [15] describe such an implementation for a nonholonomic wheeled robot.

This paper takes the same approach and describes the successful implementation of a two-step approach to generate a control neural network using the NEAT algorithm for a pultrusion system like the one displayed in Figure 1 and Figure 2.

The pultrusion system is represented by a Finite Element, short FE, model, but it is too slow to complete enough training rounds in a reasonable time. Therefor, an ANN is first trained to approximate the output of the FE model of a pultrusion process. This approximation deviates in accuracy and stability, but is significantly faster than the original model. The time gained by the increased speed is used to train a control algorithm based on NEAT. This evolutionary based algorithm benefits strongly from high-speed simulation due to its high parallelizability and the many training rounds required.

This is an alternative approach to the previously presented method [16], a more traditional control algorithm for a pultrusion based on data received on the resonant ultrasonic spectroscopy [17].

2. Pultrusion Model and Neural Network Training

Training a reinforcement learning based neural network for controlling real engineering systems requires a large amount of training cycles. Using an FE model for this purpose can lead to unreasonably long training times, especially if the task is not yet reasonably defined as is briefly explained later. An alternative to using the FE model directly is to train an ANN so that it behaves like the FE Model. The ANN is much faster than the FE model with comparable accuracy. Both the model and the process used to train the ANN FE model are described in Figure 3. A total of 196,620 samples with a 32 samples per batch are fed to the network during training. The input and output temperature is normalized in the following way

$$T_{ANN}=\frac{T_{FE}}{255}.$$

(1)

The degree of cure does not have to normalizes as it is already in range of 0 and 1. The corresponding FE model is described in more detail in [18].

It consists of two intertwined partial differential equation models shown in the upper half of Figure 3.

The ANN based on an FE model is faster than the original FE-Model by a factor of 1000. This allows incredibly fast training cycles in a very complex environment even without the use of high performance computing (HPC). However, the ANN itself has to be trained on an HPC, especially because hyper-parameter tuning is necessary to maximize accuracy. The ANN later runs in a loop and feeds its own output into the input for the next iteration. This exponentially increases any instability and degradation between FE-model and neural network. Hyperparameter tuning minimizes this divergence. Figure 4 compares the accuracy of the ANN with the original model by calculating the mean square error (MSE) during a series of random speed changes of the pultrusion.

The MSE of the temperature is about 2 °C and the curing error is around 1.5%, showing very good process mapping by the ANN. This is further emphasised by watching the presentation of results in Supplementary Video S1. The training loop is quite simple. A number of slices are continuously moved through the virtual mold. The change in temperature and degree of curing of each slice is calculated by the ANN. The ANN-FE model can handle multiple slices at once without increasing the computation time. The training loop is displayed in Figure 5.

The control ANN is built by applying reinforcement learning techniques. While these techniques can produce very satisfactory results through back-propagation, they require a fixed network structure. While it is feasible to use a fixed network structure together with back-propagation a different approach is chosen. An alternative to back-propagation is the use of genetic algorithms. Genetic algorithms are a good alternative especially when the network structure is not fixed. As shown in [19,20,21] genetical algorithms are not inferior to other methods especially for simple networks. For ease of implementation, the NEAT-algorithm [14] with the PYTHON implementation [22] is chosen. This algorithm is simple and quick to implement and highly scalable.

The chosen task of creating a control algorithm for the given system is similar to the game Flappybird. In this game, a player has to pass through a number of small-width gates while moving forward. If he fails to pass a gate, the  game is over. A number of healing value “gates” is defined. If the healing value $\alpha$ is not within a certain range of the healing target ${\alpha }_{target}$ after a predefined period of time, the task ends and the accumulated points are stored as that person’s fitness. The task itself is executed in several sub-processes, which are connected to the main process via input and output queues. A major advantage of the queue system is its excellent scalability. The number of sub-processes is only limited by the system resources and the speed increase is linear as long as the system can keep up.

One of the most important steps is the implementation of a reward function. This function defines the optimization plane and should have a non-zero gradient at any given point except the global optimum. For this particular task there are three things that should be accomplished:

• The further an individual gets, the higher the fitness function result should be
• If a gate is missed, the fitness function should reflect the distance from the respective gate
• The fitness function should output the highest value when the curing target is reached

Those items can be accomplished by the following fitness function P

$$P=\sum _{t=0}^{\infty }{P}_r\left(t\right)$$

(2)

with

$$P_r\left(t\right)=\left\{\begin{array}{cc}|1+\alpha -{\alpha }_{Target}|\hfill & t\ne t_{Change}\hfill \\ \left(1-|\alpha -{\alpha }_{Target}|\right)·1000\hfill & t=t_{Change}\cap |\alpha -{\alpha }_{Target}|\ge 1\%\hfill \\ 1000\hfill & t=t_{Change}\cap |\alpha -{\alpha }_{Target}|<1\%\hfill \end{array}\right.$$

(3)

This fitness function in combination with the selected scoring function allows the algorithm to determine the fitness of each individual. The next generation of individuals is then generated by mutation and heredity of the fittest individuals.

The NEAT-algorithm itself is very powerful, but requires the adjustment of a number of hyperparameters to converge in a reasonable manner and time. In particular, the mutation power and the mutation rate are of high importance. These parameter determine the convergence ability and the convergence speed. In the present case, a slower mutation rate is necessary for convergence. A valid pair of hyperparameters is contained in the Appendix A.

Figure 6 shows the results of this optimization strategy with the aforementioned parameters over several generations. All nodes use the sigmoid activation function. It is very interesting to note that despite the specification of the last 19 time steps as well as the target value for curing as input, not all inputs are used in the control ANN. Looking closely at the generated ANN, it is apparent that the ANN is emulating some kind of PD controller. The integrator is omitted because there is no memory cell. This in turn is a strong indication of the use of a memory cell or memory variable to optimize the behavior. While this is not directly provided by the NEAT Python algorithm, an implementation is possible by adding an additional output that is fed back at the next iteration.

Figure 7 represents the solution of the evaluation problem for the best individual after 270 Generations. The speed is limited between 0.05 m/min and 0.2 m/min, which is typical for pultrusion. The black horizontal lines in the upper figure represent the target curing value while the orange curve shows the simulated curing by the ANN-Model. The lower figure shows the speed in the pultrusion system. The figure clearly shows that the algorithm is capable of controlling the FE-ANN to stay within 1% of the target cure value by adjusting the pultrusion speed. The 1% distance is the result of the chosen fitness function. There seems to be a constant deviation of around 1%, which could be reduced by further training or a change in the fitness function. However, a 1% deviation is within an acceptable range and the deviation has to be confirmed first on the FE model.

3. Discussion

As Section 2 shows, a two-step approach can lead to a significant gain in computation time, which directly affects the training time of the control algorithm. While there is a difference in the real FE model and the fitted ANN, the difference is relatively insignificant with respect to the difference in the optimized control algorithm. To demonstrate the ability of the control ANN, the system is compared to the FE model. Figure 8 clearly shows that there is only a minor difference in behaviour for the control-ANN applied to the original FE-Model of around +1%. This is a systematic error in the control algorithm.

A fast and easy fix for this systematic error is a change in the target curing value ${\alpha }_{Targe{t}_M}$ which is fed to the model like,

$${\alpha }_{Targe{t}_M}={\alpha }_{Target}-1.6\%$$

(4)

This small change nearly fully negates the systematic error and leads to a very good deviation of below $0.5\%$ for most curing targets as displayed in Figure 9.

The two-step approach of first training an ANN to approximate a FE model and then training a control algorithm on this much faster ANN model yields a significant speed advantage of several orders of magnitude. This approach can be applied to a variety of scientific problems. It works especially well for complex numerical or real world systems without a deterministic scientific model as long as a reasonable amount of training data is available.

The first part, in particular, is responsible for the high speed-up. The second part, i.e., the NEAT algorithm, is easily applicable to any problem where a clear evaluation and fitness function can be created. Especially the fitness function is crucial for a successful implementation. Local minima are particularly problematic, so a well-thought-out implementation is important. The algorithm will find and exploit any loopholes in the formulation. For example, an implementation such as

$$P_r\left(t\right)=\frac{1}{n+\left({\alpha }_{target}-\alpha \right)}$$

(5)

for small values of n might lead to oscillation since it might be more valuable to obtain a few values $\alpha$ extremely close to the ${\alpha }_{target}$, rather than a large number of values slightly further from ${\alpha }_{target}$.

4. Conclusions

This paper shows that a two-step approach, in which the system under study is first approximated by an artificial neural network. In the sense of preprocessing, this step provides a training system that can be quickly examined and used to train an ANN-based control algorithm at a reasonable speed. The speed increase in this case is more than 52×.

The control-model generated is based on the NEAT-algorithm and is implemented with an appropriate evaluation and fitness function. The evaluation function observes and evaluates whether a certain range, e.g., 1% of the target cure value, is reached after a certain period of time. Failing to do so results in an instant loss and stop all further gain in the fitness function.

This algorithm results in a usable control that emulates some kind of PD control based on the observed final structure. Applying the control algorithm on the original FE model results in a viable behavior.

In future research, both the control ANN as well as the FE-ANN, can be further enhanced to not only react to a single set of epoxy curing parameters, but rather to obtain a universal epoxy behaviour with variable parameters. This can be archived by training the FE-ANN on different epoxy parameters and even different geometries. Memory cells need to be added to the control algorithm to explore and store the system behavior of a random epoxy system.

A next step will be the implementation of an auto-encoder algorithm to obtain the degree of curing directly from the sensor signal of the real pultrusion system.