Auto-Encoder-Enabled Anomaly Detection in Acceleration Data: Use Case Study in Container Handling Operations

Abstract

The sudden increase in containerization volumes around the globe has increased the overall number of cargo losses, infrastructure damage, and human errors. Most critical losses occur during handling procedures performed by port cranes while sliding the containers to the inner bays of the ship along the vertical cell guides, damaging the main metal frames and causing the structure to deform and lose its integrity and stability. Strong physical impacts may occur at any given moment, thus in-time information is critical to ensure the clarity of the processes without halting operations. This problem has not been addressed fully in the recent literature, either by researchers of the engineering community or by the logistics companies’ representatives. In this paper, we have analyzed the conventional means used to detect these critical impacts and found that they are outdated, having no real-time assessment capability, only post-factum visual evaluation results. More reliable and in-time information could benefit many actors in the transportation chain, making transportation processes more efficient, safer, and reliable. The proposed solution incorporates the monitoring hardware unit and the analytics mechanism, namely the auto-encoder technology, that uses the acceleration parameter to identify sensor data anomalies and informs the end-user if these critical impacts occurred during handling procedures. The proposed auto-encoder analytical method is compared with the impacts detection methodology (IDM), and the result indicates that the proposed solution is well capable of detecting critical events by analyzing the curves of reshaped signals, detecting the same impacts as the IDM, while improving the speed of the short-term detection periods. We managed to detect–predict between 9 and 18 impacts, depending on the axis of container sway. An experimental study suggests that if programmed correctly, the auto-encoder (AE) can be used to detect deviations in time-series events in different container handling scenarios.

1. Introduction

Shipping containers have become the main driving force for logistics companies worldwide since their introduction, transporting various cargo, especially for ocean-going cargo ships. This is mainly due to the many advantages of efficient cargo storage and fast logistics, including the simplicity of the transportation of casual materials and their increasing volumes. However, the cost of transportation is increasing constantly due to human error, resulting in the loss of containers and cargo and halting handling/transportation processes in ports. The autonomous operation of various modes of transport in recent years has become attractive to the whole logistics industry, and it has gradually become a hotspot of academic research in recent years [1,2]. New container handling machinery, such as the autonomous Quay Cranes (QC), are now being adopted around the world, yet old problems remain the same, with no serious technological improvements in the mechanical parts of the cranes, the control of movements and estimation of real-time dynamics of both the spreader, the container and the cargo inside. Therefore, novel and updated data-driven predictive maintenance and monitoring systems have become a real necessity for logistic and insurance companies, ensuring the reliability of cargo handling operations, visibility of responsibilities in real-time, and security of the transport processes in the port environment, ensuring the future perspectives and strong agility of logistics processes, technologies, systems, and management strategies.

From an operational perspective, cargo and infrastructure damage detection problems are a real engineering headache for specialists working in the relevant fields [3], especially if carried out remotely as accurate prognostics are required. Specialists from LKAB “Smelte” container terminal in Klaipeda city demonstrated the importance of the in-time detection of damages, the frequency of their occurrences, and the entire network of responsibilities, which helped to establish the problem area presented in this work.

The detection of container impacts (examples presented in Figure 1) in the vicinity of the heavy machinery in container terminals is a critical factor, influencing the cost of transportation, the responsibilities, and operational actions. These issues are tackled by many scholars and practitioners in their respective research [4]. The detection of these damages in due time, remotely, has not been researched by scholars in this specific area. It is therefore a priority for port operators and insurance and logistics companies.

From a research and systematic point of view, these impacts can be categorized as statistical anomalies of the sensory data, regarding digital accelerators or any other means of vibration or shock detection. In general terms, anomaly detection is an important key component of all novel data-driven solutions, highly anticipated by managers and insurance companies.

Quay and yard cranes are very complicated heavy types of machinery, and due to human factor errors, environmental impacts and extremely complex mechanical/dynamic processes, the probability of abnormal occurrences during container handling is relatively high by comparison. At the same time, due to the sheer volumes of transportation, the constant stress on the operators and the elements of the control systems, the majority of shipping containers are often impossible to investigate manually by port personnel. Furthermore, it is not possible to monitor the dynamic parameters of the cargo or the entire container in a real-time manner. Physical complexity, harsh working conditions, and other relevant ICT boundaries limit the efficiency of regular solutions that require high reliability during the critical execution of the inspection tasks, so the detection of abnormal dynamic parameters fluctuation and the technologies supporting these methods have been paid much attention in the literature recently [5], but not in the containers handling and security areas.

Currently, the most advanced, container tracking and monitoring system is the “42 Container” project (“https://weare42.io”, accessed on 28 July 2022), which uses conventional means to accumulate data from the surrounding environment without any means of self-awareness and Edge computing. This smart container prototype continuously records its location and inner status, which includes: humidity and temperature parameter fluctuation inside and outside of the container, the closing or opening of the door or the container, structural vibrations and sudden shock position during the route, as well as sound and air pollution parameters. For security reasons and insurance-related affairs, the Container 42 also utilizes several cameras that take time-lapse images or start recording during specific measurable events and with the help of the Tesla 42 sensor, measures the movement of assets inside the container. The doors are sealed with advanced event-based or place-based programmable high-end SBS locks and the entire system is powered either by inner power banks or solar panels soon to be approved and certified by Lloyd’s Register.

Knowledge extraction, on the other hand, is carried out remotely by servers in a Cloud-Fog infrastructure, which imposes stress on the supporting wireless systems and computing units. This puts a significant amount of strain on the wireless channels, increasing the probabilities of packet losses in shielded environments of the container terminal. Yet, Edge computing systems that rely on inner knowledge extraction algorithms and other means of signal manipulation, with the help of many powerful computing units, pose new and prospective challenges to researchers, as well as promising results for decision support specialists. Currently, the level of fidelity of the existing systems does not allow the monitoring of individual containers with exact precision [4], limiting the applicability of any complex knowledge extraction technique in handling processes [6]. Such complexity requires understanding the dynamics of the processes, the dynamics of the machinery [7], and their influence on other systems in use. New computational methods and remote process control solutions have advanced virtualization to new heights [8] by allowing the simulation of the complexity on different levels of fidelity, controlling real processes in real-time by using remote sensors with Edge computing capabilities [9].

However, in previous studies, the auto-detection of potential damages in the heavy transportation industry has been prioritized by several authors, most notably Molodova et al. [10], who presented an automatic method for detecting railway surface defects called “squats”, using axle box acceleration (ABA) measurements on trains, and Wiseman [11], who suggested a safety tool for an incessant inspection of “SkyTran” tracks by employing digital surveillance technology. Regarding the AI paradigm and the novelty of smart IoT, the following research is mostly based on research findings in Neural networks (NN) [12,13]. While most researchers treat damage detection tasks in heavy industry in a supervised manner, which can be regarded as pattern recognition problems in most cases, supervised learning requires massive amounts of labeled training data from all possible handling scenarios to achieve efficiency [14]. However, it is nearly impossible to gain such a significant amount of data from all possible damaged conditions during container transportation in different modes while using different handling machinery. This is because the damage cases are chaotic and quite rare in real-life applications. Although some researchers attempt to fabricate the missing parameters [15], the labeled damage state data, through numerical simulation, there is no guarantee that all damage states will be covered at a high-fidelity simulation level with a high level of accuracy to be adopted in real-life scenarios. Therefore, the performance of supervised learning models for container damage detection is limited when encountering unseen damage cases, transport means, handling operations, and weather conditions. Access to genuine container handling data in our case study and the lack of tracked true cases of anomalous ones favor the adoption of the so-called trained auto-encoders (AE).

In general terms, auto-encoders are unsupervised Machine Learning (ML) models often used for anomaly detection in various industrial applications. These models use a first network called the encoder that encodes the input sensory data into a latent representation, which is then decoded by a second network called the decoder. The differences between the input data and the output show anomalies in data samples in the initial data logs. The structure of these input and output artificial neural networks (ANNs) may vary, depending on the application cases. These may include the Recurrent Neural Networks (RNN) [16], convolutional neural network (CNN) [17], and long/short term memory networks (LSTM) [18], etc. Unsupervised feature learning auto-encoder systems showed promising results in the recent literature, where Nayeeb et al. [19] developed a two-stage auto-encoder-based features enrichment technique to detect COVID-19 from chest X-ray images, and Zhu et al. [20] proposed a novel method called Adaptive Aggregation-Distillation AutoEncoder (AADAE) for unsupervised anomaly detection of sensor-based data in order to solve industrial engineering tasks, while also broadening the whole AE applications domain. Other researchers such as Yong et al. [21] optimized the adaptability of the AE in sensor-based solutions by proposing novel explanation methods based on the mean and epistemic uncertainty of log-likelihood estimates, and Haosen et al. [22] explored a novel data-driven approach for long-term real-time and robust voltage stability assessment based on variational autoencoder (VAE), solving the problem of increased uncertain elements in power systems and the extensive deployment of online monitoring devices. More interestingly, AE-based sensor systems and analytics methods are more frequently being used, even in space exploration agendas. For example, Yan et al. [23] developed a Memory-augmented skip-connected deep autoencoder (Mem-SkipAE) system, ensuring the safety and stability of the space rocket, while safely implementing accurate anomaly detection on key parts such as the liquid rocket engine (LRE), and Hong et al. [24], who, focusing on the problem of the unlabeled ISAR image clustering of space targets, proposed a new unsupervised clustering method based on an adversarial autoencoder (AAE) and density peak-spectral clustering (Dpeak-SC).

Overall, the development of information and communication technologies (ICT) stimulates the evolvement of new data-driven systems and embedded computing techniques to obtain real operational time-constrained status data, informing users only of the required operational updates about the status of the containers and the cargo. With the development of AI-enriched data-driven methods [25], real-time monitoring [26] and prognostics techniques have made great progress. The rapid progress of machine learning (ML) techniques [27] based on embedded technologies found many areas of applicability in transport and logistics for detecting anomalies in technical and highly dynamical processes. Among them, AE technology has achieved remarkable success recently due to a large number of normal data samples in anomaly detection industrial applications [28], while the number of abnormal samples is quite limited. Therefore, the adoption of the unsupervised auto-encoder computing paradigm [29] can efficiently exploit the functionality of automatic feature extraction from normal data samples remotely, identifying the sample size accordingly to identify abnormal conditions for separate events, detecting their true causes.

This paper presents the application of auto-encoders for anomaly detection in real-time sensor data. The innovation of this paper lies in the design and realization of an anomaly detection system embedded in an IoT sensor. An anomaly detection method based on an auto-encoder is developed and tested in a real operational environment. This research intends to show the potential of these techniques in Edge computing systems, solving complex engineering and monitoring tasks that are more conducive to systematic early warnings to repair personnel of the ports, managers, and operators of the cranes, intended to adopt reasonable ways to avoid further critical damage to the containers and the cargo. The main contribution of this work is listed as follows:

To the best of our knowledge, this is the first time auto-encoders have been embedded in IoT sensors for container dynamic condition monitoring for anomaly detection via time-series analytics.

2. Materials and Methods

2.1. Detection Module

The proposed detection framework primarily consists of the prototype of the detection module (see Figure 2) used in the experiments. To acquire the impact events data, a data logging system was developed with a local storage unit and a Bluetooth wireless sensor data transmission module. The system also includes components of other electronics, most notably other sensors, intended to be used in further experimental studies of the research group. The detection system was tested in a laboratory environment, collecting acceleration data and other key parameters with a scaled prototype of the quay crane. The system was mounted on the experimental spreader with a load, with more details of the test-bed already demonstrated in [30].

Edge computing capability was established using the Raspberry Pi 4 electronics unit to perform an inner analysis of the acquired data samples, performing pre-defined signal filtering and prediction tasks. The end-node device system consisted of:

• A data transmission unit using Bluetooth;
• Raspberry Pi 4 (four ARM A72 1.5 GHz cores, 8 Gb of RAM) with a 128 Gb SD UHS-I memory card;
• A SINDT-232 digital accelerometer with high-stability 200 Hz MPU6050 3-Axis acceleration, having 0.05-degree accuracy and an acceleration range of ±16 g;
• An inner 6000 mAh battery.

2.2. The Embedded Data Analytics Method and Its Experimental Setup

The key idea of the proposed Edge computing solution is to reduce the space of considered sensors’ signal functions using an appropriate low-dimensional basis adopting the auto-encoder technology [31]. At this point, model order reduction is state-of-the-art in computational engineering and is a highly active field of research in other fields of engineering, where non-linearity of the processes is observed. Hence, semi-supervised machine learning techniques based on auto-encoders can capture non-linear relationships and observe the processes by controlling operations. Container handling is not a linear process to start with, therefore, to find the exact collisions of the container and the ship’s interior structures (as an example), a semi-supervised machine learning technique based on an autoencoder to detect an anomaly in the accelerator sensor data is proposed and tested in a real-life application. A trained autoencoder was deployed on the Edge computing embedded system running on Raspberry Pi 4, logic programmed through automatic code generation from the MATLAB simulation environment (refer to Figure 3).

Recently, these techniques have proven to be very effective [28,32] because they can be trained to detect anomalies with data representing normal operations with ordinary technological processes. Data from failed operations are not needed and they are computationally practical and fast to be deployed on Edge computing devices and embedded systems. Generally, in normal conditions, when “normal” data from container handling processes are fed into the network (with known operational conditions and physically detected abnormalities), the network can restore the input. The error between the input and output is then small enough to be neglected. On the other hand, when abnormalities occur in the process and data is corrupted without knowing the level and the place of the corruption, with no time stamps and checkpoints, and such data containing anomalies is fed into the network, the network fails to restore the input, and the error becomes substantial and detectable if the network is well trained.

The research framework includes the detection module, the embedded logic used to compute events, the level or virtual fidelity of the process in the generalized GUI interface, as well as the IoT placement and detection scheme. As shown, we have developed a detection system with a graphical representation of the results, which was acquired using the system and transferred from the IoT to the user via Bluetooth in a closed experimental environment in Port. The detection was carried out along the X, Y and Z-axes (refer also to Figure 4) during container handling by the Yard crane of the Limited Liability Stevedoring Company “Klaipėdos Smeltė” container terminal.

Autoencoders are based on neural networks and the network consists of two parts: an encoder and a decoder. The encoder compresses the N-dimensional frame of the sensor data input into a code called the Latent space. It contains most of the information carried in the signal input, but with fewer data, though it is capable of capturing non-linear relationships quite accurately. The basic structure of the AE is shown in Figure 1. It is composed of the input layer, the hidden layers, and the output layer. The input layer is used to input original signal data (features vector), defined as $X_i$. The hidden layer is used to perform feature conversion concerning the input data, expressed as $H_m$. The output layer is used to reconstruct the features that were transformed by the hidden layer, and it is expressed as $\widehat{X_i}$. The process of feature conversion from the input layer to the hidden layer is called the encoder process and is generally presented as (1):

$$H_m=f\left(W_1{X}_i+b_1\right)$$

(1)

Here: $f(\cdot )$is the defined encoder activation function and $W_1$is the encoder weight matrix defined for the time frame $F_1$, while $b_1$is the bias vector for the weight matrix.

The process of feature reconstruction from the hidden layer to the output layer is called the decoder process and is generally presented as (2):

$$\widehat{X_i}=h\left(W_2{H}_m+b_2\right)$$

(2)

Here, $h(\cdot )$ is the decoder activation function. Both, the encoder and decoder use sigmoid in the activation functions that are in symmetry, and $W_2$is the decoder weight matrix $W_1=W_1^T$, which is transposed, while $b_2$ is the bias vector.

The decoder regenerates the input from the lower-dimensional code of the latent space (see Figure 5), with its size defined as $H_m$.

In general terms, the input layer is a matrix that becomes a low-dimensional matrix of the latent space (the so-called code) through a transformation-E (Encoder) and D (Decoder), resulting in the full restoration of the $\widehat{X}$. The essence of the AE method is to find the E (and D) through ANN so that the matrices X and $\widehat{X}$ be as consistent as possible. In ideal situations, when normal sensory data are fed into the auto-encoder, the auto-encoder can regenerate the input, and the error between the input and output is thus quite small, ranging between statistically neglectable errors. However, when sensory data containing crucial and undetected anomalies is fed into the auto-encoder, it fails to regenerate the input ideally, and the error increases with each new iteration.

Therefore, the objective function is to minimize the total reconstruction error for each handling procedure—minimizing the loss estimation function $L(\cdot )$by employing root-mean-square error estimation criteria—expressed as (3):

$$Minimize L_T\left(X_i,\widehat{X_i}\right)=\frac{1}{N}{\displaystyle \sum }_{i=1}^N\sqrt{{\left(X_i-\widehat{X_i}\right)}^2}$$

(3)

here:

-

the number of data frames for the measurement period T is set as N, being $N\in Z$;

-

with the latent space code being $E\left(X\right)$ , and the restored data frame being $D\left(E\left(X\right)\right)$;

-

and i being the sensor data sample input to the AE.

The structure of the ANN in our case study included the exact number of hidden neurons (representing the exact dimension of the latent space “code”):

-

A total of 10 hidden neurons, representing the input (N-dimensional frame of sensor data) with 10 weights.

-

A total of 20 hidden neurons, representing the input (N-dimensional frame of sensor data) with 20 weights.

-

A total of 30 hidden nodes, representing the input (N-dimensional frame of sensor data) with 30 weights.

The number of hidden neurons ranged between 10 and 30, with a step of 10. In our test setup, we compared two different approaches, namely the auto-encoder (AE) and the IDM. Additionally, the following pre-processing parameters of the IDM, based on [5], were used:

-

High-pass filter frequency—3.8 Hz,

-

Threshold—73%,

-

Filter queue—200.

These parameters were chosen for the IDM, as they proved to be the most accurate in detecting the impact events. Next, the device was placed on the side wall of the tested container and handling operations were performed according to the work standards and safety regulations (see Figure 6).

3. Results

3.1. Initial Results

Initial test samples, acquired during previous experiments, paved the way for the AE improvement in a laboratory environment. Testing of the AE was performed as a computational experiment using the previous data sample, also used as a training background for the embedded use case AE. As can be seen, the anomalies in the data samples, acquired via IoT remotely, are not visible. No clear results can be observed, with clearly distinguished sway of the container, obstructed with systematic noise (see Figure 7), which may be a result of multiple physical impacts, environmental impacts, the noise of electronics, etc. During a closer inspection, we observed that the regular pattern was broken at around 65th and 600th ms.

Initial results show that a chosen fit function can generate similar patterns of accelerated movement. In our example, Sub of sine with several terms equal to 4 showed the exact sway pattern, with clearly visible peaks in the amplitude and noisiness that is clearly out of shape. A similar fit function generated by the AE in the latent space in the validation phase squeezes the signal frame into a latent representation function effectively, clearly demonstrated in Figure 8 (middle section: time frames from 400th until 800th ms).

The validation of the AE using 10 hidden neurons in the ANN structure proved to effectively regenerate the signal, showing the potential time stamps of the anomalies. Error estimation is in the ±0.1 range on average, yet starting from the 400th millisecond, reconstruction of the signal tends to change drastically. Further results of the embedment of the AE with the varying structure of the decoder and encoder are demonstrated in the following sub-section.

3.2. Experimental Results Using the Proposed Method

The auto-encoder is trained on data without anomalies from handling operations without any impacts, where sway is natural with no visible obstacles. As a result, we learned that network weights minimize the reconstruction error for load signals without faults. The statistics of the reconstruction error for the training data can be used to select the right thresholds in the anomaly detection block that determines the detection performance of the pre-selected auto-encoder. The detection block declares the presence of an anomaly when it encounters a reconstruction error above the threshold.

The developed anomaly detection method proved to work in the actual container handling conditions, detecting events effecting the dynamic stability of the container.

The proposed detection method was trained using data from a single handling procedure (see Figure 9), representing the container movement procedure, identical to the actual experiment. The best detection accuracy was achieved using 20 hidden neuron structures of the AE, resulting in the minimal total average RMSE of 0.12, while the 10 hidden neuron structures resulted in 0.33, and the 30 hidden neuron structures resulted in 0.59, during the three different experimental handling procedures. It is worth mentioning that each procedure was performed accurately, with an almost identical period for each control and maneuvering operation.

The following figures demonstrate the actual impacts detected by the embedded IDM, marked by crosses, and the AE method with the Encoder-Decoder structure with minimal loss, marked by circles. As stated, the most accurate impact detections should be carried out using the X-axis as the basis, due to the limited directions of container sway when the handling procedures are performed by the spreaders of the yard or quay cranes, especially in the inner sections of the ship while handling is carried out along the vertical cell guides. Figure 9 demonstrates the prediction of impacts along the X and Y axes during the initial extraction procedures of the container from the hull of the ship, detected by the system in due time.

During the experiment, 9 impacts were predicted along the X-axis and 12 impacts were detected along the Y-axis using the AE, while the IDM method detected 49 potential impacts along the X-axis and 40 impacts along the Y-axis. It is visible that the embedded AE detected the same anomalies as the IDM, yet eliminated the less plausible ones, according to the RMSE of each data frame, set as 10 milliseconds. However, future research must include other structures of the AE to justify the adaptability in real-life operations.

4. Discussion

In this paper, we have analyzed the real-life applicability of a trained auto-encoder to detect acceleration data anomalies while examining separate frames of data in real-time. The proposed system was tested as an Edge computing unit in real handling operations, receiving experimental data about potential impacts on the vertical cell guides of the ship while transporting empty containers of the same mass during each experiment. The AE was compared to the IDM, which used pre-defined parameters calculated in the previous study, and the following results were gained: the AE method detected fewer impacts than IDM, yet IDM detected all the plausible ones and the ones also detected by the AE. The real-life applicability of this system is efficient, but more experimental studies are required with varying numbers of hidden layers, and several hidden neurons in the ANN structures of the Encoder-Decoder. Future research must also consider changing the IDM parameters to detect less critical impacts and keep the ones that matter most.

The detection of dangerous impacts to the containers is a serious computational task for the Edge computing IoT devices, because:

• The measured signals are contaminated by noise components from several other naturally occurring and unnatural processes extraneous to the natural motion of the container [33], including the quay crane and spreader dynamics, as well as environmental and electronic noise.
• This vast disparity in time scales, as well as the issues with signal contamination, pose serious signal processing and de-noising challenges for conventional methods [5], operating in harsh working conditions.

Therefore, simplified, yet efficient solutions that include optimal algorithms should be developed in further studies to efficiently monitor the container handling tasks and predict dangers, while minimizing the computational strains on the Edge computing devices.