2.1. Edge Computing
To efficiently collect machinery data for further processing, one would either need AMSs equipped with custom APIs and adequate storage to extract raw data directly or deploy custom Data AcQuisition (DAQ) systems on the machinery. However, the former is a relatively new technology that has not been widely adopted, and the latter can be prohibitively expensive due to the costs associated with integrated sensor systems. Our framework is flexible enough to accommodate any scenario, from high-end DAQ and control systems to low-cost sensor-microprocessor systems designed to simply transmit raw measurements. The framework’s only requirement is that data must be transmitted using the MQTT protocol.
MQTT was initially developed as a cost-effective solution for monitoring devices connected via satellite links, where communication costs were prohibitively high. Today, MQTT has become a widely adopted standard in the industrial IoT, valued for its lightweight publish-subscribe model and minimal energy consumption. Although its use in shipboard sensors has been limited so far, this is primarily due to the relatively recent adoption of the OASIS standard version, which was published only a decade ago. Running over TCP/IP, MQTT is particularly well suited for environments with unstable connections or limited network bandwidth, making it ideal for use on vessels. It operates by enabling devices (clients) to publish information to a central server (broker), which then distributes this information to other clients subscribed to specific topics. This method minimizes network bandwidth usage and reduces power consumption by not requiring continuous polling of data, making the protocol ideal for maritime operations, where the number of shipboard devices may be very large. In the proposed framework, the MQTT broker is the starting point of the edge pipeline, as illustrated in
Figure 2, hosted on a standalone server PC or within the vessel’s existing server infrastructure. A detailed breakdown of this pipeline’s constituents follows.
The MQTT broker allows the shipboard machinery sensors to publish their data to preconfigured topics (henceforth, the term “sensor” will refer to the entire DAQ device, not just the signal detector). This setup offers flexibility in how topics are configured, allowing sensors to be logically grouped based on the needs of data consumption. For instance, sensors might be grouped by type, with all thermal sensors sharing one topic and electrical sensors another. Alternatively, sensors could be organized by their physical location, such as grouping all sensors located in the engine room under a single topic. However, a critical grouping criterion in the proposed framework is the frequency of the sensors’ measurements, which significantly influences the edge processing strategy. High-frequency data may require different preliminary processing at the edge to manage volume efficiently, while lower-frequency data can be processed more straightforwardly. Importantly, these grouping criteria are not mutually exclusive and can be combined to optimize data consumption by clients and subsequent processing. The proposed MQTT broker is Eclipse Mosquitto [
34], due to it being lightweight and, most importantly, open-source; however, there is a plethora of other options [
35].
The next major component of the edge pipeline is a distributed streaming platform deployed to manage and store the data streams provided by MQTT. Apache Kafka [
36], widely recognized as the open-source industry standard for such workloads, is capable of handling a large volume of events with high throughput and low latency. By now, Kafka is so well established in the industry that its deployment is practically as straightforward as deploying a simple Docker container and writing some configuration files. Kafka also utilizes a publish-subscribe topic-based model; therefore, it might initially seem redundant in the same pipeline alongside MQTT. Nonetheless, it is crucial to understand that, despite their operational similarities, MQTT and Kafka serve distinct purposes. MQTT is primarily responsible for collecting data from various sources across potentially unstable network conditions. In contrast, Kafka excels at data storage and decoupling data collection from processing, enabling multiple consumers to access and process the same data streams independently.
Obviously, to facilitate the transfer of data from MQTT to Kafka, a bridge that acts as a consumer for the former and as a publisher for the latter is essential. This bridge can be custom-implemented, for example, using the Eclipse Paho [
37] library, to meet specific operational needs such as data validation, timestamp addition, or the enhancement of data with additional features during ingestion. Alternatively, solutions like the open-source Telegraf [
38] can be utilized to provide a pre-configured bridge, simplifying the integration process and reducing the need for extensive code development. Similar bridges are also required to consume data from Kafka, which are then either written directly to an IoT-optimized database or processed further before being uploaded to the cloud.
As for the first approach involving direct data writing, an IoT-optimized database such as InfluxDB or PostgreSQL with the TimescaleDB extension [
39] can be utilized. Unlike Kafka, which is utilized primarily as a transient data layer with a short retention policy to minimize storage overhead, these databases are configured with a slightly longer retention policy. This extension is not intended for analysis but to ensure that recent data are readily accessible for providing a critical backup in case of severe data flow disruptions. Additionally, by connecting the IoT database to dashboard applications like Grafana—often paired with InfluxDB for sensor data monitoring [
40]—engineers on board can perform visual checks to confirm that sensors are online and functioning correctly.
When it comes to the pathway that involves the upload of the data to the cloud, there exists an option to perform pre-processing at the edge before transmission. This step is particularly common when dealing with high-frequency sensors, where continuously uploading large volumes of data is not feasible. In such instances, necessary pre-processing, such as extracting fine-grained features essential for cloud-based analysis, is executed at the edge. Basic pre-processing tasks like aggregations or simple transformations can be handled efficiently using Kafka Streams. For more complex pre-processing requirements, custom services leveraging technologies like Pandas [
41] for data manipulation and Dask [
42] for parallelization and lazy evaluations may be employed. In scenarios with substantial computational resources available at the edge, Apache Spark [
43] can be utilized as the most efficient and scalable solution. Similar to Kafka, simply allocating more resources allows the parallelization of tasks or the execution of more intensive workloads. Regardless of the choice for pre-processing, the data are eventually uploaded to the cloud. Here, there are two main options: batch versus streaming uploads. Given the nature of maritime operations, where equipment failures typically develop over weeks or months rather than instantaneously, regular batch uploads via HTTP at pre-determined intervals or during hours of low average network usage should generally be sufficient and preferred. This also minimizes prolonged network use, which is beneficial given the often unreliable connectivity at sea. In instances where an internet connection is temporarily unavailable, such as when a vessel is traveling through remote regions, the upload process can be configured with periodical retry mechanisms. This setup allows data to be transmitted as soon as connectivity is restored, without disrupting data pipelines or risking data loss.
The proposed edge architecture is designed to be highly scalable and adaptable, capable of operating on hardware ranging from low-cost PCs to more complex server clusters, as long as such infrastructure is available on the vessel. In environments where server clusters are used, Kafka’s distributed nature can be fully exploited. Via horizontal scaling, it is straightforward to deploy more Kafka instances and partition the data that each instance processes. Additionally, more Kafka instances can be deployed as redundancies to ensure higher data availability. With more computational resources available, tools like Spark can also be utilized for seamless pre-processing. However, all of these are simply conveniences rather than requirements. Notably, the pipeline is also sensor-agnostic in the sense that it requires only that data be transmitted via MQTT, accommodating everything from high-end DAQs to basic low-cost sensors. This flexibility even allows for simulated control scenarios without requiring sensors to be specifically configured for them. For instance, in scenarios requiring high-frequency data collection at intervals rather than continuous monitoring, the system can dynamically subscribe to relevant topics for the duration needed to gather the necessary data, thereby optimizing resource use without depending on the sensors’ hardware. With such capabilities at the edge, one might question the need for cloud involvement at all. However, the cloud plays a critical role, not only in ensuring long-term data storage but also in deploying DL models that are trained on the data to provide essential insights. Due to their computational requirements, these models are less suited to run on the edge in marine environments, especially given their constraints on power usage and the impracticality of preemptively scaling physical infrastructure. Thus, transitioning to cloud-based processing not only addresses these limitations but also enhances the overall efficacy and scalability of the proposed framework.
2.2. Cloud Pipeline
Following ingestion, the data are stored within a lakehouse architecture [
26], which combines the expansive storage of a data lake with the structured query capabilities of a data warehouse. Within this lakehouse, processing and transformations are performed to extract fine-grained features to be utilized for DL. These features are used in subsequent DL experiments, including the training and evaluation of predictive models used for inference. This initiates the MLOps cycle, which involves continuous model retraining and evaluation to maintain model efficiency and relevance. A platform that facilitates all these processes as a unified ecosystem in a computationally optimized and scalable manner is Databricks [
44], which is integrated across most major cloud providers.
At this point, it is important to address an apparent contradiction regarding the design of a low-cost solution alongside the use of what are often perceived as expensive cloud services. In reality, the ongoing costs associated with the staffing, maintenance, and scaling of on-premises systems often surpass the costs of cloud services, particularly when also considering the requirements related to physical space and power consumption. Additionally, platforms like Databricks provide dynamic scaling and efficient resource allocation, reducing the need for extensive in-house expertise. Moreover, cloud providers offer guarantees regarding data security, which is critical in the competitive maritime industry where stakeholders are highly protective of their data. As discussed in the following sections, very few maritime-related datasets are open and this is one of the main reasons. It thus becomes evident that cloud platforms align with the framework’s commitment to cost-efficiency, particularly over the long term. With these considerations in mind, in the remainder of this subsection, we describe how the data pipeline works using Databricks as the core platform.
Databricks implements the lakehouse architecture through its integration with Delta Lake [
45], which is an open-source storage layer built on top of existing cloud storage services (e.g., ADLS2 for Microsoft Azure, S3 for AWS, and Storage for GCP). Among the main features that Delta Lake brings to traditional data lakes are ACID transactions, data time travel, scalable metadata handling, and schema enforcement and evolution. Moreover, Databricks incorporates Apache Spark in its runtime environments. This integration enables the optimized and parallel execution of code, as well as the distribution of data across multiple nodes, which is essential for handling a large volume of data from multiple sources.
The pipeline depicted in
Figure 3 starts with the ingestion of the sensors’ telemetry, either pre-processed at the edge or not. If the data arrive online in streams, a message bus service (e.g., Event Hubs for Microsoft Azure, Kinesis for AWS, and Pub/Sub for GCP) is necessary to serve as a buffer, bridging the data transmission at the vessel with the data ingestion from the data pipeline. On the other hand, in batch-upload scenarios, a storage service acts as the staging area where the data are received as files, preferably in read-optimized formats like Apache Parquet [
46]. To incrementally ingest the new data files as they arrive in the storage service, Databricks’ built-in Auto Loader is utilized. The Auto Loader maintains an internal state to track all processed files, thus ensuring that no file is missed while also offering features such as schema inference and handling of schema drift.
Subsequently, the data are written into tables in the lakehouse using Spark’s Structured Streaming [
47]. Structured Streaming is an engine built on top of Spark which enables the ingestion of data from a source to a sink via built-in connectors, using a high-level API. In the case of data from incoming streams, the source is the message bus service used, while in the case of uploaded files, the source corresponds to the storage service monitored by the Auto Loader. Finally, in both cases, the sink connector points to the Delta Lake.
The architecture chosen for the Delta Lake in our framework is known as a multi-hop or medallion architecture and is illustrated in
Figure 4. The idea behind this design is to organize the data into distinct layers within the lakehouse, with the structure and quality of the data being progressively enhanced as they move through the layers. The number of layers is flexible and is based on specific operational needs or data management strategies. For the purposes of the presented framework, a three-layer configuration is recommended. The initial layer, known as the bronze layer, contains tables that hold the raw data ingested via the aforementioned procedure.
Once the data are successfully ingested into the bronze layer, they undergo a refinement process to prepare them for DL applications. This process involves extracting feature vectors, which are then stored in the middle layer of the lakehouse, known as the silver layer. As discussed in the following subsection, the DL model processes multivariate time series, which are measured from various sensors installed on a single machine. Therefore, the first step of this refinement process involves mapping sensor-level data to system-level data, i.e., combining univariate time series to create a multivariate one. If the data have not been pre-processed at the edge, this typically includes aggregations (e.g., averaging or extracting median values) to ensure that the length of the univariate time series is consistent across all sensors installed on a machine without resorting to methods like padding. Further data processing might involve transformations such as min-max normalization or standardization using pre-trained scalers. This implies that historical data are available on which these scalers have been trained. All of these operations are performed using Spark, mainly to ensure short execution times and to handle data regardless of volume. It is worth noting that cleaning and filling missing values is not necessary, as it has already been handled at the edge. For example, in case a sensor malfunctions and subsequently stops operating, it will stop transmitting signals, appearing as if it has been disconnected from the DAQ’s corresponding channel. The DAQ can then transmit a pre-specified value to indicate that the sensor was disconnected, and these values can be filtered out or replaced by the last correctly measured value once they reach the onboard server.
The fine-grained data from the silver layer’s tables are input into DL models with the aim of generating insights regarding the condition of monitored devices through fault detection. Of course, the framework is versatile, allowing for the integration of “traditional” statistical or threshold-based methods, either independently or in conjunction with DL models. The results from these analyses are stored in the final layer of the lakehouse, known as the gold layer, which is primarily designated for reporting purposes. Consequently, the data in this layer are consumption-ready and tailored for end-use applications. To consume these data, a dashboard application can be implemented, leveraging the numerous connectors supported by the Delta Sharing protocol—an open protocol that allows sharing data that live in a Delta Lake without having to copy them to another system [
48]. This dashboard can be configured with a role-based access control model to maximize its business utility. For instance, stakeholders might have access to high-level metrics, such as the number of faults detected in each vessel of their fleet. On the other hand, engineers on board may be able to investigate not only which data points of a time series are identified as faults by the models but also why, as long as the deployed models are explainable.
Before proceeding to the proposed DL architecture for fault detection, it is important to highlight that Databricks not only manages the execution of the aforementioned operations but also orchestrates the entire workflow through Databricks Workflows. The arrival of new data, whether in file or stream form, automatically triggers the workflow, with various notebooks designated to handle different aspects of the data pipeline. Nonetheless, for use cases requiring broader integration capabilities, such as triggering external functions via APIs or interfacing with other cloud services, one may consider switching to a dedicated pipeline management service capable of running Databricks notebooks as jobs (Data Factory for Microsoft Azure, Glue for AWS, and Dataflow for GCP).
2.3. Deep Learning Architecture
As previously mentioned, the framework can operate using “traditional” analysis methods. However, marine environments present unique challenges that can limit the effectiveness of these techniques. If one takes vibration analysis via Fourier transformations as an example, which has been extensively employed for machine condition monitoring [
49], numerous problems immediately become apparent: variable background noise is always present because no machine on a vessel operates in isolation; machinery often runs under alternating load conditions, which can lead to non-stationary signals; and marine equipment is frequently subject to transient events, such as sudden changes in engine speed or impacts from waves, which can introduce short-lived changes in vibration signals, to name a few. To effectively address these issues, extensive experimentation is necessary—sometimes individually for each piece of equipment. This might involve setting baselines during dry dock measurements or capturing signals under varying operational conditions and then performing extrapolations. Given these complexities, maritime operations are ideal candidates for the application of DL methods, as they do not require such extensive preliminary steps, relying instead on a data-driven approach that adapts to the inherent variability of the environment. However, when available, such information is very valuable for data pre-processing, as it may enhance the model’s performance.
Naturally, the choice of a suitable DL architecture is not trivial since DL models are not one-size-fits-all solutions by default. Prior to the development of the DL architecture, specific criteria were established. Firstly, the chosen model must operate in an unsupervised manner, a necessity due to the scarcity of labeled data in the maritime domain. Secondly, any trained model instance should provide a level of explainability, i.e., it must allow the user to understand and attribute the features that most significantly contribute to the identification of faults. Finally, the architecture must be as adaptable as possible: it should be domain-agnostic and capable of handling any type of multivariate time-series data without being specialized to only certain types or patterns of anomalies. Among the various candidates satisfying these conditions, we opted for an architecture that belongs to the family of MTAD-GAT models [
50,
51], which extract spatio-temporal information from multivariate time series by leveraging Graph Attention Networks (GATs) [
52]. This choice is particularly justified in our context because shipboard machinery and their constituents can be represented as interacting sub-systems, and the relationships among these sub-systems are naturally modeled by graphs. Thus, we may flexibly define any system as a sub-graph of the overall machinery network. The architecture is depicted in
Figure 5.
To train a model instance based on this architecture, a multivariate time series with n variables that has already been properly pre-processed is drawn from the Delta Lake’s silver layer. Before being fed to the model, it is split into potentially overlapping segments of equal length, w, as the proposed model operates in a sliding-window setting. Each of these -dimensional segments corresponds to a single data point as far as the model is concerned. The end goal is to accurately predict whether the -dimensional vector at the last timestamp of a window corresponds to a normal or fault state for the machine from which the multivariate time series was extracted. However, the DL model is not trained as a binary classifier.
Instead, a joint optimization strategy is followed, attempting to jointly minimize a reconstruction-based loss function and a forecasting-based loss function, and , respectively. The idea behind this is to train the model to accurately predict upcoming measurements from the sensors installed on the machine while also capturing their data distribution. Then, newly arriving measurements can be evaluated on two axes: how close their values are to the trained model’s prediction and how likely it is that they have been generated by the distribution the model has learned. Based on the significance of the deviation for the reconstruction and forecasting process, a score is assigned to each timestamp. Finally, the state corresponding to each timestamp is identified as a fault or not, depending on whether the score is over or under a specified threshold, respectively.
At the start of the architecture lies a one-dimensional convolutional layer with unitary stride and dilation, whose purpose is the extraction of high-level information as a type of local feature engineering. This is mainly achieved due to the temporal invariance of one-dimensional convolutional layers, as they are able to detect features regardless of their shift in time, similar to how two-dimensional convolutional layers detect features in images. The size of the convolution kernel, denoted by
, is one of the architecture’s hyperparameters. This layer’s output is
-dimensional, where
. Three copies of this output are made: the first copy is not further processed, while the other two pass through two parallel GATv2 layers, one spatial and one temporal. GATv2 layers are different from the original GAT layers introduced in [
52] in that they are more expressive because their attention mechanism is not static but rather dynamic [
53].
For the spatial layer, each variable of the time series is represented by a -dimensional node in a graph, with the graph’s edges capturing the relationship between variables. This allows the GATv2 layer to detect multivariate correlations without any prior knowledge through the dynamic attention mechanism. Similarly, for the temporal GATv2 layer, each timestamp within a sliding window is represented by an n-dimensional node in a graph, with the graph’s edges now capturing temporal relationships. For both layers, the embedding dimension is equal to the input dimension, meaning that the total number of nodes, as well as their dimension, remains fixed. Therefore, the output of the spatial GATv2 layer is -dimensional, and the output of the temporal GATv2 layer is -dimensional. By transposing the latter and then concatenating both of them along with the unprocessed -dimensional copy of the convolutional layer’s output, a -dimensional matrix is produced. This concludes the first half of the architecture, which essentially performs feature engineering. Each column of the final matrix corresponds to a -dimensional feature vector at a given timestamp, which fuses the original input’s information with the spatio-temporal dynamic attention scores.
The second half of the architecture begins with a single LSTM layer with a hidden dimension d, which receives the feature matrix as its input. The purpose of this layer is to capture sequential patterns and long-term dependencies in the temporal dimension, a task at which recurrent neural networks and especially LSTMs excel. Using this layer’s output, the architecture then splits into two directions. The first corresponds to the reconstruction process, where the entire architecture so far is treated as an encoder, and a decoder is added at the end. This decoder consists of sequential LSTM layers with hidden dimension and a linear layer to map the LSTMs’ output to an -dimensional vector. The second direction involves forecasting, where sequential LSTM layers with a hidden dimension , followed by a linear layer, try to predict the n variables of the last timestamp of each window, without prior knowledge of them. To achieve this, each data point has to pass through the architecture twice, in parallel: once as it is for the reconstruction direction and once with the data corresponding to the last timestamp removed for the forecasting direction.
To train instances of this model, both
and
are calculated as the Root Mean Square Error (RMSE) between the model’s predictions and the actual measured values, although other loss functions have also been proposed for similar architectures [
54]. This is a form of autoregression; therefore, the model is trained in an unsupervised manner, as required. The total loss function
is minimized using Stochastic Gradient Descent (SGD), as implemented by the Adam optimizer. To avoid overfitting, dropout layers are inserted wherever applicable (i.e., between consecutive layers) with a probability of 20%. An additional regularization technique utilized is the introduction of a patience mechanism with limit
p: the data points used for training are split into two sets, and only one of them is used to train the model by updating its parameters via backpropagation. The other set, usually called the validation set, is only utilized to evaluate the model’s performance during each training epoch. If the validation loss does not increase for
p consecutive training epochs, then training is terminated and the model instance with the lowest achieved validation loss is saved.
With a model instance trained, a scoring function is necessary to assign scores to each timestamp’s values based on the ground truth and the model’s predictions. In addition, a mechanism to calculate a threshold is required. Suppose that the measured values at a given timestamp,
t, are the elements of
, and the corresponding predictions by the model are the elements of
and
through reconstruction and forecasting, respectively, then the label
for this timestamp is defined as
where
is the score at timestamp
t obtained from a scoring function
,
is the threshold, and a value of 0 or 1 corresponds to a normal or fault state, respectively.
As far as the scoring function is concerned, to ensure explainability for the final predictions, one may define a variable-wise score,
, as
where
is a weighting factor that controls the influence of the
i-th variable on the overall score and
is a regularization parameter that modulates the impact of the forecasting error relative to the reconstruction error. If
or
, the variable-wise score is influenced more by the forecasting error or by the reconstruction error, respectively. These parameters are useful in cases where domain expertise can be utilized to drive the fault detection process. For instance, if one knows beforehand that vibration measurements are more important than the environment’s temperature in the study of a motor, then the
-value of the former can be set higher than the
-value of the latter. Nevertheless, in cases where the data are scarce or such domain knowledge is not accessible, it is advised to start with
for all variables. Obviously, the overall score is obtained by summing over all variable-wise scores as
When it comes to the thresholding mechanism, ideally, thresholds would be derived utilizing domain knowledge. In its absence, the framework requires an automated method to derive thresholds that scales computationally well with the introduction of many machines, does not rely on labeled data, and is easy to re-execute whenever more training data become available. One such mechanism, proposed by [
55], uses LSTMs for anomaly detection and is adopted in our framework with some modifications. Suppose there are
T data points available for model training; their corresponding scores form the set
. Depending on the scores’ values relative to the threshold, these scores can be split into two sets:
By denoting the mean value and standard deviation of a finite set
by
and
, respectively, the threshold in this mechanism is given by
where
is selected as a value for which the corresponding threshold maximizes
In other words, the threshold is chosen so that if all scores above it are removed from
, the greatest percent decrease in
and
occurs, weighted by the inverse number of total faults to prevent classifying many timestamps in the training data as faults. It is noted that (
6) is expected to be maximized by multiple values of
z. Searching for one such value of
z via brute force might be computationally expensive in real-world scenarios. Therefore, as a rule of thumb, it is suggested to explore values for
z in the
range, with a step of
.
2.4. Evaluation and MLOps
Before a trained model instance can be deployed in production to perform inference on new incoming data using the scoring function of Equation (
3) and a generated threshold, its performance must be properly evaluated, as long as labeled data are available for this purpose. Even though the F1-Score is the simplest metric for such evaluations, it treats each point of the time series independently, as if it were uncorrelated to the rest of the time series. Furthermore, faults often manifest not as isolated points but rather as consecutive sequences known as events. Point-wise metrics do not capture the continuity of anomalous events, thus potentially misrepresenting their severity and often leading to more pessimistic scores for the evaluated models [
56].
To address these, a point-adjustment (PA) strategy, proposed by [
57], has been adopted by numerous authors. The idea behind PA is simple: in real-world scenarios, if any point within an event is correctly classified, a human operator will be alerted and can thus examine the monitored device to identify the fault. Based on this, the model does not necessarily need to classify all points within an event as faults; if at least one point within the event is identified, then all points within the segment can be marked as correctly predicted before the calculation of the F1-Score. However, this metric, known as the PA F1-Score, has shortcomings of its own: it tends to overestimate model performance, and it has been demonstrated that a random classifier can outperform state-of-the-art architectures when evaluated using this metric [
58].
In the presented framework, an alternative evaluation strategy is utilized that incorporates multiple metrics, as recommended in [
59]. We propose using both the regular F1-Score and the PA F1-Score for evaluation, which should ideally converge to similar values. Additionally, a third F1-Score termed the brute-force (BF) F1-Score, is calculated using a threshold obtained by a brute-force search across all possible threshold values to maximize the F1-Score. The argument in favor of this is that the previously mentioned scores evaluate not only the DL architecture but also the thresholding method given by Equation (
5) [
60]. The BF F1-Score evaluates the ability of the model itself to properly score the timestamps in a way that can successfully separate faults from normal values. Besides these metrics, the delay in detecting each event is also considered, with penalties applied for any events that remain completely undetected. Finally, training and inference times are reported, as the capability to retrain models within a limited timeframe and to process incoming measurements promptly is vital for industrial applications.
Specifically regarding the PA F1-Score, a stricter version called PA%K-L is proposed, inspired by the PA%K protocol [
58]. Under the PA%K protocol, point adjustment is applied to all
N points of an event only if the model correctly identifies more than
of them as faults, where
. In our protocol, we introduce an additional parameter,
L, which controls the number of points that undergo adjustment within an event. In particular,
L represents the number of nearest-neighbor points around each correctly identified fault that are adjusted. The example shown in
Figure 6 demonstrates how this protocol works compared to PA and PA%K. In all three cases, the event consists of 10 timestamps, and the model manages to predict only 4 of them. The original PA is obtained by setting
and
, in which case the entire event is considered correctly predicted by the model. For values of
, the PA%K protocol dictates that the event is undetected, and therefore no point adjustment is performed. However, for
and
, the PA%K protocol coincides with regular PA. Nonetheless, when setting
, only one point around each correctly identified fault is adjusted, meaning that the adjusted predictions lead to an F1-Score that is higher than the non-adjusted one but lower than its PA version.
By this point, the specifics of model training and evaluation have been thoroughly discussed. However, it is important that in a real-world application, they adhere to standard MLOps practices. Model instances must be registered and tracked through all stages of their lifecycle. During the training and hyperparameter tuning phases, models should remain in a “development” stage where they can be iteratively adjusted and optimized. Once training is complete, models transition to a “staging” stage for evaluation and testing. If they meet predefined performance criteria with respect to the aforementioned evaluation metrics, they then advance to the “production” stage, where they are deployed to perform inference on new incoming data. Orchestrating this process in a scalable and automated manner is crucial, especially when monitoring numerous machines.
The open-source platform that is selected to orchestrate the DL workflows in our framework is MLflow [
61,
62], developed and managed by Databricks. During model training, an MLflow experiment is defined to track all relevant metrics and artifacts using MLflow’s tracking server. Specifically, the total training time, along with the loss function’s value at the end of each epoch, is logged. Additionally, all model hyperparameters are archived as JSON files for reproducibility, and the calculated threshold is stored for use during evaluation and inference. Finally, the trained model instance is saved in MLflow’s model registry, along with all the details of the environment in which it was developed and trained. Once hyperparameter tuning is complete, i.e., after training and comparing the loss function’s values for different model instances, one or more of them transition to the next stage.
For model evaluation, a different MLflow experiment is initiated, where all the discussed evaluation metrics are logged, including the average time taken to perform scoring on a single timestamp’s measurements. If, after evaluation, a model instance surpasses the performance of the currently deployed production model in terms of the evaluation metrics, it is then promoted to production, while the previous instance is archived. Naturally, being able to compare model instances and determine which is superior is not straightforward when multiple metrics are involved. Hence, the following composite metric is proposed:
where
is the model’s regular F1-Score,
is the corresponding PA F1-Score obtained using the PA%K-L protocol,
is the number of events in the time series,
is the number of undetected events, and
is the model’s delay in identifying the
i-th event. Additionally,
are tunable hyperparameters that regulate the contribution of each term to the overall score, and
are event-dependent factors that quantify the importance of the
i-th event’s identification delay. It should be noted that the delay can be defined in terms of either actual timestamps or sliding windows, depending on the problem at hand. The definition of what constitutes an event is also problem-dependent.
The first two terms of define its linear dependence on the two versions of the F1-Score, with the and coefficients regulating each term’s contribution. To place more focus on the regular F1-Score than the PA one, it suffices to choose suitable values for the coefficients, satisfying . However, in addition to this, it may also make sense to penalize significant discrepancies between the two. This is where the third term comes in, with quantifying the impact of these discrepancies on the overall score. The fourth term is related to the model’s delays in identifying events. Significant delays are equivalent to large values of and therefore lead to minor contributions to the composite metric, despite the successful identification of the corresponding events. To make the impact of the delays event-dependent, the factors are introduced. Thus, an event with higher significance than others (e.g., an event that is known to lead to the complete breakdown of a machine) can have a higher -value assigned to it. Of course, such assignments require explicit domain knowledge; therefore, it makes sense (at least initially) to set in order to avoid biases that do not arise from deep insights into the system’s physics. Finally, the fifth term is introduced to explicitly penalize the unsuccessful identification of an event. Even though the fourth term’s contribution would be zero in these cases (since ), it is sensible to impose a penalty by subtracting from the overall score, instead of simply “not adding” anything to it.
In the best-case scenario, the model manages to classify all points of the time series correctly, in which case
,
, and
, resulting in
. In the worst-case scenario, where the model fails to identify any of the time series’ anomalies,
,
and
hold, leading to
due to the max function that ensures no negative scores arise. Closing the discussion on the evaluation of model instances as part of their lifecycle, it is important to stress that, apart from the composite metric of Equation (
7), an additional requirement for a model instance to transition to production is to achieve a BF F1-Score that is higher than a pre-determined threshold, which is dataset-dependent.
Ultimately, the model instances that have advanced to the production stage are deployed for inference on new data. Since these new data are stored persistently in the lakehouse, with the process already described, they are available for labeling by a domain expert (e.g., an engineer on the vessel). In this way, normal segments can be used for future retraining of the model instances, and new evaluation datasets can be constructed, containing a mixture of normal data and faults. The development–staging–production cycle can then begin anew, with existing model instances being refined to achieve better performance or new model instances replacing them altogether.