A Novel End-to-End Provenance System for Predictive Maintenance: A Case Study for Industrial Machinery Predictive Maintenance

Abstract

In this study, we address the critical gap in predictive maintenance systems regarding the absence of a robust provenance system and specification. To tackle this issue, we propose a provenance system based on the PROV-O schema, designed to enhance explainability, accountability, and transparency in predictive maintenance processes. Our framework facilitates the collection, processing, recording, and visualization of provenance data, integrating them seamlessly into these systems. We developed a prototype to evaluate the effectiveness of our approach and conducted comprehensive user studies to assess the system’s usability. Participants found the extended PROV-O structure valuable, with improved task completion times. Furthermore, performance tests demonstrated that our system manages high workloads efficiently, with minimal overhead. The contributions of this study include the design of a provenance system tailored for predictive maintenance and a specification that ensures scalability and efficiency.

1. Introduction

The increasing complexity and autonomy of predictive maintenance systems highlight a critical gap in the current landscape: the absence of robust provenance mechanisms to ensure transparency, accountability, and explainability. As these systems are designed to prevent equipment failures by predicting potential issues before they occur, they inherently rely on complex algorithms, often driven by machine learning. While these algorithms improve efficiency and reduce downtime, they introduce significant challenges in terms of understanding, validating, and justifying the decisions made by the system. This lack of clarity and responsibility can reduce user confidence, make it harder to meet regulations, and slow down the use of these systems, especially in industries where equipment failures can have serious consequences.

Provenance is the documentation of data’s history, origin, and transformations throughout their lifecycle, ensuring transparency, accountability, and explainability, particularly in complex predictive maintenance systems [1]. These systems rely on advanced algorithms and machine learning models that analyze large volumes of IoT data to predict equipment failures, but understanding these decisions can be challenging without clear data traceability.

Provenance addresses these challenges by providing a structured account of data flow, applied algorithms, and outcomes. The PROV-O (Provenance Ontology) is a standardized framework that describes provenance information [2]. Using the PROV-O framework developed by the W3C, we capture key elements like data entities, activities, and agents, facilitating the understanding of complex workflows. Our study extends PROV-O with domain-specific features for predictive maintenance, such as tracking model performance, maintenance timings, and hyperparameters. This enhanced provenance system ensures decisions are transparent, auditable, and explainable, building user trust and meeting regulatory requirements.

In industries where unplanned downtime can result in significant financial losses, ensuring the reliability and transparency of predictive maintenance systems is essential. The ability to understand and explain the decisions made by these systems is not merely a technical requirement but a strategic necessity. The transition from reactive to proactive strategies in maintenance systems represents significant progress, shifting from a fail-and-fix approach to a predict-and-prevent strategy [3]. In these strategies, machine learning is a fundamental technology used to develop intelligent prediction algorithms [4]. In our previous work [5,6,7], we proposed a proactive maintenance framework. Predictive maintenance systems are crucial in minimizing unexpected failures. However, as these systems become more complex and autonomous, it becomes even more imperative to guarantee that their decisions and actions are clear and explainable. Predictive maintenance systems use data from Internet of Things (IoT) devices to predict equipment failures in advance. The data are analyzed to predict potential errors in the system, helping to ensure timely maintenance and reduce unplanned downtime by allowing for planned interventions. Provenance information can play a pivotal role in bridging this gap by providing a comprehensive view of the data flow and decision-making processes within predictive maintenance frameworks. It enables stakeholders to verify the correctness of predictions, understand the context in which decisions were made, and ensure that appropriate actions are taken based on accurate data. By incorporating provenance information through PROV-O, these systems can offer transparency, accountability, and explainability. This enables users to track the movement of data, understand the reasons behind maintenance operations, and validate the effectiveness of predictive models.

The absence of a provenance system in current predictive maintenance practices is a major concern, particularly as these systems increasingly rely on complex machine learning models to make autonomous decisions. Without a structured provenance mechanism, there is limited visibility into how data are processed, transformed, and used to arrive at predictions. This lack of traceability makes it difficult for operators to understand or justify system recommendations, undermining user trust and raising concerns about accountability. Furthermore, in industries such as manufacturing, aviation, and healthcare—where predictive maintenance is critical—failure to provide clear explanations for decisions can lead to serious operational and safety risks.

Additionally, the absence of provenance makes it challenging to audit system performance and validate compliance with regulatory standards, which often require detailed records of decision-making processes. Provenance systems provide a comprehensive way to document data origins, transformations, and the rationale behind maintenance actions, thus addressing these critical gaps. By enabling a transparent view of how predictive models function and how maintenance actions are derived, provenance systems significantly enhance the reliability and effectiveness of predictive maintenance practices.

The motivation to explore provenance in predictive maintenance systems emerged from the increasing complexity and autonomy of these systems, which use machine learning models to predict equipment failures. Our prior research and industry consultations highlighted significant challenges in understanding and trusting these models’ decisions. This gap in transparency, traceability, and accountability inspired our investigation into provenance as a solution.

While the existing PROV-O model provides a solid foundation for capturing provenance, it lacks domain-specific features needed for predictive maintenance. To address this, we extended PROV-O by adding attributes tailored to predictive maintenance, such as metrics for model performance and time indicators for maintenance activities. These enhancements improve explainability, auditability, and traceability, thereby increasing system reliability and user confidence.

Our paper outlines a comprehensive approach for integrating this extended provenance framework into predictive maintenance systems, detailing the design, implementation, and evaluation of our model. We developed a prototype and conducted user studies to assess its usability and effectiveness. Users performed tasks related to traceability, visualization, and explainability, and the results show that the expanded provenance model significantly improved task efficiency. Additionally, performance and scalability tests confirmed the system’s suitability for real-world applications.

This study addresses the critical gap by integrating provenance information into predictive maintenance systems. By extending the existing PROV-O schema, we introduce domain-specific features tailored to the needs of predictive maintenance, such as machine learning model metrics and time indicators for monitoring maintenance processes. Our approach not only enhances explainability, accountability, and transparency but also improves system usability and trustworthiness. We aim to integrate provenance information into predictive maintenance systems to make the decisions made by these systems explainable. We also aim to improve the user experience and system reliability by evaluating the usability and effectiveness of provenance information in predictive maintenance systems. Through a prototype implementation and comprehensive evaluations, including user studies and performance tests, we demonstrate the practical benefits of our framework in addressing the challenges of provenance in predictive maintenance systems.

We present the remaining sections of the article below. The second section investigates the research problem. The third section reviews previous studies and academic literature that form the foundation of our research. In the fourth section, we provide a detailed explanation of the proposed methodology. The fifth section discusses the prototype application we developed. In the sixth section, a user study and performance tests were conducted, and the results were evaluated. In the last section, we summarize our findings and outline future research directions.

2. Research Problem

Predictive maintenance systems, designed to forecast and prevent equipment failures, heavily rely on machine learning models to analyze vast amounts of data. However, current systems lack robust provenance mechanisms that allow for detailed tracking and validation of the data and models used to make these predictions. This gap results in challenges when it comes to justifying system decisions, ensuring regulatory compliance, and building user trust, particularly in industries wherein equipment failures can lead to significant financial loss or safety hazards. Furthermore, while the PROV-O offers a standardized framework for capturing and representing provenance information, it was not developed with the unique requirements of predictive maintenance in mind. The absence of domain-specific extensions for predictive maintenance systems limits the ability to fully capture the lifecycle of maintenance decisions and the data used to inform them.

Existing predictive maintenance systems face several significant challenges and limitations in the absence of a robust provenance framework. One of the most critical challenges is the lack of transparency in how maintenance decisions are made. Predictive maintenance systems often rely on complex machine learning models to analyze data and predict equipment failures. Without provenance, it becomes difficult to trace the origins of the data, understand the transformations applied, or identify the rationale behind specific predictions and actions. This lack of transparency can lead to confusion among operators and reduce trust in the system’s recommendations.

Another limitation is the inability to ensure accountability and auditability. In industrial settings, where maintenance decisions can have serious financial and safety implications, it is crucial to have a complete record of how predictions were generated and what data were used. The absence of provenance information makes it nearly impossible to audit the system’s performance, validate the accuracy of predictions, or identify potential sources of error. This can lead to challenges in regulatory compliance and hinder efforts to improve the system over time.

Furthermore, the lack of explainability is a significant drawback. Operators and maintenance teams need to understand why certain actions are recommended, especially when these actions involve costly or time-consuming interventions. Without a provenance framework, explanations for predictions remain opaque, making it difficult to justify maintenance operations to stakeholders or optimize decision-making processes. This can result in missed opportunities for preventive action or unnecessary maintenance, both of which reduce the overall efficiency and reliability of maintenance operations.

The research problem this study addresses is the lack of an integrated, efficient, and scalable provenance system for predictive maintenance. There is a need to develop an architectural framework that can collect, store, and query provenance data, while also extending the PROV-O schema to meet the specific needs of these systems. In doing so, we aim to improve the traceability, transparency, and overall explainability of predictive maintenance systems, while also evaluating the performance and scalability of the proposed solutions. To address this research problem, we aim to answer the following questions:

Research Question 1: What is the optimal architecture for a provenance collection and query system designed for predictive maintenance systems? With this research question, we aim to explore how to design and implement the optimal architectural framework for a provenance collection and query system designed for predictive maintenance systems. This architectural design includes the efficient capture, storage, and retrieval of provenance data to enhance the system’s capability to predict and prevent equipment failures.

Research Question 2: To what extent can PROV-O Specification be extended to better support the unique requirements of predictive maintenance systems? We aim to investigate how we can extend PROV-O Specification to meet the specific needs of predictive maintenance systems. We aim to identify the additional information and relationships required to fully capture provenance information for maintenance activities and decisions. We also aim to evaluate proposed PROV-O extensions that could increase the efficiency, comprehensiveness, and effectiveness of provenance records in predictive maintenance.

Research Question 3: What methodologies can be employed to effectively integrate provenance into a predictive maintenance system? Our goal with this question is to investigate the methods and tools available for successfully integrating provenance data into predictive maintenance systems.

Research Question 4: How can the performance of the provenance system be evaluated in terms of execution time and scalability? With this question, we aim to evaluate the proposed architecture’s performance by focusing on metrics such as execution time and scalability. The study will evaluate the system’s efficiency for handling high data volumes and its performance under increasing workloads.

Research Question 5: What are the comparative advantages and disadvantages of utilizing traditional logs and extended PROV-O graphs in predictive maintenance systems? With this question, we aim to examine the advantages and disadvantages of these two approaches by analyzing traditional logs and extended PROV-O graphs used in predictive maintenance systems. By evaluating the strengths and weaknesses of both approaches, we aim to understand which method better serves the important goals of the systems, such as traceability, transparency, and explainability.

3. Literature Review

Machine learning and IoT technologies have significantly improved predictive maintenance by accurately predicting equipment breakdowns and optimizing maintenance schedules [8]. Furthermore, the domain of data provenance has evolved in recent years, specifically with the creation of PROV-O Specification, which offers a standardized structure for capturing the history and origin of data. A significant amount of research has delved into how provenance can be used in machine learning. The authors of [9] developed a tool called MLflow2PROV, which extracts evidence from machine learning experiments in a W3C-PROV-compatible format. This tool makes it easier to understand what is happening during experiments and supports building better models. Similarly, the authors of [10] conducted a study using explainable artificial intelligence in the field of predictive maintenance. PROV-Template and similar approaches have aimed at increasing the usability of provenance information while reducing transfer and storage costs [11]. The authors of [12] proposed a system to track, store, and manage metadata and provenance information in machine learning experiments. Moreover, the authors of [13] proposed PROV-ML, a new method of representing provenance data designed for machine learning. Our work differs from these studies as it not only focuses on the application of provenance in specific contexts, such as machine learning workflows or predictive maintenance, but it also extends the provenance model by proposing a novel architecture that integrates provenance into predictive maintenance systems. Furthermore, we not only expanded the PROV-O schema, but we also developed a working prototype to showcase the functionality of our proposed architecture. We conducted user study using a simulation to assess this prototype. This study focused on evaluating the significance of the provenance information obtained by our expanded PROV-O Specification. Beyond these specific studies, there has been a significant amount of research into data provenance, covering both theoretical aspects [14,15,16,17,18,19] and practical applications like visualizing provenance [20,21], implementing self-healing and transparency in IoT applications [22,23], and exploring social computing [24,25]. Our work contributes to both the theory and practice of data provenance by modifying and expanding the PROV-O Specification to meet the needs of predictive maintenance systems. We not only produced theoretical advancements, but we also developed and tested a prototype of our proposed system. Through simulation-based user study, we assessed how the provenance information captured by our enhanced PROV-O schema improves the system’s ability to handle queries, provide clear explanations, and ensure transparency. Additionally, extensive research has been carried out in predictive maintenance for various fields, including industry applications [26,27,28,29,30,31,32,33], IoT applications [34,35,36,37,38], machine learning [39,40,41], HVAC systems [42], big data systems [43], cyber-physical production systems [44], and many other related areas [4,45,46,47,48,49,50,51,52]. Unlike domain-specific studies, our research focuses on incorporating provenance into predictive maintenance systems via the extension of the PROV-O Specification. We created and implemented a prototype of this extended design, which was assessed in a user study. This study focused on evaluating the significance of the provenance information obtained through our expanded provenance specification.

4. Methodology

To effectively address the challenges of transparency, accountability, and traceability in predictive maintenance systems, we propose an enhanced provenance management system. Provenance, which documents the history and transformation of data, plays an important role in ensuring that maintenance actions and decisions are understandable and verifiable. However, traditional provenance models like PROV-O lack the domain-specific features needed to fully support the complex requirements of predictive maintenance. In response, we extend the PROV-O schema and integrate it into a comprehensive provenance management framework tailored for predictive maintenance systems, ensuring that all key processes, from data collection to fault prediction, are thoroughly documented and explainable.

4.1. Proposed Provenance Management System

Provenance is the metadata which store the origins, history, and changes in data, enabling the tracing of information’s evolution across time. PROV-O offers a systematic framework for modeling and representing provenance information. The foundation of PROV-O consists of a small set of classes and properties that facilitate the creation of simple, initial provenance descriptions. An entity is a physical, digital, conceptual, or other kind of concept with some fixed aspects; entities may be real or imaginary. An activity is an occurrence over a period of time that acts upon or with entities, including consuming, processing, transforming, modifying, relocating, using, or generating entities. An agent is an object that bears responsibility for an activity taking place, the existence of an entity, or another agent’s activity.

Provenance is chosen over other methods due to its unique ability to provide a holistic view of data lineage, encompassing the origin, transformation, and usage of data throughout their lifecycle. Traditional methods, such as simple logging or auditing, often fall short in offering the detailed, relational context needed to fully understand complex predictive maintenance workflows. Provenance, particularly through the use of frameworks like PROV-O, not only captures data-processing steps but also documents the interactions between data, machine learning models, and maintenance actions. This detailed traceability enables better auditing, facilitates the explanation of system decisions, and enhances user trust. Additionally, the structured nature of provenance supports regulatory compliance, which is critical in sectors wherein system accountability is paramount. These advantages make provenance a superior approach for addressing the challenges posed by modern predictive maintenance systems.

Provenance management is responsible for collecting, processing, storing, querying, and visualizing provenance data to enhance the transparency, traceability, and accountability of the maintenance processes. Figure 1 illustrates the proposed workflow related to the collection, processing, storage, query, and visualization of provenance data. In the proposed provenance collection workflow, there exits five modules: Provenance Adaptor Module, Provenance Notification Ingester Module, Provenance Retrieval Client Module, Provenance Graph Generator Module, and Provenance Browser Module. These five modules work together to form a robust provenance management system, ensuring that every action and decision within the predictive maintenance system is documented, stored, and made accessible for further analysis. Each module plays a role in the collection, storage, retrieval, and visualization of provenance information, contributing to the overall goals of enhancing transparency, accountability, and traceability in prediction. We discuss details of the modules of the proposed workflow below.

Provenance Adaptor Module: The Provenance Adaptor module serves as the entry point for the provenance system, processing the log files generated by the predictive maintenance system. This module takes these raw log files as input, which contain details about every action, decision, and event in the maintenance workflow. The core functionality of the Provenance Adaptor module is to convert this unstructured log data into structured provenance notification statements. These statements are formatted in accordance with the PROV-O specification, ensuring that they capture essential provenance information such as the entities, activities, and agents involved in the workflow. Each provenance notification statement represents a discrete event or action in the maintenance process, creating a clear and traceable record. The module operates in real time, ensuring that provenance notifications are generated promptly as the log data are processed. The output of this module—provenance notification statements—is passed on to the next module in the workflow for storage and further management.

Provenance Notification Ingester Module: The Provenance Notification Ingester module is responsible for ingesting the provenance notifications generated by the Provenance Adaptor and storing them in a provenance repository. This module plays a critical role in organizing and persisting the provenance data, ensuring that all notifications are stored under a unified workflow ID. By associating each notification with a specific workflow ID, the system allows for efficient retrieval and management of provenance information later in the workflow. This workflow ID acts as a unique identifier, linking all provenance notifications related to a particular maintenance process, which can then be queried as a single PROV-O document. The ingester module is designed to handle large volumes of provenance data efficiently, ensuring scalability and reliability, especially when dealing with complex or long-running workflows.

Provenance Retrieval Client Module: The Provenance Retrieval Client module is tasked with retrieving stored provenance information from the repository using a given workflow ID. As input, this module takes a workflow ID, which it uses to locate and compile all the provenance notifications associated with that specific workflow. The module then organizes these notifications into a structured PROV-O document, representing the entire lifecycle of the maintenance process. This document can be used to audit the decisions and actions taken during the maintenance workflow, providing a clear, detailed record of how data were processed, transformed, and utilized throughout. By offering an easy way to retrieve comprehensive provenance documents, this module supports traceability and accountability in predictive maintenance systems.

Provenance Graph Generator Module: The Provenance Graph Generator module transforms the PROV-O documents retrieved by the Provenance Retrieval Client into visual representations. Using the structured provenance data, this module generates graphical outputs that allow users to intuitively understand the relationships between different entities, activities, and agents in the system. These visualizations are crucial for identifying patterns, tracing decision-making processes, and explaining complex workflows. The module uses graphs and diagrams to display provenance information in a way that is both accessible and actionable. By enabling users to see a visual summary of the workflow, this module enhances transparency and aids in the analysis of maintenance operations.

Provenance Browser Module: The Provenance Browser module serves as the user interface for exploring and navigating through all stored PROV-O documents in the system. It allows users to browse through the provenance information collected for different maintenance workflows, providing search and filtering options based on various criteria such as workflow ID, timestamp, or specific maintenance events. This module is designed to help users easily access and review historical provenance data, supporting the system’s goals of traceability, explainability, and accountability. The browser module integrates seamlessly with the retrieval and visualization tools, offering users a comprehensive view of the maintenance process and the ability to analyze provenance data from multiple angles.

4.2. Proposed Provenance Specification Designed for Predictive Maintenance Systems

Provenance data is essential for predictive maintenance systems, as it provides the transparency, accountability, and traceability needed to ensure reliable maintenance processes. However, while the standard PROV-O schema offers a solid framework for capturing provenance, it does not fully address the specific needs of predictive maintenance. To address these limitations, we extended PROV-O Specification by incorporating attributes specifically designed to meet the specific requirements of these systems. The selection of the PROV-O schema for our proposed system is based on several key considerations that align with the goals of transparency, accountability, and explainability in predictive maintenance. PROV-O provides a robust framework for modeling and representing provenance information. It is designed to capture essential elements of data lineage, such as entities, activities, and agents, which are critical for understanding the flow of data and the rationale behind system decisions.

In the context of our system, the PROV-O schema offers the flexibility needed to represent the complex workflows and transformations involved in predictive maintenance. It allows us to document how sensor data are collected, processed, and used in machine learning models to predict equipment failures. Moreover, PROV-O’s extensibility enables us to incorporate domain-specific attributes, such as model performance metrics and maintenance action types, which are essential for our application.

By leveraging the PROV-O schema, we ensure compatibility with existing tools and frameworks for provenance management, facilitating easier integration and future expansion of our system. Therefore, the selection of PROV-O is justified as it provides a comprehensive and adaptable solution for capturing the provenance of data and decisions in predictive maintenance systems.

The suitability of the PROV-O schema for our predictive maintenance system lies in its standardized and well-structured approach to representing provenance information. PROV-O, developed by the W3C, is designed to capture the relationships between entities, activities, and agents, making it ideal for documenting complex workflows and data transformations. This is particularly beneficial in predictive maintenance, where understanding how data flows and how decisions are made is crucial for transparency, accountability, and regulatory compliance.

One of the key strengths of PROV-O is its extensibility. The schema allows for the addition of custom attributes specific to the domain of predictive maintenance, such as model performance metrics, maintenance action types, and timestamps for real-time operations. This flexibility enables us to tailor the provenance system to our needs while maintaining compatibility with existing provenance tools and frameworks.

Alternative frameworks, such as the Open Provenance Model (OPM) or customized provenance systems developed for specific applications, could have been considered. The Open Provenance Model is another well-known approach that provides a flexible structure for representing provenance but lacks the formal standardization and broad community support of PROV-O. Custom frameworks, while potentially more optimized for real-time applications, would require significant development effort and may lack interoperability with other systems.

In comparing these options, PROV-O stands out for its balance between standardization and extensibility, making it a practical choice for our system. While it may have some limitations in high-throughput scenarios, our implementation addresses these through optimized data-processing techniques. Future work could explore hybrid approaches or performance enhancements to further improve the system’s scalability and efficiency in real-time applications.

Figure 2 illustrates the extended PROV-O classes.

Table 1. Attributes and their explanations.

Attribute	Explanation
actionType	Specifies the type of maintenance action taken (e.g., stop machine).
actionTarget	Specifies the target of a maintenance action, such as a specific machine.
actionTime	Records the specific time a maintenance action was executed.
name	Used to label entities such as datasets and actions within the system.
vMeasure	A clustering evaluation metric that measures the quality of the clustering model.
modelName	Captures the name of the machine learning model used.
rulValue	Represents the predicted Remaining Useful Life (RUL) of equipment.
dataId	Uniquely identifies datasets, facilitating precise tracking and referencing of data.
framework	Indicates the machine learning framework used (e.g., TensorFlow, MOA).
accuracy	Measures the overall correctness of the predictive models.
precision	Measures the accuracy of the positive predictions.
$F_1$ Score	A performance metric that considers both precision and recall.
recall	Measures the completeness of the positive predictions.
RMSE	Root Mean Squared Error, measures the square root of the average squared differences between predicted and observed values.
MAE	Mean Absolute Error, measures the average magnitude of the errors in a set of predictions.
MAPE	Mean Absolute Percentage Error, an error metric used to evaluate the accuracy of predictive models.
MSE	Mean Squared Error, measures the average of the squares of the errors.
silhouette	A metric used to measure the quality of clustering.
modelType	Indicates the type of machine learning model used (e.g., supervised, unsupervised).

presents the PROV-O attributes we extended for the predictive maintenance system, along with their explanations.

Attributes such as actionType, actionTarget, and actionTime are useful for accurately documenting maintenance actions, their targets, and the specific times at which they were performed or are required to be performed. Furthermore, performance-related characteristics such as accuracy, precision, recall, F1 Score, vMeasure, silhouette, RMSE, MSE, MAE, and MAPE provide a thorough evaluation of the machine learning models utilized for forecasting maintenance requirements. Other attributes, including dataId and modelName, enhance the traceability and identification of datasets and models, while framework and version specify the machine learning platform and version used. While trainingDataSize and testDataSize provide information about the data used during model creation, model-specific hyperparameters are specified within the hyperParameters attribute. These extensions ensure that the provenance information not only follows the fundamental principles of transparency and accountability but also offers the detailed insights required for making effective and well-informed maintenance decisions.

By including these specialized attributes, we have improved the suitability of PROV-O Specification for the complex requirements of predictive maintenance, hence improving the system’s ability to monitor, evaluate, and optimize maintenance operations.

Table 2. Terminology in the proposed provenance model.

Sub-Type	Explanation	Equivalence
Data Collection	To collect data from sensors	Sensor Data
Data Processing	To process data before using in ML activities	Processed Data
Model Generation	To generate ML models	Clustering Model, Classification Model
Anomaly Detection	To detect anomalies and label data	Labeling
Failure Prediction	To predict failure	Pre-fault Detection
RUL Calculation	To calculate the time before any error occurs	RUL
Alert Generation	To generate alert and actions	Stop Machine Action
Action Execution	To execute action in a defined time	Stopping the Machine
User	A person who uses the system	User
System	Predictive Maintenance System	Maintenance System
Administrator	A person who executes actions	System Admin
Original	Data collected from sensors	Sensor Data
Processed	Sensor data after being processed	Processed Data
Labeled	Labeled data after anomaly detection	Labeled Data
Model	Generated ML models	Clustering Model, Classification Model
Value	Values generated by models	RUL Value
Record	Outputs generated by activities	Action Record

provides examples and explanations of the terminology used in the proposed provenance model. We address Research Question 2 by presenting the specific extensions and modifications made to the PROV-O schema to better support the unique requirements of predictive maintenance systems.

Application of Extended PROV-O in Predictive Maintenance Systems: In a predictive maintenance system, the process begins with data collection from various sensors attached to machines, which capture vital parameters like temperature, pressure, humidity, and vibration [53]. These data points reflect the machine’s operational status and are transferred to the system via a gateway that connects IoT-based devices to more powerful processing units.

Once the data are collected, the anomaly detection and data labeling steps use unsupervised machine learning techniques to identify outliers—data points that deviate from expected patterns and may indicate potential failures. The labeled data are then fed into fault prediction models, which use supervised learning algorithms to forecast errors before they occur. This process forms the foundation of predictive maintenance, allowing the system to detect problems early and take preventive action. The real-time pre-fault detection step continuously monitors the machine data, using the trained models to predict potential failures as soon as anomalies are detected. This phase operates faster than traditional clustering models, making it ideal for real-time monitoring. Once a pre-fault condition is detected, the remaining useful life (RUL) calculation step estimates how much operational time is left before a failure occurs. In this context, the extensions made to the PROV-O specification provide crucial benefits. The detailed provenance data ensure that every action, decision, and data point within the predictive maintenance workflow is documented and traceable. For instance, when the anomaly detection module flags an issue, the actionType, actionTarget, actionTime, accuracy, and recall attributes allow for the provenance system to record the predictive maintenance actions. Furthermore, the extended provenance data offer a full history of the machine learning models used in the system. If a model’s predictions are questioned, the provenance information can trace the data used to train the model (example attributes: trainingDataSize, testestDataSize, hyperParameters, platform, framework, version). Finally, the provenance graph generator visualizes the entire maintenance process, showing the relationships between different entities, activities, and agents involved. This makes it easier to see how a decision was made, trace any potential issues, and ultimately improve the system’s transparency.

As shown in Figure 3, we present a predictive maintenance architecture integrated with provenance. This architecture aims to create a robust, transparent, and efficient predictive maintenance system by integrating comprehensive provenance management, ensuring that all actions and decisions are traceable, explainable, and accountable. By integrating these extended provenance capabilities into the predictive maintenance system, we have addressed the core challenges of traceability, explainability, and accountability. This ensures a more robust and reliable predictive maintenance framework, capable of handling the complexity of modern machine learning-driven operations while providing detailed insights into every aspect of the maintenance process. By introducing the proposed architecture for collection, refining, querying, and visualizing provenance data in predictive maintenance systems, we address Research Question 1.

5. Prototype Implementation

To facilitate testing of the proposed provenance system, we implemented an end-to-end predictive maintenance designed for industrial machinery. The prototype combines numerous technologies and approaches. Our end-to-end implementation of the predictive maintenance system is designed to operate in real time, ensuring that maintenance actions can be initiated promptly to prevent equipment failures. The system begins with real-time data collection from IoT sensors, which capture parameters such as temperature, pressure, and vibration. These data streams are immediately processed using unsupervised machine learning algorithms within the MOA (Massive Online Analysis) framework to detect anomalies and label data in real time. Anomalies are flagged as soon as they are identified, allowing the system to take corrective actions.

The real-time pre-fault detection module employs supervised machine learning models, such as the HoeffdingTree and LeveragingBag algorithms, which are optimized for continuous data streams. These models are trained to provide rapid predictions of potential faults, enabling proactive maintenance measures. Additionally, our system calculates the Remaining Useful Life (RUL) of equipment using deep learning techniques like LSTM and BiLSTM networks, which are configured to process sequences of sensor data efficiently. The RUL predictions are generated in real time, giving operators a clear window for taking action before failures occur.

To ensure the system’s responsiveness, we implemented a data pipeline that minimizes latency at each stage. The provenance tracking mechanism operates concurrently, capturing and storing provenance data without disrupting real-time operations. This ensures that all data transformations, model decisions, and maintenance actions are documented for auditing and explainability, even under real-time constraints. The overall architecture is designed to handle high data volumes and maintain performance, making it suitable for real-world industrial applications.

The data used in our predictive maintenance system are collected from IoT sensors installed on industrial machinery. These sensors capture critical parameters such as temperature, pressure, vibration, and humidity, which are essential for monitoring equipment health. The data are collected in real time and transmitted through a gateway to our processing system. The data pipeline is designed to ensure efficient and seamless handling of high-volume, continuous data streams.

Our data pipeline consists of several key stages:

Data Collection: IoT sensors continuously monitor machine conditions and generate data, which are then transmitted to the system using a secure and efficient communication protocol. The data are collected using the Cooja network simulator to replicate real-world conditions and ensure the reliability of our setup.

Data Ingestion: The raw sensor data are ingested into our system through RESTful APIs built on NodeJS. We use Docker containers to facilitate efficient communication between the data sources and the processing units, ensuring that data are received without delay.

Data Pre-Processing and Transformation: The ingested data are pre-processed to remove noise and fill in any missing values. This pre-processing step is crucial for maintaining the quality of the data used for model training and prediction. We apply real-time data normalization and feature engineering techniques to make the data suitable for machine learning algorithms. Anomaly detection and data labeling are performed using unsupervised machine learning techniques, such as DBSCAN and CluStream, to identify outliers and label data points accordingly.

Data Transformation: The labeled data are then used for fault prediction and RUL estimation. The transformation step includes generating features from the data sequences to feed into our deep learning models. We use LSTM and BiLSTM networks to process time-series data and predict the Remaining Useful Life of the equipment. This transformation is performed in real time to ensure the system can quickly respond to potential faults.

Data Utilization and Decision Making: The transformed data are used to make real-time predictions about equipment health and trigger maintenance actions if necessary. The predictions and decisions are documented through our provenance tracking mechanism, which captures the entire data flow and the transformations applied. This ensures that every step of the data pipeline is transparent and can be audited if needed.

To ensure the reliability of these actions and to make the system’s decisions explainable, we captured provenance logs throughout the process. These logs were then used to create provenance files with a .provn extension using the ProvToolbox. Subsequently, we employed the ProvConvert tool to generate provenance graphs from these .provn files. We used Neo4j as our graph database storage. In the system that generates provenance data, we developed a module to transfer these data into the Neo4j database. This allowed us to store our data in a graph database and query them when needed. In addition to the graphs created with ProvToolbox, this setup enables us to utilize Neo4j graphs, as well.

6. Performance Evaluations

To assess the usability of the proposed provenance system, we performed performance and scalability tests. We also conducted a user study to explore the effectiveness of the system from the end-users perspective.

6.1. Performance and Scalibility Experiments

In this context, we measured the times by inserting and reading provenance data of 10 k, 25 k, and 100 k size files into the provenance database used in the system to evaluate the potential load that might be brought to the system during the recording or reading of provenance data. The performance tests were conducted on a PC equipped with an Intel(R) Core(TM) i7-8550U 1.80GHz CPU and 16 GB RAM. Provenance data of sizes 10 k, 25 k, and 100 k were inserted into and read from the Neo4j database, which we used as provenance database, and the time taken for these operations was recorded. To ensure accuracy, each operation was repeated 100 times, and the time values were collected. To avoid any influence from the system, instead of running 100 iterations for each data size sequentially, the 10 k, 25 k, and 100 k operations were run in succession within each iteration. Additionally, the first iteration was excluded from the calculations to prevent any potential database connection overhead from affecting the results. This approach aimed at ensuring that all tests were conducted under the same conditions. The times recorded for each document type are shown in Figure 4, Figure 5 and Figure 6. Furthermore,

Table 3. Store and retrieve times with average and standard deviation for different document sizes.

Document Size	Operation	Average Time (ms)	Standard Deviation (ms)
10 k	Store	4.67	0.71
25 k	Store	4.70	0.77
100 k	Store	5.35	0.78
10 k	Retrieve	2.59	0.51
25 k	Retrieve	2.60	0.57
100 k	Retrieve	2.99	0.52

presents the average times and standard deviation values for all three file sizes.

The scalability test is an evaluation method that measures how a system performs under increasing workloads and data volumes. These tests are particularly important for understanding the stability and efficiency of a system under large datasets and high processing demands. In addition to our performance tests, we conducted scalability performance measurements in a multithreaded structure. In the multithreaded scalability test, each thread performed 100 store and retrieve operations, and the average time values and standard deviation values in milliseconds were recorded for 10 K, 25 K, and 100 K files, using 10, 100, and 500 threads. Figure 7 shows the system’s performance when storing provenance data during this test. Figure 8 illustrates the performance when retrieving these provenance data from the database. The results demonstrate that, despite the increase in the number of threads, the system maintained stable performance, especially with smaller file sizes (10 K and 25 K). Even in tests with 500 threads and 100 K files, the maximum recorded time was 647 ms (0.6 s), demonstrating a highly suitable and fast performance for such a system. These findings indicate that the system successfully maintains scalability under very high workloads and provides an optimized structure.

To test and validate this system, we developed a simulation application. Figure 9 shows the provenance graph generated by our simulation system. The graph provides detailed monitoring of the entire maintenance process and historical records. This approach not only increases the transparency and accountability of the system, but also ensures that all maintenance decisions are traceable. In this section, we address Research Question 3 by presenting the methodologies used to effectively integrate provenance data with the predictive maintenance system and the prototype implementation that demonstrates these methodologies in practice. Additionally, we provide our solution for Research Question 4, which examines the performance of the provenance system in terms of execution time and scalability.

6.2. User Study Experiment

A user study was carried out to assess the system’s usability and efficacy, with a particular focus on how well it fulfilled the goals of accountability, traceability, and explainability through the extension of the PROV-O schema. Ten test participants participated in the usability study, and each session took approximately 60 min. Initially, participants established a connection via Zoom and were provided with comprehensive explanations on the design and steps of the system. The users were provided with a detailed explanation of the tasks and procedures they would execute during the research.

Participants were given the task of completing five activities in each of the domains, namely traceability, visualization, and explainability, in a simulated environment. These tasks were first executed using the log files provided by the system. After completing the tasks using the log files, participants were introduced to the extended provenance graphs generated by our system. They were then asked to perform the same tasks using these graphs. During this process, the completion time of each task was recorded. After the task was completed, participants were asked to provide feedback using a 10-item survey using a 5-point Likert scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). The collected data were analyzed based on task completion times and survey responses. This analysis allowed us to evaluate the benefits of the extended provenance graph over the log files and to gain a deeper understanding of users’ interactions with the system.

The robustness of our user study in evaluating the usability of the prototype is supported by a structured approach and carefully selected participant pool. The study involved 10 participants, which is considered a sufficient sample size for preliminary usability testing, as recommended by usability research guidelines. This sample size allows us to identify major usability issues and gather meaningful insights into the effectiveness and user experience of the system. Our participants were selected from the field of information technology, with relevant experience in using predictive maintenance systems and data analysis tools, ensuring that they were well-suited to provide informed feedback.

The demographics of the participants were chosen to represent a range of expertise levels, from junior engineers to senior data analysts, to capture diverse perspectives on system usability. While the sample size may not provide exhaustive statistical significance, it offers a solid foundation for understanding the system’s usability and identifying areas for improvement. Future studies will involve a larger and more diverse group of participants, including domain-specific experts from industries where predictive maintenance is critical, to further validate our findings.

Overall, the user study provided valuable feedback on the traceability, transparency, and explainability features of our prototype, and the results indicate that the extended provenance model significantly enhances usability compared to traditional methods. However, we recognize the need for further research with expanded participant groups to generalize our findings more broadly.

6.3. User Study Tasks

A total of 15 user study tasks were created. The tasks were categorized, enabling evaluations to be performed at both the task level and the category level.

The following tasks are categorized into three groups: traceability, visualization, and explainability. The tasks in each category address different aspects of the predictive maintenance system, from tracing data flow and agent interactions to visualizing historical data and explaining the decisions made by artificial intelligence models.

Table 4. Tasks for predictive maintenance system.

Task ID	Task
1.1	Identify the origin of the processed data.
1.2	Trace the movement of data from its source to its final usage in a classification model generation activity.
1.3	Determine which agents interacted during the action execution activity.
1.4	Find the sequence of activities that led to the stop machine action.
1.5	Identify the start and end times of all activities involved in the stop machine action.
2.1	Find the entire history of data usage for the stop machine action.
2.2	Find when and how each piece of data was used in the stop machine action.
2.3	Find all maintenance actions taken.
2.4	Find the connections and relationships between different entities in the RUL model generation activity.
2.5	Find the data flow and transformations during the failure prediction activity.
3.1	Find the AI model name used for creating the latest clustering model in the maintenance process.
3.2	Describe the framework, model type, and version used for the last classification model.
3.3	Describe the training and testing data sizes for the latest RUL model.
3.4	List the hyperparameters used in the latest AI models for predictive maintenance.
3.5	Explain the performance metrics (accuracy, precision, f1, recall) for the last classification model.

presents the tasks, while

Table 5. List of survey questions.

Question ID	Question
1	The presented information is easy to understand, and I was able to complete the given tasks easily and quickly.
2	The presented information helps me track the data flow clearly.
3	I can easily identify the reasons for actions taken by the system.
4	The system’s decision-making process is transparent.
5	I can easily see the performance information of the generated models.
6	I can easily track activities and data changes in the system.
7	I can quickly find the source of any data or decision.
8	I have confidence in the system’s accountability.
9	The presented information enhances my trust in the system.
10	Overall, the presented information improves my experience with the predictive maintenance system.

outlines the survey questions.

User Study Participants: The usability study was conducted from 1 August 2024 to 8 August 2024, with 10 test participants. The participants were selected from the information technology field to match the system’s operational requirements.

Participants started the process within the simulation system and progressed through data collection, labeling, anomaly and pre-failure detection, remaining lifetime computing, and action generation activities. They performed the assigned duties by independent completion of the system outputs, namely the system’s text logs and extended provenance graphs. Each activity was completed with participants answering questions using the Likert scale.

Time of Tasks: The aim of the usability study was to evaluate the usability of our application and the importance of extended provenance from the perspectives of traceability, visualization, and explainability with users. While completing each task, the time spent was automatically recorded in the database by the system both with classical logs and provenance.

User Study Results: When we look at the user study results, as shown in

Table 6. Comparison of Log and PROV-O responses.

Question ID	Log min	Log max	Log avg	Prov min	Prov max	Prov avg
1	1	3	2.40	4	5	4.80
2	2	3	2.60	4	5	4.80
3	1	3	2.20	4	5	4.60
4	1	4	2.50	3	5	4.60
5	1	3	2.00	4	5	4.60
6	1	4	2.30	4	5	4.50
7	2	3	2.10	4	5	4.40
8	1	4	3.00	4	5	4.40
9	1	4	2.40	4	5	4.60
10	1	4	2.80	4	5	4.80

, users rated the extended PROV-O significantly higher. While the average log scores range between 2.00 and 3.00, these values range from 4.40 to 4.80 for the extended PROV-O.

Table 7. Comparison of Log and Prov response times in ms.

Category ID	Log min	Log max	Log avg	Prov min	Prov max	Prov avg
1	26,703.0	216,352.0	94,596.1	11,395.0	63,432.0	28,730.64
2	24,426.0	177,392.0	58,717.58	14,458.0	128,949.0	28,033.98
3	30,593.0	118,683.0	57,250.86	9407.0	59,504.0	24,557.52

also displays the minimum and maximum scores given for both methods. The average scores per question and their comparisons can be more easily examined by referring to Figure 10. As shown in this figure, there are significant differences in the averages across all questions. However, making a decision based solely on the averages in the graph might not be sufficient. Therefore, in Figure 11, we present the visual values as min (minimum), max (maximum), and avg (average). This allows for an analysis of which questions have closer or greater differences.

In terms of response times, it is observed that tasks based on logs take much longer than those based on the extended-PROV-O. The min, max, and avg values provided in Table 7 clearly demonstrate this difference by task category. When looking at response times by question, these differences are evident across all questions. Figure 12 graphically shows these values.

This section has presented the measured time values and the user ratings. In the discussion section, the meaning of these results and the reasons for these outcomes will be discussed.

6.4. Discussion

The analysis reveals, among other things, a considerable decrease in task completion times when utilizing the extended PROV-O system as opposed to log files. For example, as shown in Table 7, the average task duration for Category 1 was measured at 94,596.1 ms when using log files, while our proposed extended PROV-O system only took 28,730.64 ms. This consistent trend across all categories suggests that users of the extended PROV-O system are able to complete activities faster and more efficiently. The shorter task durations indicate that the extended PROV-O system offers significant advantages in terms of usability and explainability, as the visuals help users understand and process the data more quickly. Furthermore, it can be seen that the times were higher initially and then decreased somewhat. This can be interpreted as users initially needing more time to understand the system and later completing tasks more quickly as they became accustomed to the survey system.

The user research assessments likewise showed these task completion time findings; the extended PROV-O-based system regularly received better scores. With the average Likert scale values ranging from 4.40 to 4.80, participants judged the extended PROV-O system much more highly. These good marks show how well the system offers dependable, clear, understandable information. Particularly in areas like simplicity of understanding, monitoring data flow, and general usability—as shown in Questions 1, 2, and 10—the organized and visual approach of the extended PROV-O system greatly boosted user experience.

Though they offered some degree of assistance, log-based approaches were usually judged as less successful in all the evaluated criteria. With average scores generally between 2.00 and 3.00, the Likert scale answers for logs revealed some difficulties in following data flow, grasping the information, and identifying the justification behind system choices. Particularly for showing performance metrics (Question 5) and defining the reasons behind actions (Question 3), the lower ratings underline the shortcomings of the log-based approach in supplying the required transparency and clarity for efficient decision making.

The extended PROV-O method also greatly enhanced the participants’ capacity to rapidly locate data sources or decision points (Question 7, average score 4.40) and to track activities and data changes (Question 6, average score 4.50). These results highlight the need for traceability and openness in complicated systems, where efficient decision making depends on quick access to correct knowledge.

The necessity of organized provenance information to improve system openness, responsibility, and usability is shown by the difference between log-based and provenance-based methods. The significant variations in user ratings and task durations between the two approaches imply that, while conventional log files have value, they cannot satisfy the needs of current, data-driven maintenance systems. On the other hand, by including domain-relevant traits and metrics that support more informed and accurate decision making, the extended PROV-O framework offers a stronger and more user-friendly solution.

This study shows how provenance data may close the gap between sophisticated machine learning models and human operators, thereby building confidence and enhancing industrial maintenance decision making. It also emphasizes the need to expand the provenance ontology for reasonable decisions in systems of predictive maintenance. Emphasized is the need to document and retrospectively monitor the data leading to decisions, model and software specifics, and performance measures connected to the particular model and system. Detailed provenance graphs help users understand the reasoning for maintenance activities, hence improving the transparency and accountability of the system’s operations.

The performance tests indicate that storing or reading provenance data does not introduce a significant additional load to the system. Our tests showed that data could be stored within an average of 4–5 ms and read within 2–3 ms. Additionally, as the file size increases, the impact on load remains negligible.

To answer Research Question 5, this work assessed the relative benefits and drawbacks of conventional logs compared to extended PROV-O graphs in predictive maintenance systems. Based on user survey findings and job completion times, the results show that the extended PROV-O framework not only improves the user experience but also facilitates more effective and transparent maintenance operations, therefore making it a better alternative for contemporary predictive maintenance systems.

6.5. Limitations of the Study

While our study presents a comprehensive approach to integrating provenance into predictive maintenance systems, several limitations should be acknowledged. Firstly, the sample size used in our user study is relatively small, consisting of only 10 participants from the information technology field. Although this sample size provided meaningful preliminary insights, it may not fully capture the diversity of perspectives or cover the entire range of expertise found in real-world industrial environments. Future research should include a larger and more varied participant pool to validate and generalize the findings.

Secondly, the performance evaluations of the provenance system, while showing promising results, were conducted under controlled simulation conditions. These conditions may not accurately reflect the complexities and constraints of actual industrial settings, such as varying network latencies, hardware limitations, or the integration with legacy systems. Testing the system in real-world environments is necessary to fully assess its scalability, robustness, and impact on maintenance operations.

Another limitation lies in the use of the PROV-O schema. Although PROV-O offers significant flexibility and standardization, it may not be fully optimized for real-time, high-frequency data environments typical of predictive maintenance. The system could face challenges in efficiently handling large data volumes and providing low-latency responses. Additionally, the complexity of managing and querying extensive provenance records might affect system performance. Future work should explore potential optimizations or the use of complementary frameworks to improve efficiency in real-time applications.

Lastly, our study primarily focuses on the technical implementation and usability aspects of the provenance system. It does not thoroughly investigate the economic impact or cost–benefit analysis of deploying such a system in industrial settings. Evaluating the return on investment and operational benefits will be crucial for practical adoption and scalability of the solution.

By acknowledging these limitations, we aim to provide a balanced perspective on our research and identify areas for future exploration and improvement.

7. Conclusions and Future Work

In this study, we successfully integrated provenance into predictive maintenance systems, creating a provenance-enabled framework that enhances transparency, accountability, and explainability. By extending the standard PROV-O schema with domain-specific attributes, we designed a robust system for documenting and tracking data, decisions, and actions within the maintenance workflow. This research demonstrated how provenance can be applied to predictive maintenance systems, and we outlined the key benefits it brings, including improved traceability of actions and enhanced decision-making processes.

The contributions of this study are twofold. First, we extended the PROV-O schema with attributes tailored to the specific needs of predictive maintenance systems, such as maintenance action types, model performance metrics, and data traceability elements. This extension provides a detailed framework for capturing complex machine learning-driven operations in a transparent and accountable manner. Second, the implementation of provenance in a real-world predictive maintenance system offers a new approach to understanding and managing the decisions made by these systems.

Our experimental work included user studies and performance evaluations, which provided key insights into the system’s usability and efficiency. Participants in the user study rated the provenance-enhanced system highly, citing the value of provenance information in understanding the maintenance process. The system was also evaluated for its performance under heavy workloads, and the results demonstrate that the overhead introduced by the provenance system was minimal. The system handled large data volumes efficiently, confirming the scalability and practical applicability of the approach.

The results of this study directly address the research problem by demonstrating that integrating provenance into predictive maintenance systems improves explainability and trust. The extended provenance system provides a clear, traceable record of actions and decisions, allowing operators to better understand and manage complex machine learning models. This addresses the core issue of transparency and accountability in predictive maintenance, offering a solution that enhances user confidence and system reliability.

Moving forward, further research could optimize the provenance features and expand this study to a broader participant base to validate these findings across different industrial sectors. The ongoing development of provenance tools and techniques will continue to support the growing demand for explainable and accountable artificial intelligence systems in predictive maintenance and beyond.