A State-of-the-Art Review in Big Data Management Engineering: Real-Life Case Studies, Challenges, and Future Research Directions
1
Department of Management Science and Technology, University of Patras, 26334 Patras, Greece
2
Department of Business Administration, University of Patras, 26504 Patras, Greece
*
Author to whom correspondence should be addressed.
Eng 2024, 5(3), 1266-1297; https://doi.org/10.3390/eng5030068
Submission received: 5 June 2024
/
Revised: 27 June 2024
/
Accepted: 1 July 2024
/
Published: 3 July 2024
(This article belongs to the Special Issue Feature Papers in Eng 2024)
Abstract
:The explosion of data volume in the digital age has completely changed the corporate and industrial environments. In-depth analysis of large datasets to support strategic decision-making and innovation is the main focus of this paper’s exploration of big data management engineering. A thorough examination of the basic elements and approaches necessary for efficient big data use—data collecting, storage, processing, analysis, and visualization—is given in this paper. With real-life case studies from several sectors to complement our exploration of cutting-edge methods in big data management, we present useful applications and results. This document lists the difficulties in handling big data, such as guaranteeing scalability, governance, and data quality. It also describes possible future study paths to deal with these issues and promote ongoing creativity. The results stress the need to combine cutting-edge technology with industry standards to improve decision-making based on data. Through an analysis of approaches such as machine learning, real-time data processing, and predictive analytics, this paper offers insightful information to companies hoping to use big data as a strategic advantage. Lastly, this paper presents real-life use cases in different sectors and discusses future trends such as the utilization of big data by emerging technologies.
1. Introduction
In today’s fast-paced digital world, the explosion of data has completely reshaped the business and industrial landscapes. Organizations are now surrounded by data from numerous sources, including traditional systems, social media, and IoT devices. This influx of data offers a huge opportunity to gain valuable insights for strategic decisions and innovation, but it also brings the significant challenge of managing, processing, and analyzing vast and varied datasets effectively [1].
At the core of handling this data flood is big data information engineering—a discipline that merges art and science to unlock data’s transformative power. It is the foundation on which data-driven companies build their strategies, helping them extract actionable insights, optimize operations, and stay competitive [2]. However, managing large-scale data involves complexities like scalability, data variety, and real-time processing, requiring advanced methods and technologies [3].
This paper explores big data information engineering in detail, highlighting its importance in our data-centric world. We provide a thorough analysis of its key components and principles, offering organizations a guide to using big data effectively for innovation and strategic goals. From data collection to storage, processing, analysis, and visualization, every aspect is crucial in revealing the value hidden in large datasets.
By examining the methodologies, tools, and best practices of big data information engineering, we aim to give organizations the knowledge they need to navigate today’s complex data environment. Using real-world examples, case studies, and strategic insights, we seek to equip decision-makers, data practitioners, and industry stakeholders with the skills to leverage big data as a strategic asset. Embracing these principles will not only help organizations survive but also thrive in an era defined by data-driven innovation and disruption.
In this paper, as outlined in Table 1, we review and present the state-of-the-art papers in the field of big data management, describing their scope. Additionally, after discussing each work, we emphasize the scope of our research, consolidating all essential up-to-date knowledge about big data management.
Table 1.
Summary of papers on big data management.
| Reference | Survey | Scope |
|---|---|---|
| [4] | TinyML algorithms for big data management in large-scale IoT systems | Introduces a set of Tiny Machine Learning (TinyML) algorithms designed to improve big data management within large-scale IoT systems. These algorithms—TinyCleanEDF, EdgeClusterML, CompressEdgeML, CacheEdgeML, and TinyHybridSenseQ—address various aspects such as data processing, storage, and quality control using Edge AI capabilities. |
| [5] | Role of IoT technologies in big data management systems: A review and Smart Grid case study | Explores how IoT devices generate vast amounts of data and the subsequent challenges in processing, storing, and analyzing these data efficiently. |
| [6] | Efficient and secure medical big data management system using optimal map-reduce framework and deep learning | Focuses on managing and securing large-scale medical data using a combination of optimal map-reduce frameworks and deep learning techniques in a cloud environment. Proposes a system that includes patient authentication, big data management, secure data transfer, and big data classification, highlighting improvements in data processing efficiency, security, and classification accuracy. |
| [7] | Big data optimization and management in supply chain management: a systematic literature review | Aims to provide comprehensive insights into big data management and optimization technologies in SCM, highlighting current applications and identifying research gaps for future exploration. |
| [8] | Research on spatial big data management and high-performance computing based on information cloud platform | Explores the management of spatial big data and the implementation of high-performance computing (HPC) on an information cloud platform. Focuses on optimizing data storage, processing, and analysis to improve efficiency and performance in handling large-scale spatial datasets. |
| [9] | Integration of big data analytics and the cloud environment in harnessing valuable business insights | Explores the integration of big data analytics with cloud computing to generate valuable business insights. Discusses the selection of cloud service providers and tools, addressing challenges in data processing, storage, and security. |
| [10] | Research on the application of big data management in enterprise management decision-making | Explores how in-depth analysis and big data management can enhance decision-making ability and execution efficiency, promoting the realization of corporate strategic goals. |
| [11] | Big data management performance evaluation in Hadoop ecosystem | Examines various big data management tools within the Hadoop ecosystem, focusing on three levels including distributed file systems, NoSQL databases, and SQL-like components. Provides a comprehensive performance evaluation of typical technologies such as HDFS, HBase, MongoDB, and Hive, among others, comparing their features, advantages, and performance metrics. |
| Our work | A state-of-the-art review in big data information engineering: real-life case studies, challenges, and future research directions | Explores the complexities and opportunities in big data information engineering. Covers the full spectrum of big data management, including data collection, storage, processing, analysis, integration, and visualization. Emphasizes methodologies, technologies, and best practices essential for leveraging big data to drive strategic decision-making and innovation across various industries. Addresses challenges such as data quality, governance, scalability, and presents real-life case studies and future research directions. |
The purpose of this work is to investigate, highlight, and contribute to the understanding and improvement of big data management engineering by presenting key factors and methodologies. This includes the following:
- •
- Presenting an overview of state-of-the-art big data management through a comprehensive, specific, and up-to-date analysis of significant methodologies, tools, and best practices in the field.
- •
- Analyzing real-life case studies and their implementations across various industries to showcase practical applications and outcomes, highlighting the successes and challenges encountered.
- •
- Identifying and exploring unresolved issues and potential research directions in big data management, thereby creating a roadmap for future studies and innovations in academia and industry.
- •
- Offering a thorough survey that aids readers in understanding the broader scope of big data management by summarizing knowledge from various sources, without the necessity to review all recent works individually.
Our paper also elaborates on the symbiotic relationship between big data and deep learning. We explore how big data technologies support the data lifecycle from collection to processing and analysis, which in turn facilitates the development and deployment of deep learning models. For instance, deep learning models for image recognition are trained on vast amounts of labeled images to achieve high accuracy [12]. Similarly, natural language processing models rely on extensive textual datasets to understand and generate human language [13].
The remainder of this article is organized as follows: Section 2 provides an overview of big data management engineering. Section 3 presents case studies, showcasing successful implementations of big data projects across different industries. Section 4 discusses challenges and future research directions. Finally, Section 5 concludes this article by summarizing key points and suggesting potential future directions. Figure 1 shows a general overview of a Big Data Management Framework.
Figure 1.
Big data management overview.
2. Overview of Big Data Management Engineering
In the current digital era, the term “big data” refers to datasets that are so large or complex that traditional data processing applications are inadequate. The characteristics of big data are commonly described by the “5Vs” framework, encompassing volume, velocity, variety, veracity, and value, which are described as follows:
- (a)
- Volume: This refers to the vast amounts of data generated every second. Organizations collect data from various sources, including business transactions, social media, sensors, and more, leading to an explosion in data volume that requires scalable storage and processing solutions. For instance, social media platforms generate petabytes of data daily, which need to be managed efficiently.
- (b)
- Velocity: This denotes the speed at which new data are generated and the pace at which they need to be processed. In the age of IoT and real-time analytics, the ability to process data streams rapidly and efficiently is crucial. Real-time data processing frameworks like Apache Kafka and Apache Flink are often employed to handle the high velocity of data inflow, enabling real-time decision-making.
- (c)
- Variety: This aspect covers the different types of data, both structured (e.g., databases) and unstructured (e.g., text, images, videos), that organizations must manage and analyze. For example, healthcare data can range from patient records (structured) to medical imaging and doctors’ notes (unstructured).
- (d)
- Veracity: This refers to the trustworthiness and accuracy of the data. Data quality and reliability are critical for making informed decisions. Big data environments often deal with data from various sources, which may include noise, biases, and abnormalities. Techniques such as data cleansing and validation are essential to ensure high data veracity.
- (e)
- Value: This represents the worth that can be extracted from data. Despite having large volumes of data, the real challenge lies in turning these data into actionable insights that can drive business decisions and innovation. Big data analytics, through methods like predictive analytics and machine learning, can unlock significant value by uncovering patterns, trends, and correlations that inform strategic decisions [14].
These five dimensions form the foundation of our exploration into big data management engineering. The core of handling this data flood is big data information engineering, a discipline that merges art and science to unlock data’s transformative power. Our analysis is based on advanced methods and technologies necessary for managing large-scale data complexities such as scalability, data variety, and real-time processing.
2.1. Data Collection and Storage
At the start of any data project, the initial phase focuses on gathering data, serving as the crucial first step in information engineering. This stage involves collecting data from a variety of sources, including traditional databases and business systems as well as new technologies like sensors, IoT devices, social media platforms, and web scraping tools. By utilizing this array of sources, businesses can access a range of data types—structured, semi-structured, and unstructured—thereby enhancing their ability to analyze and derive valuable insights.
The methods employed for data collection are as diverse as the sources themselves, with organizations leveraging a combination of technologies and techniques tailored to their specific needs and objectives. For instance, sensors and IoT devices offer real-time data streams [4], providing instantaneous insights into operational metrics, environmental conditions, and consumer behaviors. Social media APIs enable the extraction of valuable sentiment analysis, demographic trends, and market sentiments, while web scraping tools empower organizations to gather data from the vast expanse of the internet, including news articles, forums, and product reviews.
After collecting data from various sources, the next crucial step is to store it efficiently for easier processing and analysis. In the realm of big data, where data volumes can reach petabytes and beyond, traditional storage options fall short of meeting scalability and performance needs. Therefore, companies opt for a variety of storage technologies and structures tailored to handle the specific demands of big data.
Among these technologies, distributed file systems such as the Hadoop Distributed File System (HDFS) emerge as a cornerstone, offering a scalable and fault-tolerant framework for storing vast datasets across clusters of commodity hardware. By distributing data across multiple nodes, HDFS not only ensures high availability and fault tolerance but also enables parallel processing, facilitating rapid data retrieval and analysis [15]. Furthermore, the emergence of NoSQL databases, including MongoDB, Cassandra, and HBase, provides organizations with flexible and schema-less alternatives to traditional relational databases, catering to the diverse data structures and access patterns prevalent in big data applications.
The rise of cloud computing has transformed data storage, complementing on-premises systems [9]. Cloud storage services like Amazon S3, Google Cloud Storage, and Microsoft Azure Blob Storage offer scalability, flexibility, and cost-effectiveness to businesses. They allow organizations to expand their storage capabilities easily as data needs fluctuate through a pay-as-you-go model [16]. Additionally, these cloud solutions come with managed services and data tools that streamline data management tasks, freeing up organizations to concentrate on innovation and extracting insights [8]. Figure 2 represents the architecture of cloud computing.
Figure 2.
Cloud computing architecture.
Data gathering and storage serve as the foundation for managing big data, paving the way for further steps such as processing, analyzing, and presenting information. Through the use of advanced technologies and structures, companies can not only receive and store large amounts of data but also establish a base for uncovering practical insights and guiding important decision-making in the age of big data.
2.2. Data Processing and Analysis
In the realm of big data information engineering, data processing stands as a cornerstone, representing the pivotal transition from raw data to valuable insights that drive informed decision-making and strategic initiatives [10]. This transformative process encompasses a spectrum of operations, including data cleansing, transformation, aggregation, and enrichment, aimed at harnessing the latent potential within vast datasets and converting it into actionable knowledge.
At the forefront of data processing technologies lies a suite of powerful frameworks and platforms tailored to meet the diverse needs of modern data-driven enterprises. Among these, Apache Hadoop emerges as a stalwart, offering a distributed processing framework that enables the parallel execution of data-intensive tasks across clusters of commodity hardware [11]. Through the MapReduce programming model, Hadoop facilitates the efficient processing of massive datasets by partitioning them into smaller, manageable chunks and distributing them across nodes for concurrent processing [5]. This enables organizations to tackle complex analytical tasks, such as batch processing and large-scale data transformations, with unparalleled scalability and fault tolerance. Figure 3 shows the main architecture of MapReduce.
Figure 3.
MapReduce architecture.
Apache Spark complements Hadoop by offering a fast, in-memory processing engine that aims to overcome the constraints of conventional MapReduce methods. Utilizing resilient distributed datasets (RDDs) and a user-friendly API, Spark accelerates data processing tasks by storing interim results in memory, thereby reducing disk I/O demands and improving performance [17]. This feature makes Spark ideal for iterative algorithms, interactive queries, and live stream processing, enabling businesses to extract valuable insights from data quickly and efficiently. Figure 4 visualizes the architecture of Apache Spark.
Figure 4.
Apache Spark architecture.
Furthermore, Apache Flink emerges as a leader in the field of real-time stream processing. It provides low-latency and high-throughput capabilities for analyzing continuous data streams in a manner that is close to real-time [18]. Figure 5 shows the architecture of Apache Flink.
Figure 5.
Apache Flink architecture.
Following the completion of the processing and transformation of the data, the next important step is to utilize rigorous analysis and exploration in order to extract the insights contained within the data. In this regard, a wide variety of approaches and procedures for data analysis come into play. These include more conventional statistical analyses and hypothesis testing, as well as more modern machine learning algorithms and data mining techniques [19]. For companies, statistical analysis acts as a basic pillar, allowing them to obtain descriptive and inferential insights from data. These insights might range from summary statistics and distributional studies to correlation and regression analyses. These techniques provide valuable insights into the underlying patterns, trends, and relationships within data, thereby informing strategic decision-making and guiding resource allocation.
Machine learning is becoming increasingly powerful in revealing insights and extracting useful information from data. By using techniques like classification, regression, clustering, and anomaly detection, businesses can discover patterns, predict upcoming trends, and streamline decision-making processes [20]. This gives them an advantage in fast-paced and unpredictable environments. Table 2 shows the most common Machine learning techniques.
Table 2.
Machine learning techniques.
| Machine Learning Algorithm | Use Case and Description |
|---|---|
| Classification | Predicts the class or category of new observations based on training data. Commonly used for sentiment analysis, image recognition, and customer segmentation. |
| Regression | Models the relationship among variables to predict continuous outcomes. Used for sales forecasting, pricing optimization, and demand prediction. |
| Clustering | Identifies natural groupings within data, based on similarity. Helps in customer segmentation, anomaly detection, and pattern recognition in unlabeled datasets. |
| Anomaly Detection | Detects outliers or anomalies in data that deviate from the norm. Useful for fraud detection, network security monitoring, and predictive maintenance in manufacturing. |
In addition, data mining techniques make it easier to find patterns and relationships within data that were not previously recognized. This allows for the discovery of significant insights that may have been masked by noise or complexity. Uncovering actionable insights, recognizing market trends, and optimizing business processes can all be accomplished by businesses through the utilization of exploratory data analysis, association rule mining, and clustering algorithms. This helps firms drive continuous improvement and innovation.
2.3. Data Integration and Visualization
The process of data integration acts as a key bridge in the complex terrain of big data information engineering. It connects different data sources and harmonizes heterogeneous datasets in order to produce a unified and coherent perspective of the information that lies under the surface. In its most fundamental form, data integration is the process of combining data from a variety of sources in a seamless manner [21]. This process encompasses organized, semi-structured, and unstructured forms, and it is designed to facilitate comprehensive analysis and decision-making.
Data integration involves a variety of tasks, starting with data cleansing, a process to fix errors, inconsistencies, and duplicates in the data. By using methods such as removing duplicates, identifying erroneous data points, and correcting mistakes, companies can ensure that the combined dataset is accurate, complete, and reliable, setting the stage for analysis and generating insights [22]. Also, data transformation is crucial in the data integration process as it helps unify data formats, structures, and meanings. This includes converting data from their original form into a standard representation to enable smooth compatibility across systems [23]. Through methods like standardization, summarization, and enrichment, companies can improve the usability and relevance of the combined dataset, leading to uncovering insights and promoting collaboration across different departments.
Furthermore, combining data structures into a single cohesive model through schema integration is crucial for organizations. This process involves aligning and connecting data elements, entities, and relationships from various sources to establish a shared semantic structure. It promotes analysis and decision-making regardless of the source or format of the data.
In tandem with the process of integrating data, data visualization is emerging as a strong tool that can translate complicated facts into insights that are easy to understand and can be put into action. These insights connect with stakeholders from all around the company. Organizations are able to condense complicated analytical results into visually captivating tales by utilizing charts, graphs, dashboards, and interactive visualizations. This enables stakeholders to understand essential insights at a glance and make choices with confidence that are informed by the information [24].
Data visualization plays a crucial role in communication, helping organizations present data analysis findings in a visually understandable way that transcends language barriers and specialized knowledge. By leveraging our ability to interpret visual information quickly, data visualization allows stakeholders to spot trends, detect patterns, and draw practical conclusions from data. This, in turn, aids decision-making and fosters a culture of data-driven decisions within the organization [25]. Additionally, data visualization promotes sharing insights and knowledge throughout the organization, enabling stakeholders to interact with data dynamically. With charts, dynamic dashboards, and self-service analytics tools, organizations empower stakeholders to explore data on their own terms, uncovering valuable insights that drive continuous growth and innovation.
It is important to note that data integration and visualization are two parts of big data information engineering that are interrelated. Data integration serves as the foundation for unified analysis, while data visualization helps to improve the dissemination of insights and provides support for decision-making capabilities. Big data may be utilized to their full potential by organizations through the combination of data integration and visualization [26]. This enables organizations to transform raw data into actionable insights that aid strategic decision-making, innovation, and growth.
2.4. Real-Time Data Processing
In today’s fast-paced and interconnected digital world, quickly analyzing real-time data has become crucial for companies looking to stay ahead, react promptly to new trends, and seize fleeting opportunities. Real-time data processing marks a departure from traditional batch methods, allowing organizations to analyze and respond to data as they come in, leading to instant insights and actions based on the most up-to-date information. At the core of real-time data processing is the concept of data streams—limitless sequences of data that flow constantly from various sources such as sensors, IoT devices, social media platforms, and transactional systems [27]. These streams contain insights, passing trends, and important events that require immediate attention and action, underscoring the necessity of real-time processing for organizations navigating dynamic and rapidly changing landscapes. Figure 6 shows Real-time data processing.
Figure 6.
Real-time data processing.
Real-time data processing is built on a collection of cutting-edge technologies and frameworks that are designed to manage the speed, volume, and diversity of streaming data. This gives real-time data processing its basis. Among them, Apache Kafka stands out as a cornerstone because it provides a platform for distributed messaging that acts as the foundation for real-time data pipelines. Kafka makes it possible for businesses to ingest, process, and publish streams of data with low latency and high throughput [28]. This makes it possible for Kafka to provide seamless interaction with the downstream processing systems and analytics engines to be implemented. Figure 7 depicts the architecture of Apache Kafka.
Figure 7.
Apache Kafka architecture.
Kafka pairs well with Apache Storm, a framework designed for high-speed data stream processing. Storm allows organizations to analyze streaming data quickly and accurately by utilizing real-time processing and fault-tolerant mechanisms [29]. This capability helps in performing analytics, detecting anomalies, and recognizing patterns in real-time data, leading to timely decision-making and proactive actions. Figure 8 shows the architecture of Apache Storm.
Figure 8.
Apache Storm architecture.
In addition, Apache Flink has emerged as a leading contender in the field of real-time stream processing. It has low-latency and high-throughput capabilities, making it possible to analyze continuous data streams with a latency of less than one second. By utilizing a pipelined execution paradigm and stateful processing primitives, Flink gives businesses the ability to perform windowed aggregations, event-time processing, and complicated event processing on streaming data. This, in turn, enables organizations to obtain deeper insights and make decisions in real time. Figure 9 shows Real-time stream processing with Apache Flink.
Figure 9.
Real-time stream processing with Apache Flink.
In a wide variety of industries and use cases, real-time data processing is utilized in many applications that are both comprehensive and diverse. In the field of finance, real-time data processing offers opportunities for risk management, the identification of fraudulent activity, and algorithmic trading. These capabilities enable organizations to react rapidly to changes in the market and new hazards. In the retail industry, real-time processing makes it feasible to implement personalized marketing, inventory control, and dynamic pricing. This enables firms to give customers individualized experiences and swiftly capitalize on trends in customer behavior [30]. Real-time data processing is also utilized in the manufacturing, telecommunications, healthcare, and other industries. In these sectors, it assists businesses in enhancing their client experiences, streamlining operations, and fostering innovation via the utilization of data-driven decision-making. When businesses make use of the possibilities offered by real-time data processing in today’s fast-paced and data-driven world, they have the opportunity to capture new opportunities, decrease risks, and gain an advantage over their competitors [31].
2.5. Data Quality and Preprocessing
In the field of data information engineering, data quality is crucial for ensuring the trustworthiness, accuracy, and relevance of analytical findings and decision-making processes. Data quality measures how well data align with their intended purpose, considering factors such as correctness, completeness, consistency, timeliness, and significance. In the realm of big data, characterized by extensive, diverse, and ever-changing datasets, maintaining high data quality is essential for extracting actionable insights and achieving strategic objectives [32].
One cannot stress enough how important data quality is to big data information engineering. The legitimacy and efficacy of data-driven projects can be undermined by incorrect findings, poorly considered actions, and lost opportunities brought about by poor data quality. Biased insights, faulty models, and less-than-ideal results can all originate from insufficient or inaccurate data. Furthermore, duplicate and inconsistent data might bring inefficiencies and mistakes into subsequent procedures, which would reduce operational effectiveness and impede creativity [33].
The term “data preprocessing” refers to the collection of many methodologies and approaches that are utilized by organizations in order to address the challenges that are posed by the level of data quality in big data environments. In the context of data, the term “preprocessing” refers to a collection of techniques that are intended to enhance the usefulness, quality, and relevance of raw data, laying the groundwork for meaningful analysis [34].
Data preprocessing involves a step known as data cleaning, where errors, inconsistencies, and anomalies in the dataset are identified and corrected. This process may include tasks such as eliminating duplicate entries, correcting typos, filling in missing information, and resolving discrepancies in data formats or measurements [35]. By ensuring data cleanliness, companies can reduce the chances of errors and biases in analysis outcomes, ultimately improving the reliability and credibility of the insights obtained from the data. Figure 10 depicts the Data cleansing process.
Figure 10.
Data cleansing process.
An additional key component of data preparation is the process of normalizing data by converting it into a standard format or scale. This is performed in order to facilitate meaningful comparisons and analysis. A few examples of this include the normalization of categorical variables, the modification of skewed distributions in order to establish symmetry, and the scaling of numerical features to a common range. The capacity to assure consistency and comparability between different datasets and features is afforded to organizations that employ the process of normalizing their data representation. The process of identifying outliers, which involves locating and removing data points that significantly deviate from the expected or normal trend, may also be taken into consideration during the process of data preparation [36]. Outliers can be caused by a number of factors, including measurement noise, errors in data collection, or actual anomalies in the phenomenon that is being measured. Identifying and resolving outliers before they have a significant influence on analytical results and conclusions is one way for organizations to improve the accuracy and robustness of the insights that are generated from the data.
In addition, the methods of data quality assurance play a crucial part in assuring the correctness and dependability of analytical outputs in the field of big data information engineering. A wide variety of actions, including data profiling, validation, and monitoring, are included in these processes. The purpose of these activities is to evaluate and maintain the quality of the data during its entire lifespan. Through the implementation of stringent quality assurance methods, companies are able to discover and repair data quality concerns in a proactive manner, hence reducing the likelihood of mistakes and biases in the results of analytical processes.
2.6. Data Lifecycle Management
DLM stands for “data lifecycle management”, which refers to the processes and procedures that are utilized in the management of data from the time they are created until they are finally discarded. The whole lifecycle of data is encompassed by it, beginning with the collection and storage of data and continuing through processing, analysis, and, finally, archiving or destruction. Effective data lifecycle management (DLM) ensures that data are maintained efficiently, safely, and in line with legal requirements throughout their entire duration [37]. This, in turn, maximizes the value of the data and minimizes the associated risks. Figure 11 shows the Data Lifecycle Management.
Figure 11.
Data Lifecycle Management.
Stages of the Data Lifecycle
Acquisition: The data lifecycle kicks off with the acquisition stage, where data are gathered from a variety of sources such as internal systems, external databases, sensors, and IoT devices. This stage involves identifying data sources, deciding on data collection methods, and setting up protocols for data ingestion [38].
Storage: After data are collected, they move to storage. This could be in databases, data lakes, or cloud storage systems. Key decisions here include choosing data formats, structures, and access controls, along with planning for scalability, performance, and redundancy [39].
Processing: Once stored, data need to be processed to turn into useful information. This involves cleaning and enriching the data, aggregating various data points, and normalizing the datasets. This step can involve batch processing, real-time streaming analytics, or interactive querying to draw insights and add value [40].
Analysis: During the analysis stage, the processed data are scrutinized to identify patterns, trends, and correlations. This analysis can inform decision-making and drive business outcomes using statistical methods, machine learning, data mining, and visualization techniques [41].
Archiving: As data become older or less frequently accessed, they move into the archiving stage. Here, data are stored in long-term, cost-effective solutions. This stage involves setting retention policies, managing the data lifecycle, and ensuring that archived data remain intact and accessible for future reference or compliance purposes [42].
Managing data throughout their lifecycle is of the utmost importance for optimizing value and reducing risks associated with data assets. By adhering to best practices in data management and establishing strong data governance structures, companies can ensure compliance, efficiency, and reliability across the data lifecycle. This approach leads to better business outcomes and helps maintain a competitive advantage in today’s data-focused environment.
3. Case Studies Showcasing Successful Implementations of Big Data Projects across Different Industries
3.1. Case Study: GE Healthcare—Predictive Maintenance for Medical Imaging
Challenge: The difficulties that GE Healthcare had with unplanned downtime and the price of maintenance for medical imaging equipment caused major interruptions to patient care and placed a financial burden on operating budgets. MRI (Magnetic Resonance Imaging) machines, CT (Computerized Tomography) scanners, and X-ray machines are examples of the types of medical imaging equipment that are essential assets in healthcare institutions. These instruments play an important part in the diagnostic processes and the treatment plans that are developed for patients. When it comes to providing timely and high-quality care to patients, the dependability and availability of these types of equipment are of the utmost importance.
Unexpected interruptions in the functioning of imaging devices can have severe repercussions. When these machines become unexpectedly unavailable, pre-scheduled patient appointments and procedures may face delays or cancellations, causing patient dissatisfaction, longer waiting periods, and potential negative effects on outcomes. In emergency situations requiring immediate trauma care or urgent diagnostic assessments, the inability to access imaging services promptly can jeopardize patient safety and quality of care.
Apart from the implications for patient care, unplanned downtimes also lead to operational expenses for healthcare providers. The costs associated with maintaining and repairing imaging equipment can be high, particularly when emergency repairs are needed outside of regular maintenance schedules. Additionally, downtime results in missed opportunities for revenue generation, as idle equipment cannot provide services during inactive periods. Prolonged downtimes can strain resources within healthcare facilities and diminish staff productivity, ultimately affecting operational efficiency and workflow management.
GE Healthcare and other manufacturers of medical equipment have begun looking into proactive maintenance programs that make use of the Internet of Things (IoT) and predictive analytics as a reaction to the problems that have been identified. Real-time monitoring of health metrics and equipment performance, which is made feasible by technology that enables predictive maintenance, helps medical professionals to anticipate potential issues before they develop into more significant failures. Through the analysis of data obtained from sensors that are included in imaging equipment, predictive maintenance algorithms are able to identify irregularities, identify new problems, and present alerts for early action intervention [43].
By adopting predictive maintenance solutions, GE Healthcare and healthcare facilities can shift from responding to equipment issues after they occur to anticipating and preventing them. Predictive maintenance helps reduce the chances of downtime by allowing for timely interventions, scheduling preventative maintenance based on equipment conditions and usage trends, and optimizing the management of spare parts inventory. This method not only enhances the reliability and availability of equipment but also cuts down on maintenance expenses and prolongs the lifespan of medical imaging resources.
The difficulties that GE Healthcare has had with unplanned downtime and maintenance expenses highlight the need to utilize new technologies like predictive analytics and the Internet of Things for the purpose of performing preventative equipment maintenance in healthcare organizations. It is possible for healthcare providers to improve the delivery of patient care, maximize operational efficiency, and reduce the financial risks associated with equipment downtime and maintenance charges by using techniques for predictive maintenance. When it comes to the management of medical imaging equipment, predictive maintenance is a game-changing technique that guarantees dependability, availability, and performance while also supporting the aim of providing timely and high-quality patient care.
Strategy: With the help of big data analytics, GE Healthcare put in place a revolutionary predictive maintenance system meant to prevent unexpected downtime for medical devices and actively manage equipment dependability. The goal of this project was to improve operational efficiency in hospital settings by using data integration and sophisticated analytics to anticipate equipment breakdowns before they happened.
The project involved integrating data from diverse sources critical for predictive maintenance, including real-time sensor data collected from medical devices, historical performance metrics, and maintenance logs. The medical dataset of patients contains hundreds of thousands or more data entries. The input dataset is initially preprocessed to improve data quality and reduce processing time [6]. By aggregating and analyzing these multidimensional data, GE Healthcare could gain comprehensive insights into equipment health, identify potential anomalies or patterns indicative of impending failures, and take proactive measures to prevent disruptions in service delivery.
Predictive maintenance at GE Healthcare was based mostly on sophisticated analytics methods, especially machine learning algorithms that could learn from past data to forecast future events with accuracy. These algorithms were trained on massive datasets including a variety of criteria like usage patterns, ambient conditions, sensor readings, and maintenance records. By real-time monitoring and analysis of various data streams, the predictive maintenance system might identify early warning indicators of equipment deterioration or failures, allowing for prompt intervention and preventative measures.
The initiative greatly benefitted from big data integration, as it provided a comprehensive view of equipment performance and health from various perspectives. By merging real-time sensor data with maintenance logs, GE Healthcare obtained a complete understanding of equipment performance and lifespan trends. This holistic perspective enabled decision-makers to focus on maintenance activities, allocate resources efficiently, and manage budgets more effectively using predictive analysis.
The introduction of predictive maintenance at GE Healthcare is a representative example of the revolutionary influence that big data analytics may have on the operations of healthcare facilities. Through the utilization of data-driven insights, healthcare providers are able to transition from reactive to proactive maintenance strategies, decrease the operational costs that are associated with unplanned downtime and emergency repairs, and ultimately improve the outcomes of patient care by ensuring the availability and dependability of essential medical equipment.
It is also important to note that this initiative highlights the larger ramifications that might result from the integration of advanced analytics and data into healthcare settings. Through the adoption of digital transformation and the utilization of big data, organizations such as GE Healthcare have the ability to foster innovation, maximize the utilization of resources, and open the door to healthcare delivery models that are more intelligent and efficient. The term “predictive maintenance” refers to an expenditure that is estimated in order to use data assets to produce insights, promote operational excellence, and eventually revolutionize the supply and administration of healthcare services.
Outcome: By proactively detecting maintenance requirements and scheduling interventions based on predictive insights, GE Healthcare was able to cut unexpected downtime by 20% and maintenance expenditures by 10%. This led to an increase in the dependability of the equipment, an improvement in the continuity of patient care, and an optimization of the allocation of resources [44]. Figure 12 shows GE healthcare’s platform.
Figure 12.
GE healthcare platform.
3.2. Case Study: Capital One—Personalized Customer Insights for Financial Services Industry
Challenge: The effort of Capital One to improve client engagement and retention through customized banking experiences is a strategic focus on using data-driven insights and sophisticated analytics to provide customized services and solutions to each customer. Customer loyalty, contentment, and eventually keeping important clients are all greatly influenced by customer-centricity and individualized experiences in the cutthroat financial services industry of today.
In order to accomplish this goal, Capital One unquestionably initiated a comprehensive data-driven strategy, which included the use of transaction histories, demographic data, online activities, and behavioral patterns, amongst other sources of customer information. Through the use of big data analytics, machine learning, and artificial intelligence technologies, Capital One endeavored to obtain a comprehensive understanding of the specific requirements, financial behaviors, and preferences of each and every customer [45].
In this phase, Capital One would have started by combining and organizing various datasets to create a unified overview of each customer’s information. This comprehensive data system likely included both structured and unstructured data sources, such as transaction records from banking activities, social media interactions, customer service logs, and external market data. By establishing a robust data management foundation, Capital One could access detailed, real-time insights about customers. By taking advantage of this wealth of information, Capital One could then use analytical methods to derive practical insights and identify tailored banking opportunities for individual customers. By utilizing machine learning algorithms, they could group customers based on their habits, forecast individual preferences, and anticipate future requirements. This would have enabled Capital One to offer targeted promotions, personalized product suggestions, and bespoke services that aligned with each customer’s unique financial objectives and preferences.
The integration of customized banking services extended across various points of contact, including digital platforms (such as mobile apps and online banking sites), customer service interactions, and marketing messages. By tailoring engagements based on individual preferences and behaviors, Capital One aimed to provide smooth and captivating experiences that enhance customer satisfaction and foster lasting connections. Moreover, Capital One’s approach to tailored banking services highlights a trend toward prioritizing customers in the financial sector. By using data-driven tactics and focusing on customer-centric strategies, companies like Capital One can stand out in a competitive market, cultivate customer loyalty, and ultimately achieve sustainable growth.
The initiative taken by Capital One to improve client engagement and retention through the provision of tailored banking experiences is a prime example of the revolutionary impact that data analytics and customer-centric initiatives provide in the financial services industry. Capital One’s goal is to establish long-lasting connections with its customers, foster customer loyalty, and establish itself as a leader in the provision of innovative and personalized banking products. This will be accomplished by utilizing data-driven insights to comprehend, anticipate, and meet the requirements of individuals.
Strategy: Capital One’s implementation of a data analysis system reflects a strategic effort to use data-driven insights effectively to improve customer interactions and provide personalized services in the financial sector. By utilizing data analytics, Capital One aimed to study extensive customer transaction records, spending habits, and engagements across different platforms to gain valuable insights and offer customized solutions to individual clients. The project encompassed various aspects of data analysis and artificial intelligence, starting with data consolidation. Capital One merged diverse datasets from multiple sources, such as transaction logs, client profiles, demographic details, online engagements, and external market statistics. By centralizing this information into a single platform, Capital One established a holistic understanding of each customer’s financial activities, preferences, and requirements.
During the course of the project, the process of feature engineering was extremely important. This involved the identification of pertinent features and characteristics from the integrated dataset by data scientists and analysts in order to construct predictive models. For the purpose of capturing relevant patterns and insights, features such as transaction frequency, expenditure categories, geographic location, and customer segmentation were developed. This procedure included the transformation of data, the normalization of data, and the enrichment of data in order to prepare the dataset for the construction of machine learning models.
One of the most important aspects of Capital One’s analytics platform was the creation of machine learning models, which made it possible to provide tailored product suggestions and targeted marketing offers. For the purpose of analyzing client behavior and preferences, data scientists utilized sophisticated algorithms such as collaborative filtering, clustering, and recommendation engines. The purpose of these models is to find chances for cross-selling or upselling related financial products and services by predicting future purchase patterns and learning from previous data.
For Capital One, the utilization of this big data analytics platform makes it feasible for the company to provide individualized customer experiences on a large scale. Through the utilization of machine learning-driven insights, Capital One has the ability to personalize marketing materials, special offers, and product ideas in accordance with the distinguishing traits and behaviors of each individual customer. This tailored method increases relevance, improves consumer participation, and ultimately encourages customer loyalty and happiness from the perspective of the customer.
Capital One’s dedication to utilizing data analytics highlights a strong focus on innovation and customer satisfaction within the financial industry. By adopting data-driven technologies and analytical tools, companies like Capital One can discover avenues for expansion, differentiate themselves in the market, and foster stronger connections with customers by understanding and meeting their evolving financial needs. The implementation of a data analytics system by Capital One serves as a prime example of how data-driven approaches can revolutionize personalized customer interactions and improve business outcomes. Through the application of analytics and machine learning methods, Capital One demonstrates how businesses can leverage data to offer enhanced services, boost customer engagement, and gain a competitive edge in today’s fast-paced and data-centric environment.
Outcome: Capital One was able to achieve a 15% boost in customer satisfaction and a 20% improvement in the efficacy of cross-selling by utilizing big data analytics [46]. A better level of consumer involvement, enhanced loyalty, and improved business performance were all outcomes that resulted from the individualized insights. Figure 13 shows the Capital One’s big data analytics platform.
Figure 13.
Capital One’s big data analytics platform.
3.3. Case Study: Walmart—Supply Chain Optimization with Real-Time Analytics
Challenge: Walmart’s initiative to improve inventory management, prevent stockouts, and streamline their supply chain demonstrates their commitment to leveraging data-driven insights and advanced technology to meet the evolving needs of customers in a fast-paced retail environment. As one of the largest retailers globally, Walmart faces numerous challenges in managing inventory, ensuring an efficient supply chain, and maintaining customer satisfaction. By implementing innovative solutions based on data analysis and technology, Walmart aimed to boost operational efficiency, enhance product availability, and provide top-notch customer service.
Walmart most likely utilized a mix of big data analytics, machine learning, and Internet of Things (IoT) technology in order to accomplish these goals. This would have allowed the company to obtain real-time visibility into inventory levels, demand trends, and supply chain dynamics. Creating a complete and accurate view of inventory across the retail network would have been the goal of the project, which would have entailed the integration of data from a variety of sources, such as point-of-sale (POS) systems, inventory databases, vendor systems, and external market data.
Walmart relies on demand forecasting as a key component of its inventory management strategy. By analyzing sales data, seasonal patterns, customer preferences, and external factors such as weather and promotions, Walmart can predict future demand more accurately. Leveraging forecasting models powered by machine learning algorithms allows Walmart to anticipate customer needs, adjust inventory levels accordingly, and prevent stock shortages while enhancing inventory turnover efficiency. One other aspect Walmart is interested in is optimizing its supply chain. The company utilizes logistics and routing algorithms to streamline transportation routes, reduce delivery times, and lower operational expenses. Through real-time tracking of inventory and shipments using IoT devices, Walmart can closely monitor inventory movements, identify supply chain bottlenecks promptly, and proactively resolve any disruptions in the supply chain flow.
In addition, it is quite probable that Walmart makes use of data analytics in order to successfully execute dynamic pricing strategies and promotional campaigns that are based on demand patterns and the price of competitors. Walmart is able to alter its pricing and promotional plans in order to optimize income, boost sales, and improve customer happiness by conducting real-time analysis of customer behavior and the dynamics of the market. Walmart’s dedication to simplifying supply chain processes also includes environmental programs like waste reduction and resource efficiency enhancement. Aligning with Walmart’s larger sustainability objectives, data analytics are essential in maximizing inventory levels to decrease food waste and prevent overstocking.
Walmart’s efforts to optimize inventory management, reduce stockouts, and streamline supply chain operations exemplify the transformative impact of data analytics and technology in the retail industry. By leveraging data-driven insights to make informed decisions, Walmart can enhance operational efficiency, improve product availability, and deliver exceptional customer experiences. This customer-centric approach underscores Walmart’s commitment to innovation and continuous improvement in meeting the diverse and dynamic demands of modern retail consumers.
Strategy: The adoption of a real-time big data analytics platform by Walmart, with the goal of optimizing inventory management and demand forecasting, exemplifies the retailer’s dedication to utilizing data-driven insights in order to improve both operational efficiency and consumer pleasure. Walmart’s goal was to obtain real-time visibility into sales patterns, market dynamics, and supply chain performance by utilizing advanced analytics techniques and combining data from a wide variety of internal and external sources. This allows Walmart to make proactive choices and manage inventory levels.
The project involved combining data from various sources, such as internal sales figures, inventory records, weather trends, supplier performance metrics, and market data. Walmart standardized these diverse datasets into a single analytics platform to create a comprehensive data environment for thoroughly analyzing inventory management and supply chain operations. A significant aspect of Walmart’s efforts focused on demand forecasting using analytics. By examining sales data, seasonal patterns, promotions, and external factors like weather conditions, Walmart developed advanced predictive models to predict future demand accurately. These models helped Walmart anticipate changes in customer demand, optimize inventory levels effectively, and reduce the risk of shortages or excess inventory.
Walmart also quite likely utilized real-time data analytics in order to monitor and respond to crises involving customers and the market promptly. Walmart can adjust its inventory levels, enhance its product variety, and fine-tune its pricing tactics in order to better satisfy the preferences of its customers and optimize its profitability. This is accomplished by continuously monitoring sales patterns and supply chain performance indicators in real time.
The implementation of a real-time big data analytics platform also facilitated improved collaboration with suppliers and partners. Walmart can share actionable insights and performance metrics with suppliers, enabling collaborative demand planning and inventory management. This collaborative approach enhanced supply chain visibility, reduced lead times, and strengthened relationships with key stakeholders across the supply chain ecosystem [7]. Walmart’s use of sophisticated analytics for inventory management also mirrors a larger retail sector movement toward operational excellence and data-driven decision-making. Big data analytics may be used by Walmart to improve inventory management procedures, cut operating expenses, and eventually provide better customer experiences by guaranteeing product availability and prompt order fulfillment.
Walmart’s use of a real-time big data analytics platform for the purpose of inventory management and demand forecasting exemplifies the revolutionary nature of data analytics in terms of its ability to enhance the operational efficiency and competitive advantage of commercial retail businesses. Through the utilization of data-driven insights to optimize inventory levels and supply chain procedures, Walmart has the potential to enhance the satisfaction of its customers, boost its profitability, and position itself as a pioneer in the field of data-derived retail innovation. Walmart is demonstrating its commitment to using technology and analytics in order to meet the ever-evolving expectations of modern customers and to give exceptional value across all of its retail operations through this endeavor.
Outcome: Through the utilization of real-time analytics, Walmart was able to achieve a decrease of 10% in the number of out-of-stock situations and an improvement of 15% in the turnover of inventory. As a consequence of the supply chain being streamlined, operational efficiency was increased, carrying costs were decreased, and customer satisfaction was increased [47]. Figure 14 shows Walmart’s online marketing platform architecture.
Figure 14.
Walmart’s online marketing platform architecture.
3.4. Tesla—Quality Control and Production Optimization in the Automotive Sector
Challenge: Tesla’s goal of improving product quality, minimizing flaws, and streamlining production methods demonstrates an emphasis on utilizing data analysis, automation, and continuous enhancement techniques to meet the growing market demand for electric vehicles (EVs) while upholding high standards of excellence and efficiency. As a pioneer in the automotive sector, Tesla encounters distinct challenges related to expanding production capacity, ensuring reliability, and providing top-tier products that align with customer preferences.
Tesla most likely used a multidimensional strategy that incorporated data-driven insights and sophisticated technologies across the whole product lifecycle, beginning with the design and manufacturing stages and continuing through quality assurance and customer delivery. This was performed so that Tesla could fulfill these aims. The purpose of this effort was to optimize processes, discover problems at an early stage in the production cycle, and drive initiatives for continuous improvement by utilizing real-time data analytics, machine learning, and automation.
Tesla’s strategy focuses on using data to ensure quality control and detect defects. Tesla relies on data analysis and machine learning to examine sensor data, production records, and quality measures in real time. By monitoring key performance indicators (KPIs) and identifying deviations from the norm, Tesla can proactively detect and address potential issues or irregularities during production, reducing rework and enhancing overall product quality. In addition, Tesla likely utilizes predictive maintenance techniques to optimize equipment performance and minimize downtime at their manufacturing plants. Using sensors and predictive analytics, Tesla can monitor the condition of their production machinery, forecast maintenance requirements, and plan maintenance tasks in advance to prevent unexpected interruptions in operations, thereby maintaining peak production efficiency.
Tesla’s dedication to ongoing development also encompasses automation and process efficiency. Using data analytics, Tesla can find possibilities to streamline production processes as well as bottlenecks and inefficiencies. Tesla can scale production volumes, lower cycle times, and improve operational efficiency by putting automation technologies, robots, and AI-driven solutions into place to satisfy the expanding market demand for EVs without sacrificing quality or safety.
Tesla’s approach to improving product quality and optimizing production processes exemplifies the transformative impact of data analytics and technology in the automotive industry. By leveraging data-driven insights to drive operational excellence and innovation, Tesla can deliver high-quality electric vehicles that exceed customer expectations, differentiate itself in a competitive market landscape, and drive sustainable growth. The emphasis Tesla places on improving product quality, lowering defects, and automating and data analytics manufacturing processes emphasizes how crucial it is to use cutting-edge technology to promote ongoing development and operational effectiveness in the automotive sector. Tesla is positioned to address the changing needs of customers for high-performance, environmentally friendly electric cars while preserving its reputation for quality and innovation in the automotive industry by using the potential of data-driven insights and technology-driven innovation.
Strategy: Tesla’s use of data analysis and machine learning in manufacturing is a key strategy to improve efficiency, reduce defects, and enhance production quality. By utilizing real-time data analytics and proactive maintenance techniques, Tesla aims to streamline manufacturing processes, boost product quality, and meet the rising demand for vehicles while upholding high standards of performance and reliability.
Tesla’s project involved integrating data from various sources in its production lines, such as sensor readings, production logs, quality control metrics, and equipment performance data. By consolidating and aligning these datasets within a single analytics platform, Tesla established a comprehensive data environment that enabled continuous monitoring and analysis of manufacturing operations in real time. A crucial aspect of Tesla’s efforts was utilizing machine learning algorithms to examine manufacturing data and identify the root causes of defects. Through the use of analytical methods, Tesla could detect patterns, anomalies, and deviations from expected performance metrics. Historical data were used to train machine learning models to anticipate defects, fine-tune process parameters, and suggest corrective measures to prevent quality issues and reduce production interruptions.
In addition, Tesla has incorporated predictive maintenance techniques to enhance the performance of its equipment and reduce downtime at its manufacturing plants. By using sensors and predictive analysis, Tesla can monitor the status and condition of production machinery in real time. Maintenance algorithms analyze data from equipment sensors to identify early signs of wear and predict potential malfunctions. This approach allows for the scheduling of maintenance tasks to avoid unexpected downtime and ensure efficient production operations.
The integration of data analysis and artificial intelligence into Tesla’s manufacturing processes underscores the importance of data-driven decision-making and the adoption of technological advancements in the automotive sector. By utilizing real-time data analysis, Tesla can continuously refine production methods, enhance product quality, and achieve operational excellence to meet the rapidly changing demands of the electric vehicle market.
Tesla’s deployment of big data analytics and machine learning for manufacturing optimization exemplifies the transformative impact of data-driven technologies on driving efficiency, quality, and innovation within automotive manufacturing. By leveraging advanced analytics and predictive maintenance strategies, Tesla is able to maintain a competitive edge, deliver high-quality electric vehicles, and uphold its commitment to sustainability and innovation in the automotive sector. This initiative demonstrates Tesla’s strategic vision and commitment to leveraging technology to redefine manufacturing practices and deliver superior products to customers worldwide.
Outcome: By leveraging big data for quality control and process optimization, Tesla achieved a 20% reduction in manufacturing defects and a 30% improvement in production efficiency. The data-driven insights enabled Tesla to deliver high-quality vehicles at scale and maintain a competitive edge in the automotive market [48]. Figure 15 shows the Predictive maintenance process in the automotive sector.
Figure 15.
Predictive maintenance process in the automotive sector.
4. Discussion
4.1. Challenges and Future Research Directions
4.1.1. Edge Computing
In the realm of data processing and analytics, edge computing is a revolutionary trend that shifts the focus away from centralized cloud servers and toward distributed computing resources. These resources include Internet of Things devices, sensors, and edge servers, all of which are located closer to the source of the data [49]. This paradigm shift is being driven by the demand to minimize latency, reduce bandwidth usage, and reduce dependency on traditional cloud infrastructure. This is especially true in circumstances when real-time responsiveness and low-latency applications are absolutely necessary. By utilizing their proximity to data sources, businesses are able to process and analyze massive volumes of data that are generated by Internet of Things devices and sensors that are located at the network edge. This is made possible by edge computing [50].
Edge computing advances real-time analytics and decentralized decision-making by allowing data processing to occur closer to the source. This enables data analysis and decision-making without the need to transmit large amounts of data to centralized data centers. It is particularly useful in applications requiring immediate responses based on real-time data, such as autonomous vehicles, industrial automation, and remote healthcare monitoring. Additionally, edge computing enhances context-aware applications by utilizing local data and environmental signals to provide personalized and adaptive user experiences [51].
Edge computing provides increased data privacy and security, which is another key breakthrough given by this technology. When enterprises handle sensitive data locally at the edge, they are able to reduce the exposure of data during transmission and storage, hence minimizing the risk of data breaches and unauthorized access. Edge computing makes it possible to secure sensitive information while still adhering to data protection requirements by enabling approaches such as data encryption, access control, and privacy-preserving methods. In addition, edge computing has the potential to improve resilience against cyber attacks by lowering the attack surface and making it possible for distributed edge systems to respond quickly to incidents [52].
Despite the fact that it has a number of benefits, edge computing also has a number of obstacles that need to be solved in order to achieve widespread acceptance and scalability. When dealing with dispersed edge settings, which are characterized by different and heterogeneous devices with varying capabilities and connections, one of the challenges that must be overcome is guaranteed dependability and consistency. It is of the utmost importance to effectively manage and orchestrate edge resources in order to guarantee consistent performance and dependability throughout a distributed infrastructure. Furthermore, in order to address security problems that are associated with the transmission and storage of data at the edge, it is necessary to implement effective encryption, authentication, and security protocols in order to safeguard the integrity and confidentiality of data in settings that are dynamic at the edge [53]. Table 3 shows the key aspects of edge computing, including trends, advancements, and challenges.
Table 3.
Key aspects of edge computing, including trends, advancements, and challenges.
| Aspect | Description |
|---|---|
| Trend: Edge Computing | Edge computing is an emerging paradigm shifting data processing and analytics closer to data sources (IoT devices, sensors, edge servers) rather than relying solely on centralized cloud servers. This reduces latency, bandwidth usage, and dependency on cloud infrastructure. |
| Advancements |
|
| Challenges |
|
4.1.2. Federated Learning
Federated learning is a new approach in machine learning that enables decentralized model training across various devices or edge nodes. This eliminates the need to collect sensitive data into centralized repositories, which is a common practice in traditional machine learning platforms [54]. Each participating device or node trains a local machine learning model using its own data in this manner. Only model changes (such as gradients) are exchanged with a central server or coordinator. This strategy is referred to as the “local machine learning algorithm”. The decentralized training paradigm enables companies to make use of dispersed data sources for the purpose of model training, while also limiting privacy problems that are connected with the centralized aggregation of user data [55].
Federated learning has made significant strides in privacy-focused machine learning. By training models directly on devices and keeping data local, this approach minimizes the risk of sensitive data exposure to central servers. This not only addresses privacy concerns but also ensures compliance with data protection laws. It allows organizations to utilize datasets spread across various devices or locations without compromising individual user privacy, making it ideal for sectors like healthcare, finance, and IoT [56]. Furthermore, federated learning enables the creation of personalized AI models tailored to specific users or devices. By training models on local data, personalized recommendations, predictions, and adjustments can be made based on user preferences and behaviors. This functionality enhances user interaction and experience while upholding data privacy and security standards.
While federated learning offers numerous benefits, it also presents several challenges that must be addressed to ensure smooth implementation and scalability. One major challenge is the communication among devices during model training and aggregation, especially in situations with limited bandwidth or unreliable network connections. Another key hurdle is maintaining model convergence and consistency across devices with varying computational capabilities and data distributions [57].
To tackle these issues, techniques such as adaptive learning rates, ensuring differential privacy, and employing secure aggregation methods are used to facilitate robust model training in federated learning setups [58]. Additionally, addressing concerns related to data distribution and bias among data sources is crucial for the success of federated learning. It is essential to have representative training data from all participating devices to create accurate and unbiased AI models. Strategies such as data sampling, augmentation, and model averaging play a significant role in managing discrepancies in data distribution and enhancing overall model performance within federated learning environments [59]. Table 4 shows the key aspects of federated learning, including trends, advancements, and challenges.
Table 4.
Key aspects of federated learning, including trends, advancements, and challenges.
| Aspect | Description |
|---|---|
| Trend: Federated Learning | Federated learning is a decentralized machine learning approach where multiple devices or edge nodes collaboratively train a shared model, without centrally aggregating data. Each device trains its local model using local data and only model updates are shared with a central server. |
| Advancements |
|
| Challenges |
|
4.1.3. Explainable AI (XAI)
Explainable artificial intelligence, often known as XAI, is a prominent movement in the field of artificial intelligence. Its primary objective is to improve the interpretability and transparency of machine learning models. XAI approaches are designed to give explanations that are both obvious and intelligible for the judgments and predictions that are produced by artificial intelligence systems. This will ultimately improve the trustworthiness, accountability, and acceptance of AI technology across a variety of areas. The rising demand to simplify complicated machine learning algorithms and provide people with the ability to grasp and confirm the logic behind judgments powered by artificial intelligence is the driving force behind this development [60].
One of the main developments made possible by XAI approaches is the creation of methodologies for local interpretability, model-agnostic explanations, and feature significance analysis. The analysis of feature significance helps determine which input characteristics or variables have the biggest impact on model predictions, thus illuminating the fundamental principles behind AI judgments. Model behavior may be understood by users without a particular understanding of the internal architecture of a machine learning model thanks to model-agnostic explanations, which concentrate on methods applicable to any machine learning model. Local interpretability techniques give end users clearer and more understandable explanations at the level of individual forecasts, hence improving AI judgments [61].
Explainable AI (XAI) plays a crucial role in various fields, such as ensuring regulatory compliance, assessing risks, and enhancing collaboration between humans and AI. In heavily regulated industries like finance and healthcare, XAI techniques allow auditors and regulators to verify model decisions and ensure they meet legal requirements [62]. Additionally, XAI aids in risk assessment by providing insights into the factors influencing model predictions, helping organizations identify biases or errors in AI systems. Furthermore, explainable AI promotes collaboration between humans and AI technologies by empowering users to trust and engage with AI systems effectively [63].
Despite significant progress, XAI still faces hurdles that must be overcome to unlock its potential and scalability fully. One key challenge involves balancing model complexity and interpretability. For instance, more intricate models like deep learning algorithms might prioritize performance over transparency.
Deep learning, a subset of machine learning, involves neural networks with many layers (deep networks) that excel in analyzing large datasets. The success of deep learning algorithms is highly dependent on the availability of big data, as these models require substantial amounts of data for training in order to achieve high accuracy and performance. Big data provides the diverse and extensive datasets needed to train deep learning models effectively [64].
Furthermore, big data analytics tools help preprocess and manage the data required for deep learning, ensuring that the data fed into the models are clean, consistent, and relevant. This preprocessing includes tasks such as data normalization, augmentation, and transformation, which are critical for enhancing the performance of deep learning models [65].
Ongoing research focuses on developing XAI methods that can elucidate the decisions made by these complex algorithms. Moreover, it is important to ensure that the explanations provided by XAI techniques are not only meaningful and understandable but also actionable for end users. This is vital for building trust and acceptance of AI-powered solutions [66]. Table 5 shows the key aspects of Explainable AI (XAI), including trends, advancements, and challenges.
Table 5.
Key aspects of Explainable AI (XAI), including trends, advancements, and challenges.
| Aspect | Description |
|---|---|
| Trend: Explainable AI | Explainable AI focuses on developing machine learning models that provide transparent explanations for their decisions and predictions. XAI techniques aim to enhance the interpretability, trustworthiness, and accountability of AI systems. |
| Advancements |
|
| Challenges |
|
4.1.4. Large Models in Big Data
Large models, such as ChatGPT and Sora, have fundamentally transformed the landscape of big data processing and analysis. These advanced neural networks leverage vast datasets to learn and generate valuable insights, making them indispensable tools in various applications. For instance, ChatGPT has found extensive use in natural language processing tasks, including text generation, translation, and summarization [67]. Similarly, Sora excels in analyzing and extracting patterns from massive datasets, making it highly effective for large-scale data tasks [68].
The deployment of these models has demonstrated significant benefits. One of the primary advantages is their ability to process and analyze unstructured data, such as text and images, which constitutes a considerable portion of big data. This capability enhances the comprehensiveness of data analysis, enabling the extraction of more complex insights. Furthermore, these models support improved decision-making through advanced predictive analytics and pattern recognition, which can identify trends and forecast outcomes with high accuracy. Additionally, the automation of complex tasks, such as customer service, content creation, and data extraction, leads to increased operational efficiency, allowing organizations to focus on more strategic activities [69].
Despite the considerable benefits, integrating large models into big data applications does not come by without issues that need to be addressed. One of the most significant obstacles is the requirement for substantial computational resources. Training and deploying large models demand immense computational power and memory, often leading to high costs that can be prohibitive for many organizations. Scalability also poses a considerable challenge. As datasets continue to grow, maintaining the performance of these models without degradation becomes increasingly difficult. Furthermore, ensuring data privacy and security is paramount, especially with stringent regulations such as the General Data Protection Regulation (GDPR). Protecting sensitive data during the training and inference processes is of great importance if we are to prevent breaches and unauthorized access [70].
The integration of large models with existing big data frameworks and infrastructures [71] is another complex challenge. This process often requires significant modifications to current systems, which can be resource-intensive and time-consuming. Additionally, the interpretability of these models remains a pressing issue. Understanding the decision-making process of large models is crucial for building trust and ensuring ethical use, yet it remains a challenging task because of the inherent complexity of these models [72].
To address these challenges and enhance the effectiveness of large models in big data applications, several future research directions are proposed. One key area is an improvement in model efficiency [73]. Developing more efficient algorithms and hardware can significantly reduce the computational and energy costs associated with training large models. Another vital area is the creation of scalable architectures. These architectures need to handle increasing data volumes seamlessly while maintaining performance, ensuring that the models can grow alongside the datasets they are designed to analyze.
Advanced privacy techniques, such as federated learning and differential privacy, are essential for protecting sensitive data while leveraging the power of large models [74]. These techniques allow models to learn from data without the need to centralize them, thereby enhancing privacy and security. Furthermore, hybrid models that combine the strengths of large models with traditional big data processing frameworks can offer a balanced approach, leveraging the best of both worlds.
Lastly, improving model interpretability is also an important research direction. Developing methods to make the inner workings of large models more transparent and understandable will enhance trust and usability, facilitating their broader adoption in the business and the academic world [75]. By focusing on these areas, future advancements can ensure that large models continue to play a pivotal role in extracting value from big data while effectively addressing the associated challenges.
4.2. Challenges and Potential Solutions in Big Data Management
When it comes to contemporary data management and analytics, one of the most critical issues is the management of data sources that are becoming increasingly complicated and diverse. As businesses amass varied datasets from a variety of sources, such as structured and unstructured data, Internet of Things devices, social media platforms, and enterprise systems, they are confronted with issues relating to the integration of data, quality assurance, governance, and scalability [76].
One of the biggest obstacles is data integration, which calls for the harmonization and consolidation of many datasets from many sources to produce a single perspective for analysis. Problems with interoperability, data silos, and inconsistencies arise from the complexity of the integration process brought on by the range of data formats, schemas, and storage systems. As data sources and quantities increase, conventional integration techniques become insufficient, and scalable and adaptable solutions to manage various data kinds and formats are needed [14]. Making sure data are consistent and of high quality from several sources is another problem. The reliability and efficacy of data-driven analytics and decision-making can be negatively impacted by data quality problems like missing values, duplication, and errors. Organizations that want to guarantee correct and trustworthy insights from heterogeneous datasets must put in place strong data quality assurance procedures, such as data cleansing, standardization, and validation, as data complexity grows [77].
Data governance also presents issues when it comes to managing data sources that are both complex and varied. In heterogeneous settings with distant data sources, the process of establishing and implementing policies for data governance, metadata management, and access restrictions becomes increasingly complicated. An additional degree of complexity is added to data governance processes by the necessity of ensuring compliance with legal requirements, privacy rules, and data security standards [78].
The capacity to scale is a significant obstacle to overcome when working with big amounts of complicated data originating from a variety of sources [79]. There is a possibility that traditional data management systems will have difficulty meeting the scalability requirements of big data analytics, real-time processing, and distributed computing. For the purpose of accommodating the increasing volume, velocity, and diversity of data sources, it is vital to use scalable architectures, cloud-based solutions, and distributed processing frameworks [80].
To address these challenges, organizations can leverage several potential solutions including the following:
Advanced Data Integration Tools: Utilizing up-to-date data integration tools and platforms is crucial for companies handling a mix of data sources and intricate data setups. These tools are specifically designed to tackle the complexities of merging datasets that come in different formats, have diverse schemas, and are stored in various systems. A notable feature of data integration tools is their support for schema-on-read, which offers flexibility in interpreting and processing data. Unlike schema-on-write methods that require predetermined schemas before storing data, schema-on-read allows for dynamic interpretation of data during query execution, accommodating a wide range of evolving data structures.
Data virtualization is yet another essential element that contemporary data integration systems make available to their users. Through the use of data virtualization, it is possible to have access to data from a variety of sources in real time without having to physically move or duplicate the data. Through the utilization of this strategy, companies are able to generate a unified, virtual picture of data that encompasses numerous systems, applications, and databases, hence allowing the smooth access and integration of data. Through the reduction in data movement latency and storage costs, data virtualization enables agile data provisioning and speeds up the creation of applications that are driven by data.
Moreover, contemporary data integration solutions often include features for data federation, which facilitate the coordination of data from various sources and environments. Data federation enables organizations to gather and analyze data in real time from diverse systems such as local databases, cloud services, IoT devices, and external data repositories. This functionality is crucial for establishing a cohesive data infrastructure and supporting complex analytical scenarios that require information from multiple origins. In addition to streamlining data integration processes, modern tools support data manipulation and coordination, empowering organizations to adapt swiftly to evolving business needs and data requests. Real-time data processing capabilities also bolster streaming analytics, event-triggered architectures, and dynamic data workflows—enabling organizations to derive insights and make informed decisions almost instantly.
Companies can solve the obstacles that are encountered in heterogeneous data sources by utilizing current data integration tools and platforms that enable schema-on-read, data virtualization, and data federation. This allows enterprises to gain better agility, flexibility, and efficiency in data management and analytics. Unlocking the full potential of their unique data assets for business innovation and competitive advantage is made possible by these technologies, which enable businesses to adapt to changing data landscapes, embrace decision-making that is driven by data, and embrace data-driven decision-making.
Data Quality Management Practices: It has become imperative for organizations to prioritize data quality management practices to maintain the accuracy, reliability, and consistency of their data assets. Data quality management involves processes and methods aimed at evaluating and enhancing data quality throughout the data lifecycle. One important aspect is data profiling, which entails analyzing datasets to understand their structure, content, and quality attributes. Effective data governance plays a critical role in ensuring compliance with regulations, bolstering data security and integrity, building trust in data-driven decisions, and promoting organizational responsibility and transparency. Organizations must invest in data governance practices to optimize the value of their data assets while minimizing risks related to data misuse, security breaches, and regulatory non-compliance.
Scalable Infrastructure and Technologies: Organizations addressing the difficulties of effectively managing large-scale data processing and analytics must implement scalable infrastructure solutions. Often, the volume, velocity, and variety of data produced from many sources are too much for traditional on-premises infrastructure to handle. Big data processing frameworks like Hadoop and Spark, as well as cloud computing and distributed databases, provide the resources and flexibility needed to efficiently handle and analyze enormous amounts of information.
Providing on-demand access to computer resources, storage, and services over the internet, cloud computing is a game-changing technology that has the potential to revolutionize several industries. Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform are examples of cloud platforms that provide scalable infrastructure components. These components include compute instances, storage solutions, and data analytics capabilities. By utilizing cloud computing, organizations are able to expand their resources dynamically in response to the needs of their workload. This allows for more cost-effective data processing and analytics rather than being constrained by the restrictions of on-premises technology.
Distributed databases utilize a multitude of nodes or servers for storage and processing in order to manage massive amounts of data. As data volumes increase, distributed database systems such as Apache Cassandra, MongoDB, and Amazon DynamoDB enable organizations to scale horizontally by adding nodes to the database cluster. By virtue of their fault tolerance, high availability, and efficient data retrieval capabilities, distributed databases are ideally adapted for scalable data storage and management.
Processing large amounts of data is made easier with tools like Apache Hadoop and Apache Spark. These platforms use distributed computing to analyze datasets by running tasks simultaneously across clusters of standard hardware. This approach allows for the execution of data operations, machine learning algorithms, and real-time analytics. By utilizing distributed computing, companies can speed up data processing and manage various workloads effectively.
Organizations may enhance resource usage and facilitate scalability by utilizing serverless architectures and containerization, in addition to cloud computing and distributed databases. Serverless computing systems, such as AWS Lambda and Azure Functions, allow enterprises to execute code without the need to allocate or oversee servers. These platforms automatically adjust their capacity to match workload requirements. Containerization technologies like Docker and Kubernetes offer lightweight and portable environments for delivering and managing applications. They enable the effective allocation of resources and scalability across dispersed infrastructure.
Large-scale data processing and analytics encounter scalability issues that enterprises can address by implementing scalable infrastructure solutions like cloud computing, distributed databases, big data processing frameworks, serverless architectures, and containerization. These technologies let businesses support flexible, economic data-driven projects, optimize resource use, and expand resources dynamically. Putting money into scalable infrastructure sets the stage for using data as a strategic asset and promoting innovation in the data-heavy corporate environment of today.
5. Conclusions
The field of data information engineering is a diverse and evolving area that addresses the complexities and opportunities of handling, processing, and analyzing large volumes of data. It encompasses methods, technologies, and strategies designed to manage the challenges of today’s data environments. By applying the elements and concepts discussed in this document, companies can effectively utilize big data to achieve significant business results, stay competitive in the market, and foster innovation across various sectors.
Data collection and storage that is both effective and efficient is one of the fundamental foundations that support big data information engineering. Distributed file systems, such as the Hadoop Distributed File System (HDFS), NoSQL databases, such as MongoDB and Cassandra, and scalable cloud storage solutions, such as Amazon S3 and Google Cloud Storage, are examples of the sophisticated technologies that are utilized by organizations. Organizations are able to ingest, store, and manage vast amounts of structured and unstructured data from a variety of sources, such as sensors, Internet of Things devices, social media platforms, and corporate applications, with the assistance of these technologies.
Data processing and analysis are essential components of big data information engineering. Apache Hadoop, Apache Spark, and Apache Flink represent essential technologies for managing extensive data processing activities, including batch processing, real-time streaming analytics, and intricate data transformations. Machine learning algorithms, statistical analysis, and data mining techniques are utilized to extract important insights, detect trends, and generate educated predictions from large datasets. This enables enterprises to obtain actionable knowledge from their data assets.
In data information engineering, an important aspect is the integration and management of data. Companies aim to unify datasets from various sources to create a cohesive view of their data. This process includes cleaning data, transforming it, and integrating schemas to ensure data accuracy and consistency. Effective governance structures are established to uphold data policies, adhere to regulations, and safeguard data security and privacy throughout the data lifecycle.
It is also important to note that big data information engineering places an emphasis on the significance of data visualization and interpretation. In order to convey complicated data insights in a manner that is visually intelligible, data visualization techniques like charts, graphs, and dashboards are utilized. This helps to facilitate effective decision-making and the sharing of insights among stakeholders. Additionally, approaches such as explainable artificial intelligence (XAI) are utilized in order to improve the interpretability and transparency of machine learning models. This makes it possible for stakeholders to comprehend the reasoning behind decisions that are driven by AI.
As the volume and complexity of data continue to grow exponentially, the role of big data information engineering becomes increasingly important in enabling data-driven decision-making and insights-driven strategies. Organizations that invest in robust big data practices and technologies are better positioned to leverage their data assets for strategic decision-making, innovation, and sustainable competitive advantage in today’s data-centric business environment. The evolution of big data information engineering will continue to shape the future of industries, empowering organizations to unlock new opportunities and navigate the complexities of the digital era.
Author Contributions
Conceptualization, L.T. and A.T.; methodology, L.T.; formal analysis, A.T.; investigation, Y.S.; writing—original draft preparation, A.T.; writing—review and editing, L.T.; visualization, A.T.; supervision, Y.S.; project administration, Y.S. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
Not applicable.
Conflicts of Interest
The authors declare no conflicts of interest.
References
- Aljumah, A.I.; Nuseir, M.T.; Alam, M.M. Organizational performance and capabilities to analyze big data: Do the ambidexterity and business value of big data analytics matter? Bus. Process Manag. J. 2021, 27, 1088–1107. [Google Scholar]
- Wang, Y.; Kung, L.; Byrd, T.A. Big data analytics: Understanding its capabilities and potential benefits for healthcare organizations. Technol. Forecast. Soc. Chang. 2018, 126, 3–13. [Google Scholar] [CrossRef]
- Günther, W.A.; Mehrizi, M.H.R.; Huysman, M.; Feldberg, F. Debating big data: A literature review on realizing value from big data. J. Strateg. Inf. Syst. 2017, 26, 191–209. [Google Scholar]
- Karras, A.; Giannaros, A.; Karras, C.; Theodorakopoulos, L.; Mammassis, C.S.; Krimpas, G.A.; Sioutas, S. TinyML Algorithms for Big Data Management in Large-Scale IoT Systems. Future Internet 2024, 16, 42. [Google Scholar] [CrossRef]
- Al-Ali, A.R.; Gupta, R.; Zualkernan, I.; Das, S.K. Role of IoT technologies in big data management systems: A review and Smart Grid case study. Pervasive Mob. Comput. 2024, 100, 101905. [Google Scholar]
- Rajeshkumar, K.; Dhanasekaran, S.; Vasudevan, V. Efficient and secure medical big data management system using optimal map-reduce framework and deep learning. Multimed. Tools Appl. 2023, 83, 47111–47138. [Google Scholar] [CrossRef]
- Alsolbi, I.; Shavaki, F.H.; Agarwal, R.; Bharathy, G.K.; Prakash, S.; Prasad, M. Big data optimisation and management in supply chain management: A systematic literature review. Artif. Intell. Rev. 2023, 56 (Suppl. S1), 253–284. [Google Scholar] [CrossRef]
- He, Z. Research on Spatial Big Data Management and High Performance Computing Based on Information Cloud Platform. In Proceedings of the 2021 5th Annual International Conference on Data Science and Business Analytics (ICDSBA), Changsha, China, 24–26 September 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 23–28. [Google Scholar]
- Rana, M.E. Integration of big data analytics and the cloud environment in harnessing valuable business insights. In Proceedings of the 2021 International Conference on Data Analytics for Business and Industry (ICDABI), Virtual, 25–26 October 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 149–156. [Google Scholar]
- Zhuo, Z.; Zhang, S. Research on the Application of Big Data Management in Enterprise Management Decision-making and Execution Literature Review. In Proceedings of the 2019 11th International Conference on Machine Learning and Computing, New York, NY, USA, 22–24 February 2019; pp. 268–273. [Google Scholar]
- Liu, Q.; Fu, Y.; Ni, G.; Mei, J. Big Data Management Performance Evaluation in Hadoop Ecosystem. In Proceedings of the 2017 3rd International Conference on Big Data Computing and Communications (BIGCOM), Chengdu, China, 10–11 August 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 413–421. [Google Scholar]
- Shafiq, M.; Gu, Z. Deep residual learning for image recognition: A survey. Appl. Sci. 2022, 12, 8972. [Google Scholar] [CrossRef]
- Dogra, V.; Verma, S.; Kavita Chatterjee, P.; Shafi, J.; Choi, J.; Ijaz, M.F. A Complete Process of Text Classification System Using State-of-the-Art NLP Models. Comput. Intell. Neurosci. 2022, 2022, 1883698. [Google Scholar]
- Bagga, S.; Sharma, A. Big data and its challenges: A review. In Proceedings of the 2018 4th International Conference on Computing Sciences (ICCS), Jalandhar, India, 30 August 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 183–187. [Google Scholar]
- Elkawkagy, M.; Elbeh, H. High performance hadoop distributed file system. Int. J. Networked Distrib. Comput. 2020, 8, 119–123. [Google Scholar] [CrossRef]
- Yang, P.; Xiong, N.; Ren, J. Data security and privacy protection for cloud storage: A survey. IEEE Access 2020, 8, 131723–131740. [Google Scholar]
- Tekdogan, T.; Cakmak, A. Benchmarking apache spark and hadoop mapreduce on big data classification. In Proceedings of the 2021 5th International Conference on Cloud and Big Data Computing, Liverpool, UK, 13–15 August 2021; pp. 15–20. [Google Scholar]
- Deepthi, B.G.; Rani, K.S.; Krishna, P.V.; Saritha, V. An efficient architecture for processing real-time traffic data streams using apache flink. Multimed. Tools Appl. 2024, 83, 37369–37385. [Google Scholar] [CrossRef]
- Dogan, A.; Birant, D. Machine learning and data mining in manufacturing. Expert Syst. Appl. 2021, 166, 114060. [Google Scholar] [CrossRef]
- Jayatilake SM DA, C.; Ganegoda, G.U. Involvement of machine learning tools in healthcare decision making. J. Healthc. Eng. 2021, 2021, 6679512. [Google Scholar] [CrossRef]
- Yousif, O.S.; Zakaria, R.B.; Aminudin, E.; Yahya, K.; Sam, A.R.M.; Singaram, L.; Munikanan, V.; Yahya, M.A.; Wahi, N.; Shamsuddin, S.M. Review of big data integration in construction industry digitalization. Front. Built Environ. 2021, 7, 770496. [Google Scholar] [CrossRef]
- Cho, D.; Lee, M.; Shin, J. Development of cost and schedule data integration algorithm based on big data technology. Appl. Sci. 2020, 10, 8917. [Google Scholar] [CrossRef]
- Lutfi, A.; Alsyouf, A.; Almaiah, M.A.; Alrawad, M.; Abdo, A.A.K.; Al-Khasawneh, A.L.; Ibrahim, N.; Saad, M. Factors influencing the adoption of big data analytics in the digital transformation era: Case study of Jordanian SMEs. Sustainability 2022, 14, 1802. [Google Scholar] [CrossRef]
- Qin, X.; Luo, Y.; Tang, N.; Li, G. Making data visualization more efficient and effective: A survey. VLDB J. 2020, 29, 93–117. [Google Scholar] [CrossRef]
- Dimara, E.; Zhang, H.; Tory, M.; Franconeri, S. The unmet data visualization needs of decision makers within organizations. IEEE Trans. Vis. Comput. Graph. 2021, 28, 4101–4112. [Google Scholar] [CrossRef]
- Naqvi, R.; Soomro, T.R.; Alzoubi, H.M.; Ghazal, T.M.; Alshurideh, M.T. The nexus between big data and decision-making: A study of big data techniques and technologies. In Proceedings of the International Conference on Artificial Intelligence and Computer Vision, Settat, Morocco, 29 May 2021; Springer International Publishing: Cham, Switzerland, 2021; pp. 838–853. [Google Scholar]
- Mehmood, E.; Anees, T. Challenges and solutions for processing real-time big data stream: A systematic literature review. IEEE Access 2020, 8, 119123–119143. [Google Scholar] [CrossRef]
- Peddireddy, K. Streamlining Enterprise Data Processing, Reporting and Realtime Alerting using Apache Kafka. In Proceedings of the 2023 11th International Symposium on Digital Forensics and Security (ISDFS), Chattanooga, TN, USA, 11–12 May 2023; IEEE: Piscataway, NJ, USA, 2023; pp. 1–4. [Google Scholar]
- Vyas, S.; Tyagi, R.K.; Jain, C.; Sahu, S. Literature review: A comparative study of real time streaming technologies and apache kafka. In Proceedings of the 2021 Fourth International Conference on Computational Intelligence and Communication Technologies (CCICT), Sonepat, India, 3 July 2021; IEEE: Piscataway, NJ, USA, 2021; pp. 146–153. [Google Scholar]
- Leow, K.R.; Leow, M.C.; Ong, L.Y. A New Big Data Processing Framework for the Online Roadshow. Big Data Cogn. Comput. 2023, 7, 123. [Google Scholar] [CrossRef]
- Ochuba, N.A.; Amoo, O.O.; Okafor, E.S.; Akinrinola, O.; Usman, F.O. Strategies for leveraging big data and analytics for business development: A comprehensive review across sectors. Comput. Sci. IT Res. J. 2024, 5, 562–575. [Google Scholar] [CrossRef]
- Elouataoui, W.; Alaoui, I.E.; Gahi, Y. Data quality in the era of big data: A global review. Big Data Intell. Smart Appl. 2022, 994, 1–25. [Google Scholar]
- Taleb, I.; Serhani, M.A.; Bouhaddioui, C.; Dssouli, R. Big data quality framework: A holistic approach to continuous quality management. J. Big Data 2021, 8, 76. [Google Scholar] [CrossRef]
- Mishra, P.; Biancolillo, A.; Roger, J.M.; Marini, F.; Rutledge, D.N. New data preprocessing trends based on ensemble of multiple preprocessing techniques. TrAC Trends Anal. Chem. 2020, 132, 116045. [Google Scholar] [CrossRef]
- Kotiyal, B.; Pathak, H. Big Data Preprocessing Phase in Engendering Quality Data. In Machine Learning, Advances in Computing, Renewable Energy and Communication: Proceedings of MARC 2020; Springer: Singapore, 2021; pp. 65–74. [Google Scholar]
- Prakash, A.; Navya, N.; Natarajan, J. Big Data Preprocessing for Modern World: Opportunities and Challenges. In International Conference on Intelligent Data Communication Technologies and Internet of Things (ICICI) 2018; Hemanth, J., Fernando, X., Lafata, P., Baig, Z., Eds.; ICICI 2018. Lecture Notes on Data Engineering and Communications Technologies; Springer: Cham, Switzerland, 2019; Volume 26. [Google Scholar]
- Liu, X.L.; Wang, W.M.; Guo, H.; Barenji, A.V.; Li, Z.; Huang, G.Q. Industrial blockchain based framework for product lifecycle management in industry 4.0. Robot. Comput. -Integr. Manuf. 2020, 63, 101897. [Google Scholar] [CrossRef]
- Munawar, H.S.; Qayyum, S.; Ullah, F.; Sepasgozar, S. Big data and its applications in smart real estate and the disaster management life cycle: A systematic analysis. Big Data Cogn. Comput. 2020, 4, 4. [Google Scholar] [CrossRef]
- Lim, K.Y.H.; Zheng, P.; Chen, C.H. A state-of-the-art survey of Digital Twin: Techniques, engineering product lifecycle management and business innovation perspectives. J. Intell. Manuf. 2020, 31, 1313–1337. [Google Scholar] [CrossRef]
- Stark, J. Product lifecycle management (PLM). In Product Lifecycle Management (Volume 1) 21st Century Paradigm for Product Realisation; Springer International Publishing: Cham, Switzerland, 2022; pp. 1–32. [Google Scholar]
- Wang, J.; Xu, C.; Zhang, J.; Zhong, R. Big data analytics for intelligent manufacturing systems: A review. J. Manuf. Syst. 2022, 62, 738–752. [Google Scholar] [CrossRef]
- Sabireen, H.; Kirthica, S.; Sridhar, R. Secure data archiving using enhanced data retention policies. In Proceedings of the Data Science Analytics and Applications: First International Conference, DaSAA 2017, Chennai, India, 4–6 January 2017; Revised Selected Papers 1. Springer: Singapore, 2018; pp. 139–152. [Google Scholar]
- Çınar, Z.M.; Abdussalam Nuhu, A.; Zeeshan, Q.; Korhan, O.; Asmael, M.; Safaei, B. Machine learning in predictive maintenance towards sustainable smart manufacturing in industry 4.0. Sustainability 2020, 12, 8211. [Google Scholar] [CrossRef]
- Meinert, E.; Milne-Ives, M.; Surodina, S.; Lam, C. Agile requirements engineering and software planning for a digital health platform to engage the effects of isolation caused by social distancing: Case study. JMIR Public Health Surveill. 2020, 6, e19297. [Google Scholar] [CrossRef]
- Cook, E.; Merrick, J.R. Technology Implementation at Capital One. INFORMS J. Appl. Anal. 2023, 53, 178–191. [Google Scholar] [CrossRef]
- Naseema, N.; Akhtar, S.; Al Hinai, A.A. Disrupting Financial Services: A Case Study on Capital One’s Fintech Odyssey. In Harnessing Blockchain-Digital Twin Fusion for Sustainable Investments; IGI Global: Hershey, PA, USA, 2024; pp. 363–383. [Google Scholar]
- Neebe, K. Sustainability at Walmart: Success over the long haul. J. Appl. Corp. Financ. 2020, 32, 64–71. [Google Scholar] [CrossRef]
- Hamdan, A.; Ibekwe, K.I.; Ilojianya, V.I.; Sonko, S.; Etukudoh, E.A. AI in renewable energy: A review of predictive maintenance and energy optimization. Int. J. Sci. Res. Arch. 2024, 11, 718–729. [Google Scholar] [CrossRef]
- Cao, K.; Liu, Y.; Meng, G.; Sun, Q. An overview on edge computing research. IEEE Access 2020, 8, 85714–85728. [Google Scholar] [CrossRef]
- Khan, W.Z.; Ahmed, E.; Hakak, S.; Yaqoob, I.; Ahmed, A. Edge computing: A survey. Future Gener. Comput. Syst. 2019, 97, 219–235. [Google Scholar] [CrossRef]
- Breitbach, M.; Schäfer, D.; Edinger, J.; Becker, C. Context-aware data and task placement in edge computing environments. In Proceedings of the 2019 IEEE International Conference on Pervasive Computing and Communications (PerCom), Kyoto, Japan, 11–15 March 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1–10. [Google Scholar]
- Alwakeel, A.M. An overview of fog computing and edge computing security and privacy issues. Sensors 2021, 21, 8226. [Google Scholar] [CrossRef] [PubMed]
- Shahzadi, S.; Iqbal, M.; Dagiuklas, T.; Qayyum, Z.U. Multi-access edge computing: Open issues, challenges and future perspectives. J. Cloud Comput. 2017, 6, 30. [Google Scholar] [CrossRef]
- Zhang, C.; Xie, Y.; Bai, H.; Yu, B.; Li, W.; Gao, Y. A survey on federated learning. Knowl.-Based Syst. 2021, 216, 106775. [Google Scholar] [CrossRef]
- Nilsson, A.; Smith, S.; Ulm, G.; Gustavsson, E.; Jirstrand, M. A performance evaluation of federated learning algorithms. In Proceedings of the Second Workshop on Distributed Infrastructures for Deep Learning, Rennes, France, 10–11 December 2018; pp. 1–8. [Google Scholar]
- Chronis, C.; Varlamis, I.; Himeur, Y.; Sayed, A.N.; Al-Hasan, T.M.; Nhlabatsi, A.; Bensaali, F.; Dimitrakopoulos, G. A survey on the use of Federated Learning in Privacy-Preserving Recommender Systems. IEEE Open J. Comput. Soc. 2024, 5, 227–247. [Google Scholar] [CrossRef]
- Kairouz, P.; McMahan, H.B.; Avent, B.; Bellet, A.; Bennis, M.; Bhagoji, A.N.; Bonawitz, K.; Charles, Z.; Cormode, G.; Cummings, R.; et al. Advances and open problems in federated learning. Found. Trends® Mach. Learn. 2021, 14, 1–210. [Google Scholar] [CrossRef]
- Xu, C.; Liu, S.; Yang, Z.; Huang, Y.; Wong, K.K. Learning rate optimization for federated learning exploiting over-the-air computation. IEEE J. Sel. Areas Commun. 2021, 39, 3742–3756. [Google Scholar] [CrossRef]
- Li, Q.; Wen, Z.; Wu, Z.; Hu, S.; Wang, N.; Li, Y.; Liu, X.; He, B. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. IEEE Trans. Knowl. Data Eng. 2021, 35, 3347–3366. [Google Scholar] [CrossRef]
- Angelov, P.P.; Soares, E.A.; Jiang, R.; Arnold, N.I.; Atkinson, P.M. Explainable artificial intelligence: An analytical review. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2021, 11, e1424. [Google Scholar] [CrossRef]
- Došilović, F.K.; Brčić, M.; Hlupić, N. Explainable artificial intelligence: A survey. In Proceedings of the 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, Croatia, 21–25 May 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 210–215. [Google Scholar]
- Ahmed, I.; Jeon, G.; Piccialli, F. From artificial intelligence to explainable artificial intelligence in industry 4.0: A survey on what, how, and where. IEEE Trans. Ind. Inform. 2022, 18, 5031–5042. [Google Scholar] [CrossRef]
- Arrieta, A.B.; Díaz-Rodríguez, N.; Del Ser, J.; Bennetot, A.; Tabik, S.; Barbado, A.; García, S.; Gil-López, S.; Molina, D.; Benjamins, R.; et al. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 2020, 58, 82–115. [Google Scholar] [CrossRef]
- Zhang, Q.; Yang, L.T.; Chen, Z.; Li, P. A survey on deep learning for big data. Inf. Fusion 2018, 42, 146–157. [Google Scholar] [CrossRef]
- Jan, B.; Farman, H.; Khan, M.; Imran, M.; Islam, I.U.; Ahmad, A.; Ali, S.; Jeon, G. Deep learning in big data analytics: A comparative study. Comput. Electr. Eng. 2019, 75, 275–287. [Google Scholar] [CrossRef]
- Rawal, A.; McCoy, J.; Rawat, D.B.; Sadler, B.M.; Amant, R.S. Recent advances in trustworthy explainable artificial intelligence: Status, challenges, and perspectives. IEEE Trans. Artif. Intell. 2021, 3, 852–866. [Google Scholar] [CrossRef]
- Alawida, M.; Mejri, S.; Mehmood, A.; Chikhaoui, B.; Isaac Abiodun, O. A comprehensive study of ChatGPT: Advancements, limitations, and ethical considerations in natural language processing and cybersecurity. Information 2023, 14, 462. [Google Scholar] [CrossRef]
- Qin, R.; Wang, F.Y.; Zheng, X.; Ni, Q.; Li, J.; Xue, X.; Hu, B. Sora for computational social systems: From counterfactual experiments to artificiofactual experiments with parallel intelligence. IEEE Trans. Comput. Soc. Syst. 2024, 11, 1531–1550. [Google Scholar] [CrossRef]
- Chang, Y.; Wang, X.; Wang, J.; Wu, Y.; Yang, L.; Zhu, K.; Chen, H.; Yi, X.; Wang, C.; Wang, Y.; et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 2023, 15, 1–45. [Google Scholar] [CrossRef]
- Raiaan, M.A.K.; Mukta, S.H.; Fatema, K.; Fahad, N.M.; Sakib, S.; Mim, M.M.J.; Ahmad, J.; Ali, M.E.; Azam, S. A review on large Language Models: Architectures, applications, taxonomies, open issues and challenges. IEEE Access 2024, 12, 26839–26874. [Google Scholar] [CrossRef]
- Coussement, K.; Benoit, D.F. Interpretable data science for decision making. Decis. Support Syst. 2021, 150, 113664. [Google Scholar] [CrossRef]
- Myers, D.; Mohawesh, R.; Chellaboina, V.I.; Sathvik, A.L.; Venkatesh, P.; Ho, Y.-H.; Henshaw, H.; Alhawawreh, M.; Berdik, D.; Jararweh, Y. Foundation and large language models: Fundamentals, challenges, opportunities, and social impacts. Clust. Comput. 2024, 27, 1–26. [Google Scholar] [CrossRef]
- Chen, T.; Kornblith, S.; Swersky, K.; Norouzi, M.; Hinton, G.E. Big self-supervised models are strong semi-supervised learners. Adv. Neural Inf. Process. Syst. 2020, 33, 22243–22255. [Google Scholar]
- Madan, S.; Bhardwaj, K.; Gupta, S. Critical Analysis of Big Data Privacy Preservation Techniques and Challenges. In International Conference on Innovative Computing and Communications; Khanna, A., Gupta, D., Bhattacharyya, S., Hassanien, A.E., Anand, S., Jaiswal, A., Eds.; Advances in Intelligent Systems and Computing; Springer: Singapore, 2022; Volume 1394. [Google Scholar]
- Thunki, P.; Reddy SR, B.; Raparthi, M.; Maruthi, S.; Dodda, S.B.; Ravichandran, P. Explainable AI in Data Science-Enhancing Model Interpretability and Transparency. Afr. J. Artif. Intell. Sustain. Dev. 2021, 1, 1–8. [Google Scholar]
- Hariri, R.H.; Fredericks, E.M.; Bowers, K.M. Uncertainty in big data analytics: Survey, opportunities, and challenges. J. Big Data 2019, 6, 44. [Google Scholar] [CrossRef]
- Ridzuan, F.; Zainon, W.M.N.W. A Review on Data Quality Dimensions for Big Data. Procedia Comput. Sci. 2024, 234, 341–348. [Google Scholar]
- Nair, S.R. A review on ethical concerns in big data management. Int. J. Big Data Manag. 2020, 1, 8–25. [Google Scholar] [CrossRef]
- Gupta, D.; Rani, R. A study of big data evolution and research challenges. J. Inf. Sci. 2019, 45, 322–340. [Google Scholar] [CrossRef]
- Karras, A.; Giannaros, A.; Theodorakopoulos, L.; Krimpas, G.A.; Kalogeratos, G.; Karras, C.; Sioutas, S. FLIBD: A Federated Learning-Based IoT Big Data Management Approach for Privacy-Preserving over Apache Spark with FATE. Electronics 2023, 12, 4633. [Google Scholar] [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
MDPI and ACS Style
Theodorakopoulos, L.; Theodoropoulou, A.; Stamatiou, Y. A State-of-the-Art Review in Big Data Management Engineering: Real-Life Case Studies, Challenges, and Future Research Directions. Eng 2024, 5, 1266-1297. https://doi.org/10.3390/eng5030068
AMA Style
Theodorakopoulos L, Theodoropoulou A, Stamatiou Y. A State-of-the-Art Review in Big Data Management Engineering: Real-Life Case Studies, Challenges, and Future Research Directions. Eng. 2024; 5(3):1266-1297. https://doi.org/10.3390/eng5030068
Chicago/Turabian StyleTheodorakopoulos, Leonidas, Alexandra Theodoropoulou, and Yannis Stamatiou. 2024. "A State-of-the-Art Review in Big Data Management Engineering: Real-Life Case Studies, Challenges, and Future Research Directions" Eng 5, no. 3: 1266-1297. https://doi.org/10.3390/eng5030068
