Enhancing Data Management Strategies with a Hybrid Layering Framework in Assessing Data Validation and High Availability Sustainability

Abstract

Individuals and digital organizations deal with a substantial amount of collected data required for performing various data management strategies, such as replacing, upgrading, and migrating existing data from one system to another, while supporting the data’s complexity, authenticity, quality, and precision. Failures in data migration can result in data and service interruptions, financial losses, and reputational harm. This research aims to identify the specific challenges of a data management strategy, develop a comprehensive framework of data migration practices, and assess the efficacy of data validation and high availability for optimizing complex data and reducing the need to minimize errors during data migration. Combining trickle and zero-downtime migration techniques with a layering approach, a hybrid-layering framework was designed to encompass the entire spectrum of data migration techniques, beginning with system requirements and data transformation, rigorous functions, and evaluation metrics for sustainable data validation. The evaluation metric criteria are defined to evaluate data migration based on data consistency, integrity, quality, accuracy, and recall. The experiment demonstrated a real-world scenario involving a logistics company with 222 tables and 4.65 GB of data. The research compared various data migration strategies. The outcomes of the hybrid-layering framework’s examination of the final system’s functionality are satisfactory, emphasizing the critical importance of data migration sustainability to ensure data validity and high availability. This study is useful for individuals and organizations seeking to sustainably improve their data management strategies to minimize disruptions while preserving data integrity.

1. Introduction

Data are essential elements for the operation of many different computer systems, ranging from smartphones and Internet of Things (IoT) devices to powerful server infrastructures, as well as significant operations for modern business environments, including business analysis, operational decisions [1], driving innovation [2,3], and enabling competitive differentiation. Data, in their volume and variety, affect every aspect of operations, making them not only a strategic asset [4] but also something that must be managed systematically and efficiently. Effective data management is necessary for success-making opportunities and challenges in the ever-evolving landscape of present business and organizational growth. Data management encompasses a vast data volume, diversity, and velocity perspective to support organizations’ data management strategy, data quality standards, governance, security and operations, platform and architecture, and sustainability [5]. In addition, the complex nature of data management concerning the central role of various platforms and architectures focusing on database management systems (DBMS) has resulted in numerous obstacles when an organization spreads out data. There are numerous types and brands of DBMS software available. The two types of DBMS software products and licensing models [6] are commercial and open-source software. Some companies may utilize commercial database management systems like Microsoft SQL Server [7], IBM DB2, or Oracle DBMS. Meanwhile, free, open-source alternatives are provided, including PostgreSQL, MongoDB, MySQL, and MariaDB [8].

While digital organizations have expanded with an enormous amount of information, the present issues for data management are the endless increase in data, the complexity of datasets, and data quality problems. A data management strategy is one of the goals for achieving an organization’s roadmap, including communications, data management functions, business case, and funding [5]. Data migration is one of the data management functions and the most challenging task [9] for data-driven system implementation [10], consolidation, changes, and upgrades [11]. The principle of data migration enables data transfer from one or more storage spaces to the same or different computer system platforms or locations to improve data scalability, accessibility, and portability, as well as reducing the cost of information technology operations in businesses and organizations [12,13,14,15,16]. In recent years, numerous data migrations have been focused on storage [17], application [18], business process [19], DBMS [20,21], and platforms [13,20,21,22]. Many studies have identified significant concerns in data migration strategies, such as data validation [23], integrity checks, efficient migration techniques [24,25], heterogeneous data migration [26,27], and data migration frameworks for sustainable decision making [28]. In addition, existing data migration methodologies and frameworks are typically generic regarding the data pipeline and computation platform integration [29]. Migration frameworks frequently do not include methods for large-scale experimental testing and validating the accuracy and completeness of migrated data. Recently, most businesses and organizations alternatively migrate existing data from proprietary data management systems to open-source ones [30]. Due to the complexity and difficulty of data migration, the business or organization may cease services or fail [31]. Without applying an appropriate data migration technique, a failed data migration can result in unreliable data, service interruptions, financial losses, and brand damage [32].

Many recent studies have utilized data migration techniques, such as big bang data migration [33,34,35], trickle data migration [33,36,37], and zero-downtime data migration [21,38,39]. While trickle data migration allows for manageable increments [36] of utilizing the database to improve real-time stream processing without an automated migration process, the big bang migration method [34] involves migrating all data at once, which requires automated migration, testing, and validation. The zero-downtime strategy is a method that integrates the principles of continuous delivery with the objectives of high availability and data consistency [39]. However, the most challenging obstacles encountered in this approach are scheduled and unplanned data downtime. Many studies highlight only one technique out of big bang, trickle, or zero-downtime data migration, whereas others point out their limitations. This disparity underscores the need for a combination technique of data migration to support a large-scale and a complex data relationship to ensure continuous data availability throughout the entire migration process. Moreover, no comprehensive, sustainable frameworks or methodologies integrate data validation and high availability strategies with data migration, particularly in practical scenarios involving the development of cost-effective high availability solutions that balance data accessibility with resource optimization, specifically for organizations with limited financial resources.

The implementation of layering technology in system and software development yields numerous advantages. The importance of testability and error-proofing is highlighted in the concept of separate concerns in data design and development [3,40]. The ability to filter out unnecessary information and easily adapt functions are also emphasized [41,42,43].

Therefore, this study aims to address these research gaps by combining the data migration approaches of trickle and zero-downtime with a layering technique named the hybrid-layering approach. The entire data migration process is divided into layers consisting of procedures from an analysis phase to a system reconciliation phase. This study aims to identify the specific challenges of a data management strategy, to develop a comprehensive framework of data migration practices, and to evaluate the effectiveness of data validation and high availability for optimizing complex data and reducing the need to minimize errors during data migration. In order to accomplish the research objectives, the following research questions have been formulated:

-

What are the challenges of developing a comprehensive framework that integrates data validation and high availability techniques within a data migration practice, ensuring data integrity and continuous accessibility?

-

How can a comprehensive framework be evaluated for the sustainability of data validation techniques that can detect and address data quality issues, ensuring data accuracy throughout the data migration?

-

How can the effectiveness of sustainable criteria metrics and methodologies be defined and established for evaluating the efficiency and effectiveness of different data migration techniques, thereby providing decision-makers with clear guidelines for evaluating migration options?

This study makes research-significant contributions to data management strategies in data migration by providing a comprehensive hybrid-layering framework and by presenting practical case studies concentrating on the sustainability concerns of offering validation measurement together with high availability, thus supporting technological adaptation and reducing data and service disruptions.

• The framework combines hybrid data migration approaches, layered approaches, data validation, and high availability strategies. The collaborative approach to stakeholder collaboration and communication within the framework of data mapping and transitions is essential for successful data migration and promotes effective communication between team members and stakeholders. Implementing the recommended data migration techniques ensures business continuity during migration, reduces data and service corruption risks, and improves productivity, efficiency, and availability.
• Further, the study defines recommendations for evaluating and validating the accuracy and completeness of migrated data. This guidance is crucial for organizations to ensure data reliability and correctness during migration.
• This research offers practical insights into the effectiveness of the data migration framework by providing a real-world case study of converting proprietary databases to open-source ones. This empirical demonstration lends credibility to the framework for hybrid layering.
• The study assists organizations in adapting to new database system technologies when streamlining technological change and business expansion, emphasizing the significance of understanding and implementing data migration techniques to safeguard data migration strategies and sustain technological agility.

The subsequent sections of this article are structured in the following manner: Section 2 provides a theoretical background pertaining to data migration techniques and a hybrid framework. Section 3 of this study presents a comprehensive analysis of the research model and methodology that underpin the design, development, and evaluation of the proposed framework for data migration. Additionally, the study provides an overview of the outcomes obtained from a real-world case study that employed the hybrid-layering framework, and these outcomes are subsequently summarized. The discussion of the results is presented in Section 4, whereas Section 5 summarizes the migration framework and the findings derived from this study.

2. Theoretical Background

2.1. Comparison of Existing Data Migration Techniques and Hybrid Framework

Successful data migration is essential for organizations seeking to adapt to changing technological landscapes, to optimize system performance, and to guarantee data accessibility and scalability. With various data migration techniques available, such as big bang data migration, trickle data migration, and zero-downtime data migration, each approach is essential to carefully evaluate the merits and suitability of each method for the needs of a particular project.

Table 1. Comparison of data migration techniques in the preprocessing phase.

Methodology Steps	Big Bang Data Migration	Trickle Data Migration	Zero-Downtime Data Migration	Proposed Hybrid Data Migration Framework
Research and Analysis	Conduct an in-depth, comprehensive analysis of the current data environment and project specifications.	Assess the part data characteristics and project objectives to determine and meet suitability gradually.	Evaluate the data infrastructure and project goals to ascertain feasibility without any failure.	Conduct a thorough analysis of the current data environment for the entire organization, considering factors such as data complexity, volume, and business continuity requirements. Evaluate the need for continuous system availability during migration and the feasibility of incremental data transfers to reduce downtime.
Selection of Migration Technique	Choose big bang migration for low-complexity, low-volume, and short-downtime-tolerance projects.	Optimize trickle migration for projects with moderate complexity and data volume, allowing gradual data transfer to minimize downtime.	Select zero-downtime migration for projects requiring continuous system availability with complex data and a high tolerance for zero downtime.	Utilize a combination of trickle and zero-downtime migration strategies to strike a balance between gradual data transfer and continuous system availability. Choose this strategy for projects with complex data, a high tolerance for zero downtime, and the requirement of seamless data integration without protracted interruptions in mission-critical operations.
Planning and Preparation	Develop a detailed migration plan outlining tasks, timelines, and resource allocation.	Plan and prepare for incremental data transfers, ensuring coordination and data integrity across phases.	Create a comprehensive migration plan, focusing on replication and synchronization to maintain system availability during the migration.	Develop a comprehensive migration plan incorporating the incremental data transfer method of trickle migration and the continuous system availability strategy of zero-downtime migration. Assign resources and establish deadlines to ensure efficient execution. Prepare a comprehensive back-up and recovery plan to protect vital data during migration.
Data Profiling and Preprocessing	The data are profiled to identify potential issues and are preprocessed to ensure compatibility with the target data management system.	Conduct data profiling to understand data complexities and incremental preprocessing of data for seamless integration.	Conduct data profiling to ensure data compatibility and fault-tolerant preprocessing of data for synchronization with minimal disruption.	Conduct exhaustive data profiling to comprehend the complexities and nuances of data during the trickle and zero-downtime migration phases. The data must be preprocessed to ensure compatibility with the target data system and maintain data consistency during the gradual data transfer, thereby ensuring rollback capability for data recovery.

presents a comprehensive methodology encompassing these three migration techniques, providing a holistic and well-informed approach for guiding organizations through the data migration process, as well as our proposed combination approach.

The methodology aspects of migration approaches consist of research and analysis, technique selection, planning and preparation, data profiling and preprocessing, testing and validation, migration execution, post-migration evaluation, documentation, and knowledge transfer. By examining the complexities of each technique and integrating them into a cohesive methodology, organizations can make informed decisions, minimize risks, and achieve successful and seamless data migrations, thereby enhancing data management practices and bolstering their competitive advantage in a dynamic digital landscape [44,45].

The combination of trickle and zero-downtime data migration techniques is presented in Table 1 as a comparison of data migration techniques in the preprocessing phase. This strategy aims to capitalize on the advantages of both methodologies, providing a balanced solution for complex data migration projects. Organizations can sustainably achieve seamless data integration and maintain system functionality throughout the data migration process through trickle migration and zero-downtime techniques. The methodology emphasizes meticulous planning, testing, and documentation to ensure data integrity, accuracy, and optimal system performance, empowering organizations to manage their data migration projects effectively and efficiently with minimal disruption and improved user experience.

Table 2. Comparison of data migration techniques in the post-processing phase.

Methodology Steps	Big Bang Data Migration	Trickle Data Migration	Zero-Downtime Data Migration	Proposed Hybrid Data Migration Framework
Testing and Validation	After data migration, establish a dedicated testing environment to verify data accuracy and consistency.	Perform rigorous testing at each migration phase to validate data integrity and identify inconsistencies.	Incrementally implement thorough testing, simulating real-time scenarios to ensure data consistency and system functionality without impacting users.	Establish a dedicated testing environment to verify the efficacy of the combined migration strategies. Perform rigorous testing at each successive phase, ensuring data integrity, consistency, and system functionality without interfering with ongoing operations. Conduct real-time simulations to evaluate system performance and ensure availability throughout the migration.
Execution of Migration	Transferring the entire dataset to the target data in a single migration operation.	Migrate data incrementally, moving subsets of data in controlled stages while ensuring data consistency.	Implement real-time data replication and synchronization mechanisms to achieve continuous availability and data consistency during migration.	To incrementally transfer subsets of data, execute the migration in controlled stages utilizing the trickle method. Implement real-time data replication and synchronization mechanisms, similar to zero-downtime migration, to ensure continuous data availability throughout the entire migration process. Maintain data accuracy and system stability through vigilant migration monitoring and the prompt resolution of any problems.
Post-Migration Evaluation	Evaluate the success of the migration process, addressing any post-migration issues that may arise.	Analyze the effectiveness of the migration, considering data consistency and user feedback to refine the process.	Assess the migration’s performance, address post-migration issues, and gather stakeholder feedback to optimize the technique.	Consider data consistency, system performance, and user feedback in evaluating the success of the combined migration strategy. Evaluate the efficacy of the trickle and zero-downtime techniques in achieving the project’s goals. To refine the methodology for future migrations and optimize data management strategies, migration result is necessary to gather input from stakeholders and end-users.
Documentation and Knowledge Transfer	Document the entire migration process, challenges faced, and lessons learned for future reference.	Create comprehensive documentation of the migration steps and knowledge transfer sessions for IT teams and end-users.	Document the migration methodology, share knowledge and insights gained, and provide training to ensure a seamless transition to the new system.	For future reference, document the entire migration process, including the combined trickle and zero-downtime techniques, obstacles encountered, and lessons learned. Create detailed documentation outlining the methodology and knowledge transfer sessions for IT teams and end-users to ensure a smooth transition to the new data environment.

compares and contrasts the various post-processing data migration strategies. Following the migration procedure, the post-processing phase is essential for ensuring data accuracy, consistency, and operational efficiency. This table compares big bang migration, trickle migration, zero-downtime migration, and the proposed hybrid layering data migration framework. These techniques are analyzed and compared in depth, considering crucial aspects such as testing and validation, migration execution, post-migration evaluation, documentation, and knowledge transfer. This study aims to conduct a comprehensive analysis of these methodologies so that organizations can make well-informed choices when selecting the most appropriate approach for their data migration requirements. This comparative analysis is useful for IT professionals and decision-makers engaged in data migration projects.

2.2. Novel Criteria Metrics of Effectiveness Evaluation in Data Migration Sustainability

The criteria depicted in

Table 3. Key criteria of data management for evaluating effective data migration sustainability.

Criteria	Description
Compatibility [19,46]	Compatibility refers to the ability to successfully transfer data and functionality from one data management system to another without losing functionality, compromising data integrity, or requiring significant changes to the application code. This factor pertains to the assurance that the target data management system has the necessary capabilities to manage data competently and meet application requirements.
Transformation [47]	Transformation involves modifying or converting data, schema, or other components from the source data to a format suitable for the target data. Adjustments are made to ensure compatibility, data integrity, and optimal performance in the new data environment.
Integrity [48]	The integrity of data migration refers to the maintenance of data precision, consistency, and dependability throughout the migration process, including constraints, triggers, and referential integrity between source and target data. It ensures that the data is migrated from the source to the target data without loss or corruption.
Performance Optimization [19,21,49]	Performance optimization in data migration is the process of improving the responsiveness and efficiency of a data management system during and after the migration process. It requires identifying and implementing techniques to improve query execution, data retrieval, and overall system performance in the new data environment.
Licensing and Cost [21,49]	Licensing and cost considerations in data migration involve evaluating and managing the financial facets of migrating from one data management system to another. Understanding the licensing models, costs, and potential financial implications of the source and target data management systems is required.
Accuracy [21,50,51]	Accuracy refers to the exactness and accuracy of the data being transferred from the source data to the target data. It involves ensuring that the migrated data represents the original data accurately and maintains its integrity throughout the migration process.
Reliability [48,49]	In data migration, reliability refers to the dependability and uniformity of the migration process and the resulting data management system. It entails ensuring that the migration is carried out successfully, with minimal errors or interruptions, and that the migrated data functions dependably regarding data integrity, availability, and performance.
Transparency and Transformation Processing [15,51,52]	Transparency in data migration refers to the visibility and clarity of the migration procedure. In contrast, transformation processing refers to the data and structure modifications and conversions performed during migration. Transparency provides clear and comprehensive visibility into the migration process, enabling stakeholders to comprehend and track the migration’s progress, actions, and potential consequences.

are necessary for evaluating the sustainability and efficacy of data migration activities within data management.

Maintaining the alignment and efficiency of primary components, rapidly changing technologies, and knowledge of computer systems, including hardware, software, peopleware, and data, requires data migration. The computer system includes leveraging hardware advancements, adopting the most recent software capabilities, enhancing people’s skills, fostering collaboration, consolidating data, and ensuring data integrity while transforming data. With data migration skill enhancement, peopleware requires collaboration between multiple teams, including data administrators, developers, IT operations, and business stakeholders. A challenging aspect of data migration is the heterogeneity of suitable methods for data compatibility, accuracy, reliability, transparency, transition preparation, and persistent operations [19,53]. Multiple considerations must be optimized for data migration concerns. Therefore, a criteria matrix for an effectiveness evaluation is proposed to support the final data migration process prior to an organization’s implementation of a new open-source data management system. Each criterion plays a vital role in evaluating the efficacy, dependability, and long-term viability of data migration projects, ensuring that data are transferred without interruption while retaining their quality and integrity throughout the migration technique.

3. Research Model and Methodology

This section describes the research materials and methodology used to explain the data migration methods, as depicted in Figure 1. This study focused on the data transformation technique required to develop a hybrid layering data migration framework emphasizing data validity and high availability sustainability, as well as studying data migration techniques, as presented in Table 1 and Table 2. In addition, reviewing the sustainability of data management for effective and efficient data migration, designing sustainable assessment criteria for data migration, as in Table 3, evaluating and justifying the practical case using the data migration, and summarizing the research methodology results were presented. The proposed methodology utilized a layering approach and a hybrid migration of trickle and zero-downtime data migration strategies. An industrial practical success case demonstrates evaluation accuracy by applying the hybrid layering framework based on designed assessment criteria, yielding valid and high availability results for an efficient data migration method. The authors propose a novel hybrid layering framework and sustainable migration criterion metrics to address the research challenges associated with advancing the understanding of the evaluation in data migrations for sustainability of validity and high availability.

From Figure 1, several steps of this study’s methodology transfer the existing commercial data management system to the target open-source data management system. The following steps are included in the methodology for data migration transformation processes: analyzing the current source data management system, including data models, data relationships, data constraints, database structures, and functionalities. The source data included database names, table names, field names, primary keys, candidate keys for each table, constraints, and data storage locations. Significant characteristic differences of source and target data management systems, including the database model, data definitions, data constraints, data manipulating languages, database system commands, license, and cost, affected and were designed to analyze strategic concerns.

3.1. Methodological Development of Hybrid-Layering Migration Framework

As illustrated in Figure 2, system information is gathered in the requirement planning phase of a system transformation, and a data migration strategy is developed. This phase entails gaining an understanding of the current system, which is based on the source data management system, and developing a strategy for transforming data from the source data management system to the target system environment by communicating with and involving stakeholders, such as database and system administrators, users, and the chief information officer (CIO). The team examines the data models, schema, tables, attributes, data constraints, data records, and application architecture to identify potential mapping incompatibilities between the two systems. This step is essential for ensuring an effortless migration, laying the groundwork for subsequent processes in later phases, identifying the requirements of the new database system, and determining the source server database’s compatibility with the target server. The procedure entails the creation of a migration plan outlining the migration timeline, the sequence of the migration of datasets, and the roles and responsibilities of team members and stakeholders involved in the data migration. A back-up and recovery plan should also be included in the migration plan to ensure the safety of data management system services with zero downtime during the migration process.

This execution stage is subdivided into several phases with distinct activities and responsibilities. This stage consists of data migration verification and functional/performance accuracy testing phases that involve the practical migration of data. In the data migration verification phase, the focus shifts to the actual data migration, including compliance and security considerations, back-up and recovery preparations, incremental migration datasets, and data validation and verification processes with stakeholder approval. The team executes the data extraction, transformation, and loading procedures, transferring the data from one system to another table by table. Following the completion of the migration, the dataset will be transferred. Data consistency and integrity checks ensure that all crucial data have been successfully migrated. The data migration technique concludes with the data verification stored in data storage devices, where the migrated data is validated and verified for precision and completeness. In addition to the termination data on the source data management system, the migrated datasets on the target data management system are accessible for servicing application connections. In the functional/performance accuracy testing phase, the team conducts rigorous verification tests to validate the accuracy and completeness of the transferred data in conjunction with the stakeholders’ verification tests.

The framework methodology emphasizes careful planning, validation, and testing, contributing to the data migration’s overall precision and success. By focusing on data extraction, transformation, and loading processes, the framework ensures the accuracy and completeness of migrated data while minimizing the risk of errors and inconsistencies, as concluded in Table 4.

This study presents a comprehensive and systematic framework for efficient data migration from source and target data management systems to facilitate successful and accurate data migrations. Five practical processes define the framework: system transformation, data migration verification, functional/performance accuracy tests, system tuning, and system delivery. Each procedure is meticulously crafted to address specific obstacles and ensure a seamless data migration experience for organizations undergoing technological transformations. The framework allowed the company to transfer critical data tables gradually without interruptions or downtime.

Table 4. Efficient practice processes to increase the probability of a successful data migration.

Framework Phase	Efficient Practice Activity	Description
System Transformation	Thorough Planning	Develop a comprehensive migration plan, including analysis, objectives, timelines, and milestones.
	Data Assessment and Mapping	Assess the source data, map it to the target data, and ensure compatibility and integrity.
	Expertise and Collaboration	Engage experienced professionals and collaborate closely with the migration team for comprehensive understanding.
	Risk Management and Contingency Planning	Identify potential risks and develop contingency plans to mitigate them during the migration process.
System Transformation Data Migration Verification Functional/Performance Accuracy Tests	Compliance and Security Considerations	Ensure compliance with relevant data regulations and implement appropriate security measures in the new data.
System Transformation Functional/Performance Accuracy Tests Delivering System	Communication and Stakeholder Engagement	Maintain open communication with stakeholders and involve them throughout the migration process.
Data Migration Verification Functional/Performance Accuracy Tests	Testing and Validation	To ensure accuracy, completeness, and integrity, perform extensive testing and validation of the migrated data.
System Transformation Data Migration Verification Functional/Performance Accuracy Tests	Back-up and Recovery	Implement robust back-up and recovery mechanisms to safeguard the data during the migration process.
Data Migration Verification Functional/Performance Accuracy Tests	Incremental Migration	Consider migrating data into smaller batches or modules to facilitate monitoring, validation, and troubleshooting.
	Data Validation and Verification	Validate the migrated data’s accuracy, completeness, and integrity through various verification techniques.
Tuning System	Monitoring and Performance Optimization	Continuously monitor the migration process and optimize the performance of the target data.
Delivering System	Documentation and Knowledge Transfer	Maintain thorough documentation to aid future reference and knowledge transfer to other team members.
	Post-Migration Support and Maintenance	Provide support and maintenance after migration to address any post-migration issues or performance optimization.
	Training and Knowledge Transfer for End-Users	Conduct training sessions and knowledge transfer activities to ensure end-users are familiar with the new data.
	Regular Performance Monitoring and Maintenance Reviews	Regularly monitor the performance of the target data and conduct maintenance reviews to optimize functionality.

Table 4. Efficient practice processes to increase the probability of a successful data migration.

Framework Phase	Efficient Practice Activity	Description
System Transformation	Thorough Planning	Develop a comprehensive migration plan, including analysis, objectives, timelines, and milestones.
	Data Assessment and Mapping	Assess the source data, map it to the target data, and ensure compatibility and integrity.
	Expertise and Collaboration	Engage experienced professionals and collaborate closely with the migration team for comprehensive understanding.
	Risk Management and Contingency Planning	Identify potential risks and develop contingency plans to mitigate them during the migration process.
System Transformation Data Migration Verification Functional/Performance Accuracy Tests	Compliance and Security Considerations	Ensure compliance with relevant data regulations and implement appropriate security measures in the new data.
System Transformation Functional/Performance Accuracy Tests Delivering System	Communication and Stakeholder Engagement	Maintain open communication with stakeholders and involve them throughout the migration process.
Data Migration Verification Functional/Performance Accuracy Tests	Testing and Validation	To ensure accuracy, completeness, and integrity, perform extensive testing and validation of the migrated data.
System Transformation Data Migration Verification Functional/Performance Accuracy Tests	Back-up and Recovery	Implement robust back-up and recovery mechanisms to safeguard the data during the migration process.
Data Migration Verification Functional/Performance Accuracy Tests	Incremental Migration	Consider migrating data into smaller batches or modules to facilitate monitoring, validation, and troubleshooting.
	Data Validation and Verification	Validate the migrated data’s accuracy, completeness, and integrity through various verification techniques.
Tuning System	Monitoring and Performance Optimization	Continuously monitor the migration process and optimize the performance of the target data.
Delivering System	Documentation and Knowledge Transfer	Maintain thorough documentation to aid future reference and knowledge transfer to other team members.
	Post-Migration Support and Maintenance	Provide support and maintenance after migration to address any post-migration issues or performance optimization.
	Training and Knowledge Transfer for End-Users	Conduct training sessions and knowledge transfer activities to ensure end-users are familiar with the new data.
	Regular Performance Monitoring and Maintenance Reviews	Regularly monitor the performance of the target data and conduct maintenance reviews to optimize functionality.

3.2. Evaluation Criteria Metrics and Formula of Data Validation Sustainability

The hybrid layering framework is essential for assessing the accuracy of data migration to ensure the successful, efficient, and dependable transformation of data from the source data management system to the target ones. In this study, several novel key formulas were developed and utilized to evaluate the hybrid layering migration framework accuracy quantitatively. Data consistency rate (DCR), data integrity score (DIS), data accuracy index (DAI), error rate (ER), data completeness percentage (DCP), data matching ratio (DMR), data consistency index (DCI), and data accuracy percentage (DAP) are all proposed factors. In addition, precision and recall factors contribute significantly to evaluating the framework’s accuracy [54]. According to

Table 5. Effectiveness and accuracy evaluation metrics and formula of data migration framework.

Category	Evaluation Criteria	Formula	Description
Compatibility	Data Consistency Rate (DCR)	(Number of Consistent Records/Total Number of Records) × 100	Measures the level of consistency between the migrated data and the original source data.
	Data Integrity Rate (DIS)	(Total Number of Correct Values/Total Number of Migrated Values) × 100	Assesses the accuracy and completeness of the migrated data to ensure data integrity is maintained.
	Data Accuracy Index (DAI)	(1 − (|Total Number of Inaccurate Values|/Total Number of Migrated Values)) × 100	Provides an index to measure the overall accuracy of the migrated data.
	Error Rate (ER)	(Number of Error Records/Total Number of Records) × 100	Indicates the percentage of data records that contain errors or discrepancies in the migrated data.
Transformation	Data Completeness Percentage (DCP)	(Number of Complete Records/Total Number of Records) × 100	Measures the degree to which the migrated data are complete and reflects the entirety of the source data.
	Data Matching Ratio (DMR)	(Number of Matched Records/ Total Number of Records) × 100	Evaluates the accuracy of data matching during the migration process, ensuring correct record associations.
Integrity	Data Consistency Index (DCI)	(1 − (|Total Number of Inconsistent Values|/Total Number of Migrated Values)) × 100	Provides an index that reflects the consistency of data values between the source and target data.
Performance Optimization	Data Accuracy Percentage (DAP)	(Number of Correct Values/Total Number of Migrated Values) × 100	Represents the overall accuracy of the migrated data.
Effectiveness and Accuracy Evaluation	Precision	(True Positives)/(True Positives + False Positives)	Measures the accuracy of the positive predictions made during the migration.
	Recall	(True Positives)/(True Positives + False Negatives)	Measures the percentage of true positive matches found.

, the authors noted that comprehensive insights into the precision and dependability of results can be obtained.

Precision and recall are widely employed performance metrics that provide valuable insights into the quality of the migration activity, introducing a crucial aspect of evaluating the accuracy and dependability of the system’s results [55,56]. A framework for data migration can ensure data consistency, identify potential issues, and make informed decisions by measuring precision and recall results to enhance the migration process and preserve data integrity. Positive matches are data records that have been successfully migrated and match the source and target data. Negative matches, on the other hand, are records that were improperly migrated and did not match between the two data. Establishing clear criteria for positive and negative matches is essential for accurate assessment. Positive matches and negative matches should be included in the validation dataset. The evaluation result can identify true positive matches (correctly migrated matching records), false positive matches (incorrectly migrated matching records), true negative matches (correctly migrated non-matching records), and false negative matches (incorrectly migrated non-matching records) by comparing the data records between the source and target data. As a result, precision is calculated as the ratio of true positive matches to the total number of positive matches predicted by the migration process, thereby reflecting the accuracy of positive predictions made during migration.

The hybrid layering data migration framework (Figure 2) was developed iteratively, with regular feedback and input from stakeholders, to ensure that the framework is effectively tailored to the specific needs and requirements of the data migration project. However, this study’s conditional requirement was to migrate the entire database system. By mapping and schema design between source and target database systems, the target schema was designed using entity relationship diagrams (ERD) for the samples of entity relationships, tables, attributes, data type, keys, and constraints, as shown in Figure 3 and Figure 4. In addition, the preparation process included data normalization, data quality checks, and data transformation to ensure compatibility and integrity in the target system. In Figure 5, the final programming implementation of the data migration sustainability for data validity and high availability was to develop and validate the successful transfer of data from the source data to the target data management system, which includes data extraction, transformation, loading, and reconciliation.

On the other hand, the recall ratio is the proportion of true positive matches to the total number of actual positive matches in the validation dataset and measures the migration process’s ability to identify positive matches accurately. Precision and recall results are indispensable to gaining insight into the framework’s performance for the migration process. A higher precision indicates fewer false positives, indicating that the migrated data are less likely to match non-matching records and vice versa incorrectly; a high recall indicates that the migration process accurately identifies a more significant proportion of the actual positive matches.

3.3. A Real-World Case Study to Evaluate the Hybrid Layering Framework

A real-world case study of data migration begins with a company whose enterprise resource planning (ERP) systems have used a commercial database server to manage sales, products, ordering customers, purchasing orders, suppliers, and logistics data for several years. Due to changing business requirements, the company has decided to migrate the entire database to the target database management system. The goal of the migration project is to transfer the data without loss or corruption of all vital data tables, including those for supplier, product, product details, customer, purchasing transaction, sale transaction, logistic transaction, and product status. Data structure, data integrity, minimization of downtime, and data volume are just a few of the significant issues that arise from complex data migration processes. The source and target database systems have distinct data structures, including different data types, syntax, and query syntax. Over the years, the system’s provider has amassed a substantial amount of data. The entire dataset consisted of 223 tables and contained 4.65 GB of data. The processes must ensure that all data can be transferred to minimize downtime and disruptions, as prolonged system downtime could negatively affect customer and supplier satisfaction. In addition, it is essential to maintain data integrity and precision during the migration process to prevent inconsistencies and errors. Figure 3 and Figure 4 depict the entity–relationship diagram (ERD) for the sample data scenario of the company. The example ERD consists of data for the purchasing order (PO) information management system, which includes the supplier, product, product details, customer, sale transaction, product status, and supplier–product business. Each table has its unique primary, foreign, and candidate keys to ensure data integrity and uniqueness.

In data migration, it is essential to consider every aspect of migration results involving a change within the same DBMS model, a move between different DBMS models, or a move between different DBMS models while maintaining the same entity–relationship (ER) model. Each of these situations requires a unique set of considerations and outcomes. This is similar to DBMS model modifications when the current data management system is stable and meets the organization’s operational needs well. An intelligent option for migration within the same DBMS model is to either upgrade the version or optimize the data. This method typically reveals opportunities to enhance the current data structure and overall system performance. Typically, adhering to the same DBMS model reduces the number of problems associated with how tasks are carried out.

When modifying models of data management systems, it is rational to switch the DBMS model when the target model offers advanced functionality, scalability, economics, or features that work well with the organization’s changing operational needs. Data mapping and transformation issues are frequently raised between various DBMS models. Different schema structures, query languages, and data storage methods must be planned and implemented with precision. The most important step is evaluating the new DBMS’s support infrastructure, licensing terms, and overall compatibility, as switching to a different DBMS model requires a substantial investment of time and resources. This study demonstrates that changing DBMS models while maintaining the same ER model is preferable to maintaining a uniform data structure. However, the process is not simple, as mapping the ER model to different DBMS implementations can be difficult. Despite the consistency of the ER model, data types, constraints, and query languages vary across DBMS models. This study illustrates the importance of meticulous planning in complex data transformation projects. Developing solid data transfer plans and conducting a comprehensive analysis of how well the new DBMS models integrate with the existing applications and tools should always be the primary focus in these circumstances. Preparations must be made to adapt or improve applications so that the transition has no negative effect on operations and is seamless.

3.4. A Practice Implementation for the Sustainability of the Hybrid Layering Framework

As shown in Figure 5, the implementing diagram flow represents the particular concerns of sustainable data migration processes. This study involves numerous users, including developers, administrators, end-users, and stakeholders, who must be effectively communicated with by migration teams. The Python package should be installed to support pyodbc, psycopg2, csv, and time to implement an efficient and enduring data migration using Python. The pyodbc library connects to the source database management system, and the csv library is used to export the source dataset in CSV format. This practical case study connects to the target database management system using the psycopg2 library. The implementation modules begin with the CSV export of the source dataset. To execute the migrated processes, numerous functions are defined, including the definition of target table function, the definition of the migrate data function, the definition of the migrate sequence function, the definition of the check consistency function, the definition of the check integrity function, the definition of the check constraint function, and the definition of the perform migration function. The dataset will be rolled back if a single data migration function returns an error. The migrate_sequences process migrates sequences (used for serial primary keys in the target dataset) from the source database management system to the target database management system. This procedure ensures that the equivalent sequences in the target dataset are assigned the correct value based on the current identity values in the source tables and dataset. The check_consistency process is designed to check the consistency of data between the source and target datasets during data migration by comparing the data retrieved from both the source and target datasets, row by row, and determining whether the data matches in each row. To check the integrity of the data migration from source to target database management system, the integrity process must verify that the schema elements (such as tables, columns, constraints, and indexes) are consistent between the two databases, followed by a check to ensure that the data types and constraints were replicated correctly. Each table’s constraints between the source and target database management systems will be verified. If there is any inconsistency, an error message will be generated to indicate the discrepancy. If there are no errors, the result will indicate successful migration. Following the completion of data migration, the perform migration procedure verifies all processes. The script will indicate which checks failed if any do not pass. If all validations pass, the script will output a success message. The differences between Figure 1, Figure 2 and Figure 5 are, respectively, the overall research methodology to develop the data migration, the strategies and processes of the layering framework for efficient data migration, and the information system architectures of practical implementation.

The functional/performance testing accuracy phase involves validating and verifying the migrated data to ensure that transferred data are accurate, application-compatible, and complete. The testing procedure evaluates the database’s functionality and performance. The team conducts a series of tests to confirm that the database functions as expected in the target environment, with the approval of the primary stakeholders. Functional tests evaluate the behavior of the new system’s applications and queries to ensure that all functionality is intact. Simultaneously, performance tests are conducted to evaluate the database’s response time and overall efficiency, thereby identifying potential bottlenecks that could affect performance. This phase is divided into two processes, each with its own activities and responsibilities. The data validation process involves comparing the migrated datasets on the target database system to the original source server database to ensure no data loss or corruption issues. The application validation process ensures that migrated datasets function correctly within applications.

The tuning system phase focuses on optimizing the database’s performance and efficiency. The data migration team identifies areas requiring fine-tuning to improve system responsiveness and resource utilization based on the results of performance tests. Index optimization, query optimization, and configuration adjustments are performed in the target database system environment to optimize the database’s performance. This phase prepares the system for anticipated workloads and user demands. To ensure the success of the data migration from commercial to open-source database management systems, the data migration must adhere to the framework outlined above. This framework includes recommended planning, execution, and validation procedures for migration processes. In addition, data migration success requires a skilled and experienced team to manage the migration process. This team must include stakeholders such as database administrators, developers, and project managers.

The delivery system process is the last phase of the data migration framework. The fully migrated and optimized data are deployed to the production environment during this phase. The team coordinates the transition from the previous source server system to the new target system, ensuring minimal downtime and an effortless change. At this stage, user acceptance tests of the overall functionalities of system services may be conducted to ensure that all business processes and functionalities in the new data environment are operating as expected.

In conclusion, the framework for efficient data migration is a complex procedure requiring meticulous planning, execution, and validation. By adhering to the framework mentioned above and employing a skilled and experienced team, organizations can migrate their data to a new system without experiencing data loss or corruption.

3.5. Data Collection

The demographic data collection of the actual data migration, as presented in

Table 6. The unsuccessful test results of data migration approaches.

Support Source	Support Target	Solutions	Approach	Results
Data–Entity relationship	Data–Entity relationship	SQL server integration services (SSIS)	Incremental, SQL scripts, built-in features.	The system functions cannot be validated and verified despite the successful migration. The system was inoperable.
		Replication	Incremental, SQL Scripts, PostgreSQL foreign data wrapper (FDW) to read data from SQL server tables and write it into PostgreSQL.	The system functions cannot be validated and verified despite the successful migration. The system was inoperable.
		Incremental migration—Manual data pumping	Incremental SQL Scripts migrate data in smaller batches or based on manual-specific criteria.	The system functions cannot be validated and verified despite the successful migration. The system was inoperable.

, contains eight departments, including public relations (PR), purchasing order (PO), sale order (SO), human resources (HR), goods receipt (GR), delivery notes (DN), job order (JO), and hand over (HO). The minimum number of tables is 16 (human resources), and the maximum number of tables is 45 (purchasing order). Human resources (HR) are the category of the most minor concern, with 30 tables and 49,200 records accounting for 13.36 percent of the total data, making it the largest category in terms of both tables and records. Purchasing order (PO) is the least problematic category, with 13 tables and 17,810 records comprising 5.93% of the total data, making it the category with the most tables and records. These descriptive statistics can be used to effectively plan and execute the data migration process, ensuring data integrity and accuracy throughout the migration.

3.6. Data Analysis and Results

To determine the success rate of a tested data migration, developer teams must verify the data model and collect the relevant data migration results based on a practical, specific scenario. Each iteration of data migration would involve analyzing the number of successful detail evaluations for the migration. Every iteration of data migration would compute the number of successful and unsuccessful migrations, as well as any other relevant metrics that define success or failure in the tested data of the model. After collecting the necessary data, the following formula can be used to calculate the success rate:

(Number of Successful Migrations/Total Number of Migrations) × 100 = Success Rate

A table of migration results is used to monitor the success rate of individual data entities during a migration. The success rate can be determined based on whether each entity successfully migrated or encountered problems.

In our comprehensive analysis of different migration solutions presented in Table 6, this study previously assessed three methodologies: SQL server integration services (SSIS), replication, and incremental migration—manual data pumping.

The purported compatibility of these solutions with a range of source and target databases, such as MS SQL, Oracle, MySQL, and PostgreSQL, was asserted. It is imperative to acknowledge that the verification and validation stages of the migration process presented substantial challenges for each of the three approaches, ultimately resulting in their inability to succeed. These approaches demonstrate the significance of successfully implementing rigorous verification and validation procedures to migrate complex data management systems. The importance of careful planning and implementation is highlighted by our research, particularly concerning migration techniques. The previously tested migration approach highlights the significance of maintaining data integrity and validating.

Table 7. Evaluation results of data migration approaches for data validity and high availability.

Department	Data Source		Data Target
			Trickle Migration			Zero Downtime Migration		Proposed Hybrid Layering Framework
	Tables	Records	Tables	Records	Downtime	Tables	Records	Tables	Records	Downtime
Public Relation (PR)	19	18,940	19	18,940	3	Failure occurs	-	19	18,940	0
Purchasing Order (PO)	13	17,810	13	17,810	2	Failure occurs	-	13	17,810	0
Sale Order (SO)	24	69,030	24	69,030	5	Failure occurs	-	24	69,030	0
Human Resource (HR)	30	49,200	30	49,200	4	Failure occurs	-	30	49,200	0
Good Receipt (GR)	29	16,700	29	16,700	4	Failure occurs	-	29	16,700	0
Delivery Notes (DN)	23	12,190	23	12,190	3	Failure occurs	-	23	12,190	0
Job Order (JO)	26	69,030	26	69,030	4	Failure occurs	-	26	69,030	0
Hand Over (HO)	24	2370	24	2370	4	Failure occurs	-	24	2370	0
Warehouse Stock (WS)	23	28,760	23	28,760	3	Failure occurs	-	23	28,760	0
Others	11	16,520	11	16,520	0	11	16,520	11	16,520	0
Total	222	300,550	222	300,550	29	11	16,520	222	300,550	0

thoroughly examines data migration outcomes in different departments using three distinct methodologies: trickle migration, zero-downtime migration, and the proposed hybrid layering framework. The trickle migration method ensures that the same number of tables and records are migrated from the source data system for all departments. In contrast, the zero-downtime migration method fails to migrate the source data entirely. Despite that, both systems experience a significant number of migration failures, resulting in lengthy periods of inactivity, with an average of four failures per organizational unit. Zero-downtime can negatively affect the continuity of operations and the ability to access data. In contrast, the proposed hybrid layering framework emerges as a more sophisticated methodology, demonstrating operational continuity across all organizational units. This methodology ensures continuous access to data throughout the migration process, reducing the likelihood of operational disruptions. In addition, it guarantees the retention of the same number of tables and records as observed in the original system, indicating accurate data migration. The analysis demonstrates that the proposed hybrid layering framework effectively achieves data validity and high availability without causing operational disruptions, thereby establishing it as the preferred option for organizations executing complex data transfers.

Table 8. Evaluation results of successful data migration for data validity and high availability.

Department	Migration Rounds	Incremental Migration	Compatibility				Transformation		Integrity	Performance	Success Rate (SR)
			DCR	DIS	DAI	ER	DCP	DMR	DCI	DAP
PR	1	32%	53%	100%	100%	0%	53%	53%	100%	100%	100%
	2	64%	74%	100%	100%	0%	74%	74%	100%	100%	100%
	3	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
PO	1	54%	68%	100%	100%	0%	68%	68%	100%	100%	100%
	2	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
SO	1	25%	29%	100%	100%	0%	29%	29%	100%	100%	100%
	2	50%	52%	100%	100%	0%	52%	52%	100%	100%	100%
	3	75%	73%	100%	100%	0%	73%	73%	100%	100%	100%
	4	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
HR	1	20%	15%	100%	100%	0%	15%	15%	100%	100%	100%
	2	40%	27%	100%	100%	0%	27%	27%	100%	100%	100%
	3	60%	63%	100%	100%	0%	63%	63%	100%	100%	100%
	4	80%	78%	100%	100%	0%	78%	78%	100%	100%	100%
	5	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
GR	1	14%	17%	100%	100%	0%	17%	17%	100%	100%	100%
	2	35%	33%	100%	100%	0%	33%	33%	100%	100%	100%
	3	52%	58%	100%	100%	0%	58%	58%	100%	100%	100%
	4	76%	77%	100%	100%	0%	77%	77%	100%	100%	100%
	5	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
DN	1	26%	34%	100%	100%	0%	34%	34%	100%	100%	100%
	2	57%	56%	100%	100%	0%	56%	56%	100%	100%	100%
	3	83%	74%	100%	100%	0%	74%	74%	100%	100%	100%
	4	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
JO	1	27%	38%	100%	100%	0%	38%	38%	100%	100%	100%
	2	54%	59%	100%	100%	0%	59%	59%	100%	100%	100%
	3	77%	74%	100%	100%	0%	74%	74%	100%	100%	100%
	4	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
HO	1	54%	67%	100%	100%	0%	67%	67%	100%	100%	100%
	2	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
WS	1	35%	26%	100%	100%	0%	26%	26%	100%	100%	100%
	2	74%	56%	100%	100%	0%	56%	56%	100%	100%	100%
	3	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%
Others	1	100%	100%	100%	100%	0%	100%	100%	100%	100%	100%

shows the evaluation metrics of the migration results confirming the successful implementation of the hybrid layering data migration technique for various departments utilizing the proposed framework. The high success rates (SR) and outstanding performance in metrics described in Table 5, such as compatibility, transformation, integrity, and data completeness, demonstrate the efficacy of the migration framework strategy. The table displays the percentage values for each metric, indicating to what extent the migration process achieved the desired results. The findings of the data table contribute valuable insights into the field of data migration, providing organizations with crucial information for planning and carrying out successful migration projects.

A complex interaction of operational, financial, and strategic factors determines the number of migration rounds. A successful migration requires meticulous planning, execution, and adaptation to minimize disruptions and ensure the seamless adoption of new systems and technologies within an organization. The results demonstrated that the most excellent effectiveness is typically observed when different departments undergo between two and five rounds of migration. The disparity in the number of migration rounds can be attributed to various factors and organizational considerations, with the complexity of the migration process playing a crucial role. Departments with intricate systems or complex data structures may require more rounds to facilitate a seamless transition. Moreover, the size and scope of a department’s operations are essential considerations. Larger departments or those responsible for mission-critical functions frequently require additional migration rounds to ensure the successful migration of all operational aspects without causing service interruptions. The availability of resources, including personnel, budget allocation, and technological infrastructure, substantially affects the rate and number of migration rounds. Departments with abundant resources can execute migrations more efficiently and thoroughly. Integrating each migration round, testing, and validation procedure contributes to round number variation. Comprehensive testing, essential for ensuring the accuracy and reliability of data migration, may extend the overall process, leading to additional rounds. Risk mitigation also plays a significant role in the determination of migration rounds. Especially if their operations are mission-critical, prudent departments may opt for a phased approach, distributing migration efforts across multiple rounds to minimize risks. Technical dependencies that span interconnections between systems or data within an organization exert influence. For seamless integration, complex technical interdependencies may necessitate a phased migration strategy.

Priorities within an organization are a crucial factor. Departments may have varying goals and timelines. Some organizations prioritize rapid migration to adopt new technologies, whereas others have alternative strategic objectives. The changing technological landscape necessitates migration adjustments. As technology evolves, departments may need to realign their systems and data to take advantage of emerging capabilities, which can necessitate multiple rounds of migration.

The results of each department’s data migration are depicted in Figure 6 from the viewpoints of incremental data migration rounds and system compatibility for migrated data consistency and integrity percentage rates. Figure 7 illustrates each department’s data migration outcomes regarding system compatibility, migrated data accuracy and error rates, and transformation-based data completeness rates. Figure 8 represents each department’s data migration outcomes in terms of transformation percentages for data matching ratio, integrity-based data consistency, and performance-based data accuracy.

The data migration of the public relations (PR) department was conducted in three rounds, and all three rounds were 100 percent successful. The incremental migration strategy was utilized, ensuring data consistency and integrity (DCR: 100%, DIS: 100%) and successfully migrating data to the target data (DAI: 100%). The performance of the migrated system was also satisfactory (ER: 0%), and the data completeness percentage indicated that all data were migrated accurately (DCP: 100%). Similarly, the purchasing order (PO) department underwent two migration rounds that were both 100 percent successful. The migration process maintained compatibility (DCR: 100%) and data integrity (DIS: 100%) while ensuring precise data transformation (DAI: 100%). Both the performance of the migrated system (ER: 0%) and the data integrity (DCP: 100%) were flawless.

The sales order (SO) department underwent four migration rounds, with each migration round achieving 100 percent success. Throughout the migration, compatibility (DCR: 100%) and data transformation (DAI: 100%) were maintained as a result of the incremental migration process. Data integrity was also maintained (DIS: 100%), and the performance of the migrated system revealed no errors or discrepancies (ER: 0%).

The human resources (HR) department underwent five migration rounds, each 100 percent successful. The migration process maintained compatibility (DCR: 100%) and data integrity (DIS: 100%) while ensuring precise data transformation (DAI: 100%). The flawless performance of the migrated system (ER: 0%) indicates that no errors were present in the migrated data.

The goods receipt (GR) department’s data migration was completed in five rounds, with each round achieving a 100 percent success rate. The migration procedure successfully maintained compatibility (DCR: 100%) and data integrity (DIS: 100%) while transforming the data precisely (DAI: 100%). The performance of the migrated system (ER: 0%) and data completeness (DCP: 100%) provided additional evidence of the migration’s success.

The delivery notes (DN), job order (JO), hand over (HO), warehouse stock (WS), and other departments all had identically high success rates of 100%. All of these departments successfully completed their rounds of migration, maintaining compatibility, data integrity, and accurate transformation while demonstrating outstanding performance and data completion.

4. Discussion

To improve the effectiveness of data management on data migration utilizing the hybrid layering framework, the impact of data migration on sustainable, efficient business, technology, analysis, data migration techniques, and accuracy evaluation will be detailed in the discussion section.

4.1. Data Migration’s Impact on Efficient Business and Technology

The data migration from commercial to open-source data management systems significantly impacted the organization’s business and technological characteristics. The target data system provides enhanced performance, scalability, and durability, resulting in improved data management processing and storage capabilities, supported by [46,56]. With the target data management system’s support for multiple platforms, such as Windows, Linux, and macOS, the organization gains flexibility and reduced reliance on a particular Microsoft operating system. The target data management system also adapts to shifting technology landscapes for improved data manipulation needs. According to several studies [29,56,57], migration improved data accessibility and scalability, enabling faster information retrieval for decision-making processes. The migration also improved data integration and analysis, enabling more informed business strategies to be implemented using various open-source applications. Nonetheless, the migration process encountered some initial difficulties, resulting in a brief period of downtime. Despite the minor disruptions, the overall effect on business operations was positive, as the advantages of the migrated system outweighed the temporary inconvenience.

Data migration to an open-source data management system can substantially benefit an organization’s business operations. The target data management system enabled organizations to utilize open-source technology. The data management system’s robust and efficient data retrieval capabilities enabled faster access to vital information, allowing employees to make timely, well-informed decisions. This increased data accessibility enhanced the organization’s productivity and adaptability. Additionally, the migration improved data integration and analysis. The migration process afforded the chance to reevaluate and optimize data structures, resulting in more organized data and more insightful data model analysis. In addition, the data migration project resulted in increasingly tailored data security and compliance checks. The target data management system’s advanced security features and encryption support assisted in bolstering data protection and ensuring compliance with industry regulations. This enhanced data security instilled greater confidence among customers and stakeholders, fostering stronger relationships and strengthening the organization’s reputation. Due to the fact that the target data management system is an open-source data management system, the company gained access to a larger, more vibrant, and active community of developers and contributors. The community-driven ecosystem provided continuous support, frequent updates, and abundant resources, ensuring long-term stability and growth while reducing the organization’s technology expenditures.

4.2. Hybrid Layering Data Migration’s Efficient Techniques

Depending on data migration techniques, the impact of data migration on a business’s overall operations and objectives will vary. The migration objectives have achieved the intended business goals, such as increased efficiency, cost savings, enhanced data analysis capabilities, and improved decision-making processes. The business impact must be assessed via both quantitative and qualitative measures. After migration, the new data management system is closely monitored to identify and address any potential issues or obstacles. Regular and timely analysis of the migrated data management system provides users with a reliable data servicing system and addresses any post-migration concerns expeditiously. The effectiveness of the support evaluates and resolves the problem-solving procedure to ensure a smooth transition to the new data management system. Document the entire data migration process, including any obstacles encountered, lessons learned, and effective practices implemented, to prevent future problems from arising and to facilitate their prompt resolution. These techniques are a valuable resource for future migrations and the ongoing enhancement of migration procedures.

Several recommendations can be made to improve the accuracy and efficiency of practical data migrations based on evaluating and comparing the migration techniques supported by [20,51,58,59,60]. Organizations can adopt a hybrid layering strategy by combining the advantages of zero-downtime migration for continuous access and trickle migration for gradually expedited migration completion. During the planning phase, a more thorough evaluation of the system’s requirements and data structure model should be conducted to identify potential obstacles and ensure greater preparedness. The framework could also benefit from incorporating automated testing and validation processes to improve data accuracy and reduce migration-related manual errors automatically. Moreover, collaboration and communication between members of the migration team and stakeholders are essential for resolving any issues that may arise during the migration process. Regular monitoring and reporting should be a top priority for users and database administrators to detect and resolve discrepancies expeditiously.

The data migration project was also met with obstacles. The migration process necessitated meticulous planning and execution to avoid potential business disruptions, as stated in [49,61]. Early on, the scope and purpose of the data migration should be defined. The source data’s model, tables, attributes, constraints, and functionalities are essentially defined for the target data. This procedure facilitates the identification of the precise set of tables required for the migration. In the source data, the relationships between tables are examined to identify the primary key, foreign key, and constraints and to comprehend how data are linked across tables. The analysis process aids in determining which tables that are interconnected and vital to the core business processes should be moved together to preserve data integrity. Prioritize necessary tables to ensure that data tables and essential functionalities are present in the target data. Each migrated table must be analyzed for the same volume of source data. Large tables with numerous records should necessitate additional migration planning and optimization. To ensure a smooth migration, a migration process should prioritize the transfer of smaller tables and implement an incremental migration with zero-downtime. This technique decreases the possibility of errors and downtime during the migration process. Involving key stakeholders, application developers, data analysts, and end-users in selecting data tables is essential for providing input and understanding the valuable data necessary for making informed decisions. A data migration test of the selected group of tables requires a simulation to ensure data consistency and accuracy prior to the actual migration. This validation will assist in identifying potential issues and ensure the data are transfer-ready. A comprehensive back-up and recovery strategy protects data during the migration process. A back-up ensures that data can be restored to their original state in case of unforeseen issues. The successful implementation of data reconciliation techniques led to identifying and resolving data inconsistencies among expert team members that would have otherwise gone unnoticed. This proactive approach to data validation enhanced the data migration’s dependability and quality.

4.3. Successful Data Migration of Data Validity and High Availability Sustainability

This study demonstrates the most critical concerns for data migration using the proposed framework to ensure a high level of practical evaluation during the migration process. The data migration’s effects on data compatibility [19,46], transformation [47], integrity [48], performance optimization [19,21,49], licensing and cost [21,49], accuracy [21,50,51], dependability [48,49], transparency and transformation processing [15,51,52], and persistent operations [15,51,52,62] supported one of the primary benefits. In addition, novel and effective evaluation metrics were utilized to evaluate the migration’s success regarding data integrity, consistency, and accuracy. The accuracy of the migrated data was quantified using metrics such as data consistency rate (DCR), data integrity score (DIS), data accuracy index (DAI), error rate (ER), data completeness percentage (DCP), data matching ratio (DMR), data consistency index (DCI), and data accuracy percentage (DAP). The accuracy evaluation revealed that the migration process maintained data consistency and integrity with commendable results. The DCR and DCI scores indicated few discrepancies between the source and target data, indicating a high data consistency. The DAI and DMR metrics demonstrated accurate data transformation, guaranteeing that the data in the migrated system accurately reflected the original Microsoft SQL server database. The low ER and DAP values also indicated a low error rate, indicating a successful migration with a high level of precision.

The accuracy evaluation ensured that the migrated data in the open-source database accurately reflected the desired state with minimal deviations. Data accuracy and consistency are crucial for any organization to assess direct impacts on decision making, data analysis, and overall business operations [54,57]. By conducting a rigorous evaluation of accuracy, the organization could confirm the integrity and consistency of the migrated data, thereby instilling confidence in the new data management system’s dependability. In addition, the accuracy evaluation revealed potential issues or inconsistencies during the migration process. Identifying and resolving migrated issues prevented costly data errors and potential business process disruptions. The significance of the accuracy evaluation extended beyond the migration project, laying the groundwork for future data-driven initiatives and dependable data management. Innovative testing methodologies and techniques were incorporated into the accuracy evaluation process to ensure comprehensive data validation.

4.4. The Hybrid Layering Framework Applying in the Case of Non-Relational Data

The proposed framework for data migration is adaptable and can be applied to non-relational data for numerous persuasive reasons. Understanding the system’s data models, schema, tables, attributes, data constraints, and records is the initial step of the framework. The analysis easily extends to non-relational data, such as NoSQL, with unique data structures. Analysis comprehension is essential for migration success and emphasizes the development of a transformation strategy to facilitate the transfer of data from the source to the target data management system. Document-oriented, key-value, column-family, and graph data are examples of non-relational data, whereas relational databases employ structured data paradigms. Thus, the framework’s adaptability enables transformation strategies to be tailored to the unique data models of non-relational databases.

The proposed framework also requires the active participation of stakeholders, including database administrators, users, and the chief information officer. Participation of stakeholders is essential regardless of the data type. Non-relational databases are utilized in a variety of specialized settings. Therefore, soliciting stakeholder input guarantees that the migration meets these complex requirements. The framework relies on a comprehensive back-up and recovery plan to maintain data integrity during migration. This requirement highlights the importance of data protection during data migration for all data types, including transactional databases. The framework’s execution phase includes data migration and extraction, transformation, and loading (ETL) procedures. Due to differences in data models, the ETL process for non-relational databases may differ, but data migration, validation, and verification ensure data accuracy and completeness. The framework stresses the importance of data validation and verification for migration. The technique’s concern is the significance of data quality during migration for relational and non-relational databases. In addition to these essential features, the framework permits meticulous customization to meet the specific requirements of non-relational databases. The flexibility allows organizations to tailor the framework to their specific requirements, thereby enhancing the efficiency and effectiveness of data migrations in non-relational database environments. The framework’s adaptability makes it a potent instrument for managing and executing data migrations in non-relational databases. With careful customization and adherence to its principles, organizations can confidently and accurately migrate data regardless of the database paradigm.

5. Conclusions

This study presents a comprehensive framework for accurate and successful data migration with applied efficiency. The framework’s design is based on five practical procedures: system transformation, data migration verification, functional/performance accuracy tests, system tuning, and system delivery. Through meticulous planning, validation, and evaluation, the framework confirms that organizations undergoing technologically efficient transformations can have an effortless migration experience.

5.1. Theoretical Contributions

The hybrid layering data migration framework contributes to the alignment of evaluation criteria with hardware, software, peopleware, and data, which are the primary sources of data management. The proposed framework enables organizations to make informed decisions and optimize the performance along with the dependability of the data migration technique, particularly data migration, by considering computer system components. Innovative evaluation metrics, including data consistency rate, data integrity score, and data accuracy index, provide quantitative measures to assess the accuracy of the migration process, thereby improving data-driven decision making for real-world applications. In addition, the proposed framework reveals data migration processes that can be used to migrate data in any data management system and showcases the framework’s efficacy in managing data migration tasks and ensuring compatibility transformation between source and target data.

5.2. Practical Implications

Adoption of the proposed data migration framework offers a number of practical benefits. The migration to the open-source data management system enables customization, enhanced performance, scalability, and data processing. Multiple platforms are supported by open-source data services, and open-source technology promotes greater flexibility and reduces reliance on specific operating systems, resulting in lower technology investment costs. Data migration presents an opportunity to enhance the integration and analysis of advanced data technologies, enabling more informed business strategies and better decisions during digital transformation. Additionally, the improved data security and compliance forms inspire confidence among stakeholders and enhance the organization’s reputation. In this study, the proposed framework provides a practical and systematic method for ensuring the success and accuracy of the data migration process. By considering the impact of data migration on sustainable business and technology, any organization can successfully apply the principles of the proposed data migration framework to improve the data migration process, which includes phases such as planning, data extraction, data transformation, data loading, data verification, and application validation. This study also emphasizes the importance of collaboration and communication between stakeholders, application developers, and database administrators for successful and sustainable data migration.

The study’s findings confirm the general applicability of the proposed framework. At the same time, specific efficient practices may vary based on project requirements and the specifics of the migration. Determining the probability of success sustainability for a data migration is difficult since it depends on several variables, such as the complexity of the data, the size of the data, the migration strategy, the migration team’s expertise, and the availability of resources. Data migration success rates can vary from project to project. With proper planning, adherence to actual demonstrating practices, and exhaustive testing, however, the likelihood of a successful migration can be significantly increased. It is essential to employ seasoned professionals with knowledge of source and open-source data management systems. They can evaluate the migration’s specific requirements, analyze the database schema and data, and design an appropriate migration strategy. In addition, performing a comprehensive assessment and analysis of the data and their dependencies, performing extensive testing and validation of the migrated data, and having robust back-up and recovery mechanisms can increase the likelihood of a successful migration. It is also essential to note that thorough planning, testing, and contingency plans should be in place to mitigate any potential risks and minimize the impact of unanticipated problems during the migration process. In the end, the success rate of a data migration depends on each project’s specific conditions and variables. Organizations can increase the probability of a sustainable successful migration by employing a well-defined methodology, utilizing experienced professionals, and conducting exhaustive testing and validation.

5.3. Limitations and Future Work

Despite the evaluation merits of the proposed framework, the data migration process in real-world applications may face certain challenges. The initial migration phase necessitated meticulous planning and execution to prevent operational disruptions. In addition, the migration process can become complex, requiring careful consideration and optimization, particularly for large tables with numerous records and structures of data models, including data types, tables, attributes, data types, keys, and constraints. During the data migration process, the results must support data consistency and accuracy, necessitating proactive data validation techniques. While the framework addressed the limitations to a great extent, continuous monitoring and reporting are still essential for addressing post-migration issues.

The applicability of the proposed framework can be extended to other data management systems. The effectiveness of additional data migration can be compared across various data migration scenarios or additional data models, including non-tabular databases such as NoSQL databases. Automated testing and validation processes can further improve data precision and reduce manual errors. Furthermore, hybrid migration strategies, which combine the benefits of different migration techniques, can be investigated to expedite the migration process with minimal disruptions. The long-term effects of data migration factors on organizational performance, data analysis capabilities, and business operations can also be the subject of further studies.