MAISTRO: Towards an Agile Methodology for AI System Development Projects

Abstract

The MAISTRO methodology introduces a comprehensive and integrative, agile framework for managing Artificial Intelligence (AI) system development projects, addressing familiar challenges such as technical complexity, multidisciplinary collaboration, and ethical considerations. Designed to align technological capabilities with business objectives, MAISTRO integrates iterative practices and governance frameworks to enhance efficiency, transparency, and adaptability throughout the AI lifecycle. This methodology encompasses seven key phases, from business needs understanding to operation, ensuring continuous improvement and alignment with strategic goals. A comparative analysis highlights MAISTRO’s advantages over traditional methodologies such as CRISP-DM and OSEMN, particularly in flexibility, governance, and ethical alignment. This study applies MAISTRO in a simulated case study of the PreçoBomAquiSim supermarket, demonstrating its effectiveness in developing an AI-powered recommendation system. Results include a 20% increase in product sales and a 15% rise in average customer ticket size, highlighting the methodology’s ability to deliver measurable business value. By emphasizing iterative development, data quality, ethical governance, change and risk management, MAISTRO provides a robust approach for AI projects and suggests directions for future research across diverse industries context for facilitating large-scale adoption.

1. Introduction

Artificial Intelligence (AI) has established itself as a driving force in several sectors, providing innovative solutions to complex problems and expanding the horizons regarding what technology can achieve. AI applications in areas such as healthcare, finance, and manufacturing have transformed processes and enabled more informed and efficient decisions [1]. However, despite these achievements, the successful implementation of AI projects is still not a reality for many organizations. Nevertheless, the successful implementation of AI projects remains a challenge, due to the complexity involved and the lack of an integrated methodology that addresses all phases of the development lifecycle [2].

One of the main challenges faced in the implementation of AI projects is the need to align technological capabilities with strategic business needs. This disconnect often results in solutions that, although technically feasible, do not fully meet organizational objectives [3]. Furthermore, AI project management requires the coordination of multidisciplinary teams and the integration of advanced technologies, such as machine learning and natural language processing. Without a clear methodology, these challenges can lead to failure to meet deadlines and budgets, or to the delivery of results that fall short of expectations [2].

The MAISTRO methodology distinguishes itself from traditional approaches by integrating agile practices, such as Scrum and Kanban, with advanced AI techniques and ethical considerations. Unlike conventional methodologies, which often fail to deal with the dynamics and complexity of AI projects, MAISTRO offers a flexible and adaptable framework that not only improves the efficiency and transparency of processes but also ensures the sustainability of AI systems. This methodology directly addresses the gaps identified in traditional practices, such as the difficulty in defining clear sprints in a highly exploratory AI environment and the cyclical and continuous integration of intermediate results, ensuring that the proposed solutions are always aligned with the organization’s strategic objectives, as well as monitored and controlled throughout the project development cycle.

This research addresses a gap in the implementation of AI projects, where the lack of solid methodologies leads to delays, budget overruns, and underutilization of technological solutions. We seek to propose MAISTRO as an integrated response to this situation, combining agility, governance, and ethical commitment. The major difference is that each phase of the framework prioritizes both business results and transparency and accountability mechanisms. We believe that, by providing clear and iterative guidelines, MAISTRO will provide AI initiatives to achieve measurable and sustainable results. Thus, this work aims to consolidate robust processes, providing a guideline that allows organizations of assorted sizes and cultural contexts to adapt and generate continuous value. This work proposes an integrated methodology for the development of AI projects to be applied in different organizational contexts, promoting their efficiency, transparency, and sustainability. The MAISTRO methodology aims to fill the gaps of traditional approaches, which often fail to deal with the complexity and dynamics of AI projects. It proposes a framework that integrates best project management practices, advanced AI techniques, and ethical and transparency considerations.

This paper is organized into sections that follow a progressive logic to achieve the established objectives. In the next section, we present the Literature Review, where we discuss the existing methodologies and identify their limitations in relation to the current challenges of AI projects. Then, in the section on the Proposed Methodology, we detail each step of the suggested framework, explaining how it integrates project management practices and AI techniques and tools. In the Results section, we discuss the application of the methodology in case studies, analyzing the impacts and effectiveness of the proposed practices. The Discussion section addresses the implications of the results and the possible limitations of the methodology. Finally, we conclude the paper with the Final Considerations and suggestions for future research, where we reflect on the contributions of the study and the opportunities for deepening and expanding the research.

2. Related Works

In recent years, the use of agile methodologies such as Scrum, Kanban, and Extreme Programming (XP) have been adapted for AI project management, due to their flexibility and ability to promote collaboration between multidisciplinary teams. However, the literature points to specific challenges in the application of these methodologies in AI, such as the difficulty in defining clear “sprints” due to the exploration and uncertain nature of many AI projects. Martin [4] discussed these limitations, while more recent studies, such as that of Obreja et al. [5], highlight the need to integrate ethical and cultural considerations into agile processes, something that is particularly challenging in the field of AI.

The MAISTRO methodology aims to overcome these challenges by combining agile practices with advanced AI techniques and tools, ensuring that sprints and iterations are not only technically feasible, but also ethically aligned with organizational objectives. Dignum [6] emphasizes the importance of initiative-taking approaches to ensure social responsibility in AI projects, reinforcing the relevance of MAISTRO in this context.

The management of projects involving AI requires specific practices due to the complexity and iterative nature of these projects. Traditional frameworks such as CRISP-DM, OSEMN, SEMMA, and KDD provide a basis for structuring the process, from data collection to model implementation and evaluation [7,8].

However, as pointed out by Müller et al. [9], the rapid evolution of technology and the increasing complexity of data have created new challenges for these frameworks, including the difficulty in maintaining data consistency and quality throughout the project lifecycle.

Projects involving data science face many challenges that need to be overcome to ensure their effectiveness and not generate ambiguous results. The application of adaptive project development approaches, such as SCRUM and KANBAN, according to Amirian et al. [10], allows for incremental deliveries, but requires a methodology combined with this agile process, which avoids infinite iterations and the non-continuous generation of value, culminating in customer dissatisfaction.

The MAISTRO methodology, by integrating MLOps (Machine Learning Operations) principles as suggested by Kreuzberger et al. [11], seeks to solve these problems by promoting continuous monitoring and adaptation of AI models, ensuring that they remain efficient and effective even in constantly changing data environments.

The implementation of AI projects faces numerous challenges, ranging from technical complexity to managing stakeholder expectations. Aldoseri et al. [12] highlight that data quality and availability are often critical factors that limit the success of these projects. Furthermore, the integration of multidisciplinary teams, which include data scientists, software engineers, and business domain experts, presents significant challenges for project cohesion.

MAISTRO proposes a framework that facilitates the integration of these teams, promoting clear communication and continuous collaboration, which are essential for the success of complex AI projects. In addition, the methodology addresses risk management proactively, incorporating practices that ensure transparency and traceability of decisions throughout the project and multidisciplinary collaboration, as suggested by recent studies [13].

Ethical and transparency issues in AI have gained prominence in literature, focusing on the need for AI systems that are not only technically sound but also socially responsible. Recent literature, such as the work of Palumbo et al. [14], has identified the lack of objective metrics to assess ethics in AI, which makes it difficult to implement ethical practices consistently.

In addition, the implementation of ethical guidelines, such as those proposed by IEEE and the European Union, has been advocated as an essential step to ensure that AI systems are fair and accountable, including fundamental right to the protection of personal data and its safeguard [15]. However, the practical application of these guidelines still faces challenges, particularly in the balance between transparency and protection of intellectual property or data privacy [16,17].

MAISTRO differentiates itself by integrating ethical guidelines from the beginning of the project, using tools to continuously monitor and evaluate the ethical alignment of the proposed solutions. This includes incorporating principles of transparency and clear explanation, as suggested by Vainio-Pekka et al. [18], ensuring that decisions made by AI models are understandable and justifiable to all stakeholders.

Despite significant advances in AI project management methodologies, several limitations persist. As pointed out by Brendel et al. [19], many traditional methodologies fail to keep up with the speed of technological change and business requirements. Furthermore, the difficulty of integrating ethical and transparency considerations into all phases of the project remains a significant barrier.

The recently introduced ISO 42001:2023 [20] standard, despite not being considered a methodology, aims to guide organizations in the governance and management of Artificial Intelligence initiatives. Although it provides a fundamental framework for AI strategy, risk assessment and compliance, identification and mitigation of algorithmic biases, protection of privacy and ensuring fairness in AI systems, the ISO 42001:2023 standard does not comprehensively address certain aspects of iterative and multidisciplinary project management addressed by MAISTRO methodology and the importance of strategic alignment for value generation. Specifically, it does not offer detailed guidance on iterative development cycles, continuous stakeholder engagement, and ethical checkpoints built into each project phase–elements that are crucial in AI systems that rapidly evolve with changing data. On the other hand, MAISTRO places significant emphasis on agile sprints, transparent workflows, value generation and continuous ethical oversight, thus complementing and expanding what ISO 42001:2023 [20] currently covers in the domain of AI project governance.

As reported above, although there are frameworks that address technical and management aspects, few offer clear guidance on how to incorporate these ethical principles in a practical and effective way [21].

The MAISTRO methodology was developed to address these limitations, offering a more flexible and adaptable approach that not only supports the complexity of AI projects, but also ensures that solutions are ethically responsible and transparent.

Based on the considerations of the authors’ research above, the evaluation items in

Table 1. Main gaps in current AI methodologies.

	CRISP-DM	OSEMN	SEMMA	KDD
Adherence to Business Strategic
Pipeline Data Flow Application
Risk and Uncertainty Management
Legal and Ethical Concerns
Organizational and Cultural Considerations
Multidisciplinary Team Colaboration
Social and Environmental Impact
Agile Project Management Approach
Updated with Emerging Technology
Better User Experience
Continuous Improvement

Source: prepared by the authors.

were highlighted. We can see that the main methodologies currently used in the market for developing AI systems present gaps that must be covered so that projects can be successful in accordance with contemporary demands.

In view of the above, it is worth noting that the literature review conducted has some limitations, especially regarding worldwide representativeness and methodological diversity. These limitations were mitigated as far as possible but still represent areas of opportunity for future research.

3. Materials and Methods

Best project management practices can contribute to the success of artificial intelligence projects. One way to achieve the expected value more assertively is to divide the effort into four successive and well-defined macro-stages in the project lifecycle, as described by Pradeep [22]. Each of these stages uses an appropriate approach to develop projects in the dynamic and changing scenarios we face today, seeking the ideal solution to meet business needs within a specific context:

(a)

Explore the entire problem clearly: techniques or methods such as brainstorming; design-thinking; design-sprint are used for a broad discussion of the problem and identification of workable solutions that can be forwarded for a technical and economic-financial feasibility analysis.

(b)

Define the most effective solution: the ideas that were indicated in the previous stage are evaluated through proof of concept; prototypes; use cases, until the most appropriate solution is found.

(c)

Perform efficiently: the project development approach is defined to meet the delivery cadence and adaptability to changes, and project execution is monitored through best monitoring and control practices.

(d)

Continuous improvement: continuous value generation depends on lessons learned from previous projects and user feedback, which must be applied in solution scaling efforts and future projects.

The need to adapt quickly to changes and continuously generate value [23] requires the application of successful practices that meet business challenges in a timely manner.

The MAISTRO methodology was developed to provide a comprehensive and systematic, yet flexible, framework that guides development teams through all phases of the Artificial Intelligence systems development project lifecycle, from understanding business needs to operationalizing the AI system. It contains 7 phases, as listed below:

(I)

Business needs understanding.

(II)

Data understanding.

(III)

Data preparation.

(IV)

Project execution.

(V)

Evaluation of results.

(VI)

Deployment.

(VII)

Operation.

Figure 1 shows the graphical representation of the MAISTRO Methodology framework, with its 7 phases and the processes groups defined for each phase of the AI project development lifecycle.

In the first, upper layer, we have four macro-stages of the project lifecycle described above: problem definition; design and validation of the solution to be developed; execution; and improvement with continuous value generation.

The second layer establishes the seven phases proposed for the systems development cycle, with the MAISTRO solution for AI projects.

In the line immediately below, we have the process groups of a project, which are recommended by the Project Management Institute [23], so that best management practices are adopted: initiation, planning, execution, closure, and post-implementation.

And in the boxes on the lower layer, we have the processes groups to be executed in each of the 7 phases of the system development cycle, which are recurring and adaptable, aligned with agile project management practices. Thus, the circles in each of the phases, with a clockwise rotation, denote the possibility of iterations until the process is mature and can be evolved to the next phase.

Each one of the 39 processes defined in our methodology uses the scheme of Inputs, Tools and Techniques, and Outputs, called the ITTO model, described by the Project Management Institute in previous editions (particularly in the fifth or sixth, although the seventh edition has a different structure). This ensures our methodology is aligned with well-established project management’s best practices.

Each phase of the development cycle, and the processes involved, are presented, and detailed below.

3.1. Phase I: Business Needs Understanding

The initial phase of any AI project is crucial to align the technological capabilities of AI with the strategic objectives of the organization. This process begins with a detailed analysis of the problem that the AI system intends to solve, considering all relevant aspects to determine how AI can effectively contribute to the solutions [24]. This is followed by an assessment of the suitability of AI by analyzing its capabilities in relation to the specific problem and selecting the most appropriate AI tools and techniques based on the nature of the challenge [25]. It is also essential to identify which components of the project will benefit from the implementation of AI by establishing successful criteria and defining ethical and security standards, in addition to the necessary competencies for team [26,27].

3.1.1. Issue Clarify

The success of an AI project fundamentally depends on the clarity with which the problem is defined and understood. The analysis must delve into the needs of users and the existing gaps that AI can fill, focusing on real demands and not on mere technological possibilities [28]. Advanced data analysis and collection techniques, including the use of natural language processing (NLP) and clustering algorithms, are essential to detail and quantify the identified problem [29]. This step also involves assessing the technical feasibility of the proposed solution, ensuring that investments are directed at viable initiatives aligned with the real needs of the business [30].

3.1.2. Success Measuring Definition

It is important that the measures of success expected for the project are clear from the beginning. Kloppenborg et al. [31] suggest that traditional project controls on the “iron triangle” (scope, time, and cost) are incomplete and need to be expanded to reflect business success. Key performance indicators (KPIs) should be highlighted at this initial stage and monitored throughout the project to ensure that desired results are achieved, and stakeholders are satisfied. In other words: what are the project success factors that influence the achievement of a successful result that generates value for the business [32]?

Organizational, technological, procedural, and environmental aspects need to be assessed and addressed [33]. What technical criteria and indicators will be assessed? For example, accuracy, latency, precision, and error rates can be used. And what business criteria and indicators will be monitored? For example, user adoption percentage, operational efficiency, user satisfaction, weekly and monthly progress comparisons, etc. can be used.

Validation must consider indicators and metrics determined depending on the type of business and the environment in which the use case is applied. Therefore, in an industrial installation with a production line, successful indicators must be determined quantitatively and will be linked to quality variability, error rate and productivity. In a credit card fraud prevention operation, the indicators will also be quantitative and should point to the accuracy and precision of transactional responses, false positive rates, and percentage of amounts recovered. In an application for recommending travel packages for tourists, the indicators for measuring success rates may be qualitative and linked to customer satisfaction surveys. In an electronic service application, the key indicators must be qualitative, with satisfaction surveys and quantitative, pointing out response times, abandonment rates in URAs (Audible Response Unit), service chats, or purchase sites, and attribution of service levels.

To obtain fast and reliable results, it is interesting to first implement initiatives that generate an expressive value and can be available in a short period of time with low cost and a high positive impact, such as automation of repetitive tasks; simple predictive analyses based on existing data, which can reduce service time, or provide a demand forecast.

3.1.3. Insights Prioritization

At this stage, it is vital to identify and analyze available data to extract insights that guide strategic and operational decisions. The selection of insights should balance short-term solutions with long-term strategic investments, using a risk-reward analysis to assess the viability of each AI initiative [34]. The formation of an AI committee or multidisciplinary team is recommended to ensure inclusive and informed decision-making that respects privacy standards and regulatory requirements [35,36].

3.1.4. Ethics and Security Patterns

Developing AI systems ethically and securely is essential, adopting guidelines based on international standards and best practices from design to the operation of the systems [37]. It is essential to incorporate AI ethical principles that promote the protection of human rights and dignity, ensuring that systems are fair and transparent [38]. Palumbo et al. [14] established metrics for evaluating ethical principles throughout the lifecycle of an AI project. The implementation of human oversight systems and technological trust teams are important measures to prevent the reproduction of biases and ensure accountability in AI operations [39].

3.1.5. Social and Environmental Impacts

Considering social and environmental responsibility is essential in the development of AI projects, especially given the growing impact of these technologies on society and the environment. In this process, clear criteria must be established to evaluate the social and environmental impacts of proposed solutions. This includes identifying and mitigating possible negative externalities, such as the digital divide, excessive energy consumption in processing for AI training, and the impact on workers replaced or affected by technology implementations [40,41].

Additionally, the project must adopt practices that promote inclusion and equity, such as ensuring that the data used reflect diversity and minimizing algorithmic biases that can perpetuate inequalities. Concrete measures could include the use of renewable energy sources in training processes and the offsetting of carbon emissions, as well as partnerships with local initiatives to maximize the social benefits of implementations.

Social and environmental responsibility indicators must be defined at the beginning and monitored throughout the project lifecycle, being integrated into the business success metrics. This alignment not only contributes to sustainability but also strengthens stakeholder trust and promotes acceptance and legitimacy of the solutions developed [42].

3.1.6. AI Tools and Techniques Needs

The choice of tools and techniques should be determined by the latest technological innovations, ensuring that the project is in harmony with the current capabilities of AI technology. The adaptation of AI models, such as Low Rank Adaptation (LoRa) and quantization, and the integration of APIs and microservices, are essential to optimize operational efficiency and applicability in different contexts [43,44].

3.1.7. Risk Analysis

Identifying and planning for risk mitigation is essential, given the complexity and rapid evolution of AI technologies. Transparency in the documentation of AI solutions and the implementation of real-time monitoring are crucial to respond to potential threats and ensure the robustness of models in dynamic environments [45,46].

Therefore, the early development phase of AI projects lays the foundation for successful integration of technology with business objectives, requiring a meticulous approach to problem definition, technology selection, and establishment of ethical standards. Continuous collaboration across disciplines and active stakeholder engagement are essential for project alignment and success. Detailed documentation and effective communication of risks and proposed solutions are essential for transparency and continued support of AI initiatives.

3.2. Phase II: Data Understanding

After understanding the business requirements, the data understanding phase is crucial in the development of AI projects. This involves identifying and evaluating the necessary data, as well as defining processes for effective data collection and manipulation [47]. Data quality is a key determinant of the effectiveness of AI models, with data management consuming up to 80% of the time of an AI project [48].

3.2.1. Project Charter

The Project Charter becomes a guide for projects describing objectives, scope, schedule, budget, risks, stakeholders, and other knowledge areas of a project [23]. The first section of a project charter should detail the main objectives, which should be specific, measurable, achievable, relevant, and time-bound (SMART). For example, if the goal is to improve the accuracy of demand forecasts, it should be articulated with specific metrics, such as reducing forecast errors by a certain percentage. Project charter should identify stakeholders and their responsibilities, promoting collaboration and effective communication. It should also contain a macro-schedule and an initial cost estimate, with milestones for validation throughout the project lifecycle. Every project has risks that must be identified and addressed from the beginning, or even before work begins. Finally, a description of the data required for the AI project should be included, addressing the types of data, sources, quality criteria, acquisition and preparation processes, and the technologies and tools that will be employed [49].

3.2.2. Requirements Collection

Requirements collection is a process that involves interacting with stakeholders to understand the needs of the AI project [50]. This includes identifying, documenting, and prioritizing stakeholder needs, assumptions, and constraints. Clearly defining project objectives and goals is essential, as is collaborating with stakeholders to gain insights and perspectives. Data requirements, which are the backbone of any AI project, must be determined, including data sources, formats, and quality standards [51]. Capturing functional and non-functional requirements is crucial for the successful implementation of an AI project. Taking an iterative and agile approach to requirements gathering is beneficial for AI projects, allowing the project to adapt to evolving needs [52].

3.2.3. Internal and External Data

Identifying and utilizing data effectively is crucial in the development of AI projects [53]. The collection and integration of internal and external data forms the basis for building robust and accurate models. Internal data are generated and maintained within the organization, while external data are obtained from external sources [54]. The quality and structure of these data can vary, and a significant cleaning and preparation process is required to make them usable in AI models.

External data can enrich AI models by bringing new perspectives and additional information that is not present in internal data. However, integrating external data presents challenges, such as the need to validate data quality and ensure compatibility with existing internal data.

The effective integration of internal and external data requires a robust and flexible technology infrastructure. In addition, the use of advanced ETL (extraction, transformation, and loading) techniques and the establishment of automated data pipelines are best practices to facilitate integration and ensure that data is ready for analysis in a timely manner [55].

3.2.4. Large Volume Technologies

Managing and processing large volumes of data are natural challenges in AI projects [56]. In this context, the application of Big Data solutions requires advanced technologies to deal with scalability and complexity [57]. Big Data technologies are categorized into data storage, data processing, and data analysis. NoSQL databases, such as Mongo, Cassandra, and HBase, can be used for storage [58]. Processing is performed using distributed processing frameworks, such as Apache Hado and Apache Spark [59,60]. Tools such as Apache Fink and Dask can be used for complex real-time analysis [61,62]. The choice of Big Data technologies should be guided by the needs of the project, such as data volume, processing speed, and nature of the data [56]. The combination of different technologies may be necessary to meet distinct aspects of the data lifecycle. An effective approach involves the continuous evaluation of emerging technologies and the adaptation of data infrastructures to take advantage of the latest technologies and maintain competitiveness in the field of AI [63].

3.2.5. Quantitative and Qualitative Evaluation

The evaluation of quantitative and qualitative data is crucial in AI projects. This process involves analyzing the quantity and quality of available data, ensuring that they are suitable for the development of accurate and effective models. Quantitative evaluation refers to the analysis of measurable aspects of data, such as volume, variety, and velocity, while qualitative evaluation focuses on aspects such as accuracy, relevance, and consistency of data [64]. In this context, big data tools are used to manage large volumes of data [65]. Qualitative assessment involves analyzing the accuracy and completeness of data, as well as applying data cleaning techniques that are essential to improve data quality [66]. Data consistency is vital in qualitative assessment. Continuous assessment and monitoring of data quality constitute best practices that cannot be neglected in AI projects [67].

3.2.6. Manipulative Capability

Data manipulation capability is especially important in AI projects [56]. Technological infrastructure, including scalable storage systems and cloud computing technologies, plays a significant role [68,69]. Data processing tools such as Apache Spark and Hado allow for the efficient manipulation of large volumes of data [59,70].

The skills and competencies of the team are critical, including knowledge in data science, data engineering, and machine learning. Data governance and security are essential components, involving the implementation of policies and processes to ensure data quality, privacy, and security [71]. Automating data manipulation processes can improve the efficiency and accuracy of processed data [60].

The data understanding phase is structured in several interrelated steps, which together ensure that the data used in the AI project is of high quality and suitable for achieving the project objectives. The success of this phase depends on a meticulous and collaborative approach, involving multiple stakeholders and using innovative technologies to manage and process data effectively.

3.3. Phase III: Data Preparation

The Data Preparation Phase is a critical step in the development of AI systems, where data preparation is focused, following the clear definition of business and data requirements. This phase is dedicated to transforming raw data into a usable and high-quality format, suitable for feeding AI algorithms. The activities involved in this phase include data transformation, cleaning, aggregation, enrichment, labeling, normalization, and manipulation. These activities aim to adjust the data to the specific needs of the project, remove inconsistencies and ensure compliance with the model requirements [72].

3.3.1. Clearing and Processing

Data quality is a critical factor in AI projects and directly influences the performance and effectiveness of machine learning models. The data cleaning and processing stage involves several operations that aim to clean and shape the raw data so that they can be used effectively in analysis and modeling processes. Data quality is crucial to the performance of AI algorithms, and data cleaning seeks to remove elements that may introduce noise or bias into the results [73]. Data cleaning is an essential preliminary step that involves identifying and correcting missing or invalid data, using techniques such as imputation or removing incomplete records if they represent a small proportion of the dataset [74].

Another critical aspect of data processing is standardization, which ensures the cohesion and comparability of data from diverse sources. For example, normalizing values to a common scale is essential to ensure that data are compatible with AI models. Studies indicate that data normalization can significantly improve the performance of algorithms that are sensitive to the scale of data [75].

Data transformation is a practice that involves converting raw data into a format more suitable for modeling. This may include encoding categorical variables into numerical variables, creating derived variables, or applying mathematical transformations to reduce skewness in data distribution. These transformations aim to extract the most relevant patterns from the data and improve the performance of predictive models, thus increasing the robustness and generalizability of AI models [66].

Integrating data from multiple sources may be necessary to form a robust dataset. This may involve merging different datasets and eliminating duplicates, providing a more complete view of the data. Modern approaches, such as the use of automated data pipelines, simplify these processes, ensuring that data are combined efficiently [74]. Data cleaning and processing should be performed iteratively and adaptively, allowing for continuous adjustments as new data is acquired or when there are demands or needs for the AI project [73].

3.3.2. Quality Assurance

Quality Assurance (QA) is an essential component in data preparation for AI projects, directly impacting the accuracy and robustness of predictive models. Well-defined quality metrics are essential, covering data accuracy, integrity, consistency, and relevance. Automated monitoring tools are needed to track these metrics and identify deviations in real time, ensuring rapid interventions to maintain quality standards [2].

Data cleaning processes are implemented to ensure that data meet quality criteria before being used in AI models. In addition to technical aspects, QA also considers data integrity and ethics, ensuring compliance with privacy regulations and ethical principles, protecting sensitive information, and verifying the reputation of suppliers [76].

The continuous monitoring of data quality is essential in dynamic environments, which highlights the need for real-time monitoring tools and initiative-taking alerts to detect any degradation in quality. Collaboration between data science, data engineering, and quality management teams is emphasized as a key point for effective QA, enabling better problem identification and resolution [2].

3.3.3. Human Data Manipulation

Human data manipulation is a critical step in AI projects that require direct human intervention to ensure data accuracy and relevance [77]. This is particularly important in tasks involving data annotation for training supervised models. The quality of annotations is essential for model performance, and human intervention is necessary to provide complex judgments that automated algorithms cannot accurately perform [78]. In AI projects, data annotation can include activities such as image labeling and text categorization, which are essential for creating training datasets that are representative of production scenarios.

Effective implementation of human manipulation in a data pipeline requires a systematic approach. Annotators must be selected and trained appropriately to ensure that they deeply understand the annotation guidelines and required quality criteria. Methods such as calculating inter-annotator agreement are used to assess and improve the consistency and accuracy of annotations. In addition to initial annotation, the ongoing review of annotations is crucial to ensure accuracy and timeliness over time. Human data processing must also consider ethical and private concerns, ensuring that data are managed with care and in accordance with data protection regulations [77].

3.3.4. Data Enrichment

Data enrichment is a practice that involves enhancing and expanding existing data with additional information, aiming to increase its usefulness and effectiveness for training models in AI projects. It aims to improve the quality and diversity of data, making it more informative and robust.

A fundamental approach in data enrichment is the combination of information from diverse sources. By integrating internal data from the organization with external data, such as public or third-party sources, a completer and more enriched dataset is created. This fusion allows adding new dimensions and depth to the analyzed context. This strategy can significantly improve the robustness and performance of predictive models [79].

In domains such as image recognition, where obtaining annotated data is scarce or costly, data augmentation plays a fundamental role. This technique consists of creating new instances of data from existing ones, applying transformations such as rotation, scaling, adding noise or changing colors. The result is a more diverse and representative set. Data augmentation helps to generalize AI models by making them more robust to variations and noise in the data [80].

In addition to data combination and augmentation, the generation of synthetic data is an advanced approach. Using algorithms such as Generative Adversarial Networks (GANs), data that is statistically like real data, but not identical, is created. This technique is especially useful when real data are sensitive or difficult to obtain. GANs allow the creation of large datasets without compromising the privacy of individuals [80].

In data enrichment, it is essential to consider ethical and privacy aspects. Practices such as anonymizing personal data, obtaining appropriate consent, and continually evaluating ethical implications are essential practices. By following these ethical principles, we protect the individuals involved and strengthen trust in the projects and the organizations that conduct them. Maintaining ethical practices in the management of human data is vital in AI projects [81].

Given the above, the Data Preparation Phase is essential for AI projects, as it lays the foundation for the success of predictive models. The practices of data cleaning, processing, quality assurance, human manipulation of data, and data enrichment are not just technical components, but strategic elements that ensure the relevance, accuracy, and integrity of the data used. Each of these actions ensures that the data are accurate, relevant, and ethical, allowing AI models to function effectively and reliably. Careful attention to these processes not only optimizes the performance of models but also protects the individual rights of individuals and maintains trust in the project.

3.4. Phase IV: Project Execution

In the Project Execution Phase, the focus is on the effective construction of the system to produce a product, transforming theoretical concepts into practical reality. This begins with the conversion of raw data into a machine learning model, choosing the appropriate architecture, and using previously trained models. The selection of tools and frameworks is essential, prioritizing open-source options and configuring hardware and computing infrastructure to ensure scalability. Model training and optimization involve parameter adjustment and careful choice of algorithms, recognizing that there is no universal solution [82].

3.4.1. Pretrained Models

The use of previously trained models has proven to be an efficient and promising strategy in the development of AI systems. These models, initially trained in large volumes of data, capture rich and complex representations that can be used for specific tasks without the need for complete training from the beginning. In this context, in addition to the already established BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer) models, other models have gained relevance, such as T5 (Text-to-Text Transfer Transformer) and RoBERTa [83,84].

The BERT model, proposed by Devlin et al. [85], revolutionized natural language processing (NLP) by introducing a bidirectional pre-training technique. By considering the context of words on both sides of a sequence, BERT outperformed previous approaches that processed text in only one direction. On the other hand, the GPT-3 model, developed by OpenAI, impressed with its ability to generate coherent and contextualized text. Radford et al. [86] explain that GPT-3 was trained with a large volume of textual data, allowing it to learn contextual nuances and complex patterns. This model has been applied to intelligent conversational systems, as well as to automatic content generation.

In addition to these templates, T5 stands out for its unified approach to several NLP tasks, converting all of them into a “text-to-text” format. This versatility allows T5 to be applied to translation, summarization, classification, and other tasks related to language processing [84]. RoBERTa, an optimized variation of BERT, also demonstrates how robust training with more data and the removal of next-sentence segmentation can lead to significant improvements in performance [83].

On the other hand, the fine-tuning strategy, in turn, is a technique that involves adjusting the weights of a pre-trained model to suit a specific task. Instead of training a model from scratch, fine-tuning leverages the knowledge and weights learned by that model on a larger and more general dataset. Its advantages include reducing computational costs and leveraging state-of-the-art models without having to train them from scratch. However, it is essential to evaluate the quality and quantity of data available for training, as well as the computational resources required for the process [84].

In addition to NLP, pre-trained models also impact the application environment in computer vision. Architectures such as ResNet [87] and EfficientNet [88] were pre-trained on vast image banks, allowing object and scene recognition with minimal adjustments. Reusing these architectures not only accelerates the development of AI systems but also increases the accuracy of models in specific tasks.

The combination of pre-trained models and fine-tuning strategies offers a promising path for the development of more efficient and adaptable AI systems. The choice between specific models and the application of fine-tuning should be made considering the characteristics of the problem, the available resources, and the objectives of the application.

3.4.2. Algorithm Selection

The appropriate selection of algorithms is a decisive step in the development of AI projects. This choice directly impacts the success of the project and should be guided by the characteristics of the data and the specific requirements of the problem to be solved. The “there is no free lunch” theorem, discussed by Domingos [89], emphasizes that there is no universal algorithm for all scenarios. Therefore, it is essential to evaluate and compare different approaches to determine which offers the best performance in each specific case.

Decision trees are widely used due to their simplicity and interpretability. These algorithms divide data into subsets based on specific criteria, facilitating the understanding of the decisions made by the model. They are effective in classification and regression problems, especially when interpretability is an important requirement [90]. However, it is important to consider that the decision trees may be subject to overfitting, especially when applied to noisy or complex data.

Neural networks have revolutionized areas such as computer vision and natural language processing. These models are capable of learning hierarchical representations of data, capturing complex relationships. However, training neural networks can be computationally intensive and requires large volumes of data to achieve optimal performance [91].

Support Vector Machines (SVMs) are robust and efficient algorithms in high-dimensional spaces. These algorithms perform well when there is a clear separation between classes. In addition, SVMs are less susceptible to overfitting due to the use of maximum separation margins [92]. However, they may be less efficient on large datasets or those with many irrelevant features.

Ensemble algorithms, such as Random Forests and Gradient Boosting, combine multiple models to improve accuracy and robustness. The Random Forests algorithm combines several decision trees, reduces the risk of overfitting, and improves model generalization [93]. The Gradient Boosting algorithm builds models sequentially, correcting the errors of previous models, resulting in high accuracy, although it requires adequate regularization to avoid overfitting [94].

The selection of algorithms must be judicious, considering the specific characteristics of each problem and the advantages and limitations of each approach. The appropriate combination of these methods can lead to more robust and accurate results in AI projects.

3.4.3. Training and Optimization

The development of AI systems requires the rigorous application of model training and optimization techniques. These processes aim to adjust the model parameters to obtain the best possible performance, minimizing errors and maximizing accuracy. In this context, the search for optimal hyperparameters plays an essential role since these values significantly influence the predictive capacity of the models. By applying automation methods, traditional manual methods can be surpassed in terms of efficiency and results, providing a more robust and accurate model [95].

Two relevant tools for hyperparameter optimization deserve to be highlighted in the literature. One of them is the Optuna tool, introduced by Akiba et al. [96], a framework for hyperparameter optimization. Its study-and-test approach allows efficient exploration of different configurations. Optuna has been widely adopted by AI developers due to its ability to perform complex optimizations effectively. The other crucial tool is Hyperopt, developed by Bergstra et al. [97], which uses Bayesian optimization techniques to explore the hyperparameter space. This approach offers a powerful alternative to exhaustive or random search methods. Hyperopt has demonstrated its effectiveness in a variety of machine learning applications, making it a popular choice among researchers and practitioners in the field.

The proper training of AI models is essential to ensure that they generalize when applied to previously unobserved data. In addition, hyperparameter optimization allows the model to be adjusted for complexity, avoiding both overfitting and underfitting. The combination of large datasets increased computational power, and advanced optimization techniques has enabled the creation of increasingly accurate and efficient deep learning models. Hyperparameter optimization is vital to fine-tune the architecture of neural networks and improve their performance in complex tasks. The constant evolution of these techniques enhances the ability of AI to solve complex problems and positively impacts different areas of its application [91].

3.4.4. Machine Learning

Machine learning is at the heart of the development of AI systems, where data are used to train models capable of making predictions and decisions. Machine learning involves algorithms that improve automatically through experience [98]. In recent years, techniques such as deep learning have revolutionized businesses, enabling significant advances in areas such as computer vision and natural language processing [91].

Deep neural networks, a subfield of machine learning, use multiple layers of artificial neurons to model complex patterns in data. The combination of large datasets, increased computing power and advanced algorithms has enabled the creation of increasingly accurate and efficient AI models [91]. These advances have been fundamental to the development of practical applications such as speech recognition, automatic translation, and AI-assisted medical diagnoses, among others.

In addition to deep neural networks, other machine learning methods, such as support vector machines (SVMs), decision trees, and ensemble methods, continue to play important roles. Vapnik [92] demonstrated that SVMs are particularly effective in classification tasks with high-dimensional datasets, while Breiman [93] showed that ensemble methods, such as Random Forests, can improve the robustness and accuracy of models by combining multiple predictions.

The practice of machine learning also involves preprocessing and feature engineering techniques, which are crucial for preparing data before model training. Proper feature selection can significantly improve model performance by reducing data dimensionality and eliminating unnecessary noise [99].

The evaluation and validation of machine learning models are critical steps to ensure their effectiveness and robustness. Techniques such as cross-validation allow the model to be assessed on different subsets of data, providing a more reliable estimate of its generalization capacity [100]. In addition, metrics such as precision, recall, F1 score and AUC-ROC are commonly used to evaluate the performance of models, ensuring that they meet the specific requirements of the tasks for which they were developed.

From the above, Phase IV of the Execution of an AI project requires a focused, critical, and multifaceted effort that requires a systematic and integrated approach. In this context, each stage plays a crucial role in building efficient and effective AI systems. The success of an AI project depends on the constructive collaboration between these distinct stages presented, ensuring that the final system not only meets the initial requirements, but is also capable of evolving and adapting to new challenges and new data.

3.5. Phase V: Evaluation of Results

Evaluating the results of AI models is a crucial phase to ensure that the models meet the established business objectives. This stage involves a detailed analysis of several performance metrics, including accuracy, false positive and negative rates, and key performance indicators (KPIs), as well as aspects related to security, risks, and ethics [101]. Correctly evaluating the model’s performance is not only valid in terms of its technical effectiveness, but also strengthens the trust of stakeholders, promoting acceptance and support for the project, as well as attesting to the adherence of the solutions to business needs. Recent studies emphasize the importance of an integrated approach that considers the impact of the model on business and society, and of techniques to avoid problems such as overfitting and underfitting [102].

The results evaluation phase is structured in four main stages: the identification of overfitting and underfitting problems evaluation of accuracy and performance metrics; the analysis of training, validation, and testing curves; the evaluation against established KPIs; and the preparation of the system for continuous monitoring and future iterations [103]. Each step is crucial to ensure that the model not only works well with unseen data, but also remains relevant and effective over time, allowing for continuous adjustments and improvements as added information and data become available.

3.5.1. Underfitting and Overfitting

Underfitting and overfitting are common problems in the development of AI models that directly affect the model’s generalization ability. Underfitting occurs when the model is too simple, failing to capture the complexities of the training data, resulting in low accuracy on both training and testing data. On the other hand, overfitting occurs when the model is quite complex and overfits the training data, capturing noise and non-generalizable variability, which leads to high accuracy on training data but low accuracy on testing data [104].

Detecting and mitigating these issues involves using techniques such as cross-validation, regularization, and careful feature selection. Cross-validation is a statistical technique used to assess the generalization ability of a model based on different subsets of training data. Regularization, in turn, introduces a penalty for overly complex models, encouraging simpler and more robust solutions. Careful feature selection involves choosing variables that truly contribute to the model’s performance, eliminating those that introduce noise [104].

In addition to these techniques, it is important to continually monitor the model’s performance on new and unseen data, adjusting them, as necessary. This may include reevaluating the training data, updating regularization techniques, and reviewing the selected features. Implementing a continuous feedback process helps ensure that the model remains effective and relevant over time, responding appropriately to changes in the data environment and business requirements [102].

Effective mitigation of underfitting and overfitting is essential for the development of robust and reliable AI models. These issues, if not addressed properly, can compromise the model’s ability to provide valuable insights and make informed decisions. Therefore, an initiative-taking and well-structured approach to detecting and correcting these issues is critical to the success of any AI project [101].

3.5.2. Performance Metrics

Evaluating performance metrics is essential to quantifying the effectiveness of an AI model. The main metrics used include precision, recall, F1 score, and area under the ROC (Receiver Operating Characteristic) curve. Each of these metrics offers a unique perspective on the model’s performance. Precision, for example, measures the proportion of true positives among all predicted positive cases, while recall assesses the model’s ability to identify all true positive cases. The F1 score harmonizes precision and recall, offering a useful balance when there is a disproportion between false positives and false negatives [105].

The area under the ROC curve provides an overall measure of performance when varying decision thresholds. This metric is particularly useful for comparing different models and selecting the most appropriate one for a given dataset and business objective. ROC is a graphical representation that illustrates the performance of a classification model across multiple thresholds, providing a comprehensive view of its ability to distinguish between positive and negative classes [106].

The importance of choosing metrics aligned with specific business objectives should be emphasized for a more relevant and accurate evaluation. For example, in healthcare applications, recall may be more critical than precision, given the potential cost of not identifying a positive case. In contrast, for recommender systems, precision may be more valued to ensure that the suggestions made are relevant and of high quality [107].

A deep understanding of business objectives and stakeholder needs should guide the choice of performance metrics. This involves close collaboration between data scientists, business experts, and other stakeholders to clearly define which metrics are most important and how they will be measured and monitored throughout the model’s lifecycle [103].

3.5.3. Training, Validation, and Testing Curves

Training, validation, and testing curves are powerful visual tools for understanding model behavior throughout the learning process. The training learning curve shows how the model performs on the training data as more data are added. The validation curve reflects how the model performs on a separate dataset that was not used for training, serving as a crucial indicator for detecting overfitting. The testing curve finally evaluates the model’s performance on a completely new dataset, providing a final estimate of the model’s generalizability [108].

The analysis of these curves allows fine-tuning of the model, such as changing hyperparameters and regularization techniques, to achieve an optimal balance between bias and variance. The training curve should show a trend of continuous improvement in performance, while the validation curve should stabilize or improve without signs of significant degradation. If the validation curve diverges significantly from the training curve, this may indicate overfitting [104].

The proper interpretation of training, validation, and testing curves can guide decisions about the need for more training data, adjustments to learning algorithms, or modifications to model architecture. For example, a validation curve that shows continuous improvement may suggest that more training data would be beneficial, while a testing curve that shows inconsistent performance may indicate the need for a different approach to modeling [105].

Given the above, it is important to use these curves as part of an iterative model development process, where insights gained in one iteration inform improvements in the next. This continuous feedback loop is essential for developing robust, well-tuned AI models capable of delivering reliable and accurate results in production environments [107].

3.5.4. Performance Indicators

Evaluating the AI system against business performance indicators (KPIs) is an essential step to ensure that the model is not only technically correct but also aligned with the organization’s strategic objectives. Relevant KPIs may include metrics such as increased revenue, cost reduction, improved operational efficiency, and customer satisfaction [109]. Analyzing these indicators allows you to validate whether the model is contributing positively to the organization’s objectives, enabling the necessary adjustments before final implementation.

It is recommended to use dashboards and automated reports to continuously monitor these KPIs, facilitating a continuous and agile improvement approach. Dashboards provide a real-time view of the model’s performance, allowing you to quickly identify areas that need attention and adjustment. Automated reports, in turn, ensure that stakeholders receive regular and detailed updates on the model’s progress and impact [110].

In addition to monitoring traditional KPIs, it is important to consider social and ethical impact metrics, especially in AI projects that directly affect individuals and society. This may include analysis of algorithmic biases, fairness in the treatment of distinct groups, and transparency in automated decisions. Initiative-taking approaches to identify and mitigate these impacts can significantly improve acceptance and trust in the AI system [111].

The integration of business KPIs with technical performance metrics and social impact indicators creates a comprehensive and balanced view of the model’s effectiveness. This integrated approach not only ensures that the model is operating in line with strategic objectives, but also that it is contributing positively to society by promoting sustainability and ethical responsibility in the use of AI technologies [112].

Based on the above, the evaluation of results is an ongoing and dynamic process that requires a multifaceted approach. Integrating technical performance metrics, business KPIs, and social impact indicators ensures that the AI model is not only technically sound but also aligned with the organization’s values and goals. This process not only validates the model’s immediate success, but also sets the stage for its continued evolution, ensuring that it remains relevant and effective as business and societal needs change.

3.6. Phase VI: Deployment

The deployment phase is a stage in the lifecycle of an AI project where the developed and adjusted models are put into operation in real environments. This moment represents the transition from the controlled development environment to a practical and dynamic scenario, where the AI solution must prove its value and efficiency. Effective deployment requires not only the technical integration of the model, but also significant organizational and operational adaptation. It is necessary to ensure that the AI solution is accepted and understood by end users, providing adequate training and ongoing support. In addition, the deployment phase must ensure that business impacts are continuously monitored and managed to ensure that the expected results are achieved and sustainable in the long term [113].

The deployment approach must be meticulous and strategic, involving a series of interrelated processes that aim at the complete integration of the AI solution into the organization’s existing systems. Detailed planning and careful execution are essential to avoid operational disruptions and to ensure that all system components function harmoniously. Using modern DevOps practices and automated deployment frameworks can speed up the process and increase efficiency, reducing risks and enabling rapid response to potential problems. In addition to technical aspects, the implementation phase must consider human and organizational factors. Training programs and clear communication about the benefits and functionalities of the system can mitigate resistance and facilitate adaptation. User validation, continuous assessment of business impact, and documentation of lessons learned are essential steps to ensure that the AI solution not only meets its initial objectives but can also evolve and adapt to new demands and challenges over time [114].

3.6.1. Change Management

Change management is essential to ensure a smooth and successful transition of the AI system to the operational environment. This step involves preparing users and organization for the changes brought about by innovative technology. Training programs and workshops are recommended to enable users to use the AI system effectively. In addition, clear and continuous communication about the benefits and functionalities of the system can mitigate resistance and facilitate adaptation [115].

Implementing a structured change management plan is essential to deal with the natural resistance that arises during the introduction of innovative technologies. This includes identifying key stakeholders, assessing the impact of changes, and developing strategies to involve and educate users. Using continuous feedback methods can help fine-tune the transition process, ensuring that user needs and concerns are addressed effectively. It is also important to update organizational policies and procedures to reflect the new practices and processes introduced by the AI system [116].

The change management approach should be holistic, incorporating both technical and human aspects. It is necessary to foster a supportive environment where users can express their concerns and obtain prompt and effective responses. Transparency in the change process and active user participation can significantly increase acceptance and adaptation to innovative technology creating an organizational culture that values innovation and continuous learning is critical to the long-term success of implementing AI systems [117].

In addition, it is essential to continually monitor and evaluate the impact of implemented changes, adjusting strategies, as necessary. Documenting lessons learned during the change management process can serve as a valuable resource for future technology implementation initiatives. In this way, the organization not only improves its ability to manage change but also develops a knowledge base that can be applied in future projects [118].

3.6.2. Implementation Strategy

The implementation strategy is to ensure that the AI system is effectively integrated into the operational environment. This step includes setting up the execution environment, integrating with existing systems, and migrating the necessary data. Detailed technical implementation planning helps avoid disruptions and ensures that all system components work harmoniously [119].

The use of automated deployment frameworks and DevOps practices can accelerate and improve the efficiency of the deployment process. Techniques from ITIL-Information Technology Infrastructure Library [120] also support these processes. These practices enable continuous delivery and integration, facilitating the rapid identification and correction of problems. In addition, the implementation strategy should include a detailed rollback plan to mitigate the risks associated with deployment failure [121].

Collaboration between development and operations teams is essential to ensure a smooth and efficient transition to the production environment. It is essential that teams work together from the beginning of the project to identify potential obstacles and develop initiative-taking solutions. Continuous communication and regular feedback between teams help ensure that the AI system meets expectations and operational requirements [112].

In addition to technical aspects, it is important to consider human factors during the implementation of the AI system. Providing adequate training and learning reinforcement to end users and ensuring that they fully understand the functionalities and benefits of the new system can significantly increase the acceptance and success of the implementation. Detailed documentation of the implementation process and lessons learned can serve as a valuable resource for future implementations [114].

3.6.3. Results Visualization

The visualization of results is an essential component in the deployment of AI systems, as it facilitates the understanding of the insights generated by the model. Data visualization tools allow users to easily interpret results, identify patterns, and make informed decisions. Implementing interactive dashboards and customizable reports can significantly improve the user experience by providing a clear and intuitive view of the model’s performance [122].

In addition, effective visualization helps to communicate the value of the AI system to stakeholders, increasing acceptance and support for the project. It is important to select the appropriate visualization tools that meet the specific needs of end users and that can represent data in a clear and accessible manner. Integrating drill-down functionalities can allow for more detailed analysis and a better understanding of the underlying data. The visual presentation of results should also be accompanied by a clear and concise explanation, facilitating the correct interpretation of the data [123].

The choice of visualization tools should consider ease of use and customization capabilities. Tools that allow the customization of dashboards and reports according to user needs can increase efficiency in data interpretation. In addition, it is essential that the tools are compatible with the diverse types of data generated by the AI system and thus ensure an accurate and useful representation of the information [122].

Finally, the visualization of results should be worked on as an ongoing process, with constant updates and improvements based on user feedback. This ensures that visualization tools remain relevant and useful over time. Adopting an iterative approach to data visualization can help identify new opportunities to optimize the presentation of results, thus improving data-driven decision-making [124].

3.6.4. User Validation

User validation is a step towards ensuring that the AI system meets the needs and expectations of its end users. Involving users in the validation process can provide valuable insights into the usability and effectiveness of the system. User acceptance testing (UAT) should be conducted to identify potential issues and necessary adjustments before full launch [125].

Direct feedback from users helps to refine the system, ensuring that it is intuitive, efficient, and aligned with business objectives. Conducting continuous feedback sessions and implementing incremental improvements based on user suggestions can increase the acceptance and success of the AI system. It is also important to document the results of validation tests, and the changes made, creating a development history that can be consulted in future updates [51].

In addition, using AI systems for real-time validation of relevant information can improve the user experience, allowing for more efficient and accurate interactions. Tools that offer intuitive graphical interfaces for non-experts can facilitate the validation process and increase the adoption of AI systems in different environments [126].

Detailed documentation of the validation process is essential to maintain transparency and provides a basis for future improvements. This includes recording all changes made based on user feedback and keeping a record of the problems encountered and the solutions implemented. This practice not only helps with the continuous improvement of the system but also provides a valuable reference for the development team [127].

3.6.5. Security and Compliance

Ensuring that the AI system meets all security and compliance requirements is essential before going into operation in production environments. This step involves conducting security audits, penetration testing, and verifying compliance with applicable regulations. Implementing robust security measures helps protect sensitive data and ensure the integrity of the system [128].

Furthermore, compliance with privacy and data protection regulations is essential to avoid legal penalties and maintain user trust. Ensuring that all data collected and processed by the AI system are managed in accordance with information security using best practices is crucial. It is also important to establish processes for rapid response to security incidents and for continuously updating protective measures in the face of new threats and vulnerabilities [129].

The application of technologies such as blockchain can provide an additional level of security, ensuring the integrity and immutability of data processed by AI systems. The use of smart contracts can automate the application of security policies, making the system more resilient to attacks and fraud. The integration of such technologies should be considered as part of a comprehensive security and compliance strategy [130].

Finally, creating an organizational culture that values information security is essential. This includes ongoing training for all employees in security and compliance best practices, as well as fostering a mindset of constant vigilance against potential threats. Documentation of all security policies and procedures must be kept up to date and accessible to all members of the organization [128].

3.6.6. Business Impact

Assessing the impact of the AI system on business is essential to measuring the success of the project. This assessment involves monitoring specific key performance indicators (KPIs) defined during the early phases of the project. Analyzing the impact on operational processes, productivity, and customer satisfaction allows you to verify whether the strategic objectives are being achieved. Continuous assessment of the impact of the AI system can also identify areas for improvement and opportunities for further optimization, ensuring that the system continues to add value to the business overtime [131].

In addition, measuring the impact of the AI system must consider both tangible and intangible benefits. Tangible benefits include, for example: increased operational efficiency and cost reduction, while intangible benefits may involve, for example: improvements in decision-making and customer satisfaction. A comprehensive analysis of these aspects allows a complete view of the value generated by the AI system and helps to justify future investments in technology [132].

To conduct an accurate assessment, it is essential to use both quantitative and qualitative methods. Quantitative methods, such as data analysis and statistics, provide an objective view of the results, while qualitative methods, such as interviews and surveys, offer insights into the perception of users and stakeholders. The combination of these methods enriches the analysis and provides a deeper understanding of the impacts of the AI system [133].

Finally, effectively communicating the results obtained by the AI system to stakeholders is crucial to maintaining support for the project. Detailed reports and clear presentations that demonstrate the value added by the AI system can help justify future investments and promote a culture of innovation within the organization. Documenting the lessons learned during the assessment process can also serve as a reference for future projects [134].

3.6.7. Lessons Learned

Documenting the lessons learned during the implementation of the AI system is an essential practice to improve future projects. This step involves reviewing the practices adopted, the challenges faced, and the solutions implemented, providing a valuable knowledge base for the team and the organization. Lessons learned help identify best practices and avoid recurring mistakes, promoting a culture of continuous improvement. According to research, dynamic documentation in AI systems facilitates the understanding and evaluation of these systems, highlighting the importance of recording lessons learned throughout the implementation process [135].

Sharing these lessons with all stakeholders ensures that the knowledge acquired is used to improve future processes and strategies. In addition, documenting lessons learned can serve as a guide for new teams and projects, facilitating knowledge transfer and building a solid foundation for continuous development. Holding workshops and retrospective meetings can be an effective way to collect and disseminate these lessons [136]. The importance of proper documentation is emphasized in studies on internal auditing of AI systems, which highlights the need for a semantic framework to support accountability and auditing of these systems [137].

To ensure that lessons learned are effectively documented and shared, it is important to adopt a systematic approach. This includes creating standardized templates for collecting information, defining clear responsibilities for documentation, and implementing a centralized system for storing and accessing this information. In addition, it is essential to foster an organizational culture that values continuous learning and constant improvement. The integration of transparent and purpose-driven data cards has proven effective for documenting datasets, which can be adapted to document lessons learned in AI projects [138].

Finally, lessons learned analysis should be an ongoing and iterative process. This means that in addition to documenting lessons learned at the end of a project, it is important to revisit these lessons periodically to assess their relevance and applicability in new contexts. Integrating lessons learned into the lifecycle of AI projects can significantly contribute to the success and sustainability of AI initiatives in an organization. Studying the sociotechnical envelope of AI reveals that continuous iteration between human and AI agents is crucial to the success of organizational implementation [139].

The deployment phase of AI systems, as described in this section, involves thorough planning and a strategic approach to ensure the long-term effectiveness and sustainability of these systems in operational environments. Each subsection has addressed critical aspects ranging from change management, implementation strategy, to visualization of results and validation by users. These elements are essential to ensure that the AI solution not only meets technical and operational needs but is also well received and understood by end users, providing tangible and ongoing value to the organization.

At the end of this implementation process, documenting lessons learned and continually assessing the system’s impact on the business play crucial roles in fostering a culture of continuous improvement and innovation. Integrating an integrated approach that considers both technical and human factors, as well as clearly communicating the results with stakeholders, is key to the success and adaptation of the AI system to new demands and future challenges. In this way, the AI project lifecycle is completed with a solid foundation for future developments and optimizations, ensuring that the organization reaps the full benefits of implementing AI solutions.

3.7. Phase VII: Operation

The operation phase in an AI project represents the use of the developed system in the real production scenario, where it begins to interact directly with organizational processes. This phase is crucial, as it requires the implementation of robust monitoring, version management, and governance practices to ensure that the system remains aligned with the organization’s strategic objectives. Continuous monitoring is essential to detect and correct deviations or problems, ensuring that the system continues to function as expected and deliver value. Version management is equally important, as it allows updates and improvements to be conducted in a controlled manner and with minimal impact on operations. Proper governance ensures that decisions about the operation of the system are made in accordance with organizational policies and guidelines, ensuring transparency and accountability throughout the AI system operation process [140].

In addition, the operation of AI systems should be seen as a dynamic process that requires continuous cycles of improvements based on real feedback and changes in business needs. The implementation of iterative adjustments, informed by data collected during operation, allows the model to evolve and adapt to new challenges, maintaining its relevance and effectiveness over time. For example, the integration of automated monitoring and continuous improvement techniques, such as the use of CUSUM (Cumulative Sum Control Chart) charts, which are tools used in statistical process control, can be effective in maintaining system performance and stability under varying operating conditions [141]. In this way, the operation phase not only sustains the value of the AI system, but also promotes its continuous evolution, ensuring its positive impact within organization [60].

3.7.1. Production Monitoring

Production monitoring is an essential practice to ensure that AI systems operate as expected and continue to generate value for the organization. The implementation of continuous monitoring tools allows for the early detection of anomalies, enabling rapid intervention that mitigates negative impacts on system performance. As highlighted in the literature, the efficiency of an AI system is highly dependent on the ability to identify operational problems in a timely manner, thus ensuring its robustness and reliability [142]. The use of appropriate metrics and advanced monitoring technologies enables a detailed view of the system’s behavior, which is essential to maintain its effectiveness in dynamic and complex environments.

Furthermore, continuous monitoring facilitates the adaptation of the AI system to changes in the operational environment, since it allows the adjustment of the model’s settings and parameters based on real performance data. This adaptability is crucial in scenarios where operational conditions can vary significantly over time. The study by Milić and Kožičić [143] reinforces that the adaptability of AI systems is a determining factor for their longevity and relevance, since the ability to quickly adjust to new circumstances ensures the maintenance of their optimal performance.

The effectiveness of production monitoring is not limited to detecting failures but also extends to the continuous assessment of the system’s impact on organizational processes. This includes measuring model performance in terms of key performance metrics (KPIs), which allows verifying whether the organization’s strategic objectives are being met. According to [142], effective monitoring should be able to identify not only technical failures, but also deviations from expected results, enabling initiative-taking adjustments that align system performance with business objectives.

For production monitoring to be truly effective, it is essential that it be integrated into a well-structured MLOps strategy, which encompasses everything from data collection to the implementation of continuous improvements to the system. The integration of MLOps practices, as suggested by Bodor et al. [144], is important to sustain the operation of AI systems and ensures that they remain agile, efficient, and aligned with the organization’s constantly evolving needs.

3.7.2. Success Measuring

Measuring the success of an AI system in operation is a complex process that involves evaluating metrics, techniques, and analyzing the impact of the system on the organization’s strategic objectives. To ensure that the system is delivering the expected value, it is essential that the Key Performance Indicators (KPIs) defined in the initial phases of the project are determined and addressed throughout the entire journey. These indicators must be carefully chosen to reflect not only the technical performance of the model, but also its contribution to business results. The definition and continuous monitoring of these KPIs are crucial to validating whether the system’s performance is in line with the organization’s expectations and goals [145].

An integrated approach that considers both technical and business metrics is essential for the long-term success of AI systems. This is because a model that presents excellent technical results may not be aligned with the organization’s strategic objectives if its impact on the business is not evaluated. Therefore, success measurement must include a comprehensive analysis that addresses both the efficiency of the model and its effectiveness in achieving the desired organizational results through OKRs–Objectives and Key Results). Recent studies highlight that by focusing only on technical metrics, there is a risk of losing sight of the importance of strategic alignment [146].

In addition, integrating an MLOps (Machine Learning Operations) approach into success measurement helps ensure that the AI system remains effective over time. MLOps enables automation and continuous monitoring of model performance in production, facilitating rapid adjustments in response to changes in data or business needs. This practice is vital for the system to not only meet the KPIs initially established, but also to continue to evolve in line with the organization’s strategic objectives [144].

Therefore, for the measurement of the success of an AI system to be complete and effective, it is necessary to adopt an integrated vision that considers both technical and business aspects, combined with a robust MLOps strategy. This combination is essential to ensure that the AI system not only meets initial expectations but also continues to generate value throughout its useful life, adapting to changes in organizational and market needs.

3.7.3. Versioning and Updates

Managing code versions and updates is an essential process for maintaining the effectiveness of AI systems over time. This process involves the application of rigorous versioning practices, which ensure that each recent version of the system is rigorously evaluated before being deployed in the production environment. This approach minimizes the risk of introducing failures and ensures that the system’s functionalities are maintained or improved in a controlled manner. According to Merhi [33], the application of Continuous Integration and Continuous Deployment (CI/CD) techniques in AI systems facilitates the implementation of multiple versions simultaneously, which speeds up production cycles and minimizes interruptions.

Implementing planned updates is essential to ensure that the AI system remains relevant and effective in the face of constant technological evolution. Updates must be carefully scheduled and executed, based on an in-depth analysis of the system’s needs and the environment in which it operates. This includes identifying new features that can improve system performance, as well as fixing vulnerabilities and optimizing existing processes. Rula et al. [147] highlight that adopting large-scale update strategies, such as blue-green deployment, significantly reduces system downtime, ensuring a smooth and efficient transition between versions.

In addition, considering rollback strategies is an essential practice in version and update management. These strategies allow the organization to revert to a previous version of the system in case of significant update failures, thus minimizing the negative impact on operations. The study by Sakthidevi et al. [148] proposes a system architecture that includes a model repository component, which facilitates versioning and continuous updating of AI models and ensures operational continuity and data security.

Therefore, the effective version and update management requires a structured and meticulous approach, ranging from planning and implementing new versions to preparing for potential setbacks. By adopting these practices, organizations can ensure that their AI systems continue to operate efficiently and securely, continually adapting to new market demands and technological innovations.

3.7.4. AI Governance and Compliance

Governance and compliance in the context of AI systems are elements to ensure that the operation of these technologies is in line with current regulations and ethical best practices. The implementation of robust governance frameworks is essential to mitigate risks and ensure transparency at all stages of the system’s lifecycle. According to Ortega et al. [149], an effective governance framework should incorporate principles such as human-centered design and continuous governance throughout the product lifecycle, ensuring that ethical issues are addressed from the beginning of development to the ongoing operation of the system.

Compliance with regulations and adherence to ethical practices play a crucial role in building stakeholder trust in AI systems. This is particularly important in contexts where decisions made by AI systems can have significant impacts on people’s lives. As pointed out by Camilleri [150], the intersection between AI governance and corporate social responsibility should be emphasized, highlighting the need for frameworks that not only ensure compliance but also promote responsible and sustainable use of AI in production environments.

Effective governance not only ensures compliance with regulations but also facilitates the scalability and safe adoption of AI systems. Eitel-Porter [151] highlights that the implementation of governance frameworks overseen by ethics boards, supported by appropriate training programs, is vital to reduce the risks associated with large-scale AI implementation. Such frameworks help align technological development with the organization’s ethical values and principles, promoting a culture of accountability and transparency.

Therefore, the creation and implementation of AI governance frameworks that integrate ethical and regulatory considerations are essential to ensure the responsible and safe operation of AI systems. These frameworks must be dynamic and adaptable, capable of evolving as new regulations and ethical challenges emerge, thus ensuring continued compliance and ethical use of AI technologies.

3.7.5. Scalability and Resource Management

Scalability and resource management are basic aspects in the operation of AI systems, especially in environments that require high flexibility and adaptability. To ensure that an AI system can scale efficiently without compromising its performance or generating excessive costs, it is necessary to implement robust resource management practices. According to Satyanarayan [152], strategies such as the use of machine learning algorithms for intelligent workload planning and demand forecasting can significantly improve the efficiency of resource management in cloud computing systems.

The use of cloud solutions is one of the pillars for managing scalability in AI systems in dynamic environments. Tuli et al. [153] highlight the importance of co-designing resource management decisions in cloud computing environments, showing improvements in execution cost, inference accuracy, energy consumption, and response times with the adoption of the SciNet method. The combination of optimization techniques with cloud technologies allows AI systems to scale efficiently, meeting variable demands without compromising performance.

In addition, the adoption of real-time management tools, such as SPACE4AI-R, described by Filippini et al. [154], offers innovative solutions for the placement of AI application components and the scalability of resources in computing continuums. Such tools use stochastic search algorithms to optimize resource allocation, resulting in significant cost reductions and efficient performance in heterogeneous environments.

Finally, the integration of edge and fog computing techniques, as discussed by Walia et al. [155], is essential for adaptive resource management in distributed computing paradigms, especially in IoT applications. These techniques enable AI systems to manage resources effectively, ensuring they can scale quickly to meet operational needs while maintaining efficiency and reducing latency.

3.7.6. User Experience (UX) in AI Operations

User experience (UX) in AI systems is a critical factor in ensuring not only acceptance but also system efficiency. A well-designed UX can increase user confidence by facilitating interaction and ensuring that the system meets their expectations in an intuitive way. Recent studies have shown that a good UX can significantly increase user satisfaction and system effectiveness, as demonstrated by Sivakumar et al. [156], who explore the impact of artificial intelligence on human-computer interaction design, showing how a well-designed interface can improve the overall user experience.

Continuously monitoring and improving the UX is essential to ensure that the system remains relevant and useful over time. This approach includes continually collecting user feedback and analyzing interactions to identify areas for improvement. As noted by Kuang [157], collaboration between UX evaluators and AI, through visualizations and conversational assistants, can improve productivity and confidence in UX evaluation, promoting a more satisfying and efficient experience.

In addition, the adaptability of the system to the needs of users is a vital aspect to be considered during the operation of AI systems. Xu [158] suggests that the application of interactive learning methods can personalize the user experience, adjusting to their preferences and behaviors over time. This customization not only improves user satisfaction but also increases the operational efficiency of the system.

Therefore, considering UX as a central aspect during the operation of AI systems is essential to ensure that the technology is well received and used effectively by end users. Investing in user-centered design practices, continuous monitoring, and personalization can result in more intuitive and effective systems, capable of meeting the changing demands of users and improving the overall interaction experience.

3.7.7. Continuous Improvement

Continuous improvement is a fundamental practice in the operation of Artificial Intelligence (AI) systems, ensuring that these systems can evolve and adapt over time. This approach is crucial to ensure that the system remains relevant and effective in a constantly changing business environment. According to Wu et al. [159], the practice of continuous improvement in AI operations, especially in continuous learning environments, highlights the importance of incremental adjustments that, over time, result in an exponential increase in the system’s proficiency, aligning with the dynamic needs of users and the market.

Implementing feedback loops that identify areas for improvement is essential to maintaining the system’s effectiveness and relevance in constantly evolving business environments. These feedback loops allow AI systems to continually adjust and improve their operations, ensuring that responses to user needs and market conditions are rapid and accurate. According to Veprikov et al. [160], the introduction of feedback loops in machine learning systems helps to address issues such as error amplification and concept drift, enabling continuous improvement and a deeper understanding of long-term impacts on the operational environment.

Continuous improvement in AI systems should be seen as an iterative and collaborative process, where constant learning and adaptation are key to sustaining the relevance of the system. Integrating user feedback directly into AI-based workflows facilitates the organic and continuous improvement of AI solutions and ensures that these tools can better meet user needs over time. Liu et al. [161] highlight that creating autonomous frameworks that integrate user feedback into large-scale AI systems is an effective approach to continuously improve the user experience and system functionality, promoting a process of constant evolution that responds directly to changing user needs.

Therefore, incorporating continuous improvement practices with feedback loops and an iterative approach is essential for AI systems to remain effective and aligned with the ever-changing needs of users and the market. This collaborative and adaptive approach ensures that AI systems can evolve over time while maintaining their relevance and effectiveness in a rapidly changing business environment.

As explained above, the operational phase in AI systems is the integration of robust monitoring practices, version management, governance, scalability, and continuous improvements, not only ensuring the efficiency and relevance of the system over time but also ensuring that it is aligned with the strategic objectives of the organization. This phase is where the system truly proves its worth, interacting directly with organizational processes and demonstrating its ability to generate positive impact. The success of this phase depends on the careful and meticulous implementation of each of the aspects discussed, from continuous adaptation based on real feedback to maintaining ethical practices and regulatory compliance.

The ability of an AI system to evolve and adapt in a dynamic environment, through efficient governance, continuous updates, and a user experience that meets the needs of end users, is what determines its long-term success. By taking a holistic and integrated approach that considers both technical and strategic aspects, organizations can ensure that their AI systems not only meet current expectations, but also proactively adapt to changing market and business needs. In this way, the operational phase becomes not just a time of transition, but a continuous period of growth, learning, and improvement that is essential to the sustainability and long-term success of AI systems within the organization.

4. Results

The MAISTRO methodology was used in a simulated way in a created exercise with the students of SENAC-SP University Center, which included an application in a supermarket, PreçoBomAquiSim (fictitious name), to increase revenue and improve the experience of its customers. The results obtained with this proof of concept are described below.

4.1. Application of MAISTRO Methodology in PreçoBomAquiSim Supermarket

The application of the MAISTRO methodology in the PreçoBomAquiSim supermarket project aimed to develop an intelligent recommendation system, aiming to improve the customer experience and increase revenue. This hypothetical case study serves as an experiment to demonstrate the effectiveness of MAISTRO in a realistic retail scenario, where personalization and data analysis play fundamental roles. The partnership with MAISTRO allowed for a structured, agile, and ethical approach throughout the development of the project, aligning business strategies with the technological capabilities of AI.

During implementation, MAISTRO guidelines were applied in all phases of the project. In the initial phase, business needs were understood by identifying PreçoBomAquiSim’s strategic objectives and collecting relevant internal and external data. The data understanding phase enabled the collection and evaluation of information from various sources, ensuring its quality and integrity. Next, in the data preparation phase, data cleaning, processing, and enrichment techniques were performed, in accordance with the practices recommended by the methodology.

The efficient integration between the technical and business teams was another highlight of the MAISTRO application, resulting in a recommendation system aligned with the company’s expectations. The methodology allowed development to be guided by a logical and organized sequence, adapting to the changes and challenges presented throughout the process. The results, which will be discussed in the following sections, demonstrate the success of the implementation and the benefits obtained with the use of MAISTRO.

4.2. Results Obtained from the Simulation of the Methodology MAISTRO

The application of MAISTRO in the PreçoBomAquiSim project generated significant benefits. The new system provided a 20% increase in sales of recommended products, while the average ticket per customer increased by 15%. In addition, customer satisfaction improved, as measured by feedback surveys. These numbers indicate MAISTRO’s effectiveness in achieving the business objectives proposed and the key performance indicators established at the beginning of the project.

A comparative analysis of AI tools and techniques (TensorFlow 2.18.0 was compared to PyTorch 2.5.1 and Scikit-learn 1.6.1) and due to its robustness, scalability, open source, and active community, in addition to the enormous number of resources and support, TensorFlow was preferred for use in that project.

The techniques applied, such as enrichment with demographic data from external sources, and the selection of appropriate machine learning algorithms (Neural Collaborative Filtering—NCF, in comparison to the Wide and Deep Learning and AutoRe due to the need to use pre-trained models and to capture the specificities of customers and PreçoBomAquiSim supermarket products), were decisive for the system’s performance.

During the evaluation phase, overfitting problems were identified and corrected, ensuring that the developed model presented superior results in different scenarios. The deployment process was conducted safely and in compliance with regulations, strictly following the governance guidelines defined by MAISTRO.

These results confirm MAISTRO’s potential for AI projects, especially in retail contexts where personalization is essential. Detailed analysis of metrics and comparisons with traditional methodologies will be addressed in the next section, highlighting MAISTRO’s advantages.

4.3. Comparative Analysis with Traditional Methodologies

When comparing MAISTRO with other methodologies such as CRISP-DM and OSEMN, we observed that MAISTRO presents superior flexibility and adaptability to the specific needs of AI projects in retail. While traditional methodologies tend to be more linear and rigid, MAISTRO allows for continuous iterations and adjustments according to the demands of an agile project approach (Figure 2). In the case of PreçoBomAquiSim, this flexibility was crucial to deal with changes in data requirements and the evolution of consumer behavior.

A natural sequence is established between the processes of the methodology stages so that stage 2 (data understanding) is dependent on the establishment of business needs (stage 1) and so on until stage 7 (operation) can use the solution implemented in stage 6 (deployment). However, given the nature of artificial intelligence projects, recurrences between non-consecutive stages are also sometimes necessary, for example: after determining the results in stage 5 (evaluation of results), it may be necessary to collect higher quality data in stage 2 (data understanding), or to make some refinement in the execution of the project (stage 4) to obtain accuracy more suited to the needs of the problem in question. The same may occur in stage 4 (project execution), which may require more adequate data preparation and return to stage 3 (data preparation) on a recurring basis or even return to stage 2 (data understanding) to obtain data from a more complete source, until a satisfactory level is achieved. And so on, making the MAISTRO methodology flexible, adaptive, efficient, and effective in generating continuous value for the business. Another important differentiator of MAISTRO was the integration of ethical and governance considerations from the beginning of the project. In traditional methodologies, these aspects are often neglected or addressed superficially. In the PreçoBomAquiSim project, ethics and data security guidelines were strictly followed, ensuring compliance with regulations such as the LGPD-Brazilian General Data Protection Law [162]. MAISTRO also facilitated communication between multidisciplinary teams, promoting effective strategic and technical alignment, adherence to business strategy and respect for the unique cultural and organizational aspects present at PreçoBomAquiSim supermarket.

This comparison reinforces the relevance of MAISTRO as a modern and efficient methodology for developing AI projects. The following sections will discuss the challenges encountered and the recommendations derived from these experiences.

4.4. Limitations and Challenges Identified

Although the results obtained with that project were positive, we faced some challenges throughout its execution. One of the main ones was the need to train the team in the use of the MAISTRO methodology. As the methodology is new, the team members needed time to adapt to their guidelines and practices, within an expected learning curve. Another technical challenge was the integration of large volumes of data from internal and external sources, which required robust processing and storage solutions.

MAISTRO’s guidelines for data preparation and manipulation were strictly followed, mitigating many of these challenges, but the technical complexity inherent in the project required additional effort from the IT and data engineering teams. Furthermore, during the operational phase, it was necessary to establish continuous monitoring and version management processes, ensuring the long-term efficiency of the recommendation system.

Cultural and ethical aspects and uses and customs specific to each organization must be integrated into the processes of the various stages of the methodology, and this requires the collaborative participation of several departments and governance based on best management practices. A project office could have coordinated these actions in the case of the PreçoBomAquiSim Supermarket project. Understanding these limitations is essential for future improvement of the methodology and for better management of AI projects involving large volumes of data and integration of emerging technologies. To consolidate these assessments and allow reflection on their potential and challenges, we created the SWOT matrix on the MAISTRO Methodology (Figure 3).

Concerns about issues related to ethics, transparency and social and environmental responsibility of projects, alignment with business objectives, sustainability, adaptability, and use of best governance practices, together with flexibility, scalability, and multidisciplinary integration and collaboration constitute strengths of the MAISTRO methodology.

The growing demand for artificial intelligence solutions creates enormous opportunities for the use of a complete and integrative methodology such as the one proposed in this article. Machine learning pipeline automation tools and coupling with emerging technologies drive the adoption of MAISTRO, facilitating the implementation of continuous processes and the expansion to large-scale natural language models (LLM), augmented retrieval (RAG) and graph-based machine learning (Graph AI). Partnerships with universities, research centers, and technology companies can be established to validate and refine the approach. Critical, highly regulated sectors of the economy such as finance, healthcare, and industry require robust methodologies to ensure compliance and security in their artificial intelligence models, which represent a promising possibility for the application of MAISTRO.

On the other hand, since it is a recent proposal that involves multiple disciplines and management practices, the initial adoption of the methodology still requires time to be known, tested, improved, and disseminated, which is a challenge for teams without previous experience with structured processes in artificial intelligence projects. This comprehensive scope of the various participating areas generates a long learning curve and refinement cycles to reach prominent levels of maturity, which makes the relative costs of its implementation high. The application of all phases and recommendations implies the availability of version control platforms, experiment management, automated data flows, and robust storage solutions, which can be an obstacle for organizations that do not have adequate infrastructure.

The speed at which innovative technologies emerge, and others become obsolete, requires that any methodology be constantly revalidated and updated. However, resistance to change needs to be overcome, especially in organizations with rigid and inflexible processes. New regulations and constantly changing legislation generate the need for adjustments to maintain the relevance and applicability of models, and the emergence of other proposals like MAISTRO completes the threats identified in the SWOT diagram.

Recommendations based on these aspects are discussed below.

5. Discussion

5.1. Interpretation of Results in Relation to MAISTRO Objectives

The results obtained in the fictitious case of the PreçoBomAquiSim supermarket indicate that the MAISTRO methodology achieved the proposed objectives. The increase in sales and customer satisfaction demonstrates that MAISTRO is effective in aligning AI solutions with business needs. The iterative and flexible structure of the methodology allowed continuous adjustments throughout the project, ensuring that challenges were addressed effectively, with good customer experience among the stakeholders of the project.

The application of rigorous techniques in algorithm selection and data quality control played a fundamental role in the success of the project. In addition, compliance with ethical and governance guidelines ensured that the recommendation system was adherent and compliant with current regulations and standards. This alignment between business strategy and AI allowed the MAISTRO methodology to generate value for the PreçoBomAquiSim supermarket.

Therefore, the interpretation of the results confirms the effectiveness of MAISTRO as a robust and adaptable methodology for the development of AI systems.

5.2. Implications for AI Project Development

The experience gained from applying MAISTRO in this project suggests that the methodology can have a significant impact on the way AI projects are developed. Its integrated approach encompasses both technical and managerial aspects, which is crucial in the current context, where complexity and uncertainty are constant in AI projects.

The flexibility provided by MAISTRO facilitates risk management, enabling more informed and agile decision-making. In addition, the methodology emphasizes governance and transparency, ensuring that projects are conducted ethically and responsibly. This is especially important in sectors such as retail, where privacy and the responsible use of customer data are fundamental to maintaining trust. The strengths of the methodology highlighted in the SWOT diagram in Figure 3 establish a new level in the development of artificial intelligence systems. On the other hand, the weaknesses of the methodology and the threats visualized in the SWOT diagram in Figure 3 need to be addressed and resolved satisfactorily by establishing training, applying the framework in different business areas and achieving its maturity based on real cases.

5.3. Theoretical and Practical Contributions of Methodology

From a theoretical perspective, MAISTRO offers a valuable contribution by integrating agile practices with ethical governance and security considerations. This distinguishes it from other traditional methodologies, such as CRISP-DM, SEMMA, OSEMN, and KDD, which do not address these issues comprehensively.

In practical terms, the implementation of MAISTRO in the PreçoBomAquiSim supermarket project demonstrated its viability in a realistic retail context. The methodology’s best practices and well-defined phases provided a solid foundation for the development of an efficient recommendation system and satisfactory results.

At a time when organizations are discussing their processes and adherence to disruptive practices due to digital transformation and the advent of innovative technological solutions, the opportunities reported in the SWOT diagram in Figure 3 represent motivators for driving this study in this direction.

5.4. Future Perspectives and Related Work

Prospects for MAISTRO include its application in diverse sectors such as healthcare, finance, and manufacturing, where AI systems are gaining increasing importance [40]. In addition, future research can explore the integration of MAISTRO with MLOps and DevOps practices, expanding its applicability in continuous AI operations [1].

Additionally, system development practices using AI Generative Pre-trained Transformer (GPT) should be considered, as well as LLOps (Large Language Model Operations) practices that propose models involving four levels of maturity [163].

The methodology proposed here needs to be applied in different sectors to be refined. Thus, the processes contained in each phase can be validated and a breakdown involving models, methods, techniques, and tools can be established, creating favorable conditions for preparing training on the appropriate use of the framework.

6. Conclusions

6.1. Summary of MAISTRO’s Contributions

The MAISTRO methodology has proven to be an effective and comprehensive approach for developing AI projects. Its application in the PreçoBomAquiSim project resulted in an intelligent recommendation system that improved customer experience and increased the company’s revenue. Previous studies have already pointed to the importance of methodologies that integrate technical and ethical aspects in AI development, which was corroborated by this practical application [40]. MAISTRO stood out by providing a flexible and iterative structure, perfectly aligning with the specific needs of the project [164].

MAISTRO’s contributions include the integration of agile practices, a focus on data quality, and the consideration of ethical and security aspects, which have often been neglected in other traditional methodologies, such as CRISP-DM and OSEMN [165]. These combined elements facilitated the development of AI systems that not only deliver solid results but are also sustainable and aligned with the organization’s strategic objectives and social expectations.

This synthesis reinforces the relevance of MAISTRO as a modern methodology, capable of meeting the demands of AI projects in dynamic and complex environments, promoting the alignment between technology and organizational and ethical values [1].

The methodology proposed in this work offers several significant contributions to the field of AI. It presents a flexible framework, covering all phases of the development cycle of an Artificial Intelligence project, which serves from the complete understanding of business needs to its implementation and operation, with the application of best project management practices and advanced AI techniques and tools. It also incorporates the treatment of ethical and transparency issues that are essential for the acceptance and sustainable implementation of AI systems. Furthermore, by offering a flexible and adaptable framework, the methodology can be applied in a variety of organizational contexts, which makes it a valuable tool for both academic and industrial.

6.2. Final Considerations

The application of MAISTRO in the PreçoBomAquiSim project demonstrated that well-structured methodologies are fundamental to the success of AI systems. In the context of this project, MAISTRO effectively addressed the technical, operational, and strategic challenges, promoting the collaboration and adaptability necessary to achieve the proposed objectives [166].

Although challenges arose throughout the project, such as the need for team training and the integration of large volumes of data, these difficulties did not compromise MAISTRO’s effectiveness. On the contrary, they reinforced the importance of following a well-defined and adaptable methodology to overcome obstacles and ensure the delivery of satisfactory results [40].

These considerations demonstrate that MAISTRO is a promising tool for organizations seeking to implement AI in an efficient, responsible manner, and aligned with their long-term goals, especially in sectors that require the personalization of services with large volumes of data, such as retail [164].

6.3. Suggestions for Future Work

It is suggested that future work explore the application of MAISTRO in a variety of sectors and types of projects, evaluating its effectiveness in different organizational contexts. This will allow a broader assessment of its adaptability and impact on projects involving different volumes of data, organizational structures, and specific regulations.

In addition, the development of tools and resources that support the implementation of MAISTRO can facilitate its large-scale adoption. Integration with MLOps and DevOps practices is another promising area of research, allowing MAISTRO to be used in continuous AI operations, increasing its applicability in business environments that require constant updates and system monitoring.

Comparative studies with other AI development methodologies, as well as the analysis of success stories, can contribute to the continuous improvement of MAISTRO. Collaboration between academia and industry will be essential to expand the knowledge and practices associated with the methodology, creating a virtuous cycle of learning and innovation.

Although the present study offers a broad view of the structure of the seven phases of the proposed MAISTRO methodology, a more granular analysis is necessary so that the methodology can be used by everyone. Detailed process-level studies in ITTO format (Inputs; Tools and Techniques: Outputs) will appear in later publications, exploring each of the 39 processes listed in the framework above. This will expand the documentation of the methodology and allow for its broad application.

7. Patent

The work reported in this manuscript proposes an innovative and unprecedented methodology on the market, which can be widely used in both industry and academia, providing great gains to its users. The authors of this article therefore decided to name it the MAISTRO-Agile Methodology for AI Projects to Overcome Challenges with Efficiency and Transparency. In this way, the MAISTRO brand was registered with the responsible Brazilian government body-INPI (National Institute of Industrial Patents).