Chatbot Design and Implementation: Towards an Operational Model for Chatbots

Abstract

The recent past has witnessed a growing interest in technologies for creating chatbots. Advances in Large Language Models for natural language processing are underpinning rapid progress in chatbot development, and experts predict revolutionary changes in the labour market as many manual tasks are replaced by virtual assistants in a range of business functions. As the new technology becomes more accessible and advanced, more companies are exploring the possibilities of implementing virtual assistants to automate routine tasks and improve service. This article reports on qualitative inductive research undertaken within a chatbot development team operating in a major international enterprise. The findings identify critical success factors for chatbot projects, and a model is developed and validated to support the planning and implementation of chatbot projects. The presented model can serve as an exemplary guide for researchers and practitioners working in this field. It is flexible and applicable in a wide range of business contexts, linking strategic business goals with execution steps. It is particularly applicable for teams with no experience in chatbot implementation, reducing uncertainty and managing decisions and risks throughout the project lifecycle, thereby increasing the likelihood of project success.

1. Introduction

The chatbot is often referred to as a “conversational interface” or “conversational user interface”, indicating a technology that enables communication between people and information systems (ISs), using a human language [1]. While chatbots are usually associated with text-based communication in a messenger or chat window [2], conversational interfaces can also include voice assistants integrated into smart or wearable devices, smartphones, social robots, autonomous vehicles, and other devices. The Cambridge Dictionary [3] defines a chatbot as a “computer program designed to have a conversation with a human being, especially over the internet”. In practice, modern chatbot platforms vary in behaviour and complexity and can be delivered as standalone applications or as a service based upon another technology platform.

Over the past decade, the growth in the successful use of chatbots has often been linked to advances in the development of AI technology. Virtual assistants have been developed for a range of different functions but all use AI technologies and natural language processing (NLP) algorithms [4]. For the past decade, new approaches based on non-linear neural networks have become increasingly evident, and the use of recurrent neural networks (RNNs) has demonstrated significant success in solving typical NLP problems [5]. Bernardini et al. [6] have shown how competitive pressures in global markets and rapid advances in hardware performance—making new technologies increasingly accessible—have led to a significant increase in interest in the potential of AI. In 2018, Devlin et al. [7] introduced BERT (Bidirectional Encoder Representations from Transformers). This model significantly improved conversational interface performance, and for chatbot developers, BERT provided new opportunities to solve applied NLP tasks typical of most chatbots: user intent recognition, question answering, and classification of what? [8].

The subsequent improvements in Large Language Models (LLMs) and related text generative models have had a significant impact on the development of chatbots. The latest and most significant development in the field of NLP is the creation of the GPT (Generative Pre-trained Transformer) model by OpenAI in 2018. It shows excellent results in tasks involving the processing of human-language text. The GPT model is trained on a wide range of open text data, including Wikipedia, and is able to generate coherent text of almost indistinguishable quality from human-written text [9]. GPT-4, released in March 2023, is the latest and most advanced version of the GPT family of models to date. It is a multimodal model that works not only with textual information but also with images and has pioneering abilities to handle a wide range of real-world tasks performed by humans. Benchmarks show that the model scores higher than the average human on many popular standardised exams in maths, physics, chemistry, statistics, biology, and other fields [10]. Today, chatbots and generative AI are widely used by large enterprises as well as small- and medium-sized enterprises (SMEs) in many industries. They commonly automate processes in marketing and sales, customer support, human resources, and other areas [11,12].

Chatbots are attractive to businesses as they facilitate customer service through a conversational style that is natural and intuitive for humans [13]. Despite the rapid development of technology in this area, the increasing capabilities of AI algorithms and the growing number of successful real-world applications, chatbot implementation projects face a number of difficulties. The expectation of quick and immediate success from chatbot implementation that many companies had in the middle of the last decade has been replaced by a more sceptical attitude towards the potential of virtual assistants [14]. Practical experience has shown that chatbots cannot be used effectively in all situations, whilst creating a high-quality service requires significant investment. The technological complexity and unpredictability of project outcomes present additional difficulties for implementation teams, who often lack practical experience and theoretical knowledge [15,16,17].

This article explores the range of issues that confront those involved in chatbot implementation projects. Building upon earlier research by the authors [18], a new model is formulated and presented, which can support IT managers (from large- and medium-sized companies), IT integrators, and other industry practitioners and specialists involved in the creation and deployment of chatbots. The research identifies common patterns and rules that can be useful for a wide range of organisations whose activities are related to customer service, which can be transformed into a client–chatbot interaction, benefiting both the company and the client. More specifically, this article addresses the following research questions (RQs):

RQ1. What are the main challenges which companies face when implementing chatbot projects?

RQ2. What are the critical success factors (CSFs) that contribute to the successful implementation of chatbots in companies?

RQ3. What operational model can support the chatbot implementation process and help responsible managers deliver expected chatbot project outcomes?

Following this introduction, the article comprises five further sections. In Section 2, the different aspects of the research method are discussed. This is followed by a review of the relevant literature and the development of the provisional conceptual framework for the study in Section 3. Section 4 then sets out and discusses the research results and addresses the research questions. This is followed by a Discussion section which not only examines the possible application of the model in practice but also looks at the wider issues relating to the future deployment of chatbots and AI in industry. Finally, Section 6 considers the limitations of the study, and points up possible future avenues of research in this field.

2. Research Method

2.1. The Case Study Approach and Research Philosophy

The research centres on a case study of a chatbot implementation in one of the world’s leading steel producers (presented here as an alias—ESteelCo) that took place in the two-year period from March 2021 to March 2023. The ESteelCo case study provided an in-depth inquiry into the complexity of chatbot implementation and was the basis for the generation of new knowledge in this subject area. The company’s global strategy has a strong client focus and the company and its clients employ popular instant messengers (e.g., WhatsApp, Telegram) as the preferred means of communication. In addition, the company launched a chatbot service, which offers a new modern way to solve clients’ problems. ESteelCo offers its products to a variety of different customer groups, and the launch of the chatbot service provided an immediate response to client enquiries.

The overall research aim was to develop a model based on findings obtained from analysing the chatbot implementation process. The authors adopted an interpretivist philosophy to observe the chatbot integration process within the selected organisation. First, however, a scoping literature review was undertaken to examine the relevant sources. A scoping review involves a “broad scan of contextual literature” through which “topical relationships, research trends, and complementary capabilities can be discovered” [19] (p. 351). It provides an initial overview of the subject matter “to draw the big picture” [20] (p. 1). IEEE Xplore, Google Scholar, and Science Direct were amongst the academic databases accessed to search for the existing literature, using appropriate search strings. Bell et al. [21] note that such a review can provide the researcher with insights into key themes, which here included the challenges and critical success factors for chatbot design and implementation. These key themes facilitated the development of the provisional conceptual framework for the subsequent primary, interview-based research, and for the design of the pre-interview questionnaire.

New and different perceptions of the key issues were obtained via an analysis of the unique experiences of the people involved in the project. The selected interpretivist philosophy aligns with the application of qualitative methods of analysis to answer the RQs. The experiences of practitioners and the personal views, criticisms, and beliefs of individuals involved in the study represent essential data that were recorded and analysed to explore the phenomenon and develop rules for the new chatbot implementation model. An interpretivist philosophy is also appropriate for an in-depth case study of the social phenomenon in an existing company environment [22]. The research process comprised four main phases as depicted in Figure 1. The key aspects of data collection and results validation, particularly important in a qualitative interpretivist case study, are now discussed in more detail below.

2.2. Data Collection

The study adopted a multi-method approach to data collection, employing both survey and observation to collect various sets of qualitative data in the case study company. Yin [22] (p. 99) argues that the use of various data sources within a case study can significantly improve the quality of the research data, and here the personal experience of specialists participating in the chatbot implementation project made a significant contribution in terms of variety, completeness, and quality of data. This was complemented by observations by one of the authors who played a role within the project team and had the opportunity to observe the implementation of the chatbot project.

Semi-structured interviews were used, and the list of topics and structure of questions were designed to address RQ1 and RQ2 and validate and develop the provisional conceptual framework, discussed below. This was an exploratory study in which chatbot experts and practitioners responded to a pre-defined list of questions, the answers to which formed the basis for formulating a new model for implementing chatbots. At the same time, respondents had the opportunity in the interviews for wider discussion, to recall their recent experiences, think about why certain events took place, and suggest what could have been done better to integrate the chatbot more effectively with other company systems.

There were 15 respondents/interviewees who took part in the study. Eight participated directly in the chatbot implementation project in ESteelCo, and the other seven were external experts from various banking, IT, retail, and pharmaceutical companies in Russia, China, and Germany (

Table 1. List of interviewees.

Interviewee Code	Job Title	Industry	Country	Interview Language	Internal or External
01	Project manager	Metallurgy	Russia	Russian	Internal
02	Project manager	IT integration	Russia	Russian	Internal
03	Analyst	Metallurgy	Russia	Russian	Internal
04	Analyst	Metallurgy	Russia	Russian	Internal
05	Senior Marketing Manager	Pharmaceutical	Germany	English	External
06	Business Processes Director	Pharmaceutical	Germany	English	External
07	Project manager	Banking	Russia	Russian	External
08	Product owner	Metallurgy	Russia	Russian	Internal
09	Analyst	Metallurgy	Russia	Russian	Internal
10	Marketing director	Metallurgy	Russia	Russian	Internal
11	CRM director	Online retail	Russia/China	Russian	External
12	Director	Metallurgy	Russia	Russian	Internal
13	IT Director	Online retail	Russia	Russian	External
14	IT Director	Fintech	Russia	Russian	External
15	IT developer	Online retail	Russia	Russian	External

). The interviewees were selected “because they have particular features or characteristics which will enable detailed exploration and understanding of the central themes and puzzles which the researcher wishes to study” [23] (p. 78). They were a combination of experienced internal and external specialists with various technical, industrial, and cultural backgrounds, which would support the generalisability of the research results and the formulation of a model for managing chatbot projects in other industry contexts. Guest et al. [24] (p. 78) suggested that “a sample of six interviews may have been sufficient to enable the development of meaningful themes and useful interpretations”, and Kuzel [25] (p. 41) similarly recommended that, for qualitative research, 6 to 8 interviews are enough for a homogenous sample, but 12 to 20 would be preferable when “trying to achieve maximum variation”.

In terms of the observations made during the study, a narrative approach was adopted for the recording and presentation of research records, which is characteristic of a qualitative study [26]. One of the authors was involved as a project team member and had the opportunity to observe the issues relating to chatbot integration and operation through the various project stages, from the initial design to product launch and service support. Significant daily events were recorded in the form of short notes, which were subsequently expanded and presented in a narrative form. Although this approach proved effective in providing relevant information and insights, bias in the perspectives taken is inevitable. For this reason, observation is used as a secondary data collection technique, which supplements the findings obtained from the semi-structured interviews.

2.3. Results Validation

The validity and reliability of data and results are important criteria that directly impact the quality of the study. Trochim and Donnelly [27] assume that participants are the ones who can best review research findings and evaluate how much they accurately match their opinions. The level of trust in the results heavily depends on how correctly researchers record and document the views of the people who take part in a study. Participant validation is, thus, a technique that allows researchers to achieve credibility in their research data by involving participants in the process of reviewing results [28]. Here, considerable attention was paid to establishing procedures for verifying the data obtained with the direct participation of the interviewees. Interviews were conducted remotely via videoconferencing using screen-sharing functionality, and interviews were recorded. During the interview conversations, the key thoughts and statements of the interviewee were noted down by the interviewer, and in case of doubt, clarification was requested. In addition, the transcript was emailed to the interviewee for final review and corrections. Only after confirming that all opinions were recorded and formulated correctly were the data used for processing and analysis, thereby minimising possible bias in the interpretation of the findings.

Guba and Lincoln [29] identify dependability and confirmability as two important criteria which measure research reliability. They argue that to some extent, generalisation can be achieved via a careful and exhaustive description of all stages of a study (dependability). A description of the research methodology, a justification of the chosen methods, a detailed description of the ESteelCo case study, and the process of collecting and analysing data are all integral parts of this research work. All these activities aim to establish the quality of the research and advance the possibility of reproduction by others. The second criterion, confirmability, is associated with the possibility that research findings and conclusions stated by a researcher can be confirmed by others. To this end, five experts were contacted to check the accuracy and validity of the research findings. Of these five, two were from the pool of interviewed participants, whilst the other three were experts in this field of research but not previously involved in the case study (

Table 2. Validation experts consulted in the study.

Expert No.	Was Involved as an Interviewee?	Characteristics
01	Yes	Senior IT manager and department head in a large company. Responsible for managing a programme of new IS integration projects including CRM, e-commerce platforms, AI applications, and chatbots. Experienced methodologist who organised the production of IT projects in an international IT integrator. Participated in several chatbot implementation projects as project leader.
02	No	Head of the project implementation department in a large IT integrator. Leads the development of multiple projects, including new CRM integration, website development, and chatbot integration.
03	No	Professor in Digital Skills with experience in AI, cyber security, and data science.
04	Yes	Director of digital transformation in a large enterprise. Responsible for managing a programme of IT projects including ERP, CRM, websites, and various AI application projects including chatbots. Previously involved in several chatbot implementation projects, either as a project lead or key stakeholder. Holds a PhD.
05	No	IT Director in a medium-sized company. Experienced in managing IS integration projects that include ERP, CRM, eCommerce platforms, AI applications, and chatbots.

).

Triangulation is a well-proven approach which helps to improve the validity and credibility of a study. One of the methods proposed by Patton [30] is associated with multi-method studies and the use of two or more data collection techniques. The use of different independent data sources and the application of various methods to analyse them and find the answer to the same research question help to verify and validate qualitative analysis. In this context, the results from the interviews, which directly addressed the RQs, are supplemented by observation. Using two different data collection techniques to answer the same questions helps to examine the problem from different perspectives.

3. Literature Review and Provisional Conceptual Framework

This section reports on the findings from an initial scoping literature review that Bell et al. [21] (p. 97) have observed can provide “a means of gaining an initial impression” of relevant themes and that “the narrative review may be more suitable for qualitative or inductive researchers, whose research strategies are based on an interpretative epistemology”. The scoping literature review aimed at identifying the key themes from the literature that could provide the basis for the development of a provisional conceptual framework for the research and identify some of the key literature that would support the generation of responses to the RQs. The scan of the relevant literature allowed the discovery of “topical relationships, research trends, and complementary capabilities” [19] (p. 351), and the following three sub-sections report on this literature as it relates to the three RQs. Finally, Section 3.4 sets out the provisional conceptual framework derived from this literature review.

3.1. Chatbot Implementation Challenges

Despite the potential benefits of chatbots, many companies still find the prospect of chatbot implementation challenging. According to CB Insights [14], high user expectations are not always justified as the technologies and methods for creating chatbots are far from perfect. Srinivasan et al. [31] surveyed more than 300 business executives and IT directors in 12 countries and identified the most prevalent challenges companies face when implementing chatbots: scepticism from users in accepting a new way of communicating; the limited capabilities of chatbots and their inability to provide an acceptably personalised service; and the high number of errors in handling user requests. In addition, the complexity of the technical component of chatbots and organisational barriers are other hurdles that companies must overcome. Srinivasan et al. [31] also point to the difficulty of addressing data security issues and compliance with local data protection laws, the lack of skilled labour, high integration costs, and the difficulty of platform selection due to high vendor fragmentation. This is confirmed by Kaushal and Yadav [32], who argue that companies have difficulties with platform selection and integration when implementing chatbots. The majority of chatbot platforms do not provide the necessary functionality for testing and customisation, and the resulting behaviour of the virtual assistant may fail to meet user expectations regarding quality, responsiveness, and ability to correctly process a request formulated in human language.

Despite justified criticism from businesses regarding the conversational capabilities of modern chatbots, the results of research groups and demonstrations of the capabilities of chatbots from major technology companies show that the quality of a virtual assistant dialogue is almost indistinguishable from that of a human. In practice, however, companies are severely limited in what they can do because many NLP-related tasks are challenging [33]. The creation of a perfect interactive assistant requires significant computing and financial resources to collect and process large amounts of data to train the language models, as well as the availability of appropriate specialists [34].

Challenges for chatbot developers are not always directly related to technology, organisational issues, or costs. Even a perfectly implemented chatbot that exploits all the technologies currently available may not be popular with users, some of whom may be conservative in their attitude and prefer the current status quo. Previous negative experiences with virtual assistants may also discourage potential new users [35].

Fach et al. [15], based on feedback from more than 90 leaders from different organisations worldwide, categorised the main challenges as follows:

• Setup challenges. Activities related to obtaining, preparing, and processing big data; bot training; implementation and launch; and support and maintenance.
• User/customer acceptance. Not all users are ready to interact with virtual agents. This situation is expected to gradually improve, so in the long term, such difficulties should not prevent the introduction of new technology.
• Language challenges. Effective communication with users can be hampered by technical limitations, difficulties in understanding various accents, as well as the ability of developers to deal with natural language processing (NLP) tasks.
• Regulatory restrictions and data security. Both internal company guidelines and national or international regulations (for example, GDPR) can impact system architecture and methods of receiving and storing customer data. This complicates the process of developing virtual assistants in practice and increases the cost of implementation.
• Technology-related challenges need to be addressed. It is advisable to implement technologies from strong vendors that have mature solutions, innovative features, and flexible APIs for integration with other company systems.

This categorisation overlaps considerably with the perspectives given by Srinivasan et al. [31] and other authors.

3.2. Critical Success Factors for Chatbot Projects

The project management triangle, sometimes called the “iron triangle”, has been used for over 50 years to estimate project success. Time, quality, and budget are the three parameters which determine project objectives and allow for the definition of measurable goals [36]. However, the above review of the challenges faced by companies implementing chatbots showed that most of the difficulties and failures are related not only to aspects of project management but also to a wide variety of other factors, these being external or internal to the organisation, and having a technological, social, or other nature. This suggests that the traditional approach to determining the success of a project, exemplified in the parameters of the “iron triangle”, could usefully be complemented with the identification of critical success factors (CSFs) for chatbot projects.

CSFs are well established as an approach to project management. Rockart [37] used CSFs as a method for planning information systems in organisations, and Bullen and Rockart [38] characterised CSFs as project-related areas in which sufficient results will guarantee the successful achievement of the project goals. The existing literature, documented case studies, and IT integrator guidance available on the internet were analysed to find appropriate CSFs for chatbot projects. These CSFs and their sources are presented in

Table 3. CSFs identified from the extant literature.

No.	Critical Success Factor
1	Identify use cases and assess the suitability of using a conversational User Interface (UI) [39,40,41]
2	Focus on one business objective [42,43]
3	Analyse integration capabilities of the selected platform [39,42]
4	Define business value metrics aligned to the organisation’s strategy [39]
5	Consider security and compliance requirements [31]
6	Explore vendor options in selected industry field and locality [42]
7	Analyse build and buy approaches to define the strategy [44]
8	Use the buy (not build) approach and explore your local vendors [44]
9	Choose a platform that can provide detailed reports about chatbot performance [42]
10	Offer users an option to chat in their favourite messenger and provide multiple messenger support [41]
11	Apply agile iterative approach and MVP method [40,43]
12	Design an experience that provides value for both the customer and the business [43]
13	Use information about the customers to offer personalised service [45,46]
14	Focus on chatbot productivity. The ease, speed, and convenience of using chatbots is vital. Bots should solve users’ problems in a more efficient way than any other communication channel [43,47]
15	Use entertainment elements and social interaction elements for additional motivation [47]
16	Design chatbot personality [39,41,46,48,49,50,51,52]
17	Evaluate performance and identify which requests were not processed properly [42,43]
18	Invest time and resources in bot training [40,43,53]
19	Blended communication enables excellent service. Switch dialogue to a human when a bot cannot handle a user’s request [36,41,46]
20	Create a dashboard that displays key chatbot performance metrics [43]
21	Study the users’ needs by collecting feedback and plan new features accordingly [39,43]
22	Continuously improve the chatbot by applying recent AI technologies [43]
23	Advertise the bot’s capabilities to its users [39]

and the variety of factors suggests that a range of different subject areas impact the outcome of chatbot projects. These include new technology adoption, resource availability, vendor and platform selection, software licensing and implementation costs, and organisational and project management aspects.

3.3. Relevant Models, Methods, and Frameworks

A number of technology adoption models are of relevance to chatbot projects. DePietro et al. [54] developed the Technology–Organization–Environment (TOE) framework, which has been adapted and adopted in a range of technology studies [55,56,57,58,59]. The framework suggests that three main factors influence technology innovation in the corporate environment (Figure 2). First, the technology factor concerns the IT infrastructure, hardware, network communication, software packages, and services available to the company, and the processes that facilitate their deployment. Second, organisational culture and structure, core business processes, and resource availability are seen as key elements in successful transformation. Third, the environmental factor refers to the external market within which the business operates, and the activities of other organisations—competitors, service providers, and government regulators.

Another model from the same era, but still of equal relevance today, is that developed by Earl [60], who set out a multiple methodology for IS strategy formulation. In fact, Earl envisaged three interlocking strategies covering IT, IS, and IM (Information Management), but IS is considered here. Earl identified three approaches to developing strategy—top-down, bottom-up, and inside-out. The top-down method is an analytical deductive approach that identifies business plans and IS needs. The analysis of CSFs is used via interviews, workshops, or document analysis to establish a connection between business goals and IS requirements. The bottom-up approach assumes that some existing ISs have strategic potential and their functionality can support other business activities that provide a competitive advantage. Thus, an audit of current systems is required to recognise which ones need to be updated, what kind of add-ons or extension modules exist, and what the potential for enhancement and further development is. The inside-out approach identifies innovation opportunities and discovers the new strategic potential for the company: investments in R&D, human talent, and the development of technological culture across the organisation will foster innovation and speed up new technology adoption.

There are a number of stage models used by IS researchers or IT practitioners to aid the planning and analysis of projects. These include the approach developed by the Project Management Institute (PMI), which distinguishes between initiation, planning, execution, and closeout phases [61], and the waterfall model [62] in which each phase is validated by stakeholders before moving on to the next phase. It is irreversible and does not allow the previous phases to be reviewed once they have been completed. The advantage of the waterfall model is its comprehensive approach, which helps to develop a well-integrated and documented system. The involvement of stakeholders at each stage ensures their participation and helps to minimise further complaints. The model is often criticised for its long system development process and the risk that requirements may change during development. This can lead to an implemented system that is out of date by the time it is completed [63].

Nicholas and Steyn [64] propose a stage model for IS implementation that illustrates, in a simplified form, typical project phases that are also found in other stage models. The four stages are as follows: (1) conception, in which the drivers of change are identified, feasibility is assessed, and external providers are short-listed; (2) definition, which includes a detailed analysis of the concept, system, and functional requirement specifications and detailed project planning; (3) execution, when the IS is implemented, deployed, and necessary training and process adjustments are undertaken; and (4) operation, when system handover to the end-users is finalised, and system maintenance procedures and service levels are put in place.

Agile methodologies for software development are also of relevance to chatbot projects. This way of developing new software is based on self-organised teams that can swiftly develop and deliver new software features, sometimes termed “product increments” [65]. There are a number of methodologies that provide guidelines for undertaking such development, including Scrum, Kanban, lean software development, and feature-driven development (FDD) [66,67]. The MVP (Minimum Viable Product) approach is based on agile methodology and is generally seen as a version of a new product which allows teams to collect the maximum amount of validated learning about customers with the least effort [68]. The main idea of this approach is to build the core functionality of the product, deploy it, and see it used by early adopters as soon as possible. Given the complexity involved in AI-related projects, agile methodologies are of particular value in that they are based on adaptive processes and strong communication amongst project participants [69].

Chatbot projects are characterised by a high degree of uncertainty in the early stages, unpredictability of the outcome, and the need to study user feedback in order to define new requirements. Many IT integrators support the use of the MVP method to address this uncertainty in their projects. Chakrabarti et al. [70] argue that companies should grow their service from the basic form, continuously improving its functionality by adding new features, integrations, and skills moving towards the advanced level. This approach is supported by others who argue that the MVP method should encourage teams to design, develop, and launch a quick pilot version of a chatbot to understand and assess customer reactions [39,43,54,71]. Once the first version with the core functionality is in operation, continuous improvement and optimisation activities can commence. They usually include customer feedback and behaviour research, AI model training and improvement, planning and adding new features incrementally, and additional user training.

3.4. Provisional Conceptual Framework

The provisional conceptual framework (PCF) for the research draws upon aspects derived from the literature discussed above. The CSF approach is adopted, but the list of factors presented in Table 3 is not seen as a definitive set of tools for the research. Further organisation and prioritisation are required, and to this end, the TOE framework is used, albeit in a modified manner. The Technology and Organisation categories of the framework are retained, but the Environmental element is excluded from the PCF, as there is little evidence of such factors in the CSFs listed in Table 3, and the likelihood of such aspects influencing chatbot projects is relatively low. However, “User Needs” is seen as a logical third category, which aligns with Brandtzaeg and Folstad’s [47] assertion that productivity is the most significant chatbot criterion and that this relates to a positive user experience.

Chatbot projects, like many other new IS implementations, undergo certain similar phases. Every project stage can be characterised by a certain set of the most important events and key actions that the team takes. Thus, for each stage, there are corresponding CSFs that positively influence the success of a particular stage as well as create the prerequisites for subsequently successfully solving these problems. An analysis of the preliminary list of CSFs (Table 3) suggests that many of them are relevant to a particular project stage of a project. Classifying the CSFs according to a particular project stage brings a project-oriented perspective to the PCF, and to this end, Nicholas and Steyn’s [64] four-step project lifecycle model is utilised to provide further structure to the PCF.

In summary, the PCF comprises the following elements in an X/Y axis used to categorise CSFs: a modified TOE framework which classifies the various sources that influence chatbot projects using three change categories (Technology, Organisation and User Needs); and project stages, based on Nicholas and Steyn’s model [64]. The CSFs listed in Table 3 are provisionally allocated to the appropriate cell in the X/Y axis as depicted in

Table 4. Provisional conceptual framework for researching chatbot implementation.

Project Stages/Change Categories	Conception	Definition	Execution	Operation
Technology	Analyse build and buy approaches to define the strategy. Explore vendor options in selected industry field and locality. Analyse the integration capabilities of the selected platform. Choose a platform that can provide detailed reports about chatbot performance.	Consider security and compliance requirements. Offer users an option to chat in their favourite messenger and provide multiple messenger support.	Create a dashboard that displays key chatbot performance metrics.	Continuously improve the chatbot by applying recent AI technologies.
Organisation	Identify use cases and assess the suitability of using a conversational User Interface. Focus on one business objective. Define business value metrics aligned with the organisation’s strategy.	Apply agile iterative approach and MVP method.		Invest time and resources in bot training. Advertise the bot’s capabilities to its users.
User Needs		Design an experience that provides value for both the customer and the business. Use information about the customers to offer personalised service. Apply a blended communication approach.	Focus on chatbot productivity. The ease, speed, and convenience of using chatbots is vital. Use entertainment elements and social interaction elements for additional motivation. Design chatbot personality.	Evaluate performance and identify which requests were not processed properly. Study the users’ needs by collecting feedback and plan new features accordingly.

. This represents the researchers’ assessment of these CSFs based on the existing literature sources. The resultant framework indicates that the CSFs are distributed more or less evenly over the cells of the matrix. This suggests that each of the change categories (Technology, Organisation, and User Needs) has an impact on the project at each project stage and that the identification of new CSFs and the refinement of existing ones can be effectively represented within this framework, which provides the basis for the primary case study research, through which its structure, elements, and listed success factors are analysed, refined, expanded, and validated.

4. Results

This section directly addresses the three RQs drawing on the literature review presented in Section 3 and on the primary interview material, using interviewee codes noted in Table 1. The documented notes from researcher observations are also cited, as appropriate, as a further data source.

4.1. RQ1: What Are the Main Challenges Which Companies Face When Implementing Chatbot Projects?

The three data sources noted above were analysed in conjunction to ascertain the key challenges. The challenges derived from an analysis of the literature were categorised and allocated to the appropriate cell in the PCF matrix. Then, the interview findings were analysed to produce a summary list of challenges raised in the interviews, and the individual statements of each interviewee were grouped into more general and universal formulations. In addition, the notes from the observations were similarly scanned and challenges were identified in the narrative text. The challenges identified from these three sources were then combined, via a process of aggregation and reformulation. The results contain items from three sources, grouped by challenge “category”, as defined in the PCF and ranked by a “score” value (

Table 5. Challenges when implementing chatbot projects.

No.	Challenge	Score
Technology Category
Challenge 1	Complexity and amount of system integration work.	23
Challenge 2	Difficulty in finding the best product and vendor which can solve your problem and will offer its service for a long time.	15
Challenge 3	The overall complexity of chatbot projects and the necessity to deal with many software systems and IT technologies.	13
Challenge 4	Chatbots require many calculation resources. Many companies are not ready to purchase the required hardware.	10
Challenge 5	Few companies are able to collect, analyse, and use their internal data to develop appropriate chatbot behaviour.	9
Challenge 6	The complexity of regulation and information security tasks.	9
Challenge 7	Current chatbot technologies are complex but not yet perfect and bots often fail to process users’ requests properly.	9
Challenge 8	The licence and support costs of a chatbot system from leading vendors are too high, which forces teams to create their own solutions.	6
Organisation Category
Challenge 9	Difficulty in formulating project objectives, success criteria, and measuring KPIs.	24
Challenge 10	Lack of in-house chatbot expertise and limited availability of relevant specialists in the market.	21
Challenge 11	Top managers often underestimate chatbot integration project complexity and costs and are not ready to invest a significant budget in new technology adoption.	17
Challenge 12	Planning and managing chatbot projects are challenging due to their unpredictability and high uncertainty.	11
Challenge 13	Low quality of integration services provided by vendors and IT integrators.	10
User Needs Category
Challenge 14	User scepticism about the capabilities of chatbots and unwillingness to interact with them.	27
Challenge 15	User behaviour in chats is very different from other channels like phones, websites, and mobile apps. High-quality UX design skills and user training are required.	11
Challenge 16	The scenario-based approach for creating bots prevails but limits their capabilities.	8
Challenge 17	Difficulty in creating a chatbot which solves the company’s problems and provides users with an excellent service at the same time.	4

).

Kaushal and Yadav [32], who explore different platforms for creating chatbots, point to the lack of out-of-the-box integration with popular software products, as well as the amount of work required to customise platforms and integrate with enterprise IS. This view was confirmed by many interviewees. Interviewee 1, for example, states that “the amount of work involved in integrating a chatbot with data sources is often underestimated during planning”. Interviewee 4 adds that “integrating a chatbot with other business systems is particularly difficult and unpredictable”. These statements are also supported by an analysis of meeting minutes and lessons learned during the implementation of the chatbot at ESteelCo, which indicates that the team underestimated the amount of work required to integrate the chatbot with the CRM system, resulting in a significant shift in the project start date. Dealing with complexity and the amount of system integration work is, thus, one of the key challenges (Table 5).

The Score parameter reflects the result of a quantitative assessment of the results obtained. It is calculated as follows: each interviewee is assigned an experience level from 1 to 3, measured in points. Junior employees with little chatbot experience are assigned 1 point, senior specialists or project managers with more chatbot experience receive 2 points, and chatbot experts with solid experience in integrating chatbots receive 3 points. The score is calculated as the sum of the points of the interviewee who mentioned the challenge. This provides a way of ranking the challenges so that chatbot implementation teams can better assess project risks and plan appropriate activities.

4.2. RQ2: What Are the Critical Success Factors (CSFs) That Contribute to the Successful Implementation of Chatbots in Companies?

The process of identifying CSFs was similarly based on the principle of triangulation, using three sources of data: the extant literature, the interview data, and the observation notes. The critical success factors identified in the literature review were noted in Section 3.2 and presented in Table 3.

During the interview sessions, interviewees were asked to identify critical success factors for successful chatbot implementation. The interview script structure was entirely consistent with the PCF depicted in Table 4. The respondents named success factors concerning the four project stages (Conception, Definition, Execution, and Operation) and three change categories (Technology, Organisation, and User Needs) and discussed respective actions which help companies meet their objectives. The process of merging CSFs from the three sources was carried out by finding identical or similar formulations, and by aggregating similar or complementary ideas into more general and universal statements to ensure that one CSF is not repeated in similar formats. This process is shown schematically in Figure 3.

As an illustrative example of this process, a number of the literature sources maintained that the use of an agile, iterative approach and the MVP method is recommended when implementing chatbots [40,42,43]. This was discussed more broadly by several interviewees. Interviewee 12, for example, recommends minimising the size of the first MVP by “implementing 2–3 of the simplest but most popular scenarios in the first iteration”. Interviewee 13 supports this, explaining that “launching a minimum viable product version quickly allows testing of the hypothesis and adjustment of the project roadmap”. This view is confirmed by the observation findings and analysis of the Lessons Learned meeting minutes, which took place after the launch of the chatbot. Specifically, the team concluded that “an iterative approach and MVP method can be an effective way to manage chatbot development projects, especially when the expected results and outcomes are not clearly defined”. In this way, after processing the information obtained from the analysis of publications, interview transcripts, and observations, more general formulations of the CSFs were developed and located in the appropriate cell of the PCF.

This process of combining statements and calculating the Score value uses a similar approach to that described in Section 4.1. Here, items that received a score of 3 and below are excluded from the results, as in most cases they represent individual judgements of single respondents that have not been confirmed by other sources. The results are presented in

Table 6. Critical success factors for chatbot implementation.

Project Stages/Change Categories	Conception	Definition	Execution	Operation
Technology	Identify and choose the most suitable technology and vendor which meet your business requirements. (Score = 26) Explore vendor options and external service availability in your field and locality. (Score = 24) Choose a platform that can provide detailed reports about your chatbot performance. (Score = 19) Analyse build and buy approaches to define your strategy. (Score = 17) Analyse the API of the selected platform and how to integrate it with the company’s IS. (Score = 10) Study project portfolio of vendor candidates and request recommendations from their clients. (Score = 9) The selected platform should have a powerful dialogue editor. (Score = 6)	Invest in the analysis and planning of system integration and data migration tasks. (Score = 23) Define scalability requirements and realistically estimate the required hardware. (Score = 13) Obtain necessary access to ISs and solve information security issues. (Score = 13) Strategically plan the compatibility of the bot platform with the company’s internal IT systems. (Score = 12) Make sure you know exactly how you can develop the bot on your own or change the integrator company, if necessary. (Score = 8) Formulate comprehensive functional requirements that determine the result. (Score = 5)	Create a dashboard for tracking chatbot behavioural metrics and success indicators after the launch. (Score = 11) Perform load testing, plan and prepare infrastructure for high load. (Score = 9) Prepare and use development, testing, and production environment. (Score = 9) Use SLAs (service level agreements) to set performance requirements for involved IS. (Score = 6)	Monitor a chatbot’s availability and usage. (Score = 19) Continuously improve the chatbot by applying recent AI technologies, updating language models, and installing platform updates. (Score = 14)
Organisation	Identify your aim, key success indicators, and associated improvement metrics. (Score = 24) Identify use cases and assess the suitability of using a conversational UI. (Score = 19) Find an expert to join your team. (Score = 10) Prepare the ground for agile working. (Score = 9)	Form a product team on the company’s side which includes a product owner, analyst, dialogue designer, data scientist, UX designer, and technical leader. (Score = 23) Use the MVP method and an iterative approach. (Score = 18) Find a responsible project leader from the business team who has the necessary expertise. (Score = 9) Involve top management in the process. (Score = 6) Communicate your aim to your team, contractors, and top management. (Score = 5)	Apply an agile iterative approach and MVP method. (Score = 23) Define the chatbot maintenance team and their responsibilities and train the key users. (Score = 9) Test your bot internally with your team and with business experts in the company. (Score = 7) Pay attention to project control, reporting, and communication with the contractor. (Score = 4)	Collect dialogue data and invest time and resources in bot training. (Score = 13) Develop the chatbot expertise and grow your team in-house. (Score = 9) Continue to use the agile development approach. Turn it into a routine. (Score = 9) Make sure that the chatbot systems are accepted by IT support technical specialists. (Score = 7) Advertise bot features and share success stories within your company. (Score = 5)
User Needs	Focus on one business objective. (Score = 20) Analyse alternatives to the chatbot service. Specify why the chatbot is the best way to solve your problem. (Score = 13) Make sure the chatbot is not only good for the company but also designed to make the customer’s life better. (Score = 10) Make sure the chatbot is the most convenient way for users to solve their problems. (Score = 5)	Focus on chatbot productivity. The ease, speed, and convenience of using chatbots are vital. (Score = 16) Use CJM and Jobs-To-Be-Done methods to capture user experience and specify chatbot requirements. (Score = 16) Choose the “easy-to-develop” functionality. Publish limited but well-developed functionality. (Score = 14) Design a chatbot personality which fits your brand communication style. (Score = 13) Focus on intuitive and simple UX. (Score = 9) Apply a blended communication approach. (Score = 9) Plan methods and tools for collecting user feedback. (Score = 8)	Test every product iteration on a small pilot group of real customers using real data. (Score = 23) Test UX and employ measurement tools which your platform offers. (Score = 13) Determine your bot personality which fits your brand communication style. (Score = 10)	Monitor chatbot performance and identify which requests were not processed properly. (Score = 23) Study your user needs by collecting feedback and plan new features accordingly. (Score = 20) Monitor chatbot behavioural metrics and success indicators. Ensure that the chatbot achieves its objectives. (Score = 20) Advertise bot capabilities to its users and create respective promotion plans. (Score = 15) Collect qualitative data about chatbot performance to improve functionality. (Score = 12)

, using the structure of the PCF. The identified CSFs allow implementation teams to strategically plan and execute chatbot projects and increase the chances of a successful chatbot launch.

4.3. RQ3: What Operational Model Can Support the Chatbot Implementation Process and Help Responsible Managers Deliver Expected Chatbot Project Outcomes?

The PCF played a key role during the practical and analytical phases of the study, helping to shape the data collection and processing activities. The CSFs provide a foundation for building the broader operational model for chatbot project development and implementation. The PCF structure facilitates the allocation of the CSFs into the appropriate cell in the matrix of change categories (Technology, Organisation and User Needs) and project phases (Conception, Definition, Execution, and Operation), as depicted in Table 6. This represented an initial operational model, but further reflection and adjustment produced an enhanced, more organised and dynamic operating model with a more compact presentation.

First, a critical evaluation of the initial model suggested that such a large amount of information may prove difficult to work with in practice. The total number of CSFs was 59, far exceeding the typical number of 5–8 items suggested by Rockart [37]. Researchers have used different approaches to organise CSFs into different hierarchical structures and groups, and here grouping and aggregation were used to identify seven main CSFs while maintaining a longer list of secondary factors.

Second, a number of CSFs that apply across several of the categories and stages of the model were identified, and these were taken out of the tabular structure of the matrix to sit above the others as more generally applicable. This also reflected the view of some interviewees that relating certain CSFs to specific cells in the PCF was problematic, suggesting more general factors to be followed across the stages of the project.

For example, the CSF ‘Focus on chatbot productivity’ is listed in the definition phase of the User Needs category, although this factor is more global in nature. Interviewees 2, 3, 4, 6, 7, 8, and 9 mentioned this factor as important but were unsure of the exact location of this factor in the PCF. Interviewee 8 explained that “he would like to attribute this CSF to the definition stage because it is important to think about chatbot productivity already when you plan your project and create specifications. However, this is not really limited by the Definition stage, but rather a general rule which should be considered by the teams at each stage”. Focusing on chatbot productivity can be justified in almost any action to solve a variety of problems, such as choosing a technology and platform, setting goals, formulating technical requirements, defining functionality, planning the development backlog, testing, etc. So, this CSF was removed from within the PCF matrix and relocated outside of it (Figure 4—transition A).

Similarly, some CSFs were deemed relevant to a particular category but across all project stages. However, at each stage, this CSF is characterised by specific actions that are relevant to that particular stage. For example, “Use the MVP method and iterative approach” is assigned to the Definition phase, but there are other entries associated with it in all the other project stages (Figure 4—transition B). The use of MVP and agile methodologies is a recommendation that is relevant throughout the project. As the CSF factor is formulated in a more general way, it no longer corresponds to a specific cell in the two-dimensional matrix and can be taken out of it. At the same time, at each stage in the Organisation line, following this recommendation can lead to specific actions that correspond to the current stage of the project. In this way, these entities can be linked hierarchically—the CSF, which exists in the context of the whole project, and the associated activities, which are derivatives of this CSF and are recommended for execution according to the project stage. This dependency is shown schematically in Figure 5.

This process of aggregating model elements resulted in a reduced list of more general CSFs that fall outside the scope of the tabular structure of the PCF. The remaining CSFs are left within the cell matrix but have been reformulated as local and specific tasks or “Key Actions”. These can be seen as derivatives of the CSFs and respond to the key issues that teams need to address within the specified project stages. They are of value in moving the model from a mainly strategic level to facilitate project execution. Seven CSFs are placed outside the matrix, organised horizontally, and are more general entities which provide a broad conceptual and operational platform upon which the Key Actions can be grounded.

Third, the existing literature suggests CSFs can be used to link strategic objectives to operational activities. In the context of IS strategy, Earl [60] put forward a three-level structure in which the CSFs are the element that links the company’s strategic goals to tactical planning activities. A similar construct is presented by Bullen and Rockart [38], in which CSFs link an objective to key activities.

The incorporation of a strategy level based on objectives enhances the operational model, making it more applicable to project planning and control activities in chatbot integration projects. This can be achieved by moving the CSF “Focus on one business objective” to a higher level and adjusting its wording. Trying to implement a chatbot in several different business processes at the same time can lead to a lack of focus and insufficient investment in the implementation of each of the planned functions, which significantly reduces the quality of the final result and most likely leads to the failure of the chatbot. Solving a single but important task will significantly improve the likelihood of a successful project launch. In project management, the key metrics that measure the success of the project are usually associated with the project goal or objective. This is reflected in the model, which assumes that the activities to define the project aim, the success indicators and the corresponding metrics are carried out in the Definition phase and fall under the Organisation category. The updated construct resulting from the creation of the new Objectives layer is shown in

Table 7. New operational model for chatbot implementation.

Objective			One Business Objective: (Define)	Success Metrics: (Define)
CSFs	The chosen platform and technology best suit the current needs. The chatbot software fits the objectives and the company’s regulations.	System integration and data migration tasks are in focus. Chatbot usually needs to be integrated with other company systems.	Chatbot expert is in your team. Involve an experienced manager or consultant in your project. From your company or outside.	The MVP method and an iterative approach are used. Study the user experience and adjust the plan after each iteration.	Chatbot is the best way to solve users’ problems. Ensure the chatbot can perform more efficiently than other company services.	Chatbot productivity is the top priority. The ease, speed, and convenience of using chatbots are vital.
		Conception ►	Definition ►	Execution ►	Operation
Recommended Key Actions	Technology	Identify and choose the most suitable technology and vendor which meet your business requirements. (Score = 26) Explore vendor options and external service availability in your field and locality. (Score = 24) Choose a platform that can provide detailed reports about your chatbot performance. (Score = 19) Analyse build and buy approaches to define your strategy. (Score = 17) Analyse the API of the selected platform and how to integrate it with the company’s IS. (Score = 10) Study project portfolio of vendor candidates and request recommendations from their clients. (Score = 9) The selected platform should have a powerful dialogue editor. (Score = 6)	Invest in the analysis and planning of system integration and data migration tasks. (Score = 23) Define scalability requirements and realistically estimate the required hardware. (Score = 13) Obtain necessary access to ISs and solve information security issues. (Score = 13) Strategically plan the compatibility of the bot platform with the company’s internal IT systems. (Score = 12) Make sure you know exactly how you can develop the bot on your own or change the integrator company, if necessary. (Score = 8) Formulate comprehensive functional requirements that determine the result. (Score = 5)	Create a dashboard for tracking chatbot behavioural metrics and success indicators after the launch. (Score = 11) Perform load testing, plan and prepare infrastructure for high load. (Score = 9) Prepare and use development, testing and production environment. (Score = 9) Use SLAs (service level agreements) to set performance requirements for involved ISs. (Score = 6)	Monitor a chatbot’s availability and usage. (Score = 19) Continuously improve the chatbot by applying recent AI technologies, updating language models, and installing platform updates. (Score = 14)
	Organisation	Identify your aim, key success indicators and associated improvement metrics. (Score = 24) Identify use cases and assess the suitability of using a conversational UI. (Score = 19) Find an expert to join your team. (Score = 10) Prepare the ground for agile working. (Score = 9)	Form a product team on the company’s side which includes a product owner, analyst dialogue designer, data scientist, UX designer, and technical leader. (Score = 23) Use the MVP method and an iterative approach. (Score = 18) Find a responsible project leader from the business team who has the necessary expertise. (Score = 9) Involve top management in the process. (Score = 6) Communicate your aim to your team, contractors, and top management. (Score = 5)	Apply an agile iterative approach and MVP method. (Score = 23) Define the chatbot maintenance team and their responsibilities and train the key users. (Score = 9) Test your bot internally with your team and with business experts in the company. (Score = 7) Pay attention to project control, reporting, and communication with the contractor. (Score = 4)	Collect dialogue data and invest time and resources in bot training. (Score = 13) Develop the chatbot expertise and grow your team in-house. (Score = 9) Continue to use the agile development approach. Turn it into a routine. (Score = 9) Make sure that the chatbot systems are accepted by IT support technical specialists. (Score = 7) Advertise bot features and share success stories within your company. (Score = 5)
	User Needs	Focus on one business objective. (Score = 20) Analyse alternatives to the chatbot service. Specify why the chatbot is the best way to solve your problem. (Score = 13) Make sure the chatbot is not only good for the company but also designed to make the customer’s life better. (Score = 10) Make sure the chatbot is the most convenient way for users to solve their problems. (Score = 5)	Focus on chatbot productivity. The ease, speed, and convenience of using chatbots are vital. (Score = 16) Use CJM and Jobs-To-Be-Done methods to capture user experience and specify chatbot requirements. (Score = 16) Choose the “easy-to-develop” functionality. Publish limited but well-developed functionality. (Score = 14) Design a chatbot personality which fits your brand communication style. (Score = 13) Focus on intuitive and simple UX. (Score = 9) Apply a blended communication approach. (Score = 9) Plan methods and tools for collecting user feedback. (Score = 8)	Test every product iteration on a small pilot group of real customers using real data. (Score = 23) Test UX and employ measurement tools which your platform offers. (Score = 13) Determine your bot personality which fits your brand communication style. (Score = 10)	Monitor chatbot performance and identify which requests were not processed properly. (Score = 23) Study your user needs by collecting feedback and plan new features accordingly. (Score = 20) Monitor chatbot behavioural metrics and success indicators. Ensure that the chatbot achieves its objectives. (Score = 20) Advertise bot capabilities to its users and create respective promotion plans. (Score = 15) Collect qualitative data about chatbot performance to improve functionality. (Score = 12)

.

The new model has, thus, been developed from the PCF to comprise three conceptual and operational layers: the Objectives layer, the CSFs, and Recommended Key Actions. The top Objectives layer concerns information about the strategic objectives of the project and specifies appropriate metrics for assessing the achievement of these objectives. The model calls for limiting the functionality of the first version of the chatbot, strictly focusing on one business objective and choosing one or more metrics that match that objective.

The central part of the model is the CSF layer which contains six CSFs that are applicable to different projects and should be discussed by the team in strategic meetings. The Recommended Key Actions (the bottom layer) are derived from the CSFs and should be viewed as a flexible set of tasks that can be modified, removed, or added to, depending on the organisational context and specific project requirements. They are allocated to the three change categories that affect chatbot projects: those of a technological nature, organisational aspects, and user needs; and to the four phases, which are typical for most projects: Conception, Definition, Execution, and Operation.

The application of the model facilitates a holistic and comprehensive approach to planning and managing virtual chatbot implementation projects. By linking the strategic level of business objectives, critical success factors and key actions, the new model creates the foundation for successfully building and launching virtual assistants. This was confirmed by experts during the validation process, who confirmed their view that this model can make chatbot projects more manageable and increase the chances of success.

The validation process involving five experts consisted of three steps. First, the experts were provided with a new model for chatbot implementation, along with a detailed description. Next, a discussion of the model took place between the researcher and the experts, focusing on the model’s structure, content, and application recommendations. Finally, the experts independently completed a questionnaire evaluating four aspects of the model: evaluation of the model structure, evaluation of the CSFs, evaluation of the Key Actions, and evaluation of the new model in general. Overall, the experts were positive in their support of the model, albeit with some suggested minor modifications which have been incorporated in Table 7.

5. Discussion

This section looks at some of the wider issues raised by the research results. In the first sub-section, the applicability of the model is discussed, providing some outline guidance for the use of the model in practice. In Section 5.2, data privacy issues raised by the growing deployment of chatbots are discussed. In the final sub-section, differing perspectives regarding the future of chatbots within the rapidly evolving use of AI in business are examined, including some assessment of the future impact of ChatGPT.

5.1. Model Application and Guidance

Many authors, as well as a Gartner study [72], point out that there is no perfect solution for building chatbots. There are hundreds of vendors in the global market offering their solutions. Depending on a variety of factors, encompassing technological, regional, and organisational issues, each of these platforms may seem the most appropriate for teams in their projects. For this reason, the study is not limited to any one technology or vendor but formulates general principles that are independent of the technology used. At the same time, the new model emphasises the critical importance of choosing the most appropriate platform and recommends a series of actions for choosing it.

The model is best seen as a flexible tool that can be used alongside established planning methods and techniques. It is most effectively deployed by adopting a top-down approach. The preliminary stage requires the development of the concept and formulation of the project idea. Then, the project team is best served by focusing on one business objective, after which time the model can be utilised. In addition to agreeing on the key business objective, one or more metrics that directly measure the achievement of the objective should be specified. If this proves difficult to achieve at the outset of the project, metrics can be added or refined later in the Conception stage.

After defining objectives and selecting success metrics, the CSF and Key Action layers of the model should be explored sequentially. The six CSFs are general formulations that can be applied in different phases across the duration of projects, so teams are encouraged to discuss them during planning and review sessions to establish a link with the real context of the organisation. By exploring the relevance of each CSF to a particular project, project stakeholders gradually prioritise their requirements and gain a comprehensive understanding of the challenges, issues, risks, and opportunities associated with the upcoming chatbot implementation.

The CSFs are reflected and interpreted in the Key Action layer in the form of recommended tasks and actions. The Key Action layer is the most detailed yet flexible part of the model, and teams may find it necessary to adapt it to particular project circumstances. For example, the task “Find an expert to join your team” may not be relevant for all teams. At the same time, the specifics of the organisation’s business, IT environment, internal rules, and procedures may require important specific tasks to be solved that are not listed in the operational model. For example, there may be a number of important tasks related to integration with specific ISs, creating new micro-services to implement specific planned bot functions, and preparing necessary data.

From a project lifecycle point of view, it is advisable to use the operating model as early as possible—as soon as the idea of potentially using a chatbot in a company has been raised. The model is designed to help teams with strategic planning and to shape the project’s action plan, so its role is most important at the very beginning. However, the project manager and other team members can refer to the model periodically throughout the course of a project. This can help to check how well the current tasks are aligned with the CSFs. If there are significant gaps or problems, or if significant changes to the project plan are needed, the operating model can be a useful tool for re-planning. Repeating the previous planning activities, from validating the key objective and reviewing the CSFs to defining the Key Actions, will facilitate the review and update of the project plan.

The model can also be used post-implementation as the final stage in the model (Operation) includes the list of actions and factors that are relevant to project servicing and maintenance. An iteratively evolving chatbot service will return periodically to the Execution phase to develop and release new versions with improved capabilities and features. Using the parts of the model associated with these stages, teams can be given guidance on how to adjust their work and improve their approaches. At the same time, the model can support the ongoing management of chatbots as a checklist against current management approaches, and project outcomes can be assessed and adjusted as appropriate. A review of the overall objective, metrics, CSFs, and Key Actions can facilitate adjustments that improve project organisation and outcomes. The model can help identify and address gaps, solve known problems, or improve the performance of an already successful project.

5.2. Chatbots and Data Privacy

As AI continues to penetrate many areas, there is widespread anticipation of its transformative impact. At the same time, experts from a variety of backgrounds are raising concerns about the emerging risks associated with the use of this technology. Jessani [73] points out that although AI has been applied for a relatively long time, the recent growth of generative AI presents new challenges from a data privacy perspective. ChatGPT, as well as many similar LLMs, uses huge amounts of data from the open internet and other sources to conduct their work. This leads to various confrontations between the creators of language models and content authors. For instance, The New York Times sued OpenAI and Microsoft for copyright infringement over the use of millions of its articles to train chatbots that now compete with the newspaper’s own material [74].

The General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) are the major data protection regulations which concern data privacy in the EU countries and California (USA), respectively. Additionally, some industry-related acts supplement them with more specific requirements and standards related to particular industries. Jessani [73] argues that these existing regulations are quite applicable to chatbot developers and largely capable of regulating this industry. However, the author also points to many new ways for AI to collect, process, and use information that cannot be clearly assessed from the perspective of existing data protection laws. For example, the information provided by the user when interacting with a chatbot can be used by the latter to adapt its behaviour and provide the user with a personalised service, as well as to display targeted advertising. It is difficult to say whether the law requires the user’s consent for the information they provide to be used in this way, but many users may find such behaviour unethical. Carmichael [75] adds that the media regularly presents new cases of the incorrect and harmful behaviour of AI, which include examples of gender and racial discrimination, as well as other forms of bias. Such cases may not always be in breach of current legislation, but they trigger negative public reactions and widespread debate.

Sebastian [76] summarises common methods for maintaining the privacy and security of data used when training or operating chatbots but claims that many of these methods could limit chatbot capabilities, increase development costs, and affect the user experience. Yang et al. [77] also note that companies developing chatbots face a significant challenge in balancing the need for data security with maintaining the effectiveness of their services. The authors analyse the security threats related to chatbots and highlight the challenges that chatbot developers face when trying to meet high privacy and security standards. The lack of comprehensive rules on how to keep chatbots safe is associated with high risks and the need to invest in new technical solutions. This is particularly difficult for smaller companies with limited budgets. Development teams also vary in their understanding of security, so companies need to invest in proper training for their employees to avoid security issues. Gaining the trust of chatbot users is another big privacy issue because if users do not fully understand how their data are being used, they are more likely to avoid using chatbots. Concerning the practical application of the GDPR for development companies, Wolford [78] states that data protection principles should be a priority. Accordingly, data protection requirements should form the basis of the design and analysis of every new digital product and service.

5.3. The Future of Chatbots and AI

The future of chatbots in business is directly connected to how companies will address many of the challenges identified in this study and what new methods and tools will be created to help them. Some of these challenges are related to the high volume and complexity of integration tasks. In order for a chatbot to perform its functions, companies need to find a way to pass the necessary data to it and create an interface via an API or other method to support an effective two-way data exchange. Many of the existing ISs in organisations are not set up to accommodate for this. Looking to the future, this is likely to be solved in one of two ways: either the creators of chatbot platforms will develop special connectors for quick connections to popular enterprise ISs; or software vendors will embed chatbot solutions into their products, providing a way for companies to deploy chatbots relatively quickly.

The creation of domain- and function-specific chatbots is another trend that will aid companies in addressing the multiple challenges associated with the complexity of integration work, the high cost and duration of implementation tasks, the poor quality of vendor services, and unjustified user expectations. By investing significant resources in creating high-quality niche solutions, such as chatbots for HR functions, IT helpdesk, sales and marketing and other specific process areas, and considering the specifics of individual industries, chatbot solution providers can significantly improve the quality, performance, and applicability of chatbot solutions whilst at the same time reducing the cost of chatbot implementation projects.

In this context, Chakrabarti et al. [70] predict that chatbots in companies will be organised into groups of virtual assistants that perform different functions but have a single interface for the customer. This view was supported by two of the interviewees at the IT Director level (Interviewees 13 and 14), who predicted that chatbots would evolve as a system of virtual agents within an organisation. Each individual bot will be quite autonomous, performing one or a very small number of functions. However, unlike Chakrabarti et al. [70], the interviewees suggested that the virtual assistants should be represented by different virtual personas. This, in many ways, replicates how customer service performs in certain environments today with human beings. So, for example, in an airline company, one chatbot may be responsible for selling tickets, a second one will perform check-in operations, and a third one will help issue a ticket refund if necessary. This is expected to help solve some of the problems associated with the dialogue interface and the limited functionality of chatbots, as the inconvenience of using chatbots is often due to users not understanding the functions of a chatbot, namely what it can and what it cannot do.

The future role of chatbots is also integrally linked with the future of AI, which has the potential to radically change whole industries, completely transforming the way in which people use traditional services. The forthcoming AI revolution aims to substitute humans in performing mental tasks and potentially create machines that can complete all jobs now executed by humans, only faster and more accurately. The key question then concerns the future role of humans in these circumstances. Makridakis [79] suggests four possible perspectives on this: the optimistic view, the pessimistic stance, the pragmatic interpretation, and the doubters. Without exploring all four in detail, the optimistic perspective suggests that achievements in nanotechnology, robotics, and genetics will extend human abilities with extra memory, calculation resources, and solutions that make it possible to share knowledge between humans and machines. Robots will take responsibility for all work, leaving people to spend time as they want, performing only those activities that they wish to pursue. In this scenario, chatbots, as a communication interface between humans, are largely redundant as machines and human brains can communicate directly via a network. However, a dialogue in oral, written, or mental form may remain a popular means to give orders to robots. The pragmatists also take a positive view, predicting that powerful AI can be controlled by regulations, special algorithms, or a chip that can prevent any dangerous behaviour. Markoff [80] mentions two scenarios related to the forthcoming future of AI and people. The first one presumes that AI will substitute humans in completing their jobs better and more cheaply. In the second scenario, new technology will extend humans’ capabilities and improve their overall performance. Being driven by their commercial interest, businesses push chatbots to follow the first way and anticipate having bots instead of expensive cell centre operators, service managers, sales personnel, and others. The doubters do not accept that AI will ever become a threat to people. Even when AI becomes powerful enough to perform better than humans, it fails to compete with the creativity of people. Jankel [81] describes the advantage of humans as an ability to break the rules and be “anti-algorithmic”. Such behaviour is impossible for robots due to their algorithmic nature, whereas creative breakthroughs are usually performed by leaders who sometimes think irrationally. In the pessimistic future, robots will become more intelligent while social problems become more complex. Humans let the machines participate in many vital activities which they perform better than humans. This results in a dangerous situation in which the machines make all the critical decisions, but chatbots remain an essential interface to communicate with machines.

The creation of ChatGPT and its release to the public at the end of 2022 shows that the development of AI is moving much faster than previous research suggested [82,83]. Goldman Sachs Research [84] concluded that more than 75% of the 900 jobs analysed can be automated to some extent using ChatGPT. At the same time, the authors argued that there is no reason to believe that robots will completely replace humans in the near future. Rather, in most cases, AI will complement a person, extending their capabilities, performing routine tasks, and leaving decision-making and creative tasks to humans. The latest Microsoft research [85] shows that the most recent release of ChatGPT (GPT-4) demonstrates breakthrough results in solving original problems from various fields, including mathematics, programming, physics, medicine, psychology, etc. In addition, the model perfectly generates high-quality, concise text and, according to many tests, can be considered a largely Artificial General Intelligence (AGI) system [86].

Irrespective of which future materialises, AI will play a critical role in our business life and wider society. The machines may substitute people in certain areas or augment their skills, they may become a threat or an aid to humanity, but they are already becoming a part of our daily lives. A retrospective analysis of the industrial revolutions of the past suggests that each technological breakthrough produces more new jobs than those that it eliminates [87]. This supports an optimistic view of the future, despite the many unknowns that lie ahead.

6. Conclusions

This article has put forward a new model for chatbot design and implementation, based on an initial scoping review of the relevant literature, the development of a provisional conceptual framework, and an analysis of primary interview material and field observations. This Conclusion section first summarises the overall contribution of this work, then looks at its limitations, and then outlines possible areas for future research in the field.

6.1. Contribution to the Theory and Practice

The main contribution of this research is the new model for successful chatbot implementation, which is designed to support the development process and deliver the expected outcomes of such projects. This will be of interest to researchers and practitioners involved in the implementation of virtual assistants and other projects related to new technology adoption. It provides answers to common questions about chatbot integration and offers a holistic, universal approach that links strategic business objectives to the practical steps needed to achieve successful chatbot projects.

The design of the model builds upon several widely used models and frameworks, the elements of which are combined into a new structure. This research confirms the universality and wide applicability of the TOE framework [54], as well as the possibility and suitability of making modifications to its structure depending on the context of the application. The stage model by Nicholas and Steyn [64] became a part of the design of the new model, adding a project management perspective to the implementation of chatbots, and illustrating how such models can be used effectively in combination with other models, complementing the design with a process-oriented approach. The top-down approach proposed by Earl [60] has proven its flexibility and applicability in the context of addressing contemporary project planning issues. Finally, the popular CSF method [38] was used as a tool to investigate the factors that determine the success of chatbot implementation projects. This study thus extends the existing knowledge of the models and frameworks mentioned above and demonstrates the possibilities of their application in a new context, as well as in combination with each other.

Flexibility, versatility, and its generic nature are the characteristics of this model, which is designed to support teams from different countries and regions in chatbot development projects. It is relevant to projects across a range of industries, especially those aimed at automating communication between companies and their customers. It is designed for large- and medium-sized companies that want to automate routine processes, although the model can also be used in small company projects, in which case managers applying the model are advised to omit redundant elements. The model does not specifically address voice assistants, but teams building voice bots may still find it valuable in considering the specific aspects of voice development.

Although the model may be of most use to project managers responsible for implementing chatbots in organisations, it can also be useful for other team members and stakeholders, including the project sponsor, IT directors, and representatives from the contractor companies. IT integrator teams that provide chatbot implementation services can also benefit from its use by improving their processes and understanding the needs of client organisations and meeting their expectations.

6.2. Limitations and Future Research

Despite the contribution of this research outlined above, it clearly has its limitations. From a methodological viewpoint, the selection of interviewees was limited by the availability of appropriate personnel. However, fifteen respondents took part in the study, representing six industries, three countries, and speaking two languages. This diversity allowed for the identification of the characteristics of chatbot projects that are applicable cross-industry and internationally. Nevertheless, additional interviews could enhance and improve the possibilities for the generalisation and wider applicability of the model.

Although the research focused on just one case study, the use of a multi-method approach and two data collection techniques—interviews and observation—maximised the value of the available data. Further research could include parallel case studies in different companies in various industry sectors that would provide a richer and more diverse set of research data. The model is generic enough to be configurable and customisable. The Key Actions layer, in particular, should be considered as a preliminary list of broad recommendations that can be significantly expanded and adapted to the particular circumstances of each project. The practical testing of the model in real project implementations can help evolve the model, and the use of an action research strategy would provide an opportunity to study successive cycles of chatbot development, thereby enabling iterative improvement and optimisation. A focus on specific issues such as approaches and methodologies for platform selection and the impact on project outcomes would also be of value.

Finally, it is essential that the framework be regularly reviewed and updated in line with the latest technological advances in the industry to ensure the relevance of the model and its potential application in practice. In recent years, there has been a rapid development of LLMs and their application to solving practical problems, including the creation of chatbots. OpenAI’s ChatGPT, for example, has shown impressive results since the end of 2022, and many experts predict that this technology will transform entire industries in the near future. The rapid evolution of this technology will require the assessment and incorporation of updates and revisions to the model put forward here. ChatGPT can be considered one of the most advanced NLP tools, but nevertheless, the issues addressed in this research, as well as the new model to aid teams in solving chatbot problems in the context of an organisation, remain relevant today.