Integrating Chatbot Media Automations in Professional Journalism: An Evaluation Framework

Abstract

Interactivity has been a very sought-after feature in professional journalism ever since the media industry transitioned from print into the online space. Within this context, chatbots started to infiltrate the media sphere and provide news organizations with new and innovative ways to create and share their content, with an even larger emphasis on back-and-forth communication and news reporting personalization. The present research highlights two important factors that can determine the efficient integration of chatbots in professional journalism: the feasibility of chatbot programming by journalists without a background in computer science using coding-free platforms and the usability of the created chatbot agents for news reporting to the audience. This paper aims to review some of the most popular, coding-free chatbot creation platforms that are available to journalists today. To that end, a three-phase evaluation framework is introduced. First off, the interactivity features that they offer to media industry workers are evaluated using an appropriate metrics framework. Secondly, a two- part workshop is conducted where journalists use the aforementioned platforms to create their own chatbot news reporting agents with minimum training, and lastly, the created chatbots are evaluated by a larger audience concerning the usability and overall user experience.

1. Introduction

The turn of the 20th century was characterized by many things, but perhaps one of the most important and influential changes was the introduction of information and communication technologies (ICT) that revolutionized many different facets of daily life [1]. A variety of professional fields got affected by the arrival of these technologies, and a prime example of that was the media and news industry, a sector always renowned for its unique entanglement with technological developments [2]. The way these new technologies were incorporated into the workflow of modern journalists were many and spanned across various fields of application, including the likes of content production and communication between the journalist and the audience [3]. Perhaps one of the most intriguing uses found for these new technologies was that of automated conversational agents. Commonly referred to as “chatbots”, these programs were capable of communicating with users via the use of natural language [4], and it didn’t take long for the media industry to realize the potentially massive benefits these digital tools could have in the process of creation, and particularly dissemination, of news content. Their rate of adoption varies based on a multitude of factors, but it is likely to increase with the rise of compatibility and the existence of more related expertise in the field [5]. In accordance with those parameters, chatbots in the field of journalism started as a relatively rare commodity, but as the technology matured, more and more media organizations started implementing them in their list of means to capture audience attention. As the development of conversational agents started becoming more and more focused over the years, and chatbot technology reached its commercial stage, the wider accessibility of online creation platforms allowed the average journalists to involve themselves in this interactive procedure and create new and useful tools, even without the need for coding skills or specialized ICT knowledge.

Ever since the transition to WEB 2.0, there has been one attribute of this new and ever-evolving media landscape that has been considered particularly advantageous for its ability to better engage with audiences: interactivity [6]. Media organizations constantly seek more ways to captivate the attention of the public, and chatbots seem to fit that bill perfectly, seeing as they are a type of tool designed based on principles that center on communication and conversational intelligence [7]. To this end, this manuscript attempts to highlight and overview some of the most well-known and readily available contemporary digital tools that journalists have at their disposal for chatbot creation. At the same time, it aims to present an evaluation framework for said tools, which includes allocating them to certain categories based on the amount and type of interactivity that they offer, as well as judging their performance and accessibility in general. The selected programs are specifically targeted to not require any amount of prior coding experience on behalf of the journalist for them to fully function and create a finished product capable of interacting with audiences and establishing new communication channels.

The motivation behind this work stems from the general question of whether it is feasible to integrate chatbots in professional journalism and, if so, under what circumstances. This question can be broken down into whether the existing platforms are intuitive and ready to be used by professional journalists to create chatbots and whether those bots are helpful, useful, and engaging for a broader audience. To that extent, it is important to design a framework for their evaluation that can set the directives of how they can be improved upon. This paper considers some of the most important platforms that are being used right now and presents a three-fold evaluation of them, addressing the scientific questions that have been set.

Other than this introductory segment, the remainder of the paper is organized as follows: In Section 2, the related work on chatbots is presented, including a brief overview of the term, research on journalistic chatbots, and the role of interactivity. Section 3 introduces the proposed framework for integrating chatbots in professional journalism alongside the evaluation methodology that has been adopted. The selected creation tools are also presented there. In Section 4, the evaluation results are discussed, and finally, Section 5 summarizes the conclusions and the future research plans for this project.

2. Related Work

2.1. Chatbot Categories and Architecture

The term “chatbot” can be defined in several ways, and many of them include a wide variety of programs. For this study, we are going to define a chatbot as a software application that utilizes natural language to communicate with humans [8]. Such programs have a very wide variety of applications thanks to their flexibility, and they are systematically utilized by sectors like customer service for their ability to converse with humans in a relatively natural and meaningful manner [9].

When it comes to a taxonomy of the available chatbot models, one of the major ways in which conversational agents can be distinguished from one another is their architectural design. Specifically, there are two prevailing categories that all chatbots fall under, depending on what procedure they follow to respond to users: “retrieval-based” chatbots and “generative” chatbots [10]. The first category–that of retrieval-based chatbots–comprises programs designed to communicate with the user via predetermined responses [11]. Conversational agents that fall under this model operate by searching for a reply in a pre-established repository and serving it to the user according to their input [12]. The large majority of chatbots that can be found on the web today follow this model of operation due to its simpler structure compared to the alternatives [11]. However, even though the architecture of retrieval-based chatbots presents many advantages–especially for users that don’t have any coding experience or specific ICT knowledge–this method is subject to some limitations that can potentially restrict the scope of the final product. Those limitations mainly relate to the fact that the procedure of choosing between a set of already existing responses makes it very challenging to customize the chatbot for particular situations and relegates it to having a more “passive” role in conversations [12,13].

On the other end of the spectrum, the category of generative chatbots consists of software capable of creating new responses from scratch to better match the queries of the user with the help of techniques such as Machine Translation [14]. The result of this is an often fairly convincing effect, where the chatbot can uphold a conversation with a user in a very natural manner. This procedure is a lot more demanding than its retrieval-based counterpart since it requires a substantial training dataset to function properly, but it provides the significant benefit of being able to respond to user inputs for which predetermined responses don’t exist, something that the retrieval-based model falls short of. At the same time, however, it is far more likely for it to exhibit major grammatical errors, or make other similar mistakes, compared to a solution that relies on retrieving responses from a designated archive [15].

Natural Language Processing has been advancing rapidly over the past years, and the utilization of new and innovative techniques, such as Deep Learning, suggests that generative models will be the future of chatbots moving forward [16]. However, the typically enormous datasets required for properly training such systems, as well as the complexity of text generation, which constitute two of their primary characteristics [15], dictate that retrieval-based solutions will remain the far more common alternative for the time being, especially when it comes to chatbot creators that are less knowledgeable in the programming department.

2.2. Chatbots and Media Automations in Journalism

Many people consider chatbots to be a fairly recent development, given the fact that commercial use of this technology has only started to become prominent in the past decade or so. In reality, however, efforts by researchers to create a program capable of conversing with humans naturally have been going on for the better part of a century. Often regarded as the predecessor of all chatbots, ELIZA was created by the German scientist Joseph Weizenbaum [17] and managed to engage its conversational partners to such a degree that many of them reported that they believed they were addressing a real person [18]. From that point onwards, many researchers attempted to simulate human communication with computer programs, which led to the creation of many prominent examples of capable conversational agents over the years.

Despite the rich historical development of chatbot technology, however, what indeed constitutes a relatively new phenomenon is the inclusion of these programs in the process of creating and disseminating news. This was a by-product of the general tendency for computational procedures to graduallywork themselves into the practices of the journalistic profession, subsequently creating what was later called “automated journalism”, frequently also referred to as “robotic”, “algorithmic”, or “computational” journalism [19]. Chatbots in journalism can be used for efficient user interaction in the sense that they provide a more human-like way of navigating and accessing news-related content. Generally speaking, chatbots are a sub-section of the more general term AI agent, which is used to describe artificially intelligent software capable of performing a variety of tasks. In the case of rule-based chatbots, these tasks center around providing the user with content based on their answers to the predefined questions of a conversation. In a more sophisticated approach, chatbots can also generate original content, through algorithmic techniques like text generation, document summarization etc., by applying basic concepts of algorithmic journalism.

As far as chatbots are concerned, many decades had to come by until the first proper use of one for purely journalistic purposes could be identified. Specifically, in 2014, the Los Angeles Times employed the services of an automated AI agent that was capable of data extraction and simplified content creation. The program, aptly named “Quakebot”, was tasked with monitoring data from the U.S. Geological Survey and utilizing any information it could find regarding seismic activity to write and publish simple reports in realtime. Even though Quakebot only amounted to a very simple use of this type of technology–and even at the time, it was being compared to nothing more than an intern with a lot of time in their hands [20]—it still embodied the essence of what the introduction of such programs meant for the journalistic profession: cheap and easy to maintain labor, that could substitute, or even surpass, human journalists in some tedious and time-consuming tasks.

On that basis, what followed during the next half of the decade was a massive increase in the introduction of automated elements into journalism, with chatbots being one of the technologies at the forefront of that wave of change, as, over time, many news organizations started realizing the potential benefits of automatic content dissemination [21]. Companies gradually started utilizing the unique qualities of conversational agents to spread their content more efficiently-particularly through websites like social media–which led to the creation of the term “News Bots”, a sub-category of chatbots that specialize in interacting with audiences and spreading news information [22]. The inclusion of these automated programs in the distribution of news content made the whole process much more efficient and allowed audiences to interact with news organizations in new and more engaging ways that proved to be very effective, thanks in no small part to the interactivity and personalization they offered [23].

At the same time, following the example of Quakebot, chatbots capable of data extraction and content creation started becoming more prevalent as the news industry attempted to adjust to the rapidly evolving media landscape. Faced with the ever-increasing load of information available on the internet, journalists started using AI agents capable of sifting through large amounts of data and collecting relevant results for their work in a process called “Data Mining”. This procedure was able to help media staff identify stories that have editorial value, according to the parameters set by the journalist, as well as provide aid in more specific tasks such as information verification and event monitoring [24]. Similarly, automated content production proved to be yet another way to take advantage of these versatile programs in the realm of AI-assisted news. Chatbots and related algorithms were developed to produce content on their own, with little to no human intervention. Machine-generated news content is often powered by artificial intelligence or similar machine learning algorithms [25], and it has been one of the more defining factors of the journalistic profession in the past few years [26], as it has led to many upsets in the industry, provoking many researchers, as well as practitioners into questioning whether or not these programs could potentially prove to be a threat for industry workers [27,28]. The prevalence of these new digital tools is in part responsible for the cultivation of a work environment in which digital literacy is one of the most important aspects, with the imminent re-defining of journalistic skill sets coming to the forefront [28,29]. Despite that controversy, however, what remains an undeniable fact is that chatbots and other AI agents like news writing algorithms are seeing extensive use in news media production today, with many industry-leading organizations like Forbes and the New York Times utilizing them as content creators, with the final product being almost impossible to distinguish from human writing [30]. The combination of these abilities exhibited by automated software–to comb vast amounts of data and then morph the relevant information they find into a compelling narrative–has already expanded the news writing universe in a major way, and they will, in all likelihood, continue to do so as the technology improves [31], further fueling the domain of media automation in professional journalism, which can also augment the interaction through the incorporation of media agents [32].

All of the chatbot categories mentioned above have seen extensive use in the field of journalism over the past decade. Among them, however, the category of news dissemination stands out as the field of application that incorporates chatbots the most [23]. These types of chatbots are also the most relevant ones to be examined for this study, as they are designed with the explicit goal of improving audience engagement and introducing more interactivity into news distribution [33]. This practice has transformed how media is shared and consumed since it was adopted by many prominent news organizations such as CNN, The Guardian, and The Washington Post, all of which created their own versions of news-sharing chatbots by the end of 2016, which was a year that saw a particularly large surge in chatbot creation in general [34]. Figure 1 reflects the current perspective of chatbots in the context of media automation in journalism in relation to the evaluation framework, which will be proposed later in the study. More specifically, this conceptual diagram highlights the two-way relationship that is present between the individual processes in journalism and how this feedback loop is utilized in order to reach the concessions needed for the proposed evaluation framework. In turn, the evaluation framework also feeds into this information cycle in its own way by providing a way to accurately assess the usability and usefulness of relevant tools.

Overall, a variety of literature exists in the realm of chatbot usage, specifically for journalistic purposes. Researchers over the years have covered a wide spectrum of chatbot applications in the field and have tackled issues like their usage in automated news dissemination [11,22,35], information gathering [34], user newsfeed personalization [23], as well as the use of conversational agents to establish a multi-media approach in the news [33,36], or as a means of studying the relationship between the journalist and the audience [37] as well as the cross-cultural social context those bots are being employed in [38]. Despite that, however, the majority of this research–with only a few notable exceptions–seems to focus more on the potential practical uses of these programs and less on a valid framework with which chatbot creation tools can be evaluated. Even though conversational agents are undoubtedly very useful in the media sphere, it is important to realize that the average journalist is often unfamiliar with the more technical aspects of chatbot creation. To that end, there exists a distinct lack of research focusing on evaluating easy-to-access tools for the average worker in the media industry, a flaw that this paper aims to remedy by prioritizing the opinions of individuals who are familiar with the field of journalism and integrating them via a practical use-case scenario.

2.3. Interactivity in Chatbot Design

Interactivity in media is a concept that can be defined in many different ways, depending on the angle through which one approaches it. Different parties have attempted to provide different definitions for it, depending on whether the interaction is an observer between people, between a user and a machine, or even based on the ability of a user to control and alter a given message [39]. Perhaps the most all-encompassing definition for the term, however, was given by Liu and Shrum [40], who describe it as “The degree to which two or more communication parties can act on each other, on the communication medium, and on the messages and the degree to which such influences are synchronized”. This admittedly broad definition manages to encapsulate every relevant aspect of interactivity when it comes to its applications in online media and lends itself well to the examination of programs like chatbots within the medium. After all, a prerequisite for interactivity is the multi-directional flow of information between the user and the producer of the content, where the audience is the recipient of information but also provides some sort of feedback on it [41].

Interactivity is generally regarded as the most defining factor of new media, and oftentimes it can even be considered a necessity for them [42]. Based on that, media companies started to focus more and more on interactive features for their content ever since the internet became paramount for the news industry [43] by introducing several different features that were impossible to include in traditional media. Those features are implemented because they provide numerous benefits, like human-to-human interaction (or, in some cases–like the ones we will examine later–human-like interaction), emphasize socialization, and–other than audience engagement–they can also help in the creation of online communities of readers [44] which is an added benefit, associated with even more positive outcomes for media organizations. Generally, interactivity seems to be a driving force behind longer audience engagement in media, which is not at all surprising, given the fact that it has also been proven to significantly boost user engagement, even throughout vastly different mediums like public displays [45] and digital games [46]. There is also a strong association between the need of the audience for entertainment and the use of medium interactive features to fulfill that need [44].

Many researchers have suggested models with which interactivity can be measured or segmented into different types depending on a variety of factors. One such model for categorizing interactivity that is particularly relevant to this study was proposed by Jensen [47], whose definition of interactivity is similar to the one given in this chapter, “the measure of a media’s potential ability to let the user exert an influence on the content and/or form of the mediated communication”. Based on that definition, four sub-types of interactivity are proposed: Transmissional, Consultational, Conversational, and Registrational interactivity. The first one, Transmisional interactivity, refers to a medium’s ability to allow the user to choose something out of a constant stream of content in a system where there is no two-way communication, like a Teletext service, for example. Consultational interactivity is similar but requires the user to be able to choose something on demand, out of an already pre-produced library of information, like an online encyclopedia, or a streaming service, which suggests the existence of a return channel. The third one, Conversational interactivity, applies to systems that allow the user to create and input their own information in a two-way media channel like an e-mail client. Finally, Registrational interactivity refers to a medium’s ability to register information from a user and adapt or respond to it. This applies to more “intelligent” systems that can answer specific requests and cater to a user’s needs. It is worth noting that–even though this isn’t explicitly mentioned by Jensen in his original paper [47]–this segregation seems to be presented in ascending order of sorts, with each subsequent category allowing more and more liberties to the user, and providing them with more freedom to influence the outcome of the interaction. As part of the proposed evaluation framework, this paper is going to attempt to categorize some of the most notable examples of chatbot creation platforms into the above categories, as well as take a more in-depth look into their specific characteristics to distinguish between them in a manner that will be further elaborated below, using a three-way evaluation approach.

3. Integrating Chatbots in Professional Journalism

3.1. A Framework for Journalism-Oriented Chatbots

As stated in previous sections, journalism is already intertwined with chatbot usage to a significant degree. Media organizations, as well as journalists as individuals, routinely resort to chatbots as a tool for their professional needs. There are still, however, some open questions concerning the integration of chatbots in journalism. These refer to the use cases where chatbots can improve news reporting, the relationship between an organization and the audience, and also how capable, confident and engaged journalists and the audience feel when it comes to using them. For professional journalists, this means programming chatbots to bring their work forward, and for the audience, it means using a chatbot to navigate news content. Even though there is no question that news dissemination is by far the most widely used application for chatbots in the field of media, one would be correct to assume that these versatile programs can also be utilized for other, more unique scenarios. For example, the interactive platform provided by conversational agents can be used as a “news assistant” of sorts by transferring additional information to the reader regarding the topic of interest when they request it. This approach has been examined by embedding chatbots within news articles [33], and similarly, these applications can also be utilized in other creative ways, such as gathering information from the public [34]. It is possible to envision the usage of chatbots for other creative work as well, although in most cases, more advanced conversational intelligence would be required in order for these programs to present a compelling case for their usage in these scenarios.

In this manuscript, we focus on how this symbiotic relationship between chatbot and journalist can be improved upon by examining some of the most accessible alternatives for chatbot creation aimed at workers in the media sphere that don’t possess any programming knowledge or any other similar expertise in the field of ICT. Given the above, to design a use case scenario for integrating approachable chatbot building in professional journalism, we need to account for two discrete aspects of chatbot usage in journalism: Front-end usage, also known as the part of a program the user interacts with, as well as Back-end usage, which refers to the part that the designer of a program interacts with.

The frontend refers to the interface of the end product–in this case, the finalized News Bot- and it is targeted at news consumers. This is the part of the program that the average user will be interacting with to receive news updates, as it will be explained in later sections. For the purpose of the front-end evaluation, we will be examining how intuitive the end product is, how much it helps the average news consumer, and how interactive and helpful the chatbot is perceived overall.

When it comes to the backend refers to the chatbot creation platforms themselves. These are going to play the role of a mediator in the chatbot creation process, as the journalist will be interacting with each of them to build a chatbot from scratch. For this part of the evaluation, we focus on the ease of use when it comes to creating a new chatbot, the interactivity features that are included in each platform, as well as all the monitoring and quality of life features that are presented in the journalist upon creation of the program. The usefulness of each platform to the journalist is also taken into account. With all of that in mind, the following use case scenario was developed by the researchers for the subsequent evaluation of the examined tools and platforms.

Use Case Scenario

This use case scenario aims to provide context as to how the chatbot creation platforms included in this manuscript will be evaluated. It does so by highlighting a practical, real-world example from the perspectives of the main stakeholders in the procedure while aiming to be both realistic and clearly understood.

The primary actor in this scenario is a professional journalist that works in the media industry on behalf of a news organization. This person will be the one interfacing with the back end of the chatbot construction platforms to create the desired product. The specific type of journalist represented in this scenario is well accustomed to digital technologies and has at least a basic understanding of current information and communication technologies. Having said that, however, no programming skills or any other specialized ICT knowledge is assumed, as the chatbot creation platforms have been specifically chosen to provide a coding-free experience and can be used by any professional willing to invest time in them. The primary motivation of this actor is to create a product that will adequately serve the audience of their news organization and increase user engagement.

On the other end of the spectrum, the secondary actor is the end-user of the product, which in this case is the news consumer. This person will be the one receiving news content by interacting with the product created by the journalist. Similar to the primary actor in this scenario, this type of user has at least a basic understanding of current technologies and chooses to primarily receive their news online. The basic motivation of this actor is to consume news articles in an easy and digestible way.

The systems used in this particular use case scenario are the three chosen chatbot creation platforms that are being evaluated, which will be presented in the following section. The journalist, as the primary actor, is tasked with creating a chatbot on behalf of their organization to better disseminate news content. To create a successful product for journalistic use that accommodates the needs of the presented research, this chatbot needs to align with certain parameters. For this study, it was determined that the final product needs to be able to disseminate news content:

• Automatically;
• Systematically;
• Interactively.

The automation of the news dissemination procedure is the reason why chatbots are being used in the first place. In addition to that, however, the requested application needs to be able to prescribe certain characteristics to the news consumption process by actively making it more engaging. For this reason, the journalist needs to be able to utilize the environment of the chatbot creation platforms to accomplish these goals in the most accessible way possible. To that end, parameters like the complexity of available features, the overall responsiveness as well as the ease of use were chosen as some of the most important variables to be taken into account for the final evaluation.

For this work, we assume that the end-user will be contacted by the chatbot in intervals that range from a single day to an entire week, as per their subscription preferences. In that scenario, the chatbot creation platform needs to be able to provide an end product capable of engaging with the user systematically and interactively without overwhelming them while keeping the entire procedure as simple and fluent as possible. Additional options, like features that accommodate back-and-forth communication between the primary and secondary actors (the journalist and the news consumer), are also taken into account.

3.2. Methodology for the Evaluation of Chatbot Platforms for Journalism

As already explained, a big aspect of the motivation of this work is to analyze the feasibility of integrating chatbots into professional journalism. To answer this, it is important to analyze whether journalists can design chatbots without prior training and whether the audience feels confident and engaged in using the frontend. For this reason, we have taken into consideration three important platforms that were evaluated in the aforementioned directions. In the next sections, the main characteristics of the platforms, as well as the evaluation experiments, are presented and explained.

3.2.1. Chatbot Creation Platforms

Chatbots, like most technologies in their infancy, started entering the commercial space slowly and experimentally. Over the years, however, the market started becoming more and more saturated with a variety of platforms capable of chatbot creation. The ability to utilize those programs for easier news dissemination, among other things, presented many benefits for media organizations, as discussed in earlier chapters, and thus it was adopted fairly quickly, to the point of becoming an industry standard within a few years. Specifically, a large surge in chatbot integration was observed during 2016, with numerous media companies announcing their implementations within the span of a few months [34]. This wave of chatbot innovation can be partially attributed to social media, specifically Facebook’s decision to open up its ecosystem to developers by natively supporting chatbots through its messaging service. This marks a turning point for the media landscape, not only for the news organizations themselves but also for the consumers, as the wider availability of chatbots also plays a major role in the democratization of certain services since these programs can be made available to a very large number of users via platforms like social media and messaging applications [48]. Nowadays, online tools exist that allow the average worker in the media industry to take advantage of their coding-free environment and create a fully functional conversational bot for their own journalistic purposes. This approach has even been adopted by many prominent media organizations that decide to utilize these platforms for their chatbot needs. Some of these companies, like CNN, preferred to outsource the creation of their bot to a third party, in this instance, a chatbot-building company named SPECTRM. Many other industry giants, however, decided to try their hands on these platforms and experiment with what those online tools could offer them by allowing their journalists to create their own version of a chatbot for use in the newsroom.

It is important to note that none of the currently existing online creation platforms were explicitly developed to build chatbots suitable for journalistic work. Most of the software platforms available online cater to a generalist audience and, in many cases, specialize more by offering extra features for some sectors that use chatbots very extensively, like customer service and marketing. In that context, the media industry has been trying to take advantage of the already existing tools to fulfill the needs of the audience by adapting to their limitations and nuances. Since this paper aims to evaluate the most readily available online tools that can be used by the average worker in the media industry, the criteria with which the choice of platforms was made are as follows.

First off, the creation platform should include a user-friendly environment that doesn’t require any coding expertise or comprehensive ICT knowledge on behalf of the journalist. The reason behind this, when it comes to the scope of the paper, is the need to identify the tools that can appeal to the widest possible audience while taking into account that many media industry workers are not entirely familiar with concepts like programming and creating an application for the general public. In addition to that, however, the accessibility of the user interface plays a particularly important role, as the chosen platforms will be evaluated–in part–via the use of a workshop, as will be explained in detail in the upcoming sections. This narrows the list of selected platforms down to options that lend themselves to be easily presented and taught in a workshop setting, where the participants can follow through with creating their chatbot before being asked to evaluate the user experience.

Additionally, the selected platforms must offer a usable free plan that journalists can utilize to create a fully functional bot. Most companies in the field of chatbot creation operate under a hybrid business model, where they offer a free plan with limited functionality and a premium one with more features (Business models for the platforms used in the study can be found here: (1) https://quriobot.com/pricing (accessed on 28 September 2022) (2) https://snatchbot.me/pricing (accessed on 28 September 2022) (3) https://chatfuel.com/pricing (accessed on 28 September 2022)). We are mostly interested in tools that allow users the freedom to begin experimenting with chatbot creation without any commitments, and thus only platforms with a usable free plan were taken into account. Finally, the chosen tools must exhibit characteristics that align with usage in a journalistic environment. This includes a list of criteria that will be further elaborated on in the evaluation section. Based on the above, the three following chatbot-building tools were chosen and will be examined below: Quriobot, SnatchBot, and Chatfuel. What follows is a short description of the way each one of the selected platforms operates.

Quriobot

The first chatbot creation platform examined was Quriobot (https://quriobot.com) (accessed on 28 September 2022). It consists of an all-in-one solution for chatbot creation, as it provides the user with comprehensive options for creating and customizing the final product. This platform has been utilized in the past for journalistic purposes and, among other things, for implementations, including information collected from users [34]. The Dutch public broadcasting company KRO-NCRV is also listed as an official partner on the company website. The Quriobot platform operates by allowing users to string together several “steps” to create the final product (Figure 2). The end-user can navigate the conversation by clicking on pre-assigned buttons that guide them through the step sequences. Each step represents a specific action with predetermined ways of interaction between the user and the bot, so the complexity of the chatbot hinges on the variety of steps available. More steps are being added over time, but as of the time of writing, a good selection already exists. Examples of steps include questions with a simple button for a Yes/No answer, open-ended questions that allow users to type out a response, contact forms, fields for uploading files, and so on. While there is a certain degree of choice for the user during their interaction–as with any chatbot that is properly thought out–Quriobot isn’t capable of recognizing text input as a means of navigating the interface. As mentioned above, users can still input text in specific scenarios, but that text is only stored as an answer inside an internal repository for the journalist to decipher at a later time.

SnatchBot

SnatchBot (https://snatchbot.me) (accessed on 28 September 2022) is the first creation tool on the list that exhibits natural language processing capabilities and does so in a coding-free manner, in addition to the more basic creation method that resembles the other two platforms (Figure 3). This tool is capable of understanding the user’s intent, proper training, and acting accordingly to fulfill their requests. The way this is accomplished is by having the chatbot distinguish between two different types of text inputs, named Entities and Intent. Entities correspond to anything that can be named, such as objects, places, time, and so on. Intent, on the other hand, refers to the purpose behind the user’s words in a sentence, like, for example: “I want to see the latest news”. Aspiring chatbot creators can use this model by providing it with a training dataset for each of the two categories and then running the “train” command within the interface. The platform’s processing capabilities will then take over to “teach” the chatbot to recognize certain things based on the provided data. If journalists are after the creation of a specialized chatbot, they can import—or create from scratch within the interface–the appropriate training data. However, SnatchBot also provides a handful of pre-trained models, even for free users, that can be utilized to recognize things like places, dates, currency-related terms, and even negative and positive words, which can prove very helpful for certain tasks like sentiment analysis. These models are actually recommended by the platform as the default setting since they cover a wide variety of situations.

Chatfuel

Out of the three tools examined in this paper, Chatfuel (https://chatfuel.com) (accessed on 28 September 2022) is the platform with the most public association with the media industry. News agencies like MSNBC, BuzzFeed, and the Australian Broadcasting Corporation, to name a few, have all been listed in the past as official partners of the company on the Chatfuel website. On top of that, Forbes has also publicly stated their partnership with Chatfeul for the creation of their Telegram bot. This creation tool attempts to combine ease of use with a variety of advanced features. The way the platform operates is not dissimilar to Quriobot in the sense that all actions are represented by “blocks” (Figure 4). Each individual block can contain multiple actions, thus making them more feature-rich compared to the previously seen “steps”. The creator can use visual programming to combine these blocks into a sequence that is usable by the audience. The most appealing characteristic of Chatfuel, however, is its ability to allow the audience to navigate the interface not only with buttons but with the use of natural language as well by typing out exactly what they want to do next, which allows them to bypass the predetermined path set by the journalist in favor of jumping directly to the task that is most relevant to them. The way this works from a creation standpoint is via the use of pattern matching, as the journalist can assign certain keywords to specific actions. The program will then try to identify those keywords within the user’s text and forward them to the most relevant block. The keyword creation process can be as simple as the addition of a few shortcuts or as complex as a full list of recognized phrases, creating the illusion of an intelligent conversational agent. While this process is often very successful in convincing the end-user that the chatbot understands them, in reality, there is no natural language processing going on, but rather a comparison between their text input and a repository of responses in order to determine the most relevant answer, unlike the previously seen example of SnatchBot which is also capable of building “generative” chatbots, given the proper training.

3.2.2. Heuristic Metric-Based Evaluation of Creation Interactivity

This manuscript aims to catalog the interactivity features of the selected platforms and facilitate a more in-depth look into their specific characteristics to distinguish between them. For that purpose Heeter’s [49] model will be used, a theoretical construct that is considered by many to be one of the best attempts at categorizing interactivity, specifically in the realm of media and communication. Heeter’s model has primarily been used for online websites, but it can be adapted very well for our purposes, as it allows us to quantify the interactivity of chatbot creation platforms based on the following six parameters:

• The complexity of choice available: a metric of the provided features and the ability to customize certain parameters;
• Effort users must exert: a measurement of how user-friendly the platform is, where the higher the score, the less effort is required for its proper use;
• Responsiveness to the user: the extent of the ability of the end-user (audience) to contact the creator of the chatbot (journalist) through the interface;
• Facilitation of interpersonal communication: the chatbot’s ability to act as a medium through which users can communicate with each other;
• Ease of adding information: a user’s ability to contribute to the product by providing their own information and/or feedback;
• Monitor system use: the ability of the creator to track and measure certain parameters regarding their chatbot, such as user statistics and behaviors.

These six dimensions allow for a very comprehensive and quantitative characterization of the available features found in online creation platforms. As the first step in the proposed three-way evaluation process, a rating between 1 and 5 on a Likert scale was assigned for each one of these categories by the authors. As for the specific labels of the scale, those will be explained in more detail in the next section. In addition to the above, other than rating the selected creation platforms using Heeter’s model, each tool was also assigned to one of the interactivity categories described in Section 2.3, according to the model proposed by Jensen [47]. We utilized these two models in tandem for this step of the evaluation, as one of them serves as a more general categorization of all the examined tools, whereas the other provides more specific details when it comes to the individual aspects of each platform.

3.2.3. Chatbot Creation Workshop and Back-End Usability Evaluation

To evaluate the intuitiveness of the back-end of each platform (the programming of each chatbot) and the feasibility of the creation of chatbots by journalists with minimum background knowledge in computer science, a two-part workshop was organized. To begin with, as the second step in the evaluation process, a small group was formed, consisting of eight students from the School of Journalism of the Aristotle University of Thessaloniki. All participants were enrolled in the course of Human-Computer Interaction, an introductory course on application UX design and principles and usability evaluation. The participants had no previous experience with chatbot design. The workshop provided a short hands-on tutorial on every platform. The attendees were then requested to create a simple, retrieval-based chatbot for personalized news reporting. Participants spent 30 min on each platform, including the short tutorial and the time given to build the chatbot. In the end, they were asked to complete a survey concerning the usability of every platform and their experience with it. The workshop concluded with a short discussion, where participants had the opportunity to express their opinions and specific suggestions on their overall experience, as well as the different platforms.

After cataloging the experiences of individuals that are adjacent to the field of journalism, it was deemed necessary to also incorporate the opinions of working professionals in the field. To that end, the same methodology was used in order to organize a second workshop, where a small discussion group was formed consisting of seven professional journalists who underwent the exact same procedure described above. All participants were currently working–or have recently worked–in the field of journalism as of the time of the workshop and held no experience in regard to chatbot design. After the procedure was concluded, they were asked to share their opinions in regard to the creation platforms and their end products, in addition to filling out the aforementioned questionnaire.

3.2.4. Front-End Usability Evaluation

For the evaluation of the front end of each platform (interacting with each chatbot), an online survey was conducted as the final step of the proposed evaluation framework. In this experiment, the goal is to evaluate how the audience interacts with journalistic bots that have been created using the platforms under evaluation. Combined with the results from the workshop, this experiment is designed to provide insight concerning not only the usability, usefulness, and engagement of the process of chatbot programming but also how the result appeals to a broader audience. To address this, three simple chatbots were created by the research team, offering the same functionality for personalized news reporting (The sample chatbots that were used for the purposes of the survey can be found here: (1) https://botsrv2.com/qb/aPW6jrq9yvrR4ZXQ/eBYgZbjDd4r3l7jA?mode=preview (2) https://webbot.me/2eb10100d18b67dadeb5e732e5e9ff861aeed0639f80039e2c1e8c5d099b3bc8 (3) https://www.messenger.com/t/2253824708194900) (accessed on 30 May 2022). The respondents were asked to interact with each chatbot and then fill out a survey concerning the usability and their user experience with every chatbot. The survey was based on two of the most well-known frameworks for application evaluation, focusing on user interface satisfaction [50], as well as perceived usefulness and ease of use of the software [51], and a questionnaire was adapted for the specific needs of the conducted experiment. The survey was completed by a total of 162 participants, all students of the School of Journalism of the Aristotle University of Thessaloniki, enrolled in classes related to digital media.

During the survey preparation, all ethical approval procedures and rules suggested by the “Committee on Research Ethics and Conduct” of the Aristotle University of Thessaloniki were followed. The respective guidelines and information are available online at https://www.rc.auth.gr/ed/ (accessed on 8 June 2022). Moreover, the declaration of Helsinki and the MDPI directions for the case of pure observatory studies were also taken into account. Specifically, the formed questionnaire was fully anonymized, and the potential participants were informed that they agreed to the stated terms upon sending their final answers, while they had the option of quitting anytime without submitting any data.

4. Results and Discussion

4.1. Metric-Based Evaluation Results

Starting with the heuristic metric-based evaluation, the three chatbot creation platforms were judged depending on the features they offer. To begin with, each one of the selected tools was cataloged according to Jensen’s categorization [47], as explainedin previous chapters. After this process, the three chatbot creation platforms were assessed by the authors. Each one of the four judges spent the same amount of time with each of the platforms in question and then proceeded to rate the features offered by ranking them according to the six interactivity parameters mentioned in Heeter’s model [49], evaluating them using a Likert scale from 1 to 5. Clear guidelines were established for the rating procedure and for what each one of the scores on the scale represents. Specifically: A score of 1 denotes the complete absence of a feature. A score of 2 represents the existence of a feature that has a very barebones implementation (for example, very limited user-to-user interactions). A 3 is used as a baseline and is assigned to a feature that is decently developed, but within this standardized categorization is stillmissing some features compared to the competition. A score of 4 indicates a very well-implemented feature that addresses a subject from different angles (like the inclusion of multiple different ways for the audience to contact the creator of the bot). Finally, a 5 indicates a highly advanced implementation for that specific characteristic that includes a multitude of features compared to the competition (for example, a very robust analytics system that records detailed interactions between user and bot). To ensure inter-rater reliability between the four judges, a percent agreement system was used [52], and the formation of the subsequent result matrix revealed a very high agreement between the raters, with a very small amount of outlying scores. Specifically, following this clearly outlined system, the four raters proposed the exact same score for all categories, with the only exception being the SnatchBot platform, in regards to the category “Effort users must exert”. This category was rated differently between the four reviewers and received an average score of 3, as there was some slight degree of fluctuation in the opinions of the raters in regard to the ease of use of some of the features provided by this tool. This outlying observation could be the result of the highly modular system presented by SnatchBot, which left some room for interpretation as to the degree of competency required by the average user to create a functional bot. An overview of the final verdict of the interactivity features comparison, which consists of the mean values of all four raters, can be seen in

Table 1. Chatbot creation platforms and their interactivity features ranked.

Interactivity Parameters	Quriobot	SnatchBot	Chatfuel
Complexity of choice available	3	5	5
Effort users must exert	4	3	4
Responsiveness to the user	4	4	5
Facilitation of interpersonal communication	1	2	2
Ease of adding information	4	4	4
Monitor system use	5	2	3
Total:	21	20	23

below.

The Quriobot platform was the first one to be examined, and based on the implementation of the features described in the previous chapter, it was assigned to the category of Consultational interactivity tools. Users are able to request specific things from a chatbot created through this platform. However, that can’t be accomplished through text input but rather through scripted interactions. As for its individual scores based on the Heeter model, they are as follows: Complexity of choice available: 3, Effort users must exert: 4, Responsiveness to the user: 4, Facilitation of interpersonal communication: 1, Ease of adding information: 4, Monitor system use: 5, for a total of 21.

The available features Quriobot offers during the creation process can be characterized as adequate compared to other similar tools. The standout characteristic of this platform, however, is its extensive monitoring capabilities, which are part of the reason why this tool excels in the creation of chatbots aimed at data acquisition from users. All information collected by the bot during its interactions is saved and can be accessed by the journalist in an excel sheet-style interface. This includes direct user input like text and uploaded files, the exact “Step Path” that a user followed when navigating the interface, as well as comprehensive metadata like time of access, the user’s operating system, and even location. On the other end of the spectrum, Quriobot doesn’t facilitate user-to-user interaction in any way. Thus the lowest possible score was assigned for the interpersonal communication category. Specific steps that present contact forms, text input, and the ability for users to rate the chatbot also exist, so the other categories also receive an above-average score. Overall the ease of use for this platform is quite high, as it allows for visual programming, which can ease the uninitiated into chatbot creation for the first time.

The capabilities exhibited by SnatchBot make it the only one of the examined tools that belong in the category of Registrational interactivity, as it can potentially understand the users and the intent behind their queries. Of course, this doesn’t mean that all products created with this platform will necessarily belong in this category, as it is up to the user as to how much they want to invest in the training of the NLP models, if at all, as SnatchBot also provides all the necessary tools to create a non-intelligent agent as well. As for its scores based on the Heeter model: Complexity of choice available: 5, Effort users must exert: 3, Responsiveness to the user: 4, Facilitation of interpersonal communication: 2, Ease of adding information: 4, Monitor system use: 2, for a total of 20.

These natural language processing capabilities open up a whole realm of possibilities for the journalist, and even though their full scope of applications exceeds the narrower focus of this paper, SnatchBot still presents a compelling alternative for those who require a more engaging user experience out of their bot. The ease of use does suffer slightly since the whole process of training an NLP model isn’t quite as straightforward as the simple visual programming of other platforms, but user responsiveness remains high because of the combination of NLP and the existence of a specific plug-in to pass the conversation onto a human. Multiple fields for information input exist–although there is no specific rating feature–and the monitoring statistics offered to free users are fairly limited, which is reflected in the final score of the above categories. In the past, integration with Chatbase–a popular chatbot analytics service provided by Google–used to be present, but unfortunately, this doesn’t mitigate the monitoring shortcomings of the platform, as this service was officially discontinued in 2021 after operating in maintenance mode for an extended period, leaving SnatchBot without any noteworthy monitoring capabilities to speak of.

Lastly, the options present in the third and final platform, Chatfuel, firmly place it in the realm of Conversational interactivity. This is due to its ability to interact with users both with predetermined buttons as well as free text, but without presenting options that can result in a truly generative dialogue between the bot and a human. Going into specifics, Chatfuel received the scores below: Complexity of choice available: 5, Effort users must exert: 4, Responsiveness to the user: 5, Facilitation of interpersonal communication: 2, Ease of adding information: 4, Monitor system use: 3, for a total of 23.

Complexity of choice and user responsiveness are the strongest suits of this platform. The sheer amount of options that Chatfuel’s pattern-matching solution offers will likely be enough to satisfy all but the most demanding use cases in the realm of journalistic work. The keyword system essentially makes the final product proportionally intelligent to the amount of work the user is willing to invest, making it versatile but not overwhelming to newcomers, thus the high score in the first two categories. Additionally, Chatfuel provides high-quality features in the “Responsiveness to the user” category, as it is the only platform capable of accommodating a live chat with the bot administrator, in addition to the typical conversation handover feature that can pass the text chat on to a human. Some very limited user-to-user communication features are present, but as expected from most chatbots, this isn’t the main feature of the platform. Variables can also be used to extract some user information (for example, a name to address them by) in the case that users engage with the bot via a registered account, like, for example, their Facebook profile. Finally, while there is analytics available, a significant number of features are inaccessible to free users, as they require a subscription, and so the “Monitor system use” category receives a middle-of-the-road score.

What can be surmised from the categorization based on Heeter’s [49] model is that, despite an abundance of features in most cases, the particular interactivity characteristic where chatbot creation platforms underperform is the facilitation of interpersonal communication. This is understandable to an extent, as, in most instances, the use-case for conversational agents dictates the need for back-and-forth communication between the chatbot and the audience–or in some cases, the audience and the creator of the program—but it doesn’t account for the user-to-user department. The market could attempt to adapt to the lack of features in this area, as user-to-user interaction via chatbots could facilitate the easier creation of online communities based around media organizations or even the enhancement of already existing communities through more direct and interactive connections among members.

Most of the examined platforms can satisfy the interactive needs of journalists in a variety of scenarios, as user responsiveness and ease of adding information remain high across all platforms. Another thing that could be improved, however, is the monitor system use category, as some inconsistencies can be observed over different platforms, with some of them offering extensive features and others only supplying the bare minimum for the bot administrator. This can be specifically observed in platforms that create a large separation between their free and premium plans, as analytics seems to be presented as one of the prime incentives for users to upgrade.

Overall, a decent spread of features seems to be present in the current market, as options exist even for more demanding users that require natural language processing (SnatchBot) as well as users looking for more basic–but still robust–interactions, with emphasis on information gathering (Quriobot). Chatfuel specifically represents an example of a platform that can potentially appeal to an even larger part of the media sphere because of its more generalist approach and the complexity of choices it offers. The existence and variety of these platforms, as well as their accessibility, suggest a future where news organizations can lean even more into their interactive side and create a news-sharing environment that will provide users with a level of back-and-forth communication previously unseen across the media industry.

4.2. Back-End Workshop and Evaluation Results

As described in Section 3.2.3, a two-part workshop was used as an evaluation method for each of the chatbot creation interfaces. This endeavor aimed to directly compare and contrast the usability of all the features present in each of the platforms and to examine how each one of them could realistically be utilized both by professional journalists as well as students in the field of media. A hands-on approach was adopted, where each participant was asked to create their own chatbot by the end of the procedure. The first part of the workshop, pertaining to the opinions of the students, took place during the first days of March 2022 and was approximately two hours long. The second part, which studied the perceptions of professional journalists, took place at the beginning of September 2022 and had a similar length.

For the purposes of this workshop procedure, it was important to procure participants that were all adjacent to the field of journalism but with varying degrees of familiarity with the profession. When it comes to the first part of this two-part workshop, the participants (n = 8, 4 male and 4 female) were between the ages of 18 and 30, with no prior experience or specialization in the field of ICT or chatbot creation. They were all students in the School of Journalism of the Aristotle University of Thessaloniki, enrolled in the course on Human-Computer Interaction, thus making them qualified to accurately judge the design process through a journalistic approach. As for the second part of the workshop, the participants (n = 7, 4 male and 3 female) were all professional journalists with working experience in the field but without any specialization in ICT or prior knowledge when it comes to chatbot creation. Their age group ranged from 31 to 60 years old.

After a detailed presentation of the creation process for each one of the platforms, the attendees were asked to allocate some time to create a sample chatbot for each one of the selected tools. Instructions were given for the specifications of the sample chatbot, but no extra help was provided to them to more accurately gauge their level of understanding for each of the creation interfaces. To ensure the validity of the results, the subjects were not allowed to communicate with each other during this stage of the study. After this process was concluded, participants were asked to fill out a questionnaire pertaining to their experience. The evaluation survey results are shown in Figure 5, and they are measured on a seven-point Likert scale where 1 represents a negative connotation, and 7 represents a positive one.

As can be seen from the above, the results of the workshop show that all three platforms were generally well-received. Quriobot, however, was consistently ranked higher than its peers in all categories. This lead ranges from small to relatively significant, and even though it spans all metrics, it seems to stand out, particularlyin terms of ease of use and straightforwardness. Specifically, Quriobot received an average rating of 5.91 (SD = 1) in the ease-of-use category, compared to the other two platforms, which ranged from 4.25 (SD = 1.3) for SnatchBot to 4.58 (SD = 1.5) for Chatfuel. Similarly, the process of correcting user mistakes was deemed to be noticeably easier on Quriobot with a mean score of 5.75 (SD = 0.8) as opposed to SnatchBot (M = 4, SD = 1.3) and Chatfuel (M = 4.33 SD = 1.6). Statistics such as these swayed user opinions into ranking the chatbot creation interface of Quriobot higher than the other two alternatives, as participants concluded that it is more suited for all levels of users (M = 4.75, SD = 1.2) compared to the alternatives (SnatchBot: M = 2.91, SD = 0.9; Chatfuel: M = 3.41, SD = 1.5).

When it comes to specific comments left by the participants during the discussion portion of the workshop, the declared opinions seem to substantiate the recorded results. During the first part of the workshop, Quriobot was deemed the most accessible platform overall, with one participant proclaiming that “it had the most friendly and self-explanatory environment”. Despite that, however, another participant did point out a visual bug they noticed in the interface of the program. Chatfuel had a similar, albeit less enthusiastic, reception, with attendees noting that it sported a “beautiful design” with “easy to connect flows” but also noticing that “it was slightly harder to initiate the process of chatbot creation on it”. Lastly, SnatchBot was the interface that received the most critical comments overall, with participants suggesting that “it seems only suitable for pro users” and saying that “it was difficult to understand my mistakes in this platform”. Despite that, however, it was still positively received overall, as can be seen by the aggregate results shown earlier, with two users pointing out that it presented them with the greatest variety of options, themes and topics.

When it comes to the second part of the workshop procedure, the professionals were mostly surprised by the simplicity and accessibility of modern approaches to chatbot design. All three platforms had a positive reception overall, with Quriobot standing out as slightly easier to use compared to the other two. One participant who had a disappointing experience trying to program a chatbot in the past and abandoned the endeavor very early now felt confident using the three presented tools. Another one declared that they had no experience, but they felt confident in engaging with the presented platforms regardless, without having any expertise. The same participant stated that even though they “appreciated the intuitive design of Quriobot”, they would like to explore the other two platforms more to make use of what they perceived as “their extended functionality”. Concerning the integration in journalistic workflows, one attendee found the possibility of constantly updating personalization parameters through a chatbot a particularly interesting use case, especially for smaller news organizations. Some concerns were also voiced by a workshop participant in regard to how the chatbot could operate with open-ended questions in a more conversational environment. Furthermore, the same professional noted that they would like to see evaluation results concerning their long-term use to see proof that parametrization and personalization will, in fact, enhance engagement over time. One concern that also arose during this second part of the workshop was that, according to a participant’s experience, modern users tend to mostly browse article titles and short descriptions, and they might not dedicate extra time in conversation with a chatbot for a deeper exploration of a news agenda.

Overall, besides the usability metrics, all three platforms were also perceived as useful for journalistic purposes, which is particularly relevant for the purposes of this study, as it adheres to the notion that these platforms can be utilized by the average journalist for their work. This notion extended to both university students, as well as professional journalists. Once again, Quriobot seems to be a standout in terms of perceived usefulness. This lead, however, is smaller compared to other categories and given the limited sample size of the workshops, as well as the possibility of utilizing each platform for different purposes in the real world, it is safe to assume that all three of the alternatives present a compelling case for being implemented into modern journalistic practice.

4.3. Front-End Evaluation Results

A total of 162 participants (49 male, 113 female) responded to the questionnaire, which was disseminated via the LimeSurvey (https://www.limesurvey.org) (accessed on 30 May 2022) platform during April and May of 2022. Ninety-two percent of them were in the age group of 18–24, 2% were between 25–40 and 6% were in the category of 40–60. Fifty-seven percent of them held a high school diploma, and 43% had a university or postgraduate degree. Seventy-seven percent declared that they have more than average experience with IT and communication technologies, and 81% of them have interacted in the past with a chatbot or another automated application, using a personal computer (48%), a mobile phone (44%) or other devices (2%). The results of the online questionnaires are shown in Figure 6.

Results show that users were, on average more eager to communicate with a chatbot created by Quriobot or Snatchbot, as they found the environment of Chatfuel unnecessarily complex and less easy to use compared to the other two. This increases the likelihood that a user might require assistance from a technical expert to use this tool, which seems rather inconvenient, given the examined use-case scenario. Participants believed that most people would quickly learn to use Quriobot and Snatchbot, while they felt that they would need to learn more things before using Chatfuel. Functions seemed to be overall better integrated into Quriobot (M = 3.83, SD = 0.80), as this platform was considered quite consistent, in contrast to Chatfuel (M = 3.62, SD = 0.79) and Snatchbot (M = 3.18, SD = 1.08) which exhibited slightly lower scores in this category. Overall, respondents felt more confident using Quriobot as an automated personalized newsagent.

It is worth mentioning that these results refer to users interacting with the chatbots for approximately 1–2 min each, so they represent a first contact scenario with the platforms. This means that results may not be indicative of the long-term user experience, but rather they more accurately represent the learning curve of each platform. In addition, despite the gradation of the results and the subsequent highlighting of the user’s preference for the Quriobot agent, all three platforms were considered easy to use, and participants generally felt confident using them after only small testing. This fact is particularly encouraging for the feasibility of using automated chatbots for personalized news reporting, as it suggests that journalists without a technical background, prior training, or any further instructions can immediately feel confident and engaged using them. This is part of the gap that this paper aims to fill in the related literature, and it proves to have a significant connotation when observed in conjunction with the evaluation results of the frontend of each application.

5. Conclusions and Further Work

The operational principles of conversational agents render them interactive pieces of software, almost regardless of the context they are being used. Those intrinsic characteristics make them an invaluable ally to the modern journalist, as they can be used to enhance particular aspects of their workflow that could benefit from systematic automation and interactivity, like, for example, news dissemination. To that end, this study focused on a multifaceted evaluation approach for existing chatbot creation platforms, bringing the average journalist to the forefront and examining the feasibility of integrating chatbot automation into day-to-day journalistic practices.

Chatbot creation nowadays is not as inaccessible as it once was, as even the average worker in the media industry can pick up and use one of the many available online tools that are based on visual scripting, and the market is saturated with a variety of different options to cover the needs of different users. As suggested by the workshop that was carried out during this study, the average journalist today is likely to be capable of creating a basic chatbot by utilizing the available online tools, with minimum to no guidance, even without previous experience. Furthermore, what the results of the study uncovered were that the proposed three-way evaluation approach can potentially be used to provide a more well-rounded view of existing tools that includes an examination of both the front-end as well as the back-end of an application, in addition to specific interactivity features that are considered essential for use in a media environment. This spherical examination can lead to the formation of a more balanced opinion when it comes to selected tools and platforms, one that can highlight potential inconsistencies between the performance of a specific tool when it comes to its creation process as opposed to the process of using it. For instance, out of the curated list of online tools chosen for this study, Quriobot was overall the most well-received one, both in terms of usability, as well as usefulness. This preference extends to both the back-end, as well as from the front-end of the application, as it was ranked the highest both during the two-part workshop, as well as the end-user survey. However, some inconsistencies can be observed between the scores of the other two platforms, depending on the evaluation scenario. For example, workshop attendees characterized SnatchBot’s interface as relatively hard to use, noting that it caters to more professional users. This was especially notable on behalf of the university students during the first part of the workshop and less so during the second part, which included professional journalists, although it was still prevalent. On the other hand, however, this was not true for the end-users, who evaluated the end product of SnatchBot as relatively easy to use. This dissonance highlights an important point for this study, which is the fact that when it comes to chatbot usage in a journalistic environment, all aspects of a potential tool should be examined separately, as usability and usefulness are both multifaceted terms that affect the journalist and the end-user in different ways, due to the unique nature of conversational agents.

Overall, all three platforms received relatively high scores for usefulness and usability, making them all suitable alternatives for aspiring chatbot creators in the media industry. Concerning the opinions of professional journalists specifically, the overall conclusion was that participants felt confident in using all three platforms for chatbot design. This is a noteworthy result, considering that before the short presentation of the three platforms, the attendees were either unaware or skeptical when it came to the easy integration of such tools in their work. Having said that, there are a few particular areas where improvements could still be made to the available tools in the market to ensure they cater to all types of interactivity and refine the already available features. As an example–stated in the corresponding section–some interactivity features, like the facilitation of interpersonal communication, are not yet integrated into many of the available platforms. Their incorporation could help improve those tools and saturate the market with more niche options for chatbot creation in the field of journalism and media. Furthermore, professionals raised a few concerns in regard to chatbot usage and how it relates to user engagement, which is an interesting point that could be addressed in future research over a real-world case study. Taking this into account, this work could also be elaborated on in the future with the inclusion of different evaluation techniques that focus on hyper-specific aspects of current journalistic practices and figuring out which available platforms better suit each task at hand. For example, a potential study could include several platforms, which would all be evaluated solely on the basis of how they perform when it comes to information gathering from audience members or other similarly distinct tasks.