Using the Engagement Profile to Design an Engaging Robotic Teaching Assistant for Students

Abstract

We report on an exploratory study conducted at a graduate school in Sweden with a humanoid robot, Baxter. First, we describe a list of potentially useful capabilities for a robot teaching assistant derived from brainstorming and interviews with faculty members, teachers, and students. These capabilities consist of reading educational materials out loud, greeting, alerting, allowing remote operation, providing clarifications, and moving to carry out physical tasks. Secondly, we present feedback on how the robot’s capabilities, demonstrated in part with the Wizard of Oz approach, were perceived, and iteratively adapted over the course of several lectures, using the Engagement Profile tool. Thirdly, we discuss observations regarding the capabilities and the development process. Our findings suggest that using a social robot as a teaching assistant is promising using the chosen capabilities and Engagement Profile tool. We find that enhancing the robot’s autonomous capabilities and further investigating the role of embodiment are some important topics to be considered in future work.

1. Introduction

University courses are designed based on requirements of students and teachers, to engage students and encourage them to learn actively. During the last century, a transition has taken place from behaviorist learning, featuring passive knowledge transfer and repetitions, to cognitive learning leveraging knowledge of how people process information, and constructivist learning considering subjective needs and backgrounds of students. Recently, online collaborative learning theory has been proposed, which facilitates social collaborations via transformational digital technologies [1]. Digital technologies have been described as not just an aide for teaching, but rather as something which has changed how students learn and our concept of learning [2]. For example, through “multi-inclusive” and multimodal designs, such technologies can facilitate different learning styles (e.g., being intelligible to both “serialists” and “holists”), engaging students via visual, auditory, or kinesthetic stimuli [3]. In particular, the promise of robotic technologies for engaging students is being increasingly recognized, with the result that we are now in the midst of a “robotics revolution” in education, in which robots are being used more and more in classrooms around the world targeting various age groups and disciplines [4].

One kind of robot which is increasingly being used in such applications as teaching where the social aspect is of the essence is what is being called a social robot; this refers to a (semi-)autonomous system with a physical embodiment that interacts and communicates with humans or other agents by following social behaviours and rules attached to its role [5]. As such, social robotics is part of the larger field of human-robot interaction (HRI) [6]. One instance of how a social robot could be used in teaching is that it could act as teaching assistant.

Some positive benefits of robot teaching assistants can be seen in relation to alternatives such as employing more human teachers or other digital technologies. With an increasing number of students seeking to receive university degrees, teachers can have little time to prepare due to other responsibilities, and it would be helpful to offload some of the teachers’ work. From the student perspective, opportunities for one-to-one interactions with the teacher in typical university classes are limited; robots can help with the lack of teachers.

Furthermore, robots have surpassed humans in many capabilities: e.g., sensing, memory, arithmetic, the ability to continuously work and concentrate without sleep or breaks, and the ability to communicate basically in many languages. Thus, robots could use their ability to collate student data to adapt the pace of learning, provide extra wait time for answers, and be patient if a task must be carried out many times. By this we do not suggest that human teachers should be replaced by robots; rather we imagine a situation in which both humans and robots can complement one another.

Several studies have provided evidence that robots show marked benefits over screen-based technologies in education, also at the university level, both in terms of learning outcomes and motivation [7,8,9] and the use of social robots in education [10]. Yet, there could also be disadvantages to using robots in a university classroom. In the same way that digital technologies such as slide presentations, have been described as “soporific or dazzling” [11], it could be damaging if robots were seen as a facetious distraction, or as an excuse for teachers to avoid having to deal with bothersome students. Moreover, socially interactive robotics is an emerging field which is not fully mature; imperfect capabilities could disappoint and disillusion rather than engaging, and yield results which might be different from more refined systems in the future. For robot technology to succeed in teaching, useful capabilities and potential pitfalls should be identified and considered.

The contribution of the current paper is reporting on some experiences designing and deploying a robot teaching assistant in an engineering course at the university level over a three week period; we designed the robot’s capabilities based on interviewing some teachers at a Swedish university, then adapted and analyzed the capabilities using the Engagement Profile tool. Considerations and future challenges are also described, toward informing next investigations.

2. Related Work

In a variety of studies, robots are the subject of learning [12,13,14,15,16], where they are used as a learning material. In such settings, students can experience engagement by conducting practice-based learning, assembling robots and using them to test their hypotheses. In other settings, robots can be used as tools for enabling remote attendance, typically for children as the target learning group. For example, the Pebbles robot was used to allow sick children to remotely attend classes [17]. Also, Telenoid, a remotely operated robot, was used for children’s groupwork; increased participation and pro-activeness were reported, along with the suggestion that communication restrictions imposed by using robots can actually facilitate collaboration [18]. Additionally, a tele-teaching approach was reported with the AV1 robot, where students can use an avatar to remotely attend classes and be present in the classroom; in this approach, the classic robot aspects are more in the background [19].

Pioneering work on companion robots was conducted by Kanda and colleagues, who deployed a humanoid robot in an elementary school [20,21]. Children were free to interact and play with the robot outside of class time during a thirty minute break after lunch. The robot was described to the children as only speaking English, which allowed controlling the complexity of the interactions and also motivated the students to use and learn English. As a result, the children learned new vocabulary with the robot present. Such robots have also been used as tutors to to stimulate scientific curiosity in engaged children [22], as well as to improve children’s performance, engagement and motivation in learning sign language [23].

One question is how such a tutor robot should interact during learning. One study reported that children seemed to learn better when a NAO robot acted more like a peer than a teacher [24]. However, in general, serious robots might be more effective in serious tasks than playful robots [25], and role assignment in educational HRI has been observed to be a complex and dynamic process which can be difficult to control [26]. Additionally, although social behaviors can be effective for engaging students, they can also in some cases distract children, and hence should be incorporated into interactions with care [27]. Furthermore, positive effects have been observed from personalizing the behavior of two autonomous robots, which acted as learning companions for children [28].

In other studies, robots acted as teaching assistants in the classroom. A RoboSapiens was used in an elementary school for five weeks, which read stories using different voices, led recitals, provided friendly feedback, gave quizzes, and moved when students asked [29]. Similarly, a NAO robot was used with some children to read out vocabulary words with pictures shown on slides behind it, and pantomime the meanings, while also providing entertainment such as singing and dancing [30]. As a result, it was observed that students learned faster and more words, compared to a control group. Additionally, a NAO robot was used in two studies with children with autism spectrum disorder and with children that were second language learners, observing increased participation and involvement in language learning [31].

Thus, various robotics studies have been conducted with children. Further details can be found in a number of reviews which have been made available (e.g., regarding the use of social robotics for early language learning [32]), and we also provide some examples of robots in education in

Table 1. Some examples of previous work on robot agents in a teaching context, classified by the role of the robot. Benefits refers to the advantages of using a robot, as compared to a human.

Reference	Benefits	Robot	Capabilities	Class	Outcomes
Role: Tutor outside class:
[20,22,33]	Prevent bullying, provide friendship	Robovie	Recognize children, quizzes, gaze, entertaining	Elementary school language and science classes in Japan, 1–2 months	Better retention, some increased curiosity
[34]	-	iCat	Read, gaze, feedback w/facial expressions, nod, shake head, idling	16 10–11 year old elementary school children in Holland, 1 h	Social behaviors facilitated learning
Role: Avatar:
[19]	Tele-teaching, avoid loneliness	AV1	Remote operation, avatar	9 12–16 year old adolescents at school in Norway	Users provided positive feedback
[18]	Change social interaction, expand human capabilities	Telenoid	Convey operator’s voice, arms move	28 9–10 year old elementary school children in Japan, 2 days	Limitations of robot had positive effects on collaboration
Role: Teaching Assistant:
[29]	Repeatability, digitization, fantastic appearance, different voices	RoboSapiens	Read (feedback), remote control	Elementary school for five weeks, in Taiwan	Students were motivated, and suggestions for improvement were made
[35]	Repeatability, AI, sensors	NAO	Read words, pantomime, entertainment (sing, dance).	12 year old students in Iran	Students learned faster and more words, compared to control
[31]	Second language learning tool	NAO	Listen, repeat, feedback	pre-school children in Norway; children with autism (ASD)	Increased participation and involvement

.

Fewer studies have been conducted with adults. An idea was reported that science students with disabilities could use robots to conduct experiments remotely [36]. Furthermore, a study carried out with university students found that the physical presence of a robot tutor facilitated learning outside the classroom [8]. Outside of robotics, an artificial intelligence was used for an online course to answer frequently asked questions, which worked so convincingly that students reportedly did not know they were interacting with a non-human teaching assistant [37,38].

Why have studies on robot agents up until now focused predominantly on children? It has been pointed out that children can have problems with making friends and bullying [33]. Also it has been stated that “in general younger children are more enthusiastic about robots” [39], citing a work which dealt with elementary to high school students but not adults [34]. Furthermore, this latter work stated: “It is also noteworthy that the research interest in HRI in classrooms has been largely skewed towards elementary school settings.” We speculate that there could also be a feeling that robots can be enjoyable due to their novelty, and therefore more applicable to the domain of children—whereas the adult world has sometimes been perceived in the past as a place for work and seriousness rather than emotions like enjoyment, although this perception is changing [40].

It is not yet clear if the results of studies with children will directly apply also to adults, e.g., university students. In general, adults typically have rich specialized knowledge and experience, especially at the master’s level, and are more self-directed and needs-driven than children [41]. As well, there can be large differences in the degrees of knowledge adults possess, which can cause stress for teachers [42]. Robots can be programmed with a wide range of encyclopedic knowledge, which could be useful to overcome knowledge differences. Moreover, language studies have shown that robots are more successful when the students had some ability and interest [7]. Additionally, in adult classrooms, where the students have spent years gaining expertise, face-saving becomes more important, as being judged by another adult can be humiliating [41]; making mistakes in front of a robot could be less embarrassing than in front of a human teacher.

In regard to self-direction, it could be easier for a robot to keep adults’ attention. Some children have been reported as not listening to a robot’s quiz and covering its eyes with their hands [22]. Various abusive behaviour including kicking and punching by young people toward robots has also been reported [43,44]. By contrast, adults at universities have more freedom to select what they will study, picking majors and courses. However, adults can also experience various needs, responsibilities, and worries which are not typical for children—from financial factors such as part-time work, loans, and mortages, to caring for dependents, and age-related health problems—which can lead to mental fatigue. In such cases, the communicative power of robots using visual, aural, and, possibly, haptic modalities, could facilitate learning.

Thus, previous work suggested that robots could also be useful in the context of adult learning, but it was not clear to us what kinds of robot capabilities would be desirable or how interactions could be structured to be engaging.

3. Materials and Methods

To investigate how a robotic teaching assistant can be used at the university level, we performed an experiment at the Department of Intelligent Systems and Digital Design (ISDD) at Halmstad University in Sweden, as described below. After gathering some requirements from the engineering teachers, we selected a course and a robot to facilitate lectures as a teaching assistant. We used an iterative approach, starting with an initial design of the robotic teaching assistant, that was analyzed using the Engagement Profile tool, and this design was updated during the course of three weeks.

3.1. Robot Teaching Assistant Capabilities

A wide range of tasks can potentially be performed by teaching assistants, including tutoring support, grading assignments and tests (also invigilating), assisting students with special needs, replying to emails, and questions during office hours. To select capabilities which might be useful to be incorporated into a teaching assistant robot, we conducted a brainstorming session during a regular weekly meeting of the teachers at the ISDD. We asked for any comments the teachers might have about where a robot could be helpful, especially regarding problems the teachers had faced in the classroom before. As a result, we identified challenges regarding, e.g., fatigue (specifically of the voice), absences, ambiguity, distraction, and physical chores. From this list, we suggested six capabilities of interest for the robotic teaching assistant: C₁—reading, C₂—greeting, C₃—alerting, C₄—remote operation, C₅—clarification, and C₆—motion. We describe these capabilities in

Table 2. List of capabilities for the robot teaching assistant.

	Capability	Description
$C_1$	Reading	to orally present material such as quizzes.
$C_2$	Greeting	to greet the students.
$C_3$	Alerting	to alert the teacher.
$C_4$	Remote operation	to facilitate communication with persons at remote places.
$C_5$	Clarification	to present extra material on request.
$C_6$	Motion	to perform physical tasks by means of locomotion and object manipulation.

and elaborate in more detail below.

Speech is an important communication modality in classroom teaching, but excessive speaking can be tiring for lecturers, who typically have a heavy workload, and monotonous for students. As examples, our staff mentioned that lectures typically last two hours and oral exams can last two days for large classes. To address this challenge, a robotic assistant can orally present material and moderate quizzes ($C_1$). Further, a robotic assistant can be used to greet the students ($C_2$).

Students and teachers can miss classes due to various reasons, such as illness and traveling. In such cases, video conferencing is an option, but can require people who are present to spend time setting up computers (e.g., microphones, speakers, and angles of viewing). Robotic teaching assistants can be used to overcome this challenge by using remote operation ($C_4$).

Teachers try to scaffold students’ understandings while together tackling appropriately challenging material, but for various reasons, e.g., because prior knowledge typically varies by student, additional help can occasionally be desirable. An example given at the brainstorming meeting at the ISDD was the challenge in visualizing data when teaching topics such as machine learning. This challenge can be addressed by clarification ($C_5$) and reading ($C_1$); i.e., automatically supporting the teacher by looking up and showing topics on a robot’s display while the teacher talks.

It can also be hard for a teacher to divide their attention during class between multiple factors, such as lecture content, timing, and students, which can lead to errors [45]. Examples included making coding mistakes that students did not point out, blocking part of the view of a presentation, forgetting students’ names and backgrounds, and not immediately seeing a student with their hand up. This challenge can be addressed by an alerting functionality ($C_3$), where the robotic assistant alerts the teacher when needed. Further, the greeting functionality ($C_2$) can be useful, in case the teacher forgets the names of students.

Also, the class’s time can be reduced and effort can be required to complete common tasks, such as handing out materials, closing and opening doors and windows, and lowering projection screens. Using motion capability ($C_6$), the robotic assistant can provide locomotion and object manipulation to conduct such physical tasks.

Given the exploratory nature of our study, we decided to focus most on capability $C_1$ (reading) and especially on quizzes, which was described as a useful low-hanging fruit and could engage students by encouraging active learning. In our work, we did not further consider additional suggestions that were not clearly related to a problem; these included incorporating playfulness, personalizing interactions by calling students by name or speaking their native languages, and using a range of different voices and dialects.

To realize these capabilities, a mid-fidelity prototyping approach was followed, to balance obtaining accurate insight into how a completed robot would be perceived, with allowing observations to be collected quickly and practically [46]. In line with grounded theory and an intention to explore through observations, we also drew insight from an idea from previous work that any mistakes made by a robot can have positive effects in helping students to feel less self-conscious about performing perfectly [35].

3.2. The DEIS Course

As a testbed for designing our robotic teaching assistant, we selected a course called the Design of Embedded and Intelligent Systems (DEIS), which is a double-credit compulsory course for second year master’s degree students in the Embedded and Intelligent Systems Programme at Halmstad University. The course aims to improve both the breadth and depth of the students’ conceptual and practical knowledge in a collaborative, creative, and critical manner. Students attend lectures and labs which are supervised by eight teachers, while also working independently on a problem-solving project in small groups. Lectures cover a wide range of topics, including statistical inference, robotics, sensor fusion, embedded programming, motion planning, simulation, communication, and image processing; the project involves developing platooning capabilities for some small wheeled robots. Learning is evaluated via a de-contextualized oral exam and contextualized written report (50% each). The oral exam is in “ordered outcome format”, in alignment with the structure of observed learning outcomes (SOLO) taxonomy of knowledge, in which students receive questions requiring uni-structural, relational, and creative responses [47].

We assumed that this course would be appropriate due to its contents, as it seemed intuitive to use a robot to teach a course about robots; the students study robotic components during the course, and can enrich their knowledge by seeing these components function together in a working robot. As the course is difficult to teach, a need for teaching assistance occurred; a high level of student engagement is required because students, coming from many different backgrounds such as data mining, intelligent vehicles, electronics, and communications, are expected to gain knowledge that is both wide-ranging and deep.

3.3. The Baxter Robot

Several robots were available for use at the ISDD, including NAO robots [48] and Turtlebot [49]. A Baxter robot on a Ridgeback mobile base (shown in Figure 1 and hereafter referred to as Baxter), was selected due to its versatile interactive capabilities and engaging appearance. To interact with people, Baxter has various actuators: two seven degrees-of-freedom arms capable of moving objects up to 2 kg, an omni-directional mobile base enabling movement within a classroom, speakers, and a display. Baxter also has a number of sensors, including a microphone, cameras in its head and wrists, and force sensors in its arms. 13 sonar sensors situated in a ring around its head, IR range sensors in its wrists, and a laser and inertial measurement unit in its base were available but not used in the current study. Baxter’s height (180 cm) was also considered to be a potential advantage as height plays a key role in how attractive, persuasive, and dominant a robot is perceived to be [50], and such qualities are linked with engagement [51].

Visual and aural recognition was conducted using the open source computer vision library (OpenCV) [52] and the speech recognition toolkit CMU PocketSphinx [53]. Robot behaviors were triggered and robot states changed by a teacher by pressing buttons on a graphical user interface running on a desktop, using robot operating system (ROS) [54] for inter-machine communication. A face to show on the robot’s screen was designed by people from the communication department of Halmstad University, from which we constructed variations to convey various emotions and states. Gestures were recorded and played back using Baxter’s software development library for Python.

3.4. The Engagement Profile

For the evaluation of the capabilities and the design of the robotic teaching assistant, we desired some way to assess the degree to which our robot engaged students. The Glossary of Education Reform [55] refers to engagement as follows: “In education, student engagement refers to the degree of attention, curiosity, interest, optimism, and passion that students show when they are learning or being taught, ..., and the concept of `student engagement’ is predicated on the belief that learning improves when students are inquisitive, interested, or inspired, and that learning tends to suffer when students are bored, dispassionate, disaffected, or otherwise ‘disengaged.’” According to this glossary, forms of engagement include (a) intellectual, (b) emotional, (c) behavioral, (d) physical, (e) social, and (f)cultural engagement.

In our work, we use the Engagement Profile, which was originally developed to assess engagement factors for exhibits in science centres and museums [56]. These are considered as informal teaching arenas. We posit that the Engagement Profile can be applied to a setting where the robotic teaching assistant is used in a formal learning environment. Similar to installations in science centres and museums, the robotic teaching assistant represents an artifact that the students interact with during their studies and classes. Thus, we can assume that increased engagement by the students will contribute to inspire and facilitate learning, as well as increasing the learning outcome.

The Engagement Profile [57] quantifies the characteristics of installations along eight dimensions, each of which is given a value between 0 and 5. The dimensions of the Engagement Profile represent the degrees of competition (C), narrative elements (N), interaction (I), physical activity (P), user control (U), social aspects (S), achievements awareness (A), and exploration possibilities (E). External influences are not taken into account in the Engagement Profile since these are not properties of the direct learning environment. Physical factors, such as noise, light or smell could play a role in the perception of engagement, but need to be handled outside the Engagement Profile. Properties that belong to the context, such as social factors, institutional factors, or recent incidents personally or globally, are excluded. However, these factors still need to be taken into account in the assessment process, e.g., as suggested for a different setting [58].

To adapt the Engagement Profile to a more formal learning environment with a robot teaching assistant, we replaced references to the original domain (i.e., installations in museums and science centres) by terms that are related to the use of a robot teaching assistant. The short form of the adapted version of the Engagement Profile is shown in Figure 2.

4. Study

The Baxter robot was used in four classes over three weeks in Autumn 2017 conducted by the course responsible. In the first week there were two classes on Thursday and Friday, and in subsequent weeks only on Thursdays. The classes were conducted from 10:15 to 12:00. The classroom was kept constant with a layout that is shown in Figure 3. The room was well-lit, and there was little noise from the outside. The study was conducted with 24 students (average age 26.8 years, SD = 4.7; 8 females, 16 males; from approximately ten different countries, with a majority from Asia).

4.1. Setup of the Study

The study followed the iterative design process described for a different application area [56]. Out of the description of the capabilities of interest and the context, we identified the ranges of preferred and suitable values of the Engagement Profile, as shown in

Table 3. Influence of the robot capabilities on the preferred ranges in the Engagement Profile. Values outside the suitable range are counter-indicative to the intentions of the teaching robot.

Capability	Preferred Range	Suitable Range
$C_1$, Reading	$N\ge 1$	$N\ge 1$
$C_1$, Quiz	$C=2-4;A=1-4$	$C=1-5;A=0-5$
$C_2$, Greeting	$I\ge 2$	$I\ge 2$
$C_4$, Remote operation	$S\ge 2$	$S\ge 2$
$C_5$, Clarification	$N\ge 2$	$N\ge 2$
$C_3$, Alerting	$N\ge 2;A\ge 1$	$N\ge 2;A\ge 1$
$C_6$, Motion	-	-
classroom setting	$P=1-3$	$P=0-3$
one robot in front	$S=3,4$	$S\ne 2$
use of robot in general	$U=1-3;E=0-4$	$U=0-5;E=0-5$

. We also identified the influence of the classroom setting to the values of the Engagement Profile. The Engagement Profile of the robotic teaching assistant at the start of the experiments is explained in

Table 4. The Engagement Profile of the robotic teaching assistant at the start of the experiments.

C:	2	Competition with robot. The students will discuss and respond to quiz questions in pairs in front of the class. Some students might implicitly perceive themselves to be competing with others to some extent, but in general the students will work together as a class to answer the robot’s questions.
N:	2	Limited narrative structure. The robot follows a simple storyline: introducing itself, why it is participating and what it should do, conducting its task, and saying goodbye.
I:	2	Limited interactivity. The students will respond to the robot’s quizzes, but the responses will not change how the interaction proceeds.
P:	0	Look only. The students will get the chance to also pilot the robot via a controller if they wish during the break, and they will maybe also receive handouts from the robot, but in general the students will mostly look only.
U:	1	Linear chronology. The robot will give quizzes in a predefined sequence during the lecture. The users can affect how many quizzes are given by the time they take to answer (lectures can last only two hours, so if time runs out, quizzes can be given at a later date).
S:	3	One student, others cheer and engage. The robot will conduct social behaviors aimed at the group, greeting and quizzing.
A:	1	Immediate feedback. Answers to quizzes will be given in general very soon after students respond. We do not plan to give scores currently, to avoid having some students worry about losing face, although scores could be a fun way to motivate some students.
E:	0	Defined view. The students will investigate topics through a standard lecture view, and also from an applied view in participating in quizzes, but both perspectives are predefined and the robot will only be involved with the applied/quiz perspective.

and visualized in Figure 4. In this diagram, the green area shows the preferred values for our teaching setting, while the blue hatches show the values for the implementation of the first lecture.

Each week during the experiment the students were asked to answer a questionnaire by sending them a URL for the respective questionnaire. The URL was sent after lectures 2, 3, and 4, respectively. The questionnaires contained the questions given in s

Table 5. The questions for the student opinion using the following scale: $-2$ (much less), $-1$ (less), 0 (as now), 1 (more), 2 (much more).

Question	Formulation
$Q_C$	Should there be more or less competition between groups and participants in the learning experience?
$Q_N$	Should the storyline and roles in the learning experience be more evident or less evident in the learning experience?
$Q_I$	Should there be more or less feedback on the choices you did in the learning experience?
$Q_P$	Should there be more or less physical activity in the learning experience?
$Q_U$	Should the learning experience be more or less influenced by what you did during the experience?
$Q_S$	Should more or less be done in a group (as opposed to individually) during the learning experience ?
$Q_A$	Should there be more or less feedback on how well you are doing during the learning experience?
$Q_E$	Should there be more or less possibilities to go in depth with extra content to explore on your own?

,

Table 6. Formulation of the additional questions using the scale 1 (disagree) to 5 (agree).

Question	Formulation
$Q_1$	I liked the learning experience.
$Q_2$	The learning experience was engaging.
$Q_3$	I learned much during the learning experience.
$Q_4$	I recommend the learning experience to other students.
$Q_5$	I would like to have this type of learning experience for future course content.

and

Table 7. Formulation of the questions about robot capabilities using the scale 1 (disagree) to 5 (agree).

Question	Formulation
$C_1$	The ability to read material (e.g., giving quizzes) will be helpful for a robot teaching assistant.
$C_2$	The ability to greet people (e.g., saying hello and goodbye at the start and end of a class) will be helpful for a robot teaching assistant.
$C_3$	The ability to alert the teacher (e.g., if the teacher has forgotten to mention something or an explanation is unclear) will be helpful for a robot teaching assistant.
$C_4$	The ability to be remotely controlled (e.g., for people who cannot attend class due to illness or travel) will be helpful for a robot teaching assistant.
$C_5$	The ability to provide additional information (e.g., visualizing data, or adding information about topics which the teacher or students are discussing) will be helpful for a robot teaching assistant.
$C_6$	The ability to interact physically with people (e.g., fetching and handing out materials, handshakes) will be helpful for a robot teaching assistant.

, two questions about gender and age group, as well as a field for a free form comment. The questionnaires were identical each week. They were implemented using Google Forms [59] with a new form each week.

The students were involved in the assessment by answering a standardized questionnaire with eight questions about each of the dimensions of the Engagement Profile. This questionnaire is given in Table 5. Further, we asked five more questions about their satisfaction with the learning experience, as shown in Table 6. To be better informed about which capabilities the students prefer, we asked a further six questions that are shown in Table 7.

4.2. Description of the Experiment

In the iterative design of our experiment, the experience design was changed each week in two ways: (1) to address changes desired by students, and (2) to test new possibilities and content for each robot capability. This led to a chain of designing, implementing, observing, and obtaining feedback to adjust the requirements for the next iteration, as is summarized in

Table 8. Overview of capabilities and feedback.

		Day 1	Day 2	Day 3	Day 4
		Week 1		Week 2	Week 3
Design		lecture with six behaviours	sound increased	voting via waving, links	explore sessions
Implementation
	Reading	basic	basic	on request	for small groups
	Greeting	outline/roles	basic	basic	outline/roles
	Remote operation	locomotion	recording	video conference	gaze
	Clarification	teacher	students	extra material	teacher and students
	Alerting	omission	clarification	event	switch topic
	Motion	sheet	handshake	handshake	robot kits
Feedback		good, more volume	more exploration, user control, physical, social interaction	more narrative, exploration, control, awareness	more narrative, social interaction, awareness, exploration

. In the following diary, we describe the actions and observations we made; some examples of the teaching assistant robot interacting with the class are also shown in Figure 5.

4.2.1. Day 1

The initial design was based on the outcome of a brainstorming session with the teachers of the ISDD, as described in Section 3.1, which highlighted the six potentially useful capabilities $C_1$–$C_6$. Except for the robot, the basic structure of the course held in the previous year (lecture format) and contents were retained.

C₁

Reading: quiz content was split between the lecture slides and the robot, based on the assumption that the slides would be better for clearly communicating some complex information such as equations, while the robot would be more interesting in general for communicating simpler content such as quiz questions. Six slides in the lecture presentation were allocated for quizzes. On reaching a quiz slide, the teacher pressed a button on the GUI to trigger the robot to ask questions and show a puzzled face on its display. Quiz topics included computational logic and time complexity. For example, one quiz slide showed a deterministic and non-deterministic state machine and some strings; the robot asked the students to consider which strings would be accepted by each.

C₂

Greeting: the robot was set up to introduce itself at the beginning of class, stating its name, describing its role as teaching assistant, and priming students’ expectations that it was a work in progress, while waving a hand and smiling; at the end of class it said thank you and goodbye, again waving.

C₃

Alerting: the robot looked toward the teacher and stated that the teacher had forgotten to explain a topic.

C₄

Remote operation: students were invited during a break to teleoperate the robot using a handheld controller.

C₅

Clarification: the robot automatically showed some example images in its display based on recognizing keywords spoken by the teacher: specifically, the names of some common charts, such as ’Venn diagram’, ’histogram’, ’pie chart’, and ’Gantt chart’.

C₆

Motion: the robot took an attendance sheet from the teacher in its gripper and moved forward to hand it to the nearest student.

After the lecture, the students were asked to anonymously describe their experience. Over half the class described the initial experience with the robot in class as positive (15 people), using the adjectives good (5), awesome (4), cool (2), and fun (2); one third thought it was engaging (8), describing the experience as interesting (5), exciting (3), and motivating (2). Five people had various neutral questions about the robot, and nine people voiced suggestions for improvement, six of which were to improve the sound.

4.2.2. Day 2

Based on the feedback from day 1, which was mostly positive, the system was kept the same, just increasing the volume of the robot’s speech. Further, we tried to enhance the experience as follows:

C₁

Reading: six quizzes were conducted, in regard to circuits, connectors, computers, math, and programming languages.

C₂

Greeting: the robot greeted the class at the beginning and said goodbye at the end of the lecture.

C₃

Alerting: the robot advised the teacher at one point that a description was not clear.

C₄

Remote operation: the students listened to a former master’s student describe her experience by speaking remotely through the robot.

C₅

Clarification: the robot recognized keywords which the students said and displayed related images on its display.

C₆

Motion: The robot shook hands with students who wished to do so during a break.

At the end of day 2, the students were asked to answer the questionnaire with the questions shown in s Table 5, Table 6 and Table 7. The students were given the time until the next lecture to respond to the questionnaire. Nine students answered this questionnaire. The analysis of this questionnaire indicates for the dimensions of the Engagement Profile that the students desired more exploration (E), user control (U), physical activity (P), and social interaction (S), as can be seen in Figure 6 (w1).

For the questions $Q_1$–$Q_5$, all were on the positive side. Specifically $Q_5$ and $Q_1$ indicated that the students would like to repeat the experience, and that they liked it. However, the students were not so convinced about the learning effect of the experience. Regarding the capabilities, remote operation and extra content scored highest. The detailed results are shown in the section for week 1 of

Table 9. Results from questionnaires, week 1 for $Q_1$ to $Q_5$ and $C_1$ to $C_6$.

	${\mathit{Q}}_1$	${\mathit{Q}}_2$	${\mathit{Q}}_3$	${\mathit{Q}}_4$	${\mathit{Q}}_5$	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$
	Like	Engage	Learn	Recmd.	Again	Read	Greet	alert	Remote	Content	Interact
week 1 $(n=9)$
mean	3.8	3.4	3.4	3.7	4.1	3.7	3.7	4.1	4.0	4.0	4.0
median	4.0	4.0	3.0	4.0	4.0	3.0	3.0	5.0	4.0	4.0	4.0
90 %	5.0	4.2	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0
10 %	2.0	2.0	2.0	2.0	3.0	2.8	2.8	2.8	2.8	2.8	2.8
variance	1.4	1.0	1.3	1.5	0.6	1.3	1.3	1.4	1.0	1.0	1.3
positive	67%	56%	44%	56%	78%	44%	44%	67%	78%	78%	67%
neutral	11%	22%	33%	22%	22%	44%	44%	22%	11%	11%	22%
negative	22%	22%	22%	22%	0%	11%	11%	11%	11%	11%	11%
week 2 $(n=5)$
mean	4.6	4.4	4.2	4.6	4.6	4.2	4.4	4.4	4.2	4.6	4.2
median	5.0	4.0	4.0	5.0	5.0	4.0	4.0	5.0	5.0	5.0	4.0
90%	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0	4.6
10%	4.0	4.0	3.4	4.0	4.0	3.4	4.0	3.2	2.8	4.0	4.0
variance	0.3	0.3	0.7	0.3	0.3	0.7	0.3	1.8	1.7	0.3	0.2
positive	100%	100%	80%	100%	100%	80%	100%	80%	80%	100%	100%
neutral	0%	0%	20%	0%	0%	20%	0%	0%	0%	0%	0%
negative	0%	0%	0%	0%	0%	0%	0%	20%	20%	0%	0%
week 3 $(n=12)$
mean	4.4	4.2	4.0	4.3	4.3	4.0	3.9	4.4	4.3	4.3	4.1
median	4.5	4.0	4.0	4.0	4.0	4.0	4.0	5.0	4.0	4.5	4.0
90%	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0	5.0
10%	4.0	4.0	3.0	3.1	3.1	3.0	3.0	3.1	3.1	3.1	3.0
variance	0.5	0.4	0.6	0.6	0.6	0.6	1.4	0.6	0.6	0.6	0.6
positive	92%	92%	75%	83%	83%	75%	75%	83%	83%	83%	75%
neutral	8%	8%	25%	17%	17%	25%	17%	17%	17%	17%	25%
negative	0%	0%	0%	0%	0%	0%	8%	0%	0%	0%	0%

.

Based on these results, we applied the following changes for day 3: (a) To increase exploration, the robot suggested some sources for extra learning. (b) To increase exploration and user control, the robot gave students the choice to hear more about some topics, or take additional quizzes. (c) To increase the physical and social interaction, the students were asked to wave their hands to indicate interest and to come see the robot during the break.

4.2.3. Day 3

Based on the feedback and evaluation from the previous day, the setup of the robot for day 3 was as follows:

C₁

Reading: math, pattern recognition, statistics

C₂

Greetings: hello, bye (2).

C₃

Alerting: the robot alerted the class that it was time to go for a short outing to a workshop with tools.

C₄

Remote operation: video conference with remote person (the second author in Oslo, Norway).

C₅

Clarification: the robot described some extra resources.

C₆

Motion: in break, handshakes and face recognition.

At the end of day 3, the students were asked to answer the questionnaire with the questions shown in s Table 5, Table 6 and Table 7. The students were given the time until the next lecture to respond to the questionnaire. Only five students answered this questionnaire. The analysis of this questionnaire indicates the dimensions of Engagement Profile that the students desired; more narrative (N), user control (U), visible achievements (A), and the possibility for exploration (E), as can be seen in Figure 6, (w2). We interpreted this that a) the storyline and roles should be more evident, b) there should be more possibilities to go in depth with extra content to explore on your own, c) the interaction should be more influenced by what the students did; and d) there should be more feedback on how well the students are doing.

For the questions $Q_1$–$Q_5$, as well as for the capabilities $C_1$ to $C_6$, we abstain from comments, as the number of responses is too low. See the section for week 2 of Table 9 for details.

4.2.4. Day 4

Based on the feedback and evaluation from the previous week, we added several new elements: (a) The robot presented an outline of the “storyline” for that day’s class (what activities would be conducted and why) and clarified roles. (b) For each main topic of this lecture (robotics and computer vision) the robot gave the students some free time to study and take quizzes; the robot reminded when it was time to move on. (c) Baxter was set up to recognize faces and provide feedback for specific students at the end of the lecture.

The following functionality was implemented:

C₁

Reading: sensors, actuators, computer vision, summary (6).

C₂

Greeting: hello (storyline, roles), bye (2).

C₃

Alerting: time to change topics or switch between listening to lectures and exploring.

C₄

Remote operation: make the robot’s gaze follow people moving left to right or vice versa.

C₅

Clarification: showing quiz questions and answers on the robot’s display.

C₆

Motion: the robot handed out robot kits to a representative from each project group.

For day 4, additional functionality was implemented: facial recognition for personalisation, and a looking-around feature to show awareness. Face recognition was implemented using OpenCV [52] and a support vector machine (SVM) classifier with local binary pattern features trained on data collected from the students. The robot recognized faces using its head camera and displayed the faces in its display.

For the look-around feature, the intention was for the robot to look toward the part of the classroom which was most active to show awareness. For example, if a student waved her or his hand, the robot could demonstrate awareness by looking toward them. Background subtraction was used to extract motion from the students. In other words, moving an arm in front of the robot’s camera with a non-skin-colored background resulted in a difference between the colors of pixels in image frames over time which can be quantified. Images from the robot’s head camera were split into two regions, left and right. The robot was given two states, “head left” and “head right”, which entailed looking toward students on the left or right of the classroom. The regions were defined with overlap (hysteresis) based on the robot’s state to avoid jitter between states due to motion in the middle of the image. Further, the robot was made to wait a short time after moving to seek to avoid reacting based on its own motion.

At the end of day 4, the students were asked to answer the questionnaire with the questions shown in s Table 5, Table 6 and Table 7. The students were given the time until the next lecture to respond to the questionnaire. Twelve students answered this questionnaire. The analysis of this questionnaire indicates for the dimensions of the Engagement Profile that the students desired more narrative (N), social activity (S), visible achievements (A), and the possibility for exploration (E), as can be seen in Figure 6, (w3).

For the questions $Q_1$–$Q_5$ all were on the positive side. Specifically $Q_1$ and $Q_2$ indicated that the students liked the experience and found it engaging. Still, the learning effect scored lowest here. Regarding the capabilities, the alert functionality, remote operation, and extra content scored highest. The detailed results are shown in the section for week 3 of Table 9.

4.3. Observations

Our experiment design addressed a complex scenario and was performed in an exploratory way. While the experiments were ongoing, unexpected events happened that influenced the further path of our experiments. One of the authors, therefore, took on a role as an observer. A diary of events is presented below, and some examples are shown in Figure 7.

Day 1

The robot’s voice which had seemed sufficiently loud during development was not perceived as loud enough by the students. Also, when handing an attendance sheet to a student, the student reached out their hand to grab the sheet but when the robot did not immediately let go, the student retracted their hand and the sheet fell to the ground.

Day 2

Speech recognition was difficult because students sometimes did not speak loudly despite being requested to do so, possibly out of shyness, and the teacher often had to repeat the students’ words in order for the robot to react.

Day 3

After class some students described a difficulty with waving to vote for or against exploring material, due to feeling reluctant to oppose the wishes of their fellow students. One suggestion was to use an online poll for anonymity, and to assign tasks for a longer period of time during which students could be free to move around, to also allow for physical activity.

The robot’s scheduled reminder was not used because a student reminded the teacher before the robot could. The teacher had said that the class would leave for their outing at 11:35, but a student reminded the class at 11:30. Thereafter, the students started to pack their belongings, and because of the commotion, the last comments of the robot were also not effective; the teacher had to call for the students to give their attention while the robot was speaking. Handshakes with face recognition were also not demonstrated due to insufficient time, as more time than expected was required by the students to solve the math problems given by the robot.

Day 4

Due to a scheduling mistake another class was held directly before the DEIS course class, and there was no time to prepare the robot; therefore face recognition could not be shown during class time and was demonstrated afterwards. Quizzes appeared to work well, with some students gathering around the robot during each time slot allocated for exploration. The downside was that small groups could interact but not the whole class. Also handing out materials was slow, as the robot’s motions were not fast out of safety concerns. The robot’s phrasing when reminding of the time elicited some laughter. When it said “It is time”, a student said, “Time for what?” and laughed.

Afterwards

The robot’s power adaptors were stolen, luckily after the last day. The classroom was shared with other classes, and although the classroom was to be locked outside of class time, some hurry or mishap resulted in the door being left unlocked.

Another interesting find was that the robot’s power cables were consistently removed from the wall whenever we tried to let the robot charge overnight. Moreover, someone opened a panel on the robot and disconnected power coming from the robot’s main battery to an inverter. We do not know if this was a security guard or teacher, but such problems could affect the use of robots in venues which are shared by many different users.

5. Discussion

In our research design, we explored possibilities for using a robot as a teaching assistant by using the Engagement Profile to vary six distinct capabilities, that were measured against user feedback regarding satisfaction and some engagement factors. The Engagement Profile and such methodology has been applied earlier in connection with evaluating exhibits in science centres and museums. Using this design methodology, one performs several iterations where the design of the robot teaching assistant is altered by changing its capabilities, followed by an evaluation step that gives evidence how to make further changes.

In our research, we used the Engagement Profile as an evaluation platform. For this purpose, we adjusted the Engagement Profile to the use case of robots in a teaching context. The transition from installations in science centres and museums, for which the Engagement Profile originally was designed, to robots in a teaching context seemed straightforward. Further, the categories in the Engagement Profile, i.e., competition, narrative, interactivity, physical user control, social, achievements, and exploration, appeared to be suitable for the analysis. Also, the use of the questionnaires for the analysis integrating the Engagement Profile, the user satisfaction questions, as well as the capability questions appeared to be a suitable procedure for analysis.

By using a mid-fidelity prototyping approach we were able to implement the six capabilities and adapt them each week. Although performance was not always perfect, we observed as previous studies have, that weakness could be perceived as a plus, as when mistakes lighten the mood of the class [35], or perhaps by eliciting altruistic feelings to protect and help an imperfectly functioning robot [60]; for example, in the case of the students laughing when the robot reminded the class it was time to do something but did not specify what it was time for.

Since the actual design of the robotic teaching assistant setup was on the low end in the desired ranges (cf. Figure 4), we observe that the majority of the participants desired more of all eight Engagement Profile dimensions. However, it seems puzzling that the participants did not want much more of the physical dimension, since being physical is a vital part of the nature of a robot, which as noted previously would seem to make robots be perceived as more engaging than virtual agents or objects. A possible reason is because the prototype motion capabilities were quite simplified, and students might not have been able to imagine them resulting in a significant reduction of the teacher’s workload, which could benefit the class. However, one could also reason that appliances that are less capable of physical interaction than a robot (such as Google Home or Amazon Alexa) could be sufficient for use as a teaching assistant, which might be desirable due to reduced cost. However, we believe that the property for a robotic teaching assistant to be recognized as a tangible entity which can communicate and physically affect us and the world around us will be essential, especially in the future as designs become more complex and powerful.

5.1. Limitations

The current work has various limitations regarding the study design, our approach to designing for engagement, and the robot’s capabilities. We discuss these limitations and their potential impact below.

Our study was conducted with young adults of various nationalities at an engineering master’s course, but there are many variables including age, culture, gender, and expertise and more, which can affect perception of robots. For example, age in adult classrooms can differ substantially. Young adults can be uncertain about their status as adults, with low rates of marriage, parenthood, and occupational experience [61], and can also have different media preferences than older adults [62]. As well, university classes in Sweden exhibit high cultural diversity, especially at the Master’s level [63], where cultural differences in attitudes toward robots have been noted [64]. Moreover, men have expressed more positive attitudes toward robots than women [65].

In regard to the Engagement Profile, the online questionnaires were convenient for collecting statistics but alone did not provide a way to find out why the students thought the way they did (i.e., the kind of probing which interviews allow for), which complicated efforts to improve the system.

Also, the students had a week to answer the questionnaires, which was not optimal. It would have been better if the responses had been given immediately after the learning experience, as there is evidence in the literature that results can be biased when the memory is not fresh [66]. However, there were practical reasons for performing the surveys as described previously.

Additionally, participation in questionnaires was low, possibly because the questions were the same each week, we did not give incentives to the students, and they were to be completed outside of class time. For this reason, the responses might have been biased and not representative of the class as a whole. Thus, the low number of responses to our questionnaires limited our analysis and led to some outcomes that we could only use as indications. We also note that we wished that the number of responses to our questionnaire could have been higher, specifically in week 2 where only five students responded. Therefore, the analyses using statistics might not be conclusive for such low numbers of responses. However, the results still give some indications that the participants reacted positively to the experiments, as shown in Table 9.

Also, the impact of changes made in the design was not as large as expected. Possibly, the introduction of a new artifact such as a robot teaching assistant had a greater influence on satisfaction and engagement than adjusting the robot’s capabilities. In hindsight, we recognize that the changes to the capabilities might not have been large enough between the iterations to show an impact in the Engagement Profile and satisfaction ratings. Probably, more iterations would have been useful, so that the novelty factor of using a robotic teaching assistant would be reduced, giving the changes between the iterations more room for comparison by the users.

Some limitations also apply to the mid-fidelity prototyping approach we employed, where capabilities were implemented only as prototypes or mock-ups. Thus, the teacher needed to construct situations and effectuate some actions by the robot manually, as capabilities were not fully autonomous. Being forced to press buttons and cause certain trigger-events required some need to concentrate on the robot instead of teaching. We assume that this had an impact on what the students experienced and satisfaction. Further, content for the robot such as quizzes and gestures needed to be prepared in advance. As there is no authoring system available, this can require time and resources, as well as being inflexible.

Moreover, the six capabilities we investigated were derived from functionality requirements and consideration of their usefulness was conducted from a theoretical perspective. We did not explicitly consider the abstract property of a robotic teaching assistant being recognized as a tangible entity that one can communicate with, or the degree to which the robot being evaluated is actually perceived as being capable of different capabilities. For future studies, we suggest that the impact of such properties be included in the research design.

In conducting our study, some unexpected behaviors were also observed, as was described, like in mistaking the timing for the robot to hand out an attendance sheet. Such events can influence how the students thought about the robot, but we think this was not a critical problem for our work for the following reasons: First, the contribution of the work is elsewhere, on reporting on how we can apply an iterative design tool (Engagement Profile) from a different domain to designing a robot teaching assistant, based on requirements we identified from teachers. Also, the students reported a positive general impression of the robot despite the few failures which occurred; and, the literature reports that mistakes from a robot can actually have some positive effects in helping students to feel less self-conscious about performing perfectly [35]. Moreover, it was known from the start that some failures could be expected due to the challenging setting: the paper explores testing “in the wild” in a real world setting where there is really no way to avoid all troubles from occurring [67,68]—there is no robot we know of which already has the capabilities we described, and even human teachers can make mistakes in real class settings, as our teachers described. The students also were aware from the start, from the robot’s self-introduction, that the robot was a work-in-progress.

Conversely, we think it is a strength of the adopted approach that we report on such observations, because our exploratory experiment (based on accepted methodology such as grounded theory and prototyping approaches [69]) is designed also for the purpose of trying to expose such failures early on. It is known that publication bias and the file-drawer effect are serious problems which severely impede the scientific community’s understanding [70], and hiding or designing around failures runs the risk that others will make similar mistakes, which would be desirable to avoid. Since there is increasing interest in this area, with many courses around the world starting to use robots, we think there is a use for identifying such potential failing points early on by adopting such a research design.

5.2. Future Work

As described previously, various past studies have already reported good results using robots with people of different ages, cultures, and non-engineering students (e.g., in language classes), as well as engineering students, but more work is needed. Future work will involve comparing different demographics (e.g., young adults versus elderly, different cultures, and in different fields).

For the Engagement Profile, we shall use the questions we conceived to further develop this analysis and design methodology for artifacts in a teaching context. We will also explore the possibility of dedicating time at the end of classes to answering questionnaires in order to gain more responses and how to also conduct some interviews to gain insight into why the students thought the way they did.

Regarding capabilities for a robot teaching assistant, we focused on six capabilities which our teachers indicated as desirable, but there could be various other qualities which could be useful.

Also, in general, we pressed buttons and timers to trigger robot behaviors such as quizzes and reminders, but the robot itself should be able to flexibly determine the right time to act; this includes recognizing when students are listening.

As well, content such as quizzes or suggestions for extra reading had to be prepared ahead of time manually by the teacher, and it was not possible to take into account what each student knew; future work will explore how the robot itself can construct or select content, which will have benefits such as allowing for continuous evaluation. A fundamental related problem is analysis of student behaviors and responses, which can be addressed by approaches such as educational data mining (EDM) and learning analytics (LA) [71]. For example, text mining can be conducted on a student’s verbal or written responses; the abstract structure underlying an answer can be represented using a tree or in the case of programming in terms of how input data are transformed [72]. Some other typical applications of such approaches include characterizing students by learning behaviors, knowledge levels, or personalities [73].

Such analysis should be utilized to plan a robot’s behaviors. For example, some work has already explored how deep learning can be used for an intelligent tutoring system (ITS) to select quiz questions in an optimal way for a single student based on knowledge tracing [74]. Further work is required to explore how to do so for groups of students in a class. This is not a simple question as decisions must be made about which students, if any, to prioritize: weaker students who require help to meet minimum learning requirements, or stronger students who are trying hard to learn. Additionally, another attractive prospect would be if a robot can itself learn to improve its knowledge base, e.g., from YouTube videos [75]; domain-specific learning has been demonstrated in various studies but general learning across various topics, and machine creativity for generating new content, are still desirable goals for future work.

As well, insight from various previous studies could be used to take these capabilities to a higher level of technological readiness: Greetings, a useful way to enhance engagement [76], could be more effective if personalized. For example, a system was developed which could greet people personally, proposing that robots can remember thousands of faces and theoretically surpass humans in ability to tailor interactions toward specific individuals [77]. However, much future work remains to be done in this area in regard to how content can be personalized, e.g., by recognizing features such as clothing or hairstyle, or leveraging prior knowledge of human social conventions.

As well, for alerting capability, adaptive reminding systems have been developed which seek to avoid annoying the person being reminded or making them overly reliant on the system, such as for the nurse robot PEARL [78]. We expect that this challenge will become more difficult when dealing with a group of people, where some times and actions might be good for some students but not for others. For example, some students might finish an exercise faster than others, but should a robot wait before initiating discussion until the majority of students have stopped making progress on the exercise; and if so, how can this be recognized? Possibly, a robot could detect, for each student, when a student’s attention is no longer directed toward a problem (e.g., by tracking eye gaze [79]), nothing has been written for a while, verbal behavior focuses on an unrelated topic, or potentially informative emotions such as satisfaction, frustration, or defeat are communicated—although we expect accurate inference to be quite challenging given the complexity of human behavior.

Various interesting possibilities for improved remote operation are suggested in the literature. For example, a robot could be used by students with a physical disability to carry out physical tasks [36], or students with cancer to remotely attend classes [19]. Moreover, it was suggested that robots’ limited capabilities actually facilitate collaboration, e.g., by increasing participation, proactiveness, and stimulation for operators [18,21]. Future work could consider how to share time on robots between multiple students, and provide enhanced sensing capabilities which human students might not have, like the ability to zoom in on slides. Moreover, ethical concerns should be investigated, such as if students using robots to attend classes could be vulnerable to being hassled or feeling stigmatized; or conversely, who is accountable if a student uses a robot to cause harm?

Clarification and automatically supporting the teacher (e.g., by looking up and showing topics on a display while the teacher talks) could be improved by incorporating state-of-the-art speech recognition and ability to conduct information retrieval with verbose natural language queries, such as is incorporated into IBM’s Watson, Microsoft’s Siri, Google Assistant, and Facebook Graph Search [80]. How to display such information in an optimally informative way is an interesting question for future work, both for individual students and groups of students.

Physical tasks such as handing out materials could also be improved by considering previous work: for example, approaches have been described for making hand-overs natural and effective [81], for handing materials to seated people [82], and for how to approach people to deliver handouts [83]. For example, a robot’s velocity profile can be controlled to be distinctive and evoke trust [84], in students who might not be accustomed to interacting with robots. Mechanisms for dealing with complex and narrow human environments with obstacles can also be considered; for example, a drone could be used by a large robot to deliver handouts to students in locations which are difficult to directly access.

As well, the question of whether an embodiment is truly needed arises again. As noted, various studies have suggested the usefulness of robots as an engaging medium within certain contexts. For example, a human-like robot was considered more enjoyable than a human or a box with a speaker for reading poetry [85]; the authors speculated that the robot was easier to focus attention on than the box, and that the robot’s neutral delivery allowed for more room for interpretation and for people to concentrate and immerse themselves more. However, five out of six of the capabilities we explored—all except physical tasks—do not strictly speaking require an embodiment. Not using an embodiment could be cheaper; also, using speakers and screens distributed through a classroom (e.g., as desks) could make it easier for all students to be able to see and hear to the same degree. This suggests the usefulness of some future work to examine cost/value trade-offs for having an embodiment within various contexts. Furthermore, additional mechanisms such as emotional speech synthesis could be used for capabilities such as reading, greeting, and alerting to have a more engaging delivery.

6. Conclusions

The contribution of the current study lies in reporting our experiences exploring a process for designing an engaging teaching assistant robot. This involved developing a robotic teaching assistant for a university-level engineering course and observing how this robot was received by the students. We performed an analysis once a week over a three week period using questionnaires that contained questions about robot capabilities, user satisfaction, and an adapted version of the Engagement Profile. This process is also summarized in a video accompanying the current article (Video S1: Supplemental video).

During the study, we were able to study some capabilities which were identified by teachers as being potentially desirable for a robot teaching assistant: reading, greeting, alerting, remote operation, clarification, and motion. Findings for each of these capabilities were aligned with previous results from the literature, along with recommendations for future work. Personalizing such capabilities could be useful to improve interactions with the robot teaching assistant; moreover, the role of embodiment needs to be further investigated.

As a secondary finding, the adaptation of the Engagement Profile to the case of a robotic teaching assistant and the iterative design process in developing the robot teaching assistant appeared to be useful. However, changes between iterations should have been more significant to receive stronger indications of what to improve.

The novelty of this work lies in the adaptation of the Engagement Profile to the scenario of robots in education, as well as in reporting a list of desired capabilities for a teaching assistant robot, and observations in regard to the students’ experiences. We look forward to reporting on further advances in outlined challenges, as robots are increasingly adopted to also engage adults within the context of learning.