Aiding Automated Shuttles with Their Driving Tasks as an On-Board Operator: A Case Study on Different Automated Driving Systems in Three Living Labs

Abstract

Highly automated shuttle vehicles (SAE Level 4) have the potential to enhance public transport services by decreasing the demand for drivers, enabling more frequent and flexible ride options. However, at least in a transitionary phase, safety operators that supervise and support the shuttles with their driving tasks may be required on board the vehicle from a technical or legal point of view. A crucial component for executing supervisory and intervening tasks is the human–machine interface between an automated vehicle and its on-board operator. This research presents in-depth case studies from three heterogenous living laboratories in Germany that deployed highly automated shuttle vehicles with on-board operators on public roads. The living labs differed significantly regarding the on-board operators’ tasks and the design of the human–machine interfaces. Originally considered a provisional solution until the vehicle automation is fully capable of running without human support, these interfaces were, in general, not designed in a user-centered way. However, since technological progress has been slower than expected, on-board operator interfaces are likely to persist in the mid-term at least. Hence, this research aims to assess the aptitude of interfaces that are in practical use for the on-board operators’ tasks, in order to determine the user-centered design of future interfaces. Completing questionnaires and undergoing comprehensive, semi-structured interviews, nine on-board operators evaluated their human–machine interfaces in light of the respective tasks they complete regarding user variables such as work context, acceptance, system transparency, and trust. The results were highly diverse across laboratories and underlined that the concrete system setup, encompassing task and interface design, has a considerable impact on these variables. Ergonomics, physical demand, and system transparency were identified as the most significant deficits. These findings and derived recommendations may inform the design of on-board operator workspaces, and bear implications for remote operation workstations as well.

1. Introduction

For mobility to shift away from individual mass motorization to more sustainable means of public transport, mobility services need to change fundamentally. They need to become more flexible, more easily accessible, and more readily available while remaining affordable to a wide range of users. Highly automated shuttle vehicles (SAE [1] Level of Automation 4) are expected to meet these requirements.

1.1. Highly Automated Shuttles

Typically having a capacity of up to 15 passengers, these shuttles enable ride services without requiring a driver. Rides can be tailored to individual mobility needs via on-demand services [2]. Many cities consider this means of transport a substantial component of the targeted mobility shift. For instance, the city of Hamburg, Germany, plans to have at least 10,000 highly automated and electric shuttles by 2030 in order to meet sustainable mobility goals [3].

In order to test the feasibility of highly automated driving in a real-world environment, a large number of countries and cities have set up living laboratories with automated vehicles of different kinds. The funding scheme of the German Federal Ministry for Digital and Transport on Connected and Automated Driving (CAD) lists 79 programs that have received funding since 2018 [4]. As living labs, many of the CAD projects collect data on the deployment of automated vehicles in public spaces. Although reaching “full driving automation” (Level 5 [1]) is the ultimate goal of some of the CAD projects, this goal is at stake. The reason for this is the rather sluggish development of automated driving functions. Driving automation is not progressing as quickly as previously projected [5]. Fully automated driving lies far in the future or may never become fully feasible, at least not in all situations conceivable [6]. Even when limiting the operational context to certain road types, geographic areas, or weather conditions, i.e., running a vehicle with “high driving automation” (HAV, Level 4 [1]), situations remain that require human assistance. Some examples include ambiguous object detection and deadlock situations [7].

Thus, in order to test HAVs in specific use cases while fulfilling safety requirements, many projects that deploy HAVs in the real world include safety operators on board the HAVs. These on-board operators ensure the safe operation of the HAVs by monitoring the accurate execution of the driving tasks by the vehicle automation and intervening when necessary. The interventions may either be in response to an explicit request by the automation or self-initiated by the operator, e.g., to abort an automated driving maneuver when an error occurs. Alternatively, the use of remote operators has been suggested to support vehicle automation or to take over direct vehicle control. However, this approach is linked to a plethora of challenges, e.g., the need to maintain a stable data connection between the remote operator and the vehicle [8]. Thus, it may not be a feasible solution in every project. Both operators on board HAVs and remote operators qualify in general as Technical Supervisors (Technische Aufsichten), required by law on HAVs on public roads in Germany [9].

1.2. On-Board Operators and Their Human–Machine Interfaces

In spite of the decelerated development of driving automation and imposed legal requirements, many HAV projects consider an on-board operator a temporary solution only that will soon be rendered unnecessary. Hence, the design of the on-board operator’s workspace is usually not the focus of CAD projects. Also, workspace development does not always consider the work context and tasks of the on-board operator. This is particularly true for the device that interlinks the on-board operator and the vehicle automation, the so-called human–machine interface (HMI). When developing the on-board operator’s workspace and the entailing HMIs, ergonomics, usability, and user preferences often play a negligible role. Some projects did not factor in an on-board operator from the start, and only added it in over the course of the project. Oftentimes, on-board operators are in an ambivalent role with diverging tasks that center around manually intervening in a myriad of ways when the driving automation fails or requires input [10], but also include peripheral factors such as communicating with passengers, passersby, control centers, and maintenance and cleaning staff, as well as third-party actors such as first responders.

1.3. User-Centered Evaluation

As considering the needs of human users is a crucial requirement in user-centered HMI design [11], this study aims to examine HMIs that are deployed in highly automated shuttle vehicles in light of these user needs. All of these vehicles have in common that they were operated in the context of living labs on public roads. The authors’ goal is to understand how on-board operator HMIs are evaluated by their actual users, i.e., the operators that accompany the shuttle vehicles throughout the operational phase of the living labs. We conducted this study with an awareness that any evaluation heavily depends on the concrete tasks the on-board operators are required to execute, as well as the design of their respective HMIs. Aiming for high ecological validity, the interactions between operator and HMI were examined in the concrete context of use, i.e., on the road while transporting passengers, rather than a standardized lab study. As the interviewed operators typically interacted with HMIs on a frequent basis for an extended period of time (see the Methods section), flaws in the system that surface only after extensive system use were more likely to become evident in this analysis.

This evaluation focuses on the following factors:

• Work Context: What are the circumstances that the on-board operators are exposed to during their shifts on the shuttles? The focus is on ergonomic aspects of the operators’ workspaces, physical demands imposed upon them, the technical equipment they work with, and the way their work is organized.
• Acceptance: Do the on-board operators approve of the workspaces, particularly the HMIs, they work with? Acceptance is closely tied to usability and user satisfaction [12].
• System Transparency: Is the state of the system, in this case, the highly automated shuttle, immediately clear and accessible to the operators? As the automation is responsible for the execution of the driving task by default and the operator is asked to intervene occasionally only, it is crucial for the operator to be aware of the system state at any given time. When the operator is required to intervene, the reason for this need, as well as the way of interacting with the system, is supposed to be disclosed.
• Trust: Do the operators trust the reliability and functionalities of their workspaces on board the shuttles, particularly the HMIs? Trust is closely tied to subjective safety and may originate in good usability, reliability, and user satisfaction [13,14].
• Suitability of HMI Elements for Tasks: Are the HMIs the on-board operators use for fulfilling their tasks on the shuttle actually suitable for these tasks? As tasks vary between living labs, so do HMI elements (for detailed descriptions of the respective HMI setups, see the Methods section).

Across the heterogeneity of the examined living labs (shuttles and environments) and the related operator tasks, the aim is to distill “best practices” of on-board operator HMI design that should be continued in future HMI designs. In addition, issues regarding the currently used HMIs will be highlighted from the perspective of on-board operators that frequently interact with these technologies, ensuring ecological validity.

1.4. Related Work

There is literature on several aspects of highly automated shuttles. One study [2] provided an overview of technical and regulatory perspectives on autonomous shuttle buses for public transport, [15] assessed the safety architecture of automated shuttles, and [16] discussed potential business models for shared automated vehicles. A series of studies revolve around the potential or actual users of these vehicles: [17] focused on the users’ acceptance of and satisfaction with automated shuttles in a living lab, [18] studied participants’ intentions to use automated shuttles, [19] investigated passenger trust and the comfort of first-time users, and [20] asked shuttle passengers about perceived safety and their interactions with the vehicle. Focusing on passengers as well, a study by Schuß et al. [21] attempted to examine the shuttle passengers’ acceptance of on-board operators and to identify the roles and tasks of on-board operators.

However, the specific roles of on-board operators and their perspectives on related tasks, especially the HMI between the on-board operator and the shuttle, have not been prominent objects of research when it comes to shared automated vehicles. Schrank et al. conducted an observation study in the three living labs that are also the object of this research. They investigated the degree of mental effort of on-board operators, ride duration, share of automated driving, and the share of time-critical events, as well as the quantity and quality of manual interventions ([10,22], also see the Methods section).

To the knowledge of the authors, this is the first work that systematically compares HMIs for on-board operators of highly automated shuttles across living labs, using both quantitative and qualitative methods.

1.5. Research Objectives

This paper sets out to answer the following research questions:

• How do the living labs’ human–machine interfaces (HMIs) influence factors of work context, acceptance, system transparency, and trust?
• Are the HMI elements suitable for completing these tasks?
• What recommendations for future HMI design can be derived from the observations?

2. Methods

The aim of this interview-based case study is to gain an understanding of what the on-board operator workspace looks like, the challenges on-board operators face during the operation of shuttles in living labs, and the design of HMIs for the monitoring of and intervening in the automated driving system that is located inside the vehicle. In addition to the results reported hereafter, preliminary interviews were conducted with the on-board operators. These preliminary interviews involved discussions about potential scenarios and situations that arise during vehicle operation and provided initial insights that informed the setup of this case study. The results of the preliminary interviews are not part of this paper.

2.1. Sample and Living Labs Overview

The participants were active on-board operators of highly automated shuttle buses in living labs. They were recruited through inquiries made to project partners of the German Aerospace Center (DLR) who are involved in living labs with highly automated shuttle buses, as well as other living labs in Northern Germany. The living labs were HEAT (Hamburg Electric Autonomous Transportation [23]), RealLabHH (Reallabor Hamburg, Real-World Laboratory Hamburg [24]), and TaBuLa (Testzentrum für automatisiert verkehrende Busse im Kreis Herzogtum Lauenburg, Test Center for Automated Buses in the Duchy of Lauenburg [25]), and were all located in and around Hamburg, Germany (further information in Section 2.3). The interviews were conducted in October 2021 and were carried out in person at the operating sites of the respective living labs. The shuttle types from these living labs are still widely used in research projects across Germany, underlining the persisting relevance of the analysis [26]. Nine on-board operators, three from each of the three different living labs, took part in the interview study. In the subsequent analysis, they are referred to as P1, P2, and P3 for each living lab. The participants’ ages ranged from 24 to 53, with an average of M = 41 years (SD = 10.77 years). The duration of employment as an on-board operator ranged from 6 months to 40 months, with an average duration of M = 14.56 months (SD = 12.16 months).

Table 1. Work experience as on-board operators and age per living lab.

Laboratory	HEAT		RealLabHH		TaBuLa
	M	SD	M	SD	M	SD
Work Experience (Months) 1	8.33	3.21	6.00	0.00	29.33	9.24
Age (Years) 2	29.00	6.24	44.33	6.81	49.67	5.77

¹ Duration of deployment as an on-board operator at time of interview including training phases. ² Age of participants at time of interview.

presents work experience as on-board operators and age per living lab.

The participants took part in this study voluntarily and the interviews were conducted during their work hours. This study was conceptualized and conducted in accordance with the Declaration of Helsinki. Informed consent was obtained from all participants before the interview. The participants were allowed to interrupt or end the interview at any point without justification or consequence.

2.2. Research Design

As a case study that aimed for an initial understanding of the workspaces and the HMIs of on-board operators, the research endeavor followed an exploratory approach and did not postulate hypotheses a priori. Following the approach of methodological triangulation, this study used mixed methods to gain a more detailed and valid understanding of the research context [27]. Several standardized questionnaires were completed by the operators in order to collect comparable quantitative data. This section also included a quantitate evaluation of each living lab’s shuttle HMI elements. As a source of in-depth qualitative data, additional semi-structured interviews with on-board operators were conducted using an interview guide.

2.2.1. Questionnaires

Self-report data were collected from on-board operators using four questionnaires: The Work Design Questionnaire, the Acceptance Scale, a questionnaire on transparency and trust, and a questionnaire to assess the suitability of HMI elements.

The Work Design Questionnaire (WDQ [28]) is an extensively validated tool to comprehensively describe the design of a specific job and is based on the Job Characteristics Model for the overall assessment of work activity [29]. Items from the Ergonomics, Physical Demands, and Equipment Use subscales in the Work Context category were administered. Responses to each item were collected on a 5-point Likert scale ranging from 1: “strongly disagree” to 5: “strongly agree”.

In a publication on user-adaptive art recommender systems, [30] these systems were evaluated using novel scales. This interview study used two of these scales relating to perceived system transparency and trust. In concordance with the previous questionnaire, a 5-point Likert scale ranging from 1: “strongly disagree” to 5: “strongly agree” was applied.

The Acceptance Scale [12] was developed as a standard tool for assessing how readily drivers embrace new technology. With nine items divided in two scales, it measures the usefulness of a system, associated with usability, and the user’s satisfaction with said system, similar to user experience. Responses to each item were collected on a 5-point Likert scale. The poles were semantically opposed statements relating to a construct, e.g., 1: “useful” to 5: “useless”. Items were subsequently recoded to a range from −2 to +2 in order to comply with the original publication’s evaluation guide.

Finally, the suitability of the living lab-specific HMI elements was rated by participants on a 5-point Likert scale ranging from 1: “not suitable at all to complete key tasks” to 5: “completely suitable to complete key tasks”.

2.2.2. Semi-Structured Interview

In addition to the questionnaires, the on-board operators underwent a semi-structured interview. It was subdivided into three parts. In an introductory section, an interviewer queried the on-board operator on personal and sociodemographic data including education and training, previous job positions, and work experience. Next, a detailed analysis of workspace and task design, as well as scenarios and incidents that occur in their work as an on-board operator was conducted. Finally, the interviewer asked for an assessment of the workspace and the shuttle HMI they perform monitoring and intervention tasks with. This manuscript focuses on the interview sections on assessing the workspaces and HMIs.

As the quality of the responses to the initial questions was heterogenous between on-board operators and projects, a post-hoc approach of categorizing qualitative data was applied. In this approach, a deductive method was used, which proved to be useful in gaining comprehensive insights into the text-based material of the interview study. The first step was to review and paraphrase the operators’ responses in order to make them comparable and categorizable. Second, categories were derived deductively from the literature. These categories, which were specifically defined in the area of usability and HMI, provided a theoretical framework and clear definitions. This allowed existing knowledge and established concepts to be applied to the research material, providing a solid foundation for analysis. For the HMI assessment in particular, the established Work Design Questionnaire (WDQ [28]) with its Work Context Scale was used as a scaffold to categorize operator responses on this topic (see

Table 2. Work Context Construct of the Work Design Questionnaire (WDQ [28]): subscales, items, and derived definitions.

Subscales	Items	Derived Definition 1
Ergonomics	The seating arrangements on the job are adequate (e.g., ample opportunities to sit, comfortable chairs, good postural support, etc.). The workspace allows for all size differences between people in terms of clearance, reach, eye height, leg room, etc. The job involves excessive reaching.	Extent to which the working environment allows for correct and appropriate posture and movement.
Physical Demands	The job requires a great deal of muscular endurance. The job requires a lot of physical effort.	Level of physical activity and exertion required at work.
Equipment Use	The job involves the use of a variety of different equipment. The job involves the use of complex equipment or technology. A lot of time was required to learn the equipment used on the job.	Variety and complexity of tools and technical equipment used at work.
Work Conditions	The workspace is free from excessive noise. The climate at the workspace is comfortable in terms of temperature and humidity. The job has a low risk of accident. The job takes place in an environment free from health hazards (e.g., chemicals, fumes, etc.). The job occurs in a clean environment.	Physical environment in which work is performed including explicit health risks, heat, noise, and hygienic conditions.

¹ Derived by authors as a summary of single items from respective subscale.

for used constructs and their definitions). The same scale was used as a questionnaire to collect quantitative data, contributing to methodological triangulation. Once categorization was completed, a valence rating with the three levels “positive (+)” “neutral (o)” and “negative (−)” was added, in order to classify each operator statement as favorable to, indifferent to, or in objection to the addressed aspect of the workspace and HMI, respectively. In order to ensure the intersubjective reliability of the rating, the entirety of the material was reviewed by two independent raters. These raters independently classified the operators’ responses according to the previously established criteria. After this phase was completed, the ratings were compared and any differences in interpretation were discussed. Cohen’s kappa was calculated to quantify interrater reliability, ensuring that the results were consistent.

2.2.3. Procedure

First, the experimenter briefed participants about the objectives of this study and asked them to sign an informed consent form, a non-disclosure agreement, and a data protection declaration. Subsequently, the participants answered sociodemographic questions and questions about their current workspace and its HMI in the shuttle, and filled in the questionnaires mentioned under Section 2.2.2. assessing work context, user experience, trust, system transparency, and acceptance. After that, the participants answered questions about how the workspace design can be improved with regard to the HMI and the distribution of tasks between on-board operators and HAVs. This was followed by questions related to work structures and activities, competencies, training, documentation, and knowledge management. Next, participants rated the HMI elements of their workspace individually for each project, and were asked about its strengths and weaknesses. Additionally, they were asked whether the provided information was sufficient. Subsequently, participants were asked to quantify how suitable the HMI elements were for accomplishing essential tasks. Finally, there was a short debriefing and participants were encouraged to provide overarching comments about their workspace, the HMI and this study.

2.3. Living Labs: Operating Context, Tasks, and Human–Machine Interfaces

Across Germany, a variety of living labs with highly automated vehicles (HAV, SAE Level 4) have been set up. In this study, on-board operators’ workspaces from three of these HAV projects, which focus on passenger transportation, were examined. The vehicles used in the investigated living labs were small electric shuttle buses with a capacity of approximately 12 passengers or less. They were developed as highly automated vehicles, thus not requiring a human fallback level on board of the vehicle in theory. However, due to legal or technical constraints in the living labs, a human operator had to accompany the shuttle buses at all times. The specific tasks the on-board operator had to execute depended on the living lab and the deployed vehicle type (Figure 1). Typical interventions the operators had to conduct are listed in

Table 3. Description of living labs, deployed vehicles, and typical on-board operator interventions.

Laboratory	Location	Period of Vehicle Operation	OEM 1 of Shuttle	Type of Schedule/Route	Driving Environment	Typical Interventions by the On-Board Operator
HEAT [23]	Hamburg Hafencity (urban)	October 2020– December 2021	IAV, Berlin, Germany	Fixed schedule/ fixed route.	New urban district, mixed use (residential, business, and culture), high number of pedestrians and cyclists, including tourists.	Manual driving to pass wrongly parked vehicles. Manual driving to increase distance to obstacle/comply with heavy traffic. Giving clearance to automated lane changes.
RealLabHH [24]	Hamburg Bergedorf (suburban)	July 2020–December 2021	EasyMile, Toulouse, France	On-demand schedule/ flexible route.	Residential neighborhood, narrow streets (often one-way) with cobble stone pavement, thick greenery.	Giving clearance at intersection, roundabout, or signal. Giving clearance after abrupt automated braking (e.g., due to close obstacles). Manual driving because obstacle close to virtual rail.
TaBuLa [31]	Lauenburg (small town)	October 2019– November 2021	Navya, Villeurbanne, France	Fixed schedule/ fixed route.	Old town with narrow streets, cobble stone pavement, steep inclines, sharp turns.	Manual driving to let oncoming traffic pass. Manual driving to pass wrongly parked vehicles. Manual driving to reach designated stopping area at central transportation hub.

¹ OEM: Original Equipment Manufacturer.

.

Being a system component that was not considered in vehicle development from the start, the workspaces of the on-board operators were retrofitted in all of the examined living labs. The shuttle services focused on transporting passengers on the “first/last mile“, i.e., to or from a transportation hub such as a train or metro station (HEAT, RealLabHH), or on fulfilling basic mobility needs in an urban or suburban context (TaBuLa). Details on the deployment context of each living lab are listed in Table 3.

In order to characterize the driving experience from the on-board operator’s perspective, an observation study with 32 rides in all three living labs had been conducted prior to the interview study. Shuttles were accompanied and documented by researchers regarding the mental effort the on-board operators reported, the amount of time spent in automated versus manual mode, and how safety-critical the rides were, as well as the number of operator interventions per hour. See

Table 4. Results from observation study [22].

Laboratory	Mental Effort 1	No. of Rides (Events)	Ride Duration 2 [min]	Duration in Auto. Mode 3 [%]	Duration in Man. Mode 3 [%]	Rated Criticality for Safety 4	Time-Critical Events [%]	Manual Interventions per h
HEAT	79.41	17 (101)	M = 18.12 (SD = 14.02)	51.06	17.78	M = 1.60 (SD = 0.92)	10.00	6.43
RealLabHH	55.00	11 (146)	M = 18.91 (SD = 10.61)	61.42	6.37	M = 1.93 (SD = 1.04)	15.17	30.00
TaBuLa	55.00	4 (43)	M = 23.75 (SD = 2.49)	80.53	12.37	M = 1.65 (SD = 0.69)	6.98	22.74

Auto.: automated, Man.: manual. ¹ RSME [Rating Scale Mental Effort] reaching from 0: “absolutely no effort” to 150: “beyond extreme effort” [32], for working as an on-board operator in this real-world lab in general. ² Including planned stops. ³ Remaining shares are spent during regular and irregular stops or at rest. ⁴ 1: “not safety-critical at all” to 5: “absolutely safety-critical”.

for the results from the observation study.

The following sections focus on the three living labs that were examined in this interview study.

2.3.1. HEAT

The HEAT project [23] ran as a living lab in Hamburg, Germany, with two deployed vehicles that were manufactured by IAV exclusively for HEAT. They were operated as a ring line with a length of 1.8 km at a speed of up to 25 km/h. In the final stage, five stops were offered to board or deboard the vehicle. The shuttles could ride in automated mode only along a predefined path. Whenever a shuttle needed to deviate from this path, the operator was required to steer the shuttle manually.

The on-board operator HMI design comprised six individual elements at a central control station. In the central control station, there was a centrally positioned screen displaying video images. On the exterior of the control station, two screens were installed, showing the side mirror camera feeds. Additionally, there was a screen providing information on the speed and condition of the shuttle. Above and below these screens, there were input elements that could be used to control functions such as lights, horn, hazard lights, turn signals, ramps, and start/stop for automated driving. There were buttons to return the vehicle to automated driving mode after manual intervention. There was also a joystick to control the vehicle if needed. Figure 2 depicts the HMI components of the HEAT shuttle.

2.3.2. RealLabHH

The RealLabHH project ran in Hamburg’s Bergedorf district as an on-demand service that transported passengers using three model EZ10 Generation 3 [33] vehicles from the manufacturer EasyMile. The shuttle vehicles operated at a speed of up to 18 km/h and were able to drive in automated mode in a predefined area. Unlike the two other living lab projects presented here, this feature enabled the passing of obstacles in automated mode.

The HMI inside the vehicle consisted of an information display for passengers, providing information about the vehicle’s surroundings and its operation. Additionally, there was a display featuring a street map. Above the on-board operator’s workspace, there was a screen displaying the forward-facing camera feed. Next to the on-board operator, there was a screen for giving clearance to driving maneuvers, e.g., whenever traffic had to be yielded to, to check the system state, and to modify vehicle settings. In order to control the vehicle, a control panel was worn around the neck of the on-board operator. Figure 3 depicts these HMI components.

2.3.3. TaBuLa

The TaBuLa project [25] was conducted in the Northern German town of Lauenburg an der Elbe with two Autonom Shuttle vehicles manufactured by Navya [34] that ran at a speed of 18 km/h as a fixed ring line. Similar to the living lab HEAT, the shuttles in TaBuLa could ride in automated mode only along a predefined path. Whenever a shuttle needed to deviate from this path, the operator was required to steer the shuttle manually. For more details on the operational context, see [31].

The HMI design comprised a main screen displaying a map and a control panel with diagrams and events. Additionally, the main screen provided information about the vehicle’s sensors, trajectory, schedules, speed, and other status information. In order to execute longitudinal and lateral control of the vehicle, an Xbox^® controller (Microsoft Gaming, Redmond, WA, USA) was used, which hung around the operator’s neck. Further control elements were placed on a black box. For photographs of these HMI elements, see Figure 4.

3. Results

The following section reports the outcomes of the user evaluation of the on-board operator workspaces and entailing HMIs. Results from both the questionnaires on several user-related factors and the semi-structured interview are given. Results are reported in a descriptive manner only due to the nature of the case study; the limited number of participants did not permit statistical inferences to the population.

3.1. Results from Questionnaires

The on-board operators’ responses to the questionnaires on work context, acceptance, trust, system transparency, and suitability of HMI elements for their tasks are reported below. First, applicable subscales from the Work Context category of the Work Design Questionnaire (WDQ) are stated in order to assess the on-board operators’ workspaces and HMIs.

Table 5. Subjective ratings of the work context of the on-board operators’ workspaces measured with WDQ.

Laboratory	HEAT		RealLabHH		TaBuLa
Scale	M	SD	M	SD	M	SD
Overall	2.15	0.65	3.19	0.96	3.07	1.36
Ergonomics	1.44	0.47	1.67	0.94	2.78	1.53
Physical Demands	1.78	0.90	3.44	1.15	3.11	1.61
Equipment Use	3.22	0.59	4.44	0.79	3.33	0.94

Adapted from [28]; low: 1 to high: 5 (center: 3), 1: “strongly disagree” to 5: “strongly agree”.

provides a summary of the operators’ self-report data by living lab.

Second, acceptance of the HMI was measured with Van der Laan et al.’s [12] Acceptance Scale. As shown in

Table 6. Subjective acceptance of the on-board-operation workspace measured with the Acceptance Scale.

Laboratory	HEAT		RealLabHH		TaBuLa
Scale	M	SD	M	SD	M	SD
Overall	0.17	0.42	0.13	0.56	0.89	0.80
Usefulness	0.67	0.52	0.27	0.54	1.2	0.75
Satisfaction	−0.33	0.32	0	0.59	0.58	0.85

Van der Laan Acceptance Scale [12]; low: −2 to high: 2 (center: 0.).

, the scale splits into two subscales, Usefulness, delineating usability, and Satisfaction, referring to user experience.

Third, two scales from [30] were utilized in order to capture perceived system transparency and trust in the workspace and its HMI. The results are stated in

Table 7. Subjective ratings of perceived system transparency and trust in the on-board-operation workspace and its HMI, measured with [30].

Laboratory	HEAT		RealLabHH		TaBuLa
Scale	M	SD	M	SD	M	SD
System Transparency	1.73	0.87	2.33	0.73	4.60	0.57
Trust	2.69	1.02	2.63	0.69	4.17	0.46

Adapted from [30]; low: 1 to high: 5 (center: 3). Range of values: 1: “strongly disagree” to 5: “strongly agree”.

.

Fourth, the suitability of a number of HMI components that differed by living lab was measured on a Likert scale. The results are stated in

Table 8. Suitability of different HMI components implemented in the shuttle of the respective living lab to complete key tasks.

Laboratory	HMI Component 3	Input 1	Output 2	M	SD
HEAT	Central operating station.		X	4.00	0.82
	Central screen for video stream.	X		3.00	0.82
	Peripheral screens for side mirror cameras.	X		4.00	0.82
RealLabHH	Screen that displays the immediate surroundings using sensor data.	X		2.67	0.94
	Screen that displays the surroundings over a large area using sensor data.	X		2.67	1.70
	Central screen showing the shuttle’s location accuracy (GPS) and the current route (green line).	X		3.67	1.25
	Key to switch between manual and automated mode.		X	3.67	0.94
	Information about trips.	X	X	2.00	1.41
	Touch display to make or view vehicle settings (lights, air conditioning, etc.).	X	X	3.33	1.25
	Wide-angle camera for viewing area directly in front of the shuttle.	X		4.00	0.00
	Joystick for direct control of the vehicle.		X	3.00	0.00
TaBuLa	Main screen, consisting of the following:
	(a) Map.	X		5	0
	(b) Control display for diagrams, events, errors.	X		5	0
	(c) Sensor and trajectory display.	X		5	0
	(d) Display of speed and vehicle status.	X		5	0
	Xbox controller for controlling shuttle.		X	5	0
	Control elements on black box (left in the picture 3).		X	3	2
	Operating elements around the small LCD display on the white box (right in the picture 3).		X	5	0

Low: 1 to high: 5 (center: 3). Range of values: 1: “not suitable at all to complete key tasks” to 5: “completely suitable to complete key tasks”. ¹ Information displayed to the on-board operator. ² Information entered by the on-board operator. ³ See Section 2.3 for pictures of HMI components.

.

3.2. Results from Semi-Structured Interview

Adding to results from the questionnaires, the numbers of positive, neutral and negative statements on workspace and HMI grouped by subscales from WDQ category “Work Context” and living lab are listed in

Table 9. Exemplary statements of qualitative content analysis of HMI assessment regarding work context (adapted from respective WDQ category) and valence ratings.

Subscale 1	Laboratory	Examples from Responses (Paraphrased)	+ 2	o 2	− 2	Total
Ergonomics	HEAT	“Turn indicator buttons are too far away [from my position] so I have to stretch.” (P1(−)) “Ergonomically unwise design, very poor posture because retrofitted in addition to the bench seats that had already been in place.” (P2(−)) “I do not want to work in this bent posture. I would like an individually adjustable seat like on the bus with height and distance adjustment.” (P3(−))	0	0	8	8
	RealLabHH	“I have to do a ballet performance [as frequent turning between HMI components is required].” (P1(−)) “There is no seat. Constant standing is a lot of effort and visibility [of the traffic environment] is restricted.” (P1(−)) “All electronic aids should be aligned more clearly and ergonomically in the direction of travel. This is also a safety aspect.” (P1(−))	1	0	16	17
	TaBuLa	“[Operating the shuttle] takes place while standing; I have to be careful since if I were to sit down, I wouldn’t be able to see all the traffic. A seat would be desirable, a place where I can also relax while monitoring the traffic.” (P2(−)) “The emergency call button should be at a position that is more suitable for the operator. I often accidentally touch it with my elbows and shoulder.” (P3(−))	0	0	16	16
	Sum		1	0	40	41
Physical Demands	HEAT	(None)	0	0	0	0
	RealLabHH	“Constant standing is required. It is very difficult to maintain over a longer period of time.” (P1(−))	0	0	3	3
	TaBuLa	“Workstation requires standing at all times so my legs hurt while I am observing the traffic.” (P3(−))	0	0	1	1
	Sum		0	0	4	4
Equipment Use	HEAT	“[Operating system] is easy to use and to learn.” (P3(+))	3	1	1	5
	RealLabHH	“Operation was simple.” (P1(+))	2	0	0	2
	TaBuLa	“[Learning how to operate the system] took me three months. It depends on your motivation. If you feel like learning it, it’s not hard. It was easy for me.” (P3(o))	0	3	5	8
	Sum		5	4	6	15
Work Conditions	HEAT	“We are usually on the job in pairs; one takes care of the passenger, the other drives.” (P3(+)) “Every incident given to the control center, reports written by them, but also information given to each other as on-board operators on road conditions.” (P2 (o)) “I wish that the manufacturer’s responses on incidents should also be communicated to us on-board operators. I want to know how can I avoid this error and what is the reason for it.” (P3(−))	2	3	6	11
	RealLabHH	“With three shuttles running, we don’t need a control center. I opt for reducing communication with the control center and call the police, towing service or ambulance directly, not via the control center.” (P1(−)) “It is important to keep a checklist to make sure everything, e.g., check the brakes. Everything we do is documented in a [physical] folder. I would prefer that we write tickets digitally instead of on paper, e.g., as an app. This would help us save paper.” (P1(−)) “I would like more breaks because you are always on the go and your intervention might be required. This means constant stress.” (P3(−))	1	0	8	9
	TaBuLa	“Working hours are pleasant.” (P2(+)) “Shift plans are good.” (P2(+)) “Documentation via cell phone using voice and text message and by sending pictures via chat, e.g., notes on parking offenders on notes.” (P3(o))	4	2	1	7
	Sum		7	5	15	27
Sum Overall			13	9	65	87

¹ Subscales adapted from “Work Context” category of WDQ [28]. ² Valence ratings of responses levels: (+) positive, (o) neutral, (−) negative.

. Additionally, exemplary responses for each subscale and living lab are reported in this table. Regarding interrater reliability, Cohen’s kappa was calculated to be 0.86. According to [35], this can be interpreted as an “almost perfect agreement” between the raters.

4. Discussion

Using methodological triangulation, on-board operator workspaces of highly automated shuttles were examined across three living labs. Although not a focus of user-centered vehicle and HMI design so far, the results revealed a set of benefits and deficits of these workspaces and their HMIs. This section sets out to first summarize the positive and negative remarks reported in the Results section, and then derive recommendations for the design of on-board operator workspaces in the future.

4.1. Benefits and Deficits of Current On-Board Operator Workspaces

The following section delineates the benefits of and problems with on-board operator workspaces in living labs that were reported by the operators from these labs. Positive and negative aspects of the workspaces in general will be summarized prior to referring to specific HMI elements.

4.1.1. Workspace Design

The following aspects are grouped by the WDQ Work Context subscale, following the Results section. A seemingly trivial, however neglected, aspect of workspace design in the investigated living labs concerns the ergonomics of the overall setup of the on-board operator workspace. Operators complained about ergonomically poor workspaces. Almost all of the responses on this matter were negative. For example, P1 of the living lab HEAT criticized that the “turn indicator buttons are too far away [from my position] so I have to stretch”. Additionally, in order to face the direction the shuttle was moving in, operators had to stand in front of the passenger seats and sometimes even lean over seats to view screens or operate input devices. Another issue operators mentioned was that sitting down was not possible without missing the traffic. As there was no dedicated seat for the on-board operator, the only option was to take a passenger seat. However, these were not directed toward the screens and control elements. Hence, the operator could not sit down within the risk of missing relevant situations in the traffic that might require their attention, nor view or operate the HMI. A third complaint refers to the positioning of some buttons. One operator said that they frequently hit the emergency button by accident with their elbow as it is located within a regular area of movement. This could have serious consequences as the control center or even first responders might be alerted automatically, leading to false alarms. However, there were some positive remarks as well: P3 in HEAT liked the simple operability of the HMI, P1 in HEAT enjoyed the virtual rear mirrors that provided a wider field of view than conventional rear mirrors, and P2 in TaBuLa praised the control screen for its reliability.

In relation to physical demands, operators complained that they had to stand permanently. They were afraid this might inflict physical issues such as problems with posture.

Regarding equipment use, there was a balanced number of positive, neutral, and negative responses. Operators noted that it took them considerable time to understand the HMI: “[Learning how to operate the system] took me three months. It depends on your motivation” (P3 HEAT).

Unlike the previous categories, work conditions depended significantly on the specific living lab. The on-board operator workspaces of the shuttles in HEAT and RealLabHH were criticized for their lack of transparency on what happens to the operators’ feedback (P3 HEAT), the overinvolvement of the control center in operations, and that a paper-based checklist that would have been preferred over a digital solution (both P1 RealLabHH), as well as being exposed to permanent stress due to too few breaks (P3 RealLabHH).

In contrast, TaBuLa, which had been in operation for the longest period, received mostly positive comments on work conditions, for example, regarding pleasant work hours or shift plans (both P2).

4.1.2. Specific HMI Elements

In the subsequent section, the suitability ratings of each lab’s components of the on-board operator HMI will be discussed.

The HEAT living lab received favorable ratings for the central operating station as well as the peripheral screens for the side mirror video stream. Average ratings were yielded for the front and rear video stream displayed on the central screen.

In RealLabHH, operators responded most favorably to the wide-angle camera that transmitted the video stream right in front of the vehicle. Usually not visible without dedicated cameras, this feature might have been perceived as added value to steering a regular, non-automated vehicle. Above-average ratings were also given to the central screen that displayed the shuttle’s location accuracy and current route. It was reported to provide a good overview of the ride in a wider area and might therefore positively influence situation awareness. In addition, the key to alternate between automated and manual driving mode was rated as above average. Another input device, the joystick, received average ratings. This is perhaps because it was located on a control box of considerable weight that had to be carried around the on-board operator’s neck, applying pressure on and potentially straining the neck and shoulder muscles, with the prospect of chronic pain. A similar rating was given to the touch display for adjusting vehicle settings. As it was located on the right side from the perspective of the operator, next to the door, turning was necessary to operate it. This bears the danger that the operator might divert their eyes away from the road. Below-average ratings were given to the two screens displaying sensor data. Seemingly, the operators’ work was not supported significantly by these screens. The lowest suitability rating was given to the trip information provided, implying operators did not find the information about the trip helpful or satisfactory.

In contrast, on-board operators with TaBuLa found their HMI very suitable for their tasks. All HMI components received the highest ratings possible: the main screen with a map view, control, sensor, speed, and status display, the gaming controller used to steer the vehicle in manual driving mode, and the operating elements on the white box. Only one element received an average rating: the control elements on the black box. This might be due to their small size, which meant that the captions on the buttons were barely legible, negatively affecting usability.

4.2. Recommendations for Future On-Board Operator Workspace Design

From these deficits, a number of recommendations for a user-centered design of on-board operator workspaces and HMIs can be derived. First, remarks are made on the workspace overall, before special attention is given to the on-board operator’s HMI. Ultimately, general guidelines on operator workspaces and HMI design are discussed.

4.2.1. Workspace Design

This section provides recommendations for on-board operator workspace design, structured according to the WDQ subscales. Regarding the operator feedback that falls under the subscale on ergonomics, it can be concluded that the different body heights of operators need to be considered. It is essential to enable individual adjustments of seating positions, as well as the location and viewing angle of screens and input devices. Screens that need to be attended during the ride should face in the direction of the driving trajectory, providing operators with critical information without taking their eyes off the road. Such a placement minimizes distractions and contributes to safer operation. If a seat is added for the on-board operator’s safety and comfort, its height should be adjustable. This is to provide a better overview of the surroundings in order to enhance the operator’s field of view and reduce blind spots, e.g., right in front of the vehicle. In order to not accidentally hit buttons, as was reported for the emergency button in one of the presented shuttles, it should be checked that operators can move freely without hitting any control elements by mistake. Buttons should be made readily accessible and easy to reach in case of unforeseen situations or emergencies. The placement of buttons should always be task-related. This means that a button’s position is a consequence of the task or workflow it is a part of. For example, a button to give clearance to driving maneuvers should be placed in the direction that needs to be checked in order to do so. In general, it is strongly advised that the guidelines on ergonomic work design [36] and human-centered design [37] are followed.

When it comes to the physical demands imposed on the operator, a central point is the permanent standing that was required to monitor the traffic surrounding the shuttle vehicle. An ergonomically sound and individually adjustable seat, comparable to that of a regular bus driver, combats posture problems, thus preventing neck and back pain. A positive side effect would be that longer shifts might be possible this way. If a fixed seating position was established, the placement of control units and displays could be oriented toward the operator, preventing posture issues that result from leaning forward or to the side to view or operate HMI elements, or from carrying a control unit around one’s neck (as in RealLabHH and TaBuLa).

Regarding equipment use, it took operators quite some time to understand and be able to use the operating systems. A factor to consider might be the transparency of the system. The HMI should make the system’s state, as well as the system’s decisions and reasons for decisions, evident to the operator. A lack of system transparency was reported in HEAT and RealLabHH. Both living labs yielded below-average ratings by operators. In addition to transparency, adequate training is important to enable the operators to interact smoothly with the vehicle via the HMI. It is advisable that both routine use cases and edge cases are incorporated into the operator training, in order to prepare the operator for safe and effective HMI use in as many situations as possible.

Finally, conducive work conditions are essential for effective task performance. In addition to manageable shift lengths, regular breaks, and reliable support by coworkers, this also includes aspects of communication to third parties. An example is operators being kept in the loop when it comes to feedback they provide, e.g., on incidents regarding malfunctions with the automation. Particularly in living labs that depend on continuous feedback to improve vehicle automation and operational workflow, on-board operators should be kept in the loop regarding the impact that their feedback has in the optimization process. As the operation of a shuttle vehicle is a complex sociotechnical system, all actors and components in the system have to be carefully analyzed regarding their interactions and the way they collaborate. Tasks need to be clearly assigned to an actor. If actors share tasks, their respective share of the tasks should be explicated. All actors should be involved in this process of specifying and allocating tasks. This would help prevent problems such as the perceived overinvolvement of the control center that some operators complained about. Lastly, the advantages of digitizing processes should be reaped as much as is seen fit in the particular context of use. For instance, if considered easier and less time-consuming, completing checklists digitally could replace paper-based workflows.

4.2.2. Specific HMI Elements

The following section lists recommendations for improving the design of the HMI by living labs. The main points to be improved in HEAT concerned the display of the front and rear video stream. The screens could be enlarged to help the operator recognize smaller objects that might have an impact on the shuttles, e.g., obstacles on the street ahead. In order to aid the operator in recognizing relevant objects, the salience of these objects could be increased, for example, by surrounding them with boundary boxes.

In RealLabHH, the rating for the joystick to control the shuttle was average only. This might be because of the weight posed on the operator’s body. Placing the joystick in a fixed position or making it lighter might alleviate these complaints. The touch display received similar ratings. It could be positioned in the direction of the tasks. For example, to give clearance to driving maneuvers, the display should be oriented towards the traffic environment that has to be checked beforehand, rather than next to the door. Even more importantly, as both screens for sensor data presentation were rated below average, the concept for displaying sensor data should be revised. It should be carefully reflected in which tasks and how exactly sensor data can meaningfully support the operator, rather than displaying all data all of the time. Most importantly, meta-information on the trip needs to be provided in a clear and concise way, so the operator is aware at all times of data regarding the itinerary, pathway, and passengers getting on and off the vehicle.

In TaBuLa, the only point of improvement pertains to the control elements at the black box. Legibility needs to be ensured without diverting attention from the road for an extended period of time. Alternatively, the use of easily recognizable icons could be considered.

4.2.3. General Guidelines for On-Board Operator Workspace and HMI Design

In addition to these lab-specific recommendations for on-board operator HMI design, general guidelines for workspace and HMI design are subsequently provided. Regarding the utilization of cameras to provide the on-board operator with a view of the surroundings, the integration of high-quality camera perspectives is essential. This explicitly includes rear-view cameras. A helpful video stream allows operators to anticipate distances and detect small objects. It should be ensured that the camera’s view remains unobstructed even when doors or windows are opened. Unhindered visibility reduces blind spots and ensures that operators have a clear view of their surroundings at all times.

When it comes to screens, it might be helpful if the position of the primary screen with critical information is directly in front of the operator. This enables operators to perceive important information without taking their eyes off the road. Precise and responsive touch displays are critical for user interaction. Screens should be strategically positioned to provide a comprehensive overview of relevant information. They should be mounted securely to keep operators and passengers safe. Also, there is a need for a navigation display and a display that shows the shuttle’s route and destination prominently, helping operators navigate effectively. Showing the driving path as a line on the display may assist operators during manual driving. When a new path has to be specified by the operator, the display line could help the operator to set the shuttle’s new path accurately. Furthermore, it could be helpful to keep screens, controls, and buttons easily reachable from the operator’s position. When errors occur, displaying detailed error information is imperative in order to assist operators in resolving issues efficiently, reducing downtime. This information can help operators to resolve issues through the HMI seamlessly without requiring a hard reset of the entire system, which would prolong downtime.

Output channels do not have to be limited to the visual modality. A multimodal approach may bring about crucial advantages. The implementation of auditory signals may be useful for notifying operators of new tasks or critical events that require their attention. The use of visual cues, such as illumination rings around buttons, can clearly indicate when a system is active. This may help operators to stay informed about the status of the vehicle functions. In order to make sure operators are fully aware of the vehicle’s operating mode, it may be helpful to clearly indicate whether the vehicle automation is active or not. Color-coded output can help operators quickly identify and prioritize information and separation in the application. It might be necessary to provide output in an additional manner in case of color blindness or other vision impairments.

Regarding input components, illuminating buttons and controls can help operators identify and operate them safely and unambiguously. Furthermore, buttons with tactile textures or surfaces are useful to prevent fingers from slipping during use. The position of takeover buttons is particularly useful close to the steering unit, e.g., the joystick, to allow operators to swiftly take over control when needed.

Moreover, the HMI needs to be designed in a user-centered manner in order to ensure user-friendliness. There are several aspects that should be avoided in order to ensure the HMI of the on-board operator’s workspace contributes to safe, effective, and efficient shuttle operations. In order to ascertain this, the following principles should be adhered to (see also [11,38]):

• Do not overuse touch buttons and balance them with regular physical buttons.
• Only use buttons that contain feedback mechanisms to confirm to operators that their input has been noted.
• Make sure buttons have a distinct appearance and may not be activated by mistake. For example, an ‘On/Off’ button should not closely resemble an ‘Autonomous Driving’ button.
• Ensure buttons are easily visible and accessible, and not hidden behind other objects.
• Do only display information that is needed, depending on the operator’s current task. Ideally, the display is adaptive to the operator’s current informational needs.

4.3. Limitations of Study

This study comes with some limitations. First, since this research endeavor pursued a case study paradigm with a focus on qualitative data collection, the unique operational contexts of the living labs investigated may not fully represent the diverse challenges and setups of automated shuttle systems globally. It is suggested that future research takes a more holistic approach, examining a broader variety of automated shuttle systems. This could also include a systematic consideration of external factors, e.g., passenger behavior, traffic conditions, and weather, on shuttle operations. The results from this study may provide a starting point for this.

Second, this study relies on the perspectives of a small number of current operators in specific living labs within a distinct regulatory and cultural context. Even though the operators’ perspectives cover months or years of continuous exposure to and experience with the shuttle and on-board operator HMI systems, this experience may not be representative of operators with varying levels of technical proficiency or from different cultural backgrounds. Including a broader demographic of operators as well as additional operational contexts in future research could yield additional insights into the usability and ergonomic design of on-board operator HMIs and facilitate a global application.

Third, as the chosen approach is based on on-board operators’ self-report data, no objective measurements of operator performance and system usability were conducted. Integrating quantitative data, such as reaction times, error rates, and physiological measures (e.g., stress levels), may help to assess and compare on-board operator HMIs in a more valid and reliable manner. Suitable methods may encompass systematic interview studies with users of automated shuttles [39] and eye-tracking paradigms [40].

Fourth, even though the operators interviewed in this study had considerable experience in overseeing their respective automated shuttle systems, this study collected data only once, and therefore does not account for the long-term impact of ergonomic and HMI design on operator health and system efficiency. Longitudinal studies tracking changes in operator well-being, system trust, and performance over time would provide valuable data on the feasibility of current HMI designs.

Fifth, it has to be noted that although the investigated workspaces and HMIs for on-board operators were state of the art at the time of the interviews, rapid technological advancements may have an impact on on-board operator workspace design. Thus, this study’s findings may need to be reviewed in future research in order to stay relevant. Follow-up studies should explore state-of-the-art HMI designs longitudinally at regular intervals, including adaptive HMIs, along with other new developments.

5. Conclusions

This paper reported and discussed the quantitative and qualitative responses of on-board operators of highly automated shuttle vehicles regarding their workspaces on a case-by-case basis across three living labs in Germany. In summary, it was demonstrated that the design of on-board operator HMIs, together with other factors pertaining to the operational context, has a considerable impact on the perceived ergonomics, physical demand, acceptance and transparency of, and trust in on-board operator workspaces. The suitability of HMI elements for completing the respective tasks varied widely across projects, partially influenced by the duration of the living lab and its task structure. It was observed that the examined workspaces and the HMIs used by on-board operators were not designed in a user-centered and integrated fashion, but rather, were added at a later project stage due to technical or legal requirements. Thus, they often fall short regarding important safety- and health-relevant considerations and have considerable potential for improvement regarding factors of work context, acceptance, transparency, and trust. It is recommended that the development of on-board operator workspaces and HMIs follows a user-centered design process that factors in user needs from the beginning, both pertaining to ergonomics and usability, and is well-synched with the concrete use cases and tasks the operators are exposed to.

As on-board operators are projected to remain a part of the operation of highly automated vehicles in a variety of circumstances, at least in the mid-term, the authors advocate for a deliberate and sound application of the user-centered design process when developing on-board operator workspaces and HMIs that are aligned with vehicle design, the concrete tasks of the on-board operator, and the intended context of use. Recommendations for improving existing on-board operator workspaces and HMIs were derived from the operators’ responses. Finally, it can be noted that these ecologically highly valid insights and design recommendations may be helpful as a resource in designing other workspaces as well, e.g., the remote operator workspaces [41].