Analysis of the Interaction between Humans and Autonomous Vehicles Equipped with External Human–Machine Interfaces: The Effect of an Experimental Reward Mechanism on Pedestrian Crossing Behavior in a Virtual Environment

Abstract

The advent of autonomous vehicles (AVs) has sparked many concerns about pedestrian safety, prompting manufacturers and researchers to integrate external Human–Machine Interfaces (eHMIs) into AVs as communication tools between vehicles and pedestrians. The evolving dynamics of vehicle–pedestrian interactions make eHMIs a compelling strategy for enhancing safety. This study aimed to examine the contribution of eHMIs to safety while exploring the impact of an incentive system on pedestrian risk behavior. Participants interacted with AVs equipped with eHMIs in an immersive environment featuring two distinct scenarios, each highlighting a sense of urgency to reach their destination. In the first scenario, participants behaved naturally without specific instructions, while in the second scenario, they were informed of an incentive aimed at motivating them to cross the road promptly. This innovative experimental approach explored whether motivated participants could maintain focus and accurately perceive genuine risk within virtual environments. The introduction of a reward system significantly increased road-crossings, particularly when the vehicle was approaching at higher speeds, indicating that incentives encouraged participants to take more risks while crossing. Additionally, eHMIs notably impacted pedestrian risk behavior, with participants more likely to cross when the vehicle signaled it would not stop.

1. Introduction

As the development and deployment of autonomous vehicles (AVs) continues to advance, it is becoming increasingly important to consider how they will interact with other road users, particularly pedestrians. Pedestrians are often the most vulnerable road users, and their safety is a critical concern in the design and implementation of AVs. One approach that AV manufacturers and researchers are employing to address this issue is the use of external Human–Machine Interfaces (eHMIs). These interfaces are specifically designed to facilitate communication between the AV and other road users, including pedestrians. eHMIs can take many forms, including digital displays, lights, and even sound [1,2,3].

The use of eHMIs can help to improve the safety of pedestrians by providing them with clear and concise information about the intentions of the AV. For example, an AV equipped with an eHMI may display a message on its digital display indicating that it is stopping to allow pedestrians to cross the road. This can help to reduce confusion and uncertainty and make it easier for pedestrians to navigate around AVs [4,5,6,7]. However, prolonged exposure to eHMIs can potentially lead pedestrians to excessively rely on the displayed message while potentially ignoring implicit vehicle cues [8].

Nevertheless, there are still many challenges to be overcome in the design and implementation of eHMIs. For instance, creating displays and messages that are easily understood by a diverse range of pedestrians, including those with visual or hearing impairments, can be challenging. Moreover, further research is needed to determine the most effective types of eHMIs for different situations [9,10,11,12].

Researchers have employed various methods to evaluate the suitability and effectiveness of eHMIs in conveying the intentions and behavior of AVs to pedestrians, such as self-report studies, questionnaires and interviews [5,13,14], computer-based experiments [1,15,16,17,18,19], real-world experiments and field tests [4,10,20,21,22,23,24,25], and virtual reality studies, using Cave Automatic Virtual Environment (CAVE)- and Head-Mounted Display (HMD)-based simulators [26,27,28,29,30,31,32,33]. In summary, the interaction between pedestrians and AVs is a complex and evolving topic, and the use of eHMIs is just one of the many strategies being developed to improve safety and communication on the road.

The use of virtual environments to study pedestrian behavior proves to be an effective strategy to avoid time-consuming and costly field studies, while simultaneously addressing ethical concerns. These simulations, also known as pedestrian simulators, enable the presentation of traffic scenarios from a pedestrian’s perspective, allowing for a thorough exploration of safety measures. Numerous studies have successfully employed pedestrian simulators to explore road-crossing behavior [34,35,36,37,38,39,40,41,42,43,44,45,46,47]. These setups provide a secure, adaptable, and highly controllable environment, offering precise control over variables. However, challenges may arise due to disparities between simulated environments and naturalistic traffic conditions, originating not only from sensory cue mismatches but also from difficulties in scenario design [48].

Among the most frequently simulated scenarios in pedestrian studies is the task of crossing the street, given its safety implications and relatively compact spatial demands. The decision to cross involves a delicate balance between saving time and minimizing the risk of injuries [49]. Still, the imposition of time pressure in simulations may seem unnatural, as individuals typically cross the road with a destination in mind rather than simply for the act of crossing. This perceived lack of significance in simulator studies can lead to diminished motivation, tedium, and exhaustion.

To address this challenge, one strategy involves the incorporation of game elements into non-game contexts, a concept commonly known as “gamification” [50]. The interplay between rewards and human behavior has been extensively studied across various disciplines, and gamification has shown positive outcomes, particularly in enhancing perceived competence [51,52]. This game-based strategy has also been employed to educate children on road safety [53]. Given its success in diverse applications, similar principles are likely applicable to pedestrian simulation scenarios. Notably, time pressure, often associated with temporal restrictions, has been shown to influence pedestrian behavior. For instance, children exhibited riskier behavior when instructed to act quickly [54]. These examples emphasize the potential impact of gamification elements, such as time constraints, on shaping pedestrian behavior in simulated environments.

This study aimed to investigate and analyze pedestrian crossing behavior in the presence of AVs equipped with eHMIs while exploring the impact of a game-based reward mechanism by considering a range of personal characteristics and background variables as potential influencing factors. The experiment involved two distinct scenarios presented to the participants in a CAVE-based pedestrian simulator. In the first scenario, participants were instructed to behave naturally, while in the second scenario, they were made aware of an incentive to cross the road by means of a game-based reward system. In both cases, participants were asked to act as if they were in a hurry to reach their destination. This innovative experimental approach was designed to determine whether participants, when offered motivational incentives to cross the road, would maintain their focus throughout the experiment, sustain a constant state of urgency, and genuinely perceive a sense of risk.

The researchers expected to answer three main questions in the context of autonomous driving and pedestrian–vehicle interaction: (i) how the inclusion of eHMIs contribute to pedestrian safety; (ii) how pedestrians’ risk behavior change when they are presented with an incentive to cross the road; and (iii) what factors contribute to the change in pedestrians’ behavior. This research will contribute to a greater understanding of how these vehicles should interact with pedestrians, facilitating the development of effective safety measures and policies. For the purpose of this study, the term AV refers to a fully automated vehicle that operates without a human driver, conforming to the classification of SAE level 5 [55].

2. Method

2.1. Participants

A total of 34 adult participants were recruited from the University of Minho and the surrounding community. To be eligible, participants had to have resided in Portugal for at least one year and provided informed consent before the experiment. All participants volunteered to participate in this study without receiving any economic compensation. One male participant was unable to complete the experiment due to moderate simulator sickness and fatigue and was therefore excluded from further analyses. The first two participants, one male and one female, were recruited to help calibrate the equipment and were also excluded from the analyses. Ultimately, 31 participants (17 female and 14 male) were considered for analysis. Their ages ranged from 25 to 44 years old (M = 31.2, SD = 4.54). The study lasted approximately one hour and was approved by the ethics committee of the University of Minho.

2.2. Simulator Setup and Virtual Environment

The study took place at the Centre for Computer Graphics at the University of Minho, using a CAVE-based pedestrian simulator [46,56]. The simulator boasts a large nine-meter-wide by three-meter-high screen, onto which stereoscopic images were projected using an array of three projectors. Participants were equipped with 3D glasses and headphones throughout the experiment allowing them to see the images in three dimensions. Their head’s position and orientation were tracked using an infrared multi-camera system from Vicon (www.vicon.com, accessed on 20 February 2024) which captured the position of a set of reflective markers attached to the headphones. This allowed for the adjustment of the perspective of the projected images to the point of view of the participants in real time. It also allowed for the auralization of the sounds of the vehicle and the surrounding environment which were wirelessly transmitted to the headphones. The room itself provided a controlled environment spanning approximately 36 square meters, ensuring the safety of participants as they walked during the experiment.

The virtual environments employed in this study were generated using the Blender engine software Version 2.79b (www.blender.org, accessed on 20 February 2024). These environments depicted two real-world urban settings, specifically residential areas featuring buildings and trees positioned on either side of the road (refer to Figure 1). To enhance realism and avoid a sense of emptiness, additional parked vehicles were placed nearby. In both scenarios, traffic consisted of a single vehicle traveling along the outer lane, approaching from the left side of the participant. As indicated by the study conducted by Rodríguez Palmeiro et al. [57], the direction from which the vehicle approached was found to be irrelevant to pedestrian crossing behavior. Each lane of the one-way street within the virtual environments had an approximate width of three meters. Both virtual environments were designed and rendered to offer participants with immersive and realistic experiences, aligning with the specific objectives of the study. To prevent visual monotony and repetitiveness for participants, the assignment of each virtual environment to experimental blocks was randomized before the start of the experiment.

2.3. Experimental Design

The design methodology employed in the experiment encompassed three key conditions: (i) the speed of the approaching vehicle: 35, 40, 45, and 50 km/h; (ii) the deceleration behavior of the vehicle: “no deceleration” or “conditional deceleration”; and (iii) the eHMI message displayed by the vehicle: “vehicle is yielding” or “vehicle is not yielding”. The experiment was organized into two distinct experimental blocks: (1) a block without a reward or incentive: no reward was offered to the participants during this block, and they were instructed to behave naturally; and (2) a block with a reward or incentive: a reward was presented to the participants (refer to Section 2.4 for further details).

Within each experimental block, the vehicle approached the pedestrian at the four different speeds mentioned earlier. Half of the trials were in the “conditional deceleration” condition, where the vehicle’s behavior was interactive, i.e., depending on the participant’s movement, the vehicle would either (i) decelerate until it stopped or (ii) not decelerate (refer to the “normal behavior” branch in Figure 2). The other half of the trials were in the “no deceleration” condition, in which the vehicle’s behavior was independent of the participant’s movement, even if it could result in an accident (refer to the “manipulated behavior” branch in Figure 2). This aspect aimed to assess the pedestrian’s reactions when exposed to a high volume of non-decelerating vehicles, both with and without an eHMI. Regarding the eHMI feature, half of the trials included an active eHMI, while the other half had an inactive eHMI (refer to Figure 2). The inclusion of a “no eHMI” condition was designed to evaluate the pedestrian’s response to an approaching AV solely based on kinematic cues. This design choice allows for a comparison with the research conducted by Lee et al. [6].

Overall, the study included four conditions presented at four different approaching speeds, randomly ordered, and resulting in a total of 16 experimental conditions. To ensure robust data, each participant underwent four repetitions for each condition, resulting in a total of 64 trials per experimental block for each participant.

2.3.1. eHMI Design and Messages

In this study, two distinct eHMI designs were employed, each associated with a different message. The first design featured a green-light band, which was associated with the message “vehicle is yielding”. Conversely, the second design incorporated a red-light band, indicating the message “vehicle is not yielding”. The choice to use these colors was primarily influenced by their familiarity as they are commonly used in traffic lights and have been implemented in previous eHMI designs [1,19].

To ensure that participants understood each eHMI design, the researchers provided prior information regarding their respective meanings. This information was presented to each participant before they began the practice session, ensuring they were aware of the intended message associated with each eHMI design. The green- and red-light bands were positioned near the headlights of the vehicle, using static textures sourced from the Blender engine library (see Figure 3).

2.3.2. Vehicle Behavior

Figure 4 provides a comprehensive overview of the crosswalk in one of the virtual environments, and a depiction of the approaching vehicle can be observed in Figure 1. The other crosswalk maintains similar measurements and features.

The following is a summary of the elements highlighted in Figure 4: (A) participant starting point: positioned three meters away from the road, this marks the initial position where the participant begins; (B) detection point: located one meter away from the road, this point serves as the location where the participant was detected by the vehicle. For non-decelerating trials, the vehicle proceeds without stopping. In trials with “conditional deceleration”, the vehicle’s behavior depends on its distance to the crosswalk; (C) the crossing point: this represents the section where the participant begins crossing the road; and (D) the end crossing point: this designates the conclusion of the crossing, after which participants return to the starting point for a new trial.

In each trial, the vehicle starts moving as the participant passes the starting point (A), with the vehicle consistently beginning its approach from a fixed distance of 50 m. In non-decelerating trials, the vehicle maintains a constant speed (35, 40, 45, or 50 km/h), passing the participant without stopping. The activation of the eHMI, if applicable, signaling yielding or non-yielding behavior, occurs when the participant crosses the detection point (B).

For “conditional deceleration” trials, the vehicle’s deceleration is dependent on its distance to the crosswalk at the detection point (B). If a calculation confirms that the vehicle can come to a stop at least 5 m away from the pedestrian, a gentle and constant deceleration of 2.4 m/s² begins [4]. If not, no deceleration occurs. The activation of the eHMI in “conditional deceleration” trials is influenced by the participant’s precise timing passing the detection point.

Points C and D mark the participant crossing point and the end of the crossing, respectively. After passing Point D, participants return to the starting point for a new randomly generated trial. This experimental design aims to simulate realistic and dynamic scenarios, namely the vehicle’s behavior and eHMI activation procedure, allowing for a thorough investigation of participant responses and reactions in various conditions.

2.4. Experimental Procedure

The participants entered the simulator room, where they received an introduction to the apparatus. Before initiating the experiment, they were asked to review and sign an informed consent form outlining the task details. The researchers then explained the task, which involved interacting with a virtual AV. Participants were instructed to walk in the direction of the crosswalk while the virtual vehicle approached at varying speeds. Their task was to decide whether to cross based on their judgment of safety. The researchers emphasized that they should behave naturally as they would when crossing an actual road, and imagine themselves in a scenario where they were running late for an important meeting, thus eliciting a sense of urgency to reach their destination.

Additionally, participants were informed about the unique frontal display on the vehicle, featuring two light bands, one green and one red, each signifying specific meanings. The green light indicated that the vehicle was yielding, while the red light indicated non-yielding behavior. Participants were explicitly informed of these meanings before the experiment, with the researchers reinforcing this information verbally during the practice session.

In addition, participants were made aware that there would be instances when the display would not be present, and the vehicle might not yield. They were explicitly instructed not to run or stop in the middle of the road. Once a decision to cross was made, participants were required to continue walking, even if it meant a potential collision. After the task explanation, the participants were allowed to ask any questions they had before proceeding.

Before the main experiment, participants underwent a brief practice session to acquaint themselves with the virtual environment and understand the task. They completed two questionnaires, one before the experiment and another afterward (refer to Section 2.4.1 for more details). During the practice session, participants encountered various conditions with different combinations of eHMIs. They were informed that virtual accidents, such as collisions, might occur if they decided to cross but failed to reach the other side safely in non-decelerating trials. In such cases, they would hear a crashing sound to indicate that they had been hit by the vehicle.

The experiment consisted of two blocks and participants were informed that each block would take approximately 15 to 20 min to complete, with a short break in between. They were instructed to fill out a simulator sickness questionnaire before and after both experimental blocks (refer to Section 3.5.3). After completing the first block, participants were informed, at that point only, about the difference in the second block. Verbal communication from the researchers conveyed the following information: “We will now proceed with the second part of the experiment. Similar to the previous one, there is one distinct aspect to this segment. You will engage in a competitive game with other participants in the study. At the study’s conclusion, the participant with the highest score will be awarded a prize. You will accumulate one point for each safe crossing and three points for each risky crossing. A safe crossing is defined as crossing the road when the vehicle is yielding, while a risky crossing occurs when the vehicle is not yielding. However, be aware that if a crash occurs, we will deduct five points from your overall score. If you have no further questions, feel free to begin whenever you are ready”.

2.4.1. Questionnaires

Before starting the experiment, participants were tasked to complete the demographics questionnaire (refer to

Table 1. First questionnaire (answered before commencing the study).

	Question	Possible Answer
1	Your sex	Female; male
2	Your age	Secondary education; bachelor’s degree; master’s degree; doctor’s degree
3	Your education level	Not at all familiar = 1 to extremely familiar = 5
4	How familiar are you with the concept of autonomous vehicles?	Strongly disagree = 1 to strongly agree = 5
5	Imagine a scenario where autonomous vehicles coexist with conventional vehicles on the road. Now, as a pedestrian, please indicate your level of agreement with each of the following statements.
(a)	I feel comfortable to cross the road in front of a vehicle which is in autonomous mode.
(b)	I expect the autonomous vehicles to be better at identifying me than a human driver.

) and the first portion of the simulator sickness questionnaire [58]. Questions 4 and 5 from the first questionnaire were used to assess participants’ familiarity with and trust in AVs, respectively. Familiarity with AVs generally refers to individuals’ understanding of how AV technology works, their level of comfort or experience interacting with AVs, and their knowledge of AV capabilities and limitations. While being able to visually identify AVs on the road may be one aspect of familiarity, it is not the single or most critical factor. In dense urban traffic, especially with buses and large passenger vehicles, relying solely on visual recognition may not be practical. Subsequently, upon the conclusion of the study, participants were requested to respond to a second questionnaire, specifically addressing their experiences with street-crossing and interactions with AVs equipped with eHMIs in the simulated environment (refer to

Table 2. Second questionnaire (answered after completing the study).

	Question	Possible Answer
A	As a pedestrian, how important do you believe it is for autonomous vehicles to be easily distinguishable from conventional human-driven vehicles?	Not important = 1 to extremely important = 5
B	As a pedestrian, how significant do you think it is for autonomous vehicles to visually communicate their intentions (such as about to yield or not stopping) to pedestrians?	Not important = 1 to extremely important = 5
C	What was the most crucial factor you considered when deciding whether to cross the road or not as a pedestrian?	Open answer

). The final question asked participants to identify the factors they considered to be most important when crossing the road. The structure of the questionnaires was developed drawing on the existing literature, particularly the study conducted by Razmi Rad et al. [19].

2.5. Statistical Analysis

Four key behavioral measures were obtained from participants’ street-crossing behavior: accepted crossings, collisions, safety margin, and crossing speed. The statistical analysis employed Linear Mixed Models (LMMs) and Generalized Linear Mixed Models (GLMMs) with a binomial response variable link using the logit function [59]. The choice of LMMs was motivated by violations of the assumption of independent observations. The significance level was set at 0.05, and post hoc tests were conducted with Bonferroni corrections when appropriate.

LMMs and GLMMs are widely utilized in the analysis of behavioral measures in studies employing simulated environments, as demonstrated in previous research [19,38,48]. These experimental setups often involve nested data structures, with participants nested within conditions or trials. LMMs and GLMMs are particularly suitable for modeling such hierarchical data, accommodating multiple observations within each participant or group. Given that participants may undergo repeated measures or experience various conditions, these modeling techniques excel in accounting for within-subject correlations and dependencies in the data. One of the key advantages of LMMs and GLMMs is their ability to model both fixed effects (such as experimental conditions) and random effects (such as participant variability). This flexibility allowed for the accounting of both the systematic and random sources of variation in the data, improving the accuracy of the statistical analysis. Additionally, LMMs and GLMMs can capture non-linear relationships between predictors and response variables, accommodating both categorical and continuous predictors. This feature allows for the exploration of complex behavioral patterns that may arise in simulated experimental conditions.

The fixed effects considered for the analysis included: (i) manipulated variables from the experimental design: experimental block (1/2), vehicle speed (35/40/45/50), and eHMI status (Off/On); and (ii) participants’ personal characteristics: age, sex (F/M), education level (1/2/3/4), familiarity with AVs (1/2/3), and trust in AVs (1/2/3). In the case of familiarity with AVs and trust in AVs, the factors were transformed from five levels to three levels to simplify the model.

The modeling process was guided by key metrics such as the Akaike Information Criterion (AIC), the Bayesian Information Criterion (BIC), the Intraclass Correlation Coefficient (ICC), Root-Mean-Square Error (RMSE), and conditional R². The assessment involved a Likelihood Ratio test using ANOVA to establish the statistical significance of introducing new variables.

To address the non-independence of observations, random effects were incorporated for the subjects. The verification of LMM assumptions was conducted, ensuring linearity, normality, homoscedasticity, no collinearity among fixed effects, and the absence of influential data points. As for GLMMs, the normality of the random effects and multicollinearity were assessed. All these assumptions were confirmed to be satisfied. The variance inflation factor was calculated for all predictors, revealing no significant correlation between the predictor variables in the models.

Model enhancement involved tests to identify relevant interactions between the factors. Focusing primarily on interactions between two factors, following the principle that simpler models are often preferable, decisions to retain interactions were based on three criteria: (i) statistical significance, (ii) improvement in model performance, and (iii) the avoidance of excessive complexity.

To address the limitation of the random intercept model, which assumes consistent behavior across all conditions for each subject, the analysis was refined by considering the incorporation of random slopes. To maintain simplicity in the random effects structure, the inclusion of random slopes was limited to one particular variable: the experimental block. This adjustment acknowledged that individual subjects may exhibit varying behavioral measures under different experimental block circumstances, leading to a more realistic representation of their behavior.

3. Results

A total of 3968 trials were conducted (31 participants × 64 trials each × 2 experimental blocks). Data recording issues resulted in the exclusion of 24 trials from the first block and 20 trials from the second. Therefore, a total of 3924 viable trials were used for the analyses, with 1960 trials in the first block. The experimental design generated 624 viable trials for the decelerating vehicle, with 280 trials in the first block, and 3300 viable trials for the non-decelerating vehicle, with 1680 trials in the first block.

3.1. Accepted Crossings

3.1.1. Non-Decelerating Trials

The rate of accepted crossings was calculated by dividing the number of crossings by the total number of trials, considering only the non-decelerating trials. In Figure 5, the rate of accepted crossings in each experimental block is depicted as a function of vehicle speed and eHMI status. For the decelerating trials, all participants successfully crossed the road, as mentioned in Section 3.1.2.

The initial phase of the analysis focused on determining the model that exhibited the best fit to the data, following the procedure described in Section 2.5. Factors contributing to the model’s enhanced performance included the experimental block, vehicle speed, eHMI status, and participant familiarity with AVs. Incorporating random slopes for the experimental block resulted in an improved model fit, as indicated by improved performance indices. While this choice may not be the most parsimonious, it did not lead to an over-fitted model. Notably, no significant interactions were identified between the factors. A summary of the model results is provided in

Table 3. Results of the GLMM estimation for the accepted crossings in the non-decelerating trials.

Predictors	Coefficient	Std. Error	p-Value
Fixed effects
Intercept	1.669 *	0.841	0.047
Block = 1	– a	– a	– a
Block = 2	1.080 **	0.358	0.003
Speed = 35	– a	– a	– a
Speed = 40	−2.158 ***	0.177	<0.001
Speed = 45	−3.599 ***	0.194	<0.001
Speed = 50	−4.833 ***	0.222	<0.001
eHMI = Off	– a	– a	– a
eHMI = On	0.239 *	0.109	0.028
Familiarity = 1	– a	– a	– a
Familiarity = 2	−0.756	0.957	0.429
Familiarity = 3	−3.601 *	1.549	0.020
Random effects
σ2id	4.84
σ2slope	3.34
Correlation	−0.26
Observations	3300
Groups	31
R2	0.739
RMSE	0.314
AIC	2329.9
BIC	2397.0
ICC	0.624

^a Reference level; * p < 0.05; ** p < 0.01; *** p < 0.001; σ²_id = random intercept variance; σ²_slope = random slope variance.

.

3.1.2. Decelerating Trials

In the decelerating trials, all participants successfully crossed the road. Figure 6 illustrates the distinct patterns observed for conditions with the eHMI present, signaling yielding behavior using the green-light band, and the eHMI absent. The estimation of the distance between the vehicle and the pedestrian occurred when the participant initiated the crossing point. Additionally, the average distance at which the eHMI was activated was determined based on when the pedestrian passed the detection point.

For experimental block 1 and experimental block 2, the average activation distances of the eHMI were calculated to be 31.8 m (SD = 2.4) and 33.2 m (SD = 2.5), respectively. This suggests that when the eHMI was active and conveying yielding information, the incentive to cross the road resulted in participants approaching the crosswalk at a faster pace.

In both experimental blocks, the crossing pattern remained largely consistent. Some participants opted to wait for the vehicle to fully stop before crossing when the eHMI was not active. However, this behavior shifted when the eHMI was active, suggesting that the message conveyed by the vehicle facilitated a quicker and safer decision for crossing. This pattern also implies that pedestrians felt more at ease crossing the road either when the vehicle was at a considerable distance, in both absent and present eHMI conditions, or when they perceived the yielding behavior from the vehicle, particularly when it signaled its intention to yield. These observations align with similar findings from a study conducted by Lee et al. [6].

3.2. Collisions

During the experiments, any virtual contact detected between the vehicle and the participant was considered a collision. To ensure participants were aware of these collisions, a sound was played every time the width of the vehicle’s front bumper reached them. The collision rate was determined by dividing the number of collisions by the total number of accepted crossings. This phenomenon exclusively occurred in the non-decelerating trials. Figure 7 illustrates the collision rate divided by experimental block, as a function of vehicle speed and eHMI status.

This analysis included a total of 1016 trials, with 434 trials in the first block. One participant exhibited an overly conservative approach by never attempting to cross the road before the vehicle reached the crosswalk and was not included in the analysis. In line with the approach described in Section 3.1.1, a similar methodology was applied to fit a GLMM. Notably, the factors that exhibited superior performance in the model were the experimental block and vehicle speed. Despite the eHMI variable being statistically non-significant, the decision was made to retain this factor in the model for further consideration. The Std. Error increases at higher speeds due to the number of trials where collisions occur being smaller at higher speeds. This reduction in observations for collision incidents and the inclusion of random effects leads to less precise estimates of the Std. Error for the fixed effects, resulting in larger values. Random slopes were considered for the experimental block, and no relevant interactions were observed between the factors. The results of the model are summarized in

Table 4. Results of the GLMM estimation for the collisions in the non-decelerating trials.

Predictors	Coefficient	Std. Error	p-Value
Fixed effects
Intercept	−10.802 ***	2.844	<0.001
Block = 1	– a	– a	– a
Block = 2	−12.477 **	4.578	0.006
Speed = 35	– a	– a	– a
Speed = 40	0.294	1.396	0.833
Speed = 45	4.258 *	1.805	0.018
Speed = 50	12.957 ***	3.762	<0.001
eHMI = Off	– a	– a	– a
eHMI = On	−0.038	0.748	0.960
Random effects
σ2id	81.56
σ2slope	376.66
Correlation	−0.26
Observations	1016
Groups	30
R2	0.989
RMSE	0.081
AIC	148.7
BIC	193.0
ICC	0.987

^a Reference level; * p < 0.05; ** p < 0.01; *** p < 0.001; σ²_id = random intercept variance; σ²_slope = random slope variance.

.

3.3. Safety Margin

The safety margin was calculated for each crossing in the non-decelerating trials, with the exclusion of the collision data. This metric was determined as the time between the moment the participant passed the path of the approaching vehicle (front end width of the vehicle) and when the vehicle reached the participant’s crossing line. Essentially, it represents the “time left before a collision” during the crossing, as described by Dommes et al. [36]. Figure 8 illustrates the safety margin as a function of vehicle speed and eHMI status and is divided by experimental block.

The statistical analysis was conducted using LMMs, following the outlined procedure in Section 2.5. Two participants were excluded: one who exhibited very conservative behavior and never attempted to cross before the car reached the crosswalk, and the other who was involved in a collision during every attempt to cross. The variables that demonstrated better performance in explaining the safety margin were the experimental block, vehicle speed, eHMI status, and participants’ age. No relevant interactions were identified between these factors, and random slopes were considered for the experimental block. The results of the model are summarized in

Table 5. Results of the LMM estimation for the safety margin in the non-decelerating trials.

Predictors	Coefficient	Std. Error	p-Value
Fixed effects
Intercept	2.315 ***	0.373	<0.001
Block = 1	– a	– a	– a
Block = 2	0.272 ***	0.038	<0.001
Speed = 35	– a	– a	– a
Speed = 40	−0.475 ***	0.014	<0.001
Speed = 45	−0.828 ***	0.016	<0.001
Speed = 50	−1.095 ***	0.021	<0.001
eHMI = Off	– a	– a	– a
eHMI = On	0.042 ***	0.011	<0.001
Age	−0.041 **	0.012	0.002
Random effects
σ2id	0.08
σ2slope	0.03
Correlation	0.36
Observations	975
Groups	29
R2	0.904
RMSE	0.171
AIC	−394.8
BIC	−341.1
ICC	0.790

^a Reference level; ** p < 0.01; *** p < 0.001; σ²_id = random intercept variance; σ²_slope = random slope variance.

.

3.4. Crossing Speed

The crossing speed was calculated by dividing the crossing distance of three meters by the time taken to cross the road. The data comprised both non-decelerating trials, with collisions excluded from the analysis, and decelerating trials where participants crossed the road before the vehicle reached the crosswalk. Figure 9 shows the crossing speed as a function of vehicle speed and eHMI status, divided by experimental block, for both yielding and non-yielding conditions.

The statistical analysis was conducted using LMMs, following the procedure outlined in Section 2.5. The variables that demonstrated strong performance in the model were the experimental block, vehicle speed, and eHMI status. Particularly, the pace at which participants crossed the road might be influenced by additional factors intrinsic to the experiment. Given that the data encompassed both yielding and non-yielding vehicles, the yielding behavior of the vehicle was also modeled as a fixed effect. The factors representing participants’ personal characteristics did not contribute significantly to the model’s improvement.

The apparent ineffectiveness of eHMI in influencing crossing speed, as seen in Figure 9, might originate from pedestrians relying more on their perception of vehicle speed than the information conveyed by the eHMI. Moreover, the time required for pedestrians to process and respond to eHMI information may be longer compared to directly perceiving the vehicle’s speed. Additionally, pedestrians’ familiarity and experience with eHMI-equipped vehicles could also impact their responsiveness to eHMI signals; however, these factors were not significant in the model.

Random slopes were considered for the experimental block. The results of the model are summarized in

Table 6. Results of the LMM estimation for the crossing speed in both decelerating and non-decelerating trials.

Predictors	Coefficient	Std. Error	p-Value
Fixed effects
Intercept	1.378 ***	0.027	<0.001
Block = 1	– a	– a	– a
Block = 2	0.076 ***	0.020	<0.001
Speed = 35	– a	– a	– a
Speed = 40	0.066 ***	0.008	<0.001
Speed = 45	0.175 ***	0.012	<0.001
Speed = 50	0.280 ***	0.014	<0.001
eHMI = Off	– a	– a	– a
eHMI = On	0.013	0.009	0.125
Yield = No	– a	– a	– a
Yield = Yes	−0.077 ***	0.011	<0.001
eHMI = Off × Yield = No	– a, b	– a, b	– a, b
eHMI = On × Yield = Yes	−0.028 *	0.014	0.040
Random effects
σ2id	0.02
σ2slope	0.01
Correlation	0.13
Observations	1632
Groups	31
R2	0.708
RMSE	0.132
AIC	−1664.1
BIC	−1599.3
ICC	0.611

^a Reference level; ^b post hoc analysis; * p < 0.05; *** p < 0.001; σ²_id = random intercept variance; σ²_slope = random slope variance.

. A significant interaction was observed between the eHMI status and the yielding behavior of the vehicle. The results of the post hoc analysis of the interaction are presented in

Table 7. Post hoc analysis of the interaction between eHMI status and yielding behavior of the vehicle.

Group	Contrast	Coefficient	Std. Error	p-Value
Yield = No	eHMI = Off	– a	– a	– a
	eHMI = On	−0.013	0.009	0.750
Yield = Yes	eHMI = Off	– a	– a	– a
	eHMI = On	0.015	0.011	0.942
eHMI = Off	Yield = No	– a	– a	– a
	Yield = Yes	0.077 ***	0.011	<0.001
eHMI = On	Yield = No	– a	– a	– a
	Yield = Yes	0.106 ***	0.011	<0.001

^a Reference level; *** p < 0.001.

(t-tests estimated using Bonferroni’s method).

3.5. Questionnaires

3.5.1. First Questionnaire

Participants were asked to indicate their level of familiarity with AVs on a scale ranging from 1 (not at all familiar) to 5 (extremely familiar). The results showed that nearly half of the participants (45%) reported being slightly familiar with the concept of AVs, while 23% stated that they were not familiar with it at all. On average, participants’ familiarity with AVs was moderate (M = 2.2, SD = 1.0).

Subsequently, participants were presented with two questions to assess their trust in AVs. The first question gauged their comfort level when crossing the road in front of an AV, and the second question explored their belief in whether AVs could better identify them compared to a human driver. Both questions were answered using a scale that ranged from 1 (strongly disagree) to 5 (strongly agree). The results revealed that, on average, participants remained neutral regarding their comfort level when crossing the road in front of an AV (M = 3.2, SD = 1.1). However, they leaned towards the belief that AVs would outperform human drivers in identifying them (M = 3.5, SD = 1.1) on the same scale.

3.5.2. Second Questionnaire

Participants were surveyed about their opinions on specific aspects related to AVs through two key questions, with responses rated on a scale from 1 (not important) to 5 (extremely important). The first question addressed whether AVs should be distinguishable from conventional human-driven vehicles. Results indicated that participants deemed this aspect to be very important (M = 3.6, SD = 1.1), suggesting a belief that AVs should have a distinct appearance from traditional vehicles on the road.

The second question focused on whether AVs should visually communicate their intentions to pedestrians. Participants assigned high importance to this feature (M = 4.4, SD = 0.8), emphasizing the need for AVs to clearly indicate their actions to enhance pedestrian understanding and safety. These findings align with a study by Razmi Rad et al. [19], which reported similar results in their survey questionnaires.

To explore potential relationships between participants’ opinions and their behaviors towards AVs, a Spearman’s correlation analysis was conducted. The analysis revealed a positive correlation (r_S = 0.491, p = 0.005) between participants who felt more comfortable crossing the road in front of AVs and their expectation that AVs would excel at identifying pedestrians compared to human-driven vehicles. This suggests that higher comfort levels with AVs may be associated with greater trust in their ability to detect pedestrians accurately.

Conversely, participants who felt less comfortable crossing the road in front of AVs showed a negative association (r_S = −0.383, p = 0.034) with the belief that it is important for AVs to be distinguishable from conventional vehicles. This implies that individuals less at ease with AVs may find visual distinctiveness a crucial factor in feeling safe and comfortable around them.

However, no significant correlation was found between participants’ opinions and their level of familiarity with AVs, indicating that familiarity did not significantly influence their opinions on the evaluated features of AVs.

Additionally, participants were asked about the most important criterion they considered when deciding to cross the road. This was an open answer, and the purpose was to avoid any potential bias that might arise from providing predefined options. Participants indicated that the speed at which the vehicle is approaching plays a crucial role in their decision process, mentioned 20 times in various responses. Participants also emphasized the significance of the vehicle’s sound and communication, with only three participants pointing out the communication capabilities of the vehicle. In a study by Razmi Rad et al. [19], participants considered the distance of the vehicle as the most important criterion. The deviation in criteria between the present study and the mentioned one may stem from the differing experimental environments. The current study was conducted in an immersive setting, while the referenced study took place in a computer-based setup. The contrasting nature of these environments could contribute to the observed differences in participant responses.

3.5.3. Simulation Sickness Questionnaire

The simulation sickness questionnaire was administered to participants both before the experiment (as a baseline) and after the experiment. Comprising 16 symptoms, participants rated the discomfort they experienced during the simulation, ranging from none (zero) to severe (three). These symptoms were categorized into three groups: nausea, oculomotor disturbance, and disorientation, following the classification by Kennedy et al. [58].

In addition to the recruited participants, the first two participants who assisted in calibrating the equipment were also included in the analysis, bringing the total number of participants to 33. The average total score, calculated using the method outlined by Deb et al. [60], was 12.6, indicating the presence of “significant symptoms” during the experiment. Scores above 20 suggest potential simulator issues, while scores below 10 indicate minimal symptoms. Notably, participants were required to walk continuously for a total of 40 min, and the symptoms that scored the highest were “fatigue” and “sweating”, which were expected given the nature of the experimental design. The third highest-scoring symptom was “eyestrain”, which was also anticipated considering the demands of the experiment.

Based on these findings, it can be concluded that the current experimental design is safe to use. However, caution is advised against increasing the duration of the experiment in future studies, as a more extended exposure to the simulator could likely exacerbate the highlighted symptoms. These conclusions align with those reported by Deb et al. [60], where a thirty-minute exposure to the virtual environment resulted in minimal simulation sickness, while a duration of forty minutes or more led to significant simulation sickness.

4. General Discussion and Conclusions

One of the objectives of the present study was to explore the contribution of eHMIs to pedestrian safety in street-crossing scenarios. Another objective was to assess the influence of an experimental reward system on pedestrians’ risk-crossing behavior in the presence of AVs. Ultimately, the researchers expected to identify the factors that contributed to the observed changes in pedestrian behavior during interactions with AVs equipped with external HMIs.

4.1. Vehicle Interaction with and without eHMIs

The presence of an eHMI conveyed a notable influence on risky crossing behavior. Participants were observed to cross more frequently when the vehicle communicated that it was not stopping. This behavior implies that pedestrians were more inclined to take risks when the vehicle signaled an intention not to stop, compared to situations where there was no communication with the pedestrian. Interestingly, this behavior resulted in fewer collisions, although the model found this to be statistically non-significant. Additionally, the safety margin exhibited a slight increase when the eHMI was active and conveyed that the vehicle was not stopping, in contrast to situations where it was absent.

The observed increase in the safety margin could be linked to participants crossing faster when the eHMI was active and the vehicle was not stopping. However, this was proven to be non-significant in the model. Notably, pedestrians crossed faster when the vehicle was yielding, irrespective of the eHMI status, suggesting that the presence or absence of the eHMI did not significantly impact pedestrian crossing speed, as revealed in Table 7.

4.2. Street Crossing with and without an Incentive

The introduction of a reward system with game-related elements notably influenced participants’ behavior during the experiment. The accepted crossings exhibited an increase from the first to the second block, suggesting that participants were more motivated to cross when provided with an incentive. Moreover, fewer collisions were observed in the second experimental block, indicating intensified participant attention during this phase.

The incentive to cross the road also had effects on other variables. The safety margin displayed an increase from the first to the second block, implying a potential risk reduction when pedestrians were motivated to cross. However, the crossing speed also increased during this period, which might have influenced the observed higher safety margins.

An increase in crossing speed was also reported in the study by Schneider et al. [48] under game-related comparable experimental conditions. In contrast, in their study, there was a reduction in the safety margin and an increase in collisions from the first block to the second block. Their experiment involved continuous traffic flow with multiple vehicles on the road with a permanent speed of 30 km/h and increasing gaps between vehicles over time. Participants faced time pressure as points were deducted in the game scenario for prolonged waiting, thereby encouraging them to accept the closest possible gap. In this scenario, the potential point deduction for waiting, as opposed to the higher point deduction for collisions, created a more pronounced time pressure compared to the present study.

4.3. Factors That Contributed to Behavioral Change

The current observations suggest that both the eHMI of the vehicle and the incentive to cross using game-related elements have influenced pedestrian behavior in the simulator. Additionally, the speed of the approaching vehicle has emerged as a crucial factor impacting crossing behavior, with higher vehicle speeds associated with a lower crossing rate. Moreover, vehicle speed significantly impacted collisions and safety margins, with higher speeds correlating with a higher frequency of collisions and shorter safety margins. This underscores the pivotal role of vehicle speed in pedestrian safety during street-crossing situations.

This aligns with the expectation that higher speeds are linked to shorter safety margins and more collisions, as observed in the present study, and supported by previous research [42]. Participants also ranked vehicle speed as the most significant criterion for deciding when to cross the road in the questionnaires (refer to Section 3.5.2). The pattern of risky behavior with high-speed approaching vehicles is consistent with findings from prior studies [36,39,61,62].

The speed of the approaching vehicle also influenced pedestrians’ crossing speed, with participants increasing their crossing speed as the speed of the approaching vehicle increased. However, this pattern may not hold for older pedestrians, as observed in other studies where crossing speed is considered an important predictor of risky behavior among the elderly [42].

Participants’ familiarity with AVs and age also appeared to influence crossing behavior. Those more familiar with AVs crossed less frequently in the non-yielding conditions, revealing that a better understanding of the technology led to more cautious behavior. Familiarity with AVs has played a crucial role in other comparable studies, such as the one by Razmi Rad et al. [19]. Additionally, the safety margin was reduced for older participants, suggesting a higher risk of road-crossing for this age group. Similar findings were observed in the study by Pala et al. [42], where reduced safety margins were noted for older participants compared to younger ones.

Another fundamental aspect of understanding behaviors towards AVs involves examining pedestrian acceptability and the willingness to pay for shared AV services. While these topics were not the focus of this study, it is essential to recognize their significance in transportation research. Investigating pedestrian attitudes and perceptions towards AVs, their trust in AV technology, and their willingness to use shared AV services provides valuable insights into societal acceptance and potential implications for future mobility services [63,64].

Based on these findings, a new set of experiments should be conducted with a larger and more diverse sample of participants, encompassing a wider range of ages. Emphasizing the role of participants’ age in road-crossing behavior could provide valuable insights into how different age groups react to varying scenarios with AVs on the road. Figure 10 illustrates the outline of the factors that contributed to the change in pedestrian behavior.

4.4. Conclusions

This study employed an experimental approach to investigate the impact of incentives and the use of eHMIs on pedestrian interactions with AVs within virtual environments, considering the challenges and ethical considerations associated with real-world experiments. Several significant factors influencing pedestrian behavior were identified, including vehicle speed, the deceleration behavior of the vehicle, eHMI status, familiarity with AVs, and age. However, gender, trust in AVs, and education level did not emerge as significant factors. The presence of an external HMI had a significant effect on crossing behavior, particularly in measures associated with risky behavior. Participants were more inclined to cross when the eHMI indicated that the vehicle was not stopping, suggesting a willingness to take risks. The safety margin also exhibited a slight increase when the eHMI was present and the vehicle was not stopping.

The introduction of a reward system during the game-based experimental block led to a noticeable increase in the number of accepted crossings, particularly at higher vehicle speeds. Although the incentive encouraged participants to take more risks, the actual risk involved decreased, indicating that the reward system influenced risk-taking behavior without compromising safety significantly. Overall, the experimental approach significantly altered participants’ behavior in all of the behavioral measures: accepted crossings, collisions, safety margin, and crossing speed.

The speed of the approaching vehicle emerged as a crucial factor influencing crossing behavior, with higher vehicle speeds associated with a lower crossing rate. Additionally, higher vehicle speeds correlated with a higher frequency of collisions, emphasizing the role of vehicle speed in pedestrian safety during road-crossing situations. The speed of the approaching vehicle also impacted pedestrians’ crossing speed, with pedestrians increasing their speed as the vehicle’s speed increased.

Participants’ familiarity with AVs and age also influenced pedestrian crossing behavior. Participants who were more familiar with AVs crossed less frequently, indicating a more cautious approach. Older participants exhibited a reduced safety margin, suggesting a higher risk of crossing the road. These findings highlight the multifaceted factors influencing pedestrian behavior in the presence of AVs.

Questionnaire data indicated participants’ moderate familiarity with AVs and a somewhat neutral stance on their comfort in crossing in front of these vehicles. However, participants had a positive expectation regarding the ability of AVs to surpass human drivers in identifying pedestrians. They emphasized the importance of AVs being visually distinguishable and communicating intentions clearly, suggesting that addressing these design and communication aspects could enhance the perception of safety and trust in AVs.

The challenges of investigating this topic in real-world settings lead to the decision to conduct experiments in a virtual environment. While controlling variables provided a systematic approach, this study has limitations, namely by simplifying real-world scenarios and, thus, limiting the generality of the findings. Controlled environments allow for precise manipulations and observations of participant behavior, but they may not fully capture environmental, social, and individual factors present in real-world situations. Therefore, interpreting the findings within this context is essential, along with recognizing potential limitations in extrapolating the results to real-world contexts.

Future research could bridge this gap by incorporating more realistic elements into simulated environments, such as virtual reality simulations or semi-controlled field experiments. This approach would comprehensively address the stochastic nature of the topic and enhance the applicability of the findings to real-world scenarios.

The study underscores the significance of eHMIs as an external cue influencing pedestrians’ interactions with AVs. The data support the idea that eHMIs play a crucial role in shaping participants’ responses and behaviors during road-crossing experiences, especially in risk-related factors. These insights contribute to the understanding of the potential impact of external cues on pedestrians’ decisions and actions when crossing roads in the presence of AVs. This is essential for designing effective communication strategies between AVs and pedestrians, ultimately contributing to safer and more efficient vehicle–pedestrian interactions.