Assessment of Driving Proficiency When Drivers Utilize Assistance Systems—The Case of Adaptive Cruise Control

Abstract

Driver assistance systems (ADAS), and especially those containing driving automation, change the role of drivers to supervisors who need to safeguard the system’s operation. Despite the aim to increase safety, the new tasks (supervision and intervention) may jeopardize safety. Consequently, safety officers address the need for specific training on ADAS. However, these tasks are not assessed in driver licensing today. Therefore, we developed a framework to assess in-practice driving proficiency when drivers utilize ADAS. This study evaluated whether the proposed framework is able to identify meaningful differences in driving proficiency between driving with and without assistance. We applied the framework to perform a qualitative assessment of driving proficiency with 12 novice drivers in a field experiment, comparable to a license test. The assistance system concerned Adaptive Cruise Control (ACC). The test showed that driving with ACC has a negative influence on self-initiated manoeuvres (especially lane changes) and sometimes led to improved adaptations to manoeuvres initiated by other road users (like merging in traffic). These results are in line with previous research and demonstrate the framework’s successfulness to assess novice drivers’ proficiency to utilize ADAS in road-traffic. Therewith, the proposed framework provides important means for driving instructors and examiners to address the safe operation of ADAS.

1. Introduction

Today’s vehicles are equipped with a large variety of Advanced Driver Assistance Systems (ADAS). The systems can assist drivers through enhancement of their situational awareness (e.g., blind spot detection warning), monitor driver behaviour (like drowsiness detection), provide additional control of braking (e.g., automated emergency braking), safeguard lateral position on the road (like lane keeping assist), or relieve drivers from manually controlling the vehicle through automating standard control tasks, like keeping distance and speed (e.g., adaptive cruise control). These technological tools may raise comfort [1], may increase traffic safety [2] and may reduce the ecological impact of road transport [3]. Not surprisingly, the amount of vehicles equipped with ADAS is rising and expected to continue its advance [4,5]. In fact, the European Commission decided in 2018 on renewed vehicle safety regulations and mandated the fitting of a set of safety functions for all new vehicles as of 2022 [6]. The mandatory functions include automated emergency braking, speed limiting, and driver distraction warning. These functions encompass similar technology as used for automated driving.

1.1. Effects of ADAS on Driving Performance

Despite the potential benefits of ADAS to assist drivers, ADAS may negatively affect driving behavior—this is especially a risk with ADAS comprising (partly) automation of the driving task [7], like Advanced Cruise Control (ACC) and Lane Keeping Assistance (LKA). This is because the reliability of automating the driving task is constrained to technical restrictions, like the detection of lane markings and gap distances [8]. Due to the large variety of road-traffic circumstances, occasions in which technical conditions are met or not may change often and fast. Making reliable decisions for ADAS’ control has therefore become very challenging [9]. It is difficult for drivers to develop a correct level of reliance in the system, often leading to problems of underreliance or overreliance [10]. Hence, the introduction of automated driving systems in the car does not only raise complexity of the vehicles, it also has a large impact on the required capabilities of drivers [11,12]. Consequently, and contrary to general thoughts, automated driving may not necessarily make the whole driving task easier [13,14,15]. It will only do so within the boundary conditions suitable for automation. Then, the driving tasks and responsibilities of the driver will change from actively operating the vehicle to being a supervisor with the responsibility for safe vehicle operation and occasionally the necessity to intervene [16,17]. An example is if the system is not able to detect lane markings. Then, the driver is required to act and retake control. The need to retake control may occur suddenly and requires fast responses. Intervention is therefore a difficult task, causing a high workload [18,19]. Then, drivers often fail to regain effective vehicle control [12]. Even experienced drivers often lack the skills to use their assistance systems effectively and safely [20].

One of the main underlying factors influencing the aforementioned issues is the driver’s mental model of ADAS [21,22]. An incorrect mental model hinders drivers in accurately understanding the situation. It may create confusion about the automation state (i.e., mode confusion) and about any actions the driver is required to make [23,24,25]. A consequence may be that a driver refrains from taking control while this is actually necessary. Mode confusion (or loss of mode awareness) has been a long identified problem in human-machine interaction research, and has led to accidents within, for example, aviation [26,27], but also in partially automated vehicles [12,28,29]. Unfortunately, several studies indicate a lack of awareness and understanding of drivers about ADAS in their car [30,31,32]. While systems’ limitations might not be known to users beforehand, experiencing them in real traffic easily leads to dangerous situations [33].

1.2. Need to Address ADAS in Driver Training and Examination

The presented issues explain how ADAS creates a gap between drivers’ abilities to interact with ADAS-technology and their responsibility for its safe operation. To reduce this gap, safety officers address the need for specific training [12,34]. There is, however, a notable lack of insights into the requirements for such training and its examination [35,36,37,38,39]. Moreover, hardly any methods for ADAS training exist [40] and insights into the effectiveness of ADAS training is scarce [41]. Furthermore, there are no specific assessment criteria towards using ADAS [42].

1.3. Current Approach to Test Driving Proficiency While Using ADAS

In 2003, Engström et al. provided an extensive overview of methods and models of how novice drivers learn to drive [43]. Despite the substantial advances in vehicles’ technology and functions, training programs have not really changed [44]. Indeed, the utilization of ADAS is hardly addressed in current driver models [45]. In practice, a common method to train for using ADAS appears to be trial-and-error [20,32,46,47]. Obviously, this easily leads to unsafe situations.

As an example, the current approach in the Netherlands for testing driving proficiency is used in this work. It comprises a theory test and a practical driving test in real traffic. This approach is in line with Directive 2006/126/EC from the European Commission [35] and is similar to testing practices in Sweden and Germany [48]. During the practical test, the main focus is on the assessment of tactical tasks, like negotiating other traffic, appropriate adaptions to speed and position, and anticipating road user’s behaviour. All tasks are a subset of the main objective to safely participate in traffic [49]. There is no obligation to use specific safety features during the practical driving test. On a voluntary basis, candidates are allowed to use ADAS (except automated parking) during the driving test [50]. If used, they will be judged on applying and controlling these systems in relation to a proper performance of the driving task. The federal body for driving examination (CBR, in Dutch: Centraal Bureau Rijvaardigheidsbewijzen) judges the overall result to safely take part in traffic. This appraisal is given independent of the question of whether a candidate used ADAS or not [51].

1.4. Assessment Concept

In order to assess driving proficiency that includes utilization of assistance systems (i.e., ADAS), we developed “D-Brief”: A Driver Behaviour Rating Index and Evaluation Form. D-Brief consists of 40 behavioural aspects that are considered characteristic to safe participation in traffic when using an assistance system. The framework is intended for application by driving instructors and examiners. Since self-reflection is an important aspect of today’s driving education [52], D-Brief is also intended for that purpose. Consequently, both experts and laymen should be able to understand the assessment framework and its statements.

1.5. Scope

The aim of this study is to evaluate the quality of our developed framework “D-Brief” (Driver Behaviour Rating Index and Evaluation Form), that assesses driving proficiency while including utilization of assistance systems (i.e., ADAS). In order to evaluate our framework, we tried out D-Brief during a field test with 12 novice drivers and two assessors who acted as examiners. In our test we focused on proficiency in using ACC during highway cruising. The following list defines the main features and boundaries of our project:

• Level of automation—ADAS may consist of different levels of driving automation. In the present study, we primarily focus on SAE-level 2 systems. According to SAE International, these systems allow the automation of longitudinal and lateral control while requiring the driver to remain attentive and available at any time to take back control [53]. Examples of Level 2 systems are ACC for automated longitudinal control and LKA for automated lateral control. The reason for our focus is that these systems are increasingly available on the market, as is equipment of second-hand vehicles.
• Novice drivers—Because of this large uptake, a majority of drivers—including novice drivers—will face the requirements for controlling these systems at any time soon. To remain concise, we specifically focus on ACC used by novice drivers. Novice drivers are often, but not necessarily, young drivers. For practical reasons of our field test we selected drivers under the age of 27 years.
• Project background—The intention is to make assessment of driving proficiency when utilizing ADAS complementary to existing procedures of driving examination. For this purpose, we take the Dutch situation as a reference. Furthermore, we primarily focus on examination for license category B (passenger vehicles). This study is covered in a larger project called Prautocol [54]. Prautocol targets the development of a framework for the certification of both vehicle and driver of autonomous driving technology. In doing so, its aim is to advise on future driving permission procedures. The project is co-financed by the Dutch Research Council NWO.

2. Materials and Methods

2.1. Assessment Framework: D-Brief

D-Brief consists of 29 behavioural aspects that assess safe participation in traffic when using an assistance system. In addition, D-Brief includes statements on driver’s controllability of the assistance-system(s) that are being used. Both are explained below:

• Behavioural aspects for assessment—The generic behavioural aspects are introduced as observational statements and require rating of their occurrence on a 4-point scale, varying from (1) “(Almost) never”; (2) “Occasionally”; (3) “Often”, to (4) “(Practically) always”. Examples of the observational statements are: “Timely decision making to avoid hindrance to others”, or “Adapts behaviour at on and off ramps”.
• Assessment of driver’s controllability of ADAS—The statements that refer to the driver’s control of a particular ADAS require judgement on a 4-point scale, ranging from (1) “Completely disagree”; (2) “Disagree”; (3) “Agree”, to (4) “Completely agree”. In this test we included 11 statements on the controllability of ACC. Examples are “Conscious attention needed for overtaking with ACC” or “User uses ACC-control devices to accelerate or decelerate”.

Appendix B shows an overview of D-Brief with the behavioural aspects (divided over categories A to D) and statements for driver’s controllability of ADAS (category E), adding up to a total of 40 assessment aspects.

The intention of D-Brief is to enhance the current practice of driving examination. In the Netherlands, a learner driver is assessed on multiple aspects summarized in an assessment matrix, see Figure 1 [55]. The matrix plots a diversity of driving ‘sub’ tasks listed in columns (e.g., speed-control or anticipation) against road-traffic situations, listed in rows (e.g., merging into traffic). Assessment of the task execution is based on descriptions of favourable behaviour. It is, for instance, defined as “When merging in it is important that the vehicle’s speed is practically identical to the speed of traffic participants at the main road.” [55]. The assessment is based on a qualitative appraisal of the learner driver’s competences. It records in what situations the candidate successfully demonstrates the desired behaviour. For the development of D-Brief we therefore also decided to rely on the appraisal of driving performance based on predefined descriptions of favourable behaviour.

The next sections explain how D-Brief has been developed. For a detailed explanation of its design and development process we refer to: Van den Beukel et al. [39].

2.2. Development of D-Brief

We used the following steps to develop our assessment framework:

• Compilation of a list with relevant skills and behavioural aspects that a learner driver is required to demonstrate.
• Comparison of the relevance of these behavioural aspects between (a) driving manually and (b) while having the responsibility for supervision of an assistance system (in our case: ACC).
• Selection of behavioural aspects that are characteristic for the driver’s supervisory role.

2.2.1. Relevant Skills and Behavioural Aspects

The first step concerned the compilation of a list with relevant skills and behavioural aspects based on literature and feedback from driving instructors. The complication included theoretical and practical considerations for acquiring driving proficiency. We used literature for vocational training of driving instructors, including the assessment matrix above [55]; The book ‘Mobility and behaviour’ [56]; An interview with a driving school director [57]; Author Veders’ experience as a driving instructor. We also reviewed theories on learning and acquiring (driving) skills, such as Bloom’s learning objectives [58]; Michon’s hierarchical description of driving tasks [59]; Rasmussen’s model for task performance and required effort [60], as well as Groeger’s papers on the acquisition of driving skills [61,62] and skills to interact with ADAS [16,63,64,65]. After taking out duplicate aspects, the review resulted in a list with 49 separate aspects, each addressing a relevant skill or behaviour. We structured this list based upon the learner drivers’ proficiency levels from the handbook for driving instructors, which are: Perception; Understanding; Ability to Act and Motivation to Act [55]. These levels were selected because they are comparable to Bloom’s generic levels of learning objectives [58]. Furthermore, we used Rasmussen’s model for task complexity to provide a further distinction in required level of task-performance, i.e.,: skill-based, rule-based and knowledge-based behaviour. The subdivision of the list with reference to these performance levels led to six categories. See Appendix A. Categories concerned for example: Understanding tactical tasks based on knowledge-based behavior—U.T.K. This category consisted of, for instance: Driver applies appropriate solutions in unfamiliar situations (U.T.K.1) and Driver needs reasoning to understand vehicle status (U.T.K.5). Appendix A presents the complete list compiled in step 1.

2.2.2. Relevance of Behavioural Aspects

Comparison of the relevance of the collected behavioural aspects (step 2) is based on a narrative review. This comparison revealed key competencies in supervising ACC compared to driving manually and is inspired by a standardized tool for assessing executive functions in adolescents to perform tasks, called BRIEF [66]. The main focus of BRIEF is on assessing cognitive processes to conduct desirable behaviour in relation to task performance, workload and efforts. Therefore, the tool fits very well with our purpose to assess drivers’ performance of their supervisory role and to safely control ADAS. The result of this step was the identification of 11 aspects that describe favourable behaviour for controllability of ACC. In step 3 these aspects were brought together in category E of D-Brief.

2.2.3. Selection of Assessment Aspects

The assessment of driving proficiency is generally based on evaluating the learner driver’s behaviour within situations that are representative of the complexity of the driving task. Likewise, assessment of driver’s supervisory skills and ability to control ADAS should be linked to relevant road-traffic scenarios. Within step 3, we therefore selected which behavioural aspects are characteristic for the driver’s role to supervise ACC within our predefined use case: highway cruising. Based on this selection the original list from step 1 was reduced. To remain as comprehensive as possible the categories were also restructured with more concise labels. For instance, the category ‘Ability to Act (operational tasks based on skill-based behaviour)—A.O.S’ changed to: Operational Control. The category ‘Understanding (tactical tasks based on skill-based behaviour)—U.T.S’ changed to: Speed and Anticipation. The restructuring resulted in the following 5 categories:

• Assessment of special manoeuvres (Operational Control), like changing lanes or overtaking. Then, drivers need not only to understand the road-traffic situation itself, but also the system mode and what the system’s capabilities are.
• Assessment to account for the interests of other traffic participants (Speed and Anticipation). ADAS (like ACC) have typically limited contextual knowledge. Safeguarding the interests of other road users through adequate anticipation and speed selection is therefore an important responsibility of the driver.
• Assessment of Basic Perception (Viewing Behaviour). Specific attention should be addressed to the locus of attention. While operating the system typically directs attention to the dashboard and within the vehicle, accounting for the interests of other road users requires attention outside the vehicle.
• Assessment of adequate Perception of Traffic and Road Situations. Whereas category C addresses the locus of attention, category D focuses on behavioural aspects following this viewing behaviour, like timely reacting on brake lights of other road users, keeping a sufficient space cushion around the vehicle and not causing hinderance to other road users when changing lanes.
• Assessment of Driver’s Controllability of the Assistance-System: System understanding in relation to traffic circumstances; understanding of appropriate circumstances in which the system can be applied and how this is influenced by changing road-traffic circumstances.

While further detailing the categories listed above, we drew up D-Brief as depicted in Appendix B.

2.3. Experimental Design

To evaluate D-Brief, a field test was set up. The test assessed D-Brief for its ability to reveal meaningful differences in driving proficiency between driving with and without ACC. The judgement of driving proficiency focused in particular on lane changes at a highway. All tests took place during daytime, after the morning rush hour and while summer holidays have started. The experiment was set up as within subject comparison between manually driving and driving with ACC, while using D-Brief for assessment of driving proficiency. The assessment (with D-Brief) was collected both by expert assessors and participants—the latter in order to try out self-evaluation. Lane position data were acquired to compare the number of lane changes between manual driving and driving with ACC.

2.4. Participants

Twelve participants took part in the test, eight male and four female. Subjects were recruited through announcements at the websites of the University of Twente and HAN University of Applied Sciences. The selection strategy focused on drivers with a maximum age of 27 years. For practical reasons participants needed to understand and speak Dutch. The average age was 23.3 years (SD = 2.53 years). Five participants drove less than 10,000 km per year. Five participants drove between 10,000 and 20,000 km. Only one drove more than 20,000 km per year as this person worked during weekends as taxi driver. All of the participants had heard about ACC. None of the participants owned a vehicle equipped with ACC and none of the participants had a vehicle with ACC regularly at their disposal. However, five participants stated that they knew ACC well and that they used it occasionally. The test took place in July 2019, during the university summer semester. Participation was voluntary and without remuneration. Participants received afterwards a small surprise gift to express appreciation for their time and efforts.

2.5. Variables

The independent variables were Driving Condition (i.e., Manually Driving and Driving with ACC) and the Assessment Method (i.e., Self-evaluation and Expert-assessment). Both driving conditions involved driving at a highway for about 40 min per trial. The highway consisted of two-lane and three-lane sections and taking a connection between two highways. The condition to drive without ACC (Manually Driving) was considered as a reference for the evaluation of driving proficiency when driving with ACC. The dependent variables were (i) Driving Performance, (ii) Drivers’ Control of ACC, and (iii) Number of Lane Changes. The first variable consisted of the 29 items defined for the behavioural assessment aspects of D-Brief. The second variable consisted of the 11 aspects on controllability of ACC (D-Brief Section E). Data for the third variable involved counts of lane changes.

2.6. Expected Observations of Changed Driving Behaviour

To consider D-Brief successful, a base requirement is the ability to distinguish differences in driving performance between manually driving and driving with ADAS (in our case: ACC). For further analysis of the tool’s quality, we collected insight (based on existing studies) about some specific behavioural changes the tool is expected to reveal.

Firstly, we expect drivers with ACC to demonstrate less anticipation to other road users than when driving without ACC. This is because utilization of ACC generally reduces drivers’ situational awareness [7,67].

Secondly, we expect that ACC influences the tactical behaviour of performing lane changes on a motorway. The ACC supplies longitudinal control and therewith influences the driver’s lateral control. Such influences may be witnessed by delayed decision making and less fluent lane change behaviour [65,68,69].

The framework is also intended to allow self-evaluation. An important observation with self-evaluation by young drivers is that underperforming drivers tend to overjudge their own skills [43,70,71]. An important point for evaluation will therefore be whether the results for self-evaluation with D-Brief reveal a similar pattern.

2.7. Apparatus

The vehicle used was a Toyota C-HR build in 2018. All assistance functions were turned off during the test except for the navigation system. The ACC system was made available during the condition Driving with ACC. Data for the first two dependent variables (i.e., (i) Driving Performance and (ii) Drivers’ Controllability of ACC) were collected with printed versions of the D-Brief questionnaire for both participant and expert. The vehicle was equipped with cameras to capture driving situations. See Figure 2 for examples. One camera was directed to the rear-left and one to the rear-right. One camera was directed to the front and one camera was directed towards the driver and left side-view outside the vehicle. The video-images were provided in real-time to the expert-assessor to enhance his/her view of the driving situation. The video-images were recorded with Observer XT software (version 14) of Noldus and made available on a laptop. The software allowed real-time annotations about the driving situation, behaviour of other road users and the behaviour of the participant. The annotations were to facilitate later analysis of the video-images. For the analysis of lane changes (third dependent variable), the Observer software was also used to record when and in what direction lane changes took place.

2.8. Procedure and Planning

Per person, the test duration was 2.5 to max. 3 h. Participants were instructed that they were responsible for safe driving and invited to drive as they would normally drive. The participant’s main task was to drive the vehicle along a predefined route on a highway. Every participant conducted two trials: one while driving manually and one while using ACC. Under the condition Driving with ACC, participants were asked to use ACC as much as possible. The test always started with Manually Driving. Before starting, each participant received an explanation about the field test, including an informed consent. The road towards the start of the highway section was used to familiarize themselves with the vehicle and took about 8 min. Before starting with the second condition, Driving with ACC, participants received an instruction about ACC. For each driving condition (Manually Driving and Driving with ACC) the trial took approximately 40 min. The predefined routes included the Dutch highway A1 and A50. The test started either in Arnhem or Deventer into the direction of Zwolle. Between trials, the car was stopped at a parking place and there was time reserved to fill out the D-Brief questionnaire as well as to leave the vehicle, have a break and to receive an instruction about using ACC.

2.9. Test Conductor

One researcher conducted the test, gave instructions and performed the expert assessment. The role of test conductor was divided between two researchers; one of them had worked in the past as a professional driving instructor. Both researchers conducted together a pilot test to attune mutual understanding, approach of the trials and assessment procedure.

2.10. Assessment

During each trial, the researcher used D-Brief to perform the expert evaluation. After each trial, the participant filled out their own D-Brief questionnaire independently from the researcher. The latter is for self-evaluation of driving proficiency.

3. Results

The data analysed in this study comprise scores for the dependent variables (i) Driving Performance, (ii) Driver’s Controllability of ACC, and (iii) Number of Lane Changes. Variables i and iii were scored under two conditions: Manually Driving and Driving With ACC. Scores for the first two variables were collected based on self-evaluation in addition to expert-assessment. We start with results for Driving Performance based on expert-assessment.

3.1. Overall Evaluation of Driving Performance Based on Expert-Assessment

Figure 3 shows the average scores for each assessment aspect of Driving Performance taken by experts for the conditions Manually Driving and Driving with ACC.

Except for aspects 5 (“Abrupt control of steering wheel”) and 8 (“Mistake required other road users to counteract”), all aspects are formulated as statements for favourable and ‘good’ behaviour. While the majority of scores indicated that the behaviour was observed either “often” (score 3) or “(practically) always” (score 4), the scores generally show that performance was rather good. The unfavourable (‘bad’) behaviour of aspects 5 and 8 was only rarely observed.

Figure 3 shows that the average scores when driving with ACC are generally lower than driving manually. Since we have a small sample size and compare paired ordinal data (i.e., a rating scale for assisted driving after manually driving) we conducted a nonparametric Wilcoxon Signed-Ranks Test to examine the differences between driving with and without ACC. This test revealed that the ranks of the participants’ average scores for the condition Driving with ACC (Mean = 3.17) were statistically significantly lower than ranks for the condition Manually Driving (Mean = 3.29), Z = −2.10, p < 0.05. This means that overall participants performed worse with ACC compared to driving manually.

3.2. Assessment of Individual Aspects of Driving Performance Based on Expert-Assessment

While the previous section revealed that participants generally performed worse with ACC compared to manually driving, we will now take a closer look at the nature of the aspects that show differences in Driving Performance by looking at the assessment aspects individually and based on qualitative evaluation. Appendix C shows the scores per Behavioural Aspect and per participant based on expert assessment for Manually Driving and Driving with ACC. The fore last and last column show that the cumulative number of participants with adverse behaviour “#(−)” respectively improved behaviour “#(+)” when driving with ACC. The data show that participants scored for the majority of aspects somewhat better when driving manually than with ACC. The average of scores’ differences over all 12 participants was found to be largest (Mean = −0.9) for “No hinder to traffic participants when changing lanes” (nr. 29).

3.2.1. Aspects with Lower Scores in Driving Performance When Driving with ACC

When we look at differences in how driving condition has influenced Driving Performance (based on expert assessment) we may of course observe different patterns for individual assessment aspects. However, the following aspects show a consistent worsened performance with ACC—meaning that with each participant performance was either the same or worse (and never better) when Driving with ACC compared to Manually Driving (The number between brackets lists the number of participants with reduced performance for that aspect):

• 29. No hinder to traffic participants when changing lanes (9x)
• 11. Timely decision making to avoid hindrance to others (6x)

There are several aspects that show predominantly deteriorated behaviour when Driving with ACC compared to Manually Driving. These aspects are listed below. (Between brackets are the numbers for deteriorated versus improved behaviour when driving with ACC. For instance, the aspect “Lane changes with appropriate speed” scored for six participants lower with ACC than when driving manually, whereas we observed one participant for which this behaviour improved when driving with ACC.)

• 1. Smooth operation of brake and acceleration pedals (8 vs. 1)
• 2. Good position on lane (5 vs. 1)
• 3. Lane changes with appropriate speed (6 vs. 1)
• 4. Timely operation of indicators when changing lanes (5 vs. 1)
• 9. Speed aligned with speed of other road users (4 vs. 1)
• 10. Speed aligned to traffic intensity (5 vs. 1)

3.2.2. Relation between Driving with ACC and Care for Other Road Users

The analysis above reveals what aspects show worsened driving performance under the condition Driving with ACC. These aspects relate to operational control and taking care for the interests of other road users. Interestingly, two other aspects that show care for the interests of other road users scored somewhat better when driving with ACC:

• 14. Attentive to the behaviour of other road users (1 vs. 4)
• 17. Provides space to merging traffic (1 vs. 4)

Aspects 14 and 17 were the only aspects that showed a predominantly improved performance with ACC compared to the condition Manually Driving.

3.2.3. Relation between Driving with ACC and Perception of Traffic and Road Situations

With regard to the perception of traffic and road situations, use of ACC shows a small advantage in maintaining sufficient space cushion around the vehicle (nr. 24). Most aspects in this category show negligible differences:

• 20. Appropriate scanning of front view (1 vs. 1)
• 21. Appropriate scanning of side views (1 vs. 1)
• 22. Regular and alternating scanning (front, rear, sides) (1 vs. 1)
• 26. Timely reaction on indicators of other road users (no differences)
• 27. Appropriate judgement how road layout progresses (1 vs. 1)
• 28. Adheres to traffic rules (specifically road signs) (1 vs. 1)

Overall, in our test it is observed that using ACC caused hinderance to other road users when changing lanes, but it did not seem to have influenced other aspects regarding the perception of road situations.

3.3. Number of Lane Changes

Based on the recording of lane changes with the Observer XT software (version 14),

Table 1. Number of Lane Changes. (a) Mean number of lane changes per trial for the two driving conditions, (b) Mean number of lane changes per minute.

Number of Lane Changes Per Trial			Number of Lane Changes Per Minute
	Mean	SD		Mean	SD
Manually Driving	29.08	9.52	Manually Driving	1.04	0.35
Driving with ACC	19.17	5.37	Driving with ACC	0.70	0.19
(a)			(b)

shows the differences in number of lane changes between the two driving conditions. Results from a dependent t-test show that participants performed significantly fewer lane changes when driving with ACC (M = 19.17, SD = 5.37) than during manually driving (M = 29.08, SD = 9.52), t(22) = 3.141, p < 0.005. This outcome demonstrates that novice drivers who are learning ACC perform fewer lane changes when using ACC. This result is in line with previous research: Many studies report that ACC has a particular influence on lane change behaviour [7,65].

3.4. Congruence between Self-Evaluation and Expert-Assessment

Table 2. Comparison between Expert-assessment and Self-evaluation. Scores refer to (a) manually driving and (b) driving with ACC. Scores are ranked from highest average scores of Expert-Assessment to lowest.

Manually Driving						Driving with ACC
Rank of participant (based on av. scores)	Participant nr	Expert assessment (average scores)	Self-evaluation (average scores)	Difference (expert-self)	Counts of identical scores	Rank of participant (based on av. scores)	Participant nr	Expert assessment (average scores)	Self-evaluation (average scores)	Difference (expert-self)	Counts of identical scores
(A)	(B)	(C)	(D)	(E)	(F)	(G)	(H)	(I)	(J)	(K)	(L)
1	11	3.96	3.11	+0.85	4	1	11	3.93	3.14	+0.79	1
2	2	3.93	3.07	+0.86	4	2	12	3.79	3.11	+0.68	5
3	9	3.82	3.50	+0.32	13	3	2	3.54	3.11	+0.43	9
4.5	3	3.75	3.43	+0.32	10	4	3	3.71	3.46	+0.25	11
4.5	12	3.75	3.07	+0.68	3	5	9	3.64	3.39	+0.25	6
6	1	3.61	3.30	+0.31	13	6.5	8	3.32	3.39	−0.07	9
7	4	3.39	3.46	−0.07	10	6.5	4	3.32	3.46	−0.14	11
8	5	3.36	3.32	+0.04	8	8	7	3.14	3.36	−0.22	16
9.5	7	3.11	3.43	−0.32	14	9	5	3.11	3.48	−0.37	5
9.5	8	3.11	3.32	−0.21	8	10.5	10	2.86	3.36	−0.50	10
11	6	2.93	3.25	−0.32	14	10.5	1	2.86	3.39	−0.53	12
12	10	2.79	3.43	−0.64	6	12	6	2.39	3.21	−0.82	7
Average		3.46	3.31			Average		3.30	3.32
Average count identical scores					8.9	Average count identical scores					8.5
(a)						(b)

ranks participants from highest to lowest scores on expert-assessment for the condition (a) Manually Driving and (b) Driving with ACC. It also compares self-evaluation and expert-assessment by counting the number of assessment aspects that were judged identically between participant and expert in column F (for the condition Manually Driving) and column L (when Driving with ACC). Identical scores are not frequently observed: From the 29 assessment aspects at max 14 times (Manually Driving) and 16 times (when Driving with ACC) aspects were judged as identical between expert and participant. At the same time, the differences between self-evaluation and expert-assessment were low: For the condition Manually Driving, the average score of self-evaluation (for all assessment aspects) was 3.31 (SD = 0.21), for expert assessment the average score was 3.46 (SD = 0.27). For the condition driving with ACC the average score of self-evaluation was 3.32 (SD = 0.17), for expert-assessment the average score was 3.30 (SD = 0.35). Apart from the generally large congruency between self-evaluation and expert-assessment, the outcomes show that the experts assessed driving proficiency in the condition Manually Driving better than the participants themselves.

3.4.1. Self-Evaluation: Under- and Over-Judgement

In Table 2, column C (for Driving Manually) and column I (for Driving with ACC) show the average scores per participant that experts gave, ranked from highest average score to lowest. When we compare this with the average scores participants gave themselves (i.e., column D “self-evaluation” for Driving Manually and column J for Driving with ACC), we see that the highest ranked participants (who scored best) underjudge themselves compared to the experts’ assessment. The lowest ranked participant (who performed worst according to experts) overjudged his/her performance. Obviously, Table 2 shows a pattern for participants in the higher performance range to under-judge their own driving performance and for participants at the lower range to over-judge their own driving performance.

We see the above pattern in both driving conditions. Wilcoxon Signed−Ranks Tests revealed that the differences between self-evaluation and expert assessment were not statistically significant—neither in the condition Manually Driving (Z = −0.94, p = 0.35), nor when Driving with ACC (Z = −0.16, p = 0.87).

3.4.2. Aspects with Large Differences between Self-Evaluation and Expert-Assessment

No individual behavioural aspects showed statistically significant differences between self-evaluation and expert-assessment. When large differences were observed this concerned in most cases over-judgements (i.e., higher scores for self-evaluation than expert-assessment). The largest differences were observed when Driving with ACC for the aspects “Checks blind spot before changing lateral position” (nr. 23) and “No hinder to traffic participants when changing lanes” (nr. 29). In both cases, experts gave on average a remarkably lower score than participants. For aspect number 23 experts gave an average score of 2.50, whereas participants rated themselves in average 3.33. For aspect number 29 this was 2.58 (experts), respectively 3.58 (self-evaluation).

3.5. Drivers’ Controllability of ACC

Figure 4 shows the results for controllability of the ACC system based on self-evaluation and expert-assessment. According to expert assessment, all participants showed understanding of the basics of the ACC system, like turning on and adjusting the time-headway, and were able to use the system on the road.

Figure 4 shows differences in opinion between expert and participant for the aspects “Remains longer on the lane when using ACC” (nr. 36); “Driving with ACC influences taking account for other road users” (nr. 39) and “More road/traffic information is perceived when driving manually” (nr. 40). The differences show that the expert’s opinion generally agrees more with these propositions than the participants themselves. While none of these differences are statistically significant, self-evaluation shows larger standard deviations. It shows that although participants were generally somewhat less aware of the influence ACC had in respect to these aspects, some were as concerned about the influence as experts. Overall, in our test the understanding of the system seems reasonable.

4. Discussion and Recommendations

To evaluate the quality of our assessment framework we will review the changes in behaviour that we collected in Section 3 and compare them with expected observations based on previous studies. We start with its base requirement: the ability to distinguish differences in driving performance between manually driving and driving with ADAS (in our case: ACC). Next, we discuss whether the test results are in line with two specific expectations, i.e.,:

• Drivers are expected to demonstrate less anticipation of the behaviour of other road users.
• ACC is expected to deteriorate lateral control, in particular lane change behaviour. This may be witnessed by delayed decision making and less fluent lane change behaviour.

With regard to self-evaluation, we will discuss the relation between experts’ score−levels and participants’ own judgement. Furthermore, we discuss the utility of our questionnaire in terms of understanding and fit to practice. We will end this discussion with a reflection on this study’s limitations and provide recommendations for further research.

4.1. Assessing Influence of ADAS on Driving Performance

Based on scores from expert assessors, the results show: (1) that D-Brief reveals a significant difference in overall driving performance between manually driving and driving with ACC, and; (2) no significant differences of the individual assessment aspects between the two driving conditions.

Since the aspects are compared within subject, the second outcome is not surprising. It illustrates that participants were generally performing consistently in both conditions. We observed that participants were willing to perform. Since they had no other task than driving and operating the vehicle, they could put all effort to a safe participation in traffic. Traffic situations were relatively ‘easy’ during the road test. Hence, the circumstances were not a complicating factor. In view of these observations, the fact that overall performance (i.e., the first outcome) was significantly lower with ACC compared to the condition Manually Driving underlines that the assistance system (in our case ACC) had a remarkably negative influence on driving performance for our participants. As a result, we consider D-Brief successful in our base requirement: It is able to distinguish differences in driving performance between manually driving and driving with ADAS.

4.1.1. Anticipation and the Interests of Other Road Users

We expected that drivers with ACC show less anticipation of the behaviour of other road users and care less for their interests. The results show some differences, but they are very small. It seems that anticipation in driving situations that are self-initiated (like a lane change) are being worsened when using ACC. In particular, the aspect “hindrance of other traffic during lane changes” (nr. 29) scores more negatively with ACC compared to manually driving. However, when driving with ACC, drivers seem to anticipate better to changes initiated by other road users. For example, the aspects “provides spaces to merging traffic” (nr. 17) and “attentive to behaviour of other road users” (nr. 14) score better when driving with ACC than without ACC. The results seem contradictory, but can be explained by the technical properties of ACC: merging traffic takes advantage of the larger follow distances that the ACC-system maintains in comparison to human drivers. Whereas during lane−changes the ACC-system accelerates to pre-set speeds only once when completely moved to the other lanes. This delay causes hindrance to other traffic.

4.1.2. Assessing Perception of the Traffic Environment

It was expected that perception of relevant aspects in the traffic environment would be lower when using ACC. The results of our tool show, however, that scanning of the sides and behind the vehicle scores identically when driving with ACC and manually driving. (Most likely this is explained by the participants’ available capacity for attention.)

4.2. Lane Change Behaviour

We observed in the condition with ACC a significantly lower number of lane−changes. This may be explained by the driver being placed outside active control of the longitudinal task. Therefore, the remaining lateral task may have become more demanding. Indeed, aspects like “lane changes with appropriate speed” (nr. 3), “timely decision making to avoid hindrance to other traffic participants” (nr. 11) and “hindrance to traffic participants when changing lanes” (nr. 29) show adverse behaviour when driving with ACC. However, the reduced number of lane−changes may also be the result of less need to change lanes, as speed differences to slower vehicles in front are rather unnoticed, solved by the ACC-system. Nonetheless, when drivers get ready to change a lane it seems that this is decided upon later and when already larger speed differences between the own lane and other lane have developed. This explains the notable lower scores reported for “no hindrance to traffic participants when changing lanes” (nr. 29) when driving with ACC.

4.3. Self-Evaluation

The results show that participants in the higher performance range under-judge their own results and participants at the lower range to over-judge their own results. We observed this equally when manually driving and driving with ACC. The tendency of underperforming young drivers to overjudge their abilities is in line with other research [41,72]. It shows that D-Brief is sensitive to these individual differences in the judgement of personal capabilities and therewith provides a valuable quality of the tool’s adequateness for self-evaluation.

4.4. Fit to Practice

The starting point was to develop a framework for the assessment of drivers’ proficiency in controlling ADAS that would serve as an addition to the existing assessment matrix for driving proficiency (see Section 3.1) used by examiners. Unfortunately, during the time−frame of our test, driving examiners were not available. Other than the experience of one of our assessors being driving instructor, feedback from practice could not be included in our results. Evaluation with driving instructors is therefore an important recommendation. Nonetheless, the depth of the differences between the driving conditions revealed with the framework shows that D-Brief succeeds as a tool for training and reflection on the controllability of ADAS. Hence, we observe that the questionnaire allows a qualitative reflection on a driver’s ability to remain in control of the assistant system. This appraisal fits very well with the examiners’ final appraisal of driving proficiency, i.e., the ability to take part in traffic safely. Hence, the framework does succeed in supporting driving examiners and instructors with the assessment of drivers’ changing role when they utilize driving assistance. Since test procedures differ between countries in Europe [48], it is nonetheless recommended to include a review of driving examinations’ practices of other countries.

Feedback from using D-Brief for self−reflection showed no concerns with respect to the utility and understanding of the questionnaire. For reasons of consistency we nonetheless recommend adjusting the statements for aspects “Abrupt control of steering wheel” (nr. 5) and “Mistake required other road users to counteract” (nr. 8) to a formulation that stipulates positive behaviour, similar to the other statements.

In current practice, precise pre-set criteria are not set: For the examination, the overall picture counts wherein a candidate driver should show the ability to safely participate in traffic. D-Brief does not provide discriminative scores to identify what performance level is acceptable or not. Although this characteristic of our framework does not differentiate from current practice, an important recommendation is to perform further research into discriminative criteria because then the framework may also be used as an indicator for the required scope of training and to monitor training progress. Based on the results of this study, training would need to focus on improving care for other road users—especially when overtaking. However, since the development of training contents was not the scope of this study, future research is also necessary to define on what aspects such training should focus.

4.5. Limitations

Several limitations with regard to this study should be taken into account. While we choose non−parametric statistical analysis methods for their robustness with smaller sample sizes, it is still possible that no differences in the individual assessment aspects were found due to the smaller sample size. An important recommendation is therefore to include a larger sample size for further testing.

4.5.1. Influence of Traffic Condition

In our test, the driving circumstances were moderately ‘calm’. There were no severe weather conditions and summer holidays had started. From the perspective of the experiment’s design, this was positive. It allowed us to presume that changes in driving performance are primarily caused by driving assistance. Driving conditions have, however, a direct influence on driving demand and hence on driving performance. Since our test did not study the influence traffic conditions have on drivers’ proficiency to utilize the assistant system, this is strongly recommended for future research. In general, the utility of a system is not only influenced by its goals, but also by the context in which the system is used [73]. Traffic density may, for example, jeopardize favourable driving behaviour when utilizing ACC [74]. It may be that the influence driving condition has on performance is larger than the assistant function is able to compensate for. Then, our assessment framework may ‘fail’ to have sufficient power to discriminate in driving performance between manually and assisted driving. It may also be that an assistant function is simply not useful in particular circumstances. For further conclusions on the framework’s applicability, it is therefore necessary to research the framework’s sensitivity to distinguish differences in driving proficiency dependent on different levels of traffic complexity.

4.5.2. Participants’ Test Behaviour

Most participants used ACC for the first time. They were very attentive and showed eagerness to perform well in traffic. It is likely that results are influenced by test behaviour of the participants, showing their pursuit to perform good driving skills. Possibly, driving conditions that would normally cause adverse behaviour may have been compensated for. Additionally, from the perspective of test behaviour, an important recommendation is therefore to include more diverse and complex situations in future research.

4.5.3. Influence of Workload

Participants took part voluntarily and had all held their driving licence for less than 5 years. They had generally no or limited experience with driving with ACC. They drove in an unfamiliar vehicle. Using ACC may therefore be considered an additional workload—also confirmed in other studies [75,76]. Participants decided themselves when to activate ACC and with what settings (e.g., follow distance). The fact that in the condition with ACC participants performed less lane−changes may be explained by workload: the attempt to compensate for extra workload by changing lanes less frequently. Workload compensation may therefore be an important explanation for the reduced differences in driving behaviour between the two driving conditions. Consequently, our recommendation is to include measurement of workload with following test trials to allow analysis of the influence of workload on the driving performance under different driving conditions (i.e., manually driving and driving with ADAS).

4.5.4. Corrections for Engagement in ACC

ACC was—of course—configured to switch off when participants braked. Although participants were instructed to use ACC during the condition Driving with ACC as much as possible, it may have been that then lane changes were undertaken while drivers were currently not (or not anymore) engaged in ACC. However, the results have not been corrected for the moments in which ACC was off during the condition Driving with ACC. Based on observations, participants were engaged as much as possible in ACC during that condition. Nonetheless, a correction for engagement in ACC may help for further understanding of the generally small differences in driving performance between the conditions Manually Driving and Driving with ACC.

4.5.5. Influence of Driving Style

Behavioural aspects from D-Brief that show (almost) indifferent results between both driving conditions primarily relate to timing and traffic situation. Examples are “Timely reaction on situations that occur” (nr. 7); “Appropriate judgement how traffic progresses” (nr. 13); “Sufficient space cushion around the vehicle” (nr. 24), and “Appropriate judgement how road layout progresses” (nr. 27). This can be explained with the relation between timing and personal driving characteristics such as driving style. Studies revealed that driving style (e.g., a calm or aggressive driving style) have more influence on preferred speed and follow distance than the conditions of the traffic situation ask for [77,78]. Hence, a recommendation is to include the assessment of driving style in further research to allow comparison between the influence of driving style and the influence of driving condition on driving performance.

4.5.6. System Condition, System Type and ADAS Experience

In our test, controllability of ACC by novice drivers seems reasonable. However, our test lacks testing of controllability under more critical circumstances such as the system’s boundary conditions or in conditions with larger traffic volumes (see recommendation in Section 4.5.1). For example, whether participants understood that ACC would not react to stationary objects standing in the vehicle’s trajectory is not evaluated. Obviously, such evaluation was excluded due to safety concerns. Another important recommendation is to test D-Brief with other ADAS. Experience with a particular system may cause more substantial differences in an individual’s driving behaviour. In our case of testing driving proficiency when using ACC, it is therefore recommended to also include tests with users who are more experienced with ACC.

5. Conclusions

ADAS, and especially ADAS containing driving automation, change the role of drivers. It is necessary that driving training and examination adapts to this changing role, so that drivers will be trained and assessed for correctly utilizing assistance systems. The recently agreed on EU proposal to make a series of assistance functions mandatory increase the necessity and urgency of such training and evaluation.

The aim of this study was to evaluate whether our developed assessment framework “D-Brief” is able to assess driving proficiency while including utilization of ADAS (in our case: ACC). The framework is intended for application by driving instructors and examiners and should also allow self-evaluation. Results from a field test with 12 novice drivers and two assessors show that D-Brief is able to reveal a significant difference in overall driving performance between manually driving and driving with ACC. With respect to the nature of these changes, D-Brief revealed that driving with ACC had a negative influence on driving performance for self-initiated manoeuvres (especially lane changes)—Which was in line with expectations, and a somewhat positive influence on reacting to manoeuvres initiated by other road users (like merging in traffic). Analysis of D-Brief’s scores for the individual behavioural aspects allowed detailed explanations for the observed performance. As a result, we conclude that D-Brief is a successful framework to assess differences in driving performance between manually driving and driving with ADAS.

Based on the results of our test we also conclude that specific changes in driving behaviour occur when drivers utilize assistance systems, even when these concern ‘low’ level systems—Like level 2 (ACC) as researched here and equipped in many vehicles today. Furthermore, we observed changes that influenced drivers’ tactical tasks like the number of lane changes and the timing to initiate these tasks. These observations underline the urgent need to expand the current appraisal of driving proficiency with the assessment of drivers’ controllability of ADAS, such as D-Brief does.

Based on the observed relation between participants’ performance and their own judgement we also conclude that D-Brief is a suitable tool for self-evaluation. It helps drivers to critically reflect on their participation in traffic. This quality of D-Brief goes well with the growing importance of self−reflection as also included in current practice of driving examination.

Most individual assessment aspects of D-Brief show only small differences in driving performance between manually driving and driving with ACC. This may indicate that with limited training, important progress to increase drivers’ proficiency in using ADAS may very well be achievable.

Commercially available cars are increasingly equipped with multiple automated functions that simultaneously control functions such as headway, collision avoidance and lane position. The technology develops at high pace and updates may even be offered ‘over the air’ during the vehicle’s lifetime. These continuous developments are in contrast to the non−recurring assessment of driving proficiency after obtaining a driving licence. The technological advances fundamentally change how we interact with our vehicles. Drivers are required to identify and understand these functions, what they do and how they need to be operated. Indeed, the importance of adequate training to operate ADAS continues to grow. Future research is therefore needed to develop adequate training programs and to validate the assessment framework for a wider variety of systems and circumstances.

We conclude that D-Brief is an effective framework to assess driving proficiency while including the utilization of ADAS. With European legislation encouraging and mandating the equipment and use of advanced assistance systems, the presented assessment framework provides an important means to adapt current driver training and testing practices to account for technical developments and the evaluation of the safe operation of driving assistance systems.