Assessment of Situational Awareness in Relation to Advanced Navigation Systems Using Ship Handling Simulators ^†

Abstract

Digitalization has revolutionized the maritime industry, particularly in navigation systems. The use of advanced tools such as the Electronic Chart Display and Information System (ECDIS) has increased the need for information processing. However, the complexity of these systems can be overwhelming for navigators. To address the concern of usability of these complex navigation systems, training with simulator data allows the crew to familiarize themselves with these systems, handle complex navigation scenarios effectively, support the transition from paper-based systems to digital systems, and help in improving their situational awareness (SA) at sea. We propose a tool that provides optimal conditions for assessing situational awareness and informing the development of intuitive systems and user interfaces. In the maritime safety domain, there is an inverse correlation between situational awareness and scenario/system complexity, highlighting the importance of effective training and assessments to improve SA. The proposed tool utilizes the Situational Awareness Global Assessment Technique (SAGAT) method, widely used in other domains, to calculate an individual’s SA score. It evaluates participants’ situational awareness in different navigational scenarios on Ship Handling Simulators, using dynamic questionnaires and contextual maps. Additionally, it integrates a rule-based system to assess participants’ performance and calculate a situational awareness score in real time, offering possibilities for assessing the SA of navigators.

1. Introduction

Digitalization and automation are revolutionizing the maritime sector, which is a key industry as it transports over 90% of the traded goods around the world [1]. It is one of the most perilous and volatile industries based on its operational environment, so technological advancements and digitalization are crucial to its increased efficiency, reliability, safety, and sustainability and to meet regulatory and customer requirements [2].

Technologies used in the maritime industry are constantly improving and evolving. Today, the technically advanced systems used in the maritime industry are an amalgam of hardware elements (like sensors, GPS, cameras, radars, lidar, etc.), software elements that collect, store, analyze, and depict information like ECDIS (Electronic Chart Display and Information System), and humans (ship operators and fleet managers), which are interdependent and need to work in perfect synergy to harness the advantages of these technologically advanced systems.

The introduction of new tools and technologies has various advantages. For example, the use of ECDIS is one of the most consequential additions to the modern bridge [3], which is expected to help in critical decision-making processes. Its accurate nautical charts, superimposed with GPS or Radar data, can help improve the situational awareness of the Officer on Watch (OOW) [4]. These new tools and technologies have increased the information that is collected, processed, and depicted, making it visually and mentally complex to handle these advanced systems [5,6]. ECDIS is the successor of paper navigational charts but information here is vectorized, nautical information is layered, and there are various features and indicators that are not intuitive or open to misinterpretation [7]. Complex stressful scenarios and large amounts of data, coupled with un-intuitive user interfaces can affect the situational awareness of the OOW and the crew. According to (Endsley and Kiris, 1995) [8], the use of automated tools also raises the concern of “Out-of-Loop” syndrome, where the OOW and crew are plagued by the lack of SA and ability to respond in case of system failure.

In the domain of maritime safety, an inverse correlation exists between situational awareness and scenario/system complexity [9], emphasizing the need for effective training and assessments for improving situational awareness [10]. We intend to carry out a study to understand the impact of these advanced systems and their user interfaces on the situational awareness of the OOW and crew. In this paper, we propose and describe a situational awareness assessment tool (SA tool) with simulator data that establishes optimal conditions for evaluating situational awareness, with insights informing the development of intuitive systems and user interfaces.

2. Situational Awareness

Situational awareness (SA) is defined as having a good perception and understanding of various elements in the surrounding environment within a certain volume of space and time [11]. Navigators working in very risky and rapidly changing environments are faced with safety-critical scenarios where important decisions need to be made in stressful and time-constrained environments; hence, situational awareness is of utmost importance for safety at sea.

Situational awareness can be illustrated as a combination of three important levels according to Mica R. Endsley and Erik S. Connors in their paper, titled “Situational Awareness: State of Art”, as shown in (Figure 1) [11]. Level 1 is “Perception”, and in this level, data from senses like vision, hearing, and touch are used to build a mental picture of the environment and then scan for relevant information pertaining to the current situation. An interesting observation was made that failure in level 1 of SA is one of the main causes of accidents due to human error based on a study carried out in the aviation industry [12]. Level 2 is “Comprehension”, and it is the combination of observations made, theoretical knowledge, and experience to understand the significance and implications of the situation. Level 3 is “Projection”, and it involves predicting the future environment and making critical decisions based on the current understanding of the scenario. For future decision-making to be correct, the data and the inference developed from it in level 1 and level 2 must be accurate.

Situational Awareness Assessment Techniques

Situational awareness is one of the vital factors deciding the performance of a navigator. A good situational awareness is crucial in the maritime industry where there is high information flow, and one poor decision has the potential to cause catastrophic damage to property and human life. Hence, situational awareness assessment is imperative in evaluating the human factor in various domains like aviation, maritime, and military operations [13,14].

Some of the most popular SA assessment techniques documented in the literature are SAGAT (Situational Awareness Global Assessment Technique) [15], SACRI (Situation Awareness Critical Room Inventory, Hogg et al., 1994) [16], SALSA (Situation Awareness Levels for Systems Assessment, Hauss & Eyferth, 2003) [17], QUASA (Quantitative Situation Awareness Assessment, McGuiness, 2004) [18], SART (Situational Awareness Rating Technique) [19], and CDM (Critical Decision Method) [20]. These SA assessment techniques use a variety of probing techniques like the freeze-probe technique, the real-time probe technique, the post-trial self-rating technique, the observer rating technique, the performance-based rating technique, and the process indices-based rating technique [21].

SAGAT was first proposed as a technique for assessing the SA of an aircraft pilot. It uses the freeze-frame probe technique to compute the SA rating [15] of the assessee, as shown in Figure 2. In the freeze-frame probe technique, the maneuvering task being performed by the assessee on the ship simulator is frozen at random times and questions pertaining to the situation are presented to the assessee to answer in a fixed duration of time. The responses are recorded for evaluation and SA score calculation. The freeze probe technique is used multiple times for each assessee for different scenarios and questions. The random freeze-frame timing makes SAGAT one of the SA assessment techniques with the highest correlation to real performance due to higher validity and statistical stability [21]. SAGAT focuses on monitoring the accuracy, credibility, cost, and performance of SA metrics.

3. Proposed Situational Awareness Assessment Tool

The objective of the proposed SA tool is to study the effect of ECDIS and its user interface on the situational awareness of the navigators. It is based on the SAGAT SA assessment technique and utilizes the freeze-frame probing technique. Back in 2016, more than 59% of the merchant ships had started the use of Electronic Nautical Charts (ENC) in ECDIS and, since then, its usage has been on the rise [22]. There are more than 30 different ECDIS manufacturers in the market. While the basic presentation requirements are standardized in resolution MSC.191(79) by the International Maritime Organization (IMO), various custom settings, menus, and features in ECDIS can differ widely based on manufacturers and the models [22,23,24]. These variations in the user interface and the data formats may influence the situational awareness of the navigators. A situational awareness (SA) training/assessment with simulator data provides an opportunity for the navigators to familiarize themselves with the user interfaces of these technologically advanced navigation systems and prepare them to collaboratively handle various complex navigation scenarios, respond, and assess their responses for better preparedness and improved situational awareness.

Experimental Setup of Situational Awareness Assessment

The proposed SA tool evaluates and assesses the situational awareness of test participants in the complex navigation scenarios on Ship Handling Simulators (SHS) integrated with advanced navigational systems. A total of seven complex ship navigation and maneuvering scenarios with varying difficulty levels, duration, and number of freezes are designed for the assessment of the test participants. To avoid order effects, the order of the scenarios was balanced, assigned to each participant, and loaded in the SHS for assessment. Each scenario has a pre-destined number and duration of freeze frame, where the task is paused to deploy a dynamic questionnaire to assess the SA of the test participant pertaining to the current navigational scenario. The questionnaire is integrated with a contextual map for a better outlook.

Broadly, the experimental setup consists of two main components:

• Ship Handling Simulator (SHS) bridge;
• Situational Awareness Assessment tool (SA tool) (Version 1.6, Fraunhofer CML, Hamburg, Germany, 2024).

The SHS bridge simulates the maritime environment for training and research purposes. It contains various advanced tools like ECDIS, Radar, VHF communication, Steering and Engine Control, Alarms on the instruments, Logbook, etc. A fully IMO-compliant Wärtsilä Navi-Trainer Professional Simulator (NTPRO) is used as the Ship Handling Simulator [25]. Different models of ECDIS from various manufacturers can be integrated into the SHS Bridge to assess and train the navigators on the same equipment as on the real ship for improved SA and preparedness. Data from various sensors like Automatic Identification System (AIS), Global Positioning System (GPS), wind, echosounder, and log are transmitted from the SHS bridge to ECDIS based on the National Marine Electronics Association (NMEA) 0183/2000 standard [26]. Figure 3 shows the SHS bridge used for the SA assessment.

The SA tool is a combination of Python-based applications running on a PC connected to the same network as that of the SHS bridge [27]. The main applications that constitute the SA tool are the Tracker application, the User Map Interface, the Simulation Control tool, and the Question Control tool, as described below:

Tracker Application: The communication and data exchange between the SHS and SA tool is facilitated by the Distributed Interactive Simulation (DIS) protocol [28]. A subset of DIS Protocol Data Units (PDUs) such as Entity State, Signal, Data, and Data Query are used to track the ship position and motion parameters [29]. The Data PDU is used to transmit simulation parameters from the SHS to the tracker application, while the Data Query PDU enables information transfer from the tracking application to the SHS for remote control of data tracking. The tracker application decodes the DIS Protocol Data Units (PDUs) and tracks all the ship objects. This application also implements a rule-based system to assess adherence to COLREGs (Collision Regulations) based on the collected data from the SHS [30].

User Map interface: An interactive user map interface is used to visualize the questions on an Electronic Nautical Chart (ENC) with the ship objects sent from the tracker application. The ENC is requested from a separate chart server using Remote Procedure Calls (gRPC).

Simulation Control Tool: This tool automates the process of scenario start, stop and freeze on the SHS based on the selected participants. The tool assigns the predefined stimulus order of the scenarios, number, and time of freezes for each participant and loads the scenario into the SHS for assessment.

After a scenario is loaded on the simulator bridge, a notification will appear on the tool requesting the participant for their readiness to start the simulation. Once the start is approved by the participant the simulation scenario execution starts, the tool automatically pauses the exercise on the simulator at the destined freeze time and initiates a pop-up to instruct the participant to stop their ship maneuvering activity and start answering the assessment questions. While answering the questions, participants cannot see the simulator, so they are forced to answer the questions from memory. There will also be an audio notification from the SHS after the successful loading of the scenario, the start of the scenario, and the freeze of the scenario to notify and guide the test participant. Figure 4 shows the example view of the Simulation Control tool.

Question Control Tool: The Question Control tool is used to load the dynamic questionnaire on an interactive map based on the current scenario and participant experience. The term “dynamic” refers to the fact that, although all test subjects are given the same questions, the situations in which the questions are presented can easily be differentiated due to the test subjects altering the situation through their driving. This unique characteristic poses a challenge for the tool, such as accurately calculating the answer options for each participant based on the individual situations. The tool loads the question stimulus order based on the participant selected. After every freeze, the participants are displayed with questions on the integrated map to answer. A rule-based system is used to assess the answers to check adherence to COLREGs (Collision Regulations). The answers and SA assessments are stored for further processing. The questionnaire consists of single-choice, multiple-choice, and short-text answer-type questions, and a custom grading schema is used for each type of question. Figure 5 shows the example view of the Question Control tool.

The unique features of the SA tool are as follows:

• The DIS protocol is selected as the underlying communication protocol to support simulation data exchange between the simulator and the SA tool. The data messages between the simulator and SA tool are exchanged in PDUs (Protocol Data Units) as defined in IEEE 1278.1-1995 [28]. The use of DIS protocol prevents the SA tool from being vendor-locked and makes it compatible for use in SHS from various manufacturers, as most of the reputed SHS use DIS protocol for data handling.
• The implementation and use of a rule-based assessment and evaluation system for the collected responses to check their adherence to COLREGS and to calculate the SA rating.
• The use of interactive maps to display the questionnaire in the freeze frame to provide a better outlook to the test participant.
• In addition, the SA tool also collects the confidence level of the test participant for each of the answered questions for gathering a better understanding of the mindset of the test participant at a particular time and test scenario.

4. Experiment Conducted Using the SA Tool

The method developed and the corresponding tools were successfully tested and validated in an initial study. A total of 40 subjects participated, divided into two groups: 20 novices and 20 experts. The novices were nautical students with prior knowledge of radar and ECDIS technologies, whereas the expert group consisted of experienced navigators with at least five years of professional experience, such as pilots. The division into novices and experts served not only to test the method itself but also to investigate potential differences in situational awareness (SA) between these two groups. However, the focus of this discussion is on the method and instruments, not on the SA results.

For the execution of this study, the SHS and the instruments were integrated into a comprehensive experimental setup. Each experimental run lasted about 3.5 h per participant, including breaks. Participants were informed in advance via email about the study and its conditions, and they could withdraw at any time without any disadvantages. At the start of the experiment, participants received detailed instructions about the procedure. This was followed by an initial questionnaire to collect personal background data and potential confounding variables, as well as a standardized introduction to the SHS, including the basic functions of radar and ECDIS.

To ensure standardization and minimize experimental effects, all instructions and questionnaires were presented via a laptop. Participants had access to two screens: one for responding to the questions from the SA tool and a second for instructions and additional surveys, including questions about workload and the perceived difficulty of the scenario. During the experiment, participants were directed by the experimenter on which tasks to perform at which times. Before the start of the actual study, the experimental procedure was conducted with a test scenario to allow participants to familiarize themselves with the tasks.

Upon starting the scenario, the subject had to navigate safely, with execution on the system performed by a “helmsman experimenter” at the participant’s direction to keep the experimental conditions consistent between subjects. If a freeze occurred unexpectedly for the participant, they were asked to turn away from the SHS and proceed to the SA screen to answer the questions generated by the SA tool regarding the current situation. After completing a scenario, participants were asked to rate the difficulty of the scenario and the effort involved in handling the task on the second screen.

After three scenarios, consisting of one test scenario and two data collection scenarios, there was a break. Subsequently, four additional scenarios were conducted. At the end of the study, a standardized survey was conducted on safety in the simulator and the user-friendliness of the ECDIS. Finally, participants received a debriefing, where they had the opportunity to comment on the experiment or ask questions. Figure 6 shows the sequence of events in the experiment.

Overall, the test procedures were conducted smoothly and without any major incidents. The correct order of the scenario freezes and questions by the “question control tool” and the calculation of the correct answer options and answers by the SA tool led to the first interesting results with this method. The coordination of the complex, individual task steps for the participants, the interaction of the SHS and the tools, as well as the entire experimental process, were successful. The feedback from the participants also shows that the entire experimental setup was successful. In total, the method and tools, as well as the complex experimental setup, were tested successfully.

5. Conclusions

Digitalization and automation are continuously introducing new tools and technologies into the foray of the maritime industry and the working environment of navigators. An experimental setup that simulates real-life scenarios and allows the navigators to train and familiarize themselves with tools and their user interfaces in a safe environment before embarking on their voyages could significantly improve the situational awareness and preparedness of navigators at sea. In conclusion, our research emphasizes the importance of usability in maritime navigation. The proposed system not only improves situational awareness but also ensures that navigators can efficiently handle complex data, ultimately leading to safer maritime operations.

The experiment setup and tools such as the SA tool and Question Control tool, which are proposed, developed, and evaluated for ECDIS-based training and situational awareness evaluation of navigators in this paper, have shown promise and received positive feedback from navigators. Further research avenues like the integration of machine learning techniques for scenario generation and situational evaluation models could be explored to enhance the proposed tools. Continuous innovation and implementation of such tools and experimental setups in the maritime industry to evaluate situational awareness and train navigators before introducing new tools will pave the way for safer, hassle-free voyages and reduce stress for navigators.

The evaluation, measurement, and transparent presentation of users’ situational awareness is especially helpful in the development of complex navigation systems and enables manufacturers of such systems to design project processes more effectively. A continuous review of the user interfaces and functionalities could lead to long-term optimizations in the area of users’ situational awareness (SA).