Rethinking Maritime Education, Training, and Operations in the Digital Era: Applications for Emerging Immersive Technologies

Abstract

The use of simulators in maritime education and training is an essential component for developing seafarer competencies. Emerging immersive technologies, such as virtual reality (VR), augmented reality (AR) and mixed reality (MR) have created new and differing possibilities for maritime simulations and simulators. The increasing advancements and technical readiness of these systems have paved the way for a new generation and category of simulators and simulation-based experiences for professional education, training, and operations, which are relatively cheaper, more immersive, compact, and accessible in comparison to traditional configurations. Although the concept of utilizing VR, AR, and MR head-mounted display technologies for professional training and operations is not new, their recent developments and proliferation now allow for practical implementation and real-world application. Particularly, the adoption and integration of these technologies into the education, training, and operations of maritime industries provide new possibilities and paradigms to support operators and operations both on land and at sea. The purpose of this paper is to discuss the concepts of VR, AR, and MR applications specifically for maritime education, training and operations, including the potential benefits, drawbacks, and limitations of these systems.

1. Introduction

Maritime-related industries, such as shipping, are complex socio-technical environments, which function within highly variable operational, regulatory, economic, political, social, and international conditions. One of the most critical elements of operating today’s modern and technologically sophisticated ships safely are the seafarers themselves [1]. Thus, seafarers require appropriate training and skills in order to successfully manage the complexity of operations. Maritime education and training (MET) is critical to the development of competent workers who contribute to the safety and efficiency of a system. The digital transformation and its effects of ever-advancing technologies have trickle down effects on the organization of maritime-related industries and the traditional tasks of seafarers [2]. Whether onboard or onshore, working in the maritime domain is becoming increasingly based on knowledge, requiring increasingly specialized skills. There are approximately 1.6 million seafarers worldwide, and the demand for highly skilled seafarers and officers is growing [3], stressing the need for effective and efficient education and training.

Different types of skills will be required as new paradigms of shipping operations continue to evolve, while once critical and relevant seafaring competencies may become obsolete. Seafarer and seafaring-related skills, which are already highly technology and technically oriented, will likely require even more continual development and updating during their formal education and throughout a career [4]. As new types of operating paradigms evolve and varying types of ships share the same waters (e.g., manned, unmanned, shore-side control, semi-autonomous and fully autonomous ships, etc.), different competencies consisting of hard and soft skill sets will require continual development. Maritime education facilities, training centers, simulator and technology providers, and shipping companies must provide operators with optimized MET programs and tools to meet the evolving demands of current and future maritime operations.

The recent advancements of virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies in consumer electronics are creating new opportunities for evolving current maritime training tools and methods. The rapidly advancing and competitive marketplace in which developers and technology companies are aggressively investing in these areas is an indication of the interest the hi-tech industry and society at large has in VR, AR, and MR. Wearable technologies, such as VR, AR, and MR head-mounted displays (HMD) pose a new and interesting addition for professional education and training. This potentially industry-disrupting technology may supplement, or in an extreme and longer-term perspective, replace, current training simulators and methods prevalent across MET facilities and maritime industries. In general, MET providers and maritime simulator manufactures have focused on high-end full mission simulators in which hardware and software combine to replicate realistic, high-fidelity working environments at sea, becoming an industry standard across the globe. There is currently a gap in the market of MET simulators for more personalized, immersive, mobile, and accessible training opportunities which are delivered through personal consumer electronic platforms (such as personal HMDs and mobile computing) (see Figure 1). Thus, the further exploration of new and emerging technologies such as VR, AR, and MR HMDs, and their wide-ranging impact on MET, maritime operations, and the maritime domain as a whole, is required. The purpose of this paper is to explore the potential effects, including benefits, drawbacks, and limitations of VR, AR, and MR technologies applied in MET and operations.

2. Simulations and Simulators in MET

Maritime officers undergo academic and practical training throughout their education, required for professional certification, recertification, and promotion throughout one’s career. The utilization of simulator technologies in MET has been commonplace for decades and is well supported by the IMO in the International Convention on Standards of Training, Certification and Watchkeeping for Seafarers (STCW) [5]. The STCW focuses on the acquisition of differing types of skills for division of support, operational, and management levels [5]. This requires a balance between both academic and vocational approaches in the education process, which has shown to be tricky in MET [6]. Traditionally, MET is delivered through a combination of theoretical classroom-based training, followed by practical training, and service at sea through hands-on experience [7]. Although this model still persists, the advent of improved technologies and advanced maritime simulators has allowed cadets to practice and develop their seafaring skills before ever having to step foot onboard a real ship. Simulator-based training is also a cost-effective and powerful solution that allows instructors to create diverse and replicable scenarios, which they would not otherwise be able to train due to safety, economic and ethical constraints of real-life training exercises [8,9]. Three-dimensional (3D) visualization can reduce the gap between simulation and real life [10], and well-designed simulation enhances learning, improves performance, and reduces errors [11]. Furthermore, simulation training improves trainees’ perception and assessment of dangerous situations [12], improves training outcomes in comparison to conventional classroom-based exercises [13], and creates more collaborative, critical thinking and case-based learning [14,15].

There are five basic types of training divisions which include: (i) team training, (ii) operator training, (iii) decision making training, (iv) procedure training, and (v) maintenance training [16]. MET simulator configurations, including software and hardware technologies, can generally be divided into several classes based on their performance capabilities and functionalities, for example, (i) full-mission, (ii) multitask, (iii) limited task, and (iv) special tasks [17,18]. In practice, this allows the training of both technical and nontechnical skills required by seafarers, including positions and competencies required both onboard and onshore. Maritime simulators and simulated training activities encompass a diverse range of skills required for differing professional positions and operations, for example, navigation and bridge operations, engine rooms, communications, vessel traffic services, ballast and cargo, anchor handling systems, firefighting, rescue, oil spill management, drilling systems, pilotage, and crane operations [5]. These simulators can range from smartphone and tablet applications to basic desktop computer software to full mission systems with 1:1 scale duplication of the work environment and its equipment [16]. MET simulators can have disadvantages in comparison to real-life training, including over-simplifications of marine systems and procedures or deficiencies in the underlying coding of the virtual models [10]. However, the advantages of simulators arguably outweigh the trade-offs, with simulation-based training becoming increasingly prominent and sophisticated in MET over the past several decades.

MET is disseminated through accredited educational institutions and companies, including maritime universities, academies, training centers, and for-profit businesses to educate, train, and license seafarers. These training facilities require a sophisticated infrastructure of people, equipment, and financing to maintain and operate. Seafarers and maritime workers can be viewed as both a customer and product of MET. Professional development, refresher training courses, and maintaining valid certifications requires visits to centralized MET facilities for training and assessment of necessary competencies, not only throughout their formal education but over the course of a seafarer’s career. As working at sea is by nature a transient, distributed, isolated, and often international, costs to bring workers in for training at traditional MET facilities can have high economic impact for individual companies and the overall industry. Thus, the economics and logistics of seafarer training impact the quantity and quality of training delivery and outcomes. This can have ripple effects on the competencies of seafarers and ultimately the safety of operations.

3. A Paradigm Shift for MET?

As the computing power and ubiquity of high-tech consumer electronics continually increases, while relative price-points decrease, the ability to integrate emerging technologies into widespread education and training methods is becoming practical. This provides new opportunities to explore possibilities for enhancing the educational experience for students and instructors. Maritime industries have developed a comprehensive regulatory framework, which ensures seafarers are competent to carry out required duties. However, the methods of developing particular competencies have evolved as technologies and operational demands evolve.

Emerging Technologies

Just as personal computing and the Internet created new opportunities for knowledge sharing by opening new possibilities for information exchange, distance education, and communication, today’s consumer electronics (e.g., smartphones, tablets, VR, AR, and MR HMD systems, etc.) provide a new landscape for ubiquitous training simulators. These technologies are relatively affordable, flexible, and portable, which allow individuals and educational facilities to connect through mobile learning and communication platforms, virtual environments, and distributed training simulations. VR, AR, and MR can be described within a continuum which ranges between “real reality” and “virtual reality” [19]. There are various definitions for VR, AR, and MR which can be dependent on context, as well as the state-of-the-art technology at the time of formulation. The following definitions and current technological examples provide a framework for the reality-virtuality continuum:

Virtual Reality (VR): is a computer generated three-dimensional (3D) graphical representation of an environment in which users can be immersed using an HMD. The HMD provides stereoscopic view and fully covers the eyes of the user. Users cannot “see” the real world when they are in VR HMDs (e.g., Oculus Rift, HTC Vive, Vive Pro, and Samsung HMDs).

Augmented Reality (AR): is an overlay of information on the real world. AR is a live direct (through transparent displays) or indirect (through live camera) view of the surrounding real-world environment enhanced and augmented with computer-generated information (such as two-dimensional (2D) or 3D graphics, video, and audio, etc.). Users are aware of what is going on in the real world when they are using AR, unlike VR (e.g., Google Glasses and Apple AR).

Mixed Reality (MR): merges real and virtual worlds to produce hybrid synthetic environments where physical and digital objects co-exist and interact in real time (e.g., Microsoft HoloLens).

4. VR, AR, and MR Applications in the Maritime Domain

4.1. In Education, Training, and Assessmet

MET has long been technologically oriented. The introduction and integration of VR, AR, and MR can both compliment current MET methods and simulator technologies, as well as create new and different training opportunities within MET pedagogical frameworks. VR, AR, and MR HMD technologies may be initially viewed as a disruptive technology by the industry, particularly manufacturers of traditional maritime simulators and MET facilities. These have typically required large investments in the development, marketing, acquisition, operation, and maintenance of traditional, high-value maritime simulators and personnel, representing significant MET infrastructure and economic investment. However, these new technologies should be viewed as a potentially valuable addition to MET, providing new ways of training which may contribute to enhanced learning outcomes, increased training opportunities, and ultimately better trained workers. There is a growing body of research investigating the effects of the latest generation of HMD technologies as tools for education and training, with results that are showing promise for increased user motivation, higher levels of immersion, and increased transfer of training [20,21,22,23]. Specifically, within the maritime domain, research and development projects are focusing on the next generation of HMDs for maritime applications, and investigating how these new technologies can be implemented within MET and operations [24,25,26].

Emerging HMD technologies also opens a market for new product offerings for maritime simulator and equipment manufacturers. High-cost simulators (e.g., full mission bridge or engine room installations) can have more favourable profit margins and associated lifecycle revenue (including installation, maintenance, and upgrading costs) by comparison. However, personal HMD hardware and software systems represent a lower price point per unit and would have a much higher volume of individual users and total units used throughout the industry.

HMD training applications have advantages in their inherent flexibility, mobility, and accessibility. Training can be implemented anywhere and anytime in a high-fidelity and immersive simulated environment. These systems inherently increase access to MET simulations and training packages, which are traditionally limited to desktop programs and simulators located at centralized training facilities. The training anywhere anytime concept allows trainees and professionals alike, who are taking courses at MET training facilities, more flexibility. It can increase opportunities for simulation training after hours or at the convenience of the trainee, regardless of geographical location (see Figure 2).

From an economic perspective these mobile technologies can reduce the direct and indirect expenditures of training throughout a seafarer’s career, including associated travel costs, overtime and salary payments. For MET facilities the overhead costs of operating simulator systems for purchase, operation, and maintenance requires significant economic capital. VR, AR, and MR systems could reduce the amount of time required at traditional MET facilities by allowing user access while onboard, travelling or at home, thus increasing opportunities for immersive simulator training.

The Role of the Instructor

The instructor plays a key role in education, providing reinforcement and expert knowledge to facilitate students, and their specific needs, throughout the teaching process. During traditional maritime simulator training the instructor and student interact collaboratively and continuously throughout a simulated scenario [27]. The direct interaction between instructor and student within simulator-based training is an important aspect for developing student knowledge, providing feedback through verbal cues and physical gestures [27]. Furthermore, besides the planned curriculum and learning outcomes, during direct interactions a “hidden curriculum” emerges which can include unplanned and unstructured learning moments between instructor and student (or group of students) that add value to learning outcomes and student experiences [7]. Instructors also rely upon scaffolding, debriefing, and meta-reflections to promote effective learning in current simulator-based MET [28].

In the development and application of virtual reality and mobile training solutions the role of the instructor and their interaction with students will evolve. Personal training solutions can be used within traditional training facilities in physical proximity with instructors and other students. It also allows for mobile and distributed training sessions which can be run live with other students and instructors through connected networks, or on-demand at an individual’s convenience. This flexibility in training timing and mobility requires different input and requirements from the instructors and assessment techniques. For example, VR HMDs do not allow instructors to follow the student’s attention or gaze patterns as typically performed in traditional simulators. Thus, new types of interaction and integrated assessment techniques between student and instructor should to be developed, independent of the availability of a live instructor, either in person or over a network. Mobile systems such as built-in tracking sensors, such as eye trackers, heart rate monitors, accelerometers, and automated performance scoring can provide more objective assessment and feedback of trainee exercises. In addition, VR simulators can provide more personalized training by adapting the scenarios based on trainee’s past performance and individual training requirements [29].

Technologies are not necessarily intended to replace traditional teaching methods and experienced well-trained instructors, but rather be implemented as a supplemental and complimentary tool to the teaching and learning framework [30]. MET curriculums cannot exclude the role and value of experienced instructors in the development of student competencies. Emerging technologies and training solutions in the form of VR, AR, MR, artificial intelligence, machine learning, and automated assessment should not be looked as a threat or replacement to traditional paradigms of education delivery, but rather a supplementation to the overall system. These are potential solutions to take advantage of and implement when and where they provide benefits. Ultimately, it is the experience and knowledge of subject matter experts and experienced instructors that provide the foundation for student education and the development of training programs based on operational requirements.

4.2. In Operations, Onboard and Onshore

VR, AR, and MR technologies can also add value in onboard and onshore operations. These technologies provide new possibilities and methods during planned and unplanned work, command and control, scenario testing, communication, and knowledge exchange between crew, ships, and ship-to-shore networks. With mobile VR HMD simulators onboard, crew can practice complex offshore operations and procedures before performing them and/or refresh their technical, high-risk, emergency or seldom used skills and scenarios. For example, navigation procedures for unfamiliar, high traffic or hazardous waters could be practiced in a mobile VR HMD simulator onboard a ship prior to being executed in real life. Similarly, specific skills and situations regarding emergency scenarios, ice navigation, challenging weather conditions, maintenance of machinery systems, and logistical planning can be practiced and refreshed during operations at sea. In future the crew could use “digital twins” of ships in VR to better understand the capabilities of the ship before performing complex operations or maneuvers. This can be particularly helpful when new crew takeover ships in order to familiarize themselves with the specific environment, equipment and operations in an effort to avoid errors and accidents.

Unlike VR, which fully removes users from reality, AR and MR HMDs overlay virtual objects and information on the real world. This can enhance the information flow in operations and help users better understand situations through superimposed computer-generated visualizations in their sightline. For example, in navigation procedures, AR and MR can superimpose route data such as the fix, course, and drafts to the navigating officer or multiple members of the bridge team, similar to the Head-up Display found in military and commercial aviation. This could be particularly useful for low visibility navigation. In emergency operations or evacuation planning and execution, AR and MR can guide users through procedures by guiding them through a structure and provide best-practice instructions and reminders in high stress situations. AR and MR can further improve the workflow and facilitate workers access to information, where HMDs provide hands-free data transfer (both sending and retrieving information to and from users), facilitating knowledge acquisition and communication. For example, AR and MR HMDs can aid marine engineers during maintenance procedures through step-by-step instructions of superimposed projected visualizations onto the real-life equipment and workspaces, augmented with text or audio directions. Furthermore, the AR and MR HMDs can transmit a live video feed to remote locations where centralized experts in a specific piece of equipment, system or procedure can guide the onsite worker in real time through the correct procedures.

As shore-based command and control of ships and marine structures increases, VR, AR, and MR technologies can be used to facilitate distributed and remote operations. In particularly, the immersive nature of VR HMDs may be a solution for shore-side control and monitoring operations to gain sufficient knowledge of a ships’ status by allowing for first-person perspectives of the onboard structure. This can allow for shore-side personnel to provide navigational feedback and input from the ship’s bridge, inspect systems, allow for structural walkthroughs, and gain access to confined spaces or remote areas of larger ships that have reduced or no crew onboard. AR and MR are powerful technologies which provide users with enhanced information and communication transfer opportunities, providing interesting additions to traditional methods and operations, as well as for future configurations of ship command and control.

5. Implications, Trade-Offs, and Future Directions

The introduction and proliferation of VR, AR, and MR technologies in the consumer electronic marketplace provides a wide range of new applications and opportunities. The technologies are creating exciting new possibilities for MET and maritime operations, enhancing and possibly disrupting the current system. There are inevitable implications and trade-offs with the introduction of new technologies. Although the rapid advancements of these technologies are opening new areas for MET and operations, we must first better understand their overall purpose, effectiveness, and potential added value to the overall system. Just because we can, does not mean we should. There must be pedagogic foundations and frameworks for implementing new technologies, such as VR, in education and training programs [31].

In particularly, the soft skill development of cadets and seafaring professions has been of focus within the industry over the past several decades. Bridge resource management training has been developed to train communication, leadership, decision making, and teamwork-oriented skills [5]. Socialization and face-to-face communication skills and evaluation is of importance in seafaring and working within complex socio-technical systems, which not only require technical competencies, but also social skills.

VR, AR, and MR technologies provide novel and exciting experiences, however, the long-term benefits and effectiveness of such systems and their applications are not yet well understood in the maritime domain. Many basic questions remain unanswered including: What types of training and operations are these technologies useful for? How can/will they be fit within the larger scope of MET curriculums? What are their weaknesses and limitations? and What is their effectiveness compared to traditional methods and technologies? Future research should investigate and answer these questions, and many other questions which will emerge as evolving technologies create new and differing opportunities for training and operations.

6. Conclusions

The utility and flexibility of advancing immersive HMD systems provide a wide range of new possibilities and applications for MET and operations. These personal devices are relatively affordable, flexible, portable, and powerful technologies that stand in a stark contrast to traditional industry-standard maritime simulators. VR, AR, and MR HMDs provide an interesting supplement, and in time, perhaps a disruptive alternative to the current methods and technologies in MET and operations, opening new commercial areas and practical applications across the industry. However, as with any new technology, especially as captivating as VR, AR, and MR HMDs, a research-based approach is required. Many issues need to be addressed and investigated to better understand their strengths and limitations for both MET programs and for maritime operations. Evidence-based investigation should result in more successful applications and integration of these new technologies, resulting in enhanced training and operational outcomes. The authors hope that this work will encourage other academics and practitioners to further investigate these technologies within existing MET and operational frameworks.