Applications of Multi-Agent Systems in Unmanned Surface Vessels

Abstract

The comprehensive and safe application of unmanned surface vessels is certainly one of the biggest challenges currently facing maritime science. Such vessels can be implemented within a wide range of autonomy levels that goes from remote-controlled vessels to fully autonomous vessels in which intelligent vessel systems completely perform all necessary operations. One of the ways to achieve autonomous vessel systems is to implement multi-agent systems that take over all functions performed by the crew in classical manned crew vessels. A vessel is a complex system that conceptually can be considered as a set of interconnected subsystems. Theoretically, the functions of these subsystems could be performed using appropriate multi-agent systems. In this paper we analyzed 24 relevant papers. A review of the current state of implementation of multi-agent systems for performing the functions of unmanned surface vessels is presented.

1. Introduction

Unmanned vessels are one of today’s trends in maritime affairs. We are witnessing numerous projects of unmanned vessels for activities such as civil, military, and research purposes. The history of attempts to achieve unmanned vessels is longer than 130 years. Nikola Tesla created the first model of a miniature remote-controlled ship [1,2]. He presented this model at Madison Square Garden in New York at the exposition in 1898. As with all his inventions, Tesla was ahead of his time. This prototype was actually the first unmanned vessel, but the development of such vessels should have waited some time. This time was needed to develop appropriate supporting technologies that would make the idea of unmanned vessels interesting to maritime industry. Today, the maritime industry is preparing for a time when human presence on ships will no longer be required. The reasons for this trend can be divided into two groups. The first group includes financial reasons, which can be seen in reducing operating costs and increasing the carrying capacity of ships due to the absence of human crews. The second group relates to security reasons. A large number of maritime incidents are caused by human factors. Therefore, it can be anticipated that the reduction in the number of people on board, and eventually the complete absence of human crews on board, will reduce the number of marine accidents. However, it is also necessary to consider which classes of security challenges this new approach to maritime will bring. When designing unmanned surface vessels, it is necessary to harmonize the designed solutions with the official documents regulating this area. In this case, these are conventions and standards such as the International Regulations for Preventing Collisions at Sea (COLREG) and the International Convention for the Safety of Life at Sea (SOLAS) imposed by the International Maritime Organization (IMO), as well as all other relevant official documents. Failure to comply with these rules is a frequent cause of a ship collisions. Prior to the introduction of modern navigation systems, 56% of ship collisions were caused by the failure to comply with COLREG [3].

There are several degrees of vessel autonomy. At the lowest level, only individual processes are automatized, and the vessel needs a human crew. At the two next levels, autonomy is enabled by remote control. The autonomy of the vessel increases as the level of remote control of the vessel increases. The highest level of unmanned vessel includes vessels with control system that can make and implement decisions fully autonomously. A vessel is a complex system that consists of different subsystems. Each of these subsystems should be made autonomous in order to develop a fully autonomous vessel. The question is how to conceptualize such a complex system, how to implement it, and how to evaluate it. The use of intelligent agents in addressing these challenges is an increasingly frequent topic in papers published by relevant authors in this field. Intelligent agents are autonomous entities that perceive the state of their environment, interpret perceptions, and act within the environment in accordance with their goals. The elements of an intelligent agent that allow it to perceive the environment are called agent sensors. On the other hand, effectors are those elements that enable an intelligent agent to operate within the environment. During the implementation of intelligent agents into existing vessels, it is possible to use parts of the existing ship equipment as the agent’s sensors or effectors. In order to increase efficiency, intelligent agents are organized as multi-agent systems, i.e., the systems in which agents act cooperatively to accomplish common goals.

There are numerous possibilities for the use of intelligent agents in areas such as transportation and logistics [4], design of critical systems [5], manufacturing [6], tourism [7], or education [8]. The aim of this paper is the review of the current state of research in the implementation of multi-agent systems in unmanned vessels, i.e., the review of advancement in unmanned vessels functionalities using multi-agent systems. Some of research described in this review uses a single agent solution. However, we included them because any use of a single agent solution in a group of vessels must be considered as a multi-agent system, i.e., the group has all multi-agent system properties. Moreover, two additional questions will be considered. The first concerns the answer to the question of whether any research includes mechanisms for interoperability between different systems of unmanned surface vessels. The second question concerns an existence of metrics that would establish a relevant comparison system of different multi-agent ship systems of the same purpose.

The paper is organized as follows. Section 2 and Section 3 give brief descriptions of unmanned surface vessels and intelligent agents, respectively. Section 4 gives review of 24 papers that use multi-agent systems to achieve any functionality of the unmanned surface vessels. In Section 5 discussion is given.

2. Unmanned Surface Vessels

The increasing maritime traffic is demanding a global increase in the number of people employed on ships of any kind. This particularly applies to experienced officers on board. Technologically advanced modern ships emphasize this problem even more. Nowadays in shipping, we have on one side the lack of experienced crews, and on the other side the growth of freight transport demands. A lack of manpower is not the only problem of maritime affairs. It is often the human factor that causes many unwanted situations at sea. The problem with human crews on ships is that they are the biggest causes of potentially devastating situations that can occur due to wrong decisions, violation of defined procedures, fatigue, or due to lack of practice [9]. Due to the development of appropriate technologies, one of the ways to solve the above-mentioned problems is the development and implementation of unmanned vessels. Such vessels are not yet commercially used. It can be expected that the technologies of unmanned vessels first will be used for military purposes. So far, the military industry is usually the first that has practical applications of new technologies. The same thing happens in unmanned vessel technologies. For example, the U.S. Navy announced 6 years ago that it has ready technology for small unmanned ships that are ready for interoperability with ships with crews [10]. The first commercial unmanned ships are not expected before 2035, but by introducing them, maritime traffic will enter a new era where conventional ships with crews and unmanned ships will sail within the same waters simultaneously [11].

The most frequently referenced definition of an unmanned vessel is the definition used by the International Maritime Organization (IMO). According to this definition, an unmanned vessel is any vessel that can operate up to a certain level without the direct participation of people [12]. The same organization also proposed a classification of the autonomy of vessels that contain at least some degree of autonomy. This classification is shown in Figure 1. At the lowest level there are those vessels that have at least some automated decision-making systems and the crew operates their functions. At the next level, there are remote-controlled manned vessels, followed by the level with remote-controlled unmanned vessels. At the highest level of autonomy are vessels whose systems are fully capable of operating the vessel.

There are other classifications of this area. For example, according to the Lloyd’s Register definition, an autonomous high-level ship should have the capability to autonomously cross the ocean, performing assessment of navigational states, identification of navigation risk, and decision making in real-time [13]. Regardless of the classification used, it is possible to classify: fully manned vessels, remotely operated vessels, and autonomous vessels.

The remotely controlled vessels are therefore a phase between fully manned and autonomous vessels. Very high requirements will continue to be set to a crew that will control a vessel remotely [14]. These requirements are expected because the vessels will still have to operate their safely navigate high seas. These vessels shall be subject to requirements affecting their ability to sail, maneuver, locate, monitor, and operate the propulsion system, as well as to any other subsystems of the vessel necessary for the safe conduct of the voyage. When considering safety issues for unmanned vessels, some new issues, such as cyber-attacks or systemic errors resulting from unforeseen system behaviors, must also be raised [15]. An additional challenge facing unmanned vessels is to operate within a highly dynamic environment where vessels with a human crew can also be found. This is important because human reactions can sometimes leave the zone of expected actions and the management systems of unmanned vessels need to be prepared for such situations.

3. The Concept of an Agent and Multi-Agent Systems

The concept of an agent was in artificial intelligence for approximately three decades. An agent theory describes what an agent is, what its properties and internal structure are, and how they are connected. In order to do this, an agent theory uses a mathematical formalism for representing an agent, its behavior, and reasoning. There is no unique definition of an agent. Several definitions are proposed. Three definitions are described below, i.e., definitions given by Wooldridge [16], Ferber [17], and Russel and Norvig [18].

Wooldridge [16] describes an agent as a computer system that is situated in some environment, and that is capable of autonomous action in this environment in order to meet its design objectives. This definition does not include intelligence. An agent is intelligent if it is capable of flexible autonomous action, where flexible mean that beside autonomy, an agent also needs to possess the following characteristics: reactivity, proactiveness, and social ability. Reactivity means that an agent can perceive its environment and has a real-time response to the observed changes. Proactiveness is an ability of an agent to exhibit goal-directed behavior that leads to the achievement of its designed objectives by taking the initiative. Social ability is a characteristic of an agent to interact with other agents (and possibly humans) in order to satisfy its designed objectives.

Ferber [17] defines an agent as a physical or virtual entity which is capable of acting in an environment and can communicate with other agents. An agent is endowed with autonomy, i.e., it is not driven by commands of the user or other agents but by a set of tendencies, which can take a form of individual objectives or of satisfaction/survival functions. It possesses own resources. An agent is able to perceive its environment to a limited extent and has only a partial representation of this environment. An agent has skills and can offer services and may be able to reproduce itself. Ferber represents an agent as a ‘living organism’ that behaves in such way to attain objectives on the basis of all available elements, i.e., perceptions, representations, actions, communications, resources, and skills.

There are two concepts of agents according to the how the knowledge is represented [17]. The first types are cognitive agents. They have symbolic and explicit representation of the world on the basis of which they can reason. They have a knowledge base with data, know how to complete tasks, and how to communicate with other agents and environment. If they have goals and explicit plans on how to achieve their goals, they are intentional agents. A well-known type of cognitive agents are the belief–desire–intention (BDI) agents [19,20,21]. They have mental attitudes: beliefs, desires, and intentions. Beliefs represent the informational state of an agent about itself, other agents, and the environment. Desires describe what the agent’s motivations and goals are. Intentions are a result of deliberation. They lead to action. The strength of this type of agents is that they are inspired by the human concept of knowledge and deliberation. This property makes them simple and intuitive for understanding their internal structure. The second types are reactive agents. Their representation of knowledge is situated at a sub-symbolic level; they have no planning mechanism or explanation of goals. An internal representation is numerical, so they use optimization methods for exploring parameter space. This type of agent originates from the study of animal behavior [22,23]. They emulate the type of animal behavior that is simple and governed by simple rules, e.g., an ant colony, a flock of birds, or fish. The strength of this type of agent is not an individual agent, but a large number of agents with their property of self-organization that leads to emergent behavior that might be described as intelligent, i.e., intelligence of the group. Ferber [17] also divides agents according to their modes of conduct on teleonomic and reflexes agents. Teleonomic agents are directed towards explicit goals expressed within the agents. A reflex agent behavior is regulated by perceptions obtained by environment.

This classification is simplified and divides agents in two completely different types. In multi-agent systems, these types can be combined in different ways depending on the problem domain, i.e., hybrid agent architecture can be developed or a hybrid multi-agent system can be created.

Russel and Norvig [18] define an agent as anything that perceives its environment through sensors and acts on that environment through actuators. An agent behavior is described by the agent function that is implemented by an agent program. The agent function maps percept sequence to an action. There exist many types of agent programs design and they depend on the nature of the environment.

Russel and Norvig [18] describe four basic kinds of agents: simple reflex agents (directly respond to percepts), model-based reflex agents (have an internal state that depends on the percept history, combines current percept with internal state, updates current state, and acts according to that), goal-based agents (act to achieve their goals), and utility-based agents (try to maximize their own utility function). All four kinds of agents can be the foundation for learning agents because they can improve their performance through learning.

As we saw from the definitions above, the concept of agent is mostly generic but gives the opportunity to be broadly applied to many different domains. Additionally, it is obvious that a single agent cannot make significant change in an environment in which it acts and consequently cannot be interesting for implementation. Therefore, agents are organized in multi-agent systems in order to resolve problem in different domains. To achieve common goals agents in multi-agent system must interact with each other. The interaction may be simple or complex, as well as may be achieved directly or through the environment. So far, numerous multi-agent systems are developed. In this paper we are interested in the use of multi-agent systems in unmanned surface vessels.

4. The Use of Intelligent Agents in Unmanned Surface Vessels

When considering multi-agent unmanned vessels, the multi-agent system ideally should be able to fully assume the role of a human crew member. Regardless of the ship’s functionality, the multi-agent system needs to take over each of the following four functional classes: (i) information acquisition; (ii) information analysis; (iii) decision and action selection, and (iv) action implementation [24]. Figure 2 shows the structure of an agent through these functional classes. Performance of information acquisition is the task of agent sensors. Agent knowledge shall contain the knowledge necessary to perform information analysis as well as decision and action selection. Finally, agent effectors perform actions. When considering the operation of the multi-agent system on board, it is necessary to conceptualize, as parts of the multi-agent system, all those parts of the ship’s equipment over which the agent may have control. Parts of the ship’s equipment that can acquire different states of its environment, e.g., smoke detector, echo sounder, etc., may serve as agent sensors. Furthermore, all those parts that can perform actions can be conceptualized as agent effectors.

The following is a review of research papers from the scope of use of multi-agent systems to achieve the functionality of unmanned surface vessels. The presented papers were obtained using search tools with keywords unmanned (or autonomous), vessel (or ship), and agent (or multiagent, multi-agent).

The agents were first used in marine research for different simulation purposes. Simulations enable testing of various items which, in a real situation, would be too dangerous or too expensive to perform, and sometimes impossible to perform. In the case of maritime research, simulations made it possible to test scenarios to raise the level of safety and functionality of ship transport. With further development of simulation platforms, such environments are also used for design and evaluation of entire systems, including systems of unmanned vessels.

Liu, Sun, and Du [25], in their work, propose the model of intelligent agent for simulating ship collision avoidance. Within this model, they defined seven basic components that such an intelligent agent should have. Each one of these basic components is specialized in performing one function. Cooperative activities of these components result in the following functions: perception, memory, communication, emotion, action, learning, and thinking. By using these functions, a ship conceptualized through an agent model is able to achieve functionalities, such as: detecting changes in the environment, gathering information about other vessels in the environment, judging the degree of danger of the current situation, communicating with other participants within the environment, making knowledge-based decisions, and acting to avoid collisions with other vessels or obstacles. With this approach, the target ship assumes the characteristics of an intelligent agent because this ship acts autonomously and adaptably.

Han, Zhang, Wang, Luo, Ran, and Xu [26] propose the multi-agent model for training unmanned surface vessels. In their model, agents are designed as the physical controllers for each unmanned surface vehicle. Within the model, an agent cooperation method is defined. The authors designed the multi-agent-based learning system to train unmanned surface vehicles. A system simulation was carried out to verify its effectiveness. This simulation showed that the proposed model is feasible and can be transformed into a policy for selecting the actions of a team of unmanned surface vehicles.

The focus of work of Xiao, Fu, Zhang, Agarwal, and Goh [27] is on planning, modelling, and testing autonomous vessels. Their model can be clearly divided into the following layers: (i) modelling analytics (sea way capacity evaluation, situational awareness, scenario evaluation, risk evaluation, and high-level decision making); (ii) multi-agent system modelling (dynamic multi-agent modelling, monitor runtime intermediate data, and generate modelling output) and (iii) data and knowledge (ENC data, AIS and radar data, regulation, spatial environment data, and other maritime transport data). The authors consider the process of autonomous maritime transport as a chain consisting of path planning, monitoring the situation, and making decisions according to the current state of the environment.

Autonomous navigation is the functionality most often referred to as the functionality taken over by the agent-based systems in autonomous ships. Navigation is a complex process that within itself possesses many requirements that need to be accomplished. These requirements represent functionalities such as avoiding collisions, path planning, detecting anomalies, or classifying objects of the vessel environment. The accomplishment of these functionalities by agents can be achieved through learning based on data on previous reactions of vessels in different situations. For the purpose of the above-mentioned training of agent-based systems, deep learning algorithms are most commonly used. The need to achieve autonomous navigation resulted in a greater number of publications dealing with appropriate deep learning methods. Thus, within the fifteen-year period (2003–2018), the number of publications discussing deep learning-based navigation methods increased 9 times. In the same period, the number of publications processing deep learning in ship collision avoidance increased 5 times and the number of publications researching deep learning for anomaly detection increased 15 times [28].

Perera [3] proposes a collision avoidance model within the framework for the autonomous navigation of ships. In the base of this model is an agent system based on deep learning systems. Deep learning systems would learn from the behaviors of the crews of nowadays ships, for example, they would learn vessel management by observing the actions of the shipyards. Similar methods are already used in practice. This concerns in particular the resolution of autonomy problems in related areas, such as the autonomy of road vehicles and drones. Nowadays, these methods can be adapted to unmanned surface vessels.

By analyzing COLREG rules, Perera finds potential regulatory problems related to the COLREG application to autonomous vessels. This author also suggests possible solutions to the discovered deficiencies of COLREG. A structured technology framework proposed by Perera would possess decision support layer supported by Internet of Things (IoT) elements that would provide the system with relevant data about the environment. He also stated that procedures for checking such autonomous systems should be established. Of course, all these checks must be coordinated with the appropriate public authorities. Perera, in collaboration with Murray, suggests the method for maritime situation control based on the prediction of ship behavior via clustering data extracted from ship trajectory in an observed geographical region [29,30]. The method uses machine learning techniques to extrapolate those segments of the trajectory that are relevant for the current calculation.

Azimi, Salokannel, Lafond, Lilius, Salokorpi, and Porres [31], in their navigational model, recommend a combination of reinforcement learning and imitation learning. Reinforcement learning models identify the main elements of the problem understudy by analyzing how an agent interacts with its environment to reach a maximum reward while pursuing an ultimate goal. The agent derives information from its environment in order to perform actions which have impact based on a predefined reward. The imitation learning is based on data collected from experienced human behavior in order to teach the system to imitate an expert behavior. The authors combine reinforcement and imitation in their agent-based simulation environment to investigate the safety of this approach and the advantages of this model in relation to traditional search-based planning and optimization algorithms.

In their approach to collision avoidance methods, authors Chen Chen, Ma, Xu, Yuwang Chen, and Wang [32] model the system as the multi-agent system. In their modelling approach, they do not start from an individual ship but observe a group of ships in a determined area. Each ship is joined by an agent. The avoidance of collision takes place through cooperation between these agents, and the applied actions are mutually decided (head-on, overtaking, or crossing). The method by which the system learns is multi-agent deep reinforcement learning.

Koznowski and Lebkovski [33] defined the multi-agent system for the control of unmanned port tugboats. This system achieves interoperability within the formation of tugboats when performing joint port operations, such as assistance in ship maneuvers, monitoring and patrolling port areas, conducting port inspections, and assistance in the resolution of marine pollution. Here, decision making is centralized, where decision making is not at the level of tugboat, but every tugboat is part of the team. Within the multi-agent system, the functionality of the interoperable action of tugboats is achieved using control algorithms and evolutionary algorithms. The authors also developed an appropriate simulation environment, which explores possible scenarios of synergy of tugboats in performing port operations.

Wu, Lei, He, Zhang, and Ji [34] also use deep reinforcement learning within their agent model. They use this model in order to solve the optimization problem of unnamed ship path planning. The waiting times at the defined port places are the values that are used for the path optimization in the model. In accordance with the reinforcement learning methodology, the learning agent interacts with the environment and gains knowledge of the best possible action.

The application of deep reinforcement learning to unmanned vessels is also the subject of research by Guo, Zhang, Zheng, and Du [35]. They modified the deep reinforcement learning method to achieve an agent-based path planning model for unmanned ships.

Xiao, Ligteringen, Gulijk, and Ale [36] defined the model for predicting ship behavior, which is based on previous behavior data of the observed ship. Data of ship movements are obtained from the automatic identification system (AIS). Based on these data, a mathematical model that provides information about the probability of future ship behaviors was created. Using this data, an autonomous ship can adjust actions according to calculated probability of other ship behavior that shares the same environment.

The next area of research of the multi-agent system application in unnamed surface vessels is safety. The range of potential safety threats to the ship is broad. Kim, Perera, Sollid, Batalden, and Sydnes [37] define challenges that are related to autonomous ship safety. According to them, the safety of autonomous ships should be considered through the following eight categories: navigational safety, ship system safety, ship structural safety, personnel safety (due to different degrees of autonomy, it is also necessary to analyze this category), equipment safety, security (piracy, cyberattacks, illegal boarding, and robbery), cargo safety, and onboard emergency management (fire extinguishing, chemical and biological issues, and emergency evacuation). It is evident from this categorization how wide the area of onboard safety is. The same goes for agent systems that support safety on board.

Under conditions where manned vessels and unmanned vessels are in use, it is necessary to further improve maritime search and rescue. It is necessary to have ready methods and procedures enabling the prompt and efficient undertaking of necessary actions in case of any need to search and rescue at sea. This is the focus of the Yu and Xue research [38]. They introduced the architecture of the multi-agent-based intelligent decision support system of maritime search and rescue. According to characteristics and procedures of the search and rescue, the whole decision process is divided into a few threads. Each thread is accomplished by one agent. These agents are: the main control agent, distress information processing agent, accident situation evaluation agent, search and rescue supporting information processing agent, search area determining agent, search and rescue resources selecting and optimizing agent, and shipwreck data mining agent. Testing of prototype systems shows that it is possible to effectively replace human decision making in search and rescue processes. The system proved credibility for decision making in the following tasks: (i) assessing disaster risk; (ii) operating in an environment where a quick response in critical situations is needed; and (iii) assessing the necessary search and rescue resources.

Jakob, Vanek, Hrstka, and Pechoucek [39] propose the multi-agent model, the main task of which is to reduce the risk of pirate attacks at sea. Pirate attacks are a serious security problem in some seas. When solving this problem, the authors not only address unmanned ships, but suggest a model that is applicable to any ship, regardless of its degree of autonomy. The model recognizes three different types of agents: merchant ships, pirate ships, and navy warships. The reduction in risk of pirated attacks is achieved by recommending the following actions: recommending transit corridors, group transit (convoys), route randomization, and recommendations of patrol actions. Of course, this model cannot be achieved without integrating a wide range of data from the real world. Therefore, this emphasizes the need for the coordination of various systems, both in trade and military navies.

Sumic, Males, and Rosic [40,41] introduced a topic of the multi-agent system-based fire protection of autonomous ships. They developed the multi-agent ship firefighting model, where the agents take over fire control of every room of a ship. Each room of a ship was assigned its own agent. This agent has access to the firefighting elements contained in his assigned room. These elements include smoke detectors, flame detectors, IR cameras, fire foam generators, and similar elements. These elements serve as sensors and effectors through which the agent receives information from its environment and takes actions that it concludes are necessary to execute. An appropriate simulation environment was developed and the defined model was tested.

The supervision of numerous onboard systems is an area where the application of agents could be useful in unmanned vessels. Perera suggests using agents for condition monitoring [3]. Condition monitoring implies monitoring of all ship functions to predict the need for maintenance. The idea is that the agent initiates maintenance through the condition-based maintenance system in case of the conclusion that a system requires maintenance. In this approach, the question arises when an urgent repair of a system is required while the vessel is, for example, in the middle of the ocean. One possible answer to this question is to achieve interoperability with robotic systems that could be used in situations such as the situation mentioned above.

When considering unmanned surface vessels, it is necessary to research an infrastructure that will support such vessels. Marine surveillance systems are an important part of this infrastructure. These systems are particularly important in the areas of dense traffic. Many vessels, and the different ways these vessels behave, create a dynamic and complex system that needs to be carefully monitored. Multi-agent systems can also have a significant application in this field.

Mano, George, and Gleizes developed the adaptive multi-agent model for maritime surveillance [42]. In this model, each vessel is represented by one agent. Each agent monitors local anomalies to react appropriately in case it concludes that intervention is necessary. The causes of ship behavior anomalies can vary from criminal intent to various malfunctions.

A kind of continuation of the previous research is the research conducted by Brax, Andonoff, and Gleizes. They proposed a multi-agent system designed to observe the abnormal behavior of ships [43]. The developed system proposed by these authors is not intended for independent decision making, but serves as a means of helping human operators in early warnings about potential problems. The human operator shall decide on the action to be taken. This information is feedback to the multi-agent system. Based on feedback, the multi-agent system adapts its future alerts related to the potential ship’s abnormal behaviors.

Singh, Nguyen, Kumar, and Lau also assign an agent to each ship within their multiagent model [44]. The purpose of their model is the management of maritime transport in a busy waterway environment. In order to avoid congestions, the system based on this model generates the speed recommendation for each vessel in the environment.

Some authors approach the problem of unmanned surface vessels through robotic technology. A certain number of such approaches can also be considered as an agent approach if the robot is considered as a physical manifestation of an agent and artificial intelligence methods are used. Often, these systems are operationalized through the cooperative activity of a large number of robots, i.e., multiple unmanned surface vessels. Song and Chen [45] conceptualize multiple unmanned surface vessels as cognitive agents who, besides sensors and effectors, possess the following modules: (i) user interface and communication module; (ii) learning module; (iii) module for modelling environmental information; and (iv) decision-making module. Luo, Bae, Min, and Kim [46,47] model multiple unmanned surface vessels for the purpose of applying teams of robotic vessels in environmental operations, such as oil cleaning operations and other sea protection operations.

Finally, two more cases will be listed. Both cases are specific when compared to previously described research. In the first one, the management of groups of fishing boats is considered, while the second refers to the problem of avoiding the destruction of the marine biosystem.

In their research, Vanhée, Borit, and Santos [48] consider a particular type of autonomous vessel, namely autonomous fishing vessels. The focus of their work is the application of multi-agent systems in autonomous fishing vessels. Their goal is the definition of autonomous fishing operation systems. This system is modeled as a multi-agent system and its task is to perform actions necessary for integrated fishing operations, i.e., search for suitable fishing locations, operations with fishing nets, and the storage and transport of the fish caught.

The multi-agent model 3MTSim is developed to simulate the spatiotemporal movement of marine mammals and maritime traffic [49]. It is based on existing telemetry data on fin, blue, and beluga whales, as well as on land-based tracking of humpback and minke whales in the St. Lawrence Estuary in Canada. This is the decision–support tool to inform management personnel in the estuary. This tool also can be used in autonomous ships passing an ecologically sensitive area, i.e., an area populated by protected marine species. The model represents the decision-making process as a function of environmental conditions, the contextual setting, and objectives that are set. This paper also shows how wide the planning process has to be when designing unmanned vessels. It is not expected that each unmanned vessel must contain an implemented system enabling the avoidance of disturbances of local marine species. This functionality can be provided as an external service to vessels when passing through bio-sensitive areas.

5. Discussion

A total of 24 papers examining multi-agent systems as facilitators of unmanned surface vessel functions are presented. A summary of the main contribution of these papers is given in

Table 1. Main contributions of the presented papers.

Author(s)	Publication Year	Contributions
Liu, Sun, Du [25]	2007	Model for simulating the ship collision avoidance
Mano, George and Gleizes [42]	2010	Adaptive model for the maritime surveillance
Parrott, Chion, Martins, Lamontagne, Turgeon, Landry, Zhens, Marceau, Michaud, Cantin, Menard, Dionne [49]	2010	Model of the marine mammals’ movements and the maritime traffic interaction
Yu, Xue [38]	2010	Intelligent decision support system of the maritime search and rescue
Brax, Andonoff and Gleizes [43]	2012	Model for observing the abnormal behavior of ships
Xiao, Ligteringen, Gulijk, Ale [36]	2012	Model for the ship behavior prediction
Jakob, Vanek, Hrstka, Pechoucek [39]	2012	Model for reducing the risk of piracy at sea
Vanhée, Borit, Santos [48]	2018	Model of the autonomous fishing vessels
Luo, Bae, Min, Kim [46,47]	2018, 2020	Environmental operations based on multi unmanned surface vessels
Singh, Nguyen, Kumar, Lau [44]	2019	Model for the management of maritime transport
Han, Zhang, Wang, Luo, Ran, Xu [26]	2019	Training model for the unmanned surface vessels
Xiao, Fu, Liye Zhang, Wanbing Zhang, Agarwal, Goh [27]	2019	Framework for design, planning, modelling, and evaluation of the autonomous shipping systems
Perera [3]	2020	Framework for the autonomous navigation of ships; collision avoidance
Perera [3]	2020	Model of monitoring all ship functions to predict the need for maintenance
Azimi, Salokannel, Lafond, Lilius, Salokorpi, Porres [31]	2020	Ship navigation model based on combination of reinforcement learning and imitation learning
Guo, Xiuguo Zhang, Yisong Zheng, Du [35]	2020	Model of the vessel path planning based on deep reinforcement learning
Murray, Perera [29,30]	2021, 2022	Model of the maritime situation control based on prediction of a ship trajectory
Koznowski, Lebkowski [33]	2021	Model for the control of unmanned port tugboats
Song, Chen [45]	2021	Conceptualization of multi unmanned surface vessels through the system of cognitive agents
Chen Chen, Ma, Xu, Yuwang Chen, Jin Wang [32]	2021	Multi-ship cooperative collision avoidance based on deep reinforcement learning
Sumic, Males, Rosic [40,41]	2021	Model for the autonomous ship firefighting
Wu, Lei, He, Zhang, Ji [34]	2022	Model for the path planning optimization based on deep reinforcement learning

.

It is shown that the most frequent application of agents in unmanned surface vessels is in navigation, since a total of nine papers directly consider that purpose. The number of papers engaged in the navigation of unmanned vessels is not exhausted. Two out of three papers presented here that are classified in the field of simulations consider navigational problems through simulations. By adding these two papers, the number of papers directly or indirectly engaged in navigation of unmanned vessels almost reaches half of all papers considered.

Safety is one of the essential elements of unmanned surface vessels. Among the presented papers, three papers belong directly to the safety category. These are the papers that consider reducing the risk of pirated attacks on the sea, fire protection of autonomous vessels, and agent-based search and rescue operations at sea. All works categorized in the field of navigation could also be categorized in the field of safety. This is understandable because, for example, every system that covers navigation must have built-in methods to avoid collisions, which is one of the functions that must be covered by vessel safety systems. However, the presented papers show that there is enormous space for additional application of intelligent agents in the field of unmanned surface vessel safety. For example, there is no appropriate multi-agent system solution for avoiding, detecting, and responding to events such as grounding, engine errors, propulsion system errors, hull damage, problems with ship stability, failure of any ship subsystem, cargo problems, or cyber-attacks on the ship’s systems. Intelligent agents could also be used in the monitoring and management of ship systems such as the engine, propulsion system, electrical system, communication system, and IT system. Only one paper was found to discuss the problem of vessel system monitoring and control.

Three papers on maritime surveillance based on agents are also presented. Maritime surveillance is not a direct function of a vessel, but this activity is essential for the sea traffic. This importance is even more emphasized in the hybrid environment, where vessels of different degrees of autonomy navigate in the same area, starting from vessels with a human crew, through remote-controlled vessels, to fully autonomous unmanned vessels.

There are two papers in the group of elaborated papers that were singled out from the previously analyzed topics. The first paper topic is about the coordination of unmanned fishing boats, and the second covers the avoidance of sensitive marine species that may inhabit areas of navigation. These papers are an indicator of the interdisciplinary potential of the area, i.e., show how interesting it can be to extend the research of multi-agent system applications in unnamed surface vessels to other related areas.

It should be noted that none of the presented papers discuss the interoperability between several multi-agent systems on unmanned surface vessels. It is reasonable to assume that the interoperability of such systems is necessary in order to achieve the full functionality of autonomous unmanned surface vessels. This is difficult to achieve without a common framework that would define common ontology, communication, and cooperation procedures for the multi-agent systems installed on the vessel.

6. Conclusions

The analyzed papers show how wide the application area of multi-agent systems in unmanned surface vessels is. An interoperability of different multi-agent ship systems is not presented in any of the research.

Finally, it is reasonable to consider advantages and disadvantages of the methods, models, or systems presented here. This is not possible now for the following reasons:

The first reason is the great heterogeneity of the analyzed area. It is unnecessary to compare, for example, a multi-agent system designed to extinguish ship fires with a multi-agent system designed to achieve autonomous navigation of a vessel. In order to precisely detect advantages and disadvantages of a model, it is necessary to compare that model with another model of the same functionality. Each of the approaches considered here achieves some progress comparing with human-based solutions. Furthermore, in analyzed papers, comparisons of the presented model with similar models are not given.

The second reason follows from the first, and that is the absence of appropriate metrics. Different metrics should be introduced for each of the different scopes presented here. For example, a special metric should be introduced to measure the efficiency of fire extinguishing on vessels, and a special metric to measure navigation issues of vessels. One way to achieve these metrics is to establish appropriate benchmark data bases. The benchmark data bases should contain problem cases that proposed models should solve. This requires the multidisciplinary approach and cooperation of various specialists.

It can be concluded that in the future we can expect more papers on the topic discussed here. An increase in the number of published papers related to agent-based navigation is to be expected, as well as the papers related to the less explored parts of this area. More papers related to interdisciplinary topics should also be expected. All this research effort is in order to achieve fully autonomous unmanned surface vessels in the near future.