*Article* **Semantic Modeling of Ship Behavior in Cognitive Space**

**Rongxin Song 1,2,\* , Yuanqiao Wen 3,4,\*, Wei Tao <sup>4</sup> , Qi Zhang <sup>5</sup> , Eleonora Papadimitriou <sup>1</sup> and Pieter van Gelder <sup>1</sup>**


**Abstract:** Ship behavior is the semantic expression of corresponding trajectory in spatial-temporal space. The intelligent identification of ship behavior is critical for safety supervision in the waterborne transport. In particular, the complicated behavior reflects the long-term intentions of a ship, but it is challenging to recognize it automatically for computers without a proper understanding. For this purpose, this study provides a method to model the behavior for computers from the perspective of knowledge modeling that is explainable. Based on our previous work, a semantic model for ship behavior representation is given considering the multi-scale features of ship behavior in cognitive space. Firstly, the multi-scale features of ship behavior are analyzed in spatial-temporal dimension and semantic dimension individually. Then, a method for multi-scale behaviors modeling from the perspective of semantics is determined, which divides the behavior scale into four sub-scales in cognitive space, considering spatial and temporal dimensions: action, activity, process, and event. Furthermore, an ontology model is introduced to construct the multi-scale semantic model for ship behavior, where behaviors with different semantic scales are expressed using the functions of ontology from a microscopic perspective to a macroscopic perspective consecutively. To validate the model, a case study is conducted in which ship behavior with different scales occurred in port water areas. Typical behaviors, which include leveraging the axioms expression and semantic web rule language (SWRL) of the ontology, are then deduced using a reasoner, such as Pellet. The results show that the model is reasonable and feasible to represent multi-scale ship behavior in various scenarios and provides the potential to construct a smart supervision network for maritime authorities.

**Keywords:** semantic modeling; ship behavior; cognitive space; multi-scale analysis; ontology

## **1. Introduction**

There is a high traffic density in some busy waterways, especially in port areas, where some severe situations have occurred. It increases the supervision difficulty to vessels for maritime authorities, such as the Maritime Safety Committee (MSC) and services. Specifically, the supervision to vessels includes static information inquiry, tracking of one or more vessels, ship behavior recognition, etc. The fact facilitates the autonomous supervision to vessels, especially whose behaviors in congestion areas are riskier than in normal areas. The rapid development of Maritime Autonomous Ships (MASS) in recent years has also placed a demand on the autonomous recognition and semantic transformation of ship behavior, which MASS should ideally satisfy to improve the perception of surrounding ship behavior. As a result, more and more researchers are paying attention to the automatic recognition and semantic enrichment of ship behavior.

**Citation:** Song, R.; Wen, Y.; Tao, W.; Zhang, Q.; Papadimitriou, E.; van Gelder, P. Semantic Modeling of Ship Behavior in Cognitive Space. *J. Mar. Sci. Eng.* **2022**, *10*, 1347. https:// doi.org/10.3390/jmse10101347

Academic Editor: Apostolos Papanikolaou

Received: 18 August 2022 Accepted: 17 September 2022 Published: 22 September 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Ship behavior is the representation of the trajectories of ships enriched with various types of semantic attributes. It is challenging to recognize ship behavior without any instructions from the human aspect for computers. In particular, complicated behaviors comprise a set of simple behaviors enriched with geographic attributes, temporal features, motion characteristics, etc. For example, the behavior of anchoring implies the place where the behavior occurred (anchorage), the time duration of the behavior (long-term), and the motion state (stationary). These behaviors are commonly used in realistic scenarios currently and require a deeper study. It is challenging to clarify all of the behaviors by computers without a proper model in which the semantic features of behaviors can be considered in depth. By contrast, a human expert can quickly and precisely understand exactly ship behavior. This is due to the excellent capability of processing information collected from multiple sources in a cognitive space for humans. Such a capability is what is required for the intelligent computers of MSC or MASS.

The semantic modeling of behaviors in cognitive space is a process of semantic reflection of the movement of physical objects, which enables computers to understand behaviors in the same way that humans think. Hence, it is a feasible way to empower a computer to be capable of recognizing behaviors enriched with rich semantics. However, there are a wide variety of behaviors with different semantics, as stated above. It is impractical to program each behavior manually. Thus, a model to extract and collate the semantic characteristics of ship behavior is desired to be provided to reach the final goal of semantic modeling.

This work is based on the previous work [1,2], focusing on modeling and reasoning of semantic ship behavior with different scales in multiple dimensions. We propose a semantic model to extract and recognize multi-scale behaviors automatically in cognitive space based on historical automatic identification system (AIS) data. In this study, the features of ship behavior that represent corresponding trajectories are analyzed from the perspective of spatial-temporal and semantic, respectively. Furthermore, a multi-scale semantic model is given to depict ship behavior in cognitive space, in which behaviors with different spatial scales are sorted out and a formalized cognitive model of behavior is presented. Moreover, by means of the ontology modeling method, multi-scale behaviors are explored and expressed further. Behaviors with different semantic scales are presented, leveraging the functions of ontology. Finally, a case study of a ship approaching into and leaving a port is given to show how the model works.

The remainder of the paper is structured as follows. In Section 2, an overview of related work is given. The analysis of the multi-dimensional feature is in Section 3. The model of multi-scale behaviors is proposed in Section 4. Section 5 constructs an ontology model for multi-scale semantic behaviors. Section 6 presents a case study to validate the feasibility of the semantic model. In Section 7, the results and discussion of the experiment are presented. Finally, the conclusion is given in Section 8.

#### **2. Related Work**

#### *2.1. Cognitive Modeling*

There is extensive literature on the topic of cognition modeling for human behaviors, which are influential in ship behavior modeling in cognitive space. A number of studies have examined the construction method of knowledge base [3–6], and knowledge reasoning [7] with ontology [8–10], which discusses cognitive modeling and knowledge reasoning for human activities. [4,6,8] constructed the ontology based on the relationships between humans and then environment to transform human behaviors from the data layer to the semantic level, which realized the recognition of human behavior intelligently. These studies are beneficial to provide some thoughts about how to construct the cognitive framework of ship behavior.

Currently, there are few works that focus on the research of semantic recognition of ship behavior. [9] designed a method to identify the ship events using AIS data that records relevant information about ship movement, such as position, speed, course, etc. [10] tries to deduce the complicated behaviors based on the method proposed in [9], providing the foundation of cognition modeling of complicated ship behavior.

#### *2.2. Semantic Expressions of Trajectory*

To address the problem of semantic behaviors recognition for computers, from the perspective of humans, some recent studies centered on semantic modeling from a human perspective have proposed semantic computational processing methods [11], using the same approach that humans use to perform semantic recognition of behaviors. Refs. [12,13] proposed a semantic computing frame to compute the trajectory generated by moving objects, such as vehicles, humans, and animals. Based on these efforts, some studies on ship semantic behaviors have also been carried out. A semantic model of ship behavior was proposed in [14], which takes into account the uncertainty of the occurring behavior; [15] mined the pattern of ship trajectories by means of semantic annotation and possibility modeling; and [1] constructed the ontology model of ship behavior, considering the temporal relationships between each other. In addition, some projects work on the semantic computing of trajectories in the maritime domain. An example of this is datAcron [10,16,17], a project focusing on the representation of semantic trajectories of aviation and maritime conceptualizations.

#### *2.3. Ship Behavior Modeling*

In order to model ship behavior explicitly, many studies have been focusing on behavior modeling from trajectory to behavior. There are two kinds of methods used to analyze it: probabilistic statistics and motion characteristics extracting and modeling. The former refers to the pattern mining of ship behavior by means of statistics analysis [18–20]. Another approach based on motion characteristics analysis accomplishes this by analyzing the relationship between the characteristics and behaviors and then modeling. Ref. [21] considered the motion characteristics of ship trajectories to construct the model. Ref. [14] proposed a model for ship behavior based the ship basic behaviors, such as turning to port side, turning to starboard side, and some semantic behavior occurred in the environment.

Few studies on ship behavior undertake basic behavior modeling and prediction, considering the structural and temporal features of complicated behaviors, which are necessary for computers to satisfy the requirements to ascertain a desirable understanding of behaviors. Ref. [2] proposed a framework for ship behavior from a cognitive and semantic modeling perspective and constructed a semantic model to represent behavior from data to trajectory to complex behavior, considering its motion data and environmental attributes.

#### *2.4. Multi-Scale Modeling of Trajectory*

There are extensive studies focusing on the topic of multi-scale characteristics analysis in geography [22,23], which have explored in detail about the multi-scale characteristics of spatial-temporal objects [24–26]. The trajectory, as the representation of the spatial-temporal characteristics of physical objects, exhibits multi-scale characteristics. Previous studies provide a benchmark of multi-scale feature analysis for spatial objects and a solid basis for building a cognitive framework for modeling multi-scale ship behavior. Ref. [27] discussed the multi-scale representation of battlefield situation. Ref. [28] proposed a multi-level model to explore the spatial-temporal patterns of crime in different spatial scales of area. They provide guidance for the construction of cognitive models of ship behavior.

It is necessary to propose a systematic approach to analyze complicated behaviors by comprehensively considering its various characteristics, such as motion characteristics, topological relationships with environmental entities, etc. In general, the modeling of complicated behaviors needs to be considered in different dimensions.

As the semantic representation of ship trajectory, the multi-scale features of ship trajectory can form the multi-scale features of behaviors in three dimensions, such as time, space, and semantics. However, few studies have considered the semantic multi-scale features of ship behavior that are crucial for behavior recognition. The relationship of

ship behavior between different levels and between different scales of the same level has not been constructed properly, which limits the development of the modular computing capability of the autonomous system for the safety supervision of behavior.

To address the problem, Firstly, we characterize ship behavior from the scale of spatialtemporal and analyze the shortcomings of modeling ship behavior in this dimension.

Secondly, we analyze the way ship managers with different cognitive mindsets perceive ship behavior and propose a cognitive model for ship behavior from the semantic dimension, dividing ship behavior into four layers of action, activity, process, and event to describe ship behavior at different spatial-temporal and semantic scales. Finally, the cognitive ontology of ship behavior is constructed, taking the typical behavior of ships in port areas as an example for ontological modeling and expression, and exploring the mechanism of multi-scale semantic expression and reasoning of ship behavior in port waters.

#### **3. Multi-Dimensional Characterization of Ship Behavior in Cognitive Space**

A ship generates a series of trajectory segments driven by the intention of the seafarer. That means that the semantics implied by the trajectory reflects the seafarer intention to navigate. From simple behaviors, such as accelerating and going straight, to advanced behaviors are the semantics implied by a ship's trajectory, such as sailing along the fairway, berthing, etc. In other words, the behavior can be represented as the semantic reflection implied by the trajectories produced by physical objects in cognitive space where human operators process information on their own temporal and logical terms. That is, ship behavior has additional semantic features in addition to the spatial-temporal motion characteristics of ship trajectories. The semantics implied by trajectories are described differently within different spatial-temporal dimensions.

#### *3.1. Previous Work for Semantic Modeling of Ship Behavior*

For semantic modeling of ship behavior, we have explored in our previous studies [2], where a framework of semantic behavior generation process from trajectories enriched with motion semantics and topological environment semantics was given. In this paper, we proposed several concepts, such as atomic trajectory, atomic behavior, topological behavior, as well as traffic behavior, representing the semantic behavior with corresponding semantic features.

Specifically, we first divide the trajectory, generated from AIS data, into atomic trajectories, as trajectory units on the basis of our classification of atomic behavior. Atomic behavior represents the behavior of maintaining a constant motion state of both speed and course simultaneously, as shown in Figure 1. That means the trajectory was segmented according to its motion status instead of sample frequency or spatial grid division with same size, which is beneficial to reduce its computation complexity. Topological semantic enrichment is based on the atomic trajectory. *J. Mar. Sci. Eng.* **2022**, *10*, x FOR PEER REVIEW 5 of 22

**Figure 1.** Classification of ship atomic behaviors adapted with permission from Ref. [2]. **Figure 1.** Classification of ship atomic behaviors adapted with permission from Ref. [2].

Following this, in order to enrich the semantics of geographical properties for trajectory unit, we introduced 15 spatial relations for the calculation of two objects involving point, line, and surface in maritime domain by adapting Dimensionally Extended 9-Intersection Model (DE-9IM) that is proposed for describing spatial relations of two regions. Following this, in order to enrich the semantics of geographical properties for trajectory unit, we introduced 15 spatial relations for the calculation of two objects involving point, line, and surface in maritime domain by adapting Dimensionally Extended 9-Intersection Model (DE-9IM) that is proposed for describing spatial relations of two regions.

resented as formula (1) performing as a sentence, where <sup>a</sup> *Ti* represents atomic trajectory

environment as the object, and <sup>a</sup> *Bi* refers to the atomic behavior as the gerund of the

{ } rao a *TTT r B li jk i* <sup>=</sup> ∩ ∩∩

This model provides a way to reach the goal of semantic unit formation, supporting further high-level semantic modeling for complicated behaviors, which can be represented with a set of traffic behaviors. In addition, we explored the temporal relations preliminarily within complicated behaviors in [1], where we expected to depict complicated

Previous works present how to enrich semantics from different respects to trajectories, but there is a lack of extensive analysis on complicated behaviors, especially the relationships between different dimensions. On the basis of these work, we try to propose an extended semantic model for complicated behaviors combining human cognitive habits.

Based on previous research, we expect to investigate how complicated behaviors can be represented in terms of basic semantic behaviors. Considering the intrinsic spatial-temporal and semantic scale features [29] of complex behaviors, we wish to propose a framework for the analysis of complex behaviors that considers spatial, temporal, and semantic

In terms of the spatial-temporal dimension, ship trajectories as a form of spatial-temporal representation generated by physical objects, the determination of the spatial-temporal scale depends on the frequency with which the trajectories are sampled [30]. Therefore, the sampling frequency and granularity of ship trajectories must be determined when analyzing and modeling ship trajectories at multiple scales purely from the spatial-

However, it is challenging to provide a standard method to determine the scale of the spatial-temporal dimension. Because people with different roles have different concerns about ship behavior, that is not appropriate. Therefore, the modeling of multi-scale features of the track also needs to be reworked around different needs for attention, which presents a higher standard and challenge for the accurate sampling of ship tracks. For

Finally, we presented how traffic behavior, as semantic behavior unit of ships, including motion status and topological semantics, are formed through atomic trajectory,

represents the

(1)

behaviors through combining simple behaviors.

dimensions. Thus, we analyze behavior in three dimensions.

*3.2. Multi-Dimensional Feature Analysis* 

temporal dimension.

sentence, respectively.

as the subject, <sup>o</sup> *Tj* represents topological behavior as the predicate, *kr*

Finally, we presented how traffic behavior, as semantic behavior unit of ships, including motion status and topological semantics, are formed through atomic trajectory, atomic behavior, topological behavior, and environment. Traffic behavior *T* r *l* can be represented as Formula (1) performing as a sentence, where *T* a *i* represents atomic trajectory as the subject, *T* o *j* represents topological behavior as the predicate, *r<sup>k</sup>* ∼∼∼ represents the environment as the object, and - *B* a *i* refers to the atomic behavior as the gerund of the sentence, respectively.

$$T\_l^\mathbf{r} = \left\{ \underline{T\_i^\mathbf{a}} \cap \underline{T\_j^\mathbf{o}} \cap \underset{\sim \sim \sim}{r\_k} \cap [\mathcal{B}\_i^\mathbf{a}] \right\} \tag{1}$$

This model provides a way to reach the goal of semantic unit formation, supporting further high-level semantic modeling for complicated behaviors, which can be represented with a set of traffic behaviors. In addition, we explored the temporal relations preliminarily within complicated behaviors in [1], where we expected to depict complicated behaviors through combining simple behaviors.

Previous works present how to enrich semantics from different respects to trajectories, but there is a lack of extensive analysis on complicated behaviors, especially the relationships between different dimensions. On the basis of these work, we try to propose an extended semantic model for complicated behaviors combining human cognitive habits.

#### *3.2. Multi-Dimensional Feature Analysis*

Based on previous research, we expect to investigate how complicated behaviors can be represented in terms of basic semantic behaviors. Considering the intrinsic spatialtemporal and semantic scale features [29] of complex behaviors, we wish to propose a framework for the analysis of complex behaviors that considers spatial, temporal, and semantic dimensions. Thus, we analyze behavior in three dimensions.

In terms of the spatial-temporal dimension, ship trajectories as a form of spatialtemporal representation generated by physical objects, the determination of the spatialtemporal scale depends on the frequency with which the trajectories are sampled [30]. Therefore, the sampling frequency and granularity of ship trajectories must be determined when analyzing and modeling ship trajectories at multiple scales purely from the spatialtemporal dimension.

However, it is challenging to provide a standard method to determine the scale of the spatial-temporal dimension. Because people with different roles have different concerns about ship behavior, that is not appropriate. Therefore, the modeling of multi-scale features of the track also needs to be reworked around different needs for attention, which presents a higher standard and challenge for the accurate sampling of ship tracks. For example, mariners are more concerned with short-term vessel behavior, such as analyzing whether the target vessel around her is performing the maneuvers specified in COLREG. In contrast, VTS officers are more inclined to obtain a longer range or time interval of behavior, such as analyzing whether vessels within their jurisdiction are engaging in illegal activities. In other words, different people have different scales of attention to the behavior of vessels, involving differences in scale not only in the spatial-temporal dimension but also in the semantic dimension.

Therefore, the analysis of ship behavior should combine the spatial-temporal dimension with the semantic dimension. From a semantic point of view, when modeling ship behavior at multiple scales, we need to describe the behavior semantically in the spatialtemporal dimension at the same time. They need to obtain a good understanding of behavior by dividing the semantic space into several appropriate semantic scales, which are closer to the human habit of perceiving behavior.

#### **4. Multi-Scale Cognitive Modeling of Ship Behavior from Semantic Dimension**

Spatial-temporal data are prevalent with multi-level, multi-grain, and multi-resolution characteristics, and the analysis and extraction of these features is a prerequisite for their awareness and modeling. In addition, the model construction based on these features is also in line with the human cognitive habits of multi-dimensional and multi-features of spatial-temporal data. Therefore, for the spatial-temporal trajectories corresponding to ship behavior, we need to consider these multidimensional features mentioned above and consider the intrinsic relationship of each dimension and the relationship between them. In view of the cognitive habits of people with different roles in the maritime domain for ship behavior, ship behavior can be analyzed and modeled from microscopic scale to macroscopic scale.

#### *4.1. Formalized Cognitive Expression of Ship Behavior*

Behavioral cognition is the result of multifaceted description and expression of ship trajectory. Based on the analysis of cognitive elements, the cognitive expression of ship behavior, *Cog*, should be considered as a cognitive set, including four elements: who, what, when, and where, which can be expressed as Equation (2).

$$\text{Cog} = \{\mathbf{o}, \mathbf{b}, \mathbf{t}, \mathbf{p}\} \tag{2}$$

where **o** denotes the object where the behavior occurs; **b** is the behavior that occurs at the object; **t** represents the time, including instant and interval; and **p** is the place where the behavior occurs.

Considering the multi-scale characteristics of spatial-temporal trajectories, this paper divides the cognition of ship behavior into four layers in the cognitive space: action, activity, process, and event, according to the expression habit of ship behavior in the semantic dimension. The division of behavior cognition is based on two aspects, including motion features and the topological features.
