Remote Sensing for Maritime Prompt Monitoring

Abstract

The main purpose of this paper is to describe a software platform dedicated to sea surveillance, capable of detecting and identifying illegal maritime traffic. This platform results from the cascade pipeline of several image processing algorithms that input Radar or Optical imagery captured by satellite-borne sensors and try to identify vessel targets in the scene and provide quantitative descriptors about their shape and motion. This platform is innovative since it integrates in its architecture heterogeneous data and data processing solutions with the goal of identifying navigating vessels in a unique and completely automatic processing streamline. More in detail, the processing chain consists of: (i) the detection of target vessels in an input map; (ii) the estimation of each vessel’s most descriptive geometrical and scatterometric (for radar images) features; (iii) the estimation of the kinematics of each vessel; (iv) the prediction of each vessel’s forthcoming route; and (v) the visualization of the results in a dedicated webGIS interface. The resulting platform represents a novel tool to counteract unauthorized fishing and tackle irregular migration and the related smuggling activities.

1. Introduction

Maritime traffic consists of more than 600,000 vessels navigating daily all over the world seas. This huge network of vehicles poses hard issues for what concerns monitoring and timely countermeasures implementation. Satellite missions rotating on sun-synchronous Low Earth Orbit (LEO) trajectories provide heterogeneous remote-sensing data about the sea surface status on a daily basis and at diverse resolutions. This information plays a crucial role for vessel traffic monitoring, since remote sensed data provide information about the vessel main features, its peculiar geometry and even its kinematics.

The main goal addressed in this paper concerns the description of a platform dedicated to sea surveillance, capable of detecting and identifying seagoing vessels. This platform is in charge of collecting and integrating data made available by multi-source, multi-sensor satellite missions for specific maritime areas of interest. High-resolution data are provided by currently orbiting missions such as ESA (European Space Agency) Copernicus Sentinel, ISI-IMAGESAT EROS-B and ASI (Italian Space Agency) COSMO-SkyMed constellation. These data are processed to detect and identify the ships located in the area of interest and to provide estimates of meaningful features, such as shape and kinematic parameters.

The implemented platform has been conceived as a sequential chain of procedures, each addressing a specific task oriented to the extraction of meaningful information (see Figure 1). First, the input radar and optical images are processed by a dedicated algorithm that attempts to discern between sea clutter, noise background and anomalous signals possibly revealing the presence of vessels. The output of this procedure is a set of submaps, each cropped to only contain a single target. Each such submap is then fed to a ship segmentation module in charge of refining the identification of the pixels expected to belong to the target. Accordingly, this operation provides additional information concerning the target, such as its length overall, beam overall and heading (possibly estimated up to a ${180}^{\circ }$ ambiguity). The geometrical and morphological information provided by the latter module can further be enriched by inspecting the water surface surrounding the ship. Indeed, it is known that the ship’s passage through the water generates a wake pattern whose peculiar visual features depend directly on the ship’s kinematics. In particular, by detecting the linear envelopes of the main wake components and performing a frequency analysis of their observable oscillations, it is possible to estimate, respectively, the ship’s heading (univocally) and the ship’s velocity module. The estimated ship’s geometry and kinematics are then exploited to predict the ship’s behavior. Based on a statistical approach, this task takes advantage of available historical data, collected from previous capture campaigns, and employed to train a dedicated multi-class classifier. The corresponding output is the probability of presence of each vessel in a certain area at a given delay from the time of image capture. The available (genuine or fraudulent) Automatic Identification System (AIS) messages sent by apparently collaborating vessels within the Area of Interest (AoI) are then integrated with the above-mentioned estimates to build a Ship Route Prediction (SRP) system, which is used to predict the next position of the recognized ships. The entire set of processing results is eventually integrated within a webGIS interface that enables the user to easily access and visualize the output products of the platform.

The rest of the paper is arranged as follows. Section 2 presents a brief overview of the state of the art concerning maritime monitoring platforms. Section 3 briefly introduces the proposed platform. Section 4, Section 5, Section 6 and Section 7 concern detailed discussions of each processing technique integrated into the platform and Section 8 describes the WebGIS infrastructure dedicated to data integration and visualization. Section 9 presents an actual implementation of the discussed platform within the framework of OSIRIS, a project funded by the European Space Agency. Section 10 concludes the paper, presenting a summary of the main results and discussing possible future developments.

2. Related Work

Maritime surveillance topics have been considerably addressed by researchers in the framework of previous international projects. Here, some examples are briefly summarized.

DOLPHIN (Development of Pre-operational Services for Highly Innovative Maritime Surveillance Capabilities) [1] has been an EU research and development project that aimed to improve space-based maritime surveillance techniques in order to support activities in the areas of border surveillance, traffic safety and fisheries control. NEREIDS [2] (New sERvice capabilitiEs for Integrated and aDvanced maritime Surveillance) has been an EU-funded project aimed to provide an integrated vision of European maritime policy and surveillance. Specifically, the goal was to enhance European capabilities for automatic ship monitoring in addition to providing advanced decision-making tools. SIMTISYS [3] (Simulator for a Moving Target Indication System) project focused on the detection and tracking of small vessels. To this purpose, a simulator for a moving target indication system has been implemented, aiming to provide a common tool, useful for monitoring the maritime borders and in particular maritime routings and traffic and the management of pollution. PERSEUS [4] (Policy-oriented marine Environmental Research for the Southern EUropean Seas) has been a research project that aimed to assess the dual impact of human activity and natural pressures on the Mediterranean and Black Seas. PERSEUS merged natural and socio-economic sciences to predict the long-term effects of these pressures on marine ecosystems. The project’s goal was to design an effective and innovative research governance framework, in order to provide the basis for policymakers to turn back the tide on marine life degradation. AMASS (the Autonomous MAritime Surveillance System) project [5] concerned the observation and security of wide critical maritime areas in order to reduce actual and potential illegal immigration and the trafficking of drugs, weapons and illicit substances. The surveillance system consisted of autonomous, unmanned surveillance buoys with active and passive sensors, such as un-cooled thermal imagers and hydrophone arrays connected as a network with wideband radio. AMASS represented a tool to be exploited by institutional actors within the framework of maritime border surveillance.

All these projects aimed at implementing tools to improve the detection and monitoring of seafaring vessels in different critical circumstances, such as illegal trafficking and migration, fisheries control and piracy. In this paper, innovative solutions are proposed to enhance and enrich some of the crucial tasks related to maritime surveillance. More in detail, the following sections present novel methods concerning refined vessel detection and segmentation, advanced procedures for vessel kinematics assessment and classification algorithms to predict ship routes.

2.1. Ship Detection and Segmentation

The goal of ship detection concerns the recognition in the captured imagery of those pixels relating to a vessel target. This task has been widely discussed in previous literature. A popular approach towards the implementation of ship detectors based on remote sensing input imagery, aims at adopting solutions that allow high computational speed for prescreening stages. In this sense, fine statistical considerations are normally overlooked in favor of fully automatic and fast detectors based on simplified statistical assumptions and a fixed maximum false-alarm rate using, for example, Constant False-Alarm Rate (CFAR) detectors. These normally work on single-channel intensity images, and consist in a statistical test over each pixel intensity against the intensities of a specified neighborhood. In very wide imaged areas, especially with heavy traffic conditions, the false alarms produced are normally too many to be rejected manually [6]. Discrimination is intended to discard them automatically, or with minimum human intervention, by exploiting further information from either the data or some kind of prior knowledge, or both. Since some form of CFAR detection seems to be a de facto standard for prescreening, here we consider everything capable of refining its output as a discrimination algorithm. The literature often refers to either polarimetry or interferometry to perform discrimination [7,8,9]. In addition, for polarimetric data analysis, a segmentation step is recommended by some authors [10,11,12]. In [10,11,13], the authors claimed an improvement over the decomposition technique discussed in [12]. This improvement is obtained through a more accurate consideration of coherent and incoherent scattering from the target, which, under some conditions, could also be used to estimate the ship pitch and roll angles. The authors explicitly claimed that their method may be useful for classification and identification purposes. Later, their alternative representation of symmetric scattering [14,15] has been proved equivalent to that described in [12]. In [16] the authors proposed a method based on single-look complex images to detect point targets from polarimetric SAR. From our point of view, this is a preliminary step for classification and recognition. In his exhaustive description of a complete SAR-based ship detection system, Crisp [17] reported an experiment done with a richer set of targets, imaged through a satellite-borne dual-polarization SAR and ground-truthed by visual analysis and AIS [18]. The result is that a combination of the HH and HV channels produces the most reliable results. In [19], a detector based on the intensity of the multi-look polarization channels is proposed. To design a CFAR detector for these data, a specific statistical model for the multi-look images must be derived. The authors proposed the $G_0$ distribution [20] as a valid option for strongly nonstationary sea clutter. In [21], a discrimination method for CFAR-detected targets based on fusion of the target-marked HH channel and two features derived from target decomposition is proposed: the diplane and helical factors. This can also be the basis for a target classification method that outperforms alternative standard strategies such as k-means clustering and notch filtering. Experiments on airborne SAR confirm the authors’ claims.

Based on a performance comparison among four operational detection systems, all the advances made in ship detection thanks to a specific analysis of the peculiar SAR quantities summarized above are recognized in [22]. These authors admited, however, that a number of residual drawbacks still make this application prone to substantial ambiguities produced by clutter, strong scatterers, sidelobes and irregular shorelines. Since the analyzed algorithms behave differently against different ambiguities, these authors proposed a voting scheme based on different detectors as a solution to improve the detection performances. The tests were performed on different detectors, including JRC’s SUMO [23], against RADARSAT-2 and COSMO-SkyMed data (see the respective works in [24,25,26]). The boundaries between prescreening and discrimination are not so sharp. As prescreening is intended to be fast and coarse, and discrimination to be slower and more accurate, a distinction could be made on speed and accuracy. A proper discrimination algorithm, however, is one that acts on small areas of interest made of the targets detected in prescreening and their immediate neighborhoods, with no reference to the rest of the image. It is worth mentioning that ESA provides the open source software toolbox SNAP [27], a platform that supplies the most relevant procedures for SAR signal processing, including an implementation of a CFAR-based ship detector that represents the starting point for the procedures described in this work.

2.2. Ship Kinematics Estimation

Several methods have been proposed to deal with the issue of estimating the kinematics of ship motion captured through SAR imaging systems. Previous research [28] has provided an accurate description of the physics underlying the wake pattern produced by ships moving through the water. Such a pattern is directly related to the velocity of the vessel over the sea surface, and is mapped in SAR images by features related, in turn, to the velocity vectors (over ground) of the vessel and the satellite platform. Since the wake morphological features are directly related to the ship’s motion (see [29,30]), a popular approach to kinematics estimation consists in detecting and later analyzing the wake pattern generated by the ship. Since wake patterns appear mainly as V-shaped forms, many have proposed wake detectors based on the preliminary recognition of the linear envelopes observed in the wake shape. Zilman et al. [31] proposed a method based on the fast discrete Radon transform to detect the linear segments that start from the wake tip and develop along the wake shape. A similar approach was adopted by Eldhuset [32], who introduced an approximation stage of the Radon transform based on the exploitation of the Chebyshev polynomials. This way, the detector performance improves in terms of reliability and robustness with respect to false alarms. Once the linear envelopes of the main wake components have been detected, the backscattered signal can be analyzed to extract relevant information. In particular it is possible to estimate, according e.g., to the authors of [33,34,35], the ship’s heading and the ship’s velocity by performing proper analysis of the signal variations observed within the detected linear regions.

2.3. Ship Route Prediction

The problem of route prediction has been investigated in different fields, covering many types of vehicles, such as urban vehicles (e.g., cars or buses) and ships. In the case of urban vehicles, all the route prediction algorithms elaborate raw data provided by GPS, Wi-Fi or cellular network, while for ships they elaborate AIS messages. In any case, these methods rely on a probabilistic approach, which tries to predict future routes on the basis of past ones. Algorithms for SRP can be classified in two groups: point-based [36,37,38], which only predict the next step and can be used only for ships of a small size, and trajectory based [39,40], which predict the whole trajectory. Pallotta et al. [41] illustrated a hybrid trajectory-and point-based approach, called TREAD (Traffic Route Extraction for Anomaly Detection). A complete survey of SRP algorithms can be found in [42].

3. Platform Main Features

Computational procedures for the analysis of Synthetic Aperture Radar (SAR) and Optical images captured by satellite-borne sensors constitute the core of the proposed platform. This section is a comprehensive overview following the operational pipeline order (Figure 2).

The employed imagery dataset refers to the southeastern side of Malta Island. It consists of: (i) SAR Sentinel 1 A and B images, with pixel resolutions around 10 m; (ii) a massive gathering, continuously carried out between 2016 and 2018, of EROS-B optical imagery, with pixel resolution typically under 1 m; and (iii) a number of CSK H-image GEC level imagery, with pixel resolution of 2.5 m. This heterogenous dataset was also acquired with the purpose of performing a quality comparison of the results of the platform for different spatial resolutions and quality of the input data.

The satellites hosting the SAR sensing devices (Sentinel and CSK) move on Sun-Synchronous Orbit (SSO) paths, with an equator crossing time close to dawn/dusk time, riding the terminator between day and night. The EROS-B imagery has been captured at daylight time, about one hour after the local midday line time. For that reason, it has been possible to synchronize the captured data only once per day.

Before the actual ship detection procedure, a preprocessing stage is performed to detect the imagery subset belonging to land areas and exclude it from the subsequent processing. This is sometimes done by using shoreline archival data, although they may be not sufficient to ensure a proper land masking. For example, these archives may not be accurate enough near coastal areas, where land regions exhibit complicated contour shapes, with many small islands, exposed rocky structures or recently built artefacts such as harbor concrete or rock groynes. In our case, a shoreline dataset based on the Electronic Navigational Chart (ENC) for coastlines and similar (OSM) was first collected. Then, a “buffer” area including natural and manmade objects is computed. Finally, the quality of the buffering mask is manually checked by overlapping it to several SAR and Optical ground truth maps available in GIS software. Exploiting the land mask data, geometrically corrected SAR and Electro Optical images were masked before applying the ship detector. With the proposed procedure, it turns out that critical land targets, such as novel man made or complex natural objects, are properly masked, thus enhancing the detector robustness with respect to false alarms. The buffering procedure is also necessary because of the inaccurate geolocation of the acquired images in sea areas, especially for what concerns the EROS-B Electro Optical imagery. For example, we observed that in the case of high accuracy 1B level image, a displacement even larger than 50 m may occur.

The Region of Interest (ROI) addressed in this work is located in the southern part of the Sicily Island, including the Pelagie Archipelago, Pantelleria and Malta Islands. During the land masking process, the shoreline shapefiles were buffered by about 100 m, resulting in the proper exclusion of new manmade constructions or natural objects. As mentioned above, this operation ensures a low false alarm rate, although it should be remarked that the detection of adjacent, near coastal or harbor vessels is not the primary goal of proposed platform, which is indeed primarily conceived for open sea application scenarios.

In the experimental circumstances presented in the paper, AIS was exploited as a benchmark to assess the proposed platform quality. In particular, as better described in Section 6 and Section 7, the AIS represented a ground truth source to test the accuracy of our procedures’ results. AIS also represented a data source to build a training set and a test set to be exploited by the Ship Route Prediction learning algorithm, as discussed in Section 7.

Nevertheless, the maritime monitoring platform we propose also aims at providing a solution to issues that emerge when dealing with the identification of specific vessel classes, such as non-collaborative vessels. In fact, in the mentioned cases, AIS can be prone to malicious counterfeiting, potentially enabling the non-collaborative vessel to conceal any kind of illicit activity. Our system aims at providing a reliable tool to identify vessels that are not correctly monitored through the currently available systems and provide, in addition, meaningful information that can be exploited to enrich the awareness of any concerned operator during the observation of a given maritime area.

4. Ship Detection

The detection and the consequent enumeration of manmade objects over large sea areas is the primary point of interest in a maritime awareness system. High and Very-high resolution remote sensing imagery [43,44] is an important data source for ship detection. Both SAR and Electro Optical images provide geo-located information of ships detected by spacecraft sensors. Here, the adopted approach emulates the human behavior while searching for detailed features in an image. During the scan, the human eye tends to neglect featureless regions while putting more effort in the analysis of more complex regions. In the same way, the implemented detector processes an input map of a large marine surface area and returns crops cut out from the starting map, representing potential targets of interest. The resulting crops, supplied with additional information (sensing time, geometric resolution, radiometric resolution, spacecraft position and kinematics) extracted from ancillary data, are finally provided to the subsequent processing steps.

In the case of positive detection, the output crops are provided as subsets of the initial image, restricted to an Area of Interest (AoI) where only one candidate vessel is visible.

The detection methods applicable to SAR imagery depend strongly on image features such as spatial and radiometric resolution, coherence, number of looks, transmitted and received polarization, etc. The detector here proposed performs its operations without the need for human intervention, since it is able to detect potential vessel signals in any input map, eventually returning an output array containing the estimated geographic coordinates of the detected candidates. The detection is totally automatic for Sentinel 1A/B and Cosmo Sky-Med data. Only in the final stages the operator is required to check for anomalies generated by ghost and/or azimuth ambiguities featuring large radiances. As mentioned in Section 2.1, the proposed ship detector takes inspiration from the popular CFAR-based algorithm that can be found in literature. In practice [6,45], an appropriate moving window (see Figure 3) is centered, sequentially, on each test pixel. This is formed by a test pixel neighborhood, called guard area, which is supposed to contain any possible target that includes the test pixel, and a second neighborhood, called boundary array, only supposed to contain background pixels, which is used to estimate the background statistics. CFAR based ship detection features some relevant weak points that are worth being discussed. The probability of success for the detection task primarily depends on a proper setting of the size of the boundary array window. Enlarging this window typically results in the reduction of the duplicate detection rate, especially in the case of large targets (typically larger than 400 m, such as big tankers). Likewise, the value chosen for the probability of false alarms, obviously affects the detector performance. A proper set up of these parameters usually results in a trade off between false alarm rate and duplicate detection rate, also taking into consideration the specific requirements addressed by the end user. Within an operational scenario, the detector should be employed semi-automatically, thus including a slight stage of human intervention, with the purpose of excluding residual false alarms. Ghosts and azimuth ambiguities can generate false positives. The detection process is fully automated because the CFAR parameters were selected after analyzing many test outputs.

For our purposes, we observed that a tradeoff value of about 1:10 for the ratio between the size of the guard area and the boundary array area, was a good choice for both Sentinel-1 A and B data. The CFAR criterion is an adaptive threshold method: once the required false-alarm rate is fixed, the background probability density function $f\left(x\right)$, where x is the pixel value, is assumed to be parametrically known or estimated from the values in the boundary array. The probability of false alarm, PFA, is then given by

$$\mathrm{PFA}={\int }_T^{\infty }f\left(x\right)dx$$

(1)

Thus, for each position of the moving window and for any fixed value for PFA, the value of T obtained by solving Equation (1) can be taken as a threshold to decide whether the test pixel belongs to a target. If $f\left(x\right)$ is assumed to be Gaussian, Equation (1) is the basis for the well-known two-parameters CFAR detector. We implemented a CFAR detector based on a cell averaging method. In our case, the threshold T is adapted for each pixel, here also named Cell Under Test (CUT), and is computed as

$$T=\alpha \times P_n$$

where $P_n$ is the noise power estimated as the average of the boundary array cells and $\alpha$ is a threshold factor computed through the following equation [46]:

$$\alpha =N\times (PF{A}^{-1/N}-1)$$

where N is the number of training cells and $PFA$ is the desired false alarm rate.

Apparently, to obtain a high probability of detection from this kind of test, the PFA must be traded off. This is why high-detection-probability prescreening methods, though very fast, normally suffer from high false-alarm rates.

The ship detection module included in the proposed platform is supplied with two processing options. Indeed, a classical implementation is provided, supplied within the ESA SNAP^® [47] software suite, while, as an alternative, a novel CFAR based implementation, where the background has been modeled as an Erlang distributed signal ([48,49,50]), has been provided by MapSat. It is worth noticing that the MapSat implementation features a larger reliability in terms of false alarm and duplicate detection rate.

Eventually, a post-processing phase produces full radiometric and geometric resolution crop files, in geotiff format, for further classification analysis. Currently, the algorithm estimates a size box centered on the target centroid and exploits this result to optimally cut the original full radiometric image. The crop window is squared, centered on the detected target and oriented along the north–south direction. The side of the crop is three times (empirically chosen) the ship length $L_s$. To provide the subsequent processing steps with a proper coordinate system, a transformation from the Universal Transverse Mercator system to latitude/longitude was performed.

The Region of Interest considered for the test purposes of this work extends over the Southern Mediterranean Sea, including Sicily Island, Tunisia and Libya. A few oil platforms located in this area were identified by the ship detection module and later excluded from the ship detection report through an additional processing step. This refined selection process was implemented by means of a dedicated geometrical filter integrated within an algorithm in charge of performing a match with the AIS data.

5. Ship Segmentation

The recognition of visual attributes within the AoI allows the relevant information in the data to be highlighted. Imaging algorithms are applied to extract geometric and radiometric features, which provide meaningful insights about the vessel’s morphology, geometry and dynamics. SAR images are often affected by distortions due to marine clutter, spectral leakage, or antenna sidelobes. These can mask the target image, thus hampering the possibility of evaluating the size of the ship. An example of this type of distortions is represented in the SAR intensity AoI image in Figure 4, where the presence of a strong, cross-shaped artifact is apparent.

To the purpose of identifying the set of pixels belonging to the ship target, a novel dedicated segmentation technique, SISS (Ship Image Segmentation from Space)1, has been developed. SISS accepts multi-sensor input data, such as SAR and optical images. In the case of SAR images, the algorithm processes the backscattered signal looking for strong reflectors, which appear as bright spots in the image. A bright connected area is considered for further analysis, since it may be related to a potential ship. In addition, in the case of optical images, the algorithm looks for regions of connected outliers.

The first step of our procedure consists in building a binary shape where the target pixels are set to 1 while the remaining pixels are set to 0. The first approximation of the binary target is done using an adaptive thresholding. The threshold chosen is a multiple of the standard deviation $\sigma$ computed from the entire input image I:

$$T_i=i·\sigma ;i=1,2,3,\dots$$

(2)

The integer multiplier i is chosen iteratively. At the ith iteration, the thresholding operation produces a binary image $I_i$; the process stops at the smallest integer $i_b$ such that $I_{i_b}$ is the blank matrix. The final output is given by the non-null set of pixels found in $I_{i_b-1}$. These pixels are processed through a morphological operator which extracts the connected components [51] that can be possibly grouped together. Then, an ellipse fitting is applied to estimate the vessel axes and their orientation. Figure 5a–g represent the iterative thresholding procedure, while Figure 5h represents the outcome of the segmentation procedure, corresponding to the penultimate thresholding iteration.

After thresholding, the target is isolated by clipping the image outside a mask including the ship and the possible artifacts. The subsequent step is to find the principal inertia axes by performing an eigenvalue analysis on the 2D inertia tensor [52], computed with respect to the barycenter of the thresholded image. Then, a rectangular mask is applied around the barycenter, so as to include the whole target while cutting out part of the artifacts. By iterating this procedure a few times, it is possible to perform a progressively refined estimation of the length overall (LOA), beam overall (BOA) and heading (see Figure 6). Indeed, when as artifact is no longer cut out, we take the unit vector of the minimum inertia axis as the estimated heading, and the maximum distances between target boundary points along the principal axes as the estimated LOA and BOA. In the absence of further information, such as a detected wake or possible morphological ship features, the heading is estimated up to a ${180}^{\circ }$ ambiguity.

6. Ship Kinematics Estimation

The proposed Ship Kinematics Estimation module implements image processing procedures to provide insights about the ship kinematics [31,53].

The wake pattern results from the combination of multiple oscillatory components, whose summation exhibits a V-shaped envelope centered on the ship route axis. The angular aperture of this V is approximately 39${}^{\circ }$. Exploiting these observable phenomena, the route direction can be estimated by first detecting the V pattern (Figure 7) through a Radon-transform-based linear detector [51], and later by identifying the wake center axis.

The wavelengths of oscillatory components observed in the external boundaries of the wake relate to the velocity of the ship itself. Hence, provided that the image resolution is fine enough to observe these wake details, a frequency analysis of the external wake components can lead to an estimate of the dominant wavelength $\lambda$. The ship’s velocity over the sea surface, v, is finally estimated by means of the following expression [31]:

$$v=\sqrt{\frac{\sqrt{3}g\lambda }{4\pi }}$$

(3)

where g = 9.81 m/s${}^2$.

A second effect derived by the satellite and the ship velocities over ground, combined with the peculiarities of SAR imaging, is the azimuth shift, consisting in an artificial separation in the reconstructed image between the moving ship and its wake. The separation length ${\Delta }_{as}$ is proportional to the vessel speed according to the following expression [32]:

$$v=\frac{V_{sat}·{\Delta }_{as}}{R_{sr}cos\beta }$$

(4)

where $V_{sat}$ is the satellite speed, normally included in the orbital metadata accompanying the input image; $R_{sr}$ is the slant range from satellite to the target, easily derived once the geographical coordinates of the target have been computed (see Section 4); and $\beta$ is the angle between the vessel’s velocity vector and the sensor azimuth direction, which can be evaluated approximately through the estimated wake center axis. Hence, the vessel speed can be obtained by measuring the separation length directly on the SAR map.

Our kinematics estimation procedure focuses on: (i) the detection of the linear envelopes of the wake to estimate the ship heading; and (ii) the estimation of the ship velocity through Equations (3) and (4).

Unfortunately, ship wakes are not always visible or too faint to be detected. Typically, due to very low Signal-to-Noise Ratios, wake detection is a task with a very small probability of success. Different approaches have been proposed (see, e.g., [54,55]) to improve the poor results obtained through classical methods, e.g., based on the straightforward application of the Radon/Hough transform on intensity images. Taking inspiration from the authors of [32,56], we propose a novel method for detecting and analyzing wake patterns through SAR map processing. First, the gradient of the input image is computed by means of a dedicated procedure, following an approach specifically suited for speckle-affected data, such as SAR images. Assuming that the linear envelopes of the central/peripheral wake sectors, represent the main observable wake features in SAR maps, the estimated gradient feeds an algorithm based on the Radon transform, whose goal is to detect those segments. Eventually, the algorithm returns a decision statement about the presence of a wake pattern. This final decision is put forward according to the specific arrangement of the detected linear segments. Further details may be found in [57]. In the case of a positive detection, the corresponding wake pattern is identified and its relevant features are employed to estimate route and velocity parameters of the ship.

An example of the processing pipeline execution is represented in Figure 8. A SAR input crop containing a ship candidate is processed by the proposed wake detector. The central turbulent band and one of the external wake arms are correctly identified. This allowed the azimuth shift and, accordingly, the corresponding vessel speed to be estimated. A successful detection of the wake pattern also enabled us to focus the analysis on its peripheral sectors, and, through a spectral estimation technique (see, for example, [58]), to assess the presence of dominant wavelengths in the spectra. Eventually, it was possible to provide estimations for route and velocity for the candidate target in Figure 8.

Table 1. Kinematics estimation output. R stands for Route and V for Velocity. The presented velocity results correspond respectively to the Azimuth Shift (A.S.V.) method and to the Fourier Analysis method (F.A.V.).

AIS R. (deg)	Estimated R. (deg)	AIS V. (m/s)	A.S.V. (m/s)	F.A.V. (m/s)
290	285	6.0	5.89	6.22

reports the results obtained through the proposed estimation method and the corresponding ground truth data, supplied by a commercial data provider.

7. Ship Route Prediction

This section concerns the exploitation of the data and results obtained so far with the goal of predicting the route evolution of the considered target. To this aim, we developed a novel procedure adopting a point-based approach. The AoI A to be monitored is divided into a $m\times n$ square grid, where each cell is identified by a pair $(r,c)$, representing the row and the column, respectively. The $(r,c)$ system is labeled through the corresponding geographical coordinates (in the presented results the coordinate origin is at the Greenwich meridian and Parallel 0). The spherical space is mapped according to a Plate Carrée projection [59], where each square has a side of 0.01 degrees. We are aware that this kind of projection is not optimal, and that distortion increases with distance from the Equator, but for the areas that have been considered for experimental validation in this work, this represented a minor issue. Figure 9 illustrates an example of AoI split in cells.

Each ship i within A is represented at a given time t by a status $X_i^t$, which can be characterized by different parameters extracted from AIS, such as the current latitude and longitude, the current speed over the ground ($sog$), the current heading, the type and flag of ship. Other parameters can be added to the status, such as weather and sea conditions. Within this work, only AIS parameters have been considered. This does not represent any generality loss, since scalability is a main feature of the proposed system and the status array may be updated whenever required. $sog$ is measured in degrees. Thus, we split heading into two parts: $s=sin\left(heading\right)$ and $c=cos\left(heading\right)$. Thus, $X_i^t=(la{t}_i^t,ln{g}_i^t,so{g}_i^t,s_i^t,c_i^t,type)$.

7.1. The K-Nearest Neighbor Classification

Given the current status of a ship, the problem we want to solve consists in determining the next cell occupied by the ship after a given time. This is a multiclass classification problem, where every class corresponds to a cell of the grid. Many algorithms for multiclass classification have been proposed in the literature, such as neural networks, extreme learning machines, naive Bayes, decision trees and support vector machines. In this paper, we propose a SRP algorithm, which exploits a K-Nearest Neighbor (K-NN) classifier.

The K-NN classifier can be thought as a black box, which receives a series of features $X=(x_1,x_2,\dots ,x_n)$ as input and returns a class label y as output. Figure 10 shows an example of K-NN classifier, which receives two features $x_1$ and $x_2$ as input, and returns one of the following classes as output: green, orange or purple. Given a training set T and a new input $x=(x_1,x_2)$ (represented by a red dot in the figure), the K-NN classifier searches the k nearest neighbors of x in T and among them selects the most frequent class. The most frequent class is defined by a similarity function represented by distance. In the case of the example in Figure 10, the $K-NN$ classifier returns $orange$ as the output classification for that specific x input. The value for k can be set according to user needs ($k=3$ in the figure).

7.2. The Proposed Algorithm

The algorithm receives as input the current status of a ship $X_i^t$ and returns as output the probabilities that each cell $(r,c)\in A$ will be occupied by i after a time $t+\delta$. $\delta$ can be set to 30, 45 or 60 min. The entire set of the probability values returned by the predictor constitute the matrix of probabilities $M_i^{t+\delta }$, which has size $m\times n$ (see Figure 11).

The SRP procedure can be conceptually described as the sequence of three steps:

• Setup builds a dataset extracted from historical AIS data and randomly extracts a training set corresponding to the 80% of the whole dataset (the remaining 20% of the dataset represents the test set).
• Training stores the training statuses and the corresponding cell of the grid after 30, 45, or 60 min. Obviously, a good coverage of the available AIS data is a key condition for the algorithm to work. During this stage, the performance of the trained model is also assessed, by means of a proper test set.
• Runtime, given the status of a ship, assigns to each cell of the grid the probability to find the ship in that cell after 30, 45, or 60 min.

Details about the above introduced stages are discussed in the following.

7.3. Setup Phase

The setup stage consisted in cleaning and analyzing the historical AIS dataset related to the AoI. AIS Data were provided by two sources (Exact Earth4 and Astra Paging5) to reduce the probability of message loss. However, in the majority of cases, the same message was collected twice or more times. Indeed, the replication of the same message is a usual event, which has often been observed also when the collected data were coming from the same AIS source. The original dataset was cleaned by implementing a specific procedure to remove duplicates, followed by another fine selection step on the dataset, based on a check on heading and speed values. We naturally expected heading values ranging from 0${}^{\circ }$ to 359${}^{\circ }$, but many records in the dataset featured values outside this range, e.g., heading equal to 511${}^{\circ }$ (meaning no information is available), thus they were discarded. In addition, AIS records were discarded whenever their reported speed values were smaller than 0.5 knots (moored or at anchor ships) or greater than 60 knots (excessive speed). This choice was motivated by the observation that the average maximum speed of a motorboat lies around 30–40 knots, while only a few extra luxury motorboats can reach a 70 knots speed.

Starting form these data, new datasets, called input datasets, qwew generated and processed by the K-NN algorithm. In details, for each record i of the AIS dataset, the status $X_i$ qA extracted, as well as its related class $y_i$. It is worth noticing that the input dataset is a subset of the AIS dataset, containing only records for which there is an associated y. To include the whole AIS dataset in the input dataset, some interpolation techniques must be used. We reserve this implementation for future work.

The value of each $y_i$ Qa calculated as follows. First, we took the Maritime Mobile Service identity (MMSI) and the date time t from record i belonging to the AIS dataset. Then, we searched in the AIS dataset a record j with the same MMSI and a datetime $t_1$ with $t+\delta -ϵ\le t_1\le t+\delta +ϵ$, where $\delta \in [30,45,60]$ min and $ϵ=\delta /3$. Finally, we extracted latitude and longitude from j and converted them into the associated grid position $(r,c)$. At the end of the setup phase, three input datasets were built, one for each value of $\delta$, thus having ${(X,Y)}_{30}$, ${(X,Y)}_{45}$, and ${(X,Y)}_{60}$. To further constrain the search, the algorithm could integrate an additional filter based on a spatial distance criterion. This option was discarded since the gathered AIS data were preliminary filtered out to restrict the search area to a relatively small AoI (South Mediterranean area). The ship search was based on the univocal MMSI identifier and performed in a spatially bounded neighborhood, hence it is unlikely that ambiguity errors related to extremely large spatial separations emerged during the search procedure.

The setup phase splits each input dataset into training set and test set. Initially, all records were marked as belonging to the training set. Then, the setup phase randomly extracted a number of messages equal to 20% of the total number of messages and marked them as belonging to the test set.

7.4. Training and Runtime Phases

The training phase trained the K-NN classifier with the input datasets, built during the setup phase. For each input dataset, a different SRP classifier was built.

Once the model was trained, the SRP classifier was ready to be used. During the runtime phase, the SRP classifier took as input the current status $X_i$ of a ship at a time t and asked for a prediction after $\delta$. The classifier then returned the matrix of probabilities $M_i^{t+\delta }$ that each cell $(r,c)$ in the AoI would be occupied by the ship at time $t+\delta$.

At runtime, the proposed classifier could be accessed by: (a) a command line tool; (b) a Web API; or (c) a Web application. Both the command line tool and the Web API receive the ship status as input and return the matrix of probabilities, represented as a GeoJSON ([60]), as the output. Additionally, the following parameters can be specified: latitude, longitude, speed, heading, and basic class. The basic class represents the ship type and, at present, can be set to 0 (small ships) or 1 (big ships).

Figure 12 illustrates a snapshot of the web application. The web application displays all the routes used to train the model as cells of the grid. The user can click on a point in the AoI and the next predicted cells are highlighted. The red circle shows the maximum distance that the ship can cover traveling for the chosen time frame at the given speed. Through a simple interface, the user can also set the ship heading, speed and type (small or big).

7.5. Combining AIS with Satellite Images

The SRP algorithm was tested in a real use case: at a given time, the status of a ship was extracted. Then, the ship route predictor was executed to predict the next position of that ship. The correctness of the predicted result was finally compared with the data extracted from the AIS messages sent by that ship.

More in detail, the test image was captured by EROS-B on 22 October 2016 at 12:59:42. Then, the ship detector returned one potential target within the input map (see Figure 13) and the bounding box of the detection resulting in: latitude in $[33.89,33.91]$ and longitude in $[12.57,12.67]$.

By performing a search within the implemented AIS dataset, a candidate was identified, with matching AIS position and time parameters. Given the MMSI, it was possible to retrieve relevant information from the Vessel Finder portal6. The candidate was classified as a floating production, storage and offloading vessel. All information gathered for that ship was fed to the SRP input, by submitting a query for the position of the ship after 30 min. Eventually, the returned prediction matched with the AIS data.

8. WebGIS Interface

The WebGIS interface enables a generic user to access a wide range of functionalities available within the proposed platform. Users can: (i) have a quick look at real time events; (ii) access historical information; and (iii) submit the request for monitoring a marine area of interest.

More in detail, the interface is composed of two main parts, the toolbar and the map. The toolbar is located at the top of the WebGIS interface and consists of a set of buttons, each providing access to one functionality from the following list:

• Search: This button allows accessing AIS historical data. It is possible to set up the search criteria by specifying the period of interest, the type of AIS data (terrestrial or satellite), the vessel MMSI or the vessel type. By acting on the dedicated time scrollbar, it is possible to visualize the evolution of the requested data, with one minute time step.
• T-AIS Real-Time: This button activates the layer of terrestrial AIS data. The WebGIS will display all the information obtained through terrestrial AIS. The visualized data are refreshed every minute.
• S-AIS Real-Time: This button activates the layer of satellite AIS data. The WebGIS will display all the information obtained through satellte AIS. The visualized data are refreshed every minute.
• Planning: This button activates a layer providing the future coverage maps of the available satellites.
• Weather: This button shows the maps of the weather and wind direction forecasts, for the displayed zone.
• Bathymetry: This button activates the layer of bathymetry data, reported as a gray-scale map.
• New Request: This button allows to submit a new monitoring request.
• Print: This button provides a printable image of the displayed scene.

The remaining part of the interface is devoted to display the requested information on a map. Within this map, it will be possible to activate several thematic layers, some of which can also be activated from the toolbar. In the case of the simultaneous activation of several layers, the user can specify the priority in the layer stack representation. For example, the layers can be stacked to bring to front the information of major interest, and each layer can be assigned with a different opacity property.

As mentioned above, once terrestrial and/or satellite real-time AIS layers are activated, the vessels located in that area are displayed on the map. Vessels are represented as colored arrows, where color represents the type of the vessel. The available information about each vessel can be accessed by clicking directly on the vessel icon in the map (see Figure 14). A dedicated panel appears, reporting about: (i) the size of the vessel (expressed as length, width and draught); (ii) the route and the velocity of the vessel; (iii) the vessel destination and the estimated time of arrival; and (iv) the registry information (such as MMSI, vessel name, call sign, and IMO). From the information panel, it is also possible to access further vessel details and track the past positions of the vessel itself.

In addition to the visualization of real-time/historical AIS data, the user can also submit the request for a new monitoring of a given area by means of the New request button. To this purpose, the user must: (i) select a RoI (rectangular or polygonal) by directly drawing on the map or by feeding the system with a proper .KML file; and (ii) specify the time range of interest. Once the request is submitted, the OSIRIS operator must accomplish a set of steps to proceed with the monitoring request. He may edit the request to update its status (pending, rejected, processing or completed) or to specify the inclusion in the monitoring procedure of additional data captured by one or more satellites.

The user can check the status of each request in a dedicated page. In case of requests in intermediate processing status, the completion percentage is also reported. Completed requests can be selected to display the results on the map. The layer reporting the results represents the identified targets and the vessels as theme icons, according to the class labels they have been assigned with. A generic detected target will be promoted to the status of identified vessel in case a reliable link with AIS record is verified. Targets can be classified according to the following rules (see Figure 15): (A) identified vessel, with recognized route; (B) identified vessel, with recognized direction only (route is known up to a 180${}^{\circ }$ ambiguity); (C) detected target, with recognized route; (D) detected target with recognized direction (route is known up to a 180${}^{\circ }$ ambiguity); and (E) detected target, with no information about route or direction. In the results layer, users can also select which of the available satellite sources they wish to consider in the visualization map. Accordingly, the vessels and targets that have been processed and recognized will be properly highlighted on the displayed map.

The user can interactively obtain additional information about the processing results related to the vessels and targets in the map. Upon selecting a target, the corresponding dedicated panel appears. The panel contains the available processing results (e.g., estimated length, width, route and velocity) and, in the case of identified vessels, a quick link to the corresponding AIS match. From the information panel, the user can visualize the input satellite crop and the predicted route for that target. As shown in Figure 16, the route prediction is represented by means of a color map, where red corresponds to large probability values and blue corresponds to small probability values. Through the information panel, it is possible to visualize the predicted route probability map for the considered vessel at the next 30, 45, 60 and 120 min.

9. Implementation

The European Space Agency (ESA) and a scientific consortium led by MAPSAT, an Italian company providing remote sensing services for continuous monitoring of the Earth’ssurface, launched in February 2016 the project OSIRIS7 (Optical/SAR data and system Integration for Rush Identification of Ship models). OSIRIS represents a first attempt in transferring the methods and techniques described in this paper to a real-world implementation of a maritime monitoring system. Indeed, the OSIRIS’ main goal is the development of a platform dedicated to sea surveillance, capable of detecting and identifying illegal maritime traffic. OSIRIS will thus represent a new tool for combating unauthorized fishing, irregular migration and related smuggling activities. OSIRIS is carried out within the ESA General Support Technology funding Programme (GSTP). The OSIRIS project consortium includes two CNR research groups, namely the Web Applications for the Future Internet Group of IIT-CNR and the Signals and Images Laboratory of ISTI, providing long-standing expertise in Machine Learning and Computer Vision. Another member of the consortium is Sistemi Territoriali, an Italian company with expertise in Geographic Information and Business Intelligence Systems. At full operating conditions, the OSIRIS technological platform will be in charge of: (i) collecting and integrating data made available by multi-source, multi-sensor satellite missions for specific maritime areas of interest; and (ii) processing the acquired data to detect and classify seagoing vessels. High-resolution data will be provided by the Sentinel-1 (C-band SAR) and Sentinel-2 (Optical) satellites, both part of the ESA Copernicus mission. Further data will be provided, in the optical band, by EROS-B (ISI - ImageSat International) and, in X-band SAR, by COSMO-SkyMed (Italian Space Agency). The data will then be processed to detect and identify the ships located in the area of interest and to provide estimates of meaningful ship features, e.g., kinematics and shape parameters. Crucial information to perform ship identification will be collected by means of popular ship monitoring systems such as the AIS service and by integrating additional information gathered from Open-Source Intelligence systems (OSINT). The data will finally be analyzed to provide predictive models for the routes of these ships.

AIS data are provided by the two companies Astra Paging and Exact Earth. Astra Paging provides terrestrial AIS data, at a very high frequency rate, collected from a large number of ground stations deployed along the Mediterranean shoreline. In turn, Exact Earth provides satellite AIS data acquired at a low frequency rate, gathered by a Launch and Early Operations Phase (LEOP) AIS satellite constellation. A web GIS platform is currently in charge of collecting and fusing the AIS data coming from the two mentioned providers and, after a filtering stage to suppress fake or repeated records, making it available to the project operators.

10. Conclusions

A software platform dedicated to maritime traffic monitoring is presented. The main goal of this platform is to detect and identify target vessels within a given sea surface area, which is remotely supervised by orbiting satellites such as Sentinel 1/2, CosmoSky-Med and EROS missions. Radar and optical images represent the main input data for the described platform. These are processed by a suite of algorithms, which are sequentially applied to the data to return information about: (i) the ship positioning within the inspected area; (ii) the main ship geometrical attributes, such as length overall, beam overall and heading; (iii) the ship kinematic status represented by its velocity vector; and (iv) the evolution of the ship kinematics.

The OSIRIS platform has been conceived to be primarily exploited in the Mediterranean maritime framework. Nevertheless, the authors aim at conceptually extending the system to make it usable in different marine scenarios, where the peculiar environmental properties may introduce novel challenges. In polar zones, the target detection module should be able to discriminate between vessels and icebergs. In fact, in the current implementation of the presented platform, icebergs may be erroneously recognized as potential ship targets, but their exclusion from the subsequent stages of the processing pipeline can be implemented by focusing on the irregularity features of the iceberg shape, as opposed to the typical rectangular contour of a vessel. A control performed on the shape of the output returned by the detection module would enable discarding an irregularly shaped candidate iceberg. In the unlikely case that the detected iceberg exhibits a rectangular shape, it could be further assessed by checking for the presence of vessel-like attributes. For example, vessel targets should exhibit a shape factor (length over width ratio) with an average value around 5. In the unlikely case that an iceberg is classified as a non-cooperative ship, the resulting image crop would eventually be analyzed by a human operator and finally rejected.

Future developments will be devoted to improve the overall performance of the platform by enhancing the accuracy of each individual stage in the processing pipeline and to improve the classification method. We observed that, with particularly noisy or distorted images, the estimates returned by the implemented segmentation module are not sufficiently robust. We noted that the result often depends significantly on the initial shape assumed, that is, on the thresholding strategy chosen. Without relying on supplemental data or preprocessing, we are now trying to find a method to suitably tune the thresholds. Our next attempts will be to integrate the results from the same target imaged through different transmit-receive polarizations and integrate the extracted features in the classification process. Furthermore, this work is intended as a step towards a more complete ship classification method, also based on scatterometric/polarimetric features, and a more reliable course and speed estimation, based on wake feature extraction and azimuth shift evaluation. Since wake patterns are hardly detectable in SAR maps, future developments will also be devoted to the refinement of the wake recognition process, based on the exploitation of additional information, such as the fine estimate of the vessel position as well as the constraints of this peculiar hydrodynamics problem, such as the theoretically expected wake angular aperture.

One of the authors’ goal is to conceptually extend the described platform in order to make it more suitable for decision makers and concerned institutions. The presented system would certainly benefit from the integration of additional procedures with the aim of increasing the amount of awareness about a given maritime circumstance. To this aim, models for the assessment of marine traffic safety through the prediction of the accidents’ risk [61,62,63] could be integrated within the vessel characterization pipeline. For example, the probability map returned by the ship prediction module could be enriched by maps indicating the probability of accidents, eventually returning an integrated powerful tool to predict in a real time fashion those marine areas that may be prone to critical circumstances, hence worthy of being more carefully monitored by the concerned operators.

The presented platform is currently being tested within the framework of the OSIRIS project, providing preliminary promising results from a real world testing workbench.