A Digital Twin of the Trondheim Fjord for Environmental Monitoring—A Pilot Case

Abstract

Digital Twins of the Ocean (DTO) are a rapidly emerging topic that has attracted significant interest from scientists in recent years. The initiative, strongly driven by the EU, aims to create a digital replica of the ocean to better understand and manage marine environments. The Iliad project, funded under the EU Green Deal call, is developing a framework to support multiple interoperable DTO using a federated systems-of-systems approach across various fields of applications and ocean areas, called pilots. This paper presents the results of a Water Quality DTO pilot located in the Trondheim fjord in Norway. This paper details the building blocks of DTO, specific to this environmental monitoring pilot. A crucial aspect of any DTO is data, which can be sourced internally, externally, or through a hybrid approach utilizing both. To realistically twin ocean processes, the Water Quality pilot acquires data from both surface and benthic observatories, as well as from mobile sensor platforms for on-demand data collection. Data ingested into an InfluxDB are made available to users via an API or an interface for interacting with the DTO and setting up alerts or events to support ’what-if’ scenarios. Grafana, an interactive visualization application, is used to visualize and interact with not only time-series data but also more complex data such as video streams, maps, and embedded applications. An additional visualization approach leverages game technology based on Unity and Cesium, utilizing their advanced rendering capabilities and physical computations to integrate and dynamically render real-time data from the pilot and diverse sources. This paper includes two case studies that illustrate the use of particle sensors to detect microplastics and monitor algae blooms in the fjord. Numerical models for particle fate and transport, OpenDrift and DREAM, are used to forecast the evolution of these events, simulating the distribution of observed plankton and microplastics during the forecasting period.

1. Introduction

The concept of a digital representation or replica of physical objects or systems has existed for several decades. Digital Twins (DTs), which originated in the manufacturing industry, refer to virtual representations of physical products throughout their lifecycle [1]. The aerospace industry, including companies like NASA and Boeing [2], adopted DTs to simulate and monitor the behavior of complex systems. However, advancements in various technologies have recently brought the term “Digital Twin” into the spotlight. IoT (Internet of Things) devices and sensors provide real-time data from physical assets, enabling the creation of more sophisticated DTs. Machine learning algorithms and artificial intelligence allow DTs to learn from and react to real-time data, making them dynamic and capable of predictive analytics [3]. Recent advancements in visualization technology have made Digital Twins more accessible and easier to understand and interact with while also enhancing their ability to optimize operations and improve decision-making. Similarly, progress in data science has made Digital Twin technology more powerful, accurate, and versatile, improving its performance and functionality and broadening its applicability across numerous domains.

DTs surpass numerical simulations and models by using real-time and historical data to represent the past and present and simulate predicted futures. Additionally, they synchronize with the real world at defined frequencies and fidelities [4]. Leveraging real-time data and tailored predictive analytics make them more flexible and hence useful for real-world applications. Recent trends in research often integrate real-world with virtual experimentation, as this combination offers numerous benefits [5]. DTs contribute to virtual experimentation by providing accurate representations of physical entities, enabling real-time monitoring, mitigating risks, saving costs and time, optimizing performance, and generating valuable knowledge. This empowers researchers to explore and innovate in a virtual environment before implementing changes in the physical world. Transferred to the world of oceans, these what-if scenarios enable the study of changes caused by climate change or human interventions. Through Digital Twins, we can assess potential outcomes and mitigate effects before they occur, while also comparing different scenarios to identify the most beneficial solution. However, the quality and reliability of these scenario models heavily depend on the assumptions and data they are based on. Ensuring scenario reliability involves strategies such as model verification, validation, and calibration. This process includes adjusting model parameters and comparing outputs with observed data, using high-quality, accurate, and comprehensive data sets to ensure the model behaves as expected under various conditions.

The application of Digital Twin technology to the ocean, referred to as “digital twins of the ocean”, began to take shape in the late 2010s. The term itself became more widely recognized around 2020 when the European Union (EU) announced the establishment of a Digital Twin of the Ocean (DTO) under the European Green Deal and Horizon Europe programs, and the UN Decade of Ocean Science for sustainable development (https://oceandecade.org/, accessed on 9 July 2024) program DITTO—Digital Twins of the Ocean [6] came to life. The initiatives aimed to create high-resolution digital models of the ocean to support marine research, policy-making, and environmental protection [7]. The concepts are a significant attempt to make a leap forward in how we understand, manage, and protect our oceans. The Iliad project (Integrated Digital Framework for Comprehensive Maritime Data and Information Services) is one of the first DTO projects/actions driven by this European initiatives, followed by the EDITO projects within the EU MISSION OCEAN, Europe’s contribution to the UN Ocean Decade.

The ocean sciences have adopted the concept of Digital Twins for Digital Twins of the Ocean. DTO, as defined by DITTO [8], combines ocean observations with ocean predictions in an interoperable data space or lake and provides an intuitive user interface to engage with the twin and trigger so-called ’what-if’ scenarios. Other initiatives look more into localized twins for a specific ocean area and for specific purposes. These twins might be distinguished from DTO as digital ocean twins (DOTs), as they not only consider the physical ocean but include human activities and social-ecological aspects of the ocean environment as well [9]. The DITTO program proposes to use confined ocean spaces with a good coverage of ocean observations and ocean models to develop and test the methodology to build interoperable twins. OceanLab in the Trondheim fjord [10] is an example of this.

The research and development of the presented DTO is part of the project Iliad [11]. This DTO representation is connected to its physical counterpart, the OceanLab area of the Trondheim fjord, through real-time data from the surface observatories and benthic stations that provide continuous power and communication to plugged-in sensors [12]. Autonomous underwater vehicles (AUVs) provide on-demand data collected over a broader spatial scale. Data from an operational ocean model for the area are available from the national meteorological institute every hour with a 2.5-day forecast.

Our motivation for the development of a DTO is to monitor the environmental conditions in the fjord like water quality, biological events, or pollution, with uses for policy makers and industry as well as research and education. As an example, the Marine Strategy Framework Directive requires EU Member States to take measures and set up monitoring programs to collect the data needed to assess, achieve, and maintain Good Environmental Status in the marine environment. Monitoring incorporates a variety of water quality features, like the introduction of novel and previously unaccounted pollutants, e.g., the presence of micro-litter (including microplastics) [13]. The seafood industry is Norway’s second biggest contributor to the Norwegian economy. The industry can benefit from DTO, like the Water Quality DTO, as it contributes to making this industry more predictable and thus supports the planning of economically and ecologically sustainable and safe operations. Oxygen anomalies, conditions for (harmful) algae blooms, and invasive species are environmental conditions that can potentially have lethal effects and pose a serious threat to aquaculture operations. Models that simulate the distribution of matter from fish farm locations in larger spatial scales can predict environmental effects and infection pressure and thus inform decision-making and operation planning. DTO that combine statistical data from the past with data from the present and short-term (days) and long-term (months to years) predictions for the future are an attractive incentive for the insurance industry with respect to asset protection and risk mitigation. The application of DTO in aquaculture has attracted significant research attention, resulting in a number of publications in the last few years [14,15,16].

This paper describes how we build a DTO pilot from the OceanLab data within the project Iliad. Section 2.1 provides the relevance and the role of the Iliad project in the context of EU and UN initiatives, such as supporting the EU Green Deal and UN Sustainable Development Goals (SDGs) and contributing to the European Digital Strategy. Section 3 delves into the general building blocks of DTO and how we put them together to implement a DTO for environmental monitoring, including water quality, biological events, or pollution. Section 4 presents, explains, and discusses the results of the DTO Water Quality pilot related to data collection with novel sensors, data visualization, and water quality modeling and includes the case study related to algae bloom. Section 5 offers a short summary of the presented work, conclusions, and prospects of the future work.

2. Context of the Iliad Project

2.1. Iliad in the Context of EU and UN Digital Twin Initiatives

The European Green Deal is an initiative of the European Commission, led by Ursula von der Leyen, that was launched in 2019 and approved in 2020 with the objective to make Europe climate-neutral by 2050. The Green Deal outlines Europe’s agenda for climate action and sustainable development. The research program Horizon 2020, that was active between 2014 and 2020, supported the Green Deal to reach its targets and funded a number of projects which were focused on renewable energy, energy efficiency, circular economy, clean transport, climate adaptation, and digital transformation. Horizon Europe (2021–2027) continues to prioritize research and innovation in line with the Green Deal objectives. The Horizon 2020 project Iliad set out to create “A data-intensive, cost-effective Digital Twin of the Ocean” which was later adapted to the common accepted notion of Digital Twins of the Ocean (plural) as defined by the UN Ocean Decade program DITTO.

The project’s objective was that the enabling technology of the Iliad DTO will contribute to the implementation of the EU’s Green Deal and Digital Strategy as well as the seven UN Ocean Decade’s outcomes in close connection with the 17 Sustainable Development Goals (SDG). To realize this potential, the Iliad DTO have followed a systems-of-systems (SoS) approach that allows for the integration of existing EU Earth Observing and Modelling Digital Infrastructures and Facilities and is not tied to a central infrastructure but based on federated, interoperable systems.

The EU Missions of the Horizon Europe funding program include a dedicated mission to restore the EU Ocean and Waters with a digital ocean and water knowledge system, known as Digital Twin Ocean, as the central enabler. The European Digital Twin Ocean, or EU DTO, comprises EDITO ModelLab [17], EDITO infrastructure (EDITO-Infra [18]) that combines and integrates key existing services, namely Copernicus Marine Service (CMEMS [19]) and the “European Marine Observation & Data Network (EMODnet)” and a series of H2020 projects including Iliad.

EDITO projects provide a public data platform and computational infrastructure as well as publicly available models to create DTO within this infrastructure. Like Iliad, the European DTO builds on the EU Digital Agenda and will be interoperable with Destination Earth. Other initiatives from the Mission program that bring additional types of data into the EU DTO include DTO BioFlow [20], which is for the integration of biodiversity monitoring data and the family of projects to integrate socio-ecological models into the EU DTO.

2.2. The Iliad High-Level Architecture for Interoperable Multi-Stakeholder Twins

Iliad is complementary to these initiatives with an additional focus on citizen science data and the semantic interoperability of data and services based on state-of-the-art vocabularies and ontologies. Iliad is working around the following main objectives:

• create and implement a DTO framework, which will function as a domain-specific precursor of the Destination Earth DTO in a federated environment instead of a central platform.
• demonstrate the value of DTO through engaging pilots for a variety of stakeholders.
• assemble a broad and diverse user community of existing and new users who will use the project’s innovative technological solutions to address their challenges.

Iliad works closely together with the EDITO project team to contribute to the EU DTO with data and applications and to ensure interoperability between the EU DTO platform and the Iliad federation. DTO data and services are documented and accessible through the Iliad Marketplace to everyone (https://ocean-twin.eu/marketplace, accessed on 9 July 2024).

Iliad follows an adapted model according to the Big Data Value chain along the subsequent Digital Twin pipeline steps. The result is a high-level architecture as shown in Figure 1. Digital Twin Data Acquisition includes observations through sensor data and access to relevant data sources and various data spaces, as illustrated in the figure. This process handles both real-time and batch data collection, utilizing associated communication and messaging technologies. It supports various data types and ensures data protection. Additionally, it includes methods to access, assess, convert, and aggregate signals, transforming them into processable and communicable data assets that accurately represent real-world entities. Digital Twin Data Representation involves mapping data to appropriate storage and preparation systems. It utilizes various storage solutions and transforms data into analysis-ready (“Gold Data Lake”) data representations for subsequent use and processing. These data representations are organized in data lakes for the various twins. The Digital Twin Core is responsible for handling data analytics, employing a range of techniques including traditional analytic methods and advanced AI/machine learning algorithms. These methods enhance predictive capabilities and support more informed decision-making. Digital Twin visualization and interaction facilitate interactions between machines, objects, people, and the environment. These interactions can influence the data acquisition and collection processes. This step also encompasses the use of actuators and decision support systems for initiating actions within the environment. These pipeline steps are exemplified further in Figure 2 that presents the architecture of the environmental monitoring DTO.

Based on this set of Digital Twins, the development of a Digital Twin of the Ocean framework is being co-designed by a multitude of stakeholders. This framework allows further future thematic and local Digital Twins of the Ocean to be developed, taking into account the multiple stakeholder requirements and needs from the different thematic and local areas. To facilitate the management of these stakeholders, Iliad has developed a stakeholder taxonomy which, in turn, allows holistic perspectives combining several thematic twins in an area or transferring concepts, approaches, and technology to different locations. A systematic collection of best practices (https://www.oceanbestpractices.org/, accessed on 9 July 2024) support the exchanges across thematic and local areas.

An environmental monitoring twin of the ocean is in itself mainly technology- and science-driven, and thus, scientists and technology developers are key stakeholders directly involved in the Digital Twin. However, due to the above mentioned multi-stakeholder approach, it is interconnected with the industry-driven Aquaculture DTO that is highly dependent on environmental conditions. Aquaculture DTO analyzes operational and historical data related to aquaculture sites for the purpose of performance, operational support, and continuous improvement with respect to fish welfare, environmental footprint, infections, counter measurements, and environmental conditions. A third Digital Twin, Aquaculture Risk Metrics Norway, is a holistic risk assessment platform that can help monitor multiple risks and provide the relevant data needed for informed decision-making, adaptation, as well as mitigation. This twin development has been driven by the insurance industry. Collectively, these three Digital Twins of the Ocean involve the multiple stakeholders forming the twins using different means of engagement such as workshops and surveys.

3. Methodology

Geographically located in Norway’s Trondheim fjord (Figure 3), the Iliad Water Quality pilot benefits from the ocean observation infrastructure OceanLab, which was built as a part of a national infrastructure project and upgraded through various EU and national projects. The part of the infrastructure employed in the pilot includes surface and benthic observatories and a fleet of AUVs.

The Water Quality Digital Twin uses the real-time data from the sensors and combines it with particle observations at one of the sensor platforms. Particles in the ocean are a great source to assess water quality. By measuring small particles in seawater and finding out what these particles are and where they are transported to, we can assess the processes that are currently going on in the ocean. It also tells us if these particles belong there (biotic (plankton, detritus) or lithogenic (e.g., sediment)) or they are a product of an ongoing or acute pollution event (microplastics, oil) [21,22]. In case of an abrupt change in observations (ocean environment or pollution), the DTO will detect these changes and trigger a response to it. The response can be a change in the frequency of the measurement, a deployment of additional sensors, e.g., on mobile platforms like AUV, or just an alarm for the operators of OceanLab to have a closer look. The current interface to the DTO is a Grafana dashboard with several panels showing the current data in a way that is useful to its users. Additional visualization under development is a data-driven gaming environment for particle visualization in an immersive 4D environment (Section 4.1). Figure 2 illustrates the architecture of the Water Quality pilot.

3.1. Data Acquisition

Sensor Platforms

The stationary sensor platforms of the Water Quality DTO pilot are two surface observatories (buoys) and two benthic stations. All platforms are equipped with a variety of ocean sensors and their sensor suite can be opportunistically expanded, depending on the application and project needs. The observatories also serve as integration platforms for the testing of new technologies and sensors. In the context of the DTO development, two micro-particle sensors have been tested on the site. Two surface buoys provide power supply and 4G telemetry to sensors close to the sea surface. The main monitoring surface buoy (a DB24000 data buoy from Hydrosphere, designed and manufactured by Mobilis) was installed in October 2021. It is located close to the Trondheim marina and can be reached by boat within 10 min (Figure 4). The buoy is a large platform designed as a test platform for R&D, in situ experiments, and as a host for a large array of environmental sensors. Inside is an autonomous profiling winch for lowering sensors up and down through the water column and four moon-pools for mounting equipment. There is also an attachment point on the side of the buoy and around the roof. The second buoy is a 3m diameter buoy supplied by OSIL and was deployed in May 2022. This buoy is placed strategically at the entrance of the fjord, and the site can be reached within about 1 h by boat from Trondheim.

The seabed (benthic) observatories or stations are specialized research facilities designed to monitor and study various aspects of the ocean environment and the organisms close to the seafloor. OceanLab operates two seabed stations at depths of 90 m and 360 m. These stations are connected to the shore via an umbilical cable and can include multiple physical structures. The one relevant for the Water Quality DTO is the instrument rig, referred to in this paper as the benthic observatory. This observatory, shown in Figure 4, distributes power to the instruments, facilitates communication between the instruments and the shore, and allows for changes in instrumentation configuration. The umbilical provides permanent power to the benthic station and ensures continuous real-time data access from shore. The station also includes a docking plate, which serves as a landing and recharging point for tetherless underwater vehicles, allowing them to transfer data to the shore.

Mobile sensor platforms, such as autonomous underwater vehicles (AUVs), can enhance stationary observation capabilities by expanding the spatial coverage of observations. These vehicles, capable of operating autonomously in the ocean for extended periods and that are supported by subsea stations, are often referred to as subsea resident vehicles or fly-out vehicles. Although resident vehicles are of significant research interest for project partners and would be an ideal addition to stationary observatories, this technology is not yet mature enough to be demonstrated in this pilot. Therefore, AUVs operated from shore or boats are used instead. Their missions are typically on-demand, triggered by specific alerts or manually initiated by the end user.

The AUVs in the Water Quality DTO are operated by the Iliad partner NTNU. Two different vehicles, shown in Figure 5, have been used, the larger 5.5 m long Eelume snake vehicle and a smaller LAUV (light AUV) vehicle. The AUV can be equipped with similar or the same sensors as the stationary sensor platforms.

3.2. Sensors and Instruments

The sensor platforms provide power and data transmission to the sensors. The permanent sensor suite of both the surface and benthic station consists of a variety of oceanographic and environmental sensors. They are multi-sensor instrumentation modules, made for long-term deployment and designed to measure various physical, chemical, and biological variables in the marine environment. All sites include CTD and ADCP sensors. CTD stands for conductivity, temperature, and depth, referring to a set of electronic devices used to monitor changes in water conductivity and temperature as they relate to depth. This equipment is used in all fields of oceanography and provides data on the physical, chemical, and occasionally biological properties of the water column. ADCP stands for Acoustic Doppler Current Profiler, which is an instrument used to measure water currents based on the Doppler effect of sound waves. ADCPs are essential tools in various oceanographic disciplines and collect data on ocean currents and their dynamics at different depths and locations.

3.2.1. ADCP and Echosounder

The main monitoring surface buoy is equipped with a downward-facing Nortek Signature100 ADCP that includes an echosounder as well. The instrument consists of 4 beams slanted at 20° for the ADCP and one downward-facing transducer for the echosounder. The echosounder can transmit narrowband pulses (70 kHz and 120 kHz) and broadband chirps (linearly increasing from 70 to 120 kHz). These transmission are reflected by objects or matter in the water column and thus can detect, e.g., biomass and change in abundance and its behavior over time. Ocean monitoring with a combination of the ADCP and echosounder [23] can assess the distribution of zooplankton and fish with concurrent current measurements, enabling the inference of biophysical interactions (case study on algae bloom in Section 4.2.2). From the echosounder, the backscatter amplitude measurements are converted to volume backscatter (Sv, dB re 1 m⁻¹) to account for spherical spreading and attenuation [24,25].

3.2.2. Video and Sonar

Apart from the regular time-series data, the benthic observatory also collects video and sonar data (Figure 6) for the monitoring of the environment around the station. Furthermore, specific instruments are deployed temporarily, on-demand or for testing. Examples include sensors to detect, quantify, and classify suspended particles that have been deployed at the main surface buoy and include the SINTEF SilCam [26] and the LEITAT microplastic sensor [27].

3.2.3. SilCam

The SINTEF Silcam (Silhouette Camera) is an in situ particle imager used to measure high concentrations of suspended particles ranging from 30 μm to several mm in diameter. The SilCam uses a backlit volume to create quasi silhouette images of particles suspended between the light and the camera. The SilCam is able to differentiate between basic particle types (gas bubbles, suspended oil droplets, etc.) and has been extended to zooplankton quantification [28]. The onboard processing and classification of the captured objects can be performed using the PySilCam software, https://github.com/SINTEF/PySilCam, (accessed on 9 July 2024) integrated to the PyOPIA workflow [29]. The interested reader can find a detailed description of the system in [26,30].

3.2.4. Sensors on the Mobile Platforms

Both Eelume and LAUVs are equipped with various payloads for communication, navigation, and in situ data acquisition. The payload includes WLAN, GSM, and Iridium modules for surface communication. Underwater communication and positioning are facilitated by the Evologics (LAUV) or Kongsberg (Eelume) Ultra Short Base-Line (USBL) underwater acoustic modems, which operates within a range of up to 1 km depending on water conditions, along with an emergency acoustic pinger. For navigation, vehicles utilize an Attitude and Heading Reference System (AHRS), a Doppler Velocity Logger (DVL) for speed over ground and altitude, and position inputs from either USBL or GNSS systems, depending on whether the LAUV is submerged or on the surface.

Eelume is a benthic, remote sensing vehicle equipped with a camera and a profiling sonar. LAUV is a water column vehicle with the in situ payload that includes oceanographic sensors such as CTD, Dissolved Oxygen, Chlorophyll, and Turbidity sensors. Additionally, a 200/333 kHz split-beam echosounder operated by a WBT mini transceiver (Kongsberg Discovery, Norway) and an onboard SilCam are integrated into the LAUV’s hull.

3.2.5. Microplastic Sensor

LEITAT has developed an optical device to inspect a continuous stream of water to search for microscopic polymer particles or microplastics (MPs), distinguishing them from natural particles commonly found in seawater, such as sand, bubbles, or seaweed.

The sensor development mainly focuses on the reliable detection of MPs down to 150–200 microns. To this end, it employs an optical detection system based on a laser light directed at a photodiode through a transparent pipe by which the water circulates. When the photodiode stops receiving the laser beam, a trigger event is sent to the image acquisition system, which consists of an RGB camera, a tunable lens, and a magnification objective, to start capturing multiple high-resolution frames of the detected element. The sensor also incorporates an auxiliary lighting system with a white LED to improve image quality and determine shape, texture, or color, as well as UV LEDs to identify fluorescent MPs and facilitate differentiation from organic particles or water effects (Figure 7). Finally, the captured images are analyzed using machine learning/artificial intelligence (ML/AI) models to accurately identify MPs and reject the remaining elements [27].

The design addresses some of the limitations of the existing literature, which has tended to focus on intricate laboratory developments [31], although these are now evolving into low-cost and portable devices to facilitate usage by end users [32]. In this case, the development aims to reduce cost through the use of off-the-shelf cameras or bespoke electronic control systems known for low-power operation, robustness, and effective protection against hazardous environments (including the sensor in an IP67 case). In addition, it keeps the sensor constantly connected to the cloud, allowing users to visualize the resulting information on the presence of plastic particles, as well as access the system remotely to manage configuration options and operate the system online and unattended [27].

3.2.6. Ocean Model Data and Other Data Sources

Data from an ocean model are available through the Norwegian Meteorological institutes who share these data daily with a temporal resolution of one hour and a spatial resolution of 800 m. The spatial resolution is not ideal for a fjord system like the Trondheim fjord, and higher resolution models are under development but not available operationally. They would have to be provided on demand, usually from third-party providers. The SINTEF SINMOD ocean model, for example, can provide simulations that include physical, chemical, and biological ocean variables on a resolution of 32 m every hour. These data will significantly enhance the DTO with respect to accuracy.

AIS data for marine traffic in the area are available and can also be incorporated. Satellite observations as well as static data like bathymetry, mapped biological resources, or maritime/marine infrastructure are examples of additional third-party data that could be used.

3.3. Data Storage and Preparation

One of the challenges when developing Digital Twins is the availability and quality of data. Modern software systems follow a data design pattern that is referred to as medallion architecture [33] where the structure and quality of data will incrementally and progressively improve within the system. Data storage for data from sensors is often optimized for data transfer from the sensor and preservation of the time-series quality of the stored data. Data consumers are concerned about data quality and data availability and less about where the data are coming from. The data quality directly at the sensor or edge is referred to as the bronze layer of the data repository. When the data are stored in a database, they have usually undergone some quality control and been enhanced with metadata, and this is now referred to as the silver layer. The gold layer is data in a format that is intended to be used in data processes like Digital Twins and to be shared with other data consumers. The gold layer refers to data products instead of data files and might, for example, combine data from several sensors into one data set (e.g., ‘all sea water temperature data in the Trondheim fjord’).

An important aspect for data and services used within a DTO is the FAIRness of these components. FAIR [34] stands for Findability, Accessibility, Interoperability, and Reusability. Findability means that data and services can be found in a data and service catalogue. Iliad makes data findable through EMODnet [35] and NextGEOSS [36]. Accessibility means that data and services can be accessed as described in the catalogues. That usually means access through an Application Programming Interface (API). The Water Quality pilot is in the process to enhance available APIs for data from sensor networks and make these available in standard formats that support analysis-ready and cloud-optimized data (ARCO) [37] with accompanying catalogue files (https://stacspec.org/en, accessed on 9 July 2024). This is in agreement with the DTO family of projects in the EU and UN Decade.

The basis for the interoperability of DTO data and services is semantic documentation, i.e., the description of the meaning of the data for the harmonization and contextualization of different data sources. Semantic data documentation allows for standardizing and integrating diverse marine data from various sources. This ensures that data collected by the platforms in OceanLab, using different methods and formats, can be combined and analyzed in a coherent and meaningful way with consistent interpretation and comparison across different datasets. In the process of data documentation, the first step is cataloging the types of data and their metadata. The second step is to enhance the existing access to the data and services to include the semantic information and serve this together with the data via the API. Reusability means that data and services come with a defined license. Even if data and services are open and free to reuse, this should be explicitly mentioned. A popular licence for open science data is CC-BY [38].

3.4. Analytics

In the Water Quality DTO, data analytics involves contextualizing data for comprehensive evaluation and analysis, guiding decision-making based on data insights. We will discuss several tools and models relevant to this work. ML/AI methods are employed for particle identification and classification, which can occur either onboard the sensor or sensor platform (edge processing) or during post-processing (top-side). Numerical modeling with OpenDrift and DREAM is utilized for particle transport modeling, while Grafana is used to embed, display, and interact with model data charts.

PyOPIA. PyOPIA aims to establish a standardized, pipeline-based workflow for analyzing particle images in oceanography. It ensures consistent particle counting across different instruments, making it easier to integrate data from diverse sources. By standardizing size binning methods, PyOPIA simplifies the combination of data from various instruments, converting different units (e.g., 1/L or 1/L/µm) into a unified unit (e.g., µL/L). Once particle measurements have been counted and binned, these particles are numerically transported with OpenDrift under assumptions on particle densities and sinking velocities.

OpenDrift. OpenDrift [39,40] is a software package for modeling the trajectories and fate of objects or substances drifting in the ocean. OpenDrift is designed to be flexible, efficient, and user-friendly, catering to diverse applications such as oil/pollutant spill modeling, search and rescue operations, and tracking marine organisms. Developed and maintained by the Norwegian Meteorological Institute (MET Norway) with contributions from the international scientific community, OpenDrift benefits from continuous updates and improvements based on user feedback and advancements in the field.

Key features of OpenDrift include the ability to model various types of particles, such as oil droplets, plankton, or any other substances transported by ocean currents. It is also highly interoperable, integrating various data sources and numerical ocean models, with outputs that can be saved in CF-compliant netCDF files. Additionally, OpenDrift offers tools for the visualization and analysis of simulation results, including the capability to simulate backward in time.

In our Water Quality DTO, OpenDrift is included to provide the distribution patterns of, e.g., plastic pollution, algae bloom, zooplankton or any other event triggered by the DTO. OpenDrift will transport particles that have been observed at the main observatory buoy using observed current data (’where are particles that have been observed 24 h ago now?’) and modeled current data from the NorKyst800 ocean model [41,42] (’where will particles that we observe now be transported to within the next 24 h?’). Additionally, we use OpenDrift for backwards modeling from the Vanvikan port to assess sources for plastic litter.

DREAM The SINTEF DREAM—Dose-related Risk and Effects Assessment Model [43]—has been developed for regular pollution discharges. Like OpenDrift, DREAM is a so-called Lagrangian model, i.e., it represents a discharge with a number of numerical particles that are transported by ocean currents and wind. In the Water Quality DTO, we employ DREAM to assess the dilution of the discharge from a sewage outlet into the fjord using currents from the ocean model. DREAM is usually used as a decision support tool for managing operational discharges to the marine environment by the offshore energy and aquaculture industry. In the Water Quality pilot, DREAM results are translated into dilution and offered on a map to users who want to assess water quality for bathing.

Microplastic detection The microplastic sensor contains a device (Jetson Orin AGX) that processes the images of detected particles in real time and decides on the nature of each entity. If the procedure results in a positive microplastic particle detection, the device sends the associated information to the cloud to update the database and classify the particle according to the shape and size. The cloud platform allows the visualization of the images including identified microplastic particles and various graphs showing the number of microplastic particles, false positives, bubbles, etc., as well as the number of processed frames in each sensor operating day.

The processing system applies a microplastic segmentation algorithm by using an Intersection over Union (IoU) metric based on deep learning techniques, which allows the classification of each pixel in an image as microplastic, other element (sand, algae, etc.), or image background. Since the image acquisition system takes several consecutive frames of each particle (both illuminated with white and UV light), the algorithm evaluates the entire set and obtains the final result according to the global data. Each frame, before applying the described algorithm, is processed with a K-Nearest Neighbor (KNN) algorithm to obtain a binary image, which is also an input to the algorithm (together with the RGB frame) to enhance the data acquisition and the classification result (Figure 8). The algorithm requires prior training with several datasets, containing empty images, MPs of diverse nature, organic elements, and water effects in order to minimize the number of false positives.

3.5. Interaction, Visualization and Access

To create a custom web application for interacting with and visualizing machine learning models, the pilot uses Grafana. In the Iliad DTO Grafana application (twin.oceanlab.no (accessed on 24 June 2024)), we have embedded particle transport maps created with Streamlit, directly from OpenDrift and from DREAM outputs via Python script together with data charts from the sensor data and weather forecast from Windy. These maps and charts update in near-real time, allowing users to zoom in, move across the chart, and interact with the data.

For any DTO, the immediate visualization of data is crucial as it enables quick detection and response to changes and anomalies. In this pilot, we use Grafana to visualize time-series data from our surface and benthic observatories. Grafana is a powerful, cost-effective open-source platform for monitoring and observability that excels in data visualization. We have chosen Grafana for its ability to handle various data sources and types, allowing for scalable and flexible implementation. Additionally, its customizable dashboards and extensive plugin availability enable tailored visualizations and functionalities. By leveraging Grafana’s features, we can gain valuable insights into water quality patterns and trends in the area, supporting contingency actions, research, and decision-making processes.

We run Grafana on our local machine (or use Grafana Cloud) and link it to our data source. While data can be stored in various time-series databases, we use InfluxDB for this pilot. Alternatively, data can be imported from JSON files. For different applications, we have created suitable dashboards and panels, selecting the appropriate visualization types (e.g., time-series, bar chart, heatmap) based on the data and the insights we aim to convey. A significant feature is the ability to configure alerts that trigger when certain conditions are met, such as when chlorophyll levels exceed a threshold. Additionally, we use the Grafana map plugin to visualize geographical data, including AIS marine traffic, oceanographic data, and asset distribution.

4D gaming visualization of particles in seawater Developing an interactive visualization of a Digital Twin of the Ocean presents a unique set of challenges, like the real-time data acquisition and integration from various sources, such as satellites and in situ sensors, and creating an interface and user experience that allows users to interact with complex data in an intuitive and meaningful way. One approach to visualize ocean particles that are too small to be seen in nature and to cater to an audience that is used to immersive visualization has been to use gaming technology. The Unity game engine, with the Cesium Plugin for Unity and the Entity Component System, allows for the dynamic representation of large datasets at runtime and, thus, provides the necessary tools to integrate real-time data from diverse sources and render them dynamically. Cesium provides a visualization of geographical features like the bathymetry of the ocean area, providing a detailed representation of underwater terrain. Furthermore, to address the user experience, principles of game design were applied with a focus on user engagement and intuitive interfaces and effective strategies for presenting complex data in an accessible and meaningful way. The SINTEF Silcam in this pilot, which detects particles in the seawater and categorizes them by size (from observations), possible type (from classification), and depth (from modeling), provides an interesting and challenging candidate for developing an application providing data-driven visualization in an interactive 4D digital environment. Particles in the ocean can range from non-hazardous entities such as small zooplankton (copepods) to potentially hazardous materials like oil droplets or microplastics. Being able to distinguish between these different types of particles and to understand their distribution and impact is crucial for both the preservation of marine ecosystems and the management of human activities at sea. The interactive 4D digital environment is seen as an opportunity to provide a more comprehensive visualization than what was achievable with traditional scientific tools like charts, maps, and graphs.

The file format NetCDF that is accepted by the ocean science community and often produced by ocean and environmental models is usually not easily digestible by map or non-scientific software. The .NET Microsoft Scientific DataSet library was used to parse the NetCDF data to common data structures in C# to create the 3D visualization of the particles over time, effectively creating a 4D visualization. The software considers features like the particles’ longitude, latitude, and depth, as well as particle type and size, to offer a comprehensive view of their distribution and movement. While at the present stage of the application, the application connects to a local file for the particle data, future versions are envisioned to connect to an API for data access and regular updates.

Section 4.1 describes the results and breakthroughs obtained from this development process.

3.6. Algae Bloom Case Study

This section describes the motivation and methodology behind the algae bloom monitoring that utilizes DTO resources. Motivation comes from the fact that the intensity and frequency of algal blooms in coastal upwelling regions is expected to increase as the intensity and frequency of local weather event increase [44]. These events are often driven by eutrophication and ocean warming, caused by climate change and other human activities [45]. On the positive side, phytoplankton are primary producers that provide food for marine organisms and play a crucial role in the carbon cycle by transferring carbon dioxide from the atmosphere to the ocean through photosynthesis. However, harmful algal blooms negatively impact ecosystems by reducing oxygen or releasing toxic substances. Understanding and forecasting these blooms could help mitigate their impact on fish populations and serve as a valuable tool for the aquaculture industry to manage risks and prevent losses.

Phytoplankton respond rapidly to environmental changes. During a bloom, zooplankton consume large amounts of phytoplankton, helping to regulate the bloom’s intensity and duration. Rising zooplankton populations trigger a chain reaction through the marine food web, impacting higher trophic levels like fish that prey on zooplankton and ultimately affecting the entire ecosystem. In healthy ecosystems, the dynamic between phytoplankton and zooplankton is a fundamental process that sustains marine life. However, disruptions to this balance can alter the frequency and intensity of algal blooms, negatively impacting zooplankton and the broader ecosystem. Therefore, measuring the distribution, species composition, and abundance of a bloom is essential for determining its ecological impact and potential for harm.

Continuous monitoring with in situ chlorophyll sensors, optical devices (SilCam), and acoustic sensors (echosounders) from buoys can indicate the onset of an algal bloom and help monitor its progress. By measuring chlorophyll concentrations in the water, we can estimate the abundance of phytoplankton. The sound scattering layers (SSLs) are a visual indication in echosounder backscatter of the presence of an aggregation of zooplankton. By monitoring the changes in the presence and backscattering strength of SSLs over long time periods, we can identify the increased presence of zooplankton, which would indicate the onset of an algae bloom. Using SilCam, we can detect and categorize the size and the type of zooplankton and feed these data into models to monitor the evolution of the bloom.

4. Results and Discussion

4.1. Visualizations

The majority of data collected by the observatories and AUVs is time-series data. The Grafana dashboard works neatly with InfluxDB where these time-series are stored and is utilized for visualizing, as illustrated in Figure 9.

To analyze the dispersion of discharges, particles, and zooplankton in the Water Quality DTO, numerical models like OpenDrift and DREAM are used. Individual Streamlit plots and OpenDrift visualizations have been integrated into Grafana dashboards via an HTML panel, including an interactive map to show different layers for the different model applications. The main pilot Grafana dashboard, illustrated in Figure 10, provides a comprehensive overview of various pilot data, including time-series, video streams, maps, and embedded applications.

The application developed by partner Breda University of Applied Sciences (BUas) and introduced in Section 3.5 allows the visualization of the micro-particles and uses the three variables associated with them (type, diameter, depth) to code them through color or darkness and to filter them on those same variables. To provide an example, this means that the users could visualize, at the same time, the different types of particles by their different colors and their depth by their darkness while filtering out (excluding them from the visualization) all the particles with a diameter over a threshold they defined. The color/darkness coding and filtering are to be considered part of the added value to the visualization provided by the BUas application compared to traditional scientific visualization. In fact, they allow users to filter and identify the particles seamlessly and dynamically at runtime. Users can also pause, play, or speed up the simulation whenever they want. Another unique feature is the possibility for users to move freely in the world. They can use the application as a 2D map and move by dragging the view around, or they can use the keyboard in the 3D environment and position the camera at any point in space, in the sky, or below the water’s surface. Moreover, the application provides three different visualizations of the map: satellite and scientific bathymetry shown in Figure 11 and street map. The application, in its version tailored around Iliad’s data (Figure 12), allows users to see the real data detected by Iliad’s sensors and displayed on their instance of Grafana. Note: The application is still under development at the time of writing this paper, so its final aspect might change.

4.2. Particles

4.2.1. Microplastic

The MP sensor was first validated in the laboratory before deployment, employing samples containing seawater and MPs of different sizes but always above the low detection limit (around 150 microns). The procedure allowed the refinement of the processing protocol and the algorithm for identifying MPs among all detected particles, going from an initial identification efficiency of 70% to a final efficiency of 90% by adjusting parameters and improving sub-processes.

In parallel to the improvement procedure, the sensor was deployed in the buoy observatory (Figure 4) to obtain data from a real environment. In this case, the probability of obtaining particles passing through the sensor is lower than in the laboratory validation since the presence of MPs in an open marine environment is more limited than in concentrated samples. However, by selecting a specific 16-day period, the graphs in Figure 13 show the data collected: the total number of particles (or other elements) detected each day, the number of particles identified as MPs, as bubbles, and those remained undefined after applying the ML/AI algorithm. In some cases, the images taken do not contain any identifiable entity, and they are labeled as lost data, which could be related to coordination problems between the optical and image acquisition systems or false positives from the optical detection (particles below the minimum size, water effects that fade during the water circulation, etc.).

Taking the reference period of 16 days, the sensor detected, on average, 66.7 particles per day. Most of these events, 38.7, were triggered by the appearance of different water effects, mainly bubbles. The remaining 28 frames corresponded to MPs and organic particles. A thorough analysis of individual frames revealed that the sensor only detected 1.4 plastic particles per day, which is not surprising as the pilot is located in the clean water of the Norwegian fjord. Figure 14 shows a frame with a detected particle that the processing system classifies as an MP and another frame that clearly shows a bubble that the AI algorithm rejects as an irrelevant entity.

4.2.2. Algae Bloom

In the algae bloom case study, we visually determined a daytime increase in the occurrence of surface SSL during the summer based on a time-series view (Figure 15). We observed the regular pattern of diel vertical migrations from the zooplankton layers present from 2 to 6 June 2024 (Figure 15A) weakened as the surface layer increased in backscattering strength from 7 to 11 June 2024 (Figure 15B). The intensifying consistent presence of the surface SSL in the Trondheim fjord triggered the decision to deploy an AUV equipped with an imaging sensor (SilCam) and oceanographic sensors such as CTD, Dissolved Oxygen, Chlorophyll, and Turbidity sensors on 11 June 2024 (Figure 16). AUVs provide wider spatial coverage and can create depth profiles, which the stationary in situ sensors cannot achieve. The LAUV was chosen for the mission that covered an area extending 1 km north of the buoy, toward the middle of the fjord. The LAUV executed a yo-yo mission within the upper 40 m of the water column. Sensor data from the LAUV mission are provided in Figure 17 left. We can observe a distinct surface water layer in the upper 12–13 m, with higher temperatures, lower salinity levels, and slightly higher chlorophyll concentrations. Data from the SilCam was processed by PyOPIA to identify the type and size of zooplankton and the depth at which it was recorded, as shown in Figure 17 right. After the deployment, we observed a decrease in surface SSL backscatter strength and a return of a diel vertical migration pattern (Figure 15C).

For this example, the increase in sound scattering layer (SSL) intensity (a proxy for increased abundance or size of zooplankton) was determined visually and monitored over a two-week period. However, an automatic layer detector could increase the objectivity and automation of the SSL detection. The layer detector comprises smoothing the volume backscatter echograms in time and depth and percentile thresholding and morphological filtering based on the thresholded echogram with scikit-image morphology [46]. Metrics are calculated based on these detected SSLs. By applying thresholds to the metrics, an alert can be implemented to determine when an SSL is present. Once we have generated SSL alerts based on the adaptive thresholds of the echosounder data, we can then trigger an AUV mission. This time, the AUV was operated from a boat, but ideally, it should become part of the benthic station i.e., become a resident vehicle that can start and execute its missions automatically. Using the PyOPIA workflow, the presence of individual copepods can be determined within the acquired SilCam data. Each particle is individually classified, and individual images of copepods can be identified, as shown in Figure 17. This integrated approach to observing blooms could be further integrated into a broader observational system, which incorporates satellite and aerial imagery, also known as an “observational pyramid” [47].

4.2.3. Dispersion of Particles

The key novelty in our DT process is automation. Alerts are generated in case of algae bloom, if the presence of SSL is determined, or in case of MP presence, if the AI algorithm categorizes particles as MPs. These alerts trigger customized AUV missions to collect more relevant data for a targeted case study. This information helps us understand the problem and provide oceanographic and particle (type, size, and concentration/density) inputs that are fed into the transport model. The results of the transport model are visualized using Streamlit as illustrated in Figure 10. Streamlit, coupled to the model, allows us to interact with the data and simulate different scenarios. In this demonstration, we also utilized SilCam particle transport with Opendrift and NorKyst800, along with pilot ADCP data, to study the dispersion of particles or zooplankton. This contributes to our understanding of ecological processes and population dynamics.

The time frame to complete the loop from sensors’ signal to result visualization will depend on the sensors and sensor platform used. DTO data from the buoy are updated regularly every six hours. In the event of an alert, the DTO can adjust the update frequency as needed, with the time frame for responding to an acute event ranging from 15 to 30 min for the entire loop, depending on the transport model setup. Customized AUV missions, if required, can be executed on the same day, although the exact timing may vary depending on the mission location, weather conditions, or the payload needed.

Figure 18 illustrates the movement of particulates at the site over a 24 h period, driven by ocean currents derived from the operational NorKyst800 model. The settling speed of the particulates is calculated using Stokes’ law and the measured size–concentration spectra, assuming a fixed particle density.

5. Conclusions

The Digital Twin of the Ocean is a rapidly emerging topic that has attracted significant interest from scientists in recent years. Key enablers of this development include advancements in IoT devices, marine sensors, big data, ML and AI intelligence, visualization technology, and data science. The DTO initiative is strongly driven by the UN Decade of Ocean science and the EU where it is part of the European Green Deal, DestinE, and the EU’s digital strategy that aims to create a digital replica of the ocean to better understand and manage marine environments. The Iliad project, funded under the EU Horizon 2020 programme, is developing a framework to support multiple interoperable DTOs and pilots using a federated systems-of-systems approach across various fields, including water quality, oil spills, aquaculture, jellyfish forecasting, insurance, risk assessment, and cultural heritage etc. Being complementary to the EU Digital Twin of the Ocean (EU DTO) infrastructure and models, it offers examples, data, and Digital Twin components that can be reused to build and replicate twins in other ocean regions and for different applications and purposes.

This paper details the building blocks of Iliad Digital Twins along the example of the Iliad Water Quality pilot, which adheres to a standard DTO architecture that is compatible with the EU DTO. This architecture combines ocean observations with ocean predictions within an interoperable data space or lake. A crucial aspect of any DTO is its data, which can be sourced internally, externally, or through a hybrid approach utilizing both. The discussed pilot emphasizes acquiring its own data continuously or on demand from sensors and sensor platforms, including two novel micro-particle sensors: a microplastics sensor and the SINTEF SilCam, which targets a broad size and range of particles such as zooplankton, sediment, and oil droplets. To realistically twin ocean processes, the pilot acquires data from both surface and benthic observatories, as well as from AUVs serving as mobile sensor platforms for on-demand data collection.

Data are processed from the source to the application following the medallion data model with a bronze layer for the raw data to the gold layer with analysis-ready, cloud-optimized data and formats that are shared via standardized API. Semantic, machine-to-machine interoperability will be realized through a semantic documentation of the data and a controlled vocabulary and ontology.

The pilot offers a user interface for interacting with the DTO and setting up alerts or events to support ’what-if’ scenarios. Grafana has proven effective for presenting and interacting with time-series data, and its use has been expanded to include more complex visualizations such as time-series, video streams, maps, and embedded applications. Another approach to interactive visualization leverages game technologies, utilizing their advanced rendering capabilities and physical computations to integrate and dynamically render real-time data from diverse sources. Additionally, game design principles have been applied to address user experience challenges.

The results section includes two case studies: the detection of microplastics and the monitoring of algae blooms in the fjord. Both studies illustrate the use of pilot particle sensors. The algae bloom study details an automated detection, tracking, and observation process initiated by an algae bloom alert from an echosounder at the surface observatory. This alert triggers an AUV mission to collect comprehensive spatial data on oceanographic conditions, chlorophyll, and zooplankton. Currently, this process is semi-automatic but will be fully automated in the future. The section then presents a method using OpenDrift and particle transport models to forecast the evolution of these events, simulating the distribution of zooplankton or microplastics during the forecasting period.

The suggested approach currently faces some limitations, primarily due to the availability of an ocean model with sufficient resolution to accurately represent details within the fjord environment. While the model used is adequate for the central areas of the fjord, a higher-resolution model for the near-coast areas and the port is under development and will be integrated into the DTO once validated data are available. Additionally, the current transport model, while sufficient, can be replaced by a more advanced model in response to events such as oil spills from ship traffic or other unexpected discharges. The application has already been tested for sewage discharge into the fjord. The DTO architecture is designed with flexibility in mind, allowing for the exchange of components as improved data, sensors, and models become available or are required for different applications, such as ship routing within the fjord.

Future work will concentrate on two main activities: finalizing the Water Quality pilot with API and semantic interoperability for all data and services as well as replicating the pilot in the public EU DTO infrastructure which is possible through the Iliad system-of-systems approach. All data that can be shared openly will be pushed to the EU data ecosystem through EMODnet. Additionally, through the work of the project DTO BioFlow, data pipelines for biological data that have been acquired through the example of the algae bloom monitoring and other contexts will be established.

The pilot is currently in the demonstration and impact assessment phase, with a focus on enhancing the interconnections between pilot modules and replication in the Baltic Sea of Northern Germany. We aim to automate alert generation for various situations to trigger actions or aid decision-making, implement and refine ocean and transport models, expand the range of analytic tools, including ML algorithms, optimize data visualization options (time-series, spatial, or blob data), and prepare the pilot for further enhancement and replication. The pilot will be open, flexible, and scaleable to allow the seamless integration of new systems and data sources from the upcoming developments in the Norwegian Centre for Ocean Technology, which will include the aquaculture and autonomous shipping sites as well as ocean laboratories for large-scale testing and maritime transport.