**4. Conclusions**

This paper represents a significant collaborative e ffort that took place during the first year of a major Canadian research and development program, the Oceans Protection Plan. The oceanography sub-initiative of the OPP was tasked to develop port-scale models to fit into the multi-scale operational

ocean prediction systems in Canada, i.e., to extend the prediction capability from global, basin and coastal scales to nearshore waters. The work included three components: (1) the development of an evaluation process, (2) the development of model configurations with two state-of-the-art open source community models, and (3) the evaluation of the performances of the two model configurations including the comparison of each with the existing operational models. Note that the second and third components were carried out simultaneously, and both model configurations were constantly improved and fine-tuned with the guidance of the evaluation.

The evaluation process includes (1) the selection of the study area, (2) the requirements for the model setup, and (3) the metrics for evaluating the models. The selection of the study area was specific to the nature of the OPP project in that one of the six OPP pilot ports was selected, but the choice of port enabled a general assessment of the models for near-shore port-scale applications. The requirements for the model setup were quite general for evaluating the performance of di fferent source models under the same setting, but by ensuring that both models were configured using the same sources of input data for the model setup and forcing, the di fferences in the model performances can be mostly attributed to the di fferences in grid structure, model numerics, and software technology. Thus, the results of this study are valuable for further improvements of the models. The metrics can be used for evaluating any model. They were selected to assess the various aspects of the model results, and collectively, they were able to detect minor contrasts between the two models.

The chosen study area, Saint John Harbour in the Bay of Fundy, features the presence of strong tides, significant river runo ff and a narrow tidal-river channel (Reversing Falls). The complicated regional oceanography posed challenges to both the NEMO and FVCOM configurations with the full baroclinic dynamics included. The common challenge was to ensure that the models ran stably, particularly in the narrow channel with strong currents and spatial gradients of temperature and salinity. For this purpose, both models included local treatment of bottom drag parameterization. Previous configurations of the FVCOM model for the Saint John Harbour area did not include atmospheric forcing. Adding atmospheric forcing, particularly the surface heat-flux, was a challenge, but the issue was later attributed to the previously mentioned bug in the code. A challenge for NEMO was the lack of experience in creating a multiple-level nested configuration for nearshore waters. Despite the urgen<sup>t</sup> timeline, both configurations were created and improved during the first year of OPP.

The evaluation metrics were defined based on the existing expertise of the team, through expert consulting and literature research, as well as the available ocean observational data. Evaluation with these metrics led to continuous improvements of both models, that is, errors in model settings (bathymetry, model parameters and forcing) were constantly identified and corrected. The evaluation results presented in Sections 3.4–3.8 were based on the model results obtained toward the end of the first year of OPP. In terms of model accuracy, compared to the observational data, NEMO and the FVCOM achieved comparable metrics for the tidal and non-tidal components of sea level and currents, seasonal variation of the sea surface temperature, vertical profiles of water temperature and salinity, and the trajectories of surface drift. Note that for the evaluation of the drifter trajectories, the impacts of winds were not fully considered. This aspect needs to be included in future work. A major di fference identified was that the FVCOM required less computer resources and ran faster than the NEMO. One possible solution to reduce the required number of the computer slots is to reduce the domain size of the finest model configuration (SJAP100), and using the two-way nesting approach to ensure the dynamic interaction across di fferent configurations are properly simulated. Initial testing on two-way nested configuration was achieved after the completion of the evaluation in the first year (results not shown). The results of the model e fficiency evaluation sugges<sup>t</sup> that both the NEMO and FVCOM meshes could be further refined, but in doing so, the NEMO model would not meet the operational requirement for runtime due to the limitations of the model numerics and computer e fficiency.

Finally, the evaluation of the models was limited by the existing observational data. With increasing resolution, the models are reaching the limits to which the ocean is observed in terms of small scale, rapidly varying, and nonlinear processes. To support the envisaged future of port-scale operational e-navigation systems, we need to reassess what observational capacities are required to support this, both from a monitoring point of view (i.e., real-time and high spatial resolution data), as well as for model development (e.g., delayed-mode data such as moorings).

**Author Contributions:** Conceptualization, S.N.; methodology, Y.L., G.C.S., N.B.B., P.M., F.D., D.G. and F.J.M.D.; software, S.H., H.B. and G.M.; validation, S.N.; formal analysis, S.H., S.N., H.B.; investigation, S.P.H., J.-P.P., M.O.-S., S.T., H.B., G.M., Y.W., L.Z., X.H., J.C., M.D. and D.G.; resources, Y.L., N.B.B.; data curation, F.P., P.M., F.D. and S.P.H.; writing—original draft preparation, S.N., Y.L., H.B., S.P.H. and G.M.; writing—review and editing, S.H., M.O.-S., S.T., G.C.S., N.B.B., P.M., L.Z., X.H., F.D., D.G. and F.P.; visualization, S.H.; M.O.-S., S.P.H., S.N. and D.G. Supervision, S.N. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Government of Canada's Ocean Protection Plan.

**Acknowledgments:** The authors would like to acknowledge the following colleagues for their contributions, guidance and support for this project: Herman Verma, Sarah Scouten, Ji Lei, Adam Drozdowski, Charles Hannah, John Chamberlain, Guoqi Han, Sherry Niven, Kim Housten, Melinda Lontoc-Roy, and Pierre Pellerin. We are grateful to the two anonymous reviewers for the constructive comments that guided the revision of the manuscript.

**Conflicts of Interest:** The authors declare no conflict of interest.
