Non-Intrusive Water Surface Velocity Measurement Based on Deep Learning

Abstract

Accurate assessment of water surface velocity (WSV) is essential for flood prevention, disaster mitigation, and erosion control within hydrological monitoring. Existing image-based velocimetry techniques largely depend on correlation principles, requiring users to input and adjust parameters to achieve reliable results, which poses challenges for users lacking relevant expertise. This study presents RivVideoFlow, a user-friendly, rapid, and precise method for WSV. RivVideoFlow combines two-dimensional and three-dimensional orthorectification based on Ground Control Points (GCPs) with a deep learning-based multi-frame optical flow estimation algorithm named VideoFlow, which integrates temporal cues. The orthorectification process employs a homography matrix to convert images from various angles into a top-down view, aligning the image coordinates with actual geographical coordinates. VideoFlow achieves superior accuracy and strong dataset generalization compared to two-frame RAFT models due to its more effective capture of flow velocity continuity over time, leading to enhanced stability in velocity measurements. The algorithm has been validated on a flood simulation experimental platform, in outdoor settings, and with synthetic river videos. Results demonstrate that RivVideoFlow can robustly estimate surface velocity under various camera perspectives, enabling continuous real-time dynamic measurement of the entire flow field. Moreover, RivVideoFlow has demonstrated superior performance in low, medium, and high flow velocity scenarios, especially in high-velocity conditions where it achieves high measurement precision. This method provides a more effective solution for hydrological monitoring.

1. Introduction

Streamflow data constitute a fundamental component of hydrological research and serve as the cornerstone for all hydrological investigations. Particularly amidst the prevalent occurrences of extreme events, such as floods and droughts, the demand for expeditious acquisition of critical information regarding river flow velocity and runoff variations has become increasingly urgent [1]. Traditional velocity measurement methods, such as those utilizing propeller current meters and an Acoustic Doppler Current Profiler (ADCP) [2], although well-established and widely employed for field measurements, are significantly constrained by factors including weather conditions, topography, and hydraulic regimes. Moreover, these methods require manual intervention and are impractical for large-scale deployment. Consequently, they have failed to meet the evolving requirements of modern hydrological measurement practices.

In response to the challenges outlined above and to accomplish remote monitoring tasks, a method for measuring water surface velocity (WSV) based on video imagery has emerged in recent years. This approach has garnered considerable attention due to its notable safety and high measurement efficiency. The fundamental process of this method is as follows: initially, a camera is installed on one side of the riverbank to capture video imagery of the river surface. Subsequently, an image-based velocimetry method is employed to estimate the pixel-level two-dimensional velocity field of the water flow from the video imagery. Following this, an affine transformation is applied to convert the velocity field into real flow velocities.

Image velocimetry can be divided into Particle Image Velocimetry (PIV), Particle Tracking Velocimetry (PTV), Space-Time Image Velocimetry (STIV), and Optical Flow Velocimetry (OFV), according to difference motion vector estimation methods. PIV, particularly the Large-Scale PIV (LSPIV) proposed by Fujita et al. [3], estimates flow velocities by analyzing the motion of tracer particles captured by high-speed cameras. However, the accuracy of LSPIV is significantly influenced by the density and distribution of tracer particles on the water surface [4]. PTV, a Lagrangian perspective-based method, reconstructs the velocity field by tracking the trajectories of individual particles within the field of view [5,6]. Nevertheless, the performance of PTV is affected by environmental conditions such as lighting, wind speed, and shadows, which are often difficult to control in the field. Although researchers have implemented various measures to reduce the interference of these environmental factors, the collected image data may still contain noise that is challenging to completely eliminate [7]. On the other hand, STIV calculates flow velocities by identifying the texture angle produced by the grayscale changes of flow lines, and while it has a distinct advantage in terms of processing speed, it is highly sensitive to noise [8]. OFV methods, based on the brightness constancy assumption, obtain motion vectors by analyzing pixel displacements between consecutive images. Xu et al. [9] proposed a sub-grid scale variational optical flow algorithm that effectively addresses the issue of sub-grid small-scale structural information not captured in traditional grid-scale optical flow estimation. Wang et al. [10] employed a dense inverse search method, combined with a frame difference approach, to propose a fast optical flow-based flow measurement algorithm that has made significant progress in improving measurement stability. Recently, deep learning-based optical flow estimation methods have demonstrated significant performance advantages in WSV measurements. For instance, the RivQNet algorithm proposed by Ansari et al. [11], which is based on the FlowNet 2.0 network structure [12], although superior to traditional optical flow methods in terms of computational results, still has room for improvement in terms of accuracy.

This study introduces a user-friendly river surface velocity measurement algorithm that combines orthorectification and deep learning-based multi-frame optical flow estimation. Firstly, an implicit camera calibration method based on Ground Control Points (GCPs) is proposed, eliminating the need for additional chessboard images to obtain camera parameters. Using the calibration results, only four GCPs are required to perform two-dimensional or three-dimensional orthorectification, transforming the image into a “bird’s eye” view. Subsequently, the VideoFlow [13] multi-frame optical flow network, which fully considers the temporal continuity of flow velocity, is employed to estimate pixel displacement. In contrast to current approaches, this research introduces an innovative method for measuring flow velocity that avoids the need for complex equipment and preliminary processing. This method is particularly valuable for practical applications in monitoring river flow velocities. To verify the feasibility of the proposed method, a multi-scenario river simulation platform was constructed in a laboratory flume with a particular focus on verifying flood-level flow velocity measurements. Experimental results indicate that this method is straightforward to use, highly efficient, and yields reliable outcomes.

2. Materials and Methods

2.1. Image Correction

Due to the large-scale field of view and oblique viewing angles often encountered in river images collected in the wild, it is necessary to use orthorectified images, free from lens distortion and perspective effects, to compute velocity and flow rates. The latter issue makes orthorectification a crucial step. When lens distortion is negligible, the pinhole camera model can be employed. Assuming that the water surface is planar, orthorectification is typically achieved through plane-to-plane projection, matching any real position on the orthorectified plane to the corresponding position on the image plane. This plane-to-plane projection is defined by 11 coefficients, which must be estimated through either implicit or, less commonly, explicit camera calibration. Implicit camera calibration is a standard estimation procedure in which the 11 coefficients are fitted using a limited number of reference points with known images and real-world coordinates [14]. Explicit camera calibration is based on a camera model, with parameters calculated from physical parameters such as camera position and angle, focal length, sensor size, and center location. The 11 coefficients of the plane-to-plane projection can then be derived from the camera model parameters. In this paper, we introduce an implicit camera calibration method and fit the camera extrinsic parameters using GCPs.

2.1.1. Camera Calibration

To perform distortion correction and orthorectification of images, both the intrinsic and extrinsic characteristics of the camera are essential. The intrinsic parameters are specific to the camera and include the focal lengths $(f_x,f_y)$ and the principal point coordinates $(c_x,c_y)$. The extrinsic parameters define the rotation and translation that transforms the world coordinate system into the camera coordinate system. Thus, the extrinsic parameters enable the transformation from the real world to the camera and finally to the image plane, as illustrated in Figure 1. The matrix formula is as follows:

$$s\left[\begin{array}{l}u\hfill \\ v\hfill \\ 1\hfill \end{array}\right]=\mathbf{K}\left[\begin{array}{cc}\mathbf{R}& \mathbf{t}\end{array}\right]\left[\begin{array}{l}X\hfill \\ Y\hfill \\ Z\hfill \\ 1\hfill \end{array}\right]$$

(1)

$$\mathbf{K}=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$$

(2)

whereas $\mathbf{K}$ is the camera intrinsic matrix $\mathbf{R}$ is the rotation matrix, transforming the world coordinate system to the camera coordinate system, and $\mathbf{t}$ is the translation vector, translating the world coordinate system to the camera coordinate system. s is the scale factor.

For the estimation of the extrinsic parameters $\mathbf{R}$ and $\mathbf{t}$, the scale factors, we employ GCP fitting. This involves minimizing the projection error of 3D points onto the image plane using the Levenberg-Marquardt [15] non-linear optimization algorithm. The error is defined as the distance between the projected points and the actual 2D image points. Given ${\mathbf{P}}_i$ as the 3D points and ${\mathbf{p}}_i$ as the corresponding 2D points, along with the camera intrinsic matrix $\mathbf{K}$ and distortion coefficients $\mathbf{D}$, we solve for $\mathbf{R}$ and $\mathbf{t}$ such that:

$$\mathrm{m}\mathrm{i}{\mathrm{n}}_{\mathbf{R},\mathbf{t}}{\displaystyle \sum _i{\Vert {\mathbf{p}}_i-\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\left({\mathbf{P}}_i,\mathbf{R},\mathbf{t},\mathbf{K}\right)\Vert }^2}$$

(3)

The project function maps the 3D point Pi onto the 2D image plane by applying the rotation $\mathbf{R}$, translation $\mathbf{t}$, distortion correction, and intrinsic projection.

2.1.2. Distortion Correction

The two main types of distortion are radial and tangential. Radial distortion causes straight lines to appear curved, which is due to the shape of the lens. This effect is more pronounced for points farther from the image center. Two types of radial distortion can occur: a positive one known as Barrel distortion and a negative one known as Pincushion distortion. These are illustrated in Figure 2. When employing image velocimetry to measure WSV, correcting for radial distortion is crucial because the magnitude and direction of the velocity vectors are affected. As the magnitude decreases, vectors farther from the center become smaller (Figure 2b); radial distortion can be corrected using the polynomial in Equation (4).

$$\begin{array}{l}\widehat{x}=x\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)\\ \widehat{y}=y\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)\end{array}$$

(4)

$\widehat{x}$ and $\widehat{y}$ are the image coordinates after radial correction, while $k_1$, $k_2$, and $k_3$ are parameters to be determined, which can be calculated during camera calibration. $r$ is the radial distance from the image center to the point, i.e., $r=\sqrt{{\left(x-c_x\right)}^2+{\left(y-c_y\right)}^2}$. In addition, a comprehensive image distortion correction model should also include tangential distortion. The most commonly used method for addressing tangential distortion is the Brown-Conrady model [16].

2.1.3. Orthorectification

Orthorectification is the process of performing geometric correction on an image to eliminate distortions caused by perspective projection, presenting the image from an overhead viewpoint while preserving the relative sizes of the objects within the image. The simplest method for capturing images or videos of a river is to capture images from the shore. However, due to the influence of the camera angle, the velocity vectors obtained from optical flow calculations may exhibit some distortion. To address this, we employ a non-parametric orthorectification approach to correct perspective distortion. Specifically, we utilize the extrinsic parameters obtained from camera calibration, including the rotation matrix ${\mathbf{R}}_1$ and translation vector ${\mathbf{t}}_1$, along with the origin, orientation defined by rotation matrix ${\mathbf{R}}_2$, and position defined by translation vector ${\mathbf{t}}_2$ of the overhead viewpoint coordinate system, to compute the homography transformation matrix, as shown in Equation (5). This transformation matrix enables conversion from the camera coordinate system to the world coordinate system, as described in Equation (6).

$$\mathbf{H}={\mathbf{R}}_2{\mathbf{R}}_1^{\mathrm{T}}+{\displaystyle \frac{1}{d}}\left(-{\mathbf{R}}_2{\mathbf{R}}_1^{\mathrm{T}}{\mathbf{t}}_1+{\mathbf{t}}_2\right)n^T$$

(5)

$$s\left[\begin{array}{l}u\hfill \\ v\hfill \\ 1\hfill \end{array}\right]=\mathbf{K}\left(\begin{array}{ccc}{H}_{11}& H_{12}& H_{13}\\ H_{21}& H_{22}& H_{23}\\ H_{31}& H_{32}& 1\end{array}\right){\mathbf{K}}^{-1}\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]$$

(6)

Unlike the RIVeR (2020b) [17] software, which is specialized for the orthorectification of PIV results, our method imposes less stringent requirements on the number and distribution of GCPs. We can perform a two-dimensional transformation using four GCPs located on the same plane as the water surface or a three-dimensional transformation using at least four GCPs not confined to the same plane. In either case, a greater number of GCPs will enhance the accuracy of the transformation. Figure 3 presents the orthorectification results of an image of the Braque River [18] captured from a bridge using four GCPs located on different planes.

2.2. Principle Analysis of OFV Method

Traditional optical flow. Traditional optical flow estimation is a crucial problem in computer vision, aiming to infer the displacement field of objects between consecutive frames in an image sequence. Traditional methods treat optical flow estimation as an optimization problem, relying heavily on handcrafted features to quantify the similarity between image pairs. While performing well in some scenarios, traditional approaches may struggle with complex image scenes, especially those with discontinuous textures or drastic changes in lighting. Hence, with the advancement of deep learning techniques, recent years have witnessed a gradual shift towards deep learning-based optical flow estimation methods, which have demonstrated superior performance compared to traditional handcrafted feature-based methods.

Deep learning-based two-frame and multi-frame optical flow. In this study, we adopted the VideoFlow optical flow estimation framework, which further develops upon existing two-frame optical flow estimation techniques. Specifically, FlowNet [19], as the first convolutional neural network (CNN)-based end-to-end optical flow network, uses an encoder-decoder structure and a feature pyramid to predict optical flow at different scales. While FlowNet and its derivatives [12,20,21] have made big strides in optical flow estimation, they sometimes struggle with multi-scale flows and subtle motions. This can result in rough predictions, which may cause errors in further calculations. To address this issue, the RAFT [22] model was integrated, which iteratively refines the flow field through correlation volumes and cyclic Conv-GRU units, significantly enhancing the accuracy of optical flow estimation. Furthermore, with the emerging advantage of transformer architectures in capturing fine spatio-temporal details, transformer-based optical flow estimation models, such as GMFlow [23] and FlowFormer [24], have gained attention. These models further improve performance by more effectively modeling spatio-temporal dependencies. The VideoFlow architecture extends the GMFlow two-frame optical flow estimation model by incorporating temporal coherence across consecutive frames, effectively computing bidirectional optical flow within multiple frames. This approach not only improves the accuracy of optical flow estimation but also enhances the model’s adaptability to complex dynamic scenes, which is crucial for flow velocity measurement in hydrology.

2.3. Model Features and Architectures of VideoFlow

VideoFlow is an advanced framework designed to estimate optical flow in video sequences. It effectively calculates bidirectional optical flows across multiple frames by utilizing temporal information. The method involves segmenting the input video into overlapping sets of three frames each. The Tri-frame Optical Flow (TROF) module processes each set individually, while the Motion Propagation (MOP) module integrates these outputs to produce a comprehensive multi-frame optical flow estimation.

2.3.1. TROF for Tri-Frame Optical Flow Estimation

The TROF module is specifically designed to forecast bidirectional optical flows within sequences of three contiguous frames $\left\{{\mathit{I}}_{t-1},{\mathit{I}}_t,{\mathit{I}}_{t+1}\right\}$. It iteratively calculates a series of bidirectional flows ${\mathit{f}}^k$, encompassing both forward flows ${\mathit{f}}_{t\to t-1}^k$ and backward flows ${\mathit{f}}_{t\to t+1}^k$, where denotes the iteration step. TROF primarily comprises four stages: extraction of features from three frames via a Twins-SVT [25] encoder, computation of dual all-pairs correlation cost volume, fusion of bidirectional motion features, and iterative updates based on super kernel blocks. It is noteworthy that, apart from employing the Twins-SVT encoder for feature extraction, the rest of the process relies on SKBlock [26] composed of large-kernel deep convolutional layers. The TROF framework is depicted in Figure 4.

The process of bi-directional motion feature fusion in TROF. The essence of the TROF method lies in the effective integration of bidirectional motion information into the central frame within the temporal dimension. Following a methodology akin to the RAFT method, TROF initially employs a feature encoder based on Twin-SVT to convert the three input images into feature maps. These feature maps are subsequently utilized to compute bidirectional correlation volumes from the central frame to the previous and next frames i.e., ${\mathit{c}}_{t\to t-1}^k={\mathit{Corr}}_{t\to t-1}\left({\mathit{f}}_{t\to t-1}^k\right)$ and ${\mathit{c}}_{t\to t+1}^k={\mathit{Corr}}_{t\to t+1}\left({\mathit{f}}_{t\to t+1}^k\right)$, quantifying pixel-wise visual similarities between pairs of images. Additionally, it guarantees that both flows emanate from a shared pixel location and form a cohesive trajectory in the temporal domain. TROF iteratively optimizes bidirectional flows by accessing multiscale correlation values associated with the current central flow. Specifically, in each iteration, based on the currently estimated bidirectional flows ${\mathit{f}}_{t\to t-1}^k$, ${\mathit{f}}_{t\to t+1}^k$, correlation values are retrieved from the dual correlation volumes and amalgamated into the central frame. The fused correlation features ${\mathit{F}}_{\mathit{corr}}^k$ and flow features ${\mathit{F}}_{\mathit{flow}}^k$ encode dual correlation values and displacements, respectively. These features are then further encoded to derive bidirectional motion features ${\mathit{F}}_m^k$, encapsulating abundant motion and correlation information crucial for predicting the remaining bidirectional flows, as elaborated further below.

$$\begin{array}{l}{\mathit{F}}_{\mathit{corr}}^k=\mathit{CorrEncoder}\left({\mathit{c}}_{t\to t-1}^k,{\mathit{c}}_{t\to t+1}^k\right),\\ {\mathit{F}}_{\mathit{flow}}^k=\mathit{FlowEncoder}\left({\mathit{f}}_{t\to t-1}^k,{\mathit{f}}_{t\to t+1}^k\right),\\ {\mathit{F}}_m^k={\mathit{MotionEncoder}}_{\mathit{TROF}}\left({\mathit{F}}_{\mathit{corr}}^k,{\mathit{F}}_{\mathit{flow}}^k\right)\end{array},$$

(7)

Updating of Bi-directional Recurrent Flow. During the bidirectional recurrent update process of TROF, a Super Kernel Block is utilized as the updater instead of Conv-GRU in RAFT. Nonetheless, it still maintains a hidden state or feature caching to refine the flow. Motion features, context features, and hidden state features are passed into the recurrent update module to update the hidden state. Context features are derived from the visual features of the central frame, and bidirectional residual flows are decoded from the hidden state.

$$\begin{array}{l}{\mathit{h}}^{k+1}=\mathit{Updater}\left({\mathit{F}}_m^k,\mathit{g},{\mathit{h}}^k\right),\\ \Delta {\mathit{f}}_{t\to t-1,t\to t+1}^k=\mathit{FlowHead}\left({\mathit{h}}^{k+1}\right)\end{array}$$

(8)

The accumulated residual flows refined at each iteration constitute the final predicted optical flow.

2.3.2. Motion Propagation for Multi-Frame

To extend the network architecture to handle more than three frames, a motion propagation module is introduced in the recurrent update decoder. This module facilitates the propagation of bidirectional motion features along the anticipated “flow trajectories”. Specifically, to effectively propagate information between adjacent TROFs, an additional motion state feature ${\mathit{M}}_t^k$ is maintained within each TROF unit. By integrating the warped adjacent TROFs’ motion state features ${\mathit{m}}_{\mathit{fwd}}^k$ and ${\mathit{m}}_{\mathit{bwd}}^k$ into the current TROF unit, the motion information gradually propagates throughout the entire sequence over multiple iterative cycles.

$$\begin{array}{l}{\mathit{F}}_{\mathit{mop}}^k=\mathit{Concat}\left({\mathit{M}}_t^k,{\mathit{m}}_{\mathit{fwd}}^k,{\mathit{m}}_{\mathit{bwd}}^k\right),\\ {\mathit{F}}_m^k,{\mathit{M}}_t^{k+1}={\mathit{MotionEncoder}}_{\mathit{MOP}}\left({\mathit{F}}_{\mathit{corr}}^k,{\mathit{F}}_{\mathit{flow}}^k,{\mathit{F}}_{\mathit{mop}}^k\right)\end{array}$$

(9)

In each iteration, temporal motion information is extracted from adjacent TROF units and concatenated with the current frame’s motion state feature ${\mathit{M}}_t^k$ to form the motion propagation feature ${\mathit{F}}_{\mathit{mop}}^k$. This feature enhances the coherence and flow attributes of the current frame by consolidating the motion features ${\mathit{F}}_m^k$ of the adjacent TROF units. Meanwhile, the motion feature state is also updated. Ultimately, through the MOP module, the hidden motion features of adjacent TROF units are gradually assimilated, allowing the temporal perception domain of the hidden motion features to expand with each iteration, thus enabling more effective handling and integration of additional TROF units, and extending the effective temporal span.

2.4. Video Acquisition and Data Processing

In the laboratory of Hohai University, we have established a multi-camera experimental platform aimed at simulating real river environments and capturing high-quality image data. The platform comprises a flume with dimensions of 4 m in length, 0.177 m in width, and 0.2 m in depth, in addition to six DS-2XA2646F-LZS cameras. These cameras are divided into three groups, each comprising two cameras. One camera in each group captures vertical images to obtain a top-down view of the riverway, while the other camera captures images at a 30-degree angle to acquire a side view of the river surface. The videos captured by each group of cameras have resolutions of 2 k, 1080 p, and 720 p. This layout is designed to acquire image data of multiple angles and resolutions to facilitate a comprehensive analysis of river hydrodynamics. The configuration of the experimental apparatus is depicted in Figure 5. Furthermore, to facilitate camera calibration and conversion of pixel coordinates to world coordinates in image processing, we affixed black tape along the edges of the flume as GCPs, with their layout depicted in Figure 6.

Considering the complexity of real river environments, characterized by varying parameters such as floating debris, water depth, and flow velocity, as well as the influence of different lighting conditions, we have employed multiple strategies to diversify the experimental scenarios. These strategies involve adjusting the density of tracer particles, the water level in the flume, the flow rate at the water interface per unit time, and the position and intensity of illumination sources. Such configurations are intended to provide targeted directions for further algorithm optimization, aiming to adapt to the diverse monitoring tasks of complex river environments in practical applications. The data collection scenarios are detailed in

Table 1. Datasets of various scenarios in the laboratory and their velocity measurement results.

Velocity-Range	Tracer	Light	Shadow	Resolution	RDV-Velocity	Mid-Velocity	Max-Velocity
0~1 m/s	√	1000 W lamp	√	2 k/1080 p/720 p	0.98 m/s	0.700 m/s	1.100 m/s
	√	1000 W lamp	X	2 k/1080 p/720 p	0.86 m/s	0.832 m/s	0.982 m/s
	√	Dark	X	2 k/1080 p/720 p	0.86 m/s	-	-
	X	1000 W lamp	√	2 k/1080 p/720 p	0.98 m/s	0.912 m/s	1.131 m/s
	X	1000 W lamp	X	2 k/1080 p/720 p	0.89 m/s	0.881 m/s	1.202 m/s
	X	Dark	X	2 k/1080 p/720 p	0.86 m/s	-	-
1~2 m/s	√	1000 W lamp	√	2 k/1080 p/720 p	1.30 m/s	1.222 m/s	2.959 m/s
	√	1000 W lamp	X	2 k/1080 p/720 p	1.30 m/s	1.139 m/s	1.529 m/s
	√	Dark	X	2 k/1080 p/720 p	1.505 m/s	-	-
	X	1000 W lamp	√	2 k/1080 p/720 p	1.30 m/s	1.241 m/s	1.677 m/s
	X	1000 W lamp	X	2 k/1080 p/720 p	1.854 m/s	1.782 m/s	2.013
	X	Dark	X	2 k/1080 p/720 p	1.505 m/s	-	-
2~3 m/s	√	1000 W lamp	√	2 k/1080 p/720 p	2.604 m/s	1.944 m/s	2.541 m/s
	√	1000 W lamp	X	2 k/1080 p/720 p	2.85 m/s	1.964 m/s	2.814 m/s
	√	Dark	X	2 k/1080 p/720 p	2.85 m/s	-	-
	X	1000 W lamp	√	2 k/1080 p/720 p	2.85 m/s	2.132 m/s	2.769 m/s
	X	1000 W lamp	X	2 k/1080 p/720 p	2.85 m/s	2.412 m/s	3.072 m/s
	X	Dark	X	2 k/1080 p/720 p	2.89 m/s	-
3~4 m/s	√	1000 W lamp	√	2 k/1080 p/720 p	3.61 m/s	1.262 m/s	1.784 m/s
	√	1000 W lamp	X	2 k/1080 p/720 p	3.61 m/s	1.597 m/s	2.205 m/s
	√	Dark	X	2 k/1080 p/720 p	3.61 m/s	-	-
	X	1000 W lamp	√	2 k/1080 p/720 p	3.61 m/s	2.555 m/s	3.490 m/s
	X	1000 W lamp	X	2 k/1080 p/720 p	3.61 m/s	2.791 m/s	3.491 m/s
	X	Dark	X	2 k/1080 p/720 p	3.61 m/s	-	-

, with each scenario recorded using six cameras at a frame rate of 25 frames per second for a duration of 120 s. Meanwhile, we conducted instantaneous velocity measurements using a hand-held radar Doppler velocimeter (RDV) at the center and sides of the flume for comparison with image-based velocity estimation methods.

3. Results

3.1. Model Training

In this study, we utilized an NVIDIA RTX 4090 GPU equipped with 24 GB of memory as the hardware platform for our experiments. The experimental environment was constructed based on Python 3.8, CUDA 12.5, and PyTorch 2.0.1. We based our work on previous FlowFormer studies [24], chose the first two stages of the ImageNet pre-trained Twins-SVT [25] model for image and context encoding, and performed parameter fine-tuning on them. Furthermore, following the SKFlow [26], we used SKBlocks instead of ConvGRU to enhance the model’s refinement capabilities.

To bolster the generalization ability of the data, we applied data augmentation techniques such as random cropping, scaling, rotation, and noise addition during the data preprocessing phase. Given the challenges of obtaining comprehensive annotated pixel displacement datasets in real river scenarios and the reliance of the VideoFlow model on dense ground truth, we opted for multiple open-source optical flow datasets to pre-train our model. Specifically, we first conducted pre-training for 125,000 iterations on the FlyingThings [27] dataset, followed by fine-tuning for an additional 25,000 iterations on a combined dataset comprising FlyingThings, Sintel [28], KITTI-2015 [29], and HD1K [30], which is abbreviated as ‘T+S+K+H’. More information about these datasets is provided in

Table 2. Details of the FlyingThings, Sintel, KITTI-2015, and HD1K optical flow datasets.

Dataset Name	Image Pairs	Scene Type	Details
FlyingThings	~39,000	synthetic	Scenes with various everyday objects flying along ran-dom 3D trajectories.
Sintel	1040	synthetic movie scenes	Includes natural scenes and motion challenges such as long-range motion, lighting changes, etc.
KITTI-2015	400	real-world driving scenes	Contains real annotations for dynamic scene flow, used for 3D scene flow estimation.
HD1K	1000	real-world urban driving scenes	Features high-resolution, high-frame-rate, and high-dynamic-range images.

. During the model training process, we employed the AdamW optimizer and implemented a one-cycle learning rate schedule. The batch size was uniformly set to 4 across all training phases. For the FlyingThings dataset, we set the highest learning rate to $2.5\times {10}^{-4}$, while for the other datasets, the highest learning rate was set to $1.25\times {10}^{-4}$. Moreover, the number of updater iterations was established at 16 to ensure that the model learned as much as possible during training.

To comprehensively evaluate the effectiveness of our proposed method, we conducted experiments and visual validation using a tank dataset collected in the laboratory, an open-source dataset of the Freiberger Mulde River [31], the Castor River [11], the Hurunui River [32], and the synthetic River datasets created by G. Bodart et al. [33].

3.2. The Simulated River in Laboratory and Analysis of Experimental Results

For a comprehensive evaluation of the method’s robustness in challenging and dynamic scenarios, we applied the RivVideoFlow optical flow estimation algorithm to datasets of 720 p water flow velocity videos collected. The detailed results of velocity estimation are shown in Table 1. Here, RDV-Velocity represents the average velocity collected by RDV in the center of the flume. Mid-Velocity denotes the average velocity estimated by the RivVideoFlow algorithm within the corresponding region of interest (ROI) with values [60:660, 570:620]. Max-Velocity represents the maximum value estimated within the water flow area. According to the results in Table 1, the RivVideoFlow model performs well in river scenes with velocities below 2 m/s, with errors within 15%, even in scenarios where there are no tracers and shadows. However, when the actual velocity exceeds 2.5 m/s, and there are no tracers present on the river surface, the accuracy of velocity estimation decreases significantly. For the experimental groups with tracers, the estimated results are relatively close to the actual values but exhibit instability and a more dispersed velocity distribution. This may be due to foam or splashes formed by internal turbulence at high velocities, causing waves to obscure or reflect camera light, thereby blurring video images and affecting ripple edge detection and contour extraction. When the river velocity exceeds 3.5 m/s, even the estimated pixel-level maximum velocity is lower than the actual velocity, indicating that the proposed model is no longer applicable. A significant observation is that because of the small size of the flume and the camera’s close proximity to the water surface, each pixel in the captured images represents a relatively small actual distance. In other words, the displacement resolution of the images, measured in pixels per meter (ppm), is significantly higher than that of images captured under real river conditions. Typically, the ppm in real river environments is approximately several tens, whereas our laboratory data achieves a ppm of up to 1220. Consequently, the RivVideoFlow method can measure flow velocities much higher than 3.5 m/s in our laboratory setting, indicating its potential to measure even higher velocities in actual river environments. Additionally, we compared our method with the LSPIV velocity estimation method. Figure 7b shows the rendering results of velocity estimation using the well-known Fudda-LSPIV 1.7.3 velocity measurement software for the data highlighted with gray shading in Table 1 [34].

3.3. Field River Measurements and Algorithm Validation

3.3.1. The Freiberger Mulde River in Field

To investigate the application of the RivVideoFlow method for surface velocimetry in natural environments, field measurement data from the Freiberger Mulde River were utilized [31]. These data were captured by a drone hovering over the area of interest at an altitude of 30 m. During data collection, the average water depth was approximately 0.87 m, the average river discharge was 5.60 m³/s, and the average flow velocity was 0.57 m/s. The first 25 s of the recorded video, with a resolution of 640 × 480 pixels and a frame rate of 30 Hz, were used for analysis. Figure 8a displays a sample frame from the recorded video. This serves as an appropriate example for verifying the proposed method’s applicability under low-flow conditions with very clear water. Reference velocity data were collected using an ADCP along the CS1 transect, as shown in Figure 8a.

Figure 8b presents the surface velocities measured and estimated at the cross-section CS1 using ADCP and four different optical flow-based surface velocimetry techniques. Figure 9 illustrates the average spatial surface velocity distribution and optical flow fields estimated by the VideoFlow architecture before and after orthorectification. Figure 8b demonstrates that the cross-sectional velocity distribution estimated by the RivVideoFlow method aligns more closely with the reference velocities than in the downstream section. This suggests that the optical flow architecture incorporating temporal cues achieves higher measurement accuracy.

Because of the “near-to-far” effect in the camera’s field of view, one pixel in the upstream section of the Freiberger Mulde River represents a much larger actual distance compared to that in the downstream section. Consequently, the velocity distribution values rendered in the upper part of Figure 9a are relatively low. After orthorectification, the image is transformed from a side view to an overhead view (Figure 9b), resulting in each pixel representing a more consistent actual distance. Therefore, the velocity distribution in Figure 9b appears to be more uniform. In other words, the results depicted in Figure 8 and Figure 9 demonstrate that the RivVideoFlow estimation method, which integrates temporal information and has undergone orthorectification, exhibits superior performance, and produces more accurate and smoother results.

3.3.2. Comparative Analysis of WSV Measurement Techniques

Data acquired from the Castor River on 4 April 2020 [11] underscores the efficacy of diverse image-based methods for assessing WSV. The video dataset, captured at 30 frames per second with a resolution of 2048 × 1536 from a camera positioned on the left bank, spans 29 s and is complemented by concurrent ADCP measurements for precise velocity assessment. On the survey day, the average values for river velocity, discharge, and depth were 1.19 m/s, 13.5 m³/s, and 3.6 m, respectively. The study conditions, marked by optimal diffuse lighting and discernible surface texture across the river’s breadth, presented an ideal scenario for image-based velocity analysis. Spatial calibration was facilitated by documenting twelve ground control points.

The comparative analysis, as depicted in Figure 10, juxtaposes the performance of various image-based velocity measurement techniques with the ADCP data. The traditional Frameback optical flow [35] method was found to be the least accurate, significantly underestimating the river’s surface velocity. Conversely, deep learning-based approaches generally yielded more uniform velocity distributions within the camera’s field of view, outperforming PIV [3] and STIV [36] methods. Notably, the RivVideoFlow technique provided velocity estimates closely aligning with ADCP measurements, highlighting its potential for accurate flow velocity assessment in riverine environments.

3.3.3. Analysis of RivVideoFlow Algorithm Applicable Velocity Range

This study utilizes data from the Hurunui River [32] to further assess the RivVideoFlow algorithm’s performance under high-flow conditions. The video dataset of the Hurunui River was captured using a DJI M210 drone and recorded at a frame rate of 24 Hz with a resolution of 3840 × 2160 pixels for a duration of 59 s. Concurrently, actual velocity measurements were obtained through ADCP sampling. Characterized by low water clarity, high flow velocities, a flat riverbed, and a uniformly dense distribution of ripples, the Hurunui River provides a challenging environment for testing the RivVideoFlow algorithm. Analysis, as depicted in Figure 11, indicates that RivVideoFlow performs effectively in measuring high velocities. Moreover, the cross-sectional velocity measurement results presented in Figure 8, Figure 9, Figure 10 and Figure 11 demonstrate that while RivVideoFlow’s performance is somewhat diminished under low-flow conditions in shallow waters, it significantly enhances its performance in high-velocity assessments. Overall, the algorithm shows a relatively strong performance across rivers with varying flow velocities.

3.4. Visualization and Analysis of Experiments on the Synthetic River Datasets

To visually validate the effectiveness and generalization capability of the model, we selected virtual river flow videos from various scenarios, including ideal conditions, sunny weather, cloudy weather, deep forest, and high-altitude drone footage for color-coded visualization of the flow field. Figure 12 illustrates the RSV estimation results for the two scenarios. Scenario one (first row of Figure 12) depicts a sunny day with the river flowing towards the lower left direction, while scenario two (second row of Figure 12) represents a scene captured on the bridge with the river flowing to the right. In scenario one, noticeable cavities are observed in the flow field visualization produced by the RAFT model Figure 12b in the lower left region, mainly attributed to the interference of light and shadows on the river surface in that area, making it challenging for the optical flow-based model to accurately track particle motion. However, in Figure 12c, produced by the RivVideoFlow, no cavities are present, indicating that multi-frame optical flow estimation leveraging temporal cues exhibits better robustness against light and shadow interference.

4. Discussion

This study introduces RivVideoFlow, an innovative video-based technique for quantifying WSV in rivers. This technique employs a homography matrix, denoted as $\mathbf{H}$, to orthorectify images, transforming perspectives from oblique angles to a plan-view orientation. This process aligns the visual coordinates of the images with their geographical counterparts in the real world. RivVideoFlow demonstrates high precision and robustness across various datasets, surpassing traditional two-frame RAFT models by effectively integrating temporal information. This integration enhances the method’s capacity to track the continuity of water flow, thereby improving the temporal reliability of WSV measurements.

Despite its reliable estimation of WSV across diverse environmental conditions and its validation against ADCP/RDV measurements, RivVideoFlow has inherent limitations. The performance of RivVideoFlow is reliant on the diversity and representativeness of the training data. The model, which is primarily evaluated under specific conditions, may not adapt seamlessly to the full spectrum of riverine environments. Additionally, while multi-frame optical flow estimation enhances spatiotemporal stability, the algorithm’s precision could be compromised during extreme flow events, such as flash floods or rapids. Furthermore, the presence of substantial river surface debris can degrade the image quality, impacting the accuracy of optical flow estimation. Lastly, RivVideoFlow primarily assesses surface velocities, which may not accurately reflect the average velocity profile across the river, particularly in deeper rivers where surface and sub-surface velocities can significantly differ.

Future research will address three primary objectives. First, we will develop a dataset with dense ground truth annotations for synthetic river videos to enhance model generalizability beyond the current constraints of limited training data diversity. Second, we will refine the optical flow estimation by incorporating physiologically informed constraints within the deep learning model’s loss function, aimed at more accurately capturing fluid dynamics on river surfaces. Lastly, we will investigate methodologies to translate surface velocity measurements into estimates of the average velocity across the river cross-section, which is crucial for a holistic understanding of river flow dynamics.

5. Conclusions

This study successfully developed RivVideoFlow, an innovative video-based method for measuring river surface velocities and monitoring river flow rates. The method integrates lens distortion correction, implicit camera calibration based on Ground Control Points (GCPs), two-dimensional and three-dimensional orthorectification techniques, and a deep learning-based multi-frame optical flow estimation algorithm. The algorithm was tested in various environments, including flood simulation laboratory platforms, outdoor settings, and synthetic river videos. Results indicate that RivVideoFlow can reliably estimate WSV across different flow rates and camera perspectives. Unlike widely used image-based velocity measurement methods such as LSPIV and STIV, RivVideoFlow does not require the deployment of tracers or the adjustment of specific scene parameters. It is a true end-to-end velocity measurement system that operates without manual intervention. We anticipate that RivVideoFlow will facilitate a broader user base, including non-experts, to engage in large-scale image-based velocity measurement applications.