A Robust Video Watermarking Algorithm Based on Two-Dimensional Discrete Fourier Transform

Abstract

Due to the continuous development and popularity of digital video technology, the copyright protection of digital video content has become an increasingly prominent issue. Digital video watermarking, as an effective means of digital copyright protection, has attracted widespread attention from academia and industry. The two-dimensional discrete Fourier transform (2D-DFT) template-based method has the advantages of good real-time performance and robustness, but the embedding capacity is small and cannot resist frame-dropping attack. To address this problem, a new template construction method is proposed in this paper, which can extend the embedding capacity from 1 bit each group of pictures (GOP) to theoretically n bits each GOP. In addition, by changing the GOP pattern, the method gains the ability to resist frame-dropping attacks. Experimental results show that the proposed method can achieve larger watermark capacity while maintaining strong robustness against image-processing attacks, geometric attacks, video-processing attacks and compression attacks.

1. Introduction

The popularity and development of digital video technology brings convenience but also causes a series of problems, such as copyright infringement and fake information. Video data hiding, including video stenography [1,2] and video watermarking [3,4] has received extensive usage and continuous study. Here, we focus on video watermarking algorithms, which can hide copyright information in videos without being perceived and are more suitable for copyright protection. In addition, by embedding watermarks such as brand logos and trademarks in videos, video watermarking technology can increase brand value and revenue, further expanding the commercial value of the video industry.

Since a robust video watermarking algorithm which can extract the watermark correctly even after intentional and unintentional attacks is a key requirement of copyright protection, we focus on robust video watermarking in this paper. According to embedding and extraction methods, the video watermarking algorithms can be classified into compression domain-based video watermarking and content-based video watermarking [5]. In compression domain-based watermarking techniques, the watermark is usually embedded during the encoding process, such as MPEG-2, MPEG-4, H.264/AVC, or H.265/HEVC. There are a number of MPEG-based video watermarking algorithms [6,7,8]. In 2017, Li et al. [9] proposed a MPEG-2 video watermarking technology based on the DC coefficient. The watermarked video only needs to be partially decoded, which can greatly improve the efficiency of the algorithm. Furthermore, in 2017, Su et al. [10] proposed an adaptive approach of MPEG-4 video based on the Watson perceptual model. For HEVC and AVC, the other two popular encoders, there are also some video watermarking algorithms [11,12,13,14]. In 2021, Sun et al. [15] proposed a scalable watermarking algorithm using the discrete cosine transform (DCT) coefficients for watermark embedding in the H.264 compressed domain. Similarly, Cai et al. [16] employed the DCT coefficients and prediction unit partition technique to embed watermarks in HEVC video.

Compression domain-based watermarking algorithms have good real-time performance but are less robust and are also vulnerable to different video codecs. Therefore, many scholars focus on content-based watermarking algorithms, which usually have better robustness and can be applied to various video codecs. In 2017, Bayoudh et al. [17] proposed a new method based on Krawtchouk moments and DCT. The method was robust to various attacks. However, different transform domains have different limitations, so researchers turned to using the combination of multiple transform domains to improve the robustness of watermarking algorithms. In 2020, Liu et al. [18] proposed a video watermarking scheme based on discrete wavelet transform (DWT) and singular value decomposition (SVD). The scheme can greatly improve the imperceptibility. By 2022, Wang et al. [19] transformed the DWT domain into the dual tree-complex wavelet transform (DTCWT) domain, performed SVD of DTCWT video frames, and implemented watermark embedding by adjusting the shape of candidate coefficients. Finally, the algorithm was made robust against temporal asynchrony attacks. Fan et al. [20] proposed a video watermarking algorithm by combining non-subsampled contourlet transform, 3D-DCT, and non-negative matrix factorization (NMF) to embed the encrypted 2D copyright watermark into the base matrix of NMF. However, although these algorithms achieve good robustness, most of them require geometric synchronization when attacked and are more time-consuming due to the embedding in the transform domain.

Sun et al. [21] proposed a method to deal with the synchronization and the real-time issue. Two spatial masks representing secret bits 0 and 1 were generated, and by using the spatial masks, the watermark embedding process could converted from the frequency domain to the spatial domain and thus greatly reduce the watermark embedding time. However, the disadvantage of this algorithm is that only one bit can be embedded in one GOP, which limits its embedding capacity. Furthermore, it is not resistant to frame-dropping attack. Aiming at these issues, we aimed to enlarge the embedding capacity and strengthen the robustness against frame-dropping attack. First, the prime factorization method was introduced to design the corresponding watermark template sequence for the n bits watermark, and the embedding capacity was extended from 1 bit to theoretically n bits per GOP. Furthermore, we modified the GOP segmentation method, which can indirectly enlarge the embedding capacity on one hand and can resist frame-dropping attack by sending the number of GOPs as a key to the verifier, thus reducing the impact of unsynchronized watermark information.

The main contributions of this paper are as follows: (1) We increased the watermark embedding capacity from 1 bit to theoretically n bits for each GOP by using a modified template sequence. (2) We changed the GOP grouping pattern and resized the GOP to an adaptive method, so that the video can remain robust in the case of frame-dropping attack. The rest of this paper is organized as follows: Section 2 briefly introduces the related concepts, including DFT properties, and the proposed watermark template sequences. Section 3 describes the embedding and extraction steps of the proposed method in detail. Section 4 gives the experimental results as well as the comparative analysis. Section 5 concludes this paper and discusses the future research directions.

2. Preliminaries

2.1. Property of 2D-DFT

The 2D-DFT method converts a discrete time-domain signal into a discrete frequency-domain signal and is widely used in image processing. The 2D-DFT approach is often adopted in image processing, and the definition is shown in Equation (1), where $f\left(x,y\right)$ is the pixel value at the position of xth row and yth column, $F\left(u,v\right)$ is the value at the $\left(u,v\right)$ position in the transform domain, the range of u and v is the same as the range of values of x and y, and M and N represent the length and width of the image, respectively.

$$F\left(u,v\right)=\sum _{x=0}^{M-1}\sum _{y=0}^{N-1}f\left(x,y\right)e^{-\frac{j2\pi xu}{M}-\frac{j2\pi yv}{N}}$$

(1)

$$f\left(x,y\right)=\frac{1}{MN}\sum _{u=0}^{M-1}\sum _{v=0}^{N-1}F(u,v)e^{\frac{j2\pi ux}{M}+\frac{j2\pi vy}{N}}$$

(2)

The definition of the inverse discrete fourier transform (IDFT) is shown in Equation (2). The linear property is one important property of 2D-DFT, which is shown in Equation (3), where $G\left(u,v\right)$ is the corresponding 2D-DFT of $g\left(x,y\right)$.

$$af\left(x,y\right)+bg\left(x,y\right)\leftrightarrow aF\left(u,v\right)+bG(u,v)$$

(3)

From Equation (3), we can see that the change in DFT coefficients can be implemented by altering the corresponding pixel value in spatial domain. That is to say, watermark embedding in DFT domain is equivalent to increasing or decreasing the pixel value of the cover image, which is more time-saving.

The DFT rotation feature is also very important. As shown in Equation (4), this means that if an image in the spatial domain is rotated by $\theta$ degrees, the corresponding content in the frequency domain is also rotated by $\theta$ degrees. The scaling property of DFT, as shown in Equation (5), shows that shrinking or stretching the image in the spatial domain is equivalent to expanding or shrinking the image in the frequency domain by the corresponding ratio.

$$f(x\ast cos\theta -y\ast sin\theta ,x\ast sin\theta +y\ast cos\theta )⟷F(u\ast cos\theta -v\ast sin\theta ,u\ast sin\theta +v\ast cos\theta )$$

(4)

$$f(ax,by)⟷\frac{1}{ab}F(\frac{u}{a},\frac{v}{b})$$

(5)

2.2. Watermark Template Construction

According to Section 2.1, the watermark embedding process can be converted from the 2D-DFT frequency domain to the spatial domain and improve the real-time performance. Consequently, we can first design two binary sequences in the 2D-DFT domain, called the zero-template and one-template, to represent watermark bits 0 and 1, respectively. Then, we can calculate the spatial mask corresponding to the watermark template by using Equation (3). Finally, watermark embedding can be conducted by adding the spatial mask to the pixel value of the cover video frame. However, two templates mean only one bit can be embedded in one video frame, and the embedding capacity is limited. Thus, in this section, we will first introduce the watermark template construction principle and then illustrate our design of n template sequence to enlarge the embedding capacity.

2.2.1. Watermark Template and Corresponding Spatial Mask

In the process of designing the template, there are three issues to consider. The first one is the shape of the watermark template. Choosing a circular template can resist the rotation attack. According to the rotation property of 2D-DFT, the circular template keeps the same position after rotation, which makes watermark extraction faster and more accurate. As shown in Figure 1c, the zero-template, which denotes watermark bit zero, is presented. The corresponding binary sequence is 10101010101010101010, where 1 forms a peak and 0 becomes a gap. To achieve this goal, a constant C can be added to the DFT coefficient at the position corresponding to the bit “1” in the sequence. In order to ensure that the video frames are still guaranteed to be real after IDFT transformation, we should ensure that the watermark templates are centrally symmetric. Therefore, we embedded the same template sequence 2 times in the embedding process.

The second issue is the location of the watermark template. Low-frequency DFT coefficients occupy the majority energy of the video frame, and modifying them would greatly reduce the visual quality. On the contrary, high-frequency DFT coefficients hold the minority energy of the image, but they are easily removed under various video-processing operations. Therefore, it is most appropriate to embed the watermark template in the mid-frequency part. Since small radius size denotes low-frequency DFT coefficients, we set the embedding radius size to 1.2 × L/2, where L is the width of the video frame. The last issue is spatial mask generation process. The specific construction process is shown as follows.

Step 1: A grayscale image with size L × L is constructed, and then all its pixel values are set to 128. This ensures that all the DFT magnitude is 0 except the center of DFT, as shown in Figure 1a,b.

Step 2: The watermark template is added to the 2D-DFT of the grayscale image. Then, the IDFT transformation was performed to form a grayscale image with watermark information, as shown in Figure 1c–e.

Step 3: The Spatial mask corresponding to the watermark template is generated by subtracting Figure 1a from Figure 1e. From the linear property of 2D-DFT, when embedding watermark information, we only need to add the corresponding spatial mask to the video frame.

2.2.2. Template Sequence

According to [21], in order to ensure good robustness in the watermark extraction process, two bits sequences, as shown in Equations (6) and (7), were selected as templates for watermark bit 0 and watermark bit 1 for watermark embedding, where S${}_0$ represents the bit sequence for watermark bit 0 and S${}_1$ represents the bit sequence for watermark bit 1. The advantage is that the normalized correlation (NC) value of the two templates remains at 0.5 regardless of the rotation strength they undergo.

$$S_0\left(i\right)=\left\{\begin{array}{c}0whenimod2=0\\ 1whenimod2=1\end{array}\right.fori=0,1,\cdots ,l-1$$

(6)

$$S_1\left(i\right)=\left\{\begin{array}{c}0whenimod4=0orimod4=1\\ 1whenimod4=2orimod4=3\end{array}fori=0,1,\cdots ,l-1\right.$$

(7)

However, two watermark templates mean only 1 bit can be embedded in one GOP, and this leads to a low embedding capacity. To solve the problem, a new approach is proposed to extend the embedding capacity from 1 bit to a theoretical n bits by designing a number of watermark template sequences.

When constructing the watermark template, the divisor, such as 2 and 4 in Equations (6) and (7), is a key parameter. First of all, the divisors in different template sequence should be distinct with each other; thus, the watermark template can be different and can be easy to distinguish in the watermark extraction process. Second, the divisor affects the bit error ratio (BER) between any two watermark template sequences, especially with the consideration of rotation attack. Here, the BER is used to calculate the similarity between the template sequences in order to facilitate the following analysis. Therefore, to ensure n bits can be embedded in one video frame, 2${}^n$ watermark template sequences and 2${}^n$ different and robust divisors should be built. To solve the problem, the operation of factorization is brought in to calculate the template sequences, which can ensure the accuracy of the algorithm.

Equation (8) shows the prime factorization of a positive integer cn,

$$cn={pp}_1^{a_1}\times {pp}_2^{a_2}\times \cdots \times {pp}_n^{a_n}$$

(8)

where ${pp}_1,{pp}_2,\cdots ,{pp}_n$ are prime numbers that are not equal to 1 and not the same as each other, and $a_1,a_2,\cdots ,a_n$ are the prime index of ${pp}_1,{pp}_2,\cdots ,{pp}_n$, respectively, and all are greater than or equal to 1. The factor of cn can be used as divisors to construct watermark template sequences, and 2${}^n$ different divisors are required. The factor number of cn can be obtained according to Equation (9).

$$\tau \left(cn\right)=(a_1+1)\times (a_2+1)\times \cdots \times (a_n+1)$$

(9)

To ensure that the number of templates embedded in the cover video is exactly 2${}^n$, i.e., the factor number of cn is 2${}^n$, we used the inverse method of integer factorization to calculate the number of factors. The inverse operation of factorization is defined in Equations (10) and (11).

$$m_1=\frac{n}{n1},m_2=mod\left(n,n1\right),1\le n1\le n$$

(10)

$$cn1={pp}_1^{2^{(m_1+1)}-1}\times {pp}_2^{2^{(m_1+1)}-1}\times \cdots \times {pp}_{m_2}^{2^{\left(m_1+1\right)}-1}\times {pp}_{m_2+1}^{2^{m_1}-1}\cdots \times {pp}_{n1}^{2^{m_1}-1}$$

(11)

cn1 must have 2${}^n$ mutually different factors, where each factor is the combination ${pp}_1,{pp}_2,\cdots ,{pp}_{n1}$. After we obtain the divisors, we can construct the watermark template sequence by using Equation (12), where q represents the divisor, $\lceil ·\rceil$ denotes ceiling function, and S${}_k$ represents the kth watermark template sequence.

$$S_k\left(i\right)=\left\{\begin{array}{c}1whenimodq<⌈\frac{q}{2}⌉\\ 0whenimodq\ge ⌈\frac{q}{2}⌉\end{array}\right.$$

(12)

Next, we elaborate the BER of any two watermark template sequences. Suppose q${}_1$ and q${}_2$ are two different divisors for two watermark templates; there are two kinds of relation between them. One is that q${}_1$ or q${}_2$ is a multiple of the other. The other one is q${}_1$ or q${}_2$ cannot be evenly divided by the other. Figure 2 shows the case of the first relation. In Figure 2a, q${}_1$, q${}_2$ are equal to 6, 12, respectively, so q${}_2$ is an even multiple of q${}_1$. The first row is the watermark template sequence S${}_1$(i) constructed by using divisor q${}_1$ = 6, and the second row is the watermark template sequence S${}_2$(i) constructed by using divisor q${}_2$ = 12. In Figure 2b, q${}_1$, q${}_2$ are equal to 6, 18, respectively, so q${}_2$ is an odd multiple of q${}_1$.

From Figure 2a, the watermark template sequence with divisor q${}_2$ always consists of two parts: the all-one sequence and all-zero sequence. In the all-one sequence, the different bit number between S${}_1$(i) and S${}_2$(i) is $⌈\frac{q_1}{2}⌉$, and the different bit number between S${}_1$(i) and S${}_2$(i) is $⌊\frac{q_1}{2}⌋$ in the all-zero sequence, where $⌊·⌋$ denotes floor function. Therefore the BER between S${}_1$(i), and S${}_2$(i) can be described as Equation (13), which can represent the BER for the case of q${}_1$, or q${}_2$ is an even multiple of the other. By adopting the same analysis, we can obtain the BER for the case of q${}_1$, or q${}_2$ is an odd multiple of the other. The example is shown in Figure 2b, and BER is described in Equation (14).

$$BER=\frac{⌈\frac{q_1}{2}⌉\times \frac{\frac{q_2}{q_1}}{2}+⌊\frac{q_1}{2}⌋\times \frac{\frac{q_2}{q_1}}{2}}{q_2}=\frac{q_1\times \frac{\frac{q_2}{q_1}}{2}}{q_2}=\frac{q_1\times ⌊\frac{\frac{q_2}{q_1}}{2}⌋}{q_2}=\frac{1}{2}$$

(13)

$$BER=\frac{⌈\frac{q_1}{2}⌉\times \frac{\left(\frac{q_2}{q_1}-1\right)}{2}+⌊\frac{q_1}{2}⌋\times \frac{\left(\frac{q_2}{q_1}-1\right)}{2}}{q_2}=\frac{q_1\times \frac{\left(\frac{q_2}{q_1}-1\right)}{2}}{q_2}=\frac{q_1\times \frac{\left(\frac{q_2}{q_1}-1\right)}{2}}{q_2}=\frac{q_1\times ⌊\frac{\frac{q_2}{q_1}}{2}⌋}{q_2}$$

(14)

Then, we should consider the BER of S${}_1$(i) and S${}_2$(i) when they are rotated since it is better if the watermark template can resist rotation attack. From Equation (12), we can see that the structure of S${}_k$(i) is the all-one sequence first and then the all-zero sequence. As illustrated in Figure 3a, the first row is the watermark sequence S${}_1$(i) with divisor q${}_1$ = 4, and the second row is the watermark sequence S${}_2$(i) with divisor q${}_2$ = 20. When no rotation appears, S${}_1$(i) and S${}_2$(i) are most aligned, which means the BER is lowest at this time. When S${}_2$(i) is rotated to a sequence which is the opposite of the original one, as shown in Figure 3b, the BER value is the largest. Therefore, we can obtain the BER range, which is displayed in Equation (15).

$$\frac{q_1\times ⌊\frac{\frac{q_2}{q_1}}{2}⌋}{q_2}\le BER\le 1-\frac{q_1\times ⌊\frac{\frac{q_2}{q_1}}{2}⌋}{q_2}$$

(15)

From the above analysis, we can see in the case that q${}_1$ or q${}_2$ is an even multiple of the other, the BER value always remains at 0.5. In the case that q${}_1$ or q${}_2$ is an odd multiple of the other, the BER value gradually approaches to 0.5 as the multiple increases. Especially, when q${}_2$ is three times q${}_1$, the BER reaches the minimum value, which is 0.33.

Equation (15) satisfies the case that q${}_1$ or q${}_2$ is a multiple of the other. For the other case that q${}_1$ or q${}_2$ cannot be evenly divided by the other, the BER between S${}_1$(i) and S${}_2$(i) also satisfies Equation (15) after extensive testing.

Next, we will take n = 3 as an example. Three bits means 2${}^3$ templates are needed for embedding. We chose n1 = 2, ${pp}_1,{pp}_2$ equal to 2 and 3, respectively. By using Equations (10) and (11), we can obtain cn1 = 24, which consists of 8 factors, including factor 1. Eight factors are used as the divisors according to Equation (12), and the watermark template sequences can be obtained as Equation (15). Please note that, for divisor 1, i.e., the eighth watermark template S${}_7$(i), it is impossible to construct it as Equation (12), so we found the sequence whose BER with S${}_1$(i), ⋯, S${}_7$(i) is nearest to 0.5 with the help of programming.

$$\begin{array}{c}\hfill S_0\left(i\right)=\left\{\begin{array}{c}1whenimod2<1\\ 0whenimod2\ge 1\end{array}fori=0,1,\dots ,23\right.\\ \hfill S_1\left(i\right)=\left\{\begin{array}{c}1whenimod3<1\\ 0whenimod3\ge 1\end{array}\right.fori=0,1,\dots ,23\\ \hfill S_2\left(i\right)=\left\{\begin{array}{c}1whenimod4<2\\ 0whenimod4\ge 2\end{array}fori=0,1,\dots ,23\right.\\ \hfill S_3\left(i\right)=\left\{\begin{array}{c}1whenimod6<3\\ 0whenimod6\ge 3\end{array}fori=0,1,\cdots ,23\right.\\ \hfill S_4\left(i\right)=\left\{\begin{array}{c}1whenimod8<4\\ 0whenimod8\ge 4\end{array}fori=0,1,\dots ,23\right.\\ \hfill S_5\left(i\right)=\left\{\begin{array}{c}1whenimod12<6\\ 0whenimod12\ge 6\end{array}fori=0,1,\dots ,23\right.\\ \hfill S_6\left(i\right)=\left\{\begin{array}{c}1whenimod24<12\\ 0whenimod24\ge 12\end{array}fori=0,1,\dots ,23\right.\\ \hfill S_7\left(i\right)=\left\{\begin{array}{c}1whenimod2<1\\ 0whenimod2\ge 1\\ \begin{array}{c}0whenimod2<1\\ 1whenimod2\ge 1\end{array}\end{array}\right.\begin{array}{c}fori=0,1,\dots ,11\\ \\ fori=12,13,\dots ,23\end{array}\end{array}$$

(16)

According to Equation (15), we calculated the BER between each two sequences, and the results are shown in

Table 1. BER values between two of the eight sequences.

BER	${\mathit{S}}_1$	${\mathit{S}}_2$	${\mathit{S}}_3$	${\mathit{S}}_4$	${\mathit{S}}_5$	${\mathit{S}}_6$	${\mathit{S}}_7$
$S_0$	0.5	0.5	0.33–0.67	0.5	0.5	0.5	0.5
$S_1$	∖	0.5	0.5	0.5	0.5	0.33–0.67	0.5
$S_2$	∖	∖	0.5	0.5	0.5	0.5	0.5
$S_3$	∖	∖	∖	0.5	0.5	0.5	0.5
$S_4$	∖	∖	∖	∖	0.5	0.33–0.67	0.42–0.58
$S_5$	∖	∖	∖	∖	∖	0.5	0.5
$S_6$	∖	∖	∖	∖	∖	∖	0.42–0.58

. It can be seen that the BER between any two template sequences is always higher than 0.33 and as high as 0.67 in some cases, most of them remaining at 0.5.

3. Proposed Watermarking System

In this section, we describe the proposed watermarking algorithm, including watermark embedding and watermark extraction. In the watermark embedding process, the video frames are first divided into multiple GOPs, then the watermark template is embedded in each GOP, and finally perceptual model is utilized to increase the fidelity of the video. In the watermark extraction process, the GOPs are first obtained according to GOP number, then the watermarks in each GOP are extracted by averaging the video frames, and finally the watermark information is obtained by calculating the correlation between the extracted sequence and the template sequence.

3.1. Watermark Template Construction

In this section, the process of watermark embedding is presented, as illustrated in Figure 4.

In addition to watermark template embedding, there are two other key steps in the embedding process. One is the length of GOP, which is related to the embedding capacity and the robustness of the proposed algorithm. The second is the introduction of the perceptual model, which can improve the invisibility of the algorithm.

Frame attacks including frame averaging, frame swapping, and frame dropping cause synchronization issues, which lead to imprecision of watermark extraction. To solve the problem, it is better to repeatedly embed watermark information. Therefore, the video frames are divided into GOPs, and one watermark bit is repeatedly embedded in each video frame of one GOP. First, we need to calculate the number of frames contained in every GOP. The selection of the GOP size needs to be calculated by the length of watermark to be embedded. The calculation formula is shown in Equation (17), where N${}_c$ is the frame number of the cover video and N${}_w$ is the bit number of the watermark information.

$$G_s=\frac{N_c}{N_w}$$

(17)

The GOP structure is shown in Figure 5. In the extreme case, each GOP contains only one frame. This maximizes the watermark capacity, but the watermark is sensitive to frame attacks. We will discuss the selection of GOP size in Section 4.2.

Inter-frame encoding is an inevitable step in video coding, which uses a reference frame to predict the current frame. However, inter-frame encoding can disturb the embedding process of watermarks. Assuming that frame F and its reference frame belong to different GOPs, they have different embedded watermark information. When encoding frame F, its reference frame, carrying its own watermark information, is used to predict it; thus, the watermark in the reference frame affects the watermark information in frame F and causes watermark extraction imprecision. To avoid this phenomenon, the last 1/3 part of each GOP is not selected to embed the watermark in the proposed method.

In order to improve the video watermark transparency, a perceptual model was introduced. In the human visual system, low-contrast areas have a great impact on image readability and visual performance because the human eye is sensitive to low-frequency noise. Therefore, in order to ensure the video fidelity, the embedding intensity in the high-contrast region can be appropriately increased, while it can be reduced in the low-contrast region. The specific operation process of information embedding can be referred to in reference [21], and the main principle is to adjust the intensity of information embedding by calculating the local contrast.

The detailed watermark embedding steps are as follows:

Step 1: Watermark template generation. Watermark templates are generated according to Section 2.2.2, and the spatial mask in spatial domain for each watermark template is saved to reduce the time cost of watermark embedding.

Step 2: GOP division of the cover video. Video frames are divided into a series of GOPs according to the bit number of the watermark. Each GOP is divided into two parts; the first 2/3 part is used for watermark embedding, the last 1/3 part remains unchanged and keeps the number of GOPs. Please note that the GOP number needs to be transmitted to the watermark extractor as a key.

Step 3: Embedding intensity adjusting. The spatial mask is adjusted according to the perceptual model. Then, the corresponding spatial mask is added to the Y-component of each video frame.

Step 4: Video encoding. The watermark-embedded video is compressed by using the video codecs.

3.2. Watermark Template Construction

In this section, the process of watermark extraction is presented and is illustrated in Figure 6.

The extraction part is composed of 5 steps.

The first step is GOP segmentation. The video frames are divided according to the GOP number, and then the GOP segmentation is performed in the same way as the embedding part.

The second step is to find the embedded sequence with DFT transform. The average frame of each 2/3 GOP is obtained and subjected to DFT. The time cost is higher if the DFT is performed on each frame. To speed up this process, according to the linear property of DFT, it is performed on the average frame of each GOP.

In the third step, some preprocessing before watermark extraction needs to be carried out. First, the DFT values on the horizontal and vertical lines in the middle of the magnitude image need to be replaced by zero. This is because DFT is implicitly periodic in both the spatial and frequency domains, which can lead to strong horizontal and vertical discontinuities on the cycle boundaries. The intensity of these pixels is often much larger than the watermark amplitude and therefore needs to be suppressed. Second, to eliminate the effect of large energy in the low-frequency part of the DFT domain, a circular region of radius R centered at the origin needs to be set up, and all DFT values in the region of R should be replaced with zero. For most natural images, it is recommended that R be set to 0.35 times the length of the smaller side of the image height or width. For enlarging the attack, a smaller R should be used to ensure that the watermark in the enlarged video is not removed. Then, we can extract the watermark sequence in the DFT domain.

In the fourth step, we need to determine which template the extracted watermark sequence belongs to. We calculated the correlations between the extracted watermark sequence and the watermark template sequence by using Equation (18),

$${corr}_{XY}\left(n\right)=\sum _{m=0}^{M-1}X\left(m\right)Y(m+n)$$

(18)

where $X\left(m\right)$ represents the extracted sequence in the DFT domain, $Y\left(m+n\right)$ represents the template sequence, and M represents the length of $X\left(m\right)$. In order to extract the watermark accurately, we need to calculate correlations for different radius r and different rotation angles. The watermark is extracted M points at a time with an angle of 360/M between each two points and a radius of r. When embedding the watermark, M points are selected as templates, so that the extracted sequences are separated by 360/M degrees. In order to ensure the correct extraction, the extraction process is carried out every 0.5 degrees. Because the embedding angle between two points is 360/M degrees, we only need to rotate 180/M times to extract the watermark correctly. Finally, the largest correlation value means the extracted sequence belongs to the corresponding watermark template.

4. Experimental Results

4.1. Experimental Setup

The proposed algorithm was employed on an AMD Ryzen 9 5900HX processor with 3.30 GHz and 32 GB RAM, and the tool used for the experiments is MATLAB 2021b on the Windows 11 operating system. To evaluate the effectiveness of the proposed algorithm, experiments were tested on four CIF videos and four 1080p videos, with the majority of them downloaded from https://media.xiph.org/video/derf/ (accessed on 13 July 2023). The parameters of these videos are shown in

Table 2. Parameters of tested videos.

Video Name	Size	Frame Number
Hall	352 × 288	300
News	352 × 288	300
City	352 × 288	300
Akiyo	352 × 288	300
Station2	1920 × 1080	313
Pedestrian-area	1920 × 1080	375
BasketballDrive	1920 × 1080	501
BQterrace	1920 × 1080	601

, snapshots of these videos are shown in Figure 7, and all videos are in YUV 4:2:0 format. The watermark is the binary sequence of “000 001 010 011 100 101 110 111 000 001”. Eight watermark templates constructed by Equation (16) were adopted, which means 3 bits were embedded in each GOP. In this paper, the performance of the proposed algorithm was tested in four aspects: imperceptibility, capacity, robustness, and time efficiency.

4.2. Selection of GOP Size

GOP size affects the robustness of the algorithm against video frame attacks, especially frame-swapping attack and frame-dropping attack, so it is important to test the effect of GOP size on the efficiency of the algorithm and select the most suitable size. In this part, GOP sizes from 3 to 30, stepped by three were tested on four different CIF videos. We tested four videos by calculating their average BER under three conditions: no attack, 30% frame-swapping attack, and 30% frame-deletion attack. The results are presented in

Table 3. Average BER of different GOP sizes under frame-swapping and frame-dropping attacks.

GOP Size	No Attack	Frame Swapping (30%)	Frame Dropping (30%)
3	0.4667	0.43	0.44
6	0	0.08	0.40
9	0	0.06	0.22
12	0	0.05	0.20
15	0	0.02	0.15
18	0	0	0.06
21	0	0	0.03
24	0	0	0.03
27	0	0	0.02
30	0	0	0

.

According to the results in Table 3, it can be seen that under the frame-swapping and frame-dropping attacks, the BER of the watermark decreases to 0 as the GOP size increases. When GOP size is greater than 15, it can resist the 30% frame-swapping attack well, while when greater than 30, the algorithm exhibits excellent robustness against the 30% frame-dropping attack. Therefore, we suggest to select a GOP size larger than 30 frames as much as possible. Please note that, in our algorithm, the GOP size is decided by the length of the watermark bit, but we suggest that it is better to set it to larger than 30.

4.3. Imperceptibility and Capacity

Imperceptibility means that the watermark should be transparent, i.e., it cannot be easily perceived by human eyes. The evaluation standard of imperceptibility includes subjective visual quality and objective visual quality.

The subjective visual quality is to determine whether the watermark is visible or not by human eye observation. Figure 8 shows snapshots of the same frame of two 1080p videos before and after watermark embedding; we can see that there are no obvious visual changes before and after watermark embedding.

In terms of objective quality evaluation, two most commonly used transparency metrics, PSNR and SSIM were adopted to calculate the fidelity of the proposed method.

PSNR, short for peak signal-to-noise ratio, is a full-reference image quality evaluation index, which is formulated as Equations (19) and (20),

$$PSNR=10\times {log}_{10}\left(\frac{{MAX}^2}{MSE}\right)$$

(19)

$$MSE=\frac{1}{mn}\sum _{i=0}^{m-1}\sum _{j=0}^{n-1}{\left[I\left(i-j\right)-K\left(i-j\right)\right]}^2$$

(20)

where $I\left(i,j\right)$ and $K\left(i,j\right)$ are the pixel values of the original frame and the embedded frame at the $(i,j)$ position, respectively.

In addition, SSIM is a commonly used image quality evaluation index. The calculation of SSIM is shown in Equation (21),

$$SSIM\left(x,y\right)=l{(x,y)}^{\alpha }\times c{(x,y)}^{\beta }\times s{(x,y)}^{\gamma }$$

(21)

where l, c, and s represent the luminance, contrast, and structure, respectively. and $\alpha$, $\beta$, $\gamma$ represent the percentage of each component in SSIM. When $\alpha$, $\beta$, $\gamma$ are equal to 1, we can obtain Equation (22),

$$SSIM\left(x,y\right)=\frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_2)({\sigma }_x^2+{\sigma }_y^2+C_2)}$$

(22)

where x and y denote the two tested images, $\mu$ denotes the mean pixel value of the frame, and $\sigma$ denotes the covariance pixel value of the frame. $C_1$ and $C_2$ are two constants used to control the denominator not being zero.

We calculated the PSNR and SSIM of the tested videos. Since the watermark was embedded on the Y component of the video, we only calculated the PSNR and SSIM for the Y component of the watermark embedded frame. As shown in

Table 4. PSNR, SSIM, and capacity of the tested videos.

Video Name	PSNR	SSIM
Hall	40.37	0.933
News	40.31	0.936
City	38.54	0.935
Akiyo	41.21	0.928
Station2	41.54	0.927
Pedestrian-area	42.14	0.929
BasketballDrive	41.09	0.921
BQterrace	40.11	0.929

, the average PSNR value of the eight tested videos reaches 40.7, and the maximum value reaches 42.14. In addition, the average SSIM value of the eight videos reaches 0.93. The results indicate that the proposed algorithm has good visibility.

In addition to invisibility, capacity is an important evaluation index of watermarking algorithms. Watermark capacity refers to how much watermark information can be embedded in a media file, and the larger the watermark capacity, the larger the amount of information that can be embedded.

We also tested the capacity of the proposed method; since we embedded 3 bits in one GOP in the experiment, the capacity can be calculated according to the frame number of the video. The capacity presents a positive correlation with frame number of the video. In our paper, we set the video embedding capacity to be 30 bits.

4.4. Robustness of the Proposed Method

Robustness refers to the ability of a digital watermark to maintain a certain level of detectability or extractability when subjected to various attacks. In this study, robustness was measured using the normalized correlation (NC) value, which is calculated as Equation (23),

$$NC=\frac{{\sum }_{i=1}^N{\sum }_{j=1}^{M}w(i,j)w\prime (i,j)}{{\sum }_{i=1}^N{\sum }_{j=1}^M{w}^2(i,j)w^{\prime 2}(i,j)}$$

(23)

where w is the original watermark and $w^{\prime }$ is the extracted watermark. The NC value ranges from 0 to 1, and the closer the NC value is to 1, the stronger the robustness of the method is. In this part, image-processing attacks, geometric attacks, video-processing attacks, and compression attacks were tested.

4.4.1. Image Processing Attacks

Image-processing attacks generally include noise attacks, histogram equalization, and median filtering. Two most common noise types, Gaussian noise and salt and pepper noise, were tested in the experiment. Gaussian noise with mean 0 and variance 0.01 was adopted. Furthermore, the salt and pepper noise density was chosen to be 0.05. In addition to noise, we tested the attacks of median filtering and histogram equalization. In the experiment, the median filter with size 3 × 3 was selected. Histogram equalization can homogenize the distribution of pixel values, thus resulting in a more uniform distribution. The sum of the histogram bins was set to 255 in the experiment.

We tested the robustness of the proposed method against the attacks of Gaussian noise, salt and pepper noise, median filtering, and histogram equalization. The NC values of extracted watermark reach 1 for all the eight tested videos after different image-processing attacks. The results show that common image-processing attacks rarely affect the robustness of the proposed method.

4.4.2. Geometric Attacks

Geometric attacks usually cause the target object in the video frame to be distorted, deformed, or shrunken and greatly increase the difficulty of watermark extraction. In this experiment, rotation attacks (1${}^{\circ }$, 30${}^{\circ }$, 45${}^{\circ }$), cropping attacks (cropping the top side and bottom side by 10%), and scaling attacks (shrinking to 3/4, shrinking to 2/3, enlarging to 5/4, and enlarging to 4/3) were tested.

According to the results in

Table 5. NC values of geometric attacks.

	Hall	News	City	Akiyo	Station2	Pedestrian-Area	BasketballDrive	BQterrace
Cropping (10%)	1	1	1	1	1	1	1	1
Scaling (2/3)	1	1	1	1	1	1	1	1
Scaling (3/4)	1	1	1	1	1	1	1	1
Scaling (5/4)	1	1	1	1	1	1	1	1
Scaling (3/4)	1	1	1	1	1	1	1	1
Rotation (1${}^{\circ }$)	1	1	1	1	1	1	1	1
Rotation (30${}^{\circ }$)	1	1	1	1	1	1	1	1
Rotation (45${}^{\circ }$)	0.96	0.93	0.96	0.96	1	1	1	1

, it can be seen that the NC value of the algorithm is 1 for all eight videos when cropping attacks are performed, so the cropping attacks rarely affect the performance of the algorithm. For most of the rotation attacks and scaling attacks, the watermark can still be extracted correctly for the eight videos. The good robustness is partly attributed to the design of robust watermark templates, and it also owing to the appropriate setting of parameter constant C referred to in Section 2.2.1, which will be discussed in Section 4.5. However, under the attack of 45${}^{\circ }$, for four tested videos, there is a slight impact. Although this causes some watermark extraction errors, the high NC values show that the proposed algorithm is still robust to rotation and scaling attacks.

4.4.3. Video-Processing Attack

Video-processing attacks can destroy the synchronization information during the watermark extraction process, which may lead to information loss or misalignment and cause serious errors in the extraction process. Therefore, the robustness of the algorithm to video-processing attacks is crucial. In this section, three main attacks, frame averaging, frame swapping, and frame dropping, were tested to evaluate the algorithm’s robustness. A frame-swapping attack involves swapping two or more frames in a video sequence, thus disrupting the temporal continuity of the video. In our experiments, 10%, 20%, and 30% frame swapping were tested, where ($a\%$) denotes that $a\%$ video frames were swapped. A frame-dropping attack involves the deletion of some frames of the video sequence, creating disorder. In this part, 10%, 20%, and 30% of the frames were randomly deleted. A frame averaging attack is implemented by averaging multiple frames of a video sequence to produce a new frame. In this experiment, an averaging attack was performed for each frame by averaging its neighboring 5 frames.

We conducted experiments on the eight tested videos with different video-processing attacks. The NC values of the extracted watermark are 1 for all the eight tested videos when the algorithm is subjected to the attacks of frame averaging, 10%, 20%, and 30% frame swapping and 10%, 20%, and 30% frame dropping. This demonstrates that the proposed algorithm has strong robustness against video-processing attacks.

4.4.4. Robustness against CRF and Frame Rate Changing

In video coding process, CRF is a parameter that controls the size and the compression ratio of a video. Frame rate changing is also a video-processing technique used to transform the frame rate of a video from one to another. Different video devices and standards usually use different frame rates, but changing the CRF value and frame rate has an impact on the extraction of the video watermark. This is because CRF affects the pixel value of video frames and thus the accuracy of watermark extraction, while frame rates affect the number of video frames and thus influence the synchronization information. Therefore, we also tested the robustness of the proposed method against CRF and frame rate changing attack. In this section, we tested the case of CRF with 23, 25, 28 and the case of converting the video frame rate to 15 and 60.

The results in

Table 6. NC values after video attack.

	Hall	News	City	Akiyo	Station2	Pedestrian-Area	BasketballDrive	BQterrace
CRF23	1	1	1	1	1	1	1	1
CRF25	1	1	1	1	1	1	1	1
CRF28	0.86	1	1	1	0.96	0.89	0.82	1
FRC(15) 1	1	1	1	1	1	1	1	1
FRC(60) 2	1	1	1	1	1	1	1	1

¹ Frame rate changing (15 fps). ² Frame rate changing (60 fps).

show that the NC value of the extracted watermark is 1 after frame rate changing for both CIF and 1080p video resolutions, which indicates that the proposed algorithm has good robustness in coping with the frame rate changing attack. This is because the frame rate changing can be regarded as a special frame dropping and frame insertion operation, and the proposed GOP grouping method can effectively resist this attack. For the compression attack, we tested the robustness of the video after different CRF compression. For example, when the CRF is 25, the NC value of all eight CIF videos is 1. However, in comparison, when the CRF is 28, the average NC value of 1080p video drops to 0.92, and the CIF video is 0.97. This is because after compression, the amplitude of the template sequence is lower, which may lead to the case of extraction error. Therefore, when dealing with compression attacks, the present algorithm needs to be improved in future research.

4.5. The Effect of Constant C

In the proposed algorithm, C values have a great impact on the robustness. C can affect the invisibility and robustness of the proposed algorithm. Usually, a larger C leads to a decrease in the video invisibility but an increase in the robustness.

Table 7. The value of NC and imperceptibility for different C.

	Hall			News			City			Akiyo
C	7000	8000	9000	7000	8000	9000	7000	8000	9000	7000	8000	9000
PSNR	42.52	41.38	40.37	42.47	41.31	40.3	40.7	39.55	38.54	43.37	42.22	41.21
SSIM	0.96	0.95	0.93	0.96	0.95	0.94	0.96	0.95	0.93	0.96	0.94	0.93
Rotation	0.71	0.94	0.99	0.93	0.98	0.98	0.98	0.99	0.99	0.99	0.99	0.99
Scaling	1	1	1	0.69	0.71	1	1	1	1	1	1	1
Frame attack	0.99	1	1	1	1	1	0.99	1	1	1	1	1
CRF28	0.59	0.82	0.86	0.69	0.96	1	0.93	1	1	0.89	0.96	1

shows the changes in transparency and robustness by setting different C values for the tested CIF videos. For the rotation attack, we calculated the average NC values of rotating 1${}^{\circ }$, 30${}^{\circ }$, and 45${}^{\circ }$. For scaling attacks, the average NC values were calculated after scaling down to 2/3 and scaling up to 4/3. For frame attacks, we calculated the average NC values after frame swapping (30%) and frame dropping (30%).

According to the data in Table 7 and Figure 9, it can be observed that for each video, the NC value gradually increases with the increase in C, while the PSNR and SSIM gradually decrease. As in the tested “News” video, when C = 7000, the average NC value is 0.86, and the PSNR is 42; when C = 8000, the average NC value increases to 0.92, but the PSNR decreases to 41; when C = 9000, the average NC value further increases to 0.99; however, the PSNR further decreases to 40. Therefore, in the proposed method, 9000 was selected as the C value of the CIF video to ensure that the watermarked video has good robustness while maintaining relatively high imperceptibility.

4.6. Time Efficiency

Time efficiency is a response to the time resources needed to process the data. A video is composed of a series of still images, and real time is a very important factor for designers of video watermarking algorithms. We tested the time required to embed and extract the watermark for different videos and also calculated the average watermark embedding and extraction time for each frame. The results are shown in

Table 8. Time efficiency.

	Embedding Time (s)	Embedding Time per Frame (s)	Extraction Time (s)	Extraction Time per Frame (s)
Hall	20.86	0.07	2.04	0.007
News	21.49	0.07	8.05	0.004
City	21.63	0.07	1.55	0.005
Akiyo	21.56	0.07	5.20	0.005
Station2	681	1.35	26.19	0.05
Pedestrian-area	509	1.35	20.26	0.05
BasketballDrive	662	1.32	24.51	0.05
BQterrace	822	1.30	30.50	0.05

.

Experiments show that for the four tested CIF videos, average embedding time is of roughly 0.07 s per frame, and the average extraction time is about 0.005 s, while the average embedding and extraction time per frame are about 1.35 s and 0.05 s, respectively, for the four tested 1080p videos. The results present that the proposed algorithm is able to complete the task of watermark embedding and extraction in a short time with high efficiency.

4.7. Algorithm Comparison

References [21,22] were selected for comparison. We performed the proposed method on the same tested videos of references [21,22], and the experimental results are shown in

Table 9. Comparison with different methods.

	Chen [21]	Sun [22]	Proposed
PSNR	40.80	39.59	40.82
Capacity	16	16	30
Gaussian noise	1	0.75	1
Salt and pepper noise	0.95	0.84	1
Geometric resynchronization	Not required	required	Not required
Rotation	0.91	0.84	1
Scaling	0.97	0.85	1
Frame dropping	Non-robust	Non-robust	1

.

In [21,22], binary strings of the same size of 16 bits were used. We instead used a binary string of size 30 bits. Theoretically, on average, 1 bit information can be embedded per 10 video frames. Reference [22] requires an average of nine video frames to embed one bit, while reference [21] requires an average of 42 video frames to embed one bit. It can be found that our embedding capacity is increased by more than three times compared to that of reference [21], while it remains basically the same as that of reference [22]. In terms of PSNR, our algorithm approaches approximately the same PSNR as in [21,22], both around 40 dB. In terms of image-processing attacks, our algorithm is significantly more robust than [22] and more robust to salt and pepper attacks compared to [21]. Our algorithm performs well in geometric attacks, and it does not require geometric synchronization as in [22]. When referring to the rotation results, the NC value of our algorithm is 0.09 higher than [21] and 0.16 higher than [22]. For scaling, our algorithm also has higher NC values than [21,22]. In addition, the experiment of [21] cannot resist cropping attacks because it breaks spatial synchronization. Finally, our algorithm is also effective against frame-dropping attack. In contrast, references [21,22] are not resistant to frame-dropping attacks because we can reset the GOP size when extracting the watermark.

5. Conclusions

In this study, the 2D-DFT template algorithm was improved. We introduced a prime factorization method to design watermark template sequences, and the embedding capacity in each GOP can be increased from 1 bit to a theoretical n bits. In addition, we modified the GOP segmentation method to provide frame deletion resistance for the proposed method.

The experiments show that the algorithm is robust to image-processing attacks, geometric attacks, frame attack, and compression attacks. Compared with other algorithms, the proposed algorithm has good guarantees in terms of robustness, real time, and capacity.

In future work, we will study the visual perceptual factors of videos to improve watermark transparency. We will also explore methods for joint attack resistance to optimize the performance of watermarking algorithms.