RoSe-Mix: Robust and Secure Deep Neural Network Watermarking in Black-Box Settings via Image Mixup

Abstract

Due to their considerable costs, deep neural networks (DNNs) are valuable assets that need to be protected in terms of intellectual property (IP). From this statement, DNN watermarking gains significant interest since it allows DNN owners to prove their ownership. Various methods that embed ownership information in the model behavior have been proposed. They need to fill several requirements, among them the security, which represents an attacker’s difficulty in breaking the watermarking scheme. There is also the robustness requirement, which quantifies the resistance against watermark removal techniques. The problem is that the proposed methods generally fail to meet these necessary standards. This paper presents RoSe-Mix, a robust and secure deep neural network watermarking technique designed for black-box settings. It addresses limitations in existing DNN watermarking approaches by integrating key features from two established methods: RoSe, which uses cryptographic hashing to ensure security, and Mixer, which employs image Mixup to enhance robustness. Experimental results demonstrate that RoSe-Mix achieves security across various architectures and datasets with a robustness to removal attacks exceeding 99%.

1. Introduction

Recent advances in machine learning (ML) have revolutionized many fields, like natural language processing [1], time series analysis [2], and computer vision [3]. Deep neural networks (DNNs) are the ML models that contribute the most to these advancements. However, their development necessitates important and huge resources, including large and high-quality datasets that are often difficult to obtain due to confidentiality constraints, expensive hardware, like GPUs, and the rare expertise of specialists for data annotation and of data science engineers to design well-adapted model architectures and hyper-parameters [4].

As stated, training a DNN is a costly task, also making them valuable assets that need to be protected in terms of intellectual property (IP). To safeguard the latter, the most promising solution is DNN watermarking [5,6,7]. This allows the owner to embed a secret in the model in such a way that it proves its ownership. DNN watermarking is classified into two settings: white-box and black-box. These techniques are defined by the level of access required by the watermark extractor. In the white-box technique [5,8,9,10,11,12,13,14], internal parameters that directly correspond to the weights of the DNN models are accessible. In the black-box technique, only the final output of the DNN model is observed. In this case, the watermark is extracted by querying a set of trigger inputs to the DNN model [15,16,17,18,19,20,21,22]. This set is called a trigger set and consists of crafted inputs and outputs to embed a backdoor in the model to prove ownership. However, these watermarking modulations have two main limitations. First, the set of samples created to embed the watermark is the same as those used to perform the verification. If the verifier is malicious, they can potentially steal this set and claim model ownership. Second, the choice of target labels is generally naive, leading to a minimal amount of work for a potential usurper to reverse engineer the input–output pairs a posteriori and then build an illegitimate IP proof. In this paper, we present RoSe-Mix, a black-box watermarking technique that takes advantage of two existing black-box watermarking techniques, RoSe [23] and Mixer [24], to exponentially increase the necessary work for an attacker to claim an illegitimate model. Our contributions are summarized as follows:

• We present the strengths of RoSe [23] and Mixer [25] and demonstrate how their combination can enhance the security and robustness of the watermark while being merged into RoSe-Mix.
• We provide a security analysis of this method by formalizing the complexity for an attacker to break the watermarking scheme, demonstrating that breaking the watermarking protocol is exponentially hard with respect to the size of the selected secret key.
• We conduct experiments on several datasets and models to evaluate the performance of our proposed method.

Section 2 provides an overview of DNN watermarking techniques. Section 3 describes our motivation to merge RoSe and Mixer and presents RoSe-Mix. Finally, Section 4 presents experimental results using different datasets and models.

2. Related Works

In this section, we introduce the notion of DNN watermarking by defining its requirements and properties. We then present state-of-the-art methods, particularly focusing on RoSe [23] and Mixer [25], the two key methods used in our proposal.

DNN watermarking is a technique that consists of embedding a secret in a model. This secret can then be extracted later and used by its owner to identify or track the model’s illegal copies. To be usable in a real scenario, the watermarking modulation should respect several properties [26]. The first one concerns the model’s fidelity, which quantifies the performance degradation on the main task of the model after watermark embedding. The second property is the robustness of the watermark, which defines whether the watermark resists various model modifications (intentional or not). Finally, security corresponds to the difficulty for an attacker to estimate the secret key linked to the watermark.

The first DNN watermarking modulation was proposed by Uchida et al. [5] and consists of embedding a binary string in the model parameters using a regularization term and a secret key. When the model owner wants to extract the secret binary string, it is mandatory to have full access to the model parameters. This constraint categorizes the method as a white-box technique [8,10,11,12,27,28]. On the other hand, black-box techniques rely on the fact that only the inputs and outputs of the model are accessible during the verification phase. This scenario fits well with machine learning as a service application where DNNs are available through an API.

Relying on this assumption, black-box watermarking techniques hide the secret in the model’s behavior. Adi et al. [15] were the first to propose such a technique. Their main contribution is the definition of the trigger set $S_e$, which is a set of crafted inputs, $\tilde{X}$, and outputs, $\tilde{Y}$, used to create a backdoor in the model. This backdoor is then used by the owner to recognize its possibly stolen model. In their paper, $\tilde{X}$ is composed of unrelated images (i.e., out-of-distribution images), while $\tilde{Y}$ is randomly selected from the set of classes, $\mathcal{C}$. Several methods have been developed focusing on how to build the trigger set. We can also cite Zhang et al. [17], who used training images with additional patterns (text or noise) to trigger the model. Other methods craft $S_e$ using different techniques, such as adversarial examples [18] or invisible masks [16]. While the majority of existing methods focus on the machine-learning aspect of the problem, i.e., how to embed a watermark that respects the fidelity and robustness requirements, few are interested in the security side.

To improve the security of the watermark, Kallas et al. [23] proposed a black-box watermarking protocol called RoSe. This method uses one-way hashing functions and the injection of image-label pairs into the DNN during training, with these pairs serving as the trigger set for verifying ownership of a possibly stolen model. This method demonstrates strong security guarantees since breaking the watermarking scheme is NP-hard. Figure 1 illustrates this process for the MNIST dataset.

In their subsequent work, they introduce Mixer [25], a black-box watermarking technique where the triggers used during training differ from those used for ownership verification. To achieve this, they use the principle of Mixup, a data augmentation technique originally proposed by Zhang et al. [24], where new training samples are created by combining pairs of samples and their corresponding labels. By doing so, the method breaks the dependency on a fixed trigger set and its limited size. Figure 2 shows the Mixer process to generate one sample with $k=2$.

3. Proposed Method

In this section, we discuss the limitations of RoSe and Mixer, which have motivated the development of our new proposal, RoSe-Mix. Rather than directly address all the limitations, the new method takes advantage of the strengths of both techniques to improve security.

3.1. Motivation

Although RoSe and Mixer offer strong protection against usurpers, where any drop in watermark recovery accuracy is closely tied to a drop in the main task accuracy, each method has inherent drawbacks:

• Trigger set vulnerability: RoSe’s main limitation lies in its trigger set, which acts as the Owner’s key. Since the same trigger set is used for both watermark embedding and verification, it is vulnerable to a potentially untrustworthy verifier. Furthermore, increasing the number of watermark samples from the training data can degrade the model’s performance on the main task, limiting the scalability of RoSe [23].
• Domain space overlap: In Mixer, the verification trigger set differs from the trigger samples used during watermark embedding. However, since both sets are derived from the same data domain, represented as a sphere in the feature space, there remains a risk that an attacker could generate keys $\lambda$ and $\mu$ that are very similar to the Owner’s keys. This could place the attacker in the same domain space, making it difficult for the verifier to distinguish between the legitimate Owner and the attacker in case of a dispute [25].

Although some of the limitations persist, our proposed method does not aim to resolve all of them. Instead, it combines the strong security features of both RoSe and Mixer, capitalizing on their complementary strengths to enhance the security robustness of the watermarking process. By merging their protective mechanisms, the new approach offers a higher level of security against potential attacks.

3.2. RoSe-Mix: A Hybrid Watermarking Approach

In this section, we introduce our proposed method, which combines the secret-keyed hash function from RoSe with the Dirichlet distribution from Mixer to generate the mixing coefficients, denoted by $\mathit{\lambda }$ and $\mathit{\mu }$. These coefficients are then used to create trigger samples and their corresponding labels, leveraging the strengths of both methods. This fusion improves the security of the watermark generation, allowing for ownership verification using unseen samples.

The generation of the trigger set begins by selecting a set of $n_e$ images, namely $\{x_1,x_2,\cdots ,x_{n_e}\}$, from the training set $D_{\mathrm{train}}={\left\{(x_i,y_i)\right\}}_{i=1}^n\subset \mathcal{X}\times \mathcal{C}$, where n is the number of training samples, $\mathcal{X}$ the input space, and $\mathcal{C}=\{1,\cdots ,c\}$ the set of classes. These images are concatenated and hashed using a cryptographic hash function, H, with the secret key, $sk$, resulting in a unique bitstream:

$$\mathrm{bitstream}=H(x_1\parallel x_2\parallel \cdots \parallel x_{n_e};sk),$$

(1)

This bitstream is then transformed into a numeric seed by applying the function ${\mathcal{O}}_H(·)$:

$$\mathrm{seed}={\mathcal{O}}_H\left(\mathrm{bitstream}\right),$$

(2)

Using this seed, a deterministic sequence of random numbers is generated, which is then used to form the vector $\mathit{\alpha }=({\alpha }_1,\cdots ,{\alpha }_k)$. This vector serves as a parameter for a Dirichlet distribution that generates the mixing coefficients $\mathit{\lambda }=({\lambda }_1,\cdots ,{\lambda }_k)$:

$$\mathit{\lambda }\sim \mathrm{Dirichlet}\left(\alpha \right),$$

(3)

The same procedure is repeated to generate the coefficients in $\mathit{\mu }$. These coefficients are used to generate the new pairs of samples by merging them according to the Mixup principle described in Mixer [25]. For inputs, we have the following rule to generate a key input, ${\tilde{x}}_s$:

$${\tilde{x}}_s=\mathsf{clip}\left(\sum _{i=1}^k{\lambda }_i{x}_i+x_{\mathrm{overlay}}\right),$$

(4)

where $x_{\mathrm{overlay}}$ is an additive overlay added to the Mixup sample to enhance secrecy, and $\mathsf{clip}$ ensures that the pixel values remain within valid limits. These Mixer samples form the set of inputs $\tilde{X}={\left\{{\tilde{x}}_s\right\}}_{s=1}^{n_e}$. Figure 3 shows an example of samples from the trigger set generated as described previously. As an example, Figure 3d is a mixed sample between a car, $x_1$, and a horse, $x_2$, where ${\lambda }_1>{\lambda }_2$.

Next, the corresponding labels for the Mixer samples are generated. For each Mixer sample, ${\tilde{x}}_s$, the true labels, $y_i$, of the constituent images, $x_i$, are converted to their one-hot encoded forms, ${\mathit{y}}_{\mathit{i}}$. The key label, ${\tilde{y}}_s$, for the Mixer sample is a combination of these one-hot encoded labels, weighted by the ${\mu }_i$ coefficients:

$${\tilde{y}}_s=\sum _{i=1}^k{\mu }_i{\mathit{y}}_{\mathit{i}},$$

(5)

where ${\sum }_{i=1}^k{\mu }_i=1$. The set of all generated labels, $\tilde{Y}$, corresponds to the Mixer samples in $\tilde{X}$ and forms the trigger set $S_e={\left\{({\tilde{x}}_s,{\tilde{y}}_s)\right\}}_{s=1}^{n_e}$.

Once the Mixer trigger set, $S_e$, is created, it is combined with the training dataset, $D_{\mathrm{train}}$, during the training phase of the model. It can also be applied with fine-tuning in the case where we distribute the model among several clients (called traitor tracking in watermarking [30]). By including these watermarked samples in the training process, the model learns to associate specific patterns with the watermarked outputs, embedding the watermark in its behavior. This process ensures that the model is capable of recognizing these mixed-up patterns, even for unseen data during verification.

To proceed with the verification, the owner provides to the verifier the secret key, $sk$, the hash function, H, and the vectors $\mathit{\lambda }$ and $\mathit{\mu }$. The verifier uses this information to regenerate Mixer samples and queries the model $F:\mathcal{X}\to \mathcal{C}$ with the newly generated set $S_d$ of size $n_d$. The outputs of the model are evaluated based on the ratio ${\rho }_{n_d}$, which is the proportion of correctly predicted labels:

$${\rho }_{n_d}={\displaystyle \frac{\left|\right\{{\tilde{x}}_s\in S_d\mid F\left({\tilde{x}}_s\right)={\tilde{y}}_s\left\}\right|}{n_d}}.$$

(6)

If ${\rho }_{n_d}$ exceeds a predefined threshold, $\tau$, the model is verified to be watermarked, confirming ownership.

3.3. Security Analysis

In this section, we analyze and derive an approximation of the computational work required by an attacker to break the RoSe-Mix watermarking scheme.

The main goal of the attacker is to find or approximate all secrets to build the ownership proof. These secrets are divided into two parts, which correspond to the work of each original method:

a.

The bitstream (in Equation (1)) that is used to generate the seed in the RoSe method (We can approximate the work required for RoSe-Mix by adopting the RoSe analysis, where, instead of guessing the correct labels, the attacker must guess the correct bitstream and thereby the seed. This assumes that the function ${\mathcal{O}}_H$ is known).

b.

$\left(\mathit{\lambda },\mathit{\mu },x_{\mathrm{overlay}}\right)$, which are the parameters used in the Mixer method.

In the rest of this section, we define the work required for RoSe and Mixer separately and merge the effort to give a lower bound of the resources needed for an attacker to break the watermarking scheme.

3.3.1. RoSe

In the RoSe method with the highest level of security, the attacker’s objective is, using a fixed generated key, $s{k}^{\prime }$, to generate adversarial examples (by modifying the last significant bits of the images) that give a good bitstream to obtain ${\rho }_{n_a}\ge \tau$.

The computational effort required for this attack involves two primary components:

a.

Generating black-box adversarial examples.

b.

Regenerating hash-based triggers for verification.

The approximate lower bound of the computational effort for an usurper to claim the model is estimated as:

$$W_{RoSe}=R{\displaystyle \frac{2t{\omega }_F}{{log}_2\left(c\right)}}+R(2^R-1){\displaystyle \frac{{\omega }_H+{\omega }_F}{{log}_2\left(c\right)}},$$

(7)

where t is the number of iterations used to generate an adversarial sample, c is the number of classes, ${\omega }_F$ is the cost of a forward pass, ${\omega }_H$ is the cost of a hash computation, and the term R represents the rarity, which grows as a function of probability $\mathbb{P}({\rho }_{n_a}\ge \tau )$. The details of this estimation can be found in [23].

3.3.2. Mixer

In contrast, the Mixer method requires the attacker to predict the correct mixing vectors, $\mathit{\lambda }$ and $\mathit{\mu }$, which are drawn from a Dirichlet distribution, but before this, the attacker must also guess the number of classes, k, to mix from the total set of c possible classes.

The probability of successfully guessing the correct number of classes k from c possible classes is ${\displaystyle \frac{1}{\left(\genfrac{}{}{0pt}{}{c}{k}\right)}}$, and the probability of predicting the mixing coefficients in $\mathit{\lambda }$ and $\mathit{\mu }$ within the tolerance $\delta$, is determined by integrating the Dirichlet density function over the $\delta$-neighborhood around the actual values of $\mathit{\lambda }$ and $\mathit{\mu }$. This probability can be approximated by considering the space of possible values for $\lambda$ and $\mu$ as being divided into $N^k$ discrete steps. Therefore, the probability of correctly predicting $\mathit{\lambda }$ within this range is:

$$P_{\mathrm{values}}={\int }_{\begin{array}{c}\widehat{\mathit{\lambda }}:\\ |{\lambda }_i-{\widehat{\lambda }}_i|\le \delta \end{array}}f(\widehat{\mathit{\lambda }};\mathit{\alpha })d\widehat{\mathit{\lambda }},$$

(8)

where $\mathit{\lambda }$ is the true vector, $\widehat{\mathit{\lambda }}$ the guessed vector, and $f(\widehat{\mathit{\lambda }};\mathit{\alpha })$ is the Dirichlet density function. The same formula can be applied to $\mathit{\mu }$. The overall probability of success for the attacker is the product of the probability of correctly guessing the number of classes and the probability of predicting the mixing coefficients:

$$P_{\mathrm{overall}}={\displaystyle \frac{1}{\left(\genfrac{}{}{0pt}{}{c}{k}\right)}}\times P_{\mathrm{values}}.$$

(9)

Now we transform these probabilities into work. The first step is to guess the positions of the k non-zero elements among the c possible positions. The number of possible combinations is given by the binomial coefficient $\left({\displaystyle \genfrac{}{}{0pt}{}{c}{k}}\right)$.

For the number of trials to guess the values, once the positions are guessed, the attacker needs to guess the values of the k non-zero elements within the specified tolerance $\delta$. The Dirichlet distribution generates continuous values, so an exact match is nearly impossible, but within a tolerance $\delta$, we can estimate the number of trials as follows.

If we assume that $\mathit{\lambda }$ and $\mathit{\mu }$ are discretized into N steps, where $\delta =1/N$ represents the tolerance within which the attacker must guess the correct coefficients, the work is equal to $N^k$. Finally, the total work to find $\lambda$ for the Mixer method is given by:

$$W_{\lambda }=\left({\displaystyle \genfrac{}{}{0pt}{}{c}{k}}\right)\times N^k,$$

(10)

where $\left({\displaystyle \genfrac{}{}{0pt}{}{c}{k}}\right)$ is the binomial coefficient that represents the number of ways to choose k classes from $\mathcal{C}$, and $N^k$ represents the number of possible combinations for the coefficients in $\lambda$ within the tolerance $\delta$. The same work can be calculated for $\mu$ :

$$W_{\mu }=\left({\displaystyle \genfrac{}{}{0pt}{}{c}{k}}\right)\times N^k.$$

(11)

3.3.3. RoSe-Mix

In the RoSe-Mix method, the attacker faces a combined challenge. They must not only regenerate the bitstream as in RoSe, but also correctly predict the mixing coefficients of $\lambda$ and $\mu$ as in Mixer. This significantly increases the complexity of the attack. To break RoSe-Mix, the attacker must (1) generate a trigger set that gives the good bitstream, (2) guess the correct number of classes k, and (3) predict the coefficients of $\mathit{\lambda }$ and $\mathit{\mu }$ within a tolerance $\delta$.

Given this, the total work required for the RoSe-Mix method is the product of the work required for RoSe (Equation (7)) and the work required to guess the correct mixing coefficients and the number of classes (Equations (10) and (11)). Therefore, the total work for the attacker is:

$$W_{\mathrm{RoSe}\text{-}\mathrm{Mix}}=\underset{W_{\mu }+W_{\lambda }}{\underbrace{2\times \left({\displaystyle \genfrac{}{}{0pt}{}{c}{k}}\right)\times N^k}}\times W_{\mathrm{RoSe}}.$$

(12)

This work is significantly larger than either RoSe or Mixer alone because the attacker must now handle both the complexity of generating adversarial examples for the hash-based triggers and guessing the correct mixing coefficients from a Dirichlet distribution. The number of possible combinations for $\mathit{\lambda }$ and $\mathit{\mu }$ grows exponentially with k and N, while the work to generate adversarial examples grows with t, the number of iterations needed to generate an adversarial example, and R, the rarity.

4. Experimental Results

4.1. Settings

4.1.1. Dataset and Models

To assess our proposed method’s performance and general applicability, we conducted experiments on four datasets with several models. Respectively, we use LetNet5 [31] with MNIST [32] and FashionMNIST [33], VGG19 [34] with CIFAR-10 [29], and ResNet50 [35] with GTSRB [36]. We have also used a ResNet50 pre-trained on ImageNet [37] with CIFAR-10 for the transfer-learning experiments. Models for MNIST and FashionMNIST are trained over 100 epochs; CIFAR10, GTSRB, and transfer-learning models are trained over 200 epochs. The last model is used to test our method by embedding via fine-tuning. A batch size of 64 is used for all models. Datasets are split 80% training, 10% validation, and 10% fine-tuning.

4.1.2. Metrics

The performance of the proposed approach is evaluated using four metrics: TA, $Re{c}_{Tr}$, $Re{c}_{Ts}$, and USR. TA, test accuracy, represents the model’s accuracy on the original task. $Re{c}_{Tr}$ is the recovery rate of watermark samples $S_e$ used for the embedding, while $Re{c}_{Ts}$ measures the recovery rate on the unseen set $S_d$. Finally, the usurper success rate (USR) reflects the correct watermark recovery rate for samples generated by a usurper, based on 500 random fake keys. Relying on [38], we also define the threshold $\tau =25\%$, which gives us a reliable ownership demonstration with confidence greater than $1-2^{-64}$ for $k=500$.

4.2. Modulation Parameters

In Section 3.2, we described our proposed method and defined some parameters for which, in this subsection, we perform experiments to determine optimal values for k (the number of classes used to generate Mixup watermark samples) and $n_e$ (the size of the trigger dataset). We evaluated the impact of k and $n_e$ on the performance of the model for each dataset. To find the best k, we tested values in the range $k\in \{2,\cdots ,c\}$, with $n_e$ fixed at 500, selecting the k that produced the highest TA and $Re{c}_{Tr}$. After identifying the optimal k, we plotted the curves of $n_e$ to further assess its influence on performance, ultimately selecting the value that resulted in the best TA and $Re{c}_{Tr}$ for each dataset.

The results are shown in Figure 4 for k and Figure 5 for $n_e$. For the k plots, we omitted some datasets with similar behavior to conserve space. The watermark recovery rates, $Re{c}_{Tr}$ and $Re{c}_{Ts}$, are measured based on the sizes of the watermark datasets, $n_e$ and $n_d$, respectively. For $Re{c}_{Tr}$, we performed experiments with $n_e\in \{100,300,500,700,1000,3000,5000,7000,10,000\}$. The TA of each model without a watermark can be found in Figure

Table 1. RoSe-Mix performance metrics: TA, $Re{c}_{Tr}$, and $Re{c}_{Ts}$ under various attacks.

Metric	Host DNN	Watermarked DNN	Fine-Tune	Dyn. Quant.	Full Uint8. Quant.	Full Int8. Quant.	Float16 Quant.	JPEG55
MNIST
TA	98.64	99.11	99.11	99.14	99.14	99.14	99.11	99.09
$Re{c}_{Tr}$	13.54	100	100	100	100	100	100	100
$Re{c}_{Ts}$	12.50	100	100	100	100	100	100	100
FashionMNIST
TA	88.44	88.63	88.63	88.62	88.62	88.62	88.63	87.73
$Re{c}_{Tr}$	8.33	89.4	89.4	90.2	90.2	90.2	89.4	88.6
$Re{c}_{Ts}$	7.8	88	88	88	88	88	88	88.4
Cifar10
TA	70.52	78	78	78.04	78.04	78.04	77.99	75.42
$Re{c}_{Tr}$	10.42	100	100	100	100	100	100	100
$Re{c}_{Ts}$	7.29	96.8	96.8	96.8	96.8	96.8	96.8	96.4
Transfer Learning
TA	86.54	86.57	71.8	86.63	86.63	86.63	86.56	80.36
$Re{c}_{Tr}$	11.46	100	67	100	100	100	100	100
$Re{c}_{Ts}$	9.38	100	70.2	100	100	100	100	100
GTSRB
TA	78.89	90.1	80.84	90.08	90.08	90.08	90.1	86.45
$Re{c}_{Tr}$	2.2	100	89.4	100	100	100	100	92.8
$Re{c}_{Ts}$	1.4	88	94.8	88.6	88.6	88.6	88.2	79.4

. To determine the optimal $n_e$, we plotted the accuracy of the model as a function of $n_e$ and selected the value that produced the highest TA and $Re{c}_{Tr}$. The best k was 2 for MNIST, FashionMNIST, and CIFAR-10, and 7 for GTSRB. Additionally, starting from $n_e=500$, both TA and $Re{c}_{Tr}$ showed significant improvement and stabilized across all datasets. However, increasing the value of $n_e$ significantly decreases TA probably because the task related to the watermark takes more importance compared to the main task in the learning process, leading to a task interference [39] in which we note a decrease in performance on the main task for complex problems (e.g., Figure 5c). As a result, we used $n_e=500$ for the remaining experiments.

4.3. Fidelity

In this experiment, we evaluate the fidelity of the model, i.e., if the watermark embedding process degrades the performance TA. In Table 1, the first two columns show that embedding the watermark has minimal impact on the performance of the model for the main task, as the difference in TA between the host and watermarked models is negligible. Both the trigger set $S_e$ and $S_d$ yield recovery rates ($Re{c}_{Tr}$ and $Re{c}_{Ts}$) of at least 88% for the watermarked model. In contrast, when the host model is tested with those trigger sets, the recovery rates remain very low, indicating the watermark’s integrity.

An additional observation is that the TA of the watermarked model sometimes surpasses that of the host model, which can be attributed to the data augmentation effect of the Mixup samples. The high recovery rates for $Re{c}_{Tr}$ and $Re{c}_{Ts}$ also suggest that the model is learning a secret manifold rather than a collection of isolated points. This eliminates the need for the verifier to use the original trigger samples for watermark embedding, making the watermarking method even more efficient.

4.4. Robustness

The watermark must also be robust against various attacks, such as fine-tuning, quantization, and pruning. Since RoSe-Mix operates in a black-box setting, its performance is evaluated using the standard recovery metrics: test accuracy (TA), training set watermark recovery Rate ($Re{c}_{Tr}$), and test set watermark recovery rate ($Re{c}_{Ts}$). The recovery metric is commonly used in black-box watermarking methods, but we extend it to training and testing. Together, the recovery and the test accuracy metrics reflect the watermark retrieval success based on the model’s behavior (output to triggers) rather than its internal parameters. Unlike white-box watermarking, where bit error rate (BER) or weight perturbation analysis might be applicable, black-box watermarking relies on behavioral patterns observable from the model’s predictions. This ensures that the verification process remains practical and does not require access to the model’s internal structure. Additionally, we incorporate dynamic quantization, full unsigned 8-bit quantization, and full signed 8-bit quantization into our robustness evaluation to ensure a comprehensive assessment of RoSe-Mix against commonly used model compression techniques. We can also highlight that each watermarked model has a better TA compared to its version without a watermark. This effect is often observed in backdoor injection [15], which acts as learning with noisy-labeled samples [40].

We begin our analysis with Table 1, which shows that RoSe-Mix demonstrates high robustness to fine-tuning and quantization attacks, as test accuracy (TA) and watermark recovery rates ($Re{c}_{Tr}$ and $Re{c}_{Ts}$) remain almost unaffected across most datasets. For instance, in MNIST, FashionMNIST, and CIFAR-10, $Re{c}_{Ts}$ remains larger than 88% across all tested attack scenarios, highlighting that RoSe-Mix preserves the watermark even under significant post-processing. The slight increase in $Re{c}_{Tr}$ for FashionMNIST under quantization (by 0.8%) suggests that the Mixup augmentation process improves the generalization of the model rather than degrading its fidelity or performance of the primary task.

The impact of JPEG compression is more significant. JPEG compression at a quality factor of 55 leads to a performance drop of 2.58%, 6.21%, and 3.65% in TA for CIFAR-10, transfer learning, and GTSRB, respectively. Additionally, GTSRB experiences the largest $Re{c}_{Tr}$ and $Re{c}_{Ts}$ reductions (7.2% and 8.6%), indicating that lower-resolution distortions can interfere with the embedded watermark. However, even in this extreme scenario, the watermark recovery rate remains sufficiently high (>$\tau$), ensuring that ownership verification is still feasible.

The robustness of RoSe-Mix can be attributed to two primary factors:

a.

Mixup-based embedding: Unlike traditional backdoor-based watermarking schemes, RoSe-Mix does not rely on a fixed set of trigger inputs. Instead, Mixup continuously generates new trigger samples, preventing an adversary from isolating and removing the watermark through fine-tuning or pruning. This also explains why $Re{c}_{Ts}$ remains significantly higher than TA under aggressive pruning—the watermark is encoded as a distributed manifold rather than discrete patterns.

b.

Cryptographic hashing mechanism: The use of one-way hash functions ensures that, even if an attacker attempts to regenerate trigger samples, they cannot feasibly recover the correct key-label mappings without access to the private key. This is evident from the minimal USR values in

Table 2. USR of RoSe-Mix compared to Mixer.

Dataset	USR (%)
	Mixer	RoSe-Mix
MNIST	51.1	20.78
FashionMNIST	39.3	15.32
Cifar10	38.4	15.0
Transfer Learning	39.5	14.7
GTSRB	12.2	4.69

, showing that unauthorized attempts to extract the watermark result in high failure rates.

Pruning evaluations (Figure 6) further validate RoSe-Mix’s robustness. While MNIST and FashionMNIST experience 30% and 90% TA reductions at a pruning rate of $0.4$, their respective watermark recovery rates remain higher than 80% for both. In contrast, CIFAR-10 suffers greater TA drops (to 55% and 40%) at pruning rates of 0.1, yet $Re{c}_{Ts}$ remains higher than or comparable to TA. This suggests that while pruning reduces the model’s expressiveness, it does not completely remove the watermark. The interplay between Mixup-generated triggers and deep feature representations ensures that the watermark remains embedded even when substantial portions of the network are removed.

Finally, any drop in $Re{c}_{Tr}$ and $Re{c}_{Ts}$ closely follows a drop in the accuracy of the main task of the model, ensuring that an attacker cannot remove the watermark without significantly deteriorating the usability of the model. This property makes RoSe-Mix highly resistant to watermark removal attacks, since the attacker must sacrifice the core performance of the model to erase its ownership markers.

4.5. Security

Let us now consider a scenario where a usurper attempts to steal the watermarked model by forging a new trigger set, assuming they know the algorithm. To do this, the usurper must first generate the correct values for $\mathit{\lambda }$ and $\mathit{\mu }$, and then craft adversarial examples and fine-tune the model on them. For the sake of analysis, we make the unrealistic assumption that the usurper knows several factors: the statistical distribution used to generate $\mathit{\lambda }$ and $\mathit{\mu }$, as well as the watermark generation algorithm—though without knowing the exact values of the parameters. This allows us to evaluate the usurper’s performance in the extreme worst-case scenario. We then measure the usurper success rate (USR), which is the watermark recovery rate of the usurper over 1000 random fake keys, with the results presented in Table 2.

Compared to Mixer [25], the usurper success rate (USR) in RoSe-Mix is approximately 2.45 times lower. This is due to the increased difficulty in guessing the multiple secrets involved in the proposed method, as reflected by the higher amount of work required in RoSe-Mix, expressed in Equation (12), which exceeds the effort needed for Mixer or RoSe alone. Even when the usurper is unrealistically granted several secret factors, the watermark recovery rate remains low, making it challenging to influence the verifier’s decision regarding model ownership.

5. Conclusions

In this paper, we introduced RoSe-Mix, a black-box DNN watermarking technique that effectively combines the security of cryptographic hashing from RoSe with the robustness of image Mixup from Mixer. The use of cryptographic hashing ensures the security of the watermark, while the Mixup-based trigger generation enhances robustness against various attacks, such as pruning, quantization, and adversarial modifications. We demonstrated that the increased computational complexity for attackers attempting to break the watermarking scheme makes RoSe-Mix more secure than previous methods. Additionally, the USR is reduced by at least 2.5× compared to Mixer. Our experimental results, conducted on architectures such as ResNet and VGG using datasets like CIFAR-10 and ImageNet, show the effectiveness of RoSe-Mix, highlighting its potential as a practical solution for protecting intellectual property in deep-learning models. We have demonstrated that our method can achieve at least 100% Rec with degradation in TA of 10% under the most severe attack. Future work could explore the application of RoSe-Mix to other domains, such as natural language processing or time series analysis. Additionally, investigating the performance of RoSe-Mix under more advanced attack scenarios, like model extraction or evasion attacks, could further validate its security properties.