Identity Diffuser: Preserving Abnormal Region of Interests While Diffusing Identity

Abstract

To release medical images that can be freely used in downstream processes while maintaining their utility, it is necessary to remove personal features from the images while preserving the lesion structures. Unlike previous studies that focused on removing lesion structures while preserving the individuality of medical images, this study proposes and validates a new framework that maintains the lesion structures while diffusing individual characteristics. In this framework, we apply local differential privacy techniques to provide theoretical guarantees of privacy protection. Additionally, to enhance the utility of protected medical images, we perform denoising using a diffusion model on the noise-contaminated medical images. Numerous chest X-rays generated by the proposed method were evaluated by physicians, revealing a trade-off between the level of privacy protection and utility. In other words, it was confirmed that increasing the level of personal information protection tends to result in relatively lower utility. This study potentially enables the release of certain types of medical images that were previously difficult to share.

1. Introduction

To enable researchers to handle medical images more securely, it is necessary to have a framework in place that ensures patients cannot be identified from these medical images even if they are leaked. To achieve this, strong noise can be added to the images. However, such anonymization processes can render the medical images useless, protecting privacy but preventing meaningful analysis from being conducted. There are no guidelines on how strong the added noise should be, and there is no theoretical guarantee of privacy protection.

On the other hand, differential privacy technology [1,2], which has been gaining attention and is increasingly being implemented in society, is believed to be capable of realizing such a framework by providing theoretical guarantees and controllable parameters (i.e., privacy budget) on the strength of privacy protection. Differential privacy includes global differential privacy, which trusts the administrator (in this case, the medical image analyst), and local differential privacy (LDP), which does not even trust the administrator. This study extends the application of LDP. The medical images obtained through the algorithm in this study provide a theoretical guarantee on the upper bound of the probability of identifying an individual, particularly in the regions outside the lesions, even if the images are leaked to external parties. Indeed, although the application of differential privacy to multidimensional data such as images is still in its early stages, this study proposes a novel method that contributes to such pioneering efforts. Until now, it has been challenging to generate medical images that maintain utility with a realistic privacy budget. However, by adopting the technology developed by Shibata et al. [3] in previous studies, which combines LDP techniques with diffusion models [4], this study aims to overcome this issue.

Previous studies are typically concentrated on retaining the identity of the subject (e.g., a patient) in an image while removing anomalous regions. This is very important, especially in medical image analysis, because obtaining the counterfactual (fake) normal image can yield an anomaly map by taking a difference between the real anomalous image and the fake normal image. On the contrary, in this study, we formulate and validate a framework that retains anomalous regions while diffusing (i.e., altering) the identity of patients using LDP techniques and diffusion models. This is the key highlight of our research. We call this framework an Identity Diffuser. This is very challenging compared with the above-mentioned “normalizer” because one must estimate almost all the body structure from an anomalous region, where typically much lesser information is contained compared with the other region.

The remainder of this paper is structured as follows. Section 2 reports related research. Section 3 formulates the proposed method and the experimental validation approach. Section 4 presents the medical images generated by the proposed method and the experimental results. Section 5 discusses the results.

2. Related Works

The present study is categorized as a special case of inpainting. Inpainting is formulated to guess unknown masked region(s) in an image. Bertalmio et al. [5] adopted the continuous diffusion equation to interpolate the masked region. This method is a mathematical model. Lugmayr et al. [6] adopted denoising diffusion probabilistic models [4] to recover the masked region using an image prior knowledge (RePaint algorithm). This method is a data-driven model. The keyword "diffusion" is common and interesting, but the algorithms are significantly different between the two. In this study, we adopt the RePaint algorithm because we can easily integrate it with the Gaussian local differential privacy algorithm explained in the next section. The RePaint is an algorithm that performs inpainting in the image space, but there are also inpainting algorithms that operate in the latent space, specifically for latent space diffusion models [7]. Tran et al. [8] applied deep learning-based inpainting techniques to CXR images, but it was not in the context of privacy protection, and differential privacy techniques were not utilized in their work. In a broader framework than the normalization of lesion structures, some studies have applied defect repair techniques to medical imaging [9,10,11].

Although the application target is facial images rather than medical images, several studies have adopted local differential privacy algorithms to erase image features [12,13]. Regarding the application to medical images, Shibata et al. have recently employed flow-based deep generative models [14] or diffusion models [3] to formulate and validate the approach. It is important to note that these approaches are different from DP-SGD [15], which applies differential privacy to the gradient information during the backpropagation process in neural network training.

3. Methods

Figure 1 shows the flowchart of the proposed method. Identity Diffuser is built based on the framework RePaint [6], which is an inpainting algorithm using diffusion models. In this study, we further combine the theory of local differential privacy (LDP) with the RePaint algorithm.

Specifically, on a supercomputer (using a single NVIDIA A100 GPU, (NVIDIA, Santa Clara, CA, USA )), we trained a diffusion model on 7808 “normal” CXR images. The batch size was set to 8, and with gradient accumulation set to 2, this effectively emulates a batch size of 16. The learning rate was set to $8\times {10}^{-5}$, and the Adam optimizer was employed. The CXR images had a resolution of 256 × 256, and FP16 mixed precision computation was utilized, but Flash Attention was not used. This was executed for 180,000 steps, equivalent to approximately 369 epochs of training. The training process took about 18 h. For inference, the model data from the 180,000th step was loaded. The RePaint algorithm was employed for inpainting, with 12 “internal iterations” executed (for details on the internal iterations, please refer to the RePaint paper [6]). The components of the proposed algorithm are detailed below.

3.1. Diffusion Models

Diffusion models [4] gradually add noise to data such as images and then model the reverse process (denoising) using deep neural networks, typically UNets [16]. The diffusion model is mathematically equivalent to indirectly minimizing the Kullback–Leibler divergence between the distribution formed by the true images and the modeled distribution. We represent this UNet by the vector function $\mathit{f}({\mathit{x}}_t,t)$, where ${\mathit{x}}_t$ is the input image and t is the denoising time step, which represents the noise intensity in the input image. Instead of restoring meaningful images directly from completely random vectors, the process gradually approaches meaningful images, i.e., ${\mathit{x}}_{t-1}\leftarrow \mathit{f}({\mathit{x}}_t,t)$ and ${\mathit{x}}_0$ is a completely denoised image (the generated image). There are numerous proposed methods for this approach, with Denoising Diffusion Implicit Models [17] being relatively well-known. Regarding the training of denoising, it is usually set up to predict the original image (or the noise itself) from images with various noise intensities. These images with different noise intensities are uniquely identified by the denoising timesteps t. The maximum value of the denoising timesteps (i.e., the number of different noise intensity data distributions prepared) is often taken to be between 200 and 1000. The settings for noise intensity at each step (noise scheduling) also vary, with sigmoid scheduling, which we adopted in this study, and linear scheduling being representative examples.

3.2. Local Differential Privacy

The local differential privacy (LDP) with the Gaussian mechanism [1,18] is formulated by

$$Pr[M\left(\mathit{x}\right)=\tilde{\mathit{x}}]\le e^{ϵ}Pr[M\left({\mathit{x}}^{\prime }\right)=\tilde{\mathit{x}}]+\delta ,$$

(1)

where $\mathit{x}$ and ${\mathit{x}}^{\prime }$ are two “neighborhood” images (here, CXR images), $\tilde{\mathit{x}}$ is the LDP-processed image. M is a probabilistic algorithm, and Pr is the probability that $\tilde{\mathit{x}}$ is obtained when we input $\mathit{x}$ or ${\mathit{x}}^{\prime }$ to the algorithm. In this study, we very loosely define the neighborhood: for a fixed CXR image $\mathit{x}$, the neighborhood image ${\mathit{x}}^{\prime }$ can take any image in the image space. This definition potentially results in a large privacy budget if we retain the utility of LDP-processed images.

In LDP for images [3,12,14,19], noise is added to the image. Specifically, Gaussian noise is added by adopting the Gaussian mechanism. This can be mathematically expressed as:

$$\tilde{\mathit{x}}=\mathit{x}+\mathcal{N}(0,{\sigma }^2),$$

(2)

where $\tilde{x}$ is the noisy image, x is the original image, and $\mathcal{N}(0,{\sigma }^2)$ represents Gaussian noise with mean 0 and variance ${\sigma }^2$. When noise is added as described, the processed image satisfies the above equation. For the theoretical background on the setting of variance, please refer to previous research [3].

3.3. Inpainting Using Diffusion Models: RePaint

RePaint [6] is a missing region restoration (inpainting) algorithm that utilizes diffusion models. The inputs to the algorithm are a binary mask image indicating the areas to be restored and the original image. In the original RePaint algorithm, it is assumed that the restoration areas are completely noisy (corresponding to a privacy budget of 0 in LDP). However, it should be noted that in this study, it is not necessarily assumed that the noise is complete.

3.4. Post Processing for LDP with Diffusion Models

Shibata et al. [3] proposed associating this noise addition process with LDP, specifically suggesting that adding noise in the diffusion process corresponds to limiting the privacy budget. More specifically, it becomes possible to back-calculate the privacy budget assumed by the Gaussian mechanism from the variance of the noise added to the image in the forward process. They found that the denoising timesteps in diffusion models correspond to different privacy budgets. Additionally, Shibata et al. [3] found that denoising in diffusion models dramatically improves the quality, or utility, of images that satisfy LDP by having noise added.

3.5. Identity Diffuser (Proposed Framework)

First, the diffusion model was trained using only normal CXR images extracted from the RSNA Pneumonia detection dataset [20] to ensure that no new lesions are generated during the medical image creation process. Specifically, we trained a diffusion model with 7808 normal CXRs from the RSNA dataset. The resolution of the CXR images was downsampled to 256 × 256 by averaging the pixel values. Second, we prepared CXR images with the boundary box, which localizes anomalous regions. These bounding boxes were adapted from the annotations that accompanied the NIH CXR dataset, which served as the basis for the RSNA dataset. These CXR images were processed as follows: In the boundary box, we retained all the information, i.e., the pixel intensities. Outside the boundary box, we added Gaussian noise based on privacy budget ($ϵ$ and $\delta$, $ϵ$-$\delta$ local differential privacy). Third, we ran the RePaint algorithm based on the trained model and obtained denoised CXR images. The implementation of the diffusion model was adopted from an open repository [21] for 2D image applications, with slight modifications, and incorporated the RePaint algorithm. Lastly, three radiologists individually investigated the denoised CXR images together with the original images and evaluated scores.

To summarize, we have the LDP-processed image $\tilde{\mathit{x}}$, which satisfies

$$Pr[M\left(\mathcal{L}\left(\mathit{x}\right)\right)=\mathcal{L}\left(\tilde{\mathit{x}}\right)]\le e^{ϵ}Pr[M\left(\mathcal{L}\left({\mathit{x}}^{\prime }\right)\right)=\mathcal{L}\left(\tilde{\mathit{x}}\right))]+\delta ,$$

(3)

where the operator $\mathcal{L}$ extracts the normal region from the entire image.

The quantification experiments conducted by the physicians were performed on a single laptop using a custom-made application. This was a WindowsForm application developed using the .NET framework, which allowed for sequential image presentation and comparison. The physicians were able to systematically input the quality of each image.

There are three evaluation criteria. The first evaluation criterion is the overall naturalness of the CXR images. The second evaluation criterion assesses whether the embedded lesions appear natural when contrasted with other parts of the image. The third evaluation criterion is the degree of identity suppression. Each criterion is rated on a scale from 1 to 5, where 1 indicates the least natural (least suppressed) and 5 indicates the most natural (most suppressed). These criteria were assessed using lesions from ten different diagnoses, two different cases, three distinct sampling results, and four different privacy budgets. This means that 240 CXR images were evaluated per person. The evaluation results from three physicians were averaged, resulting in $3\times 240=720$ values.

4. Results

Figure 2 shows the results of CXR image processing using the proposed method.

Table 1. The relationship between privacy protection and utility. Represent the count of the corresponding images on a 2D histogram ($ϵ=4.12\times {10}^6$, $\delta ={10}^{-4}$). U represents utility, and P stands for privacy protection. The larger the integer values following these letters, the stronger those characteristics.

	U1	U2	U3	U4	U5
P1	0	10	27	0	0
P2	1	12	8	0	0
P3	0	1	1	0	0
P4	0	0	0	0	0
P5	0	0	0	0	0

,

Table 2. The relationship between privacy protection and utility. Represent the count of the corresponding images on a 2D histogram ($ϵ=4.43\times {10}^4$, $\delta ={10}^{-4}$). U represents utility, and P stands for privacy protection. The larger the integer values following these letters, the stronger those characteristics.

	U1	U2	U3	U4	U5
P1	0	0	2	0	0
P2	0	3	8	0	0
P3	0	18	9	0	0
P4	0	12	8	0	0
P5	0	0	0	0	0

and

Table 3. The relationship between privacy protection and utility. Represent the count of the corresponding images on a 2D histogram ($ϵ=0$). U represents utility, and P stands for privacy protection. The larger the integer values following these letters, the stronger those characteristics.

	U1	U2	U3	U4	U5
P1	0	0	0	0	0
P2	1	5	5	0	0
P3	0	12	9	0	0
P4	0	16	12	0	0
P5	0	0	0	0	0

show the relationship (the raw data) between privacy protection and utility for CXR obtained from this experiment. Utility was evaluated by averaging the results of the first and second evaluation criteria. And the privacy protection metric was directly taken from the results of the third evaluation criterion.

5. Discussion

It can be observed that as the privacy budget increases, the strength of privacy protection decreases and utility slightly increases. Even when the privacy budget is at its maximum, the utility does not reach its peak, indicating room for improvement in the extrapolation algorithm. On the other hand, even when the privacy budget is at its minimum ($ϵ=0$), the strength of privacy protection does not reach its maximum (P5), revealing that individuals can still be somewhat identified based on the lesions alone.

This method qualitatively differs from traditional techniques in the way it embeds lesions. It is particularly useful from the perspective of data augmentation in machine learning for rare diseases where only a limited number of images are available.

Although it slightly deviates from the main subject of this study, by strictly defining image adjacency, it may be possible to evaluate the value of the privacy budget more rigorously and generate images that maintain utility with a smaller privacy budget.

Since the lesions have not been altered at all, the disease completely remains visible to the human eye, but it may slightly affect the performance of diagnostic models (classification models). The examination of the impact of the proposed method on the performance of lesion classification models is designated for future research.

Finally, methods for supplementing missing parts in medical images, such as chest X-ray images, have already been developed (e.g., [8]). However, to the best of the authors’ knowledge, there are no prior studies that deal with methods for removing individuality while preserving lesion structures, as in this study. Therefore, a direct comparison with prior studies in the strict sense is not possible.