FreeMix: Personalized Structure and Appearance Control Without Finetuning

Abstract

Personalized image generation has gained significant attention with the advancement of text-to-image diffusion models. However, existing methods face challenges in effectively mixing multiple visual attributes, such as structure and appearance, from separate reference images. Finetuning-based methods are time-consuming and prone to overfitting, while finetuning-free approaches often suffer from feature entanglement, leading to distortions. To address these challenges, we propose FreeMix, a finetuning-free approach for multi-concept mixing in personalized image generation. Given separate references for structure and appearance, FreeMix generates a new image that integrates both. This is achieved through Disentangle-Mixing Self-Attention (DMSA). DMSA first disentangles the two concepts by applying spatial normalization to remove residual appearance from structure features, and then selectively injects appearance details via self-attention, guided by a cross-attention-derived mask to prevent background leakage. This mechanism ensures precise structural preservation and faithful appearance transfer. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior structural consistency and appearance transfer compared to existing approaches. In addition to personalization, FreeMix can be adapted to exemplar-based image editing.

1. Introduction

Recent advancements in text-to-image diffusion models, such as Stable Diffusion [1] and DALL·E2 [2], have enabled the generation of both realistic and imaginative visual content. Building upon these successes, personalized image generation has emerged as an active area of research, where the goal is to incorporate user-specific objects or styles into generated images. In this domain, some methods [3,4,5,6] achieve personalization by finetuning certain parameters within diffusion models when given a collection of images of a single user-specific subject. Recently, finetuning-free approaches [7,8] have emerged, which manipulate attention mechanisms to enable the customization of multiple objects or transfer of style concepts.

Personalized image generation holds significant value across a wide range of creative and industrial applications. In digital art and game design, creators often need to apply new styles to characters while preserving their underlying structure. In e-commerce, a single product may need to be visualized in multiple appearances without reconstructing its geometry. Similarly, in virtual reality and social media, users may wish to preserve the shape of their avatars while experimenting with diverse appearances. Addressing these needs requires precise disentanglement of structural and appearance features, which is crucial for expanding the practical utility and accessibility of personalized image generation.

Despite the growing progress in this field, existing methods still face critical limitations. Finetuning-based approaches are computationally expensive and prone to overfitting, often resulting in the dominance of a single identity. Text embedding-based techniques, such as Extended Textual Inversion (XTI) [6], attempt to achieve style mixing but suffer from cross-attention misalignment, leading to inconsistencies between the generated image and the reference. Furthermore, current approaches struggle to independently manipulate structural and appearance attributes, since attention misalignment often entangles object features with background or other visual elements during disentanglement. These challenges highlight the need for a more effective method that enables fine-grained control over structure–appearance mixing.

To address these challenges, we propose FreeMix, a finetuning-free method for multi-concept mixing in personalized image generation, as illustrated in Figure 1. Unlike prior finetuning-free approaches, FreeMix achieves fine-grained control over structure and appearance by extracting detailed structural and appearance features directly through attention. We introduce Disentangle-Mixing Self-Attention (DMSA), which integrates diffusion features across different layers and time steps. DMSA consists of two core processes: structure generation and appearance transfer. Structure generation extracts the shape and spatial composition from the reference image, ensuring that the generated image retains the structural integrity of the reference object. To facilitate effective feature mixing, we apply spatial normalization to remove residual appearance information from the structure features. Appearance transfer leverages self-attention to construct an appearance attention. Then, this attention is incorporated into the structure features.

To prevent confusion between the structure and background, we utilize a mask extracted from cross-attention, ensuring that appearance transfer is applied only to the object. Unlike XTI, which relies on text embeddings, our method provides a novel mechanism to explicitly disentangle and recombine different semantic components within attention. This enables more precise control over spatial attributes and incorporates fine details that are difficult to describe with text prompts. FreeMix achieves robust disentanglement of structure and appearance, enabling more detailed and precise personalization than prior methods. Representative results comparing existing personalization methods with our approach are shown in Figure 2

We perform comprehensive experiments to evaluate the performance of our proposed framework. Experimental results demonstrate that our method consistently outperforms existing approaches both quantitatively and qualitatively.

Our contributions can be summarized as follows:

• We propose a method that generates a new image by combining the structure of a single object image and the appearance of an exemplar without requiring additional model training.
• We introduce DMSA, a self-attention mechanism that effectively disentangles and integrates structure and appearance features, enabling precise concept mixing.
• Our experimental results demonstrate that our method achieves superior performance in multi-concept mixing and exemplar-based image editing tasks compared to existing methods.

2. Related Work

2.1. Text-to-Image Generation

Text-to-image generation has advanced significantly, particularly with the emergence of diffusion models that outperform traditional GAN-based approaches [9,10]. Diffusion models [2,11,12,13] have demonstrated superior performance by effectively capturing fine-grained details and scaling to larger datasets. Latent Diffusion Models (LDMs) [1] enhance computational efficiency by operating in a compressed latent space while maintaining high-quality outputs. Building on LDM, notable models such as Stable Diffusion and SDXL [14] further improve the quality and flexibility of image generation, making them widely adopted in various applications. Moreover, ControlNet [15] extends text-to-image diffusion models by integrating additional conditions, enabling more precise control over generated images. Our work builds upon the pre-trained Stable Diffusion.

2.2. Personalized Image Generation

Personalized text-to-image diffusion models aim to render user-specific subjects from only a few images. These approaches can be broadly categorized into finetuning-based and finetuning-free approaches.

Finetuning-based methods [3,4,5,6] finetune a pre-trained diffusion model to capture specific visual concepts from reference images. This often involves modifying certain network parameters to bind a textual identifier with the subject of interest. Textual Inversion [3] encodes the subject into a learnable text embedding that represents its unique features in the latent space. DreamBooth [4] finetunes the entire diffusion model to associate a unique identifier with a given subject. Custom Diffusion [5] modifies only the cross-attention layers of the diffusion model, reducing computational costs compared to full model finetuning. XTI [6] extends textual inversion by introducing layer-wise learnable embeddings for improved representation. NeTI [16] incorporates implicit time-aware representations to enhance subject fidelity in generated images. However, these methods require extensive finetuning, which is time-consuming and computationally costly.

Finetuning-free approaches [7,17,18] eliminate the need for test-time adaptation. IP-Adapter [17] and ELITE [18] introduce auxiliary networks trained on large-scale datasets to extract reference image features. These features are then incorporated into the diffusion model, thus eliminating the need for test-time finetuning. FreeCustom [7] and StyleAligned [8] employ attention control mechanisms to achieve personalization without additional networks. FreeCustom enables multi-object composition through multi-reference self-attention, allowing better preservation of reference concepts. StyleAligned focuses on maintaining style consistency across generated images through minimal attention sharing during the diffusion process. This technique can be integrated with dreamBooth to generate images that blend different content and styles. While finetuning-based methods achieve strong personalization at the cost of efficiency and overfitting, finetuning-free approaches improve efficiency but suffer from feature entanglement, leading to unwanted leakage between structure and appearance. Moreover, prior works mainly focus on multi-object composition or style consistency, without explicitly addressing the challenge of disentangling and integrating structure and appearance from different references. This limitation motivates the need for a mechanism that can selectively and independently control these attributes in a finetuning-free manner.

2.3. Image Editing

Image editing [19,20,21,22,23] with text-to-image diffusion models has made remarkable progress through techniques that manipulate latent representations and attention mechanisms. Prompt-to-Prompt [19] edits images by modifying cross-attention maps, enabling targeted changes while preserving structure. MasaCtrl [20] improves consistency by converting self-attention into mutual self-attention, enhancing localized edits. Direct Inversion [23] disentangles source and target diffusion branches, ensuring both content preservation and edit fidelity with minimal computation. InstructPix2Pix [22] trains a diffusion model on instruction-based edits, allowing a wide range of modifications based on textual instructions. ZeST [24] proposes a novel method for transferring materials to objects in real images without requiring any additional training. This approach extracts material representations from an exemplar image using IP-Adapter. It then applies them to the target object using ControlNet for structure-aware transfer, incorporating an inpainting model. We achieve effective appearance editing without relying on adapters or finetuning by leveraging self-attention.

3. Preliminaries

3.1. Diffusion Models

Diffusion models [11,25,26] are a class of generative models that generate images by gradually removing noise from an initial Gaussian sample $x_T\sim \mathcal{N}(0,I)$. The process consists of two phases: a forward process that progressively adds noise to an image and a reverse process that gradually denoises it to reconstruct the original image. The forward process is as follows:

$$x_t=\sqrt{{\alpha }_t}{x}_0+\sqrt{1-{\alpha }_t}ϵ,ϵ\sim \mathcal{N}(0,I)$$

(1)

where $x_t$ is the noisy image at time step t, $x_0$ is the original image, and ${\alpha }_t$ controls the noise level. During inference, the reverse process employs a neural network ${ϵ}_{\theta }(x_t,t)$ to iteratively predict and remove the added noise. Text-to-image diffusion models extend this framework by conditioning the denoising process on a text prompt. Latent Diffusion Models (LDMs), such as Stable Diffusion, further optimize this process by performing denoising in a lower-dimensional latent space rather than pixel space.

3.2. Attention in Stable Diffusion

Stable Diffusion employs a U-Net architecture with attention mechanisms to guide image generation. Two key attention modules play a role: self-attention and cross-attention.

3.2.1. Self-Attention

Self-attention captures long-range dependencies within an image, allowing spatial features to interact across the image. Self-attention is formulated as:

$$\mathrm{h}=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V,$$

(2)

where $Q,K,V$ are the query, key, and value features derived from image features, and $d_k$ is the dimension of the key vectors. In Stable Diffusion, self-attention refines the spatial structure of the image, controlling layout and details across diffusion steps and layers.

3.2.2. Cross-Attention

Cross-attention enables text conditioning by aligning image features with the text embedding derived from the input prompt. Given a text prompt embedding c, cross-attention is computed by:

$${\mathrm{A}}_{cross}=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}_c^T}{\sqrt{d_k}}}\right),$$

(3)

$${\mathrm{h}}_{cross}={\mathrm{A}}_{cross}{V}_c,$$

(4)

where $K_c$ and $V_c$ are the key and value features derived from the text embedding c. ${\mathrm{A}}_{cross}$ represents the cross-attention map that determines how much influence each text token has on different spatial locations in the image. The cross-attention mechanism ensures that the generated image aligns with the semantics of the given prompt by modulating spatial features accordingly.

Our method modifies self-attention to refine structure and appearance separately, enabling effective mixing without requiring additional networks or finetuning.

4. Methodology

Our goal is to combine structural and appearance concepts in a zero-shot manner, without requiring finetuning or adapters. Given a structure image $x^s$ and an appearance image $x^a$, FreeMix replaces the original self-attention module of Stable Diffusion with Disentangle-Mixing Self-Attention (DMSA) to generate an output image $x^g$ that incorporates the structural and appearance features from reference images. Our framework is illustrated in Figure 3.

4.1. Disentangle-Mixing Self-Attention

DMSA is a mechanism to selectively transfer structural and appearance features from different reference images. We first use the Segment Anything Model (SAM) [27] to extract masks $M_s$ and $M_a$ of the concepts from each reference image while filtering out irrelevant background elements. Starting with Gaussian noise $x_T^g\sim \mathcal{N}(0,\mathrm{I})$, we iteratively denoise it to generate the final output $x^g$. At each time step t, we obtain the noisy representations $x_t^s$ and $x_t^a$ of the structure image $x^s$ and the appearance image $x^a$ through the forward process in Equation (1). Then, DMSA extracts the key and value of the self-attention layer from each reference image and injects them into the attention mechanism of the generated image. Since the diffusion features control distinct attributes, we apply structure generation and appearance transfer at appropriate layers and time steps. Additionally, we employ a masking strategy to ensure that only targeted attributes are transferred.

4.2. Structure Generation

We adopt self-attention injection and a weighted mask. Specifically, at time step t and U-Net layer l, let the query, key, and value of $x_t^g$ be $Q_{l,t}^g,K_{l,t}^g,V_{l,t}^g$, and those of the structure image $x_t^s$ be $Q_{l,t}^s,K_{l,t}^s,V_{l,t}^s$. We compute the dot products $Q_{l,t}^g{K}_{l,t}^{gT}$ and $Q_{l,t}^g{K}_{l,t}^{sT}$, and then concatenate them, resulting in $S=[Q_{l,t}^g{K}_{l,t}^{gT},Q_{l,t}^g{K}_{l,t}^{sT}]$. Similarly, we concatenate values as $V^{\prime }=[V_{l,t}^g,V_{l,t}^s]$. To integrate structural features, we apply a weighted structure mask $M_s$ extracted from SAM, which filters out irrelevant background regions. The final output is computed as:

$${\mathrm{h}}_{l,t}=\mathrm{Softmax}\left({\displaystyle \frac{M\odot S}{\sqrt{d_k}}}\right)V^{\prime }$$

(5)

where $M=[1,w{M}_s]$ is the weighted mask, w is the scaling factor and ⊙ represents the Hadamard product.

This process integrates structural features from the reference image into the generated output. However, as shown in Figure 4, this method suffers from appearance leakage, in which appearance features of the structure image are unintentionally transferred into the generated results.

To address this, we apply spatial normalization. Inspired by prior works that define appearance as feature statistics [8,28], we consider appearance transfer to be a stylization task. In contrast to style transfer methods, which align feature statistics to inject appearance, we instead remove these statistics. Specifically, we apply spatial normalization to $Q_{l,t}^g$ and $K_{l,t}^s$ for time steps $t>\tau$.

$$S=\left\{\begin{array}{c}[Q_{l,t}^g{K}_{l,t}^{gT},Q_{l,t}^g{K}_{l,t}^{sT}]t<\tau \hfill \\ [Q_{l,t}^g{K}_{l,t}^{gT},norm\left(Q_{l,t}^g\right)norm\left(K_{l,t}^{sT}\right)]t\ge \tau \hfill \end{array}\right.$$

(6)

As shown in Figure 5, we visualize the self-attention maps of the U-Net middle blocks across different time steps after normalization and observe that structural information is preserved.

Furthermore, we perform this process starting from the earliest time steps and at the middle layers of the U-Net to focus solely on shape transfer.

4.3. Appearance Transfer

At time step t and U-Net layer l, let the query, key, and value of $x_t^a$ be $Q_{l,t}^a,K_{l,t}^a,V_{l,t}^a$. We first extract the appearance-related components from $K_{l,t}^a$ and $V_{l,t}^a$ using an appearance mask $M_a$. This mask isolates appearance features and prevents irrelevant elements from interfering with the generated output. A naive approach is to directly replace $K_{l,t}^g$ and $V_{l,t}^g$ with $K_{l,t}^a$ and $V_{l,t}^a$, but this results in artifacts where background and foreground features become entangled. To overcome this problem, we utilize cross-attention maps to create a foreground object mask $M_g$ from $x_t^g$. Specifically, at time step t, we obtain the cross-attention maps of $x_t^g$ at a spatial resolution of $16\times 16$. These maps capture the structure of $x_t^s$ during structure generation. We then extract the cross-attention map corresponding to the foreground object token and use it as $M_g$. We then perform feature mixing of the key and value as follows:

$${\widehat{K}}_{l,t}^g=K_{l,t}^g·(1-M_g)+K_{l,t}^a·M_g$$

(7)

$${\widehat{V}}_{l,t}^g=V_{l,t}^g·(1-M_g)+V_{l,t}^a·M_g$$

(8)

${\widehat{K}}_{l,t}^g$ and ${\widehat{V}}_{l,t}^g$ are used in the self-attention operation with $Q_{l,t}^g$, ensuring that appearance is transferred only to the appropriate spatial regions. Unlike the appearance mask $M_a$ obtained from SAM, the cross-attention mask $M_g$ is used to localize the foreground regions in the generated image, ensuring that appearance transfer is applied only to the target object while preserving the background. The final output is obtained:

$${\mathrm{h}}_{l,t}=\mathrm{Softmax}\left({\displaystyle \frac{Q_{l,t}^g{\widehat{K}}_{l,t}^{gT}}{\sqrt{d_k}}}\right){\widehat{V}}_{l,t}^g,$$

(9)

This attention control selectively transfers appearance features to the foreground while preserving the structural integrity of the generated image. We perform this process at later time steps $t>t_{appear}$ and deeper layers in the U-Net, ensuring that appearance features are effectively injected into the generated image.

4.4. Application to Image Editing

Our method can be easily extended to exemplar-based image editing. In this paper, exemplar-based editing involves transferring the appearance of a reference object onto a specific object in the structure image, while preserving the overall structure and background. As shown in Figure 6, similarly to previous image editing approaches, we use DDIM inversion [26] to obtain the latent noise of the structure image, resulting in $x_T^g$ as the initial point for editing. Unlike personalization, which starts from randomly sampled Gaussian noise, we use $x_T^g$ and denoise it while applying DMSA to control structural and appearance attributes. In Section 4.3, appearance transfer utilizes the foreground object mask derived from the cross-attention map. However, in image editing, the structure of the object remains fixed in the result image. Therefore, we directly utilize the structure mask $M_s$ as the foreground object mask $M_g$. In contrast to [24,29], who utilize an inpainting model to retain background details, we adopt a masked blending strategy. At each time step t, after denoising $x_t^s$ and $x_t^g$ to $x_{t-1}^s$ and $x_{t-1}^g$, we blend these representations using $M_s$, ensuring that the background remains unchanged:

$$x_{t-1}^{*}=x_{t-1}^g·M_s+x_{t-1}^s·(1-M_s)$$

(10)

Our method enables precise and controllable exemplar-based image editing without finetuning or auxiliary networks.

5. Experiments

5.1. Experiment Setting

5.1.1. Dataset

To evaluate our approach, we construct a dataset consisting of 534 diverse structure–appearance pairs and their corresponding target text prompts. The images in the dataset are collected from previous studies [4,5,7] on personalization. In addition, to enrich evaluation and avoid ambiguity, each structure–appearance pair is combined with three diverse prompts, resulting in a total of 1602 generated images. This design provides both diversity and textual variation, ensuring a more comprehensive and reliable evaluation. More details are provided in Appendix A.

5.1.2. Evaluation Metrics

Following previous studies, we evaluate the performance of our model on personalization and exemplar-based image editing. For personalization, we measure image fidelity and text fidelity. Image fidelity is assessed using CLIP-I [30] and DINOv2 [31]. Specifically, DINOv2 measures structural similarity between the structure image and the generated image. CLIP-I measures the consistency of appearance details by comparing the generated images with the appearance reference. To assess text fidelity, we employ CLIP-T [30], which evaluates how well the generated images align with the given text prompts. In addition to these metrics, we also adopt the prior preservation metric (PRES) and the diversity metric (DIV), inspired by prior work [4]. PRES is computed by measuring the average pairwise DINO embeddings between generated images of random subjects of the prior class and real images of our specific subject. A higher PRES score indicates that random prior-class subjects are more similar to the specific subject, reflecting a collapse of the prior. DIV is computed as the average LPIPS cosine similarity between generated images of the same subject under the same prompt, where higher diversity indicates less overfitting and more varied outputs. These additional metrics provide a more comprehensive evaluation of generation quality by assessing both prior preservation and sample diversity. For exemplar-based image editing, we utilize three key metrics: structure distance (StruD) [32], LPIPS [33], and CLIP-I. StruD quantifies the structural consistency between the original image and the edited image. A smaller StruD indicates better structure preservation. LPIPS measures background preservation by computing perceptual similarity in non-edited areas, similar to [23]. Finally, CLIP-I evaluates the appearance similarity between the appearance exemplar and the edited result in the CLIP space.

5.1.3. Compared Methods

We compare our approach with several methods for multi-concept mixing, including XTI [6], DreamBooth [4], Custom Diffusion [5], FreeCustom [7], and StyleAligned [8]. We include StyleAligned as a baseline because there are few existing methods for mixed structure and appearance personalization, and style transfer is closely related to our task in integrating visual attributes. StyleAligned performs finetuning of the structure image based on DreamBooth-LoRA for mixing. Additionally, we compare our method with MasaCtrl [20], Direct Inversion (DirectINV) [23], and ZeST [24] for exemplar-based image editing. MasaCtrl and DirectINV utilize DreamBooth-based finetuning of the appearance exemplar to enable reference-guided editing.

5.1.4. Implementation Settings

We use Stable Diffusion V1.5 as our base model and perform sampling with 50 denoising steps. For structure generation, we set the mask weight w to 3, $\tau$ to 35, and $t_{appear}$ to 20. Stable Diffusion consists of 16 self-attention layers (indexed from 0 to 15). We replace layers 8–15 with DMSA, where layers 8–12 are used for structure generation and layers 13–15 are used for appearance transfer. All experiments are conducted on a single A5000 GPU.

5.2. Multi-Concept Mixing

5.2.1. Quantitative Comparisons

As shown in

Table 1. Quantitative comparison of multi-concept mixing methods. Best and second best metrics are highlighted in bold and underline, respectively.

Method	Base Model	CLIP-I ↑	DINO ↑	CLIP-T ↑	PRES ↓	DIV ↑
XTI	SDv1.5	0.5425	0.6827	24.57	0.6745	0.3394
DreamBooth	SDv1.5	0.6110	0.6393	23.58	0.5721	0.3734
Custom Diffusion	SDv1.5	0.6133	0.7595	25.65	0.5583	0.3876
FreeCustom	SDv1.5	0.6220	0.7691	27.32	0.4936	0.3947
StyleAligned	SDv1.5	0.6034	0.7109	21.48	0.5636	0.3681
StyleAligned	SDXL	0.6260	0.7245	22.01	0.4803	0.3846
Ours	SDv1.5	0.6377	0.7605	27.42	0.4869	0.4038

, our method consistently outperforms existing methods in multi-concept mixing across most evaluation metrics. In terms of image fidelity, our approach achieves the highest CLIP-I and a competitive DINO, demonstrating its ability to preserve the structure and appearance of reference images effectively. For text fidelity, FreeMix achieves a higher CLIP-T compared to prior works. These results highlight the effectiveness of our method in achieving multi-concept mixing without finetuning. In addition, FreeMix obtains a lower PRES score, indicating that our method preserves the ability to generate diverse images of the prior class. At the same time, FreeMix achieves a higher DIV score, showing that it produces more diverse outputs without overfitting to specific reference contexts.

5.2.2. Qualitative Comparisons

We present visual comparisons of multi-concept mixing in Figure 7. XTI, DreamBooth, and Custom Diffusion struggle to effectively integrate the structure and appearance because they rely on text embeddings or finetuning. This approach limits precise control over individual features, making it difficult to properly fuse structure and appearance. FreeCustom and StyleAligned achieve partial success in mixing, but suffer from issues where the appearance is applied not only to the object but also to the background, leading to unintended artifacts. Additionally, these methods exhibit lower prompt fidelity. In contrast, our proposed method successfully achieves a balanced mixture of structure and appearance.

5.3. Exemplar-Based Image Editing

5.3.1. Quantitative Comparisons

Table 2. Quantitative comparison of exemplar-based image editing methods. Best and second best metrics are highlighted in bold and underline, respectively.

Method	Base Model	StruD ↓	LPIPS ↓	CLIP-I ↑
MasaCtrl *	SDv1.5	0.0350	0.1435	0.7119
DirectINV *	SDv1.5	0.0335	0.1273	0.7353
ZeST	SDXL	0.0390	0.0734	0.7519
Ours	SDv1.5	0.0245	0.0712	0.8101

* use DreamBooth-based finetuning for the appearance reference.

presents the quantitative results of image editing methods. Our method achieves significant improvements across all metrics, without finetuning. In particular, the notable improvement in CLIP-I demonstrates its superior performance in appearance transfer.

5.3.2. Qualitative Comparisons

As illustrated in Figure 8, we present a qualitative comparison of our method with baseline approaches for exemplar-based image editing. MasaCtrl struggles with appearance transfer and often introduces undesired artifacts in the background. DirectINV improves background preservation compared to MasaCtrl but still fails to effectively edit the appearance. ZeST effectively transfers the appearance of the exemplar, but structural details tend to be lost, such as the head of the robot (first row). Furthermore, ZeST relies on IP-Adaptor, which lacks the capability to extract specific regions from an image. As a result, foreground and background elements become entangled during transfer, such as projecting fire onto the cake (second row). In contrast, our approach achieves superior performance in exemplar-based appearance editing without requiring additional modules and finetuning.

5.4. Ablation Study

We replace the self-attention layers 8–15 of Stable Diffusion with DMSA, where layers 8–13 are responsible for structure generation and layers 13–15 handle appearance transfer. The appearance transfer operation is applied starting from $t_{appear}=20$. To evaluate the impact of different configurations, we conduct experiments by varying the layer replacement and $t_{appear}$. As shown in Figure 9, when structure generation is applied only to layers 8–10 while appearance transfer is applied to the remaining layers 11–15, the generated images exhibit low fidelity to the original structure. In contrast, when structure generation extends to layers 8–14, the images retain a high structural fidelity. However, in this case, appearance transfer becomes ineffective. When $t_{appear}$ increases beyond 20, the structure fidelity starts to degrade, leading to artifacts and distortions. This indicates that applying appearance transfer too late in the denoising process disrupts spatial coherence, leading to unintended structural inconsistencies. These results show that an optimal balance between structure fidelity and effective appearance transfer is achieved by applying structure generation in layers 8–13, appearance transfer in layers 13–15, and setting $t_{appear}=20$.

To further evaluate the practical advantages of our finetuning-free approach, we also report inference time comparisons in

Table 3. Inference time comparison of different multi-concept image generation methods.

	XTI	DreamBooth	Custom Diffusion	FreeCustom	StyleAligned	Ours
Generation Time (s)	1183	675	342	37	680	34

. As shown, FreeMix requires 34 s per image, which is significantly faster than traditional finetuning-based methods such as DreamBooth or Custom Diffusion, and slightly faster than FreeCustom despite providing more precise structure–appearance mixing. This demonstrates that our method not only improves the quality of multi-concept mixing but also maintains competitive generation speed, highlighting the efficiency benefits of a finetuning-free design.

5.5. Additional Results

5.5.1. Customization

By using structure generation without appearance transfer and normalization, our method enables single-image customization. As shown in Figure 10, our approach synthesizes images in various contexts while preserving the structural integrity of the input image without finetuning.

5.5.2. Appearance Control

By using only appearance transfer without structure generation, our method extracts the appearance from the input image and transfers it to a new object. As shown in Figure 11, our approach generates images that remain consistent with the appearance of the input concept.

5.5.3. Results with ControlNet

Moreover, we can integrate our method with ControlNet to achieve more stable image generation. As illustrated in Figure 12, ControlNet enables additional conditions as input, allowing for more precise control over the target layout. This integration enables the generation of images where the appearance is consistently transferred to the target layout.

6. Conclusions and Future Work

In this paper, we introduced FreeMix, a finetuning-free method for multi-concept mixing in personalized image generation. We extend self-attention in diffusion models to effectively extract and integrate structural and appearance features from reference images. By selectively combining these attributes, our method generates new concepts. Furthermore, by incorporating DDIM inversion, our method can perform image editing to transfer features from the reference image. Through extensive qualitative and quantitative evaluations, we demonstrate that FreeMix achieves superior performance compared to existing approaches. We plan to explore more advanced disentanglement strategies to enhance the control over individual visual attributes. In addition, we aim to extend FreeMix to handle more than two reference inputs, enabling richer multi-concept compositions. Another promising direction is adapting the method for video generation, where maintaining temporal consistency while mixing structure and appearance remains a challenging task.

7. Limitations

Despite the effectiveness of FreeMix, several limitations remain. First, our approach relies on segmentation masks to disentangle structural and appearance features. While SAM generally produces high-quality masks, its performance may degrade in complex scenarios such as transparent objects, fine structures, or heavy occlusions. Inaccurate masks in these cases can cause unintended leakage between structure and appearance components, thereby reducing generation quality. Second, when the structure and appearance references are highly inconsistent—for instance, an object with intricate geometry combined with a highly textured or stylized appearance—the mixing process may introduce artifacts or result in incomplete transfer. Finally, cross-attention–based mask extraction used during appearance transfer is not always perfectly precise, occasionally leading to minor background interference. We consider these limitations as opportunities for future work, including refining mask precision, developing adaptive attention-based disentanglement strategies, and extending the framework to more extreme structure–appearance discrepancies.