A Doodle-Based Control for Characters in Story Visualization

Abstract

We propose a story visualization technique that allows users to control the arrangement, poses, and styles of characters in a scene based on user-input doodle sketches. Our method utilizes a text encoder to process scene prompts and an image encoder to handle doodle sketches, generating inputs for a predefined scene generation model. Furthermore, we achieve efficient model training by fine-tuning the backbone network by applying a small dataset and employing a LoRA-based fine-tuning technique. We demonstrate that our method can generate characters with various poses and styles from doodle sketches, and it can validate the advantages of our approach by comparing it with the results from other story visualization studies.

1. Introduction

Artists in various fields including comics, cartoons, web-toons and animations require a technique that visualizes a story in a series of scenes that correspond to the artists’ intention. However, most of story visualization requires significant time and effort for a visually convincing results. Recently, story visualization models based on rapidly advancing generative models have proven to be highly effective in resolving their requirements. These story visualization models generate a series of scene images based on the story and characters provided by the user. While early models were developed primarily based on generative adversarial networks (GANs) [1,2,3,4,5], recent trends in generative models have led to the emergence of diffusion- and transformer-based models [6,7,8,9,10]. Despite these advancements, most models are limited by a fixed number of scenes they can generate. Furthermore, they rely solely on text input, which constrains their ability to fully capture the user’s intent regarding scene composition, character actions, and other details.

We propose a story visualization model that overcomes existing limitations by enabling the control on the pose of characters over scene composition. The goal of our model is to generate scenes that follow a doodle sketch while preserving the identity of the characters. A doodle sketch, which represents the pose of a character using a set of few strokes, is provided by a user. Unlike other models, our approach accepts doodle-level sketches as an input, allowing users to control the pose of characters and the overall composition of the scene (see Figure 1). Simultaneously, the model incorporates the character’s identity through the generated scenes.

In order to generate high-resolution images while preserving character identity, we utilize Stable Diffusion XL (SDXL) [11] as our backbone network. This model is fine-tuned using LoRA [12] to accurately reflect the desired character identity. Additionally, in order to align the generated images with the user’s input in terms of sketch dynamics and composition, we employ a separate adapter [13] that controls the model using the provided doodle sketch.

We demonstrate the excellence of our model by quantitatively evaluating the quality and aesthetics of the generated scenes. First, we assess the quality of the generated scenes using LIQE [14] and evaluate the aesthetic aspects of the images with NIME [15]. Finally, Q-align [16] is employed for a comprehensive evaluation that incorporates both criteria. Additionally, we present the results from a user study to further validate the superiority of our model’s outputs.

The benefit of our framework is that we present an intuitive scheme for controlling the characters generated in the story visualization model, which is a key challenge. By presenting a doodle that specifies the pose of a character with a simple set of lines and curves, users can control the pose of characters for scenes encompassing visualized stories.

This paper is organized as follows. We suggest related studies in Section 2. In Section 3, our framework is explained. We present our results and analyze them through various tests in Section 4 and Section 5, respectively. Finally, we draw our conclusion and suggest future work in Section 6.

2. Related Work

2.1. Controlled Image Generation

The progress of generative models such as GAN [17] accelerates the research on image generation in a great scale. Various models employed GAN as a backbone to generate images by appying user-specified conditions [18,19,20,21,22]. Conditional GAN [18] generates images from predefined conditions, but the condition is limited only to numerics. Pix2pix [19] and ACGAN [20] allows more extended conditions such as images; However, Pix2pix requires paired dataset for image generation and ACGAN requires segmenting the dataset according to the classes of the target images.

StyleGAN [21] and StyleGAN2 [22] allow for the generation of high-resolution images with more diverse conditions. A lot of models employed StyleGANs as a backbone to generate high-resolution images. Furthermore, they tried to embed conditions in order to control their generated images. pSp [23], which is one of these models, attempts to achieve fine-grained control by dividing condition details into three levels: coarse, medium, and fine. However, these models still contain limitations, particularly in maintaining the consistency between characters in generated images and in effectively controlling character-specific attributes.

Recently, diffusion model-based schemes have emerged in generating high-quality images [11,24,25,26,27]. A distinguished point of these models is that they generate images from textual inputs. However, due to their reliance on text-based control, reflecting full user’s intent is still challenging. Furthermore, maintaining identity consistency across multiple images remains problematic. To address these limitations, Zhang et al. [28] modified existing diffusion-based models, and some other researchers [12,13,29] added adapters to obtain better control of conditions.

2.2. Diffusion-Based Image Generation

To generate user-desired images, several models [30,31,32,33] have been developed for editing existing images. Models such as Prompt-to-Prompt [30], Custom-Diffusion [31], InstructPix-to-Pix [34], and Mokady et al.’s model [32] allow users to input prompts specifying the desired modification and to generate editing output images. However, as these models rely solely on textual inputs to control modifications, applying full user’s intent in generating images is still challenging. Paint-by-Example [33] allows users to directly mask the areas they wish to modify and then perform inpainting to apply the changes. However, this approach has limitations in generating convincing results for characters or illustrations, as opposed to real photographs.

2.3. Story Visualization

Story visualization generates a series of images that reflect the input story. Story visualization is required to preserve the identities of characters while maintaining consistency across the generated images. The research in this field began with StoryGAN [1], which pioneered the study of story visualization. Subsequent works have developed various approaches using GANs as their backbone [2,3,4,5]. Although GAN-based models maintain scene consistency, they often fail to preserve character identity and suffer from low resolution. Moreover, these models require large amounts of data and extensive training time for each story.

3. Our Method

3.1. The Overview of Our Method

Our model generates scene images from both scene prompts and user-provided doodle sketches. The scene prompt is embedded using the CLIP text encoder $E_t$ and doodle sketch using CLIP image encoder $E_I$, respectively. Instead of training the entire model, we utilize LoRA [12] to train only a small number of parameters, allowing the model to better reflect the identity of the characters. Additionally, we employ a separate adapter to guide the image generation process in accordance with the input doodle sketches. The overview of our method is illustrated in Figure 2.

3.2. Fine-Tuning

To generate high-resolution scene images, we utilize a diffusion-based pretrained model $DM$ as the backbone network. Each network in the model follows an attention structure, with each attention layer F defined as Equation (1).

$$F=Attention(Q,K,V)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d}}}\right)·V.$$

(1)

In Equation (1), Q, K and V, which mean query, key, and value, respectively, are defined in Equations (2)–(4). To incorporate the identity of a character into the model, we fine-tune the model using the character image I. Instead of fine-tuning the entire model, we employ LoRA [12] to facilitate efficient learning. The process of fine-tuning is illustrated in Figure 3. Specifically, to train the character with LoRA, we learn two low-rank matrices $A\in R^{d\times r}$ and $B\in R^{r\times k}$ in place of the original weight matrix W.

To apply the character image I to each layer, we first embed it using the CLIP image encoder $E_I$ [35]. The formula for incorporating I into the model using LoRA is as follows:

$$Q=W_Q^i·{\phi }_i\left(z_t\right).$$

(2)

$$K=W_K^i·E_I\left(I\right).$$

(3)

$$V=W_V^0·E_I\left(I\right)+BA·E_I\left(I\right).$$

(4)

Equations (2)–(4) define the query, key, and value of our framework. The purpose of these equations is to train parameters $W_Q^i$, $W_K^i$, and $W_V^0$, who play the role of self-attention. The term ${\phi }_i\left(z_t\right)$ represents the flattened version of $z_t$. Our model applies the LoRA structure exclusively to V in order to learn the character image I. Since we use a pretrained model, all weight matrices W are kept fixed, and only A and B are trained.

3.3. Doodle Sketch to Scene Image

Our model requires a user-specified doodle sketch $I_{ds}$ in order to control the overall layout of a scene according to users’ intent. To incorporate this doodle sketch into the model, we treat it as a conditioning input and apply it using an adapter $\mathcal{A}$. We utilize the T2I-adapter proposed by Mou et al. [13] as our adapter $\mathcal{A}$. The adapter $\mathcal{A}$ is composed of a scale block, which consists of one convolutional network and two residual blocks, followed by four iterations of a downsampling pair. As a result, the doodle sketch $I_{ds}$ processed through $\mathcal{A}$ is embedded into four sketch vectors $F_{ds}=\{F_{ds}^1,F_{ds}^2,F_{ds}^3,F_{ds}^4\}$. These sketch vectors $F^{ds}$ are then applied correspondingly to each layer of the fine-tuned encoder $DM$, represented as $F_{enc}=\{F_{enc}^1,F_{enc}^2,F_{enc}^3,F_{enc}^4\}$. The above process can be expressed mathematically as follows:

$$F_{ds}=\mathcal{A}\left(E_I\left(I_{ds}\right)\right){\widehat{F}}_{enc}=F_{enc}^i+F_{ds}^i,i\in \{1,2,3,4\}.$$

(5)

We also reflect the input scene text y as a prompt for the generated scene image. The input prompt is embedded using the CLIP text encoder $E_T$ [35]. The loss function of our model, incorporating all the aforementioned conditions, is defined as follows:

$$\mathcal{L}={\mathbb{E}}_{x,z\sim N(0,1),t,E_T,y,F_{ds}}\left[{∥\begin{array}{c}ϵ-{ϵ}_{\theta }(z_t,t,E_T\left(y\right),F_{ds})\end{array}∥}_2^2\right].$$

(6)

In Equation (6), $z_t$ denotes noise in time step t, $E_I$ and $E_T$ are the CLIP image encoder and CLIP text encoder, respectively. y is a prompt, and $F_{ds}$ is the sketch vector.

4. Implementation and Results

4.1. Implementation

We implemented our model in a cloud platform with Intel Xeon Pentium 8480 and nVidia H100 80 GB. We also employed SDXL [11] as a backbone network and CLIP ViT-B/32 [35] as an embedding encoder.

4.2. Results

We fine-tune existing pretrained story visualization models to incorporate our method, which takes doodle sketches as inputs. Our model visualizes three stories: Flintstone, Sailormoon, and Suzume. The pretrained models for these stories are collected as follows: we select the Flintstone model [36] from Hugging Face and the Suzume [37] and Sailormoon models [38] from CIVITAI as the pretrained story visualization models. These models are capable of generating scenes by utilizing the characters that appear within the selected story contents. However, they do not provide control over the poses of the characters as intended by the user. We demonstrate that by allowing users to input specific poses using doodles for each story prompt, we can generate scene images that reflect the poses specified by the doodles and stories suggested in the prompts. Figure 1 presents the results from the Suzume story, Figure 4 shows the results from the Flintstone story, and Figure 5 displays the results from the Sailormoon story, respectively. In each Figure, the images are generated from same prompts. However, the results from our models are generated from both doodles and prompts.

5. Evaluation

5.1. Comparison

We compare our results with those from two existing studies in the field of story visualization: ARLDM [39] and StoryDallE [6]. These studies were selected because they provide visualization results for the Flintstone story, making them appropriate for comparison with our approach. We presented prompts for four prompts and generated corresponding scene images. Additionally, in our approach, we provided doodles to specify characters. The comparison between our results, ARLDM, and StoryDallE is shown in Figure 6.

5.2. Quantitative Evaluation

We execute a quantitative evaluation by applying the evaluation metrics used in previous story visualization studies, including LIQE [40], NIMA [15], and Q-align [16]. Q-align can be applied for both quality and aesthetic elements of the generated scene images.

LIQE [40] is one of blind image quality assessment metrics, which evaluates target images without reference images. It estimates the image quality based on the vision–language correspondence using transformer-based models. NIMA [15] estimates the quality of an image using the semantic-level aesthetic using a deep CNN structure. It estimates the quality of an image into a ten-point metric using the likelihood estimated from the training dataset. Q-align [16] estimates the image quality and aesthetics using large multi-model models, which executes vision–language tasks such as image captioning and visual question answering.

The results of these evaluations are presented in

Table 1. The comparison of our results with the existing models.

	StoryDallE	ARLDM	Ours
LIQE	1.94897	1.04155	2.32057
NIMA	3.33332	3.44587	4.58561
Q-align (quality)	1.881325	2.03805	3.60447
Q-align (aesthetic)	1.3723	1.6733	3.60447

, showing that our model achieves higher scores compared to the results from previous studies.

5.3. Qualitative Evaluation

We prepared three questions regarding the images presented in Figure 6. The first question assesses the correspondence between the input prompt and the generated images. The second question evaluates the coherence of character identity across the generated images for the same content. The third question examines the quality of the generated images, specifically the absence of artifacts and overall visual excellence.

We recruited 20 participants for the qualitative evaluation of our study. Among the participants, 15 were in their twenties, 5 were in their thirties, 11 were male, and 9 were female. They were instructed to rate each image on a five-point scale, and the average score for images generated by each method was calculated. The results of this evaluation are presented in

Table 2. The results of user study.

	StoryDallE	ARLDM	Ours
Correspondence	3.67	4.06	4.25
Coherence	3.81	4.27	4.39
Quality	2.65	3.94	4.84

, where it is confirmed that the images generated by our method received the highest scores.

5.4. Ablation Study

We demonstrate the effectiveness of our model by showing that our model can control generated scene images by providing doodles with subtle variations. Figure 7 presents a pair of scene images generated from the same prompt. The only difference in the doodles used for scene generation is the variation in the characters’ hairstyles. The doodles in the upper row have a single ponytail, while the doodles in the lower row have two ponytails. The characters in the generated scene images accurately reflect this difference, confirming that our model can effectively incorporate doodle inputs to control generated scenes.

We present another ablation for LoRA. With LoRA, which fine-tunes the parameter matrix, our model can preserve the identity of characters. Without LoRA, the generated characters have a higher chance to lose their identities. Figure 8 illustrate the result of this ablation study.

5.5. Limitation

Even though we have accomplished the consistency of characters regarding posing and facial mimicry, we still have many limitations in preserving consistency. While the doodle sketches used in our model can effectively control a character’s position, movement, and hairstyle, we cannot preserve the consistency of the character’s clothing. As illustrated in Figure 9, the clothes of the characters or a ribbon on a piece of clothing do not match for a sequence of generated images.

Furthermore, backgrounds such as furniture, chauffeur, or wallpapers may not match. As illustrated in Figure 9, the chauffeurs in the scenes do not preserve consistency.

6. Conclusions and Future Work

In this study, we proposed a story visualization model that allows users to control the placement, pose, and style of characters using simple doodle sketches. By fine-tuning an existing story visualization model with a small dataset, our approach enables users to effectively control characters in the story visualization process. We demonstrated our results of controlling characters across various contents using doodle sketches. Moreover, we proved the advantages of our method by comparing and evaluating our results against those from other studies.

As a future work, we will focus on developing methods to precisely control elements such as clothing, which are not fully controllable in this study. Additionally, we aim to explore techniques for generating continuous and consistent poses across consecutive scenes to implement animations.