Generative Architectural Design from Textual Prompts: Enhancing High-Rise Building Concepts for Assisting Architects

Abstract

In the early stages of architectural design, architects convert initial ideas into concrete design schemes, which heavily rely on their creativity and consume considerable time. Therefore, generative design methods based on artificial intelligence are promising for such tasks. However, effectively communicating design concepts to machines is challenging. To address this challenge, this paper proposes a novel cross-model approach for architectural design concepts using textual descriptions to assist architects, comprising a design concept extraction module and an architectural appearance generation module. The design concept extraction module adopts a contrastive learning framework to yield a text encoder with semantic extraction. Subsequently, the architectural appearance generation module proposes a novel deep sparse and text fusion generative adversarial network to convert the extracted design concept semantics into conceptual sketches, utilizing the unique sparsity of sketches. Additionally, it employs the pre-trained latent stable diffusion model to generate realistic and diverse high-rise building renderings, simulating the recreation processes of architects. The generated designs are evaluated qualitatively and quantitatively and further compared with existing real-life buildings to demonstrate the effectiveness of the proposed method. Furthermore, we demonstrate the feasibility of applying the proposed methodology in the early stages of architectural design by modeling a generated design.

1. Introduction

Architecture represents a synthesis of artistic expression and scientific principles that demands a delicate balance between creativity and precision. Whether manifested in towering skyscrapers or intimate dwellings, architectural design is a creative process for humans with novel perspectives on aesthetics and culture. Traditional architectural design relies on conceptual sketches to convey the design concepts of architects and recreates them to generate realistic architectural concepts. Conceptual sketches are central to the emergence and reinterpretation of this process [1]. However, this approach encounters challenges such as creative constraints and significant time investments. Moreover, the seemingly abstract nature of this process is difficult to depict mathematically. Fortunately, generative artificial intelligence (AI), such as a generative adversarial network (GAN) [2], offers a new path for providing cutting-edge technology to assist architects in conceptual design, which can be performed by learning mathematical mappings between probability distributions.

Various GANs have been widely employed in the civil engineering [3,4,5,6] and architectural fields. In particular, the outstanding creative generative skills of GANs have yielded noteworthy results for diverse design tasks. The first is a layout design that uses various GANs. For example, a floor plan layout is a critical aspect of design. House-GAN [7] encoded the planar space constraints among rooms into an adjacency graph structure and employed graph-conditioned GANs for automated floor plan generation. Furthermore, House-GAN++ [8] was extended to generate floor plans for non-rectangular room shapes with doors by combining a functional graph instead of an adjacency graph. However, both methods are limited to single-layer generation and ignore the layout relationships among different layers. To address this limitation, Building-GAN [9] introduced a cross-modal graph neural network and a new voxel graph presentation for automated 3D floor plan generation while considering the associativity of room assignments between different floors. Additionally, a GAN-based layout design was applied to the modular construction. Ghannad et al. [10] combined the CoGAN framework [11] and a vectorizing algorithm to generate modular housing designs. In addition to layout design, GANs have been leveraged to learn structural information depicted in images and generate structural designs that satisfy engineering requirements. For example, Liao et al. [12] proposed StructGAN to achieve automated layout designs of shear walls in residential buildings by learning existing shear wall design documents and assigning different colors. FrameGAN [13] adopted a two-stage method for generating automated component layout designs of steel frame-brace structures. In contrast to functional layouts and structural designs, the generation of aesthetic architectural forms based on AI presents a more significant challenge. Sun et al. [14] utilized the CycleGAN framework [15] to automatically generate stylized facades via image-to-image learning. As a pioneering study in the intelligent generative design of building appearances, Self-Sparse GAN [16] was proposed to creatively generate numerous diverse sketches [17]. To integrate user preferences, Qian et al. [18] proposed an innovative AI-based architectural design approach that incorporates user preferences, encompassing geometric shapes and architectural preferences, to generate customized high-rise sketches. Subsequently, images were used as user preference descriptions. However, the capacity to comprehensively convey design concepts through images is limited, underscoring the necessity for alternative approaches, especially in the context of generating aesthetic designs for architectural forms.

Therefore, textual descriptions, which are more convenient than images for describing user preferences, have emerged as a more effective means for generating architectural appearances. Recently, text-to-image transformations have been successfully implemented in the field of computer science using various conditional GANs (cGANs) [19]. GAN-INT-CLS [20] first utilized cGANs to generate 64 × 64 images based on textual descriptions. Furthermore, for enhanced image resolution and the efficient utilization of computing resources, StackGAN [21] and StackGAN++ [22] proposed a multi-stage approach to generate 256 × 256 images by combining multiple generator and discriminator pairs. However, the problem of inconsistency in the details of the generated images remains. To address this limitation, Qiao et al. [23] further enhanced the multi-stage generation process by incorporating a text-to-image-to-text framework to align the images with the corresponding descriptions. However, the complexity and entanglements introduced by the multi-stage framework render effective training challenging. To overcome these problems, Tao et al. [24] proposed DF-GAN, which adopts a single-stage framework (One-Way Output) for direct high-resolution image generation, enabling end-to-end training. Moreover, within the end-to-end training framework, various fusion methods have been introduced to facilitate the generation of refined visual content. Liao et al. [25] proposed semantic condition batch normalization by adapting a text fusion method to align individual image regions with words in the description. In contrast, DiverGAN [26] employs the attention mechanism to fuse text and image features for semantically consistent image generation. However, these approaches are not directly applicable to generating conceptual sketches from text semantics due to the inherent sparsity of sketches compared to color images. Consequently, generating high-quality architectural sketches from text semantics is difficult owing to the sparsity of sketches.

As mentioned earlier, existing AI-assisted architectural conceptual designs require a more refined method for conveying design concepts or user preferences, and text-to-image transformation is a promising solution to this limitation. It is crucial to develop an effective text fusion method to enable the generation of high-quality sketches.

Therefore, two interconnected sub-modules, the design concept extraction (DCE) and architectural appearance generation (AAG) modules, are proposed in this paper for the intelligent design of high-rise building concepts from textual descriptions. The DCE module employs a contrastive learning framework to extract abstract design concepts from textual descriptions, thus facilitating the matching of corresponding image–text pairs. Thereafter, the AAG module translates the extracted text semantics into a spectrum of realistic and diverse high-rise building concepts. To generate semantically coherent and visually realistic high-rise conceptual sketches, the AAG module employs a novel deep sparse text fusion GAN (DSTF-GAN) to generate semantically coherent and visually realistic high-rise conceptual sketches by integrating the inherent sparsity of sketches and extracted semantic information with image features across both channel and spatial dimensions. Additionally, the AAG module incorporates a pre-trained latent stable diffusion model [27] into a pre-trained rendering (PR) module to render the generated sketches into finalized images.

The novelty of this study is summarized as follows: (1) A pioneering AI-assisted approach for autonomously designing high-rise building concepts from text descriptions is proposed; (2) a generic model is introduced to obtain a pre-trained text encoder through contrastive learning, facilitating the effective matching of sketches with their descriptions; and (3) a novel generative adversarial network that integrates sketch sparsity and text semantics to generate high-rise conceptual sketches is proposed.

The remainder of this article is organized as follows. Section 2 elaborates on the proposed framework for intelligent design based on textual descriptions. Section 3 presents the implementation details and provides insights into the design process outcomes using qualitative and quantitative evaluations, as well as the comparisons between the generated results and existing real-life buildings. Section 4 presents a comprehensive discussion that covers the comparison of the proposed fusion method and alternative fusion techniques, along with an in-depth analysis of the components of the proposed fusion method. Section 5 demonstrates the feasibility of the proposed methodology in the architectural conceptual design phase by modeling a generated design. Finally, Section 6 concludes this paper.

2. Methodology

The proposed method comprises a DCE module and an AAG module, as shown in Figure 1. The DCE module uses a contrastive learning framework to obtain a pre-trained text encoder that enables semantic extraction from textual descriptions. Subsequently, the AAG module transforms the extracted design semantics into high-rise building concepts.

2.1. Design Concept Extraction Module

For text-to-image tasks, an effective text encoder is crucial for accurately extracting semantic information from textual descriptions. Contrastive learning is particularly effective for obtaining a pre-trained text encoder as it strengthens the encoder’s ability to extract salient features from text prompts by maximizing the alignment of semantically relevant pairs while minimizing the similarity of unrelated ones. According to the principles of Contrastive Language–Image Pre-training (CLIP) [28], the DCE module uses a contrastive text–image matching model (CTIMM) to obtain an effective text encoder, as shown in Figure 2. This model comprises text and image encoder networks dedicated to extracting text and image features, respectively, and maps them to a shared semantic space. By utilizing contrastive learning, the model establishes meaningful associations between conceptual sketches and their corresponding textual descriptions. The objective of CTIMM optimization is to maximize the similarity between the features of each sketch and its corresponding text features within the semantic space.

The text encoder incorporates a bidirectional Long Short-Term Memory (BiLSTM) architecture [29] to extract multi-level semantic information and a multi-head attention layer [30] that re-extracts low-level semantic information to augment text semantics and map to the shared semantic space. Within the BiLSTM, each word corresponds to two hidden states, which can be concatenated to obtain the low-level semantic information represented by ${\mathit{h}}_{\mathit{i}}\in {\mathbb{R}}^{512}(i=1,\dots ,n)$. The last hidden state ${\mathit{h}}_{\mathit{n}}\in {\mathbb{R}}^{512}$ is selected as the text semantic vector. Furthermore, the multi-head attention layer is designed to re-extract low-level semantics ${\mathit{h}}_{\mathit{i}}\in {\mathbb{R}}^{512}(i=1,\dots ,n-1)$, yielding supplementary semantic information ${\mathit{h}}^{\prime }\in {\mathbb{R}}^{512}$. Finally, by combining ${\mathit{h}}^{\prime }$ with ${\mathit{h}}_{\mathit{n}}$, the final global text semantic feature $\mathit{T}\in {\mathbb{R}}^{512}$ is obtained in the shared semantic space. The text encoder can be formulated as

$$\left({\mathit{w}}_1,\dots ,{\mathit{w}}_{\mathit{n}}\right)={\mathit{M}}_{300\times {\mathit{n}}_{\mathit{w}}}(\mathit{w}\mathit{o}\mathit{r}{\mathit{d}}_1,\mathit{}\dots ,\mathit{}\mathit{w}\mathit{o}\mathit{r}{\mathit{d}}_{\mathit{n}})$$

(1)

$${\mathit{h}}_1,\dots ,{\mathit{h}}_{\mathit{n}}=F_{BiLSTM}({\mathit{w}}_1,\dots ,{\mathit{w}}_{\mathit{n}})$$

(2)

$${\mathit{h}}^{\mathit{\prime }}=F_{Attn}({\mathit{h}}_1,\dots ,{\mathit{h}}_{\mathit{n}-1})$$

(3)

$$\mathit{T}={\mathit{h}}_{\mathit{n}}+{\mathit{h}}^{\mathit{\prime }}$$

(4)

where $\mathit{w}\mathit{o}\mathit{r}{\mathit{d}}_{\mathit{i}}(i=1,\dots ,n)$ denotes the one-hot encoding of the $i$-th word in the description based on the word pool consisting of words in the dataset, ${\mathit{M}}_{\mathbf{300}\times {\mathit{n}}_{\mathit{w}}}\in {\mathbb{R}}^{300\times n_w}$ is the semantic embedding matrix for obtaining the semantic embedding vector of each word in the description, $n$ is the number of words in the description, $n_w$ denotes the total number of words in the dataset, and $F_{BiLSTM}$ and $F_{Attn}$ denote the BiLSTM network and multi-head attention layer, respectively. The multi-head attention layer is formulated as follows [30]:

$$F_{Attn}\left(\mathit{H}\right)=MultiHead\left(\mathit{H}\right)\left[0\right]$$

(5)

$$\begin{gathered} MultiHead\left(\mathit{H}\right)=Concat\left(\mathit{h}\mathit{e}\mathit{a}{\mathit{d}}_1,\dots ,\mathit{h}\mathit{e}\mathit{a}{\mathit{d}}_{\mathit{h}}\right){\mathit{W}}^{\mathit{o}} \\ where \mathit{h}\mathit{e}\mathit{a}{\mathit{d}}_{\mathit{i}}=Attention(\mathit{H}{\mathit{W}}_{\mathit{i}}^{\mathit{Q}},\mathit{H}{\mathit{W}}_{\mathit{i}}^{\mathit{K}},\mathit{H}{\mathit{W}}_{\mathit{i}}^{\mathit{V}}) \end{gathered}$$

(6)

$$Attention\left(\mathit{Q},\mathit{K},\mathit{V}\right)=softmax\left({\displaystyle \frac{\mathit{Q}{\mathit{K}}^{\mathit{T}}}{\sqrt{d_k}}}\right)\mathit{V}$$

(7)

where $\mathit{H}\in {\mathbb{R}}^{d_m\times d_n}$ is the input matrix of the multi-head attention layer, $h$ is the number of heads in the multi-head attention layer, ${\mathit{W}}_{\mathit{i}}^{\mathit{Q}}\in {\mathbb{R}}^{d_n\times d_k}$, ${\mathit{W}}_{\mathit{i}}^{\mathit{K}}\in {\mathbb{R}}^{d_n\times d_k}$, ${\mathit{W}}_{\mathit{i}}^{\mathit{V}}\in {\mathbb{R}}^{d_n\times d_v}$, ${\mathit{W}}^{\mathit{o}}\in {\mathbb{R}}^{h{d}_v\times d_n}$ are the parameter matrices, $Concat$ denotes splicing multiple matrices along the columns, and $F_{Attn}$ denotes the extraction of the first column from the output of $MultiHead\left(\mathit{H}\right)$.

The image encoder has a ResNet-18 [31] architecture followed by an additional multi-head attention layer to map the extracted image features to the shared semantic space. The ResNet-18 architecture processes the sketch to obtain the local features $\mathit{v}\in {\mathbb{R}}^{512\times 7\times 7}$, and then it is reshaped into a local feature matrix $\mathit{i}\in {\mathbb{R}}^{512\times 50}$, with the last column representing the average of all the local feature vectors. Subsequently, the final global feature of the sketch in the shared semantic space, denoted as $\mathit{I}\in {\mathbb{R}}^{512}$, is obtained through an additional multi-head attention layer. The image encoder can be formulated as

$$\mathit{v}=F_{ResNet}\left(\mathit{x}\right)$$

(8)

$$\mathit{I}={F_{Attn}(f}_{re}\left(\mathit{v}\right))$$

(9)

where $\mathit{x}$ is the input sketch and $f_{re}$, $F_{ResNet}$, and $F_{Attn}$ denote the average and reshape operators, ResNet-18, and the other multi-head attention layer, respectively.

The loss function of the CTIMM is designed to maximize the cosine similarity between the conceptual image and text features for matching pairs while minimizing the cosine similarity between text features for mismatching pairs in the shared semantic space. The objective loss function is defined as follows:

$${\mathcal{L}}_{CTIMM}=\sum _{\begin{array}{c}i=1,j=1\\ i\ne j\end{array}}^n⟨{\mathit{I}}_{\mathit{i}}\mathrm{},{\mathit{T}}_{\mathit{j}}^{\mathit{T}}⟩-\sum _{\begin{array}{c}i=1,j=1\\ i=j\end{array}}^n⟨{\mathit{I}}_{\mathit{i}}\mathrm{},{\mathit{T}}_{\mathit{j}}^{\mathit{T}}⟩$$

(10)

where $⟨,⟩$ denotes the dot product operation between two vectors, and ${\mathit{I}}_{\mathit{i}}\in {\mathbb{R}}^{512}$ and ${\mathit{T}}_{\mathit{j}}\in {\mathbb{R}}^{512}$ are the extracted feature vectors of the $i$-th sketch and $j$-th description for the same batch, respectively. The CTIMM is trained using Algorithm 1.

Algorithm 1. Pseudocode for the training of the CTIMM.

Contrastive text–image matching model: AdamW optimizer with $\left({\beta }_1,{\beta }_2\right)=\left(\mathrm{0.9,0.99}\right)$, $eps=1\times {10}^{-8}$, $weight\_devay=0.25, lr=0.0001$; CosineAnnealingWarmRestarts lr_scheduler with $T\_0=400, T\_mult=1, eta\_min=0, last\_epoch=-1$.

1: Extract the features of sketch and textual description

$\mathrm{I}\_\mathrm{f}=\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\_\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}\left(\mathrm{I}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\right)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\#[\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e},512]$

$\mathrm{T}\_\mathrm{f}=\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}\_\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}\left(\mathrm{T}\mathrm{e}\mathrm{x}\mathrm{t}\right)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \#[\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e},512]$

2: Normalization with L2 norm

$\mathrm{I}\_\mathrm{e}=\mathrm{L}2\_\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}(\mathrm{I}\_\mathrm{f},\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{s}=1)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\#[\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e},512]$

$\mathrm{T}\_\mathrm{e}=\mathrm{L}2\_\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\left(\mathrm{T}\_\mathrm{f},\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{s}=1\right)\hspace{1em}\hspace{1em}\hspace{1em} \#[\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e},512]$

3: Calculate the cosine similarities

$\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s}=\mathrm{n}\mathrm{p}.\mathrm{d}\mathrm{o}\mathrm{t}(\mathrm{I}\_\mathrm{e},\mathrm{T}\_\mathrm{e}.\mathrm{T})\times \mathrm{n}\mathrm{p}.\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathrm{t}\right)\hspace{1em}\hspace{1em}\#[\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e},\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e}]$

4: Calculate the loss

$\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}\mathrm{s}=\mathrm{n}\mathrm{p}.\mathrm{a}\mathrm{r}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{e}(\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{s}\mathrm{i}\mathrm{z}\mathrm{e})$

$\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}\_\mathrm{i}=\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\_\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{y}\_\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}(\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s},\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}\mathrm{s},\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{s}=0)$

$\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}\_\mathrm{t}=\mathrm{c}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\_\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{y}\_\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}(\mathrm{l}\mathrm{o}\mathrm{g}\mathrm{i}\mathrm{t}\mathrm{s},\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{l}\mathrm{s},\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{s}=1)$

$\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}=(\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}\_\mathrm{i}+\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}\_\mathrm{t})/2$

#$\mathrm{t}$ is a learnable parameter

# CosineAnnealingWarmRestarts refers to Stochastic Gradient Descent with Warm Restarts.

2.2. Architectural Appearance Generation Module

When the architectural design textual descriptions have been translated into text semantic features by the DCE module, the new architectural design is performed by the AAG module, as shown in Figure 3. The DSTF-GAN of the AAG module translates the extracted design text semantics into high-rise conceptual sketches. Subsequently, the PR module performs recreation and generates realistic high-rise buildings.

The DSTF-GAN is depicted in Figure 4, and the detailed properties of layers and operators are listed in

Table A1. The architecture of the deep sparse and text fusion generative adversarial network for the generation of high-rise conceptual sketches based on textual descriptions.

$\mathbf{Generator}, \mathit{z}\in {\mathbb{R}}^{200}~\mathit{N}\left(\mathrm{0,1}\right)$	$\mathbf{Discriminator}, \mathit{I}\mathit{m}\mathit{a}\mathit{g}\mathit{e}\in {\mathbb{R}}^{256\times 256}, \mathit{T}\in {\mathbb{R}}^{512}$
Linear: $z\to 128\times 4\times 4$	Conv. $3\times 3$, stride = 1, padding = 1
DeConv. $4\times 4$, stride = 2, padding = 1, BN-128, ReLU, Sparse Fusion Block, Text Fusion Block, ReLU	Conv. $4\times 4$, stride = 2, padding = 1, BN-128, LeakyReLU
DeConv. $4\times 4$, stride = 2, padding = 1, BN-128, ReLU, Sparse Fusion Block, Text Fusion Block, ReLU	Conv. $4\times 4$, stride = 2, padding = 1, BN-128, LeakyReLU
DeConv. $4\times 4$, stride = 2, padding = 1, BN-128, ReLU, Sparse Fusion Block, Text Fusion Block, ReLU	Conv. $4\times 4$, stride = 2, padding = 1, BN-128, LeakyReLU
DeConv. $4\times 4$, stride = 2, padding = 1, BN-128, ReLU, Sparse Fusion Block, Text Fusion Block, ReLU	Conv. $4\times 4$, stride = 2, padding = 1, BN-128, LeakyReLU
DeConv. $4\times 4$, stride = 2, padding = 1, BN-128, ReLU, Sparse Fusion Block, Text Fusion Block, ReLU	Conv. $4\times 4$, stride = 2, padding = 1, BN-128, LeakyReLU
DeConv. $4\times 4$, stride = 2, padding = 1, BN-128, ReLU, Sparse Fusion Block, Text Fusion Block, ReLU	Conv. $4\times 4$, stride = 2, padding = 1, BN-128, LeakyReLU
BN-128, ReLU, Conv. $3\times 3$, stride = 1, padding = 1, Tanh	Conv. $3\times 3$, stride = 1, padding = 1, LeakyReLU, Conv. $4\times 4$, stride = 1, padding = 0

Abbreviations: DeConv, deconvolutional layer; Conv, convolutional layer; ReLU, rectified linear unit; LeakyReLU, LeakyReLU activation function; BN, batch normalization.

and

Table A2. The architecture of the sparse fusion and text fusion blocks for integrating the sparsity of sketches and text features into the generated sketches, respectively. $H_i\times W_i$ is the size of the output feature map of the deconvolution layer in the $i$-th sparse text fusion block. In $g_{i,2}^z$ and $g_i^T$, if $H_i\times W_i$ is greater than $16\times 16$, deconvolution is adopted; otherwise, convolution is used.

$\mathbf{Sparse} \mathbf{Fusion} \mathbf{Block}, \mathit{z}\in {\mathbb{R}}^{200}$		$\mathbf{Text} \mathbf{Fusion} \mathbf{Block}, \mathit{T}\in {\mathbb{R}}^{512}$
$f_i^z$$, \mathrm{Linear}: \mathit{z}\to 1024,$ ReLU    $\mathrm{Linear}: 1024\to 512,$ ReLU    $\mathrm{Linear}: 512\to 256,$ ReLU		$f_{i,1}^T$$, \mathrm{Linear}: \mathit{T}\to 128,$     LeakyReLU     $\mathrm{Linear}: 128\to 128,$     LeakyReLU	$f_{i,2}^T,$ Linear: $\mathit{T}\to 256,$     LeakyReLU
$g_{i,1}^z,$ Linear: 512$\to 128$	$g_{i,2}^z,$ Conv or DeConv: $16\times 16\to H_i\times W_i$		$g_i^T,$ Conv or DeConv: $16\times 16\to H_i\times W_i$
ReLU	ReLU		LeakyReLU

Abbreviations: DeConv, deconvolutional layer; Conv, convolutional layer; ReLU, rectified linear unit; LeakyReLU, LeakyReLU activation function; BN, batch normalization.

in Appendix A. As noted by Qian et al. [17], a simple GAN architecture is sufficient for sketch synthesis. Building on this, DSTF-GAN is designed to effectively integrate the inherent sparsity of sketches [17] and the extracted semantics of textual descriptions into the image features to generate high-quality sketches. As shown in Figure 4a, the generator of the DSTF-GAN introduces a novel block called the sparse text fusion (STF) block to seamlessly integrate both the sparsity of the sketches and text features into the generated sketches. The STF block consists of a sparse fusion (SF) block (Figure 5a) to incorporate the sparsity of sketches and a text fusion (TF) block (Figure 5b) to augment the feature maps with the conditioned text semantic features.

The SF block employs a two-headed neural network to extract the sparsity of sketches, transforming the latent vector $\mathit{z}\in {\mathbb{R}}^{200}$ into a channel sparsity coefficient ${\mathit{\alpha }}_{\mathit{i}}\in {\mathbb{R}}^{1\times 1\times C_i}$ and a position sparsity coefficient ${\mathit{\beta }}_{\mathit{i}}\in {\mathbb{R}}^{H_i\times W_i\times 1}$. Subsequently, the feature map ${\mathit{h}}_{\mathit{i}}\in {\mathbb{R}}^{H_i\times W_i\times C_i}$ is multiplied by two coefficients, and the results are summed to yield the output ${\mathit{F}}_{\mathit{z},\mathit{i}}\in {\mathbb{R}}^{H_i\times W_i\times C_i}$. Sparsity fusion in the $i$-th STF block can be expressed as follows:

$${\mathit{h}}_{\mathit{i}}=ReLU\left(f_{BN}\left(f_{DeConv}\left({\mathit{F}}_{\mathit{i}-1}\right)\right)\right)$$

(11)

$${\mathit{\alpha }}_{\mathit{i}}=ReLU\left(g_{i,1}^z\left(f_i^z\left(\mathit{z}\right)\right)\right)$$

(12)

$${\mathit{\beta }}_{\mathit{i}}=ReLU\left(g_{i,2}^z\left(f_{reshape}\left(f_i^z\left(\mathit{z}\right)\right)\right)\right)$$

(13)

$$F_{z,i}^{j,k,l}={\alpha }_i^l\times h_i^{j,k,l}+{\beta }_i^l\times h_i^{j,k,l}$$

(14)

where ${\mathit{F}}_{\mathit{i}-1}\in {\mathbb{R}}^{H_{i-1}\times W_{i-1}\times C_{i-1}}$ is the input feature maps of the $i$-th STF block; $f_{DeConv}$, $f_{BN}$, and $ReLU$ denote the deconvolutional layer, the batch normalization layer, and the rectified linear unit activation function, respectively; $f_i^z,g_{i,1}^z,g_{i,2}^z$ denotes an underlying shared layer (multi-layer perceptron, MLP) and two exclusive networks of the two-headed neural network, respectively; and $f_{reshape}$ denotes the reshape operation, as shown in Figure 5a.

The TF block employs two MLPs and a convolutional layer to predict a channel-wise coefficient ${\mathit{w}}_{\mathit{i}}\in {\mathbb{R}}^{1\times 1\times C_i}$ and a position-wise coefficient ${\mathit{b}}_{\mathit{i}}\in {\mathbb{R}}^{H_i\times W_i\times 1}$ based on the text feature $\mathit{T}\in {\mathbb{R}}^{512}$. The feature map ${\mathit{F}}_{\mathit{z},\mathit{i}}$ is then multiplied by ${\mathit{w}}_{\mathit{i}}$ and ${\mathit{b}}_{\mathit{i}}$, respectively, and the two results are added together to yield the output ${\mathit{F}}_{\mathit{T},\mathit{i}}\in {\mathbb{R}}^{H_i\times W_i\times C_i}$. Finally, the resulting feature map ${\mathit{F}}_{\mathit{i}}\in {\mathbb{R}}^{H_i\times W_i\times C_i}$ of the $i$-th STF block is obtained after applying a ReLU activation function. This block is formulated as follows:

$${\mathit{w}}_{\mathit{i}}=f_{i,1}^T\left(\mathit{T}\right)$$

(15)

$${\mathit{b}}_{\mathit{i}}=LeakyReLU\left(g_i^T\left(f_{reshape}\left(f_{i,2}^T\left(\mathit{T}\right)\right)\right)\right)$$

(16)

$$F_{T,i}^{j,k,l}=w_i^l\times F_{z,i}^{j,k,l}+b_i^l\times F_{z,i}^{j,k,l}$$

(17)

$${\mathit{F}}_{\mathit{i}}=ReLU\left({\mathit{F}}_{\mathit{T},\mathit{i}}\right)$$

(18)

where $LeakyReLU$ denotes the LeakyReLU activation function; $f_{i,1}^T$ and $f_{i,2}^T$ both denote a single MLP; $g_i^T$ is a convolutional layer; and $f_{reshape}$ denotes the reshaping operation, as shown in Figure 5b.

The discriminator is inspired by the Target-Aware discriminator in the DF-GAN [24], demonstrating its ability to ascertain the consistency between input image semantics and text. To ensure a balanced and symmetrical structure with the generator, as well as semantic consistency between the conceptual sketches and input text, a simplified version of the target-aware discriminator is adopted, as shown in Figure 4b.

The discriminator converts a sketch into feature maps, denoted as ${\mathit{F}}_6^{\prime }\in {\mathbb{R}}^{4\times 4\times 128}$, using a convolutional layer and six DownBlock modules. Subsequently, the feature map ${\mathit{F}}_6^{\prime }$ with the reshaped text feature ${\mathit{T}}^{\prime }\in {\mathbb{R}}^{4\times 4\times 512}$ along the channels is concatenated. The output of the discriminator determines whether the sketch is semantically consistent with the textual description.

To ensure the stability of the adversarial training process, the hinge loss [32] is used as the objective function of the DSTF-GAN. Additionally, the Matching-Aware zero-centered Gradient Penalty (MA-GP) of the DF-GAN [24] is incorporated into the objective function of the discriminator to reinforce the text–image’s semantic consistency. The complete loss of the DSTF-GAN is expressed as follows:

$$\begin{array}{ll}{\mathcal{L}}_D=& -{\mathbb{E}}_{\mathit{x}~{\mathbb{P}}_r}\left[\min \left(0,-1+D\left(\mathit{x},\mathit{T}\right)\right)\right]\\ & -\left(1/2\right){\mathbb{E}}_{\mathit{z}~{\mathbb{P}}_z}\left[min\left(0,-1-D\left(G\left(\mathit{z},\mathit{}\mathit{T}\right),\mathit{}\mathit{T}\right)\right)\right]\\ & -\left(1/2\right){\mathbb{E}}_{\mathit{x}~{\mathbb{P}}_{mis}}\left[min\left(0,-1-D\left(\mathit{x},\mathit{T}\right)\right)\right]\\ & + k{\mathbb{E}}_{\mathit{x}~{\mathbb{P}}_r}\left[{\left(‖{\nabla }_{\mathit{x}}D\left(\mathit{x},\mathit{T}\right)‖+‖{\nabla }_{\mathit{T}}D\left(\mathit{x},\mathit{T}\right)‖\right)}^p\right]\end{array}$$

(19)

$${\mathcal{L}}_G=-{\mathbb{E}}_{\mathit{z}~{\mathbb{P}}_z}\left[D\left(G\left(\mathit{z}, \mathit{T}\right),\mathit{T}\right)\right]$$

(20)

where $\mathit{z}$ is the noise vector; $\mathit{T}$ is the text feature; ${\mathbb{P}}_r,{\mathbb{P}}_z,{\mathrm{a}\mathrm{n}\mathrm{d} \mathbb{P}}_{mis}$ denote the real data, Gaussian, and mismatched data distributions, respectively; and $k$ and $p$ are two hyperparameters to balance the effectiveness of the gradient penalty. The DSTF-GAN is trained using Algorithm 2.

Algorithm 2. Pseudocode for the training of DSTF-GAN.

Deep sparse and text fusion generative adversarial network: Adam optimizer with $\left({\beta }_1,{\beta }_2\right)=\left(\mathrm{0.5,0.999}\right),$ $lr\_G=0.0001,lr\_D=0.0003$

1: Initialize discriminator parameters ${\theta }_D$ and generator parameters ${\theta }_G$

2: for number of training epoch do

3:  sampling real data pairs $\{\left(x_1,{Text}_1\right),\dots ,\left(x_m,{Text}_m\right)\}$, latent variables $\{z_1,\dots ,z_m\}$

4:  extract semantics: $T_i=$ $\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}\_\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r} \left({Text}_i\right)$

5:  calculate loss function: ${\mathcal{L}}_D^i={\displaystyle \frac{1}{2}}{D}_{{\theta }_D}\left(G_{{\theta }_G}\left(z_i, T_i\right),T_i\right)+{\displaystyle \frac{1}{2}}{D}_{{\theta }_D}\left(x_i, T_j\right)-D_{{\theta }_D}\left(x_i, T_i\right)+k{\left(‖{\nabla }_{x_i}D\left(x_i, T_i\right)‖+‖{\nabla }_{T_i}D\left(x_i, T_i\right)‖\right)}^p$

6:  update the discriminator parameters: ${\theta }_D\leftarrow Adam({\nabla }_m{\displaystyle \frac{1}{m}}{\sum }_{i=1}^m{\mathcal{L}}_D^i,{\theta }_D,{\beta }_1,{\beta }_2,{lr}_G,{lr}_D)$

7:  sampling latent variables $\{z_1,\dots ,z_m\}$

8:  calculate loss function: ${\mathcal{L}}_G^i=-D_{{\theta }_G}\left(G_{{\theta }_G}(z_i, T_i), T_i\right)$

9:  update the discriminator: ${\theta }_G\leftarrow Adam({\nabla }_m{\displaystyle \frac{1}{m}}{\sum }_{i=1}^m{\mathcal{L}}_G^i,{\theta }_G,{\beta }_1,{\beta }_2,{lr}_G,{lr}_D)$

10: end for

#$\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}\_\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}$ is pre-trained from the Contrastive text–image matching model

The PR module leverages the image-to-image capabilities of the pre-trained latent stable diffusion model [27] to re-create the generated conceptual sketches as follows:

$$\tilde{\mathit{x}}=F_{LDM}\left(\mathit{x}\right)$$

(21)

where $\mathit{x}$ and $\tilde{\mathit{x}}$ denote the input conceptual sketch and the generated high-rise building image, respectively, and $F_{LDM}$ is the pre-trained latent stable diffusion model.

Additionally, by integrating the pre-trained text encoder, DSTF-GAN, and the stable-diffusion-v1-5 model, we have developed a user-friendly work interface within the Python framework. This interface facilitates the seamless transition from input text prompts to high-rise building renderings. Detailed descriptions of the interface and workflow are provided in Appendix A, as illustrated in Figure A1.

3. Experiments and Results

This section provides an overview of the datasets used in the experiments. Subsequently, the training details of the DCE and AAG modules are presented. To assess the efficacy of the proposed intelligent design approach based on textual descriptions, we systematically conducted a sequence of experiments. Specifically, for the DCE module, ablation studies showed that data augmentation and the semantic re-extraction layer enhanced the accuracy of architectural design semantic extraction, resulting in a more precise textual encoder for semantic extraction. In the case of the AAG module, qualitative experiments confirmed that our method effectively generates architectural sketches consistent with the provided textual semantics. By leveraging a pre-trained diffusion model to render the generated sketches, we produced realistic high-rise building renderings that better capture the underlying semantic nuances compared to direct diffusion model generation. Furthermore, comparisons with existing real-life buildings were conducted to demonstrate the effectiveness of the proposed method.

3.1. Sketch–Text Dataset

To address the challenge of generating the appearance of high-rise buildings, we required a dataset containing paired high-rise building sketches and the corresponding textual descriptions. A total of 856 images were meticulously gathered from online sources, with a distinct emphasis on the top 1000 tall buildings worldwide. The images of high-rise buildings represent real-world structures from various countries and regions, including a variety of architectural styles (such as postmodernism, modernism, and structuralism) and functional types (including commercial and cultural buildings). This diversity ensures that the model can generalize the architectural characteristics of high-rise buildings in different styles and cultural contexts. The initial preprocessing step involved resizing the images and removing their backgrounds. Subsequently, conceptual sketches were generated using the eXtended difference-of-Gaussians operator (XDoG), as shown in Figure 6. The underlying procedure of the XDoG operation can be outlined as follows [33]:

$$g\left(x\right)=S_{\sigma ,k,p}\left(\mathit{x}\right)\ast I\left(\mathit{x}\right)$$

(22)

$$S_{\sigma ,k,p}\left(\mathit{x}\right)=(1+p)G_{\sigma }\left(\mathit{x}\right)-p{G}_{k\sigma }\left(\mathit{x}\right)$$

(23)

$$G_{\sigma }\left(\mathit{x}\right)={\displaystyle \frac{1}{2\pi {\sigma }^2}}\exp (-{\displaystyle \frac{{‖\mathit{x}‖}^2}{2{\sigma }^2}})$$

(24)

where $I\left(\mathit{x}\right),\mathit{x}$, and $g\left(\mathit{x}\right)$ denote the original image, a two-dimensional coordinate, and the filtered sketch generated by the XDoG operator, respectively; $S_{\sigma ,k,p}\left(\mathit{x}\right)$ is the reparameterization of the XDoG filter for simplified control; and $\sigma ,k,p$ are the control parameters.

Textual descriptions included architectural style (e.g., modernism, futurism, or postmodernism), high-rise type (e.g., high-rise or tower), architectural design manner (e.g., symmetrical, misaligned, or segmented), architectural structure (e.g., podium or revolving floor), and other common design concepts to denote the abstract design concepts. To mitigate potential overfitting, we leveraged the pre-trained latent stable diffusion model and ChatGPT 3.5 from Open AI to augment each sketch–text pair, as shown in Figure 7. For image augmentation using the pre-trained diffusion model, we leveraged the image-to-image functionality of stable-diffusion-v1-5, adjusting the output correlation coefficient to preserve the overall structure while modifying only the finer details, resulting in 32 variations per sketch. For text augmentation with ChatGPT, we rephrased each textual description into 10 semantically equivalent but differently phrased versions. During this process, we ensured that the rephrased sentences maintained the original semantics by applying the following established conditions, including using synonyms, reordering words, and adjusting the phrasing, aiming to generate diverse expressions with the same semantic content. Subsequently, we conducted manual evaluations to ensure that each of the 10 augmented text prompts for a sketch preserved the original meaning.

3.2. Training Details

DCE module. The image encoder utilized 32 attention heads, whereas the text encoder employed 16 attention heads. Training was performed with a batch size of eight and a total of 800 epochs. In the training of the CTIMM, the dataset was partitioned into a training set constituting $90\mathrm{\%}$ of the data and a validation set representing the remaining $10\mathrm{\%}$. This division was randomly conducted multiple times for robust assessment. The optimization was conducted using an AdamW [34] optimizer with parameters ${\beta }_1=0.9,{\beta }_2=0.999,weight\_decay=0.25$. The learning rate was set to 0.0001, and the learning rate schedule followed the CosineAnnealingWarmRestarts [35] with $T\_0=400$, $T\_mult=1$, $eta\_min=0,$ and $last\_epoch=-1$. The experiments were conducted on a workstation equipped with a 16-core AMD Ryzen 9 3950X processor, an NVIDIA GeForce RTX 2080 Ti GPU, and 64 GB of DDR4 memory. The CTIMM was implemented using the open-source machine learning framework PyTorch Lightning 2.0.0.

AAG module. Training was performed with a batch size of eight and a total of 600 epochs. The division of the dataset was the same as that used by the DCE module to obtain the pre-trained text encoder. In the training process of the DSTF-GAN, the Adam [36] optimizer was utilized with ${\beta }_1=0.5$, ${\beta }_2=0.999$. The learning rates for the generator and discriminator were set as ${lr}_G=0.0001$ and ${lr}_D=0.0003$, respectively. The experiments were conducted on a workstation equipped with a 16-core AMD Ryzen 9 3950X processor, NVIDIA GeForce RTX 2080 Ti, and 64 GB of DDR4 memory. The DSTF-GAN was implemented using PyTorch Lightning. The pre-trained latent stable diffusion model was employed to render the generated conceptual sketches, with $strength=0.6~0.8$, $prompt=‘A high-rise building, 3D’,$ $scale=30$, $ddim\_steps=100$.

3.3. Experimental Results

Training the text encoder. To evaluate and acquire an efficient pre-trained text encoder, the metric should signify the highest cosine similarity values between the corresponding image–text and text–image pairs within the validation set. Accuracy was computed using the following formula:

$$Acc={\displaystyle \frac{{\sum }_{i=1}^n{\sum }_{j=1}^{b}g({\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}_{\mathit{j},\mathit{}:}^{\mathit{i}} ,{\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}_{:,\mathit{j}}^{\mathit{i}} ,j)}{n\times b}}$$

(25)

$$g\left(\mathit{x},\mathit{}\mathit{y},j\right)=\left\{\begin{array}{l}1 , if argmax\left(\mathit{x}\right)=j and argmax\left(\mathit{y}\right)=j \\ 0 , otherwise \end{array}\right.$$

(26)

where ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}^{\mathit{i}}$ is the cosine similarity matrix of the $i$-th batch; ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}_{\mathit{j},:}^{\mathit{i}}$ is the $j$-th row of ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}^{\mathit{i}}$, representing the similarity of the $j$-th text to every sketch in the batch; ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}_{:,\mathit{j}}^{\mathit{i}}$ is the $j$-th column of ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}^{\mathit{i}}$, representing the similarity of the $j$-th sketch to every text in the batch; $n$ and $b$ are the total number of batches and the batch size, respectively; and $\mathit{x},\mathit{y}$ denote two vectors. A higher $Acc$ indicates a stronger alignment between the sketch and its corresponding text description, reflecting more accurate semantic extraction.

The dataset was randomly split into training and validation sets on five occasions, and the model’s matching accuracy was evaluated for each partition. During training, we randomly selected a sketch–text pair from a set of 32 generated sketches and the corresponding 10 distinct textual descriptions for each epoch. Figure 8 shows the $Acc$ comparisons of the CTIMM under various dataset divisions, incorporating a semantic re-extraction layer and different data augmentation techniques, including random image scaling and cropping (DA) during the training process, the pre-trained latent stable diffusion model for image augmentation (SD), and ChatGPT for text augmentation (GPT). The experimental results on the validation set demonstrate that these techniques consistently improved the matching accuracy between sketches and text descriptions, regardless of the dataset partitioning. Figure 9 shows the matching outcomes of one batch in Division 1 of the dataset. The results demonstrate that the pre-trained text encoder effectively extracted design concepts from textual descriptions and provided the necessary conditions for the subsequent generation of high-rise building images.

Generation of high-rise building images. During the training of the DSTF-GAN, we selected the data partition that achieved the highest $Acc$ along with the corresponding text encoder, as this partition is believed to better capture the features of high-rise buildings described in the textual descriptions. The performance of the trained DSTF-GAN was then qualitatively assessed on the validation set to evaluate its generative capabilities. Figure 10 shows the ability of the DSTF-GAN within the AAG module to produce a varied array of semantically consistent conceptual sketches of high-rise buildings using the same design concepts. However, the implicit biases present in the dataset can affect the generated sketches. For example, since the tower-type buildings in the dataset are primarily cultural landmarks, inputting the term “tower” tends to generate sketches that closely resemble these cultural structures. Figure 11 compares the generated conceptual sketches with the real sketches in the training dataset, demonstrating that the DSTF-GAN does not memorize the samples in the dataset but rather understands the semantics of textual descriptions and the features of the high-rise sketches. Notably, the generated sketches exhibited uniqueness, indicating that the DSTF-GAN can learn the attributes of the design concepts in the training dataset and does not lose the creative requirements of the architectural design process. Subsequently, as shown in Figure 12, the PR module effectively transformed the generated conceptual sketches into lifelike high-rise buildings using the pre-trained stable-diffusion-v1-5 model, with parameters set to $strength=0.6~0.8$, $prompt=‘A high-rise building, 3D$’, $scale=30$, and $ddim\_steps=100$. The experimental outcomes confirmed the practical viability of the proposed approach for generating a diverse and authentic collection of high-rise building concepts from textual descriptions. Figure 13 provides a direct comparison of the high-rise building images generated by the pre-trained latent diffusion model and the proposed method. The pre-trained model employs the text-to-image functionality of the stable-diffusion-v1-5 model, with the input consisting of the text description corresponding to the sketch, using parameters $scale=5$ and $ddim\_steps=100$. The results demonstrate that our approach generates high-rise building renderings with improved semantic consistency compared to those produced directly from the text-to-image functionality of the diffusion model, thus highlighting the superior performance of the proposed method. The limitations of the pre-trained latent diffusion model, such as lacking a comprehensive understanding of architecture and precise correspondence between architectural design concepts and visual features, were evident. In contrast, the proposed method, which utilizes conceptual sketches as an intermediary representation, proved to be a more effective approach.

Authenticity assessment of generated results. To further validate the authenticity of the generated results, an investigation is conducted to assess the applicability of the generated architectural sketches and renderings by comparing them with real high-rise buildings not included in the dataset. Figure 14 demonstrates the utility of the generated sketches and renderings, showing similarities with existing buildings. This indicates that the proposed methodology captures essential characteristics found in real buildings while maintaining innovation and diversity in conceptual design. For instance, Figure 14a highlights similarities between a generated rendering sample and Metropol Tower Istanbul in the Ataşehir district of Istanbul, Turkey, showcasing a cutting architectural design approach at the top and similarly decorated surfaces. These findings validate the feasibility of applying this method.

4. Discussion

4.1. Comparison of Different Text Fusion Methods

To demonstrate the efficacy of the proposed approach of fusing the textual semantics and latent visual features of sketches in the context of the sketch generation task, we conducted a comparative analysis with two other cutting-edge single-stage text fusion methodologies, namely DF-GAN [24] (denoted as “DF”) and DiverGAN [26] (denoted as “Diver”), within the framework of the DSTF-GAN. DF utilizes two MLPs to predict language-conditioned channel-wise scaling parameters $\mathit{\gamma }$ and shifting parameters $\mathit{\theta }$, which are generated from the concatenation of semantic vectors $\mathit{T}$ and noise vectors $\mathbf{z}$, to operate the feature maps. Diver utilizes attention mechanisms to derive reshape operators at both the channel and pixel levels, thereby facilitating the re-weighting of the visual feature maps. We replaced the text fusion module in the DSTF-GAN with the “DF” and “Diver” fusion techniques, while keeping other components unchanged, and evaluated the effectiveness of the proposed text semantic fusion method using quantitative metrics, including FID and the newly introduced CM precision.

The Fréchet Inception Distance (FID) [37] was employed to evaluate the different fusion methods. The FID measures the quality of the generated images by computing the Fréchet distance between the distribution of generated samples and the distribution of actual data. A lower FID value indicates that the synthetic images closely resemble the real images, reflecting their visual quality. The FID is calculated as

$$FID\left(r,f\right)={‖{\mu }_r-{\mu }_f‖}_2^2+Tr\left({\sum }_r+{\sum }_f-2{\left({\sum }_r{\sum }_f\right)}^{1/2}\right)$$

(27)

where $\sum$ and $\mu$ represent the covariance and feature-wise mean of samples, respectively, and $f$ denotes the real and synthetic images, respectively.

While FID evaluates the visual quality of generated sketches, it does not assess the semantic alignment between generated sketches and their corresponding textual descriptions. To address this limitation, we introduced contrastive matching precision (CM precision) as a complementary metric for evaluating text-to-conceptual sketch synthesis. To calculate the CM precision, we used the pre-trained CTIMM to measure the percentage of correct matches between the generated sketches and their corresponding input textual descriptions. For the new dataset consisting of the generated high-rise conceptual sketches and corresponding texts, the CM precision was calculated as follows:

$$\mathrm{C}\mathrm{M}-\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{{\sum }_{i=1}^{n_g}{\sum }_{j=1}^{b_g}g({\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}{\mathit{s}}_{\mathit{g}}}_{\mathit{j},\mathit{}:}^{\mathit{i}} ,{\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}{\mathit{s}}_{\mathit{g}}}_{:,\mathit{j}}^{\mathit{i}} ,j)}{n_g\times b_g}}$$

(28)

where ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}_{\mathit{g}}^{\mathit{i}}$ is the cosine similarity matrix of the $i$-th batch on the newly generated dataset; ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}{\mathit{s}}_{\mathit{g}}}_{\mathit{j},:}^{\mathit{i}}$ is the $j$-th row of ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}_{\mathit{g}}^{\mathit{i}}$, representing the similarity of the $j$-th text to every generated sketch in the batch; ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}{\mathit{s}}_{\mathit{g}}}_{:,\mathit{j}}^{\mathit{i}}$ is the $j$-th column of ${\mathit{l}\mathit{o}\mathit{g}\mathit{i}\mathit{t}\mathit{s}}_{\mathit{g}}^{\mathit{i}}$, representing the similarity of the $j$-th generated sketch to every text in the batch; $n_g$ and $b_g$ are the total number of batches and the batch size, respectively; and $g$ is expressed in Equation (26). A higher CM precision indicates a better match between the generated sketch and text semantics.

To calculate the FID, we generated five distinct sketches for each text description in the validation set by altering the noise vector $\mathit{z}$, and computed the FID by comparing the generated sketches with the corresponding real sketches. To calculate the CM precision, we grouped the generated sketch–text pairs into batches of eight, calculated the CM precision for each batch, and then averaged the results across all batches to obtain the final CM precision.

Table 1. FID and CM precision results of three fusion methods.

Text Fusion Method	FID ↓	CM Precision (%) ↑
Diver	116.99	6.81
DF	25.88	27.64
DSTF (the proposed method)	25.73	36.08

presents the experimental results, revealing that the proposed DSTF-GAN outperformed the other methods in terms of the FID and CM precision metrics. These quantitative comparisons provide evidence for the efficacy of the fusion approach employed in the DSTF-GAN for text-to-sketch synthesis. The results underscore that incorporating the sparsity of sketches into the generative process significantly enhances quality and semantic fusion. The conceptual sketches generated from various fusion methods using the same descriptions are shown in Figure 15. The results show that the Diver fusion method only captured certain architectural features and failed to generate complete high-rise sketches, whereas the DF fusion method could only generate realistic but not well-aligned high-rise building sketches. For instance, as depicted in Figure 15b, both DF and the DSTF-GAN produced high-rise building sketches with a broader base and a pole. However, only the DSTF-GAN incorporated a dome structure at the top, demonstrating the superior accuracy of the proposed DSTF fusion method in the synthesis of sketches.

4.2. Component Analysis

Section 4.1 has demonstrated the efficacy of the DSTF-GAN fusion method compared with two other leading single-stage text fusion approaches. To gain deeper insight into the conceptual sketch generation process, we conducted ablation studies to assess the individual effectiveness of each component within the DSTF-GAN, namely the SF block, TF block, and the order of these two blocks. Three variations of DSTF were created: “DSTF w/o SF block”, “DSTF w/o TF block”, and “DTSF”. These variations indicated the removal of the SF Block, the removal of the TF Block, and the swapping of the two blocks.

To comprehensively analyze each component, was employed an additional metric, the LPIPS metric [38], in addition to the FID and CM precision. LPIPS leverages pre-trained networks to assess the diversity of the generated sketches. This was accomplished by quantifying the average feature distance between the feature maps of the synthetic images. A higher LPIPS value signifies a greater dissimilarity between pairs of synthetic sketches, indicating higher diversity. The calculation methods for the FID and CM precision follow the same procedure as described in Section 4.1. The comparative results of the ablation studies are listed in

Table 2. FID, CM-precision, and LPIPS results of component analysis.

Architecture	FID ↓	CM Precision (%) ↑	LPIPS
DSTF (the proposed method)	25.73	36.08	0.2200
DSTF w/o SF block	26.51	30.83	0.0588
DSTF w/o TF block	32.43	4.31	0.3095
DTSF	29.43	32.01	0.2235

.

Effect of the SF block. Compared with DSTF w/o SF block, DSTF enhanced the LPIPS value from 0.0588 to 0.2200, indicating that the SF block effectively increased the diversity of sketches generated from the same description by introducing sparsity to the feature maps. Furthermore, the incorporation of sparsity during the fusion process improved the CM precision from $30.83\mathrm{\%}$ to $36.08\mathrm{\%}$. This demonstrated that integrating sparsity into the fusion process enhances the semantic consistency between the text and the generated sketches.

Effect of the TF block. The TF block improved the CM precision from 4.31% to 36.08% compared with DSTF w/o TF block. Additionally, the TF block contributed to a decrease in the FID from 32.43 to 25.73, indicating an improvement in the overall quality of the generated sketches. The TF block achieved this by incorporating channel- and position-level adjustments based on text semantics, effectively enhancing the fusion process and generating more accurate and visually appealing sketches.

Swapping the order of the two blocks. A comparison between DSTF and DTSF revealed that the selected order of the two blocks had a notable impact on the performance of the model. DSTF achieved an improved CM precision of $36.08\mathrm{\%}$ compared with $32.01\mathrm{\%}$ for DTSF. Additionally, DSTF exhibits a decreased FID of 25.73 in contrast to 29.43 for DTSF. These results indicated that the order of combining sparsity and fusing text semantics in the sketch generation process plays a crucial role in generating high-rise building sketches that are more semantically consistent.

5. Application

To assess the potential utility of the proposed pipeline to assist architects in the conceptual design phase during the professional high-rise building design process, architects are invited to develop a 3D conceptualization of a rendered building appearance in an office building type using the SketchUp Pro 2021 software. Figure 16 illustrates the elevation perspective and aerial views of the created office building model. The architects perceive this approach as significantly enhancing the efficiency of the early conceptual design phase for buildings while providing them with more design inspiration.

6. Conclusions

This paper proposes a novel method for assisting architects in the conceptual design stage by intelligently generating high-rise building concepts from textual descriptions. The model comprises a design concept extraction (DCE) module and an architectural appearance generation (AAG) module. The DCE module adopts contrastive learning to train a contrastive text–image matching model (CTIMM) to obtain a pre-trained text encoder that can accurately extract design concepts. Furthermore, the AAG module, which consists of a deep sparse and text fusion generative adversarial network (DSTF-GAN) and a pre-trained rendering (PR) module, transforms the extracted semantics into high-rise building sketches and renders them as realistic high-rise appearances. The effectiveness of the proposed method is validated through qualitative and quantitative experiments, as well as the comparison with existing buildings. Furthermore, the feasibility of applying the method to assist architects in the conceptual design phase of a building is demonstrated through 3D modeling of a generated high-rise rendering. The proposed method demonstrates the potential of leveraging deep learning techniques to facilitate the generation of high-rise building concepts from textual descriptions, offering valuable support to architects during the initial design process.

Limitations. The dataset used in this study consists of only 856 real-world high-rise sketches paired with corresponding textual descriptions. Expanding the dataset could incorporate a wider range of architectural styles and features, leading to more diverse and robust high-rise sketch generation. Furthermore, the rendering process using the pre-trained diffusion model introduces some uncertainty, which may result in variations in architectural style and features, causing a few discrepancies between the generated renderings and the original textual descriptions. Although the visual plausibility of the generated sketches has been validated through 3D modeling and comparisons with existing buildings, incorporating qualitative or quantitative evaluations from experienced architects could further enhance the validation of the method’s effectiveness.

Future work. In future work, we will collaborate with professional architects to enhance the dataset and evaluate the generated designs, employing methods similar to focus groups [39]. By incorporating additional architectural elements and inspirations, we aim to improve the model’s ability to generate high-rise buildings with diverse styles, cultural features, and structural variations while also improving its practical applicability. Additionally, we will refine the diffusion model’s rendering process for more controlled outputs and enable interactive editing. We also plan to investigate the generation of 3D architectural massing by decomposing the overall volume into sub-blocks and representing it through a graph-based representation. During the generation process, spatial and functional constraints will be incorporated to ensure design feasibility. Additionally, by integrating the generated 2D high-rise renderings, we aim to achieve a cohesive design that synthesizes both massing and form, contributing to advancements in 3D architectural design.