From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model

Abstract

The role of image data in knowledge extraction and representation has become increasingly significant. This study introduces a novel methodology, termed Image to Graph via Large Language Model (ImgGraph-LLM), which constructs a knowledge graph for each image in a dataset. Unlike existing methods that rely on text descriptions or multimodal data to build a comprehensive knowledge graph, our approach focuses solely on unlabeled individual image data, representing a distinct form of unsupervised knowledge graph construction. To tackle the challenge of generating a knowledge graph from individual images in an unsupervised manner, we first design two self-supervised operations to generate training data from unlabeled images. We then propose an iterative fine-tuning process that uses this self-supervised information, enabling the fine-tuned LLM to recognize the triplets needed to construct the knowledge graph. To improve the accuracy of triplet extraction, we introduce filtering strategies that effectively remove low-confidence training data. Finally, experiments on two large-scale real-world datasets demonstrate the superiority of our proposed model.

1. Introduction

With the growing popularity of mobile devices, the State Grid Corporation of China has started using them to photograph and document operations in scenarios such as maintenance and emergency repairs. These photos are numerous and lack label information. Currently, the State Grid Corporation of China relies on manual comparison and identification to assess risks in actual operations. For example, Figure 1d reveals that an operator on a power pole is not wearing a safety belt. To identify risks in operational behavior, it is usually necessary for individuals with expert knowledge to compare the complex operational specifications described in business standards and the operational behavior depicted in captured images. To bridge the semantic gap between operational risks in images and business standards, knowledge graphs have become a feasible solution [1,2]. By constructing a knowledge graph of the entities and their relationships depicted in an image, it is possible to compare these graphs with those of images that meet business standards, thereby identifying potential operational risks. For instance, we constructed a knowledge graph for the actual image Figure 1d. By comparing it with the standard operation images in Figure 1c,e, we observe that the entities in Figure 1d are the same as those in the standard operation image in Figure 1c, but the relationship between the entity of the safety belt and the entity of the telegraph pole is missing. Therefore, we can identify the risk of not wearing a safety belt. Potential operational risks include issues such as missing entities or incorrect relationships (e.g., not wearing a safety belt). Thus, we propose an unsupervised task of building a knowledge graph for each unlabeled image in order to improve the efficiency of identifying operational risks, eliminating the need for experts to manually compare images.

Knowledge graphs [3,4,5], as an important tool for describing semantic structures including entities and their relationships, have been widely used in various intelligent applications, such as search engines and question-answering systems. By constructing a network of relationships between entities, knowledge graphs provide support for complex queries and decisions, making information retrieval and processing more efficient and accurate [6,7]. The existing method of knowledge graph construction (KGC) mainly relies on textual data [4,8], such as extracting entities and relationships from corpora through text mining techniques like those used in Wikidata [9,10]. With the rise of multimodal data, including non-textual data such as images, attention to its information content and expressive power is increasing. The multimodal knowledge graph construction method [1,2] utilizes multiple data types to provide more comprehensive information support for understanding entities and their relationships, such as attributing a set of images to each entity. Nevertheless, as shown in Figure 1a,b, existing knowledge graph construction [2,10,11] often focuses on using a large amount of textual or multimodal data to build a large and comprehensive knowledge graph, and these methods are more focused on how to model the relationships between multiple data. In contrast, as shown in Figure 1c–e, we focus on constructing a knowledge graph for unlabeled individual data (each image), necessitating a detailed examination of the information within a single image. Therefore, existing methods are not suitable for the unsupervised task of constructing an independent knowledge graph for single data.

With the development of large language models (LLMs) in image understanding, such as MiniGPT-4 and Llama-vid [12,13], which have demonstrated the ability to accurately and detailedly describe images, using LLMs to understand images has become a feasible direction for research and application. At the same time, recent research has begun to explore the use of LLMs to assist in building knowledge graphs [5,14], particularly in modeling text data, and has made some progress. Most mainstream methods still use serialized sequences as inputs and have not fully explored the non-sequential structure of images. This limitation poses challenges in using LLMs to build a knowledge graph based on a single image, especially in effectively combining the non-sequential information in the image with the modeling of the knowledge graph. This paper aims to explore how to utilize LLMs’ ability to understand images and their potential to construct knowledge graphs, in order to improve the efficiency and accuracy of single-image-based KGC.

In this paper, we propose an innovative approach for constructing an image-to-graph knowledge representation (ImgGraph-LLM) through an LLM, which constructs an independent knowledge graph for each unlabeled image in the dataset. This method first uses a self-supervised learning strategy to compile the training set and then fine-tunes the LLM to recognize key information related to knowledge graph construction. Then, the fine-tuned LLM can directly extract relationship triplets, namely entities and their relationships, from images and then construct a knowledge graph. To improve the accuracy and confidence of triplet extraction, we adopt filtering strategies to filter out triplets with low confidence, thereby enhancing the accuracy of knowledge graph construction. This image-based knowledge graph construction method provides a new perspective for intelligent systems, with images as the center, expanding the application of knowledge graphs.

2. Related Work

2.1. Knowledge Graph Construction

Knowledge graph construction (KGC) involves creating a structured representation of knowledge, which includes identifying entities and their interrelationships [4,5,10]. Existing methods can be divided into two categories. The first category involves constructing knowledge graphs based on textual data. With the advancement of language models, recent works have employed the language models to process text data and generate knowledge graphs, such as BT5 [15], ReGen [16], and Grapher [17]. For example, Grapher [17] uses a pre-trained T5 model [18] to construct a multi-stage framework that efficiently extracts knowledge graphs from text descriptions. The second category focuses on using multimodal data to construct knowledge graphs [1,2], as seen in methods like Richpedia [19], MMKG [20], VisualSem [21] and TIVA-KG [22]. For example, TIVA-KG [22] produces universal knowledge graphs that encompass text, images, videos, and audio. Our approach differs from these existing solutions in that it is the first attempt to construct a knowledge graph from a single image, rather than integrating all data into a comprehensive knowledge graph.

2.2. LLMs for Image Understanding

The integration of LLMs as decoders in vision–language tasks has surged in popularity, leveraging cross-modal learning to facilitate knowledge exchange between the language and visual domains [23,24,25]. With the release of GPT-4 [26] by OpenAI—demonstrating enhanced visual comprehension and reasoning skills following pre-training on extensive image–text datasets—other models like VisionLLM [27], LLaMA-VID [13], and MiniGPT-4 [12] have also significantly advanced the field. These open-source models not only excel in performance but also serve as crucial tools for tailored fine-tuning. Inspired by these works, we employ an LLM to model images and complete knowledge graph construction.

3. Problem Formulation

Now, we define the knowledge graph construction problem. Let the images $I=\{i_1,\dots ,i_N\}$ denote the data sources of knowledge graph construction, where N is the number of images. The knowledge graph construction $KGC$ is a procedure that maps each image $i\in I$ onto a knowledge graph $G_i\in G$, denotes as $KGC\left(i\right)=G_i$. The knowledge graph set G is defined as $G=\{G_{i_1},\dots ,G_{i_N}\}$. Notably, the input data only contain the images without any other data, such as labels or descriptions. Hence, the knowledge graph construction problem is an unsupervised structure prediction task.

Although there has been considerable research on knowledge graph construction, there are distinct differences between our task definition and existing work, specifically manifested in the following aspects:

• Different data source: Unlike existing methods that primarily utilize text or multimodal data, our approach focuses exclusively on leveraging image data for knowledge graph construction.
• Different training method: While many existing methods rely on supervised or semi-supervised approaches, we employ an unsupervised methodology tailored to extracting knowledge directly from images.
• Different construction results: Rather than constructing a comprehensive knowledge graph from an all-input dataset, our method generates a knowledge graph from each individual input datum, specifically from a single image.
• Different implementation method: Traditional methods typically involve multistep operations such as entity recognition and relation extraction to piece together a knowledge graph. In contrast, we develop an end-to-end model that directly produces a knowledge graph from image data, streamlining the process.

4. Method

Our proposed ImgGraph-LLM comprises three components. Firstly, we extract self-supervised information from unlabeled image data to construct the training dataset. Secondly, we design an iterative fine-tuning method to use the constructed training data to fine-tune the LLM. Thirdly, we employ mutual information to implement filtering strategies. Finally, as shown in Figure 2, the fine-tuned model is capable of generating a knowledge graph for each image.

4.1. Training Data Generation

The goal of training data generation is to use self-supervised information to generate data for fine-tuning the LLM. When processing image data from the State Grid, we propose two key assumptions to ensure that the generated data can be effectively used for model fine-tuning and can help the LLM remove irrelevant information from images and focus on information related to the knowledge graph.

Assumption 1.

In general, the photos place the main operational behavior in a central position, while the content around the photos (such as background information and unrelated objects) is considered irrelevant information. We should enable the LLM to effectively discriminate and filter out this irrelevant information.

Assumption 2.

This assumption emphasizes the importance of main entities and their relationships in photos. Due to the potential impact of hallucinations on the LLM, they may result in missing or incorrectly modeled entities and relationships in actual images during output. Therefore, our method needs to have high recognition and reinforcement capabilities to ensure that the model can accurately capture and express key entities and their relationships in photos.

Guided by the above two assumptions, the aim is to promote the application of self-supervised operations in image understanding and knowledge graph construction, providing efficient and accurate training data for fine-tuning the LLM.

We design two self-supervised operations for generating training data to meet Assumption 1 and Assumption 2, focusing on the layout of different information in the image. Given the image $i\in I$, based on image size and edge width parameters, the center region $i_c$ and edge region $i_e$ of the image are divided. The size of the image is $H\times W$ with the edge width X; then, the size of each area can be calculated as follows: the height of the central region $i_c$ is $H-2X$ and the width of the central region $i_c$ is $W-2X$. Therefore, the size of the central region $i_c$ is $(H-2X)\times (W-2X)$. The edge region $i_e$ includes four edges, each with the width X. The height of the upper and lower edges is X. The width of the left and right edges is X. Therefore, the size of the edge region $i_e$ is $(2X\times W)+(2X\times (H-2X\left)\right)$. Finally, the image can be split as $i=\{i_c,i_e\}$.

Since our data lack label information, there are no labels to indicate center or edge regions. To leverage the self-supervised information within the image itself, we set the edge width X to a random value within the range of 0.05–0.15. The minimum value of 0.05 is chosen to ensure that blank space and irrelevant information are excluded, while the maximum value of 0.15 maintains the integrity of the central area. During experiments, we generated multiple random values for the edge width X for each image, creating multiple center and edge regions. For simplicity, we used the unified notation $i=i_c,i_e$ to denote the center and edge regions derived from the image, without distinguishing cases with varying edge widths.

Self-supervised Operation 1: Irrelevant information from the edge region $i_e$ is deducted. Given the original image i, irrelevant information around it (such as background and extra objects) is removed, and only the main operational behavior and key entities in the center region $i_c$ are retained. During the training process, the model learns to only focus on the important information in the center region $i_c$, while ignoring the unrelated information around it, thereby improving the model’s accurate recognition ability for key entities and their relationships.

We design a set of image transformations, $T_e$, which include transformations such as cutout. Our goal is to remove irrelevant information from the edge region $i_e$ using these image transformations $T_e$. In the transformations $T_e$, random cutout and color transformation are two key operations. The random cutout operation allows us to randomly remove a portion of the image, which often contains background or irrelevant details, thereby increasing the model’s focus on the core area of the image. Meanwhile, through color conversion, we can fill the cutout areas with blocks of different colors, ensuring that the overall structure and content of the image remain unchanged even after the color change. Through these methods, we aim to reduce interference from irrelevant information in the edge regions, ensuring the effectiveness and accuracy of model training.

Self-supervised Operation 2: Relevant information from the center region $i_c$ is disrupted. Given the original image i, entities and relationships in the center region $i_c$ are disturbed, such as by removing some entities. After that, the entities or relationships are different while maintaining the same background and operational behaviors. Using these constructed data can guide the LLM to correctly identify entities and relationships, thereby avoiding recognition errors caused by hallucinations and other common-sense.

We design a set of image transformations, $T_c$, which include jigsaw puzzles [28,29] and geometric transformations. Specifically, a jigsaw puzzle [28,29] is a classical self-supervised learning task that involves rearranging shuffled image patches into their correct spatial order to reconstruct the original image. Our goal is to disturb relevant information from the center region $i_c$ using these image transformations $T_c$. Specifically, we employ jigsaw puzzles to displace scrambled patches within the central region as shown in Figure 3. This scrambled image serves to not only disrupt entity relationships but also hinder entity recognition by controlling the size of these scrambled patches. For instance, setting the scrambled patches to 32 blocks divides the entities within the image into various scrambled segments, rendering entity recognition and their relationships unfeasible. Due to the fact that geometric transformations can effectively disrupt the structure of an image without changing the actual pixel information, we choose operations that have a significant impact on the original image, such as random cropping and flipping. The original central area can be viewed as a global view, while the transformed image forms a local view. Through these transformation operations, we can significantly change the layout and content distribution of the image, making it difficult to recognize the original entities and their relationships after transformation. For example, random cropping may cut the information of the entity into different local segments, while flipping can change the relation of entities.

Pseudo-label Generator: This is an unsupervised image labeling technique. We assume that there is a corresponding label, $l_i$, for an image, i, represented as $l_i=F\left(i\right)$. In the process of generating training data, we perform various transformation operations on the image i. Specifically, for the transformations $T_e$ around the image, the key information is not destroyed and only irrelevant information related to the construction of the knowledge graph is removed. Therefore, after these transformations, the labels of the image should still remain unchanged, that is, $l_{T_e\left(i\right)}=F\left(T_e\left(i\right)\right)$. Regarding the transformation $T_c$ of the image center, these transformations will completely destroy the information of entities and their relationships. Therefore, it is impossible to construct a meaningful knowledge graph from the images that have undergone these transformations. Therefore, the labels of these images will be set to empty, represented as $l_{T_c\left(i\right)}=F\left(T_c\left(i\right)\right)$. Through this approach, we can appropriately label transformed images in a way that reflects their utility for training data in constructing knowledge graphs.

We generate a dataset D using self-supervised operations and pseudo-labels. For any image i, we apply the transformations $T_c$ and $T_e$ to generate multiple images. Due to the randomness of the transformations $T_c$ and $T_e$, the same transformation of $T_c$ or $T_e$ produces different image results. For simplicity, we randomly generate M instances for each transformation operation, denoted as $T_c\left(i\right)=\{T_c^1\left(i\right),\dots ,T_c^m\left(i\right),\dots ,T_c^M\left(i\right)\}$ and $T_e\left(i\right)=\{T_e^1\left(i\right),\dots ,T_e^m\left(i\right),\dots ,T_e^M\left(i\right)\}$. The corresponding pseudo-labels are $L_{T_e\left(i\right)}=\{l_{T_c^1\left(i\right)}=F\left(T_c^1\left(i\right)\right),\dots ,l_{T_c^m\left(i\right)}=F\left(T_c^m\left(i\right)\right),\dots ,l_{T_c^M\left(i\right)}=F\left(T_c^M\left(i\right)\right)\}$ and $L_{T_e\left(i\right)}=\{l_{T_e^1\left(i\right)}=F\left(T_e^1\left(i\right)\right),\dots ,l_{T_e^m\left(i\right)}=F\left(T_e^m\left(i\right)\right),\dots ,l_{T_e^M\left(i\right)=F\left(T_e^M\left(i\right)\right)}\}$.

The generated dataset is based on assumed labels ($L=l_{i_1},\dots ,l_{i_N}$) and is represented as $\mathcal{D}\left(L\right)$:

$$\begin{array}{cc}\hfill \mathcal{D}\left(L\right)=& \{(i_1,l_{i_1};T_c\left(i_1\right),L_{T_c\left(i_1\right)};T_e\left(i_1\right),L_{T_e\left(i_1\right)}),\hfill \\ & \dots ,\hfill \\ & (i_N,l_{i_N};T_c\left(i_N\right),L_{T_c\left(i_N\right)};T_e\left(i_N\right),L_{T_e\left(i_N\right)})\}.\hfill \end{array}$$

Please note that the original dataset does not contain labels. The labels in the generated dataset $\mathcal{D}\left(L\right)$ are pseudo-labels created based on assumed labels ($L=l_{i_1},\dots ,l_{i_N}$) and self-supervised operations (such as $L_{T_c\left(i_1\right)},L_{T_e\left(i_1\right)}$). During the model training phase, these pseudo-labels dynamically change to fully utilize the self-supervised information.

4.2. Iterative Fine-Tuning

Based on the generated training data $\mathcal{D}\left(L\right)$, we fine-tune the large language model (LLM) using the generated data $\mathcal{D}\left(L\right)$. Due to the lack of real labels for each piece of training data (image), existing data cannot be directly used for supervised fine-tuning methods. For this purpose, we design an iterative fine-tuning method. By iteratively utilizing self-supervised information of data $\mathcal{D}\left(L\right)$, the model can learn effective representations and features even without explicit labels, gradually optimizing its ability to construct knowledge graphs and alleviating the limitations of supervised learning caused by the lack of real labels.

Iterative Pseudo-label Supervision: In our dataset $\mathcal{D}\left(L\right)$, each image i lacks a corresponding label $l_i\in L$. To address this issue, we design an iterative self-supervised training method. Firstly, we use a pretrained LLM to generate a preliminary knowledge graph of the image as the initial label $l_i^1$ (first epoch). Although the initial label $L^1=\{l_{i_1}^1,\dots ,l_{i_N}^1\}$ generated is not entirely accurate, due to the presence of self-supervised information in the generated training dataset $\mathcal{D}\left(L^1\right)$, such as the generated pseudo-labels $L_{T_c\left(i\right)}^1$, the fine-tuned LLM can better distinguish between background information and entity information. During the fine-tuning process, the model gradually optimizes and adjusts the generated knowledge graph by learning the intrinsic structure and patterns in the data. Therefore, as the number of iterations increases, the knowledge graph constructed by the fine-tuned LLM gradually becomes more accurate and reliable. Through multiple epochs of the iterative self-supervised training method, we can ensure that the final generated knowledge graph has accuracy in expressing and capturing entities and their relationships. This iterative training strategy fully utilizes the concept of self-supervised learning, gradually improving the understanding ability of complex information in images through feedback and the correction of the model itself, thereby effectively improving the performance of the model.

Lightweight Fine-tuning: Currently, many studies are dedicated to proposing lightweight fine-tuning methods for LLMs to reduce expensive computational resource consumption. LoRA technology introduces an efficient fine-tuning method for LLMs by approximating weight updates using low-rank matrices, effectively reducing the number of parameters that need to be adjusted. Therefore, we employ LoRA as the technical means for the lightweight fine-tuning of the LLM. The objective of LoRA can be computed as Equation (1).

$$\underset{{\Theta }_{LoRA}}{max}\sum _{(x,y)\in \mathcal{D}\left(L\right)}\sum _{k=1}^{\left|y\right|}log\left(P_{{\Theta }_{LLM}+{\Theta }_{LoRA}}\left(y_t\right|x,y_{<t}\}\right),$$

(1)

where ${\Theta }_{LLM}$ represents the original LLM parameters, which are frozen. ${\Theta }_{LoRA}$ denotes the LoRA parameters. $\mathcal{D}\left(L\right)$ is the constructed training data. x and y represent the image data and the pseudo-label, respectively. For instance, for a given image, i, and its corresponding initial label, $l_i^1\in L^1$, $(x,y)$ would be $(i,l_i^1)$.

Iterative Training Processing: The training process of our model involves multiple iterations, as shown in Figure 4. To ensure the sufficient optimization of the LLM, we set the total number of iterations to K rounds. This means that the proposed model undergoes K complete fine-tuning training cycles. In each fine-tuning iteration, the LLM learns and adjusts parameters based on the current training data to continuously improve its performance and accuracy. Through such multiple iterations, we expect the LLM to gradually converge to an ideal state, achieving the expected accuracy and effectiveness. Setting the total number of iterations to K ensures that the model has sufficient time and opportunity to optimize and adjust, thereby improving the final performance.

In the initial training phase, we construct a training dataset containing images and corresponding pseudo-labels, denoted as $\mathcal{D}(L^1=F_{{\Theta }_{LLM}}\left(I\right))$, while the function $F_{{\Theta }_{LLM}}$ is our model with the parameters ${\Theta }_{LLM}$. Next, we utilize this initial training dataset $D^1$ for the first iteration to optimize the following loss function shown in Equation (2).

$$\underset{{\Theta }_{LoRA}^1}{max}\sum _{(x,y)\in \mathcal{D}\left(L^1\right)}\sum _{k=1}^{\left|y\right|}log\left(P_{{\Theta }_{LLM}+{\Theta }_{LoRA}^1}\left(y_t\right|x,y_{<t}\}\right).$$

(2)

Although the pseudo-labels in the training dataset are derived from the LLM, introducing self-supervised information into the dataset and using it as the input for the first fine-tuning training can enable the model to extract additional useful information from it, thereby enhancing its performance and generalization ability.

In the k iteration, we use the labels generated from the LLM model parameters ${\Theta }_{LLM}+{\Theta }_{LoRA}^{k-1}$ obtained from the $k-1$ iteration to construct a new dataset, $D(L^k=F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^{k-1}}\left(I\right))$. Specifically, we employ the model parameters ${\Theta }_{LLM}+{\Theta }_{LoRA}$ obtained from the $k-1$ iteration to annotate all images in the training data, generating a new training dataset, $D^k$, used for the k iteration. This process aims to improve model performance by optimizing the following loss function shown in Equation (3).

$$\underset{{\Theta }_{LoRA}^k}{max}\sum _{(x,y)\in D\left(L^k\right)}\sum _{k=1}^{\left|y\right|}log\left(P_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}\left(y_t\right|x,y_{<t}\}\right).$$

(3)

By incorporating the optimization results from the $k-1$ iteration as supervisory information, the labels of the training data $D\left(L^k\right)$ become more accurate. This iterative approach gradually adjusts and refines the model through fine-tuning new data distributions, thereby enhancing its predictive capability. This iterative fine-tuning process can be viewed as incremental learning, which proves particularly effective in adapting the model to continuously changing data distributions and label variations throughout the iteration process. This adaptability ultimately leads to improved prediction accuracy over time.

4.3. Filtering Strategies

Due to our iterative fine-tuning approach relying on self-supervised information introduced from the constructed training dataset, we design a filtering strategy to assist the LLM in removing low-confidence constructed training data during the fine-tuning phase, as shown in Figure 4. The core intuition is derived from information bottleneck theory, which suggests that when information is compressed through transformations, some useful signals for prediction may be lost.

We assume that the original image i contains a richer set of semantic and structural signals compared to its transformed counterpart $T_c\left(i\right)$, especially when $T_c$ deliberately removes or distorts salient parts of the image. Therefore, when these images are processed by the LLM, the output derived from the original image should retain higher mutual information (MI) with the original semantic content than that derived from the transformed image.

To quantify this, we use mutual information, $MI(X;Y)$, which measures the amount of shared information between two variables. In our case, X is the input image (either original or transformed), and Y is the LLM’s modeling result (e.g., predicted entities or relations). A higher MI indicates that the model’s output preserves more of the original input’s informative content. We propose the following three MI-based filtering strategies.

Strategy 1: The images $T_c\left(i\right)$ generated by the transformation $T_c$ should contain less information than the original image i after being modeled by the LLM. This is because the crucial information required for identifying entities and their relationships is removed from the images $T_c\left(i\right)$, thereby reducing the information content in the LLM’s output results.

Strategy 2: The operation $T_e$ is used to produce multiple images $T_e\left(i\right)$, inputting these images into LLM to obtain multiple results. Then, compare the consistency among these results. Since $T_e$ is designed to filter out irrelevant information, each transformed image should exhibit similar information content. Significant discrepancies among these results may indicate lower confidence levels in certain transformed images $T_e\left(i\right)$, necessitating their filtration.

Strategy 3: This strategy compares two types of transformation operations, $T_c$ and $T_e$. $T_e$ removes unnecessary information from the original image, while $T_c$ removes useful information. Consequently, in the LLM’s output results, the information content derived from $T_e$ should surpass that from $T_c$.

These filtering strategies enable the effective identification and removal of generated training data with low confidence, ensuring that only high-confidence data is used for model fine-tuning. The measurement method based on mutual information ($MI$) can effectively measure changes in the amount of information. Next, we explain how to use mutual information to implement the three filtering strategies for the k iteration model $F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}$, as shown in Equations (4)–(6).

$$MI(F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}\left(i\right),F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}\left(T_c^m\left(i\right)\right)>{\alpha }_1$$

(4)

$$MI(F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}\left(T_e\left(i\right)\right),F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}\left(T_e^m\left(i\right)\right))<{\alpha }_2$$

(5)

$$MI(F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}\left(T_e^m\left(i\right)\right),F_{{\Theta }_{LLM}+{\Theta }_{LoRA}^k}\left(T_c^m\left(i\right)\right)>{\alpha }_3,$$

(6)

$T_c^m\left(i\right)\in T_c\left(i\right)$ and $T_e^m\left(i\right)\in T_e\left(i\right)$. Here, ${\alpha }_1,{\alpha }_2$, and ${\alpha }_3$ are the hyperparameters. If Equation (4), Equation (5), or Equation (6) is not satisfied, then we need to delete the corresponding training samples. For example, Equation (4) measures the mutual information between i and $T_c^m\left(i\right)$. If this value is less than $alph{a}_1$, it indicates that i is similar to $T_c^m\left(i\right)$. Consequently, the image $T_c^m\left(i\right)$ is deemed to retain significant entities and relationships that are at risk of being overlooked. As a result, $T_c^m\left(i\right)$ fails to fulfill the necessary criteria and should be removed from the dataset. This exclusion ensures that $T_c^m\left(i\right)$ will not be part of the k iteration of the training process.

4.4. Model Discussion

1. Comprehensive Comparison with Existing Solutions: Unlike most current approaches that rely heavily on textual or multimodal data, our method exclusively utilizes image data as the sole input source for knowledge graph construction. This unique focus on visual information allows us to explore underutilized semantic cues embedded in images, providing a fresh perspective on visual knowledge representation. Moreover, while conventional methods often depend on supervised or semi-supervised training, our approach adopts an unsupervised learning framework, enabling the model to automatically extract structured knowledge from raw images without requiring annotated labels or human-defined schemas.

2. Theoretical Background and Innovation: The core innovation of our method lies in its image-centric, end-to-end KG construction pipeline. Traditional pipelines typically involve multi-stage operations such as entity recognition, attribute detection, and relation extraction, often designed separately and optimized independently. In contrast, our model learns a direct mapping from a visual input to a structured knowledge representation, enabling a more cohesive and scalable solution. This end-to-end paradigm not only simplifies the workflow but also opens new possibilities for learning richer visual semantics in a unified architecture. Theoretically, our work builds on advances in contrastive learning, visual representation learning, and graph construction, integrating them into a pipeline that pushes the boundaries of visual knowledge extraction.

3. Novel Methodology and Field Impact: At the heart of our approach is an instance-level KG construction mechanism, where each individual image is treated as a unique knowledge source. Instead of building a single, unified graph across the dataset, our model generates a mini-knowledge graph per image, capturing localized semantics and supporting fine-grained reasoning. By introducing this methodology, we provide a framework that can potentially reshape how knowledge graphs are constructed and applied in image-centric applications.

In summary, our proposed ImgGraph-LLM introduces a different approach to visual knowledge extraction, characterized by unique data sources, unsupervised training, instance-specific graph outputs, and an end-to-end architecture. These innovations not only distinguish our work from existing methods but also contribute novel tools and perspectives to the broader field of knowledge representation and visual understanding.

5. Experiments

5.1. Experimental Setup

Dataset: Due to the lack of an existing real image dataset from the State Grid Corporation of China, we collected and cleaned data from real scenarios, including transmission, distribution, substation, and safety supervision scenes. Our dataset was unlabeled, aligning with the objective of unsupervised knowledge graph construction. To evaluate the performance and generalization capability of the proposed model, we implemented the following strategies: (a) Manual Annotation for Testing: We randomly selected a subset of the data and created a test set with manually annotated labels. This allowed for a direct assessment of the model’s performance. (b) Dataset Variations for Performance Testing: We prepared two datasets of different sizes, as detailed in

Table 1. Statistics for the two datasets. The test set was manually annotated.

Datasets	Train	Test
StateGrid_S	7000	1000
StateGrid_L	3500	500

, to investigate the model’s performance across varying data scales. These strategies helped in comprehensively understanding the model’s effectiveness and its ability to generalize across different data sizes.

Evaluation Metrics: To evaluate the constructed knowledge graph, we employed three standard metrics: precision, recall, and F1 scores [16,17,30]. These metrics were calculated by comparing the generated triples with the gold-standard triples from the test data. Notably, since the order of triples did not influence the results, the evaluation script identified the best possible match between the generated and gold triples by examining all potential permutations. We then utilized Named Entity Evaluation metrics to assess three levels of match: Exact, Partial, and Strict [14,30]. These categories reflected varying degrees of tolerance for how closely an entity’s match needed to align with the actual entity in both content and its placement within a triple.

Baselines: Since this was unsupervised knowledge graph generation, existing related work typically relies on supervised signals, whether for knowledge graph construction [3,4,5] or scene graph generation models [31,32,33]. To assess the efficacy of ImgGraph-LLM, we compared it against two established baselines: ChatGPT-4o and ImageCaption. ChatGPT-4o is capable of real-time visual reasoning, taking an image as input and directly producing a knowledge graph. Access to ChatGPT-4o was granted on 1 June 2024, via https://chatgpt.com. For the image-to-knowledge graph process, we also employed an image captioning model [34], which provides comprehensive descriptions of visual content and uses Grapher [17] to facilitate the subsequent development of a knowledge graph. Thus, we used image captioning (referred to as ImageCaption) as a benchmark method for constructing knowledge graphs.

Parameter Settings: Our model was fine-tuned, based on an LLM. Given the availability of several LLMs, we selected MiniGPT-4 as our backbone for fine-tuning, after considering two key requirements: the ability to process image data and access to open-source code for easy training. It is important to note that other models meeting these criteria could also be used. We set M to 32, which ensured training efficiency while maximizing the use of self-supervised information. We searched for the parameters ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$ within the set 0.1, 0.2, 0.3, 0.4, and 0.5; then, we set ${\alpha }_1$ to 0.2, ${\alpha }_2$ to 0.2, and ${\alpha }_3$ to 0.3. The parameters ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$ were =0.5. All experiments were conducted on one NVIDIA RTX A100 GPU.

5.2. Results and Analysis

5.2.1. Overall Performance

Table 2. Recommendation accuracy on StateGrid_L. The best result is highlighted in bold.

Model	Match	F1	Precision	Recall
ImageCaption	Exact	0.678	0.678	0.680
	Partial	0.684	0.686	0.685
	Strict	0.673	0.674	0.674
ChatGPT-4o	Exact	0.442	0.438	0.449
	Partial	0.463	0.456	0.472
	Strict	0.409	0.406	0.415
ImgGraph-LLM	Exact	0.808	0.801	0.822
	Partial	0.834	0.823	0.850
	Strict	0.807	0.801	0.815

and

Table 3. Recommendation accuracy on StateGrid_S. The best result is highlighted in bold.

Model	Match	F1	Precision	Recall
ImageCaption	Exact	0.675	0.676	0.679
	Partial	0.684	0.686	0.685
	Strict	0.671	0.670	0.671
ChatGPT-4o	Exact	0.440	0.437	0.448
	Partial	0.463	0.455	0.472
	Strict	0.408	0.407	0.413
ImgGraph-LLM	Exact	0.731	0.727	0.745
	Partial	0.778	0.764	0.793
	Strict	0.732	0.726	0.749

present the comprehensive experimental results across two datasets, evaluated under different metrics. We can obtain the following observations:

1. Superior Performance of ImgGraph-LLM: Our ImgGraph-LLM model achieved the best performance across all metrics. Specifically, on the StateGrid_S and StateGrid_L datasets, ImgGraph-LLM demonstrated an average improvement of 10.6% and 20.4%, respectively, compared to the best baseline model. Although the performance improvement differed between the two datasets, it was consistently superior to that of the baseline model, indicating that our model is robust and reliable across datasets of varying scales. These experimental results confirm the effectiveness of our method in unsupervised image-based knowledge graph construction (KGC) tasks.

2. Baseline Model Performance: The performance of the baseline model was relatively consistent across both datasets. Since our task involved constructing an unsupervised knowledge graph without labeled data to train ImageCaption, we used an existing pre-trained model for testing, resulting in similar performance on both datasets. Similarly, for ChatGPT-4o, we directly used OpenAI’s API for testing, and its performance remained relatively stable.

3. Comparison of ChatGPT-4o and ImageCaption: The performance of the ChatGPT-4o model was significantly lower than that of the ImageCaption model. This discrepancy arose because ChatGPT-4o not only recognizes entities in images but also models more implicit information using its logical reasoning capabilities, such as electric installation entity. This additional information modeling introduced interference from irrelevant information, leading to decreased performance. In contrast, the ImageCaption method focused solely on recognizing and modeling images without additional inference, resulting in less irrelevant information and, consequently, better performance than ChatGPT-4o.

5.2.2. Ablation Study

To investigate the effects of self-supervised operations, we separately removed the operation $T_c$ and operation $T_e$, referred to as “Our w/o $T_c$” and “Our w/o $T_e$”, respectively. According to the results in

Table 4. Ablation study of self-supervised operations using StateGrid_L. The best results are highlighted in bold.

Model	Match	F1	Precision	Recall
Our	Exact	0.808	0.801	0.822
	Partial	0.834	0.823	0.850
	Strict	0.807	0.801	0.815
Our w/o $T_e$	Exact	0.784	0.776	0.796
	Partial	0.808	0.793	0.827
	Strict	0.779	0.769	0.785
Our w/o $T_c$	Exact	0.754	0.741	0.764
	Partial	0.779	0.763	0.795
	Strict	0.748	0.740	0.742

, we observe that removing either the operation $T_c$ or operation $T_e$ led to a decrease in performance; however, both variations still outperformed all baselines. This result validates the effectiveness of the two self-supervised operations: they provide self-supervised signals that enhance the fine-tuning of LLMs, enabling the better identification of entities and relationships. Additionally, the performance of “Our w/o $T_e$” was superior to that of “Our w/o $T_c$”, possibly because the transformation operation $T_c$ directly involves the modification of entities and relationships, which is more closely related to the core information required for constructing knowledge graphs. Please note that the same trend was also present for the StateGrid_S dataset. Overall, by integrating these two approaches, which utilize distinct self-supervised signals for LLM fine-tuning, optimal performance was achieved.

5.2.3. Case Study

We conducted a case study to visually demonstrate the efficacy of our proposed model. As depicted in Figure 5, our model precisely established the relationship between the entities “seatbelt”, “telegraph pole”, and “person”, whereas baseline models failed to accurately model the relationship between “seatbelt” and “telegraph pole”. Additionally, our model effectively filtered out irrelevant background information, a feature that was lacking in other baseline models. As a result, the knowledge graph constructed by our model exhibits superior accuracy.

6. Conclusions and Future Work

ImgGraph-LLM represents a significant advancement in the domain of image-based knowledge graph construction. It not only enhances the automation and precision of knowledge extraction from images but also paves the way for downstream applications in multimodal reasoning, visual question-answering, and cross-modal retrieval. We evaluated ImgGraph-LLM using two challenging real-world datasets. The experimental results demonstrate consistent and significant performance gains, with average improvements of 10% and 20.4%, respectively, over existing baselines in relational extraction accuracy. These results validate the effectiveness of our method in capturing complex visual-semantic relationships.

While ImgGraph-LLM demonstrates strong performance in constructing knowledge graphs from image data, it currently relies on domain-specific priors to generate pseudo-labels, particularly tailored for power grid imagery, where structured components and visual patterns are well defined. This reliance on prior knowledge may limit the generalizability of our approach when applied to other domains where such visual regularities are absent or less pronounced, such as in natural scenes or consumer photography. In future work, we plan to explore the adaptability of ImgGraph-LLM to a broader range of application scenarios. Specifically, we aim to investigate domain-agnostic strategies for pseudo-label generation to enhance the robustness and transferability of our model across diverse visual contexts.