A Smart Contract Vulnerability Detection Method Based on Heterogeneous Contract Semantic Graphs and Pre-Training Techniques

Abstract

The use of smart contracts in areas such as finance, supply chain management, and the Internet of Things has significantly advanced blockchain technology. However, once deployed on the blockchain, smart contracts cannot be modified or revoked. Any vulnerabilities can lead to severe economic losses and data breaches, making pre-deployment vulnerability detection critically important. Traditional smart contract vulnerability detection methods suffer from low accuracy and limited reusability across different scenarios. To enhance detection capabilities, this paper proposes a smart contract vulnerability detection method based on heterogeneous contract semantic graphs and pre-training techniques. Compared to the conventional graph structures used in existing methods, heterogeneous contract semantic graphs contain richer contract information. By integrating these with pre-trained models, our method exhibits stronger vulnerability capture and generalization capabilities. Experimental results show that this method has improved the accuracy, recall, precision, and F1 value in the detection of four widely existing and harmful smart contract vulnerabilities compared with existing methods, which greatly improves the detection ability of smart contract vulnerabilities.

1. Introduction

Blockchain technology, since it was proposed by Satoshi Nakamoto in the Bitcoin white paper in 2008 [1], has rapidly emerged as a disruptive technology with potential applications in various fields, including finance, supply chain management, and the Internet of Things (IoT). Ethereum, as the second-generation blockchain platform, introduced smart contracts as a core feature, enabling decentralized computation on top of the blockchain [2], which significantly expanded the scope of blockchain applications.

The automation and immutability of smart contracts have increased trust in their use across various sectors; however, these same characteristics present significant security challenges. Once vulnerabilities are embedded in smart contract code, they can lead to major losses for associated accounts. According to the SlowMist Technology report [3], there were 464 security incidents in 2023, resulting in losses totaling 2.486 billion USD. Notable cases include the non-custodial lending platform BonqDAO and the crypto infrastructure platform AllianceBlock, which lost approximately 120 million USD due to vulnerabilities in BonqDAO’s smart contracts. Therefore, comprehensive security testing of smart contracts before deployment is crucial.

Currently, researchers have developed many effective tools for detecting vulnerabilities in smart contracts, which can be divided into two main categories: traditional methods, such as Manticore [4], SmarTest [5], ConFuzzius [6], Slither [7], and deep learning-based methods, such as VanillaRNN [8], LSTM [9], TMP [10]. However, these methods have certain limitations:

• traditional methods rely on complex predefined patterns or meticulously designed test cases, requiring experts to deeply analyze vulnerabilities and continually update specific rules. These methods struggle to scale with the rapid growth of smart contract deployments, which restricts their applicability and leads to lower accuracy.
• Deep learning-based methods often inadequately consider the structural and semantic information of code, leading to insufficient generalization when facing diverse smart contracts.

In order to address the shortcomings of existing methods, we select four smart contract vulnerabilities that cause the most serious and widespread losses: Reentrancy [11], Transaction State Dependency [12], Block Info Dependency [13], and Nested Call [12] as research objects. These vulnerabilities were chosen based on the research findings of Monika [12] and Chen [14], who categorized and graded the impact of smart contract vulnerabilities. Additionally, their importance has been validated through numerous studies (e.g., [15,16]) and widely recognized classifications, such as the SWC Registry [13]. See Appendix A for corresponding code examples and descriptions. This paper proposes a smart contract vulnerability detection method based on heterogeneous contract semantic graphs and pre-training techniques. Compared with the traditional data flow graph used in existing methods, the heterogeneous contract semantic graph contains richer contract structure information, which enables the model to focus on the key features of the vulnerability more comprehensively and has stronger vulnerability capture and generalization capabilities. The heterogeneous contract semantic graph is embedded in the graph neural network so that it can be used as part of the pre-training model input to detect smart contract vulnerabilities.

The main contributions of this paper are as follows:

• We designed a heterogeneous smart contract semantic graph generation method based on abstract syntax trees (AST), using variables and function calls as nodes to comprehensively represent the semantic and structural information of contracts at the statement level.
• By combining heterogeneous contract semantic graphs with pre-training techniques, our Approach achieved superior performance in detecting various types of smart contract vulnerabilities. Experimental results show that our method achieves detection accuracies of 90.96%, 89.57%, 88.46%, and 87.34% for the four types of vulnerabilities, representing significant improvements over 12 mainstream methods.

Next, Section 2 will introduce the existing work on smart contract vulnerability detection and the relevant background knowledge of this article. Section 3 will introduce the smart contract vulnerability detection method based on heterogeneous contract semantic graphs and pre-training technology. Section 4 will show the experimental results, and Section 5 will summarize.

2. Related Work

2.1. Existing Smart Contract Vulnerability Detection Tools

With the rapid development of blockchain platforms such as Ethereum, numerous smart contract vulnerability detection tools have emerged. In the following, we will briefly introduce mainstream methods for detecting smart contract vulnerabilities. We will also analyze their strengths and limitations. Representative traditional methods include those based on symbolic execution (e.g., Oyente [17], Manticore [4], SmarTest [5], Mythril [11]), fuzz testing (e.g., ContractFuzzer [18], ConFuzzius [6]), static analysis (e.g., Slither [7], SmartCheck [19]), and formal verification (e.g., KEVM [20], ZEUS [21]).

Symbolic execution-based methods, such as Oyente [17], use symbolic inputs instead of concrete ones. They simulate the execution of the analyzed program and transform program operations into symbolic expressions. This allows for the analysis of path reachability, test data generation, and detection of specific vulnerabilities. However, as the size of smart contract code increases, these methods may face challenges like path explosion and slow constraint solving. This can reduce the efficiency of vulnerability detection. Fuzz testing-based methods, such as ContractFuzzer [18], generate many random test cases by obtaining function parameters through the Abstract Binary Interface (ABI). They combine these cases with vulnerability patterns to detect issues like reentrancy vulnerabilities. However, these test cases often lack specificity and may not cover all possible vulnerability scenarios. For complex smart contracts, fuzz testing may fail to trigger deeply hidden vulnerabilities, especially those that only appear under specific input conditions. Slither [7], as a static analysis tool, directly analyzes the source code of smart contracts. It can quickly detect potential vulnerabilities without executing the contract. This makes it more efficient in speed, allowing it to handle large-scale smart contract codebases. However, static analysis tools typically rely on predefined rules or patterns. As a result, Slither may fail to detect certain atypical vulnerabilities, especially those involving complex state changes or dynamic behaviors. KEVM [20] uses formal modeling of smart contract behavior to verify whether a contract adheres to specific security properties. It provides a high level of security assurance. However, formal verification involves exhaustive checking of all possible execution paths. As a result, KEVM’s verification process often requires significant time. This can be problematic when handling complex contracts, leading to excessively long verification times.

In recent years, deep learning has been widely applied in various fields. Leveraging big data and deep learning techniques to automatically learn the characteristics of smart contract vulnerabilities has become a new research direction. Wang et al. [15] developed ContractWard, which extracts features from bytecode using n-grams. The model uses five machine learning algorithms, including Random Forest and Support Vector Machine, to detect six types of vulnerabilities, such as reentrancy and integer overflow. However, this method does not consider the structural or semantic information of the code during learning and computation. Liu et al. [22] proposed the vulnerability detection network CGE (Combining Graph features and Expert patterns). It defines expert pattern features, such as timestamp declarations, and combines them with contract graph features obtained through graph neural networks. This improves the accuracy of vulnerability detection. However, their model cannot handle heterogeneous graphs with multiple types of nodes. They addressed this by removing secondary nodes and aggregating their features into primary nodes, which led to the loss of some information. This reduced the method’s generalizability when dealing with diverse smart contracts.

2.2. Graph Pre-Trained Models

Pre-trained models begin by undergoing preliminary training on large-scale, unsupervised datasets to learn general features and representations. They are then fine-tuned on smaller, labeled datasets to adapt to specific tasks. The primary advantage of this approach is the ability to leverage the extensive information from large datasets to enhance performance on specific tasks. Models like BERT [23] and GPT [24] have achieved notable success in natural language processing tasks and are widely used. Inspired by this, researchers in the field of software engineering have proposed a series of models specifically for programming languages, such as CodeBERT [25] and CodeT5 [26], which are designed to understand and generate source code. Similar to text-based pre-training models, these models learn the structure, syntax, and common programming patterns by pre-training on large amounts of open-source code, significantly improving the automation and efficiency of programming-related tasks.

However, some existing works [25,26] treat program source code as collections of tokens, overlooking that code is not merely a sequence of words but includes various graph-like data structures such as loops, jumps, controls, and dependencies [27]. GraphCodeBERT [28] makes up for the shortcomings of previous work by incorporating semantic-level information about the code and introducing the data flow graph (DFG) of the code for pre-training. The data flow graph is a structure frequently used in program analysis [29]. In GraphCodeBERT, code is first parsed into an Abstract Syntax Tree (AST) using standard compilation tools. From the AST, a labeled sequence of source code variables is derived, and a data flow graph is constructed. This graph represents program variables as nodes and the dependencies among them as edges. The code and its corresponding data flow graph serve as inputs to the model, achieving good results in a variety of code-related downstream tasks.

2.3. Graph Structure Representation of Smart Contracts

Current research [29] suggests that programs can be transformed into graphical representations, thereby preserving the relationships between program elements. Zhuang et al. [10] have utilized this approach by converting smart contracts into graphs and employing Graph Neural Networks (GNN) for detecting vulnerabilities in smart contracts. However, their model struggles with handling heterogeneous graphs with multiple types of nodes, leading them to remove secondary nodes and aggregate their features into primary nodes, resulting in a partial loss of structural information. Wu et al. [30] introduced a method for smart contract vulnerability detection named Peculiar, which uses the tree-sitter-solidity tool https://github.com/JoranHonig/tree-sitter-solidity (accessed on 31 July 2024) to generate an AST. From the AST, they extract a Data Flow Graph (DFG) to represent the contract’s graphical structure and have designed a method to filter key variables, simplifying the DFG into a Critical Data Flow Graph (CFDG) for model input. Although the CDFG simplifies smart contract information to some extent and focuses more on variables that could lead to vulnerabilities, it only includes the data flow relationships between variables and misses a lot of structural information related to vulnerabilities, such as special function calls and execution sequences. In order to solve the problem that the existing smart contract graph structure cannot adequately represent smart contract information, this paper proposes a new smart contract graph structure representation method, which uses variables and function calls as nodes to construct a heterogeneous contract graph. Based on this heterogeneous graph, we further consider how to effectively capture the complex semantic relationships within the graph. There are several embedding methods for heterogeneous graphs, such as metapath2vec [31], HAN [32], and HGT [33]. Metapath2vec defines meta-paths to guide random walks, capturing semantic relationships in heterogeneous graphs, and then uses the Skip-Gram model to learn node embeddings. HAN utilizes meta-paths to guide attention mechanisms, focusing on interactions between nodes with significant semantic importance. HGT, based on the Transformer architecture, introduces type-specific parameterization to handle the heterogeneity of nodes and edges, and incorporates relative temporal encoding to account for the dynamic nature of the graph. This is particularly crucial for smart contract vulnerability detection, as vulnerabilities are often closely related to data flow and control flow timing between nodes. Therefore, we select HGT as the graph embedding method to fully represent the semantic information and structural relationships of smart contracts.

3. Research Method

In the methodology of this paper, we do not undertake the pre-training task ourselves; instead, the pre-training is handled by GraphCodeBERT [28]. This is appropriate because the Solidity language, specifically designed for Ethereum smart contract development, exhibits features like static typing, inheritance, libraries, and complex user-defined types, drawing influences from JavaScript, Python, and C++. GraphCodeBERT, during its pre-training phase, utilized 2.3 million functions from six programming languages, encompassing three pre-training tasks: modeling programming languages, predicting edges in the program data graph (data flow graph), and aligning variables across program source code and its graph structure. Our method requires loading the pre-trained parameters from GraphCodeBERT, followed by fine-tuning them on a smart contract dataset. The workflow of this method is shown in Figure 1 and is divided into two stages: (a) Graph Generation Stage, during which source code is converted into an AST, parsing the AST to retain different nodes and edges and generating a heterogeneous contract semantic graph. (b) Vulnerability Detection Stage, where smart contract vulnerabilities are detected based on the pre-trained model. We will now elaborate on these two stages.

3.1. Graph Generation Phase

In the process of generating the AST for Solidity code, most existing work, such as [10,30], predominantly employs the tool tree-sitter-solidity, which is known for its efficiency and incremental parsing capabilities. However, for Solidity code, the AST generated by this tool does not contain sufficiently detailed node information. For example, as shown in Listing A1, concerning the content at line 11 of the reentrancy vulnerability example code msg.sender.call, Figure 2a represents a portion of the AST nodes obtained from tree-sitter-solidity, whereas Figure 2b shows a portion of the AST nodes parsed by the official Solidity compiler, solc. It is evident that the AST nodes parsed by tree-sitter-solidity only include type and text information within their description, lacking detailed metadata. In contrast, the AST generated by solc not only includes structural information within the AST hierarchy but also provides rich descriptive information for nodes such as id, src, nodeType, and typeDescriptions, particularly offering standardized descriptions in typeIdentifier and typeString for critical nodes associated with vulnerabilities like the low-level function call call(). Given that solc has undergone multiple iterations since 2015, the AST structure and node information generated by different versions of solc can vary. The tool solc-typed-ast enables the creation of normalized type Solidity ASTs based on solc, mitigating the differences brought by various versions of the compiler. Therefore, we opt to use solc-typed-ast to generate the code AST.

The AST generated by the solc-typed-ast tool includes 56 types of nodes, which we categorize into three classes based on their functionalities. They are structure operation class nodes, variable and call class nodes, and other class nodes. The detailed classification is shown in

Table A1. Solc-typed-ast node type table.

Functional Category	Node Type
Structural and Operation	ContractDefinition, EnumDefinition, ErrorDefinition, EventDefinition, FunctionDefinition, ModifierDefinition, SourceUnit, Block, Break, Continue, DoWhileStatement, EmitStatement, ExpressionStatement, ForStatement, IfStatement, InlineAssembly, PlaceholderStatement, PragmaDirective, Return, RevertStatement, Throw, TryCatchClause, TryStatement, UncheckedBlock, VariableDeclarationStatement, WhileStatement, NewExpression, Assignment, BinaryOperation, Conditional, TupleExpression, UnaryOperation, IndexAccess, IndexRangeAccess, MemberAccess
Variables and calling	FunctionCall, Identifier, IdentifierPath, UserDefinedValueTypeDefinition, VariableDeclaration, ModifierInvocation, ParameterList, StructDefinition
other	ImportDirective, PragmaDirective, OverrideSpecifier, StructuredDocumentation, UsingForDirective, InheritanceSpecifier, ArrayTypeName, ElementaryTypeName, EnumValue, FunctionTypeName, UserDefinedTypeName, FunctionCallOptions, Literal

in Appendix A. The structural and operational class nodes encompass the structure and control information of an entire ‘.sol’ file’s AST, from top to bottom, including files, contracts, functions, statements, and operations. We utilize this information to construct the edges in the heterogeneous contract semantic graph, including both data flows and control flows. The variable and call class nodes, as components of statements, are constructed as nodes within the heterogeneous contract semantic graph. Other class nodes, which include import statements, structured documentation, and other nodes that do not contribute to vulnerability generation, or basic class nodes like names, contain information that can be encapsulated by higher-level nodes and may be omitted.

Specifically, in high-level programming languages, the basic units of execution are statements, including both simple and compound statements. Simple statements consist of a single logical line, such as expression statements (ExpressionStatement) and assignment statements (Assignment), whereas compound statements encompass other statements (groups of statements) that influence or control the execution of the included statements in some manner, such as if statements (IfStatement) and for statements (ForStatement). The logic within simple statements facilitates operations on variables and function calls. In our proposed method for constructing heterogeneous contract semantic graphs, we analyze each simple statement in the contract according to the typed-AST structure, extracting variables and function call nodes to serve as nodes within the heterogeneous contract semantic graph. It is important to note that Solidity includes certain special variables that are always present in the global namespace. These built-in variables are essentially of a basic type and are used to describe attributes of blocks and transactions, such as block.prevrandao and msg.sender [34]. Although these variables are not explicitly declared, they are still integrated as nodes in our graph. Solidity allows for the manipulation of mappings, arrays, and structures through dereferencing, but we do not treat the results of such dereferences (e.g., balances[msg.sender]) as individual nodes. Instead, we regard these structures as a whole, which facilitates their inclusion as nodes when constructing the contract semantic graph.

The relationships among simple statements—such as sequence, loops, and conditionals—act as control flows, while the operations within simple statements (such as BinaryOperation, IndexAccess, etc.) and the inputs and outputs involved in function calls are constructed as data flows. Take the code shown in Listing 1 as an example:

Listing 1. Sample Code.

The visualization of the typed-AST generated by the solc-typed-ast tool is shown in Figure 3.

Based on the structural information in the typed-AST, the example code is transformed into a heterogeneous contract semantic graph as shown in Figure 4.

The variable node features in the contract graph are represented by a four-tuple, F = (id, name, nodeType, typeDescriptions). The function call node features in the contract graph are represented by a five-tuple, F = (id, name, nodeType, typeDescriptions, parameter), where id is the node identifier, name is the variable name (function name), nodetype is the node type, typeDescriptions is the specific description of the node type, and parameter is the function parameter. The edge features in the contract graph are represented by a four-tuple, F = (order, type, $V_{start}$, $V_{end}$), where order is the time order, type is the edge type (control flow or data flow), and $V_{start}$ and $V_{end}$ are the start and end nodes. The resulting contract graph is denoted as $HCG\left(SC\right)=(V,E)$, where V is the node set and E is the edge set. Figure 4b shows the heterogeneous contract semantic graph built based on the contract with the vulnerability, and Figure 4c shows its edge features.

3.2. Vulnerability Detection Phase

This section will offer an in-depth description of how the proposed approach utilizes a graph-based pre-trained model to identify vulnerabilities in smart contracts. It will encompass details on data preparation, the model’s architecture, and the training procedure.

First is the data preparation process. This method is based on GraphCodeBERT [28], but it differs in how it handles graph-structured data. In the original GraphCodeBERT architecture, given a source code $SC=\{s{c}_1,s{c}_2,\dots ,s{c}_n\}$, a corresponding data flow graph $G\left(SC\right)=(V,E)$ can be obtained, where $V=\{v_1,v_2,\dots ,v_n\}$ is a set of variables (also the nodes in the data flow graph), and $E=\{e_1,e_2,\dots ,e_n\}$ is a set of directed edges indicating the dependencies among variables. The source code and the set of variables are merged into a sequence $I=\left\{\right[CLS],SC,[SEP],V\}$, with $\left[CLS\right]$ as a special token preceding the sets, and $\left[SEP\right]$ as a separator between the source code $SC$ and the variable set V. For each token in the sequence I, we generate the corresponding position embedding and add the token and the corresponding position embedding to represent the token. Special position embeddings are assigned to all variables to signify their roles as nodes within the data flow. The final input representation, denoted as $X_0$, is derived in this manner.

In our method, the embedding for the source code text follows the approach used in GraphCodeBERT. However, the embedding for the graph structure differs because the GraphCodeBERT method, which is suited for handling simple homogeneous graphs like data flow graphs, does not adequately capture the information in the heterogeneous contract semantic graph $HCG\left(SC\right)=(V,E)$ generated earlier. In our graph embedding process, we choose HGT [33] as the graph structure embedding method to extract the structural information of the heterogeneous contract semantic graph. The resulting node feature sequence replaces the variables sequence in the GraphCodeBERT method.

The purpose of HGT is to consolidate information from source nodes to obtain the contextual representation of target nodes. This procedure can be segmented into three distinct phases:

• The first part is the calculation of heterogeneous mutual attention. The calculation of heterogeneous mutual attention begins by examining the meta-relationships between a target node t and each of its source nodes $s\in N\left(T\right)$. These relationships are defined by the tuple $\langle T\left(s\right),\varphi \left(e\right),T\left(t\right)\rangle$, representing the source node type, the edge type, and the target node type, respectively. To accommodate the diverse and complex nature of these relationships in a heterogeneous graph, the model converts target node t into a query vector and each source nodes s into a key vector. Unlike standard Transformers that use a direct inner product for such calculations, HGT utilizes distinct attention matrices $W_{\varphi \left(e\right)}^{ATT}$ tailored to each edge type $\varphi \left(e\right)$, ensuring that the nuances of different semantic associations are captured effectively.
• The second part is the heterogeneous message-passing process. The message-passing process from the source node to the target node and the calculation of mutual attention are parallel. The goal is to merge the meta-relationships of various edges into the message-passing process. By doing so, it helps to balance the distribution disparities among different types of nodes and edges.
• The third part is the aggregation for a specific task. It uses the attention vector as the weight to calculate the corresponding information from the source node and obtain the updated vector. The updated vector is linearly mapped and connected with the original vector of t in the previous layer as a residual. In this way, the output $H^{\left(L\right)}\left[t\right]$ of the target node t in the $L_{\mathrm{th}}$ layer of HGT is obtained. Stacking L layers can obtain a rich context representation $H^{\left(L\right)}$ for each node as the input of the downstream task.

In order to be able to handle the dynamic nature of the graph, relative time coding is introduced. Traditionally, time information has been integrated by constructing a separate graph for each time slot, a method that can lead to the loss of structural dependencies across different time slots. Moreover, the representation of a node at time t might rely on edges from various other time slots. Thus, the appropriate method to model a dynamic graph is to preserve all edges occurring at various times and permit interactions between nodes and edges that possess different timings [32]. Specifically, given a source node s and a target node t with the edge $\varphi \left(e\right)$ between them, the edge information $\varphi \left(e\right)$ includes temporal information order, which is used as an index for the relative temporal encoding. This encoding is applied through sine and cosine functions to capture the relative temporal dependencies.

$$Base(order,2i)=sin\left({\displaystyle \frac{order}{{10000}^{\frac{2i}{d}}}}\right)$$

(1)

$$Base(order,2i+1)=cos\left({\displaystyle \frac{order}{{10000}^{\frac{2i+1}{d}}}}\right)$$

(2)

$$RTE\left(order\right)=Linear\left(Base\right(order\left)\right)$$

(3)

In Equations (1) and  (2), $order$ is the timing information carried by the edge in the heterogeneous graph, i is the position index, and d is the dimension of the feature vector in HGT. $Base$ represents the basic relative temporal encoding calculated by sine and cosine functions. In Equation (3), $Linear$ represents the projection of the basic relative time code to obtain the final relative time code RTE(order). RTE(order) is added to the node representations $H^{\left(L\right)}$ to obtain $H^{{\left(L\right)}^{\prime }}$. This allows the resulting node representations to capture the relative temporal information between the source node s and the target node t.

Secondly, the model architecture and training process are as follows. As shown in Figure 5, the sequence $I=\{\left[CLS\right],SC,\left[SEP\right],H^{{\left(L\right)}^{\prime }}\}$ obtained in the data preparation stage is fed to the Join layer. In the Join layer, the sequence I from the data preparation stage is converted into an input vector $X_0$. Subsequently, the input vector $X_0$ proceeds through the multi-head attention layer, undergoes layer normalization, and passes through multiple Transformer layers $(n=12)$ to produce distinct contextual representations, $X^n={\mathrm{Transformer}}_n\left(X^{n-1}\right),n\in (1,12)$. Equations (4) and (5) represent the training process of the model. In Equation (4), H and X are vectors, $MultiAttn$ denotes a multi-head self-attention operation, and $LN$ denotes a layer normalization operation. In Equation (5), $FNN$ represents a two-layer feed-forward network, where each Transformer layer comprises a structurally identical transformer. As shown in Equations (4) and (5), the output $X^{n-1}$ of the previous layer first undergoes a multi-head self-attention operation. The output of the self-attention operation is not directly passed to the next stage, but is first added to $X^{n-1}$ to form a residual connection and normalized to obtain a vector $H^n$. After the vector $H^n$ passes through the feedforward layer, which includes two linear transformation layers with an activation function in between, it also undergoes a residual connection and another layer of normalization to produce the output $X^n$. This process helps the model avoid potential gradient vanishing problems in deep networks while maintaining information from each layer’s input.

$$H^n=LN(MultiAttn\left(X^{n-1}\right)+X^{n-1})$$

(4)

$$X^n=LN(FFN\left(H^n\right)+H^n)$$

(5)

For the output $\widehat{X}$ of the multi-head self-attention in the $n_{th}$ transformer layer, the calculation process is shown in Equation (6) to (9):

$$Q_i=X^{(n-1)}{W}_i^Q,K_i=X^{(n-1)}{W}_i^K,V_i=X^{(n-1)}{W}_i^V$$

(6)

$$hea{d}_i=Softmax\left({\displaystyle \frac{Q_i{K}_i^T}{\sqrt{d_k}}}+M\right)V_i$$

(7)

$$M_{ij}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{q}_i\in \{\left[CLS\right],\left[SEP\right]\}\mathrm{or}{q}_i,k_i\in SC\mathrm{or}\langle q_i,k_i\rangle \in E\cup E^{\prime }\hfill \\ -\infty ,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

$$\widehat{X}=[hea{d}_1,\dots ,hea{d}_u]W_n^O$$

(9)

In Equation (6), $X^{(n-1)}\in {\mathbb{R}}^{\left|I\right|\times d_h}$ is the output of ${(n-1)}_{th}$ Transformer layers, X, W are vectors, and Q, K, V are triplets. $X^{(n-1)}$ is linearly projected onto the triplets of $Q_i$, $K_i$, and $V_i$ using model parameters $W_i^Q$,$W_i^K$,$W_i^V\in {\mathbb{R}}^{d_h\times d_k}$. To incorporate a graph structure within the Transformer and illustrate dependencies between graph nodes, we adopt the approach of GraphCodeBERT [28], utilizing graph-guided masked attention to depict the interactions among tokens. Graph-guided masked attention is implemented through the mask matrix M. In Equation (7), $head$ is the head in multi-head attention, $d_k$ is the dimension of the $head$, M is the mask matrix, $M\in {\mathbb{R}}^{\left|I\right|\times \left|I\right|}$, where if the $i_{th}$ token and the $j_{th}$ token are associated, then $M_{ij}=0$, otherwise it is $-\infty$. The calculation process of M is shown in Equation (8), $\left[CLS\right]$ is a special mark in front of the set, $\left[SEP\right]$ is a separator, $SC=\{sc,s{c}_2,\dots ,s{c}_n\}$ is the set of tokens in the source code, E is the set of edges $\{e_1,e_2,\dots ,e_l\}$ in the heterogeneous contract semantic graph $HCG$, representing the control flow and data flow relationship between nodes, and $E^{\prime }$ is a set indicating the association between the smart contract source code token and the nodes in the heterogeneous contract semantic graph ($HCG$). Attention computations are permissible between nodes $v_i$ and $v_j$ when they are directly connected (i.e., $\langle v_i,v_j\rangle \in E$) or are the same node (i.e., $i=j$). To represent the relationship between the source code tokens and the nodes in the heterogeneous contract semantic graph, we first define a set $E^{\prime }$. If node $v_i$ is determined by the token ${\mathrm{sc}}_j$ in the source code (i.e., $\langle v_i,{\mathrm{sc}}_j\rangle ,\langle {\mathrm{sc}}_j,v_i\rangle \in E^{\prime }$), they are allowed to perform attention computation with each other; in other cases, the attention is masked by assigning an attention score of $-\infty$. After the Softmax calculation in Equation (7), it is assigned a value of 0. In Equation (9), u is the number of heads in the multi-head attention, and $W_n^O\in {\mathbb{R}}^{d_h\times d_h}$ is the model parameter.

After the $n_{th}$ layer of the Transformer model, layer normalization is employed for regularization. Subsequently, the output y, which represents the probability of the contract containing a vulnerability, is derived using a linear layer followed by a Sigmoid [35] function, as demonstrated in Equation (10). A loss function is then formulated to measure the discrepancy between this output y and the target value, where the target is set to 1 if the smart contract exhibits a specific vulnerability, and 0 otherwise. Finally, the backpropagation algorithm is utilized to train the network.

$$y=Sigmoid\left(X^n\right)$$

(10)

4. Experiment

In this section, we undertake a comprehensive empirical analysis of the method we propose, utilizing datasets that are publicly accessible. This evaluation is designed to rigorously test the effectiveness of our approach. To systematically assess the performance of our method, we have articulated several specific research questions:

• RQ1: Is our proposed method capable of effectively identifying the four most prevalent vulnerabilities in smart contracts, and does it outperform existing methods in this regard? We address this question by comparing the accuracy, precision, recall, and F1-score metrics.
• RQ2: What is the contribution of different modules in the proposed method to vulnerability detection? This question investigates the contribution of various modules to the model, including the heterogeneous contract semantic graph and the pre-trained model. We designed ablation experiments to answer this question.

4.1. Experimental Setup

Experiment environment: To conduct the experimental analysis, we utilized the open-source tool solc-typed-ast https://github.com/Consensys/solc-typed-ast, accessed on 11 March 2024, to parse Ethereum Solidity source code into a standard AST format. This was followed by further data optimization operations, such as removing comments. We developed a heterogeneous contract semantic graph generator based on the standardized AST information. The neural network was designed and implemented using the PyTorch framework. All experiments were performed on a physical machine running the Ubuntu 22.04 operating system, equipped with an Intel Xeon Silver 4310 processor with a base frequency of 2.1 GHz, 64 GB of RAM, and an NVIDIA A10 GPU with 24 GB of VRAM. Model training acceleration was achieved using the CUDA 12.4 computing library. The development environment included Visual Studio Code software (version 1.86), the PyTorch (version 2.2) framework, a Node.js (version 20.8.0) environment, and programming languages such as Python (version 3.11) and JavaScript (ECMAScript 2023).

Comparative Methods: To evaluate the effectiveness of the proposed method, we compared its performance with 12 state-of-the-art smart contract vulnerability detection methods. These include six open-source detection methods based on different neural network models: VanillaRNN [8], LSTM [36], GRU [36], DR-GCN [10], TMP [10], and CGE [22]. Additionally, we compared our method with six traditional smart contract vulnerability detection tools from top-tier conferences and journals: Oyente [17], Contractfuzzer [18], Mythril [11], Slither [7], Smartcheck [19], and Securify [37].

Dataset: To assess the effectiveness of the proposed method, we constructed our dataset based on the open-source dataset Smartbug [38]. Smartbug is one of the most widely used public datasets in smart contract vulnerability detection research, containing 47398 Ethereum smart contracts labeled based on static analysis tools. However, due to the errors in static analysis tools, this labeling method lacks complete accuracy. In order to conduct an accurate evaluation, we carefully selected those parts of Smartbug that have been manually verified and confirmed to be accurate vulnerability labels as our dataset in combination with relevant literature [16,30]. we first reclassified the data from several manually labeled smart contract datasets according to the four types of vulnerabilities we are studying. Then, we used automatic analysis tools to detect vulnerabilities in these contracts, ensuring that our dataset contained a certain number of contracts that traditional detection tools failed to classify as vulnerabilities. In addition, we removed comments from the code, excluded contracts with less than 100 lines of code, and deleted duplicate contracts, which had different addresses but the same main functions determined by code similarity analysis. This process produced the final dataset we used in our experiments. The dataset’s variety and frequency of vulnerabilities are detailed in

Table 1. Vulnerability Counts Across Different Categories.

Reentrancy	Transaction State Dependency	Block Info Dependency	Nested Call	Safe
973	895	1030	863	1097

. The dataset is randomly divided into three parts, each accounting for a different proportion: 70% for the training set, 20% for the validation set, and 10% for the test set. The neural network model uses the training set to learn the various vulnerability features in the smart contract, while the validation set is used to adjust parameters during model training to avoid model overfitting. The test set is used to evaluate the generalization ability of the model in vulnerability detection. Such a data allocation strategy helps to conduct a comprehensive and rigorous evaluation of the model.

Parameter setting: The default parameters of the model are as follows: Batch size: 16, Initial learning rate: $2\times {10}^{-5}$, Dropout rate: 0.1, Optimizer: use AdamW, weight decay is set to $1\times {10}^{-4}$.

4.2. Evaluation Metrics

In the experiments presented in this paper, we employed four widely-used evaluation metrics to comprehensively assess the performance of the smart contract vulnerability prediction model: Accuracy, Recall, Precision and F1-score. The calculations for these metrics are based on the four crucial components in the confusion matrix: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). The specific formulas for these calculations are detailed as follows:

$$Accuracy=100\times {\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$

(11)

$$Recall=100\times {\displaystyle \frac{TP}{TP+FN}}$$

(12)

$$Precision=100\times {\displaystyle \frac{TP}{TP+FP}}$$

(13)

$$F1=200\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(14)

Note: All calculated values in the article are multiplied by 100 and expressed as percentages.

4.3. Ablation Experiment

This section aims to validate the individual contributions of each component in the proposed method. First, to verify the contribution of pre-training techniques, we normalized the parameters of the pre-trained model (GraphCodeBERT [28]) and trained the model using the same input. For the smart contract files in the dataset, the canonical abstract syntax tree (AST) is first parsed by the solc-typed-ast tool. Based on the AST, a heterogeneous contract semantic graph is constructed, and features are extracted through the HGT network. Before training, GraphCodeBERT’s parameters are normalized to eliminate the pre-training advantage. We denote this variant as $method-WP$. During the training phase, the loss rate decreased slowly. After a period of training, we tested the results on the test set, as shown in

Table 2. Performance comparison of our method and its variants on four vulnerability detection tasks.

Methods	Reentrancy				Block Info Dependency				Transaction State Dependency				Nested Loop
	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1
Method-WP	55.32	34.68	48.03	59.1	44.69	50.52	47.92	49.19	41.24	47.96	31.74	51.24	39.56	36.28	32.09	34.06
Method-HOG	80.09	81.18	72.15	76.4	81.3	80.68	78.42	79.53	74.11	76.39	68.92	72.46	70.23	65.08	71.44	68.11
Our method	90.96	91.62	89.16	90.37	89.57	87.62	91.43	89.50	88.46	86.20	90.37	88.26	87.34	84.91	89.77	87.30

. The $method-WP$ exhibited a substantial decline in performance across various types of vulnerability detection tasks. This outcome suggests that pre-training techniques can boost the generality of models, lessen the learning burden, and serve an indispensable role in the detection of vulnerabilities.

To study the contribution of the heterogeneous contract semantic graph, we compared the performance differences between homogeneous and heterogeneous graph analyses. For the smart contract files in the dataset, the canonical abstract syntax tree (AST) is first parsed by the solc-typed-ast tool. In the contract graph construction phase, the heterogeneous contract semantic graph is replaced by the isomorphic graph generated by the method proposed by Liu [22], so this variant is recorded as $method-HOG$. Then, the graph embedding is completed by GAT (graph attention neural network), and finally the obtained graph features and source code are used as input to train the GraphCodeBERT model loaded with pre-trained parameters. After training, the results tested on the test set are shown in Table 2. We observed that the original architecture of our method significantly outperformed method-HOG in all aspects. This result suggests that the heterogeneous contract semantic graph contains finer-grained semantic information in its graphical features, which is crucial for the smart contract vulnerability detection task.

4.4. Comparison with Other Deep Learning Based Methods

The experiments compared the proposed method with 12 state-of-the-art methods, including six open-source detection methods based on different neural network models: VanillaRNN [8], LSTM [36], GRU [36], DR-GCN [10], TMP [10], and CGE [22].

Table 3. Performance comparison with neural network-based methods.

Methods	Reentrancy				Block Info Dependency				Transaction State Dependency				Nested Call
	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1
Vanilla-RNN	50.12	55.84	46.33	50.72	49.77	44.59	51.17	52.15	52.43	50.20	54.65	52.40	51.98	50.80	48.76	49.71
LSTM	53.29	59.97	48.45	53.79	50.79	57.23	51.42	54.17	53.15	60.24	51.86	55.74	55.07	56.35	50.83	53.43
GRU	57.80	72.17	52.18	60.68	62.76	59.85	65.43	62.61	62.76	59.85	65.43	62.61	59.65	57.43	61.98	59.97
DR-GCN	78.96	77.05	73.24	75.09	75.53	72.89	78.11	75.51	-	-	-	-	-	-	-	-
TMP	82.15	80.90	80.39	80.64	80.87	78.39	83.45	80.88	-	-	-	-	-	-	-	-
CGE	85.32	81.25	87.46	84.23	86.21	84.10	85.96	85.03	-	-	-	-	-	-	-	-
Our method	90.96	91.62	89.16	90.37	89.57	87.62	91.43	89.50	88.46	86.20	90.37	88.26	87.34	84.91	89.77	87.30

“-” means the corresponding tool does not support detecting this type of vulnerability.

presents the Accuracy, Recall, Precision and F1-score values of our method compared to the six detection methods based on neural network models.

The initial three models utilize text sequences from smart contract codes as inputs, while the subsequent three models are based on the graph structure data of smart contracts. These models represent typical approaches in the field of smart contract vulnerability detection and have been widely adopted as benchmark methods in recent studies [10,30,37]. Considering that none of the compared methods support other block information dependency vulnerabilities except timestamp dependency, in order to ensure the fairness of the comparison, we limit the scope of method evaluation and only verify and compare the detection effect of timestamp dependency vulnerabilities. It can be seen that in smart contract reentrancy vulnerability detection, our method improves the accuracy by 52.11% compared to the best-performing text sequence modeling model, GRU, and by 6.61% compared to the best-performing graph structure modeling model, CGE. In detecting block information dependency vulnerabilities in smart contracts, our method improves the accuracy by 42.71% compared to the best-performing text sequence modeling model, GRU, and by 4.90% compared to the best-performing graph structure modeling model, CGE. We attribute this improvement to our proposed heterogeneous contract semanti graph, which contains richer semantic information and higher separation between different features compared to homogeneous graphs. Methods based on graph structure data modeling tend to outperform those based on text sequence modeling, which also proves that code is not merely a sequence of words, and ignoring structural information such as data flow and control flow can degrade performance. For the detection of transaction state dependency vulnerabilities and Nested Call vulnerabilities, as the three graph structure-based methods do not support these types, we only compare with the text sequence-based methods. The results show that compared to the best-performing GRU, our method improves the accuracy by 40.94% and 46.42%, respectively.

4.5. Compared with Traditional Tools

In addition to comparing our method with neural network-based approaches, we also compared it with state-of-the-art traditional smart contract vulnerability detection tools. Based on the detection targets and main features of these tools [39], we used Smartcheck [19], Securify [37], and Slither [7] for source code-level vulnerability detection, and Oyente [17], Contractfuzzer [18], and Mythril [11] for bytecode-level vulnerability detection. In order to meet the input requirements of Oyente, Contractfuzzer and Mythril tools, we use the compiler to generate corresponding bytecodes for the smart contracts in the dataset for them to use. The results are shown in

Table 4. Performance comparison with traditional tools.

Methods	Reentrancy				Block Info Dependency				Transaction State Dependency				Nested Call
	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1	Acc	Rec	Pre	F1
Smartcheck	40.43	41.25	39.11	40.15	52.91	39.70	51.17	52.15	30.66	33.89	40.67	36.97	47.26	50.12	55.99	52.99
Oyente	60.23	62.35	65.12	63.46	39.45	41.23	43.56	42.37	-	-	-	-	42.78	39.75	46.89	43.03
Contractfuzzer	67.54	61.44	49.26	54.68	63.99	65.68	67.89	66.75	-	-	-	-	-	-	-	-
Mythril	65.35	69.91	50.08	58.36	70.23	63.31	69.88	66.43	77.12	51.32	59.70	55.19	60.23	55.71	60.38	55.71
Securify	73.88	75.79	68.45	71.93	-	-	-	-	-	-	-	-	-	-	-	-
Slither	80.35	87.10	82.57	84.77	77.12	74.28	68.42	71.23	60.96	82.09	62.07	71.27	55.37	27.69	61.91	38.27
Our method	90.96	91.62	89.16	90.37	89.57	87.62	91.43	89.50	88.46	86.20	90.37	88.26	87.34	84.91	89.77	87.30

“-” means the corresponding tool does not support detecting this type of vulnerability.

.

It can be observed that traditional vulnerability detection tools do not perform well. The best-performing traditional tool, Slither, achieved an accuracy of 80.35% in reentrancy vulnerability detection, while our method significantly improved the accuracy by 13.20%. We believe the poor performance of traditional methods is due to their heavy reliance on simple and fixed vulnerability detection patterns. For example, Mythril uses symbolic execution, mixed concrete execution, and control flow analysis to detect vulnerabilities in smart contracts. In reentrancy vulnerability detection, Mythril determines if a contract has a reentrancy vulnerability based on whether there are internal function calls or state variable modifications following a low-level call to call.value. Additionally, tools that use bytecode as input lack source code semantic information, which can lead to decreased detection performance. For block information dependency vulnerability detection, since Securify does not support the detection of such vulnerabilities, we only compared our method to the other five methods. Our method outperforms traditional methods on all four metrics and achieves 16.1% accuracy improvement over the best traditional method, Slither. The poor performance of some traditional tools may be attributed to their simplistic approach of checking for the presence of block information statements like block.timestamp to determine the existence of timestamp dependency vulnerabilities. Since only Mythril, Slither, and Smartcheck support the detection of transaction state dependency vulnerabilities and Nested call vulnerabilities, we compared our method solely with these three tools. The results show that our method improved accuracy by 14.7% and 26% in detecting transaction state dependency vulnerabilities and Nested call vulnerabilities, respectively, compared to the best-performing traditional tool, Mythril.

4.6. Comparison of Runtime Resource Consumption

To verify the usability of the proposed method, we evaluated the average time and memory consumption of the proposed method and other baseline methods in detecting contracts in the dataset. Since only reentrancy vulnerabilities can be detected by all methods, to ensure the fairness of the evaluation and the accuracy of the results, we screened out contracts containing reentrancy vulnerabilities and security contracts from the original dataset to form a sub-dataset, and shut down all background processes for evaluation in a clean experimental environment.

Traditional tools are primarily based on techniques such as symbolic execution, static analysis, or fuzz testing. These methods have most of their computational demands concentrated on the CPU and incur memory overhead during both static analysis and dynamic execution. Therefore, we measured their memory consumption. For deep learning-based methods, the VRAM consumption during inference is more indicative of model complexity and resource utilization. Thus, we primarily measured their VRAM consumption during the inference process. To ensure the reliability of the results, we ran each experiment multiple times in the clean environment and averaged the results to obtain the final evaluation. The results are shown in

Table 5. Running time and memory consumption of different methods.

Methods	Acc	Recall	Precision	F1	Avg Time (s)	Memory (MB)	VRAM (MB)
Smartcheck	40.18	41.36	39.15	40.11	0.92	3.58	-
Oyente	60.05	62.77	65.21	63.41	2.50	174.44	-
Contractfuzzer	68.07	60.83	48.96	54.80	1.76	85.35	-
Mythril	65.13	69.36	49.83	57.95	2.39	161.40	-
Securify	73.38	75.86	68.52	71.50	2.73	170.59	-
Slither	81.15	87.86	82.00	84.71	0.89	4.27	-
Vanilla-RNN	50.46	55.78	46.40	50.24	0.35	-	71.34
LSTM	52.78	60.09	48.14	54.06	0.44	-	125.39
GRU	58.16	72.09	51.81	60.69	0.39	-	98.35
DR-GCN	79.24	77.17	73.05	74.96	0.65	-	217.53
TMP	81.54	80.61	79.66	80.08	0.73	-	393.75
CGE	85.56	81.19	88.32	83.58	0.91	-	674.20
Our method	90.96	91.62	89.16	90.37	0.84	-	650.83

.

Among traditional tools, the static analysis-based method Slither has the fastest execution speed and consumes the least memory, because static analysis usually only needs to traverse the code once, generate an abstract syntax tree or control flow graph, and use predefined rules for analysis. The symbolic execution-based method Securify generates a large number of symbolic paths and symbolic expressions during execution. For multiple conditional branches in complex code, the symbolic path may grow exponentially, resulting in a large amount of memory consumption. The execution time and memory consumption of deep learning-based methods are highly correlated with model complexity. Although the three methods using text sequences as input have faster inference speeds, they exhibit significant deficiencies in terms of accuracy. In contrast, methods based on graph structure modeling, while having increased inference time, significantly improve detection accuracy by better capturing the structural information within smart contracts. Given the substantial economic value associated with smart contracts, we believe that increasing computational resource consumption to enhance detection accuracy is both reasonable and worthwhile.

Compared to Slither, the best-performing traditional method, our approach achieves a 13% improvement in accuracy with similar execution time, greatly reducing the likelihood of missing potential vulnerabilities. Additionally, compared to CGE, the best neural network-based model, our method achieves a 6% increase in detection accuracy while maintaining comparable inference time and VRAM consumption. This demonstrates that our method not only improves performance while keeping resource consumption stable but also more effectively leverages model complexity to enhance the detection of contract vulnerabilities. Our method can provide higher security in actual smart contract audits, especially for high-value smart contracts. This performance improvement can effectively reduce the risk of economic losses caused by vulnerabilities. In addition, maintaining a balance between reasoning efficiency and resource consumption also makes our method more practical in large-scale or batch contract audit scenarios.

5. Conclusions

In this paper, we propose a smart contract vulnerability detection method based on pre-training technology and heterogeneous contract semantic graphs. Compared with existing methods, the heterogeneous contract semantic graph proposed in this paper better captures the dependency between program variables and functions. The graph embedding is achieved through the heterogeneous graph neural network HGT, and the introduction of pre-training technology enables efficient detection of smart contract vulnerabilities. Experimental results show that the proposed method performs well in detecting four of the most severe and widespread vulnerabilities. This approach not only covers a broader range of vulnerability types but also achieves higher detection accuracy compared to existing methods. It serves as an effective tool for the preliminary screening of vulnerabilities in smart contracts, significantly improving the efficiency with which developers identify potential issues. The method holds broad application prospects in the field.

However, our method has certain limitations:

• The scope of our method is currently limited to Ethereum and Solidity, and it does not support the detection of vulnerabilities in smart contracts written in other languages (such as Vyper, Rust, and Go).
• The dataset used in our experiments was manually curated based on existing research, and this process may introduce some degree of subjectivity, potentially leading to false positives.
• This study focuses on verifying the effectiveness of our approach using only four specific vulnerability types. As blockchain technology evolves and programming languages continue to advance, the types of vulnerabilities will also diversify.

6. Future Work

In future work, we aim to extend the capabilities of our method in several key areas. First, we plan to broaden the scope of our approach to support vulnerability detection in smart contracts written in other blockchain programming languages, such as Vyper, Rust, and Go. This will enable our method to be applied across a wider range of platforms and ecosystems beyond Ethereum and Solidity, making it more versatile and robust for developers working with different technologies. Second, we plan to extend our evaluation beyond the four specific vulnerability types currently used. As blockchain technology evolves, new types of vulnerabilities will emerge, and it is crucial to ensure that our method can detect a broader array of vulnerabilities. This includes incorporating emerging vulnerability types such as Gas optimization issues, flash loan attacks, etc. Furthermore, In addition, our current approach focuses on detecting vulnerabilities in a single smart contract. As blockchain technology continues to develop, decentralized applications (DApps) are becoming a mainstream use case for blockchain systems. Therefore, developing methods for detecting vulnerabilities between multiple interconnected contracts and components in DApps will become a valuable and important area for future research. In the future, we will explore the possibility of applying the current approach to DApp vulnerability detection.