Research on Intelligent Perception Algorithm for Sensitive Information

Abstract

In the big data era, a tremendous volume of electronic documents is transmitted via the network, many of which include sensitive information about the country and businesses. There is a pressing need to be able to perform intelligent sensing of sensitive information on these documents in order to be able to discover and guarantee the security of sensitive information in this enormous volume of documents. Although the low effectiveness of manual detection is resolved by the current method of handling sensitive information, there are still downsides, such as poor processing effects and slow speed. This study creatively proposes the Text Sensitive Information Intelligent Perception algorithm (TSIIP), which detects sensitive words at the word level and sensitive statements at the statement level to obtain the final assessment score of the text. We experimentally compare this algorithm with other methods on an existing dataset of sensitive Chinese information. We use the metrics measuring the accuracy of the binary classification model, where the F1 score reaches 0.938 (+0.6%), and the F2 score reaches 0.946 (+1%), and the experimental results fully demonstrate the superiority of this algorithm.

1. Introduction

With the development of technology, countries and enterprises are gradually reducing the number of paper documents in favor of electronic documents. The information contained in these documents is likely to contain sensitive information, so the organizations concerned need to conduct sensitivity checks on these huge volumes of documents. However, manual reviews are too time-consuming and costly, given the sheer volume of electronic papers. As a result, countries and enterprises have started utilizing artificial intelligence to automatically perceive and categorize electronic documents based on whether they contain sensitive information. However, the speed has been increased while the accuracy rate has been less than ideal when using conventional machine classification methods. Therefore, it has become an urgent goal to improve the quality of sensitive information in electronic documents while ensuring the speed of intelligent perception.

There are two types of electronic document classification methods, the first of which is the vector space model. These models can analyze document features by computing vectors, and they can assess the semantic relationships between words by measuring the distance of vectors in space. The earliest one-hot technique [1] turns the vocabulary into a binary code, transforming the discrete text into Euclidean space, where each word corresponds to a point, and the feature relations between the words can be determined by measuring the distance between points. The one-hot method, however, is unable to capture the semantics of words and the links between words. When the vocabulary is exceedingly broad, the word vector it represents has an excessively large and sparse dimensionality, which makes the calculation difficult. A sentence is represented by a vector whose dimension is equal to the number of words in the lexicon in the bag of words model [2], which introduces word frequency on the basis of one-hot. The bag-of-words approach partially solves the problem of vector sparsity compared to the one-hot method, but it does not take into account word order as well as relationships between words and is unable to capture contextual information. The N-gram method [3], which is based on the bag-of-words model, takes all neighboring N words into account as a whole, assuming that each word is related to the n-1 words that came before it or that the n-th word can be inferred from the preceding n-1 words. It is common to use the bi-gram method with two words as a whole and the tri-gram method with three words as a whole. The probability of the text as a whole is the product of the probabilities of the word occurrences. Although this approach includes contextual information and takes word order inside phrases into account, it results in more severe data sparsity when the vocabulary is large. The term frequency-inverse document frequency (TF-IDF) method [4] determines whether a word is unique to this document by taking into account its frequency in this document and its frequency in other documents in the corpus. This method incorporates lexical features and textual information into the word vector. However, it still does not reflect word position information and contextual information, and the semantic relationships that exist between words cannot be characterized by textual semantic relationships through frequency alone.

The second method is the language model based on neural networks, which has become very popular in recent years. Inspired by the CNN neural network structure [5], Bengio et al. [6] used the first n-1 words to predict the probability of occurrence of the n-th word using contextual modeling in order to produce a word vector representation of each word. However, manual extraction results in a restricted ability to represent complicated functions and perform challenging classification problems, as well as an incomplete and inaccurate representation of emotion. Mikolov et al. proposed the Word2vec model [7], which contains the CBOW and Skip-gram model approaches to obtain word vectors by computing the association of words with their contexts. The CBOW model predicts the middle target word given the surrounding words, whereas the Skip-gram model predicts the surrounding words given the middle words. These two methods’ computing complexity is too great when the vocabulary is broad, though. Pennington et al. proposed the Glove (global vector for word representations) model [8] to solve the issue that the Word2vec model does not take global aspects into account. This model applies global features while also taking into account local contextual data. Language model pre-training combined with word vector training has grown to be the most popular technique available. Later, Google made improvements to the RNN model’s slow parallel computation issue [9] and introduced the Transformer model [10], an encoder and decoder model made up entirely of attention mechanisms that addressed the RNN model’s inability to be calculated in parallel. Devlin et al. [11] then built the Bert model by extracting the encoder from the Transformer model, which produced the best test scores on 11 natural language processing tasks. In 2019, Baidu enhanced the Bert model’s masking technique and offered the ERNIE model with entity machine masks and phrase-level masks [12], and its trial results on five Chinese NLP tasks produced the best results. Baidu proposed the ERNIE 2.0 model [13] in 2020, which is based on the ERNIE model and continuously learns multi-tasks while performing pre-trained models on created tasks to produce the best results on 16 tasks. However, the word embedding of the result shows that the information about significant entities in it has not been properly applied.

To solve these problems, we propose a sensitive information perception model based on ERNIE and knowledge graph embedding (KG-ERNIE), which can use a pre-trained text encoder to encode the text, then the convolutional learning of intermediate vectors and the extraction of semantic information to perform intelligent perception of sensitive information on the text. The TSIIP algorithm, which uses word-level, statement-level, and text-level text detection, is proposed. The following are the significant contributions of the work in this paper:

(1) Based on THUNews and Chinese Wikipedia, a Chinese sensitive information dataset, JWBD, and sensitive vocabulary are built, and the proposed KG-ERNIE model and the TSIIP algorithm are trained and tested using these datasets.

(2) The KG-ERNIE sensitive information detection model is proposed, which employs a knowledge graph-based entity embedding technique and an ERNIE-based pre-training model to encode input text and extract semantic information and features. A convolutional neural network-based model (CNN) is then used to recognize and categorize the encoded information.

(3) The TSIIP algorithm, which perceives sensitive words at the word level and sensitive statements at the statement level to determine the final evaluation score of the text, is created and applied to the proposed KG-ERNIE model to solve the problems with low machine recognition accuracy of text-sensitive information and only taking into account single-level semantics.

The rest of the paper is organized as follows: Section 2 presents related work. Section 3 describes, in detail, the KG-ERNIE sensitive information detection model and the TSIIP algorithm proposed in this paper. Section 4 presents the experiments and analysis related to this work. Section 5 summarizes our work and provides a future outlook.

2. Related Work

With the proliferation of information on a global scale, intelligent perception of sensitive information has become an increasingly important area of research. Sensitive information keyword-matching algorithms were utilized in 2015 by Berardi et al. [14] to locate classified documents. Since keyword lexicons are manually generated, subjectivity can affect how accurately things are classified. Recurrent neural networks (RNN) were then employed by Neerbeky et al. [15] in 2017 to analyze the syntactic and grammatical structure of texts, find sensitive information in text documents, and assess the sensitivity values of the semantic component of the text structure. Neerbeky et al. [16] then developed a sensitive phrase recurrent neural network with an Acc value of 76.78 % to capture the intricacy of detecting sensitive information. In 2018, Xu et al. [17] presented their ideas and introduced a new topic-tracking algorithm. It monitors sensitive words over a period of time and assigns different weights to them, taking into account the number of occurrences of sensitive words in the text and the different positions in the text where the sensitive words appear, but the algorithm ignores the semantic information before and after the words as well as the semantic information at the utterance level. Then, Xu et al. [18] presented a novel approach for textual CNN-based sensitive information identification. More quickly and accurately than previous models and techniques, the training time of the detection model was reduced while maintaining detection accuracy. A framework to more comprehensively extract data features in 2020 was put forth by Lin et al. [19] in order to improve detection outcomes. The Bi-LSTM and CNN structures are the foundation of the system. Convolutional neural networks are utilized to effectively extract local features from unstructured documents, and Bi-LSTM networks are constructed to extract global features. In the same year, Hong et al. designed a brand-new framework to identify sensitive data using a network traffic recovery method [20]. A scalable convolutional neural network (CNN) and a bidirectional long short-term memory model (Bi-LSTM) with a multi-channel structure were proposed in 2021 by Gan et al. [21] to detect sensitive information in Chinese texts, analyze the sentiment tendency of Chinese texts, and introduce attention mechanisms to improve the model’s performance.

Due to the widespread use of the Transformer model [10] and the Bert model [11] in 2019, a number of academics have presented numerous solutions based on these two models. A sensitive information classification model based on BERT-CNN was presented by Wang et al. [22], which enhances the word embedding’s generalization capability and effectively classifies network sensitive information in short text datasets. Using a variation recognition and similarity calculation method that includes variations in synonyms, pronouns, acronyms, and word forms, followed by a combination of rules, Fu et al. [23] proposed sensitive word detection method uses variation recognition and association analysis to identify and judge sensitive words. However, the running cost is too high, and it is more difficult to apply to constantly changing sensitive words. Pablos et al. [24] used a sequence tagging algorithm based on the BERT model to find and delete sensitive data from Spanish clinical literature in 2020 to protect the privacy of personal data. A method to extract sensitive information from unstructured data using a hybrid content-based and context-based extract mechanism was proposed by Guo et al. [25] in 2021. In 2022, Cong et al. [26] combined the BERT framework and knowledge graph to form the KGDetector framework for detecting Chinese sensitive information, and the CNN+FC classifier proposed in the paper was used in their construction of a Chinese sensitive information dataset that was experimented on, and the F1 value was 93.7%. In the same year, Gibert et al. [27] successfully generated four datasets for Spanish-named entity detection in the legal sector, utilizing the MAPA project framework and the Transformer model for the de-identification of sensitive data in real-world scenarios. Lelio et al. [28] proposed a method for the automatic anonymization of personal data that can extract linguistic features from the text while masking the sensitive information found. The dataset utilized for the tests in the paper is made up of court documents from the Italian Supreme Court. In order to improve the model’s ability to recognize sensitive features and to increase the rate at which sensitive information is detected, Huang et al. [29] proposed an approach to sensitive information detection approach based on language model word embedding [30] and attention mechanisms (A-ELMo). However, the ability of the ELMo model to extract features is different from that of the Transformer model [10].

Therefore, algorithm [17] based on the review of sensitive information based only on the frequency and location of occurrence of sensitive words was improved, and we proposed the TSIIP algorithm, which perceives sensitive words at the word level and the semantics of sensitive sentences at the sentence level, and finally obtains the overall evaluation score of the input text.

3. Methodology

In this section, we will introduce the KG-ERNIE model and the TSIIP algorithm in detail. The architecture of the KG-ERNIE model is shown in Figure 1, and the KG-ERNIE model mainly consists of two parts: text encoder and classifier. The following sections will introduce these two parts in turn.

3.1. Text Encoder

The task of the text encoder is to extract useful intermediate semantic information from the incoming text and to encode it in a way that the computer can understand. It has an entity embedding module based on knowledge graphs and an ERNIE-based encoding module.

3.1.1. ERNIE-Based Encoder Module

Text consists of sentences, each of which consists of one word, but the classifier needs a vector or tensor as its input. We extract semantic information from the raw input text in a contextual representation using the ERNIE model, which represents text as vectors. The dynamic word vectors produced by the ERNIE pre-training model, as opposed to static word vectors, are better able to represent contextual cues and prevent the production of multiple meanings for a single word. The architecture of the ERNIE model is similar to that of the BERT model, which is based on the bi-directional Transformer’s encoder.

ERNIE and BERT models are both trained using a Masked Language Model (MLM). For the differences between Chinese and English, the ERNIE model is an improvement over the BERT model, which masks words as units. Since in Chinese, the meaning of a single word is often very different from that of its constituent words, and at the same time, words occur very frequently, using words alone for masking training would lose the information about the features between words, and would not allow good learning of the information and knowledge expressed in the sentence by the word in which the word is located. The ERNIE model includes both word-based and entity-based masking training strategies. A comparison of the two masking approaches is shown in Figure 2. For example, in the sentence “Harbin is the capital of Heilongjiang, an international city of ice and snow culture”, the BERT model covers up the “r” of “Harbin” and the “hei” of “heilongjiang”, the “internal” and “snow” in “international ice and snow”. “The ERNIE model covers up the complete words “Harbin” and “ice and snow”. This can better enhance the association between words better learn the feature information and knowledge of words, and strengthen the ability of word vectors to express Chinese semantic information.

Specifically, the operational flow of the ERNIE module is shown in Figure 3.

The pre-processed text sequence t of length n is input into the ERNIE module. First, t is fed into the embedding layer, which transforms the text sequence t into a vector $\left\{e_1,\cdots ,e_n\right\}$, as shown in (1).

$$\left\{e_1,\cdots ,e_n\right\}=Embedding\left(t\right),t=\left[CLS\right]t_1{t}_1\cdots ,t_n$$

(1)

where $t_1{t}_2\cdots t_n$ is the word in the text and [CLS] is the statement start symbol. Then the vector $\left\{e_1,\cdots ,e_n\right\}$ is input into the multi-headed self-attentive module (MH-ATT) [10] to obtain $\left\{w_1,\cdots ,w_n\right\}$, as shown in (2).

$$\left\{w_1,\cdots ,w_n\right\}=MH-ATT\left(\left\{e_1,\cdots ,e_n\right\}\right)$$

(2)

The obtained $\left\{w_1,\cdots ,w_n\right\}$ is input into the prefix network (FFN) [10] to obtain the final output of the encoder module of $\left\{w_1,\cdots ,w_n\right\}$, as shown in (3), consisting of two linear transformations with a ReLU activation function.

$$\left\{w_1^0,\cdots ,w_n^0\right\}=FFN\left(\left\{w_1,\cdots ,w_n\right\}\right)=max(0,\left\{w_1,\cdots ,w_n\right\}{W}_1+b_1)W_2+b_2$$

(3)

3.1.2. Knowledge Graph-Based Entity Embedding Module

The output $\left\{w_1^0,\cdots ,w_n^0\right\}$ represents the semantic information of the input context from Section 3.1.1, but the specific and valuable information of the entities contained in it may not be highlighted, for example, if the text contains entities: names of people, places, etc., then this text may be biased to introduce the content related to that entity. To this end, in addition to using the ERNIE model to encode the text input to extract the text semantic information of the text, we also need to extract the entities $\left\{e_1^0,\cdots ,e_n^0\right\}$ from the text and use the entity embedding module of the knowledge graph to obtain the representative intermediate information among them. We refer to the work of Zhang et al. [31] and train a Knowledge Graph Entity Embedding Module (KG) using CN-DBpedia to help construct semantically richer intermediate vectors with specific information of named entities.

Specifically, the operational flow of the KG module is shown in Figure 4.

We combine the semantic information vector $\left\{w_1^{(i-1)},\cdots ,w_n^{(i-1)}\right\}$ and the entity information vector $\left\{e_1^{(i-1)},\cdots ,e_n^{(i-1)}\right\}$ as inputs into layer i (i starts from 1), and the two vectors are fed into two multi-headed self-attentive modules (MH-ATTs) to obtain the layer i outputs of $\left\{w_1^{\left(i\right)},\cdots ,w_n^{\left(i\right)}\right\}$ and $\left\{e_1^{\left(i\right)},\cdots ,e_n^{\left(i\right)}\right\}$, as shown in (4) and (5):

$$\left\{{\tilde{w}}_1^{\left(i\right)},\cdots ,{\tilde{w}}_n^{\left(i\right)}\right\}=MH-ATT\left(\left\{w_1^{(i-1)},\cdots ,w_n^{(i-1)}\right\}\right)$$

(4)

$$\left\{{\tilde{e}}_1^{\left(i\right)},\cdots ,{\tilde{e}}_n^{\left(i\right)}\right\}=MH-ATT\left(\left\{e_1^{(i-1)},\cdots ,e_n^{(i-1)}\right\}\right)$$

(5)

where we integrate the contextual semantic information sequences and entity sequences and calculate the input embedding for each semantic and entity for the semantic $w_j^{\left(i\right)}$ and the corresponding entity $e_k^{\left(i\right)}$, $h_j$ is the hidden internal state of the integrated semantic and entity information, “$\sigma$” is the nonlinear activation function Gelu, b is the deviation term, $W_{ht}^{\left(i\right)}$, $W_{he}^{\left(i\right)}$, $W_t^{\left(i\right)}$, and $W_e^{\left(i\right)}$ are the corresponding weights, and $b^{\left(i\right)}$, $b_t^{\left(i\right)}$, $b_e^{\left(i\right)}$ are the corresponding bias, as shown in (6)–(8), respectively.

$$h_j=\sigma (W_{ht}^{\left(i\right)}{\tilde{w}}_j^{\left(i\right)}+W_{he}^{\left(i\right)}{\tilde{e}}_k^{\left(i\right)}+b^{\left(i\right)})$$

(6)

$$w_j^{\left(i\right)}=\sigma (W_t^{\left(i\right)}{h}_j+b_t^{\left(i\right)})$$

(7)

$$e_k^{\left(i\right)}=\sigma (W_e^{\left(i\right)}{h}_j+b_e^{\left(i\right)})$$

(8)

For the semantics of $w_j^i$ without corresponding entities, the word embedding is output directly without semantic and entity information integration, as shown in (9) and (10).

$$h_j=\sigma (W_{ht}^{\left(i\right)}{\tilde{w}}_j^i+b^{\left(i\right)})$$

(9)

$$w_j^{\left(i\right)}=\sigma (W_t^{\left(i\right)}{h}_j+b_t^{\left(i\right)})$$

(10)

The semantic vector $\left\{w_1^{\left(i\right)},\cdots ,w_n^{\left(i\right)}\right\}$ of the last layer is represented as $R_E$ and the corresponding entity vector $\left\{e_1^{\left(i\right)},\cdots ,e_n^{\left(i\right)}\right\}$ as $R_K$ is output to the classifier.

Therefore, the text encoder works as follows First, the pre-processed text sequence t of length n is input into the embedding layer of the ERNIE module, and t is transformed into the vector $\left\{e_1,\cdots ,e_n\right\}$, then the vector $\left\{e_1,\cdots ,e_n\right\}$ is input into the MH-ATT layer to extract the semantic information $\left\{w_1,\cdots ,w_n\right\}$, and finally the semantic information $\left\{w_1,\cdots ,w_n\right\}$ is fed into the FFN layer to obtain the final output of the ERNIE module $\left\{w_1^0,\cdots ,w_n^0\right\}$ and the entity information $\left\{e_1^0,\cdots ,e_n^0\right\}$. Then, the semantic information $\left\{w_1^0,\cdots ,w_n^0\right\}$ and the entity information $\left\{e_1^0,\cdots ,e_n^0\right\}$ are input into the two MH-ATT layers of the KG module to extract the semantic information to obtain the intermediate semantic information vector $\left\{{\tilde{w}}_1^{\left(i\right)},\cdots ,{\tilde{w}}_n^{\left(i\right)}\right\}$ and the intermediate entity information vector $\left\{{\tilde{e}}_1^{\left(i\right)},\cdots ,{\tilde{e}}_n^{\left(i\right)}\right\}$, and finally infuse the two intermediate vectors to receive the final output of the whole model $\left\{w_1^{\left(i\right)},\cdots ,w_n^{\left(i\right)}\right\}$ and $\left\{e_1^{\left(i\right)},\cdots ,e_n^{\left(i\right)}\right\}$, i.e., $R_E$ and $R_K$.

3.2. Classifier

We input the context semantic vector $R_E$ and the entity vector $R_K$ obtained by the KG module in Section 3.1.2 into the concat module to concatenate the two vectors into forming a 968-dimensional vector R, as shown in (11). The flow of the whole classifier is shown in Figure 5.

$$R=R_E⨁R_K$$

(11)

The classifier compares the words in the obtained vector R. The similarity calculation formula used for the comparison is shown in (12). We compare the word, $t_i$, in the input sentence with the different types of the words, $t_j$, in the sensitive word category library and take the largest Sim($t_i$,$t_j$ ) to assign to Smw($t_i$) to receive the similarity score, Smw($t_i$), for the word, $t_i$, and classify the sentence into the class c in which the $t_j$ is located, as shown in (13).

$$\begin{array}{c}Sim(X,Y)=\frac{{\sum }_{i=1}^n\left(X_i·Y_i\right)}{\sqrt{{\sum }_{i=1}^n{X}_i^2}·\sqrt{{\sum }_{i=1}^n{Y}_i^2}}\end{array}$$

(12)

$$\begin{array}{c}Smw\left(t_i\right)=max\left(Sim\left(t_i,t_j\right)\right)\end{array}$$

(13)

Then, the classifier compares the semantic vectors V($Se{n}_i$) of the input sentences with the semantic vectors V($Se{n}_j$) of the class c of sensitive sentences in the sensitive sentences library C. $Sim(V\left(se{n}_i\right),V\left(se{n}_j\right))$, the largest $Sim(V\left(se{n}_i\right),V\left(se{n}_j\right))$ is assigned to $Sms\left(se{n}_{i}i\right)$, as shown in (14):

$$Sms\left(se{n}_i\right)=max\left(Sim(V\left(se{n}_i\right),V\left(se{n}_j\right))\right)$$

(14)

The final assessment Score of the input text is calculated by attaching weights to the statements of different categories $({\alpha }_1,\cdots ,{\alpha }_m)$ is the weight of each category of sensitive statements, which is set to 1 in this paper and can be adjusted according to the requirements, as shown in (15). If the final assessment Score exceeds the set threshold, the input text is determined to be sensitive text.

$$\begin{array}{c}Score={\alpha }_1\ast Sms\left(C_1\right)+{\alpha }_2\ast Sms\left(C_2\right)\\ \cdots +{\alpha }_m\ast Sms\left(C_m\right)\end{array}$$

(15)

Therefore, the TSIIP algorithm comes from this. The operation mechanism of the algorithm is shown in Algorithm 1. We use an example to illustrate our algorithm. The input text “US-Malaysian navy assembles in South China Sea for joint military exercises” is split into “US-Malaysian navy”, “South China Sea”, “assembly”, and “joint military exercises”. First, its segmentations are compared with the sensitive words in the sensitive vocabulary, T, and the similarity score, smw, is calculated for each word to classify it as a military sentence. Then it is compared with the sensitive sentences in the sensitive sentence library C and the similarity score, sms, is calculated for the whole sentence. If the similarity score, sms, of the sentence exceeds the threshold, S, it is classified as a sensitive military sentence; otherwise, it is a military non-sensitive sensitive sentence.

Algorithm 1 Text Sensitive Information Intelligent Perception (TSIIP) algorithm.

Require: Pending articles Text, Sensitive vocabulary T, Sensitive Threshold S, Sensitive sentences library C Ensure: Categories of Text and The sensitivity judgment of Text 1: There are m types of sensitive categories in the Sensitive vocabulary T and Sensitive sentences library C. 2: Split the pending article text into sentences, split sentences into words, and remove the stop words to get n statements. 3: for$i=1ton$do 4:    $Sms\left(se{n}_i\right)=0$ 5:    while $word\ne$ “ ” do 6:      $Smw\left(t_j\right)=0$ 7:      Compute the similarity score $Smw\left(t_j\right)$ of the input word $t_j$ based on Equation (13) 8:    end while 9:    classify the sentence $se{n}_i$ into the highest score of class c 10:    Compute the similarity score $Sms\left(se{n}_i\right)$ of the input sentence $se{n}_i$ based on Equation (14) 11: end for 12: Get all similarity score of sentence Sms in the Text 13: for$i=1tom$do 14:    $Sms\left(C_i\right)=avg(allSmsofse{n}_i\in C_i)$ 15: end for 16: Classify Text as the category $C_i$ of the highest $Sms\left(C_i\right)$ based on Equation (15) 17: if Score≥ S then 18:    Determine Text has sensitive information and belong to class $C_i$ 19: else 20:    Determine Text has not sensitive information and belong to class $C_i$ 21: end if

4. Experiment

The proposed KG-ERNIE model and the TSIIP algorithm are described in detail in Section 3. To further demonstrate the advancement and superiority of our proposed model and algorithm, we apply the KG-ERNIE model and the TSIIP algorithm to sensitive information intelligent perception datasets and compare their effectiveness with that of other state-of-the-art methods to verify the effectiveness of the KG-ERNIE model and the TSIIP algorithm in solving this problem. Since, in reality, the damage caused by leakage due to misclassification of sensitive information as non-sensitive information is far greater than the damage caused by over-protection due to misclassification of non-sensitive information as sensitive information, we set a higher penalty for misclassification of sensitive information as non-sensitive information. Therefore, we put more emphasis on the recall rate than the accuracy rate, which may result in a decrease in the accuracy rate, but will improve the recall rate.

4.1. Evaluation Criteria

For binary category classification problems, a true positive (TP) indicates that the model correctly predicts sensitive information. False positive (FP) indicates that the model predicts sensitive information for non-sensitive information. False negative (FN) indicates that the model predicts sensitive information as non-sensitive information. True negative (TN) indicates that the model correctly predicts non-sensitive information. The evaluation criteria used in this paper to measure the model performance are as follows. Since we put more emphasis on the recall, we have introduced the F2 score as an evaluation metric in addition to the F1 score.

Precision: The proportion of correctly classified positive samples to the total number of samples predicted to be positive, as shown in (16)

$$\begin{array}{c}\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}\end{array}$$

(16)

Recall: The proportion of correctly classified positive samples to the total number of all positive samples, as shown in (17)

$$\begin{array}{c}\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{TN}}\end{array}$$

(17)

F1 score: Coordinated average of accuracy and recall, as shown in (18)

$$\begin{array}{c}\mathrm{F}1=\frac{2\ast \mathrm{Precision}\ast \mathrm{Recall}}{\mathrm{Precisi}+\mathrm{Recall}}\end{array}$$

(18)

F2 score: The recall rate is weighted higher than the evaluation metric of precision, as shown in (19)

$$\begin{array}{c}\mathrm{F}1=\frac{5\ast \mathrm{Precision}\ast \mathrm{Recall}}{4\ast \mathrm{Precisi}+\mathrm{Recall}}\end{array}$$

(19)

4.2. Datasets and Parameters

In order to compare different models and algorithms, we use the THUNews dataset from Tsinghua University as the white sample of the experimental dataset and the JWBD dataset, which was composed of data manually collected from overseas blogs and Wikipedia using a crawler program, as the black sample of the experimental dataset, and we also compose our sensitive library from the collected data. We used the crawler to crawl the data of several fields and filtered out the texts containing sensitive information in different fields as black samples. For example, the military category contains information about combat, the political category contains the words of officials, and the economic category contains the economic income of companies, etc. As indicated in

Table 1. Experimental dataset.

Datasets	Training Sets	Validation Sets	Test Sets	Total
THUNews	37,800	12,600	12,600	63,000
JWBD	2998	998	998	4994

, we split the dataset into training, validation, and test sets in the proportions of 3:1:1. The parameters of the experiment are listed in

Table 2. Parameters of the experiment.

Parameter	Value
Optimizer	Adam
Learning rate	4 × 10−4
Epoch	30
Batch size	32
Active function	Relu
Dropout	0.5

.

4.3. Experiment Result

We compared the KG-ERNIE model with the schemes proposed by other researchers and used the TSIIP algorithm for sensitive information detection. As shown in

Table 3. Comparisonof experimental results. The bold numbers are the best results.

Method	Precision	Recall	F1 Score	F2 Score
KG-ERNIE+TSIIP	0.925	0.951	0.938	0.946
KGDetector	0.931	0.934	0.932	0.933
CNN-BiLSTM	0.922	0.941	0.931	0.936
TextCNN	0.866	0.914	0.889	0.904
BERT-BiLSTM-Attention	0.865	0.905	0.885	0.897

, we compared the KGDectetor model proposed by Cong et al. [26] and the CNN-BiLSTM model proposed by Gan et al. [21] and the TextCNN model proposed by Kim et al. [32] and the Bi-LSTM model proposed by Guo et al. [25]. The comparison results prove that the KG-ERNIE+TSIIP scheme outperforms the other methods in terms of recall, F1 score, and F2 score. Among them, the accuracy reached 0.925, the recall reached 0.951, which is 1% better than the CNN-BiLSTM model, the F1 score reached 0.938, which is 0.6% better than the KGDectetor model, and the F2 score reached 0.946, which is 1% better than the CNN-BiLSTM model. The results were as we had previously expected, sacrificing some accuracy but improving recall, with the best experimental results for recall, F1 scores, and F2 scores.

Then, in order to study the influence of word embedding models on the experimental results and to determine whether the word embedding model used has a greater impact on the semantic integrity extraction of the text, we replaced the word embedding ERNIE model in the text encoder with the BERT model [11], the Glove model [7], and the Word2vec model [6] for experimental comparison. The results show that the word embedding ERNIE model used in this paper gets better experimental results in terms of recall and F1 score and F2 score, with the recall reaching 0.951, the F1 score reaching 0.938, and the F2 score reaching 0.946, as shown in

Table 4. Comparison of experimental results of different word embedding models. The bold numbers are the best results.

Method	Precision	Recall	F1 Score	F2 Score
KG-ERNIE+TSIIP	0.925	0.951	0.938	0.946
Bert	0.929	0.928	0.928	0.928
Glove	0.801	0.812	0.806	0.812
Word2vec	0.834	0.827	0.83	0.828

. The results are also in line with our expectations, as the ERNIE model uses a masking approach that includes full word masking as well as entity masking, extracting more complete contextual semantic information and, therefore, outperforming the Bert model in most metrics. The accuracy of the ERNIE model is slightly lower than that of the Bert model, as we sacrificed a small amount of precision to achieve a higher recall, where the accuracy is 0.4% lower than the Bert model, but the recall is 2.3% higher than the Bert model.

To verify the stability of the model, we use accuracy and recall as indicators and judge the stability of the model based on the changes in these indicators. The experimental results are shown in Figure 6. It can be seen that the accuracy and recall of the KG-ERNIE+TSIIP solution grow gradually as the number of training times increases, and the changes in accuracy and recall tend to stabilize after about the 15th epoch. This shows that the solution of KG-ERNIE+TSIIP has strong stability.

To demonstrate the generalization of our method, we used 2044 private data from a company as a test, including 1842 non-sensitive texts and 202 sensitive texts. The results are shown in

Table 5. Experimental results from the company’s private data. The bold numbers are the best results.

	Precision	Recall	F1 Score	F2 Score
KG-ERNIE+TSIIP	0.851	0.867	0.859	0.864
KGDetector	0.701	0.681	0.691	0.685
CNN-BiLSTM	0.684	0.703	0.693	0.699
TextCNN	0.653	0.681	0.667	0.675
BERT-BiLSTM-Attention	0.710	0.702	0.706	0.704

, where the precision and recall, as well as F1 score and F2 score are lower than in Table 3. According to these results, we analyze that the sensitive words and sensitive phrases in the training dataset cannot cover all the sensitive words and sensitive sentences in the real data because the construction of the sensitive sentences library and the sensitive vocabulary are not perfect and the scale of the data is insufficient.

5. Conclusions

In this research, we propose an intelligent perception algorithm (TSIIP) and the detection and recognition model KG-ERNIE for the problem of text sensitive information perception. We design an algorithm to identify sensitive words at the word level and sensitive statements at the statement level in order to obtain the evaluation score of the final text for the issue of considering only single-level semantics. We encode the input text using an entity embedding model based on the ERNIE pre-training model and knowledge graph and then use our intelligent perception algorithm to intelligently perceive sensitive information on the input text using a convolutional neural network (CNN) as the underlying architecture. For the dataset, we created a sensitive vocabulary based on THUNews and Chinese Wikipedia, as well as the Chinese sensitive information dataset JWBD. Numerous experiments were performed on this dataset, and the F1 and F2 scores for the KG-ERNIE model and the TSIIP method proposed in this study were 0.938 (0.6% improvement) and 0.946 (1% improvement), respectively. Thus, it can be said that the TSIIP algorithm and the KG-ERNIE model surpass other current methods. Specific application scenarios may include: applying the method to the review of documents transmitted by enterprises in the network to prevent the leakage of sensitive information of enterprises or individuals by perceiving whether the transmitted text contains sensitive information; it can also be applied to the review of detecting whether sensitive information is contained in electronic documents of countries or governments to improve the efficiency and quality of the review of electronic text documents.

Although this solution has achieved good results, there is still room for improvement. In future work, we can continue to expand and improve the sensitive vocabulary and sensitive sentences library and further investigate this part of the classifier to improve both the recognition precision and the recall rate of sensitive information. Furthermore, we intend to further develop our current binary classification system into a quantitative assessment of the text sensitivity level, with sensitive information text being classified into three levels: Top Secret Level, Confidential Level, and Secret Level. We will further investigate this part of the classifier to improve both the recognition accuracy and the recall rate of sensitive information.