Autonomous Data Association and Intelligent Information Discovery Based on Multimodal Fusion Technology

Abstract

The effective association of multimodal data is the basis of massive multi-source heterogeneous data sharing in the era of big data. How to realize data autonomous association between massive multimodal databases and the automatic intelligent screening of valuable information from associated data, so as to provide a reliable data source for artificial intelligence (AI), is an urgent problem to be solved. In this paper, a data autonomous association method based on the organizational structure of data cells is proposed, including transaction abstraction based on information nucleuses, symmetric and asymmetric data association based on strategies and data pipes, and information generation based on big data. To screen meaningful data associations, an information-driven intelligent information discovery method and a task-driven intelligent information discovery method are proposed. The former screens meaningful data associations by training the reward and punishment model to simulate the manual scoring of data associations. The latter is task-oriented and screens meaningful data associations by training the reward and punishment model to simulate the manual ranking of data associations related to the task requests. Through the above work, autonomous data association and intelligent information discovery are effectively realized based on multimodal fusion technology, which provides a novel data source mining approach using multimodal data sharing and intelligent information discovery.

1. Introduction

In this age of constant innovation, AI and automation are reshaping our lives, challenging us to think, work smarter, and enhance our creativity in ways we never have before [1]. What has constrained many of the major breakthroughs in AI is not a lack of advanced algorithms, but a lack of high-quality data sets [2]. Data is critical in developing state-of-the-art machine learning techniques. To ensure the proper operation of AI solutions, large amounts of high-quality data are needed to train the underlying neural networks [3]. A good example is multilingual natural language processing (NLP), which relies on the input of thousands of artificial sample data in various languages [4].

In recent years, the emergence and refinement of many data organization and management techniques have provided new ways to access complex data, representative of which are the data virtualization methods [5,6,7] and intelligent data warehouse methods [8,9,10,11]. Data virtualization is a class of data access methods that separates logical view and physical storage. The client data requirements are automatically associated to multiple storage databases via the virtual layer, and the virtual layer compiles the returned data and feedback to users. However, the construction of the virtual layer depends on a lot of manual definitions as it is difficult for computers to understand the contextual associations between data [12]. Data warehouses are central repositories for storing and managing large amounts of structured data, integrating data from multiple data sources, and storing them uniformly in the warehouse layer. Classical intelligent data warehouse modeling methods include Entity-Relationship modeling methods [13], Dimensional Modeling methods [14], Data Vault Model methods [15], etc., which enable trend analysis and fast complex querying via historical data, etc., but these modeling methods are also manually predefined. Therefore, no matter those data processing methods that provide data processing logic, such as data virtualization or those methods that store complex data directly, such as data warehouses, the key step is to provide the logic of association between data [11]. However, as society evolves, along with the emergence of massive databases, the demand for accurate data for AI is also rising [16,17]. While, ideally, we could manually create an accurate data association for each data requirement, this is too time-consuming. On the other hand, with the emergence of new databases over time, new complex data requirements are constantly emerging, and such a huge amount of work is impossible to realize only using manual labor. The association between data can generate various kinds of new information [18,19,20], but how to realize the autonomous association of extensive multimodal data and the intelligent selection of meaningful associations to provide AI with high-quantity and high-quality complex data in real time and further enhance the application of AI is the problem that needs to be solved [21,22].

The aim of this paper is to achieve the autonomous association between multimodal data and how to train the deep learning model to realize the intelligent screening of association information. Compared with the traditional data processing methods, the proposed methods should have the characteristics of more automatic, less format limitation, and stronger real-time automatic update and so on. The intelligent knowledge discovery method proposed in this paper should not only provide high-quality data sources for AI in real time, but also provide theoretical support for data processing methods that rely on a large number of manual definitions of multimodal data.

The main contributions of this paper are summarized as follows:

• Propose a data autonomous association method for multi-source heterogeneous data based on the organizational structure of data cells. Initial data cells are constructed according to the definition of the organizational structure of data cells; the concepts of symmetric random association and asymmetric random association of data pipes are defined by analyzing the symmetric and asymmetric strategic associations of data. Information nucleuses of combined data cells containing independent association information are generated using multi-level association of data pipes. All information nucleuses of combined data cells are obtained to form the set of information nucleuses of combined data cells.
• Propose an information-driven intelligent information discovery method. A reward and punishment model is constructed and the model is trained by the manual scoring data of information nucleuses of combined data cells. Ultimately, the trained model replaces the manual labor in intelligent scoring of information nucleuses and realizes the intelligent screening and storage of meaningful information nucleuses, which contain a large number of meaningful associated information.
• Propose a task-driven intelligent information discovery method. Based on realistic task requests, the collection of subject headings related to the task request is obtained by the topic model of natural language understanding (NLU). Subject headings are used to search for information nucleuses of initial data cells to obtain the set of matched initial data cells. The data pipes of matched initial data cells are randomly associated to form combined data cells, from which the set of information nucleuses of the matched combined data cells related to task requests can be obtained. A reward and punishment model is constructed and the model is trained using the manually sorted data of information nucleuses of combined data cells related to task requests. The trained model achieves intelligent sorting of information nucleuses, realizing task-driven intelligent information discovery.

The remainder of this paper is mainly organized as follows: Section 2 introduces the related works of this paper; Section 3 introduces the methods of this paper, including the organizational structure of data cells, the data autonomous association method based on the organizational structure of data cells, the information-driven intelligent information discovery method, and the task-driven intelligent information discovery method; Section 4 gives examples of the proposed method; Section 5 discusses the contribution, innovation, application fields, and feasibility of the proposed method; Section 6 introduces the conclusion of this paper.

2. Related Works

The related works involved in this paper include multi-source heterogeneous data fusion, data association, and autonomous knowledge discovery and data self-intelligence. This chapter analyzes the current research situation from the above points.

2.1. Multi-Source Heterogeneous Data Fusion

Multi-source heterogeneous data fusion is a process of cognition, synthesis, and judgment from a large number of multi-source heterogeneous data to obtain more comprehensive, accurate, and reliable information. The key to multi-source heterogeneous data fusion is to deal with the differences between different types of data, including text, pictures, audio, video, and mixed data [23].

Data fusion methods can be divided into three categories according to the fusion level [24]: the data-level methods, the feature-level methods, and the decision-level methods. The data-level fusion methods integrate the original multi-source heterogeneous data to form a new unified data set, which retains the original data information but relies on a lot of computing, so the real-time performance is poor. Liu et al. converted all data to the resource description framework (RDF) data format and input the converted data into the data fusion framework to realize information fusion [25]. Ji et al. proposed a BPR-Review-Score-Social (BRScS) data fusion model to fuse social relationships and user comments, which realized user merchant recommendations based on multi-source and heterogeneous information [26]. The feature-level fusion methods use feature fusion algorithms to generate new representative feature vectors by extracting all kinds of original data feature vectors. Compared with the data-level fusion methods, the feature-level fusion methods are more efficient because they simplify the original data, but the disadvantage is that feature extraction accompanies information loss. Liu et al. mapped the Mel-scale Frequency Cepstral Coefficient (MFCC) features of sound and the image features extracted using Convolutional Neural Networks (CNNs) using subspace, and retrieved them with Euclidean distance, thus realizing the solution of the cross-modal surface material retrieval of the hearing to the vision [27]. The decision-level fusion methods are also called post-fusion methods (decision-level fusion after classifier coding), which conduct coordination and joint decision by finding out the credibility of each mode. The commonly used methods are averaging, voting, weighting, adaptive enhancement, dynamic Bayesian network, and so on. For example, Meng et al. proposed a decision-level fusion method based on a binary classification model and the evidence theory [28], which uses logical regression and a Support Vector Machine (SVM) model to solve the binary classification problem, and then the evidence theory was used for decision confusion.

2.2. Data Association and Autonomous Knowledge Discovery

Knowledge discovery [29] is a process of automatically extracting novel and understandable potential patterns and trends from massive data, and it is an important field for discovering new knowledge via information organization. Data association is an important means of potential knowledge discovery via the integrated expression, management, and retrieval of data with different sources, structures, formats, and characteristics [30].

In recent years, scholars have put forward a series of innovative ideas on knowledge discovery based on data association. For example, Abdulahi Hasan et al. extract knowledge from large data sets using data mining methods based on clustering sampling and realize future prediction and decision analysis based on knowledge discovery [31]. With the rise and development of machine learning, the research of knowledge discovery based on data association has become more extensive. For example, Mollaei et al. realize knowledge discovery using association rules (ARs) and natural language processing (NLP) and effectively extract hidden patterns from different types of data sharing records [32]. The knowledge discovery of machine learning is becoming more and more popular, and knowledge graphs are widely used in knowledge discovery methods based on data association. Cao et al. use a knowledge graph to organize interdisciplinary knowledge and reveal potential knowledge relevance, providing strong support for interdisciplinary scientific research and innovation [20]. Deng et al. attempted the knowledge discovery of oral historical archive resources in the field of digital humanities, carried out multi-dimensional knowledge discovery research on the aspects of the overall overview of the project and the relationship between events and themes [33], and constructed a knowledge graph for experimental data sources, which formed a high-dimensional knowledge representation space which was closely related to the data examples of oral historical archives, which provided a new way to mine the hidden content of human resources and enriched the research system of autonomous knowledge discovery.

2.3. Data Self-Intelligence

The data self-intelligent methods are a series of intelligent methods that can learn and optimize data independently. Data intelligence uses technologies such as data mining and machine learning to extract valuable implication patterns from data [34] and has two characteristics: driven by big data and guided by the application scenario. The data self-intelligent methods use machine learning and data analysis technology to achieve automatic and intelligent data effective information acquisition [35], which is widely used in various fields.

At present, the data self-intelligent methods include knowledge-driven types and data-driven types [36]. Knowledge-driven-type methods use existing models, algorithms, and a series of prior knowledge to carry out multi-dimensional and interdisciplinary analyses of data on complex topics [37]. Zhang et al. use a knowledge graph to obtain the embedded representation of the data structure via the TransR model and obtain the hidden information in the data via the inner product calculation [38]. The data-driven-type methods guide decision-making via data information and patterns [39]. For example, Sarker put forward the viewpoint of “data science” from the point of view of data-driven and carried out data-driven intelligent computing and decision-making according to different application scenarios to provide advanced data analysis references for 10 practical application fields [40]. To integrate the advantages of two kinds of methods, scholars have built knowledge-driven and data-driven collaborative models [41,42,43,44,45,46]. Through the complementary advantages of the two paradigms, hybrid models are established to further realize data intelligence while following the scientific mechanism.

3. Methodology

This paper proposes the methods of autonomous data association and intelligent information discovery for multi-source heterogeneous data (the flowchart is shown in Figure 1): (1) Define the organizational structure of data cells for multi-source heterogeneous data; (2) Initial data cells are constructed according to the organizational structure of data cells, including the construction of information nucleuses of initial data cells by extracting field names and key-value names from databases or technical names used to parse other critical information, the construction of strategies by predefined multiple algorithms and scripts, and the construction of data pipes. According to the symmetric and asymmetric random association concepts of data pipes, information nucleuses of combined data cells containing associated information are generated, and strategies and data pipes are constructed to realize the construction of combined data cells. (3) Meaningful information nucleuses are exacted in two ways: The first one constructs a reward and punishment model and the model is trained using the manual scoring data of combined data cells; the trained model replaces manual scoring and intelligent screening of information nucleuses, and the cloud brain is used to store meaningful nucleuses to achieve information-driven intelligent information discovery. The second one starts with a series of task requests. The collection of subject headings according to task requests is obtained by the topic model of NLU, and subject headings are used to search for information nucleuses of combined data cells to obtain the set of matched initial data cells; data pipes of matched initial data cells are randomly associated and form the set of information nucleuses of combined data cells; A reward and punishment model is constructed and the model is trained using the manually sorted data of information nucleuses of combined data cells related to task requests. Finally, the trained model achieves intelligent sorting of information nucleuses for the given task request, and the cloud brain is used to store the optimal matching information nucleuses to realize task-driven intelligent information discovery.

3.1. Data Cells

This part introduces the organizational structure of data cells. Data cells achieve autonomous data association via independent cognitive mechanisms and data processing patterns (see Figure 2).

A data cell is composed of an information nucleus, a series of strategies, and data pipes corresponding to strategies one by one. Data cells have spatial-temporal properties. The temporality is reflected in the process of birth, existence, and apoptosis of data cells, and the spatiality is reflected in the spatial location attribute of data cells. The data pipes depend on strategies that can be associated with each other and generate combined data cells, so data cells are hierarchical. The association of data pipes is primary and secondary, as the association request is sent by the primary data pipes and the request is accepted by the secondary data pipes. Data cells formed by the association of multiple initial data cells are called combined data cells. The combined data cells composed of two initial data cells are called binary combined data cells; the combined data cells composed of $\lambda$ initial data cells are called $\lambda -tuple$ combined data cells, and the unassociated data cells are called the initial data cells. The structure of the initial data cell is shown in Figure 2a and the structure of the binary combined data cell is shown in Figure 2b. In Figure 2a, $c_u$ represents the information nucleus of the initial data cell $u$, $s_{u,i}$ represents the $i-th$ strategy of $u$, and ${oc}_{u,i}$ represents the data pipe corresponding to $s_{u,i}$. The initial data cell $u$ can be expressed as:

$$u=\{c_u,S_u,O_u\}$$

(1)

$$S_u=\left\{s_{u,i}\right\},1\le i\le n$$

(2)

$$O_u=\left\{{oc}_{u,i}\right\},1\le i\le n$$

(3)

where $S_u$ represents the set of the strategies, and $O_u$ represents the set of data pipes. Figure 2b shows the structure of the binary data cell $\left(u,v\right)$, in which the information nucleus is generated by the association of data pipe ${oc}_{u,i}$ of $u$ (the master data pipe) and the data pipe ${oc}_{v,j}$ of $v$ (the secondary data pipe). For convenience of representation, a sketch of the binary data cell $\left(u,v\right)$ is shown in Figure 3.

Without losing generality, the association of data pipes is expressed as:

$$r({oc}_{u,i}^{\ast },o{c}_{v,j}^{\ast }){,oc}_{u,i}^{\ast }\in u,{oc}_{v,j}^{\ast }\in v$$

(4)

where $u$ and $v$ are data cells ($u$ and $v$ can be the same); ${oc}_{u,i}^{*}$ is the primary data pipe; $o{c}_{v,j}^{*}$ is the secondary data pipe; $r({oc}_{u,i}^{*},o{c}_{v,j}^{*})$ represents the association of the data pipe ${oc}_{u,i}^{*}$ and the data pipe $o{c}_{v,j}^{*}$.

Since the combined data cells are composed of initial data cells, the definition of elements and the relationships among the elements of the data cells are illustrated here using the initial data cell as an example, including the information nucleus, the strategies, and the data pipes:

• Information nucleus: The information nucleus is the core constituent of the initial data cells, which carry the intelligent implementation of data. Strategies and data pipes exist based on the information nucleus. There is no one-to-one correspondence between the information nucleus and things in the real world, but a kind of abstraction of transactions with the same characteristics. The information nucleus is embeddable and objective: The embeddability of the information nucleus refers to the semi-automatic or automatic association between the information nucleus and the database system; the objectivity of the information nucleus means that although information nucleuses do not correspond to the real world one by one, they all have practical significance.
• Strategies: Strategies refer to a set of algorithms and scripts, which is the method and process of realizing specific transactions based on the data and logic provided by the information nucleus. Strategies are divided into internal strategies and external strategies. It has been explained earlier that the data cells are hierarchical, so the internal strategy and the external strategy are relative. For example, strategies within the combined data cells are internal strategies relative to the combined data cell and are external strategies to the initial data cells that make up the combined data cell.
• Data pipes: Data pipes are channels via which the initial data cells carry out information transfer and communication. Data pipes are not independent but have a one-to-one relationship with strategies. The construction of the information nucleuses of combined data cells is based on the association of data pipes of initial data cells.

3.2. Construction of Initial Data Cells

Initial data cells are directly linked to databases and their construction includes the information nucleus construction, strategy construction, and data pipe construction.

3.2.1. Information Nucleus Construction

Information nucleuses are common properties or characteristics of data in the database, which is a class of abstractions with the same characteristic transactions (see Figure 4). Some databases have fields or key-values with specific meanings, such as table-type relational databases and document-type non-relational databases (Figure 4a,b). For these databases, the field names or key-value names are directly extracted as information nucleuses. However, some special databases do not have the original specific attributes or descriptive information, such as image-type databases or video-type databases. For this kind of database, pre-processing technologies are used to transform the original data into various types of implicit information to replace the original information, such as Image/Video Semantic Analysis and Understanding [47], Image/Video Description [48], Image/Video Comprehension [49], Video Caption [50], and Video Target Relationships Mining [51], and technical names are used as information nucleuses, which represent that the data is processed using the specified technology (Figure 4c,d). The information nucleuses of initial data cells and the field names/key-value names/technical names have a n-to-n relationship:

$$B:\left\{c_1,c_2,\dots ,c_{\alpha }\right\}\leftarrow \left\{k_1,k_2,\dots ,k_{\alpha }\right\}$$

(5)

where $\left\{k_1,k_2,\dots ,k_{\alpha }\right\}$ represents the set of key-value names/field names/technical names; $\left\{c_1,c_2,\dots ,c_{\alpha }\right\}$ represents the set of the information nucleuses.

3.2.2. Strategy Construction

Strategy construction includes underlying strategy construction, advanced strategy construction, and the combination strategy construction.

Underlying strategies include input and output strategies, data normalization strategies, mathematical operation strategies, logical operation strategies, and so on.

• Input strategy and output strategy.

Input strategy and output strategy are the most basic and most frequently applied strategies in initial data cells, which are expressed as:

$$F_I=Input\left(\right)$$

(6)

$$F_O=Output\left(self\right)$$

(7)

where $Input\left(\right)$ function refers to requesting external input, and $Output\left(self\right)$ function refers to the output of the data associated with the information nucleus.

2.

Data normalization strategies.

Data normalization strategies map the data to a unified dimension to keep the data on the same metric scale, which is the premise of scale-independent operation of data:

$$F_D\left(obj\right)={dim}_{self}\left(obj\right)$$

(8)

where ${dim}_{self}\left(obj\right)$ represents the function that maps $self$ to the dimension of $obj$ such as the input layer data normalization method, the hidden layer data normalization method, the stream data normalization method, the stream data input layer normalization method, and so on [52].

3.

Mathematical operation strategies.

Mathematical operation strategies refer to the four operations of addition, subtraction, multiplication, and division:

$$F_{Math}\left(obj\right)=self\mathrm{M}obj$$

(9)

where $\mathrm{M}$ specifically represents the “addition ($+$)”, “subtraction ($-$)”, “multiplication ($\times$)”, and “division ($/$)” operations of numerical values, the logical operations of the database, and so on.

4.

Logical operation strategies.

Logical operation strategies include the and-or-non-operation of data:

$$F_{Logic}\left(obj\right)=self\mathrm{L}obj(or{F}_{Logic}\left(obj\right)=\mathrm{L}obj)$$

(10)

where $\mathrm{L}$ represents binocular logical operations (e.g., “and”, “or”, “different-or”, “same-or”, “with-no”, “or-no” operations) and monadic logical operations (e.g., non-operation, etc.).

Advanced strategies refer to complex algorithms or scripts, which include association rule strategies, spatio-temporal relationship strategies, empirical model strategies, and so on.

• Association rule strategies.

The association rule strategies are used to mine frequent items and corresponding association rules in a series of transaction sets.

Suppose itemset $C=\{C_1,C_2,\dots ,C_m\}$, $\forall$ transaction $t_{\mathrm{i}}\subseteq C$. Given a transaction set ${\rm T}=\left\{t_i\right\},1\le i\le n$. Association rules are the implication of event $X\to Y$ in the transaction set ${\rm T}$, which includes support and confidence:

$$Support\left(X\to Y\right)=\frac{p\left(X\cap Y\right)}{n}$$

(11)

where $X$ and $Y$ represent events, $p\left(X\cap Y\right)$ represents the probability of the simultaneous occurrence of events $X$ and $Y$ in set ${\rm T}$, $n$ represents the total number of transactions in set ${\rm T}$, and $Support\left(X\to Y\right)$ represents the support of the event $X\to Y$.

$$Confidence\left(X\to Y\right)=\frac{p\left(X\cap Y\right)}{p\left(X\right)}$$

(12)

where $X$, $Y$ represent events, $p\left(X\cap Y\right)$ represents the probability of the simultaneous occurrence of events $X$ and $Y$ in set ${\rm T}$, $n$ represents the total number of transactions in set ${\rm T}$, and $Support\left(X\to Y\right)$ represents the support of the event $X\to Y$.

2.

Spatio-temporal relationship strategies.

The data are spatial and temporal. Spatiality refers to the absolute or relative position of data. Temporality refers to the presentation of data in a time series. The integration of space and time considers the changes in transactions described by the data with time to realize spatio-temporal relationship mining, spatio-temporal data analysis, spatio-temporal evolution analysis, and so on.

$$Space,Time=ST\left(self\right)$$

(13)

$ST\left(self\right)$ represents the spatio-temporal relationship analysis of data and returns the spatial and temporal attributes of the transaction. $Space,Time$, respectively, represent the spatio-temporal properties of the data.

3.

Empirical model strategies.

Empirical model strategies are also known as “black box models” [53] and refer to some methods that do not analyze the actual process mechanism, but carry on the mathematical statistical analysis from the actual data related to the process and predict the relationship between the parameters and variables according to the principle of minimum error.

Combination strategies are nested calls to strategies introduced in this section to achieve complex calculations and analyses:

$$F_{combination}(\ast )=F_1\left(F_2\left(\ast \right)\right)$$

(14)

where $F_1$ and $F_2$ represent any strategies described in this subsection.

3.2.3. Data Pipe Construction

Data pipes are constructed in one-to-one correspondence with strategies. Data pipes are meaningless on their own and generate new information by associating with each other. The details of the association of data pipes are shown later.

3.3. Autonomous Association of Multi-Source Heterogeneous Data Based on the Organizational Structure of Data Cells

This section introduces the autonomous association of multi-source heterogeneous data based on the organizational structure of data cells. Data autonomous association is the process of multilevel random association of data pipes and continually generating high-level combined information nucleuses of data cells containing new information to construct high-level combined data cells. This section introduces the mode of the random association of data pipes of initial data cells and elicits the general rules of random association of data pipes and the construction of combined data cells.

Using the construction method of initial data cells introduced in Section 3.2, field names, key-value names, or technical names of multimodal databases are extracted to realize the construction of information nucleuses of initial data cells, followed by the construction of strategies and data pipes of initial data cells to realize the construction of initial data cells. All data pipes of initial data cells form the set of data pipes of initial data cells:

$$OC=\left\{{oc}_{\left(1\right)},{oc}_{\left(2\right)},\cdots {,oc}_{\left(a\right)}{,\cdots ,oc}_{\left(l\right)}\right\}$$

(15)

where ${oc}_{\left(a\right)}$ represents a data pipe of initial data cells, and $OC$ represents the set of data pipes of initial data cells. Suppose ${oc}_{\left(a\right)}$ corresponds to the strategy $s_{\left(a\right)}$ and $s_{\left(a\right)}$ corresponds to the information nucleus $c_{\left(a\right)}$. The random association of data pipes means each data pipe ${oc}_{\left(p\right)}$ in $OC$ acts as the primary data pipe, and data pipe ${oc}_{\left(q\right)}$ in $OC$(${oc}_{\left(p\right)}$ can be the same as ${oc}_{\left(q\right)}$) is associated as the secondary data pipe:

$$\left\{r\left({oc}_{\left(p\right)},{oc}_{\left(q\right)}\right)|\forall {oc}_{\left(p\right)}\in OCand{\forall oc}_{\left(q\right)}\in OC\right\}$$

(16)

where ${oc}_{\left(p\right)}$ and ${oc}_{\left(q\right)},$ respectively, represent any data pipe in $OC$; $r\left({oc}_{\left(p\right)},{oc}_{\left(q\right)}\right)$ indicates the association result of ${oc}_{\left(p\right)}$ and ${oc}_{\left(q\right)}$, which is a unified expression formed by the sequential combination or connection of $c_{\left(p\right)}$, $s_{\left(p\right)}$, $c_{\left(q\right)},$ and $s_{\left(q\right)}$. Figure 5 is an example showing the association of the data pipe of “add” from the initial data cell with the information nucleus “weight” and the data pipe of “output” from the initial data cell with the information nucleus “height”, and the result of the association is a unified expression of “weight + height”.

After the random association of data pipes, three types of unified expressions are produced: Expressions conform to arithmetic rules and have practical significance; expressions fail to conform to arithmetic rules; expressions conform to arithmetic rules but have no practical significance. The following example explains the three types of unified expressions:

Suppose the data pipe ${oc}_{\left(p\right)}\in c_u$ and the information nucleus $c_u$ are “the score of subject A (numerical type)” and ${oc}_{\left(q\right)}\in c_v$ and the information nucleus $c_v$ are “the score of subject B (numerical type)”:

If the data pipe ${oc}_{\left(p\right)}\in c_u$ is the logical addition operation strategy: $F_{Plus}\left(obj\right)=self+obj$ and the data pipe ${oc}_{\left(q\right)}\in c_v$ is the output strategy: $F_O=Output\left(self\right)$, the association of ${oc}_{\left(p\right)}$ with ${oc}_{\left(q\right)}$ produces the uniform expression $r\left({oc}_{\left(p\right)},{oc}_{\left(q\right)}\right)$: “subject A score + subject B score”, which is a “numeric A + numeric B” type operation, which means the sum of subject A score and subject B score. The unified expression conforms to arithmetic rules and has practical significance.

If the data pipe ${oc}_{\left(p\right)}\in c_u$ is the output strategy: $F_O=Output\left(self\right)$ and the data pipe ${oc}_{\left(q\right)}\in c_v$ is the logical addition operation strategy: $F_{Plus}\left(obj\right)=self+obj$, the association of ${oc}_{\left(p\right)}$ with ${oc}_{\left(q\right)}$ produces the uniform expression $r\left({oc}_{\left(p\right)},{oc}_{\left(q\right)}\right)$: “subject A score subject B score+”. The unified expression does not conform to arithmetic rules.

If the data pipe ${oc}_{\left(p\right)}\in c_u$ is the multiplication strategy: $F_{Multiply}\left(obj\right)=self\times obj$ and the data pipe ${oc}_{\left(q\right)}\in c_v$ is the output strategy: $F_O=Output\left(self\right)$, the association of ${oc}_{\left(p\right)}$ with ${oc}_{\left(q\right)}$ produces the uniform expression $r\left({oc}_{\left(p\right)},{oc}_{\left(q\right)}\right)$: “Subject A score × Subject B score” is a “Numeric A × Numeric B” type operation, which means the product of subject A score and subject B score. The unified expression conforms to arithmetic rules but has no practical significance.

The generation of binary combined data cells is based on the random association of data pipes of initial data cells. Among these random associations of data pipes, some are symmetric while others are asymmetric. Symmetric associations refer to when the primary and secondary pipes are exchanged and the new association is the same as the original association or has the same meaning as the original association. Some examples are shown in Figure 6, where Figure 6a,b are symmetric associations. The association of $A+A$ shown in Figure 6a is unchanged when the primary and secondary data pipes are exchanged. For Figure 6b, because $A\&B=B\&A$, the association $A\&B$ is also symmetric. On the contrary, Figure 6c,d are asymmetric associations because the meanings of $A-B$ and $B-A$ are different, and if the variables $A$ and $B$ are exchanged in the confidence calculation, the meaning will be changed.

Nucleuses of binary combined data cells are the unified expression produced by the random association of data pipes of initial data cells that conforms to arithmetic rules; in order to prevent redundancy, when the association of data pipes is symmetrical, either of two symmetric expressions is selected to construct the nucleuses of binary combined data cells, and when the expression is asymmetric, both the two asymmetric expressions are selected. The construction of external strategies and pipes of binary combined data cells is the same as that of initial data cells. The information nucleus of N-tuple combined data cells (N > 2) is formed by the random association of the data pipes of (N−1)-tuple data cells with the data pipes of initial data cells using the same construction principle of external strategies and data pipes of initial data cells.

3.4. Information-Driven Intelligent Information Discovery

The random association of data pipes of data cells generates a variety of information nucleuses of combined data cells that conform to arithmetic rules and contain new information. However, the determination of whether the information nucleus of combined data cells has practical significance, so as to screen meaningful data associations, is difficult because the evaluation of the practical significance of the information nucleus of combined data cells is subjective and needs to be scored manually by relevant experts. However, the scoring of mass information nucleuses of combined data cells is time-consuming and labor-consuming, so it is necessary to construct an automatic discovery method to realize the intelligent screening of mass information nucleuses. This section describes an information-driven intelligent information discovery method (see Figure 7). Via the introduction of the reward and punishment model and training of the model by simulating the manual scoring mode of information nucleuses, the intelligent scoring and storage of information nucleuses can be realized.

The reward and punishment model guides the behavior of the intelligent system by interacting with artificial prior data. According to these interactions, the intelligent system can gradually simulate artificial intelligent selection and finally realize independent decision-making. The key to the reward and punishment model is to construct the loss function to describe the difference between the model-estimated values and the manual scores. The reward and punishment model is trained using many manual scoring samples to realize the intelligence scoring of information nucleuses of combined data cells. According to the actual needs, the scoring threshold can be set to realize the automatic screening of meaningful information nucleuses. {The information nucleus, Score of the information nucleus} is used as the training data format. The scoring data of information nucleuses is generated by subjective scoring standard construction and manual scoring (see

Table 1. Subjective scoring standard of combined data cells.

Score	Practical Significance Description of Information Nucleuses
1	nucleuses with no practical significance.
2	nucleuses with a little practical significance.
3	nucleuses with general practical significance.
4	nucleuses with great practical significance.
5	nucleuses with common practical significance.

).

The subjective scoring standard shown in Table 1 divides the practical significance of information nucleuses into five grades, with a corresponding score of 1–5. Assuming that the initial score of any nucleus is 3, in order to maximize model scoring close to human scoring, the reward and punishment model is fitted by maximizing the likelihood function. This means that the scoring ability can be improved by minimizing the loss function. The loss function is expressed as:

$$Loss\left(\theta \right)=-ϵ\mathit{log}\left(\sigma \left(\left|r_{\theta }\left(x\right)-r_{\theta }^{*}\left(x\right)\right|\right)\right)$$

(17)

where $Loss\left(\theta \right)$ is the loss function of the reward and punishment model; $x$ represents information nucleuses of combined data cells; $r_{\theta }\left(x\right)$ represents the manual score of $x$; $r_{\theta }^{*}\left(x\right)$ represents the model estimated score of $x$; $\sigma (*)$ represents the sigmoid function. $log(*)$ represents the logarithmic operation and $-ϵ$ represents the negative constant. The maximization of the difference is achieved by performing a negative logarithmic operation on the result of the sigmoid function.

The cloud brain is used to store meaningful information nucleuses of combined data cells to facilitate rapid external calls, including meaningful information nucleuses of combined data cells (unified expression body), a database list of field names, key-value names, or technical names in the expression body, and scores of meaningful nucleuses. The above storage method realizes the fast call of external requests according to the database list and calling logic stored in the information nucleus. In addition, the cloud brain also stores the call records of meaningful information nucleuses, including the time of calls and the results of return. According to the comparison of the returned results of the same calls at different times, historical comparative analysis of data can be realized.

3.5. Task-Driven Intelligent Information Discovery

This section introduces the task-driven intelligent information discovery method. This method takes realistic task requests as input. Using the NLU topic model [54], the task requests are decomposed into the collection of subject headings for retrieving information nucleuses of initial data cells [55] to obtain the set of matched initial data cells; the set of information nucleuses of combined data cells are generated by random association of data pipes of matched initial data cells. Via the introduction of the reward and punishment model, combined with the task request, and training of the model by simulating the manual sorting mode of information nucleuses, the realization of intelligent sorting and the storage of information nucleuses of combined data cells is achieved (see Figure 8).

The realistic task request is decomposed by the NLU topic model into the set of subject headings, following the principles of non-repetition, non-omission, and ease of recognition:

$$Request\to SH=\left\{{sh}_1,{sh}_2,\dots {sh}_i,\dots ,{sh}_k\right\}$$

(18)

where $Request$ represents the task request; ${sh}_i$ represents the subject heading; $SH$ represents the set of subject headings.

By imitating the principle of search engines [56], subject headings are retrieved from the set of information nucleuses of initial data cells, including synonym substitution, subject heading conversion, and so on, and return to the set of matched initial data cells. Data pipes of initial data cells are randomly associated (same as Section 3.2.2) to obtain the set of information nucleuses of combined data cells related to the task request. After removing information nucleuses which do not conform to the operation rules, the reward and punishment model is trained to realize the intelligent sorting of information nucleuses of combined data cells in the set and thus obtains the optimal information nucleus. Different from the information-driven intelligent information discovery method, the task-driven intelligent information discovery method does not use direct scoring but sorts the set of nucleuses of combined data cells according to the task request and takes {task request, the list of information nucleus of combined data cells} as the training data format for model training. In order to maximize the model sorting result close to that of human sorting for task requests, the reward and punishment model is fitted by maximizing the likelihood. This means that the sorting ability can be improved by minimizing the loss function. The loss function is expressed as:

$$Loss\left(\theta \right)=-ϵE_{\left(Request,c_{\beta },c_{\phi }\right)\sim D}\left[\mathit{log}\left(\sigma \left({Rank}_{\theta }\left(Request,c_{\beta }\right)-{Rank}_{\theta }\left(Request,c_{\phi }\right)\right)\right)\right]$$

(19)

where $Loss\left(\theta \right)$ is the loss function of the reward and punishment model; $Request$ represents the task request; $D$ represents the sorting list of information nucleuses for the task request. $c_{\beta }$ represents the information nucleuses of combined data cells sorted before $c_{\phi }$ in the sorting list; ${Rank}_{\theta }\left(Request,c_{\beta }\right)$ and ${Rank}_{\theta }\left(Request,c_{\phi }\right),$ respectively, represent the ranks of $c_{\beta }$ and $c_{\phi }$ in the sorting list; $\sigma (\ast )$ represents the sigmoid function. $log(*)$ represents the logarithmic operation and $-ϵ$ represents the negative constant. The maximization of the difference is achieved by performing a negative logarithmic operation on the result of the sigmoid function.

The trained model can autonomously sort the information nucleuses of combined data cells and realizes intelligent information discovery based on task requests.

4. Examples of Applications

This section introduces some examples of intelligent information discovery methods. The information-driven method takes the example of the Cultural Income Table of the Department of Culture and the Tourism Income Table of the Department of Tourism to demonstrate the construction of initial data cells by using table fields and meaningful combined data cells generated by autonomous association of data pipes. The task-driven method takes the task request “What’s the weather today?” as an example and demonstrates the process of subject heading acquisition, subject headings–nucleuses retrieval, the autonomous association of data pipes, and the discovery of the optimal combined data cell for the task request.

4.1. The Example of the Information-Driven Method

The Cultural Income Table includes the fields: Time (IT), Place (IP), and Cultural Income Table (II); the Tourism Income Table includes the fields: Time (TT), Place (TP), and Tourism Income (TI).

For ease of representation, the different strategies are abbreviated in this section, as shown in

Table 2. Abbreviations of strategies.

Strategies	Abbreviation
Input	$\leftarrow A$
Output	$\to A$
Data normalization	$d(A,B)$
Addition	$A+B$
Subtraction	$A-B$
Multiplication	$\mathrm{A}\times \mathrm{B}$
Division	$\mathrm{A}÷\mathrm{B}$
With	$\mathrm{A}\&\mathrm{B}$
Or	$\mathrm{A}\Vert \mathrm{B}$
Non	$!\mathrm{A}$
Supportability	$S(A,B)$
Confidence	$C(A,B)$
Spatio-temporal relationship	$TimeSpace\left(A\right)$
Empirical modeling	$Model\left(A\right);Model\left(A,B\right);Model(A,B,C)\dots$

:

Initial data cells are constructed using the fields of the Cultural Income Table and the Tourism Income Table. Meaningful nucleuses of combined data cells acquired by multi-level data pipes association are shown in

Table 3. Meaningful nucleuses of combined data cells.

No.	Nucleuses	Explanation
1	←IT ←IP→II	Input a year and a city, output the city’s cultural income for the year.
2	(←IT←IP→II) − (←TT←TP→TI)	When the input satisfies “←IT = ←TT, ←IP = ←TP”, it indicates the difference between tourism income and cultural income in the given city in the given year. (When “−” is replaced by “/”, it indicates the income ratio; when “−” is replaced by “+”, it indicates the sum of income).
3	(←IT1←IP1→II1)–(←IT2←IP2→II2)	When the input satisfies “←IT1 = ←IT2, ←IP1≠←IP2”, it indicates the cultural income difference for the same year of different cities. When “←IT1 ≠ ←IT2, ←IP1 = ←IP2”, it indicates the cultural income difference in the given city in different years. (“−” replaced by “+” for sum of income and “−” replaced by “/” for income ratio)
4	((←IT1←IP1→II1)–(←IT2←IP2→II2))/(←IT3←IP3→II3))	When the input satisfies “←IP1 = ←IP2 = ←IP3, ←IT2 = ←IT3”, and IT1 and IT2 are adjacent years, it indicates the annual growth rate of cultural income of a city.
5	(((←IT1←IP1→II1)–(←IT2←IP2→II2))/(←IT3←IP3→II3)) && (((←TT1←TP1→TI1)–(←TT2←TP2→TI2))/(←TT3←TP3→TI3))	When the inputs satisfy “←IP1 = ←IP2 = ←IP3 = ←IP4 = ←IP5 = ←IP6, ←IT2 = ←IT3”, IT1 and IT2 are adjacent years, “←IT1 = ←TT1, ←IT2 = ←TT2, ←IT3 = ←TT3 “, it indicates the positive and negative correlation analysis of cultural income and tourism income of a city.
6	Model((←IT1←IP1→II1), (←IT2←IP2→II2) …(←ITn←IPn→IIn))	Empirical modeling analysis of changes in cultural income over time, or empirical modeling analysis of future cultural income, etc.
7	Model (((←IT1←IP1→II1), (←IT2←IP2→II2), …(←ITn←IPn→IIn)), ((←TT1←TP1→TI1), (←TT2←TP2→TI2), …, (←TTn←TPn→TIn)))	Mining the relationship between cultural income and tourism income, etc.

The numbers after the fields in the table are to distinguish multiple occurrences of the same field in the same information nucleus, such as “IT1” and “IT2” in Example 3.

.

4.2. The Example of the Task-Driven Method

Taking the task request “What’s the weather today?” as an example, the task request is analyzed using the NLU topic model to obtain the set of subject headings “$SH=\left\{city,date,airtemperature,temperature,airquality\right\}$”. Subject headings are used to retrieve initial data cells to obtain the set of information nucleuses of initial data cells:

$$\left\{city,region,date,airtemperature,temperature,humidity,airquality,airpollutionindex\dots \right\}$$

(20)

Information nucleuses of combined data cells are generated by the autonomous association of data pipes of data cells.

Information nucleuses are manually sorted and used to train the reward and punishment model. The trained model realizes automatic ranking of information nucleuses of combined data cells and the best matching nucleus is stored (see

Table 4. Sorting list of information nucleuses of combined data cells for the request.

Rank	Nucleuses of Combined Data Cells
1	←city*←date*←air temperature*←temperature*←air pollution index*……
2	←city*←date*←air temperature*←temperature*←air quality*←air pollution index*……
3	←city*←date*←air temperature*←temperature*
…	…
n−1	←air quality*
n	←air quality

Fields marked “*” in the upper right-hand corner are from the meteorological database.

).

5. Discussion

The method proposed in this paper can be widely applied to multi-source heterogeneous databases of different fields, types, and structures in a variety of government departments, companies, and social organizations. The autonomous data association method based on the organizational structure of data cells and two different data intelligent information discovery methods provide a new theory for the autonomous mining, management, and application of multi-source heterogeneous data, and a new idea for cross-industry information sharing that provides high-quality data source for AI. The methodology of this paper is feasible in the following three aspects:

• Theoretical feasibility: Field names, key-value names, or technical names in the database are data items of the same type divided according to some boundaries, which represent specific transactions. The data association is the analysis of the relationship between transactions. This paper takes field names, key-value names, or technical names as information nucleuses of initial data cells, and the nucleuses of combined data cells are generated by random association of data pipes of initial data cells, so the construction of information nucleuses are in accordance with the essence of data association. The construction of strategies and data pipes and the association of data pipes realize the data autonomous association. Strategies are all kinds of common algorithms and scripts, and unified expressions are generated by the association of data pipes, which realizes the association of multiple data sources with different types and formats. The process of association of data pipes is random, thus maximizing the acquisition of associated new information. According to Section 3.2.2, the unified expressions formed by the autonomous association of data pipes need to be judged using arithmetic rules and practical significance. Therefore, we select expressions that meet arithmetic rules and classify associations into symmetric and asymmetric categories to construct information nucleuses of combined data cells and the reward and punishment models from two perspectives are trained to simulate the manual screening process. The information-driven type directly filters nucleuses, and the task-driven type is task-oriented to the requirements of real applications. In conclusion, the method in this paper extracts meaningful associated information from subjective and objective perspectives and from overall and local perspectives.
• Technical feasibility: The construction of information nucleuses of initial data cells is the process of extracting field names, key-value names, or technical names, which can be obtained directly using related database statements and other critical information can be obtained using various existing algorithms, such as image-recognition algorithms. Strategies are divided into underlying strategies and advanced strategies, where underlying strategies are common basic mathematical operations and advanced strategies are some existing complex algorithms or combinations of different strategies, which can be realized via algorithm development. The association of data pipes is a sequential combination or connection of information nucleuses and strategies. Expressions that fail to conform to arithmetic rules can imitate “calculator” applications, where logical judgments are implemented using manually predefined rules. The symmetry of associations of data pipes can also be judged using manual predefinition. For the two intelligent information discovery methods, we have given loss functions and training data formats for the reward and punishment model, so the model can be implemented via more detailed designing and programming. The training data can be obtained using manual scoring or ranking of samples. When the empirical knowledge reaches a certain level, the model can spontaneously screen meaningful information nucleuses to realize autonomous information discovery. The meaningful nucleuses are stored in the cloud brain, which is essentially a relational database, and can be designed with reference to relational databases.
• Feasibility of data security: The data invoked in the actual application is according to the unified expressions of meaningful information nucleuses stored in the cloud brain. Although data cells are directly connected with the database, and the data are called and analyzed without changing the source data and source environment, the data invoked according to the unified expression have been calculated using various strategies, which are far different from the original data, and the safety of the original data is initially guaranteed. In addition, for further guarantees of data security, the managers of data can restrict the permission of information nucleuses in the cloud brain to different personnel or departments.

The data autonomous association method and associated information intelligent discovery method proposed in this paper have multiple advantages over existing multi-source heterogeneous data processing methods, as summarized in

Table 5. Comparison of our method with other multi-source heterogeneous data processing methods.

	Type and Range of Data Processed	Manual Dependency	Association Pattern between Data	Deep Mining of Data Association	Robustness
Traditional data association [18,19,20]	Several specific types	High. Requires data association definition	Relies on advance manual definition	No	Weak
Data warehouse [5,6,7]	Multimodal data around a topic	High. Requires storage layer design	Relies on advance manual definition	No	General
Data virtualization [8,9,10,11]	Multimodal data around a topic	High. Requires virtual layer design	Relies on advance manual definition	No	General
Our method	Wide range of multimodal data	Low. Only manual annotation of samples	Autonomous association	Yes	Strong

:

Our method can deal with a wide range of multi-source and multimodal data, while other data processing methods only deal with certain types of data or are limited to a certain topic. The manual dependence of our method is smaller because the main workload is the manual labelling of data in the early stage, and the manual dependence will be reduced significantly after the model has matured. The data-autonomous association method based on data cells is autonomous while other methods rely on manual definition. The intelligent discovery methods can better cope with data updates because only the association rules are stored, such as changes in data types and the generation of new associations. Therefore, our method has greater robustness than traditional methods.

6. Conclusions

The data autonomous association method based on data cells and two intelligent information discovery methods effectively solves the problem of autonomous association of massive multimodal data and the problem of intelligent acquisition of potential information generated by autonomous association. According to the meaningful association rules, a large number of complex and effective data sources can be effectively provided for AI. Via the examples of applications and feasibility analysis, we demonstrate the effectiveness and the feasibility of methods. In our future work, we intend to propose more optimized data autonomous association methods and further optimized models to discover deeper information from massive multimodal data. Furthermore, we will conduct a series of evaluation indexes to evaluate the methods.