*3.2. Quantification Analysis of National-Level Policy Documents on AI and Comparative Analysis between International Research Hotspots and Policy Keywords*

Word co-occurrence is one of the most commonly used analytical methods for bibliometrics. Two keywords appear in one document and are recorded as one co-occurrence of the topic. The more co-occurrences, the closer the relationships between the two words and the stronger the correlations; analyzing the co-occurrence relationships of keywords in existing policy documents can cluster the subject words and identify the core theme of the field; thus, the co-occurrence network of the keywords in each period and stage can be constructed, and the stages can be identified [19,51]. After integrating and modifying the co-occurrence map of AI reports, a three-stage theme co-occurrence map of AI development was obtained, as shown in Figures 5–7.

**Figure 5.** The initial exploration stage keywords co-occurrence map of China's national AI policy (2009–2012). Source: According to References [19,51].

**Figure 6.** The steady rising keywords co-occurrence map of China's national AI policy (2013–2015). (**a**) (2013–2014); (**b**) (2014–2015). Source: According to References [19,51].

**Figure 7.** The rapid development keywords co-occurrence map of China's national AI policy (2016–2018). (**a**) (2016–2017); (**b**) (2017–2018). Source: According to References [19,51].

In the initial exploration stage of AI policy (2009–2012), as shown in the co-occurrence map of policy keywords in Figure 5, "AI", "infrastructure", "information security", "technical standards", and "Internet of Things" were the core themes of relevant policies. These were closely related to the social development at that time. During this period, the Internet of Things was just beginning to emerge, and infrastructure construction about AI was ineffective, especially in some aspects such as data collection, data mining, and database construction. However, at this stage, national-level and central-level policies directly related to AI and the Internet of Things had not yet been introduced, resulting in drawbacks in intellectual property rights, property rights protection, technical standards, information security, and public safety.

At this stage, the hotspots of international research on AI in China were "artificial intelligence", "artificial neural network", "genetic algorithm", "group intelligence", "return", "model", "system", "standard", "recognition", "predict", "data mining", and others. It can be seen that there are differences between the two. For example, many hotspots of international research focused on specific technique and frontier exploration of AI (such as "artificial neural network", "genetic algorithms", "recognition", etc.), while policy keywords and hotspots were "Internet of Things", "Infrastructure", "Information Security", and so on. However, there were subtle similarities and connections, such as "data mining", "identification", "prediction", and so on, which were the foundation and prototype of policy keywords in terms of "infrastructure", "Internet of Things", "database construction" and "intelligent industry chain". The exploration of "model" and "system" in research hotspots is the exploration and foundation of future policies on "technical standards". It can be seen that in the initial exploration stage of 2009–2012, the hotspots of international research on AI in China had a certain alienation from the AI policy at the national level, because at this time, there were not many national-level policies on AI. At the same time, international research hotspots and policy keywords had a certain degree of connection. The research hotspots contained the technical basis and prototypical exploration of the current and future policies.

In the steadily rising stage of AI policy (2013–2015), as shown in the co-occurrence maps of policy keywords in Figure 6a (2013–2014) and Figure 6b (2014–2015), "AI", "Internet of Things", "big data", "infrastructure", and "information security" were the core keywords of relevant policies. In this period, China had a certain foundation in the application of big data technology and the development of the internet of things industry, and the knowledge of AI had become more and more mature, and then turned to the basic technology of AI and related practical applications. In 2015, "Made in China 2025" was released, deploying "intelligent manufacturing" in a holistic way and implementing the strategy of manufacturing power; in the same year, the guidelines for the "Internet +" Action were released, clarifying that AI was one of the 11 key development areas for the formation of the new industrial model, promoting the development of AI to the national strategic level. During this period, the hotspots of international research on AI in China were "deep learning", "feature selection", "face recognition", "artificial bee colony algorithm", "time series", "regression", "China", "strategy", and so on. From a macro perspective, the international research on AI in China gradually increased the research and analysis of "China" and "strategy" level; at the micro level, the discussions on the technology and application of AI were more specific and cutting-edge, being inseparable from the policy promotion and keywords of this period, and many specific technologies were built on the basis of AI infrastructure, big data, and databases deployed in China. Compared with the alienation relationship in the initial exploration stage of the previous period, the specific technology and application of AI in this period began to be reflected in policy formulation and social life, that is to say, the hotspots of AI international research began to be closely integrated with the keywords of national policy, but research hotspots focused more on the exploration and application of technology, and national policy was more macroscopic, representing the overall embodiment of hotspots and technologies.

In the rapid development stage of AI policy (2016–2018), as shown in the co-occurrence maps of policy keywords in Figure 7a (2016–2017) and Figure 7b (2017–2018), "AI" became the largest keyword in this stage, while "big data", "infrastructure", "Internet of Things", and "technical standards" were still relevant policy keywords. It should be noted that the keywords such as "robot", "intelligent manufacturing", and "deep learning" also appeared and increased, and the related words of "intellectual property" and "property rights protection" also began to appear. In 2016, the National 13th Five-Year

Plan of China proposed to break through AI technology. During this period, the development of virtual technology, intelligent commerce, industrial robots, and other fields marked the gradual establishment and improvement of the AI industry system, and the state began to attach importance to the intellectual property rights and property rights protection of AI. The release of the New Generation Artificial Intelligence Development Plan in 2017 marked the beginning of the new era of AI in China. At present, China has systematically arranged AI from the national level, and the Report of 19th National Congress of China put forward "promoting the deep integration of the Internet, big data, AI and the real economy" and strengthened the comprehensive support of AI for science and technology, economy, social development, and national security [4,36]. During this period, the hotspots of international research on AI in China focused on specific frontier technologies and practical commercial application technologies such as "big data", "convolution neural network", "electronic skin", "image classification", "sensors", "movies", "electronic games", and "applications", which is in line with the current policy advocating integration of artificial intelligence and the real economy at the national level. At the same time under the promotion and accumulation of previous research hotspots, current research hotspots began to closely integrate with current policy topics. There are two reasons for this: on the one hand, the planning and top-down design of policies are constantly strengthened, and the contents and keywords of policy documents often cover most of the hotspots and keywords of AI research; on the other hand, the hotspots of AI international research in this period also indicate that more scholars began to pay attention to expanding their research topics and directions to cover policy strategies and commercial applications. Therefore, in this period, the international research hotspots of AI in China were closely integrated with the characteristics of national policies, and they were deeply integrated with and covered by with each other.

In this part, some experts suggested that the research team could continue to use new algorithms and systems to further enhance performance values, e.g., HMM, modified HMM, embedded HMM, GMM, etc. [96–101], and these need to be further expanded upon by the research team in follow-up studies.

#### **4. Discussion and Conclusions**

First, this paper adopted the methods of mathematical statistics to carry out simple historical statistics and stage division of relevant Chinese scholars' papers on China's international AI research downloaded and screened from Web of Science database. According to the number and growth trend of articles published, the international research on AI in China was roughly divided into three stages, namely the initial exploration stage (2009–2012), the steady rising stage (2013–2015), and the rapid development stage (2016–2018). Subsequently, the high-yielding countries/regions and institutions of AI international research were analyzed, and the CiteSpace software was used to process related papers. Hotspot knowledge maps of international AI research were drawn, the relevant research topics, keywords, and hotspots are found, and the network results and distribution of research hotspots were obtained. Hotspots and keywords included "artificial intelligence", "neural network", "model", "system", "optimization", "genetic algorithm", "cloud computing", and so on according to the frequency. In the past three years, new hot topic words have included "China", "extreme learning machine", "face recognition", and "electronic skin", etc.

Based on the theory and method of quantitative analysis of policy documents, 262 central-level AI policy documents were collected and screened from the Government Document Information System (GDIS); the documents were also sorted by quantity and distributed into stages. Based on the documents themselves and the previous division of research hotspot stages, the documents were also divided into three stages according to the time of publication: the initial exploration stage (2009–2012), the steady rising stage (2013–2015), and the rapid development stage (2016–2018), which were the same as the research hotspot stage for comparative analysis. On the three-stage co-occurrence maps of China's national-level policy keywords, this study made a preliminary quantitative analysis of policy documents and carried out a comparative analysis of the evolution of AI international research hotspots.

In terms of stages, in the initial exploration stage (2009–2012), China's international AI research hotspots and national-level policy keywords had a somewhat alienated relationship; research hotspots were more frontier, while national level policies were few, but research hotspots formed the technical basis and embryonic exploration of the current and future policy keywords. In the steady rising stage (2013–2015), the AI research hotspots were more closely related to the national level policy keywords; the research hotspots focused more on the exploration and application of technology, and the national level policy was the overall collection of hotspots and technologies. In the rapid development stage (2016–2018), the characteristics of China's international AI research hotspots and national level policies were closely integrated, and they were deeply integrated and covered.

In general, research hotspots often focus on exploring the frontiers and technologies of AI. With the development of the times and the progress of policies, research hotspots gradually began to integrate with policies: research hotspots expanded from frontier exploration to the applications of technology and attention was paid to strategies and policies, while policies at the national level began to pay more attention to top-down design and bottom-level promotion, and the overall strategy and planning layout of research hotspots and technological frontiers were also relatively in place, being more targeted strategically and overall.

Although this research had a certain novelty in comparing and analyzing the hot spots and keywords of policy and academic circles, and involved scientometrics and knowledge map visualization analysis of relevant academic literature and policy documents, there are still many limitations and weaknesses related to the opinions of some experts and reviewers. First, the selection of academic literature and policy documents had certain limitations and could be misleading. This study only shows the general situation of China's AI research and policy-making from one side or one general trend. It is possible that different sample selections would reflect different situations and outcomes, clarification of which will require greater sample sizes in the future or more sample selections from other perspectives. Secondly, experts also suggested some novel algorithms and systems in AI, which the research team need to further study, reference, and apply to enrich the research on AI development. Future studies need to select samples scientifically, and use more new algorithms, new systems, and a variety of research methods to comb and study the development and research of AI in China and the world.

**Author Contributions:** Conceptualization, J.G. and X.H.; Methodology, J.G. and X.H.; Software, J.G. and L.Z.; Validation, J.G., X.H. and L.Z.; Formal analysis, J.G. and X.H.; Investigation, J.G. and L.Z.; Data Curation, J.G. and X.H.; Visualization, J.G., X.H. and L.Z.; Writing—Original Draft Preparation, J.G., X.H. and L.Z.; Writing—Review & Editing, J.G., X.H. and L.Z.; Funding Acquisition, X.H. and J.G.

**Funding:** This research was funded by [The National Natural Science Foundation of China] grant number [71904101, 71801169], [China Postdoctoral Science Foundation] grant number [2018M640150, 2019M650754], and [The National Social Science Foundation of China] grant number [18ZDA075, 18CTQ040].

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
