Bayesian Structural Time Series and Regression Modeling for Sustainable Technology Management

Abstract

Many companies take the sustainability of their technologies very seriously, because companies with sustainable technologies are better able to survive in the market. Thus, sustainable technology analysis is important issue in management of technology (MOT). In this paper, we study the management of sustainable technology (MOST). This focuses on the sustainable technology in various MOT fields. In the MOST, sustainable technology analysis is dependent on time periods. We propose a method of sustainable technology analysis using a Bayesian structural time series (BSTS) model based on time series data. In addition, we use the Bayesian regression to find the relational structure between technologies. To show the performance of our method and how the method can be applied to practical works, we carry out a case study using the patent data related to artificial intelligence technologies.

1. Introduction

Technology is the most important factor for changing the world [1]. Many technologies have changed the world to date, with artificial intelligence (AI) technology now dominating. Most companies have tried to increase their technological competitiveness and sustainability in the marketplace. Finding and developing sustainable technologies is a good approach for increasing a company’s technological competitiveness [2]. Many studies on sustainable technology analysis have been performed in diverse fields [3,4,5,6,7,8]. Kim et al. (2015) proposed a method of sustainable technology forecasting in the defense technology field. They applied the technology diffusion model to their proposed method. This paper showed how the sustainable technology analysis can be applied to defense technology of Korea. Park et al. (2015) studied a network model to select sustainable technology in the Ford Motor Company. In this paper, the authors explained the sustainable technology structure of the technologies included in Ford using the international patent classification (IPC) codes of Ford’s patents. Kim et al. (2018) carried out a sustainable technology analysis of AI technology. They used the visualization using social network analysis (SNA). Lee et al. (2018) conducted a study on sustainable technology transfer using ensemble modeling. They combined a topic model with an adaptive boosting algorithm for the ensemble modeling. Kim et al. (2019) proposed a study on data envelopment analysis and state space model for sustainable technology analysis. This paper used the technological keywords extracted from patent documents for the proposed model. In the most previous studies, the technology keywords or IPC codes extracted from patent documents were analyzed by statistics and machine learning algorithms without time considerations. However, technology is heavily influenced by time factors because technology has been developed over time from the past to the present. Sustainable technology likewise has been changed over time. Thus, for finding the sustainable technology in a given technological field, we have to understand the technological relationships between all sub technologies via sustainable technology analysis. However, it has been difficult to find the methods to solve this problem in previous studies. Therefore, we propose a model that considers time and identifies technological relationships. In this paper, we use the Bayesian structural time series (BSTS) to perform time-related sustainable technology analysis and the Bayesian regression to find the technological association [9,10,11]. The Bayesian statistical approach can be used effectively in sustainable technical analysis. This is because the Bayesian approach builds analytical models based on prior experience (prior distribution) and given data (likelihood function). In sustainable technology analysis, the prior distribution deals with the expert’s knowledge of a given technology domain, and the likelihood function covers the patent data related to a target technological field. The prior distribution and likelihood function are combined (multiplied) to form the posterior distribution and this becomes the final Bayesian model. Therefore, we propose a technology analysis model for sustainable technology management by using these characteristics of Bayesian statistics. Finally, we combine the results of BSTS and Bayesian regression to draw conclusions for the management of sustainable technology (MOST). The remainder of this paper is organized as follows. In Section 2, we show the research backgrounds related to our study. We show the proposed method for MOST in Section 3. The next section provides the result of our case study using AI technology. In the conclusions section, we conclude our research and describe our future works related to MOST.

2. Research Background

2.1. Management of Sustainable Technology

Companies must develop sustainable technologies to improve their market competitiveness. Sustainable technology refers to a core technology that can secure the sustainability of the company’s technologies [2,3,5]. Thus, understanding a company’s sustainable technology is a very important task in management of technology (MOT). In addition, the MOST is focused on sustainable MOT. In this paper, we study on a method for MOST using the BSTS and Bayesian regression.

2.2. Structural Time Series Models

Time series models have been used in many applications such as business, science, and engineering. The first goal of time series models is forecasting, and the next is explaining model structure. There are a number of methods in time series analysis. The structural time series model is one of them. This constructs a forecasting model by structuring the elements of time series. In the structural time series model, the observation in time t $y_t$ is defined as per following observation equation [9,10].

$$y_t=Z_t^T{\alpha }_t+{\epsilon }_t$$

(1)

where ${\alpha }_t$ is a vector of latent variables and $Z_t$ is a vector of model parameters. The error term ${\epsilon }_t$ follows a Gaussian distribution with $\mu =0$ and ${\sigma }^2=H_t$. In addition, ${\alpha }_t$ is represented as the following transition equation [9,10].

$${\alpha }_{t+1}=T_t{\alpha }_t+R_t{\eta }_t$$

(2)

where ${\eta }_t$ has a Gaussian distribution with $\mu =0$ and ${\sigma }^2=Q_t$. This equation shows the update of unobserved latent variable ${\alpha }_t$ over time. $T_t$ and $R_t$ are the transition matrix and structural parameter, respectively. $Z_t$, $T_t$, and $R_t$ contain the values 0 and 1 representing the relevance for structural computation. The Equations (1) and (2) describe the state space of observed data. Using this model we can build time series models for short- and long-term forecasting.

2.3. Bayesian Techniques for Industrial Engineering

Various Bayesian techniques have been used in industrial engineering, including MOT. Among them, the Bayesian network is one of the popular models. Many studies on MOT have used this model for quantitative data analysis. A Bayesian network is a graph model to represent the causal structure between nodes (variables) under uncertainty [12]. This builds a directed graphical model using joint and conditional probability distributions and chain rule [13,14]. That is, the joint probability of m nodes is represented as per the following chain rule equation [12]:

$$p\left(y_1, y_2, \dots ,y_m\right)={\displaystyle \prod }_{j=1}^{m}p\left(y_j|y_1,\dots ,y_{j-1}\right)$$

(3)

where $y_1, y_2, \dots ,y_m$ are m nodes of Bayesian network model. Also, this is updated by learning from data as follows [13]:

$$p\left(\theta |y\right)\propto p\left(y|\theta \right)p\left(\theta \right)$$

(4)

where y is data associated with node and $\theta$ is the parameters of the model. For example, in the infrastructure systems such as water supply and supply chain management, the Bayesian network method was used for modeling the quantification of resilience [15,16]. In addition, the Bayesian network model was carried out to analyze the sustainable technology [6]. Bayesian networks are a good model for understanding the causal structure of uncertainties in the data, but it is difficult to illustrate the time factors in the model. However, the time factor is important in technology analysis, because the technology has been developed over time. Therefore, we need Bayesian technique that can consider time factor for technology analysis. To overcome this problem, we propose Bayesian modeling with a time factor for MOT.

3. Bayesian Structural Time Series and Regression Models for MOST

Technology changes and evolves over time. As time goes on, technology has been developed and improved the quality of human life. In this paper, we analyze the patent keyword data over time using Bayesian structural time series (BSTS) modeling. Bayesian statistics has been applied to many statistical fields such as regression, classification, clustering and time series analysis [17]. Bayesian statistics are based on the Bayes theorem as follows [11,17,18]:

$$P\left(\mathsf{\theta }|\mathrm{x}\right)=\frac{P\left(x|\theta \right)P\left(\theta \right)}{P\left(x\right)}$$

(5)

where x is observed data and $\mathsf{\theta }$ is model parameter. P(x) is computed as follows: $P\left(x\right)={\displaystyle \sum }_{\theta }P\left(x,\theta \right)={\displaystyle \sum }_{\theta }P\left(x|\theta \right)P\left(\theta \right)$. $P\left(\theta \right)$ and $P\left(x|\theta \right)$ represent the prior and likelihood functions, respectively. Also, $P\left(\theta |x\right)$ is the posterior function. This is updated by learning the observed data x given $\theta$ (likelihood). In Bayesian statistics, we can obtain various analysis results according to various selection of prior distribution. The BSTS is also based on general structural time series model in Equations (1) and (2). Also, we select Gaussian distribution as the prior of our BSTS model because we use the occurred frequency values from 0 to infinite [0,∞). The following represents the basic structure of BSTS [9]:

$$y_t={\mu }_t+{\tau }_t+{\beta }^T{x}_t+{\epsilon }_t$$

(6)

where ${\mu }_t={\mu }_{t-1}+{\delta }_{t-1}+{\mu }_t$, ${\delta }_t={\delta }_{t-1}+v_t$, and ${\tau }_t=-{\displaystyle \sum }_{s=s}^{S-1}{\tau }_{t-s}+w_t$. The ${\tau }_t$ is distributed to Gaussian. We also consider the Bayesian regression model as follows [11]:

$$\mathrm{y}={\beta }_0+{\displaystyle \sum }_{i=1}^p{\beta }_i{x}_i+e$$

(7)

where e is distributed to Gaussian with mean = 0 and variance = ${\sigma }^2$. In our model, the prior of ${\sigma }^2$ is followed to an inverse-chi square (Inv-${\chi }^2$) distribution as follows [11]:

$${\sigma }^2~Inv-{\chi }^2\left(n-p,s^2\right)$$

(8)

where n and p are data and parameter sizes, respectively. In addition, $s^2$ is calculated as follows:

$$s^2=\frac{1}{n-p}{\left(y-X\widehat{\beta }\right)}^T\left(y-X\widehat{\beta }\right)$$

(9)

To build the technological structure of sustainable technologies, we combine the results of BSTS and Bayesian regression. Figure 1 shows the process of our proposed method.

The first step in our proposed method was to select a target technology for sustainable technology analysis. We collected the patent documents related to target technology from the patent databases such as the WIPS Corporation (WIPSON) and the United States Patent and Trademark Office (USPTO) [19,20]. We also extracted technological keywords from the collected patent documents using text mining techniques. In this paper, we use the R data language and its text mining techniques [21,22]. Using the result of extracted keywords, we chose the representative technologies for the target technology. To understand the representative technologies from past to present and forecast the future state of the technologies, we searched time series data related to the representative technologies. We retrieved the time series data from the Google Trends [23]. Using the data of Google Trends, we performed the time series data analysis by the BSTS model and Bayesian regression. Lastly, we combined the analytical results of the BSTS model and Bayesian regression to find sustainable technologies for target technology field.

The scope of MOT is very wide. The MOT includes everything a company needs to manage technology, including technology forecasting, new product development, technology innovation, technology roadmaps, and R&D strategies [1]. Most MOT tasks are based on analysis and understanding of technology. The MOST is the same as general MOT. In addition, technology analysis can expect improved results when prior knowledge and time factors of target technology are added to the analysis model. In this paper, we perform technology analysis using BSTS, a model that considers prior knowledge and time factors simultaneously. We also carry out Bayesian regression analysis to construct technological relationship between technologies. Therefore, using the results of the BSTS and Bayesian regression, we will be able to handle various tasks of MOST.

4. Case Study using AI Technology

In this paper, we used the patent documents related to AI to illustrate how our model can be applied to real technology field. For our case study by the BSTS model and Bayesian regression, we used the R data language and its packages [21,22,24,25]. We retrieved the AI patents from the patent database WIPSON [19]. The collected patents were applied and registered from 1995 to 2016. Finally, we selected total 13,858 valid patents by 2015. Using the text mining techniques [22], we preprocessed the patent documents and extracted AI keywords from the documents. There were 36 extracted keywords, as follows: analysis, awareness, behavior, cognitive, collaborative, computing, conversation, corpus, data, dialogue, feedback, figure, image, inference, interface, language, learning, mind, morphological, natural, network, neuro, object, ontology, pattern, recognition, representation, sentence, sentiment, situation, spatial, speech, understanding, video, vision, and voice. We divided the 36 keywords into three groups (Group I, Group II, Group III) by alphabetical order. Figure 2 shows the keyword trends by year of Group I.

In the graph, the X and Y axes represent the year and the occurred frequency of each keyword respectively. We found that the trends of keywords analysis, behavior, corpus, data and feedback are meaningful. Next the trends of the keywords in Group II are shown in Figure 3.

The keywords of image, inference, interface, language, network, and object show meaningful trends in the plots. In particular, keyword network shows a relatively more significant trend than other keywords. Finally, the trends of the keywords belonging to Group III over time is shown in Figure 4.

We knew that the patent keywords of pattern, recognition, spatial, speech, video, vision and voice represent significant trends for AI technology. Using the visualization results of collected AI patent data, we determined the technologies related to AI as follows; neural networks (NN), data analysis (DA), cognitive computing (CC), natural language (NL), machine learning (ML), pattern recognition (PR), artificial intelligence (AI), and deep learning (DL). We collected trend data for the AI technologies described above from January 2010 to June 2019 in Google Trends [23]. Using the collected data, we visualized the trend data of AI technology in Figure 5.

The trend of AI technology has been rapidly increasing since 2014. This is a result of a lot of interest in AI and breakthrough technology development for AI from 2014. The trends of neural networks (NN) and pattern recognition (PR) technologies, which are closely related to AI traditionally, are similar to AI. The cognitive computing (CC) technology peaked between 2016 and 2017, but declined thereafter. This means that the CC technology based on cognitive science and cognitive psychology has limitations for AI developing. On the other hand, deep learning (DL) technology and related machine learning (ML) and data analysis (DA) technologies are continuously increasing. Based on trend visualization of these AI technologies, more advanced analysis is performed using BTST model for AI technology forecasting. Using trend data of AI technology, we carried out the BSTS modeling for forecasting 24 months.

Table 1. BSTS(Bayesian structural time series) forecasting results.

Time	NN	DA	CC	NL	ML	PR	AI	DL
1	100.29	97.12	25.28	36.22	96.18	80.05	97.30	93.05
2	100.90	98.15	25.25	36.87	97.10	80.89	97.80	93.45
3	102.26	99.03	22.56	36.49	97.94	80.64	98.82	94.04
4	102.74	99.60	21.42	36.70	98.71	81.79	99.72	94.63
5	103.71	100.73	20.70	37.06	99.59	82.27	100.06	95.36
6	103.16	101.02	20.27	37.40	100.53	82.70	100.59	95.87
7	103.90	102.59	18.87	37.48	101.51	82.35	101.60	96.64
8	103.03	102.80	18.62	37.97	102.69	83.19	102.63	97.19
9	103.91	103.77	17.54	38.03	103.28	83.31	103.24	98.18
10	104.85	104.49	16.37	38.26	104.16	83.76	103.25	98.92
11	105.42	105.41	15.14	38.33	105.20	84.48	104.42	99.35
12	106.22	106.12	13.77	38.51	105.94	84.92	105.32	99.91
13	105.20	107.66	13.24	38.80	106.78	85.26	106.16	100.67
14	107.21	108.16	12.22	39.15	107.77	85.64	107.02	101.10
15	108.39	108.40	11.04	39.32	108.67	85.53	107.64	101.82
16	110.07	109.33	9.91	39.70	109.42	86.56	108.10	102.49
17	111.98	110.73	9.81	40.16	110.58	87.47	109.25	102.91
18	112.73	111.14	8.19	40.22	111.43	87.11	109.95	103.74
19	114.49	112.08	7.16	40.24	112.30	87.27	110.97	104.18
20	113.47	112.57	6.34	40.55	113.20	87.51	111.66	104.83
21	114.68	113.93	6.18	41.17	114.12	88.38	113.26	105.29
22	114.38	114.60	5.23	41.44	115.15	89.12	114.35	105.81
23	115.61	115.64	2.86	41.17	116.06	89.52	114.98	106.36
24	116.15	116.77	3.94	42.25	117.13	89.65	116.20	106.70
Mean	107.63	106.79	13.79	38.79	106.50	85.00	106.00	100.08

Note: NN: neural networks; DA: data analysis; CC. cognitive computing; NL: natural language; ML: machine learning; AI: artificial intelligence; DL: deep learning.

shows the forecasting results by BSTS.

The NN (neural networks) has the largest value in the average of the 24-month forecasting. We know that deep learning, an important element of AI, is based on neural networks. The mean values of DA (data analysis) and ML (machine learning) are also larger than others. The mean value of CC (cognitive computing) is the smallest in the all technologies. This is because it is an AI technology used in some fields such as psychology or cognitive science. The following figures show the predicted values for eight detailed technologies and their confidence intervals of predicted values. Figure 6 shows the visualization of BSTS forecasting for NN, DA, CC, and NL.

The increasing slope of DA is larger than NN. In addition, the confidence interval of NN is larger than DA. This means that the fluctuation of NN is big. The slope of CC is forecasted to decrease gradually. The NL is forecasted to be slightly decreased or maintained. Figure 7 shows the predicted results of the BSTS model for the remaining four technologies.

Both ML and PR, including AI and DL, are forecasted to be increased. The predictive interval of PR is larger than others. So, we expect the future trend of PR to be relatively large. From the results of Figure 6 and Figure 7, we found that technologies of NN, DA, ML, and PR are expected to be continuously developed in the future.

We carried out another experiment using Bayesian regression. We consider two models by Bayesian regression. In our first model, the AI is used as response variable, and the others are used for predictive variables as follows:

$$\mathrm{AI}=b_0+b_{1}NN+b_{2}DA+b_{3}CC+b_{4}NL+b_{5}ML+b_{6}PR$$

(10)

The second model is defined as follows:

$$\mathrm{DL}=b_0+b_{1}NN+b_{2}DA+b_{3}CC+b_{4}NL+b_{5}ML+b_{6}PR$$

(11)

In this paper, we choose the Gaussian distribution as the prior distribution for Bayesian regression modeling.

Table 2. Statistical significance of Bayesian regression parameter.

Technology	Artificial Intelligence	Deep Learning
Neural networks	0.0044	0.0001
Data analysis	0.4270	0.3055
Cognitive computing	0.0001	0.0001
Natural language	0.0001	0.2219
Machine learning	0.0001	0.0001
Pattern recognition	0.0007	0.0843

shows the probability value (p-value) representing the statistical significance of regression parameters.

We confirmed the statistical significance of the parameters under 99% confidence level. That is, we can select the predictive variables with p-value less than 0.01 for final Bayesian regression model. So, for the response variable AI, we selected NN, CC, NL, ML, and PR for the final predictive model as follows:

$$\mathrm{AI}=b_0+b_{1}NN+b_{2}DA+b_{3}CC+b_{4}NL+b_{5}ML+b_{6}PR$$

(12)

Also, we built the predictive model for the response variable DL as follows:

$$\mathrm{DL}=b_0+b_{1}NN+b_{2}CC+b_{3}ML$$

(13)

Using these selected models, we predicted the trends of Al and DL. Figure 8 shows the trends of two response variables.

We can expect a very rapid increase in the trend of AI (artificial intelligence) and DL (deep running) in the future. Especially, the slope of AI and DL are very similar to each other, and it is confirmed that the influence of deep running on artificial intelligence development will be very large. Using the results of BSTS and Bayesian regression, we build the sustainable technology structure of AI in Figure 9.

AI is directly influenced by natural language processing, deep learning, and pattern recognition technologies. In addition, deep learning is affected by the technologies of artificial neural networks, cognitive computing systems, and machine learning algorithms.

5. Discussion

The method proposed in this paper has both strengths and weaknesses. First, the strength of the proposed method is the construction of a technology analysis model that considers the time factor. In other words, for the MOST related to AI technology, a sustainable technology analysis was conducted in consideration of the time trends of the detailed technologies of AI from the past to the present. In addition, we can use the prior distribution of Bayesian statistics to incorporate the expert’s prior experiences of AI into the technology analysis model. In other words, the probability distribution suitable for prior experience was designated and multiplied by the likelihood function based on patent data to obtain the posterior probability distribution. The proposed model was based on the posterior distribution.

In this paper we used the conjugate prior as a prior distribution. This was due to the convenience of the calculation. For example, if the likelihood function has a Gaussian distribution, the prior probability distribution is also determined as a Gaussian distribution. However, various prior distributions can be used to reflect the diverse prior information of experts in the model construction as much as possible. However, if the likelihood function and the prior distribution are heterogeneous, it is difficult to calculate the posterior probability distribution. However, to properly reflect expert knowledge in the model, the most appropriate prior probability distribution should be used rather than using the conjugate distribution. In such cases, Bayesian computing such as Marko Chain Monte Carlo (MCMC) should be used to calculate the posterior probability distribution.

In our future works, we will study Bayesian deep learning for time series forecasting, and add more diverse distribution for prior probability distribution to this model. We use this prior distribution to add domain knowledge of target technology to the new Bayesian time series deep learning model, and derive the results of technology analysis for identifying the sustainable technology trends. This research will contribute to sustainable technology management, such as research and development planning that can continuously improve the technological competitiveness of countries and companies.

6. Conclusions

We proposed a method to build the sustainable technology structure of target technology field. In this paper, we applied the BSTS and Bayesian regression to our proposed modeling. We selected artificial intelligence as the target technology for practical application of the proposed method. In our case study, we collected the patent documents related to AI. We extracted the technological keywords of AI from the patent documents. Using the extracted patent keywords, we selected the representative technologies for AI. We searched the time series data of the selected technologies from the Google Trends, and analyzed them by the BSTS and Bayesian regression. We combined two results of the BSTS and Bayesian regression, and made a sustainable technology structure. The proposed method can be applied to diverse technological fields such as smart healthcare, Internet of things, self-driving cars, etc. It is important for a company to find sustainability for the technology sector in order to improve its technological competitiveness continuously in the market. In our future works, we will study more advanced methods to detect sustainable technologies. To this end, we will study new methods using not only statistics but also deep learning.