Supply Sequence Modelling Using Hidden Markov Models

Abstract

Logistics processes, their effective planning as well as proper management and effective implementation are of key importance in an enterprise. This article analyzes the process of supplying raw materials necessary for the implementation of production tasks. The specificity of the examined waste processing company requires the knowledge about the size of potential deliveries because the delivered waste must be properly managed and stored due to its toxicity to the natural environment. In the article, hidden Markov models were used to assess the level of supply. They are a statistical modeling tool used to analyze and predict the phenomena of a sequence of events. It is not always possible to provide sufficiently reliable information with the existing classical methods in this regard. Therefore, the article proposes modeling techniques with the help of stochastic processes. In hidden Markov models, the system is represented as a Markov process with states that are invisible to the observer but with a visible output (observation) that is a random state function. In the article, the distribution of outputs from the hidden states is defined by a polynomial distribution.

1. Introduction

The proper functioning of an enterprise is an important element of maintaining its continuity. Therefore, it is necessary to constantly track and analyze the signals coming from both the company itself and its environment in order to quickly respond to changes, modify adopted strategies, and introduce innovative solutions so as to adapt the company’s operation to the changing market conditions on an ongoing basis.

Logistics processes and their effective planning, proper management and effective implementation are of key importance here. The article examines the supply logistics, i.e., the process of supplying raw materials necessary for the implementation of production tasks. The specificity of the examined waste processing company requires knowledge about the size of potential deliveries because the delivered waste, which in this case is the production raw material, must be properly managed and stored due to its toxicity to the natural environment.

In the article, hidden Markov models are used to assess the level of supply. They are a statistical modeling tool used to analyze and predict the phenomena of a sequence of events [1]. It is not always possible to provide sufficiently reliable information with the existing classical methods in this regard. Therefore, the article proposes modeling techniques that use stochastic processes. In hidden Markov models (HMMs), the system is represented as a Markov process with states that are invisible to the observer but with a visible output (observation) that is a random state function. HMMs can be defined as a machine learning model, particularly as a discrete method, which is a statistical strategy for modeling systems intended to contain Markov processes with hidden states. It is possible to train HMMs on a specific sequence of observations. In addition, the sequence of observations on a specially trained HMM l can also be evaluated to determine the probability of observing such a sequence within the constraints of a particular model. With HMMs, a sequence of observations can be evaluated to see whether it fits a given model well. The higher the score, the greater the match between the observation sequence and the observational data used to train the model. In addition, HMMs allow decoding. This means that we can discover the “best” sequence of hidden states and, as a result, maximize the expected number of correct states. It is widely known that HMMs can be used in different applications [2,3], but still, the use of HMMs in many new application fields is yet to be explored [4,5].

Moreover, research results presented in literature indicate that models obtained using HMM have been successfully applied in the different areas and achieve better results compared to models obtained using, for example, naive bayes (NB), decision tree (DT), nearest neighbors (KNN), support vector machine (SVM), artificial neural networks (ANN) or radial bias function (RBF) [6,7,8].

The main problem in the construction of the hidden Markov models is the proper selection of the number of states and the type of connections between them as well as the estimation of the model parameters that will ensure its high predictive ability. The use of HMMs allows for the prediction of the most probable sequence of the next states based on the observed sequence of events. This issue is discussed in detail in the article on the example of a manufacturing company. On the basis of the weight of the product received at the plant, Markov states were distinguished and specific amounts of raw material delivery were assigned to them. Five observed states were assumed corresponding to five levels of supply from very low, through low, medium, high, to very high. On the basis of the observed sequence of events, a sequence of the most probable hidden states was identified, the occurrence of which depends only on the state in the preceding moment. For the sequence of the hidden states, a matrix of transition probabilities between the states was determined. The identification of deliveries using HMM allows to provide the company with the information (with a certain probability) regarding subsequent deliveries. In this article, the distribution of outputs from the hidden states is defined by a polynomial distribution.

The article is organized as follows. Section 2 contains an overview of the literature on the subject. Section 3 characterizes the subject of the study, i.e., the analyzed company. Section 4 presents mathematical methods used in the article, presents the characteristics of Markov chains, defines hidden Markov models (HMMs) and discusses the problem of their identification. In Section 5, the parameters of the hidden Markov model for the supply process of the surveyed company were estimated. It allowed to draw conclusions about the size of the subsequent deliveries. The article ends with a summary of the conducted analyses, final conclusions and an indication of the directions for future research (Section 6).

2. Literature Review

Prediction in logistics systems is a topic widely discussed in the literature. Forecasts regarding demand, supply, deliveries, and orders are a valuable source of information for the preparation of logistics plans and their implementation in an enterprise as well as for building flexible systems responding to deviations of reality from the forecast. Both short-term and long-term forecasts are important in this area as they can support the enterprise management at the operational and strategic levels, respectively [7,8,9,10,11,12,13,14]. Short-term forecasts are useful in planning daily or weekly operations, while long-term forecasts are useful when planning the development of the company’s infrastructure or other long-term investments [10,11,15].

Most quantitative methods use historical data to produce a forecast. The information regarding the expected level of demand can support decisions in the area of production, workforce planning, pricing and defining the marketing strategy. Obviously, it is not possible to construct forecasts that perfectly correspond to future events, and absolutely certain forecasts do not exist. Therefore, the methods of prediction are constantly developed and improved in order to achieve the highest possible level of precision. The vast majority of the studies available in the literature in this area are based on time series analysis. Simple models of exponential smoothing [10,11,16], moving average [17], and finally ARMA class models are most often used due to their high flexibility of application both to stationary and non-stationary series. For that reason, they are very popular in forecasting within a supply chain [10,18]. However, in case of time series, a big problem involves rapid changes as well as discontinuity in demand and high seasonality, e.g., in relation to the products sold only in certain months of the year. The difficulty in finding regularities is not conducive to accurate predictions. This volatility of the market and its randomness make it necessary to implement new methods that counteract external factors.

Machine learning methods have been developing dynamically recently. This is particularly due to the possibility of obtaining precise and extensive data on their functioning from logistics systems, the so-called big data [19,20,21]. Various methods are used in this regard, including traditional neural networks [22,23], fuzzy neural networks [24,25], support vector machines [26,27], decision trees [28,29,30], random forests [31,32] or k-means algorithms [33]. Machine learning results, compared to empirical data, often bring more satisfactory results [34,35,36], often provide more accurate forecasts and are a real alternative to time series. In addition, they are less demanding in terms of pre-processing and data processing.

Nevertheless, in each case, it is necessary to take into account a certain inaccuracy of forecasts and search for such methods of logistics planning that will take into account the possibility of significant deviations of the actual results of the company from the assumed assumptions in the best possible way. In addition, it should be emphasized that even despite the not always satisfactory verifiability of the forecasts, mathematical models are still the foundation of logistics planning in enterprises so the methods of their improvement should be constantly developed. Therefore, the authors in this article proposed an alternative method that uses hidden Markov models in order to identify deliveries and they presented its implementation on the example of a manufacturing company.

Thus, the following three research objectives were adopted:

(1)

To propose a method for forecasting the level of subsequent supplies of raw material for production using hidden Markov models.

(2)

To identify the hidden states fulfilling the Markov property which describe the dynamics of raw material supplies.

(3)

To present a method of modeling using stochastic processes that can be used primarily in relation to irregular deliveries with high randomness.

3. Description of the Case Study Company

The case study company deals with the recovery and processing of waste into alternative fuels, RDFs (refuse-derived fuels) [37]. RDFs are created as a result of the transformation of waste the energy potential of which is sufficient to obtain energy or the properties of which allow it to be processed into products that can be used for energy. RDF is a specific type of fuel, characterized by high calorific value and homogeneous particle size. Its production consists in separating the combustible fraction (paper, plastics, textiles, wood, rubber) from municipal waste by sorting it and subjecting it to a multi-stage process of shredding followed by briquetting [38]. The production of alternative fuels is an excellent way to minimize the overall amount of waste and to obtain energy, the recipients of which in Poland are cement plants. Therefore, the examined plant is located in the Cement Plant Chełm. It specifically deals with the production of rubber granules using the material resulting from the recycling of the used car tires and other rubber components. A sample of the final product is presented in Figure 1.

Waste is delivered to the indicated plant by trucks or vans. After entering the plant and undergoing a qualitative assessment, the waste in the form of tires is weighed using a truck scale and then it is directed to the unloading storage yard. The technological line for the processing of rubber waste consists of a pre-shredding section, proper grinding as well as magnetic and pneumatic separation. The technological line includes shredders, granulators, sets of special sieves, belt conveyors, mixers, and magnetic and pneumatic separators [37]. The preliminary shredding section includes machines and devices responsible for the initial shredding of waste into the 20–40 mm fractions (pre-shredders), then, in the proper shredding section, through the sets of granulators and specialized grinders, proper shredding takes place, resulting in the appropriate fractions of 0.01–30 mm granules, combined with simultaneously isolating rubber, wire and textile cord to separate devices for further processing [32]. The isolated raw materials are directed to the machine groups responsible for the production of final products. The shredded rubber is directed to mills and pneumatic separators in order to clean it and eliminate undesirable impurities. Then, it is packed in collective packaging. The steel wire is directed to magnetic separators and then to other grinding devices and a sieve classifier which cleans the wire of impurities. The cleaned wire (steel cord) is directed to the places of temporary storage in the storage yard [37].

The textile cord is isolated throughout the process with the use of pneumatic separators, and then, in part, it is directed to cleaning on devices used for the production of a stabilizing additive for mineral–asphalt mixtures. After cleaning, the cord is used for further processing in the machine called homogenizer where the filling emulsion and mineral filler are added to the textile cord. After adding the substrates, the mixing of the whole takes place. The mass prepared in this way is transported to the granulator where it is granulated into pellets [37].

At each of the stages of separation and cleaning of rubber, steel cord, and textile cord in the technological process, waste is generated in the form of low-quality rubber and other solid impurities derived from processing. The waste is an excellent quality alternative fuel that is processed in the combustion process in the cement plant. The general overview of the process is presented in Figure 2.

The company processes a total of approx. 47,000 tons of waste per year, including used tires as well as plastics and rubber. In connection to the functioning of the installation, the waste in the form of ferrous metals and alternative fuels is generated. Both types of finished products, i.e., rubber granulate and a stabilizing additive for mineral–asphalt mixes, are widely used in the manufacturing of products, e.g., car mats, slabs and rubber paving stones for playgrounds, etc. Tires are stored in special boxes outside (Figure 3).

The functioning of this type of company requires a lot of staff involvement and the possession of a modern technological line. The energy potential of waste varies and requires a proper selection and segregation. Therefore, a properly planned, organized and conducted observation of the delivered waste is crucial. The model proposed in the article can successfully support the supply logistics management processes in the surveyed company.

4. Materials and Methods

4.1. Markov Chains

Below, we assume the following: $R$—the set of all real numbers; $N$—a set of natural numbers including zero; $\left(\Omega ,F,P\right)$—probabilistic space.

Definition 1.

A stochastic process${\left\{X_t\right\}}_{t\in T}$is a family of random variables$X_t:\Omega \to S$defined on the probabilistic space$\left(\Omega ,F,P\right)$for any$t\in T\subset ℝ$and$S$is the set of realizations.

Definition 2.

A stochastic process${\left\{X_t\right\}}_{t\in T}$is a Markov process [1,35] if for every$n$, for any finite subset$\left\{t_1,t_2,\dots ,t_n\right\}$, where$t_1<t_2<,\dots ,<t_n$,$t_i\in T$, and for any states$\left\{x_1,x_2,\dots ,x_n\right\}$,$x_i\in S$,$1\le i\le n$the property

$$P\left(X_{t_n}<x_n|X_{t_{n-1}}=x_{n-1},X_{t_{n-2}}=x_{n-2},\dots ,X_{t_1}=x_1\right)=P\left(X_{t_n}<x_n|X_{t_{n-1}}=x_{n-1}\right)$$

(1)

is satisfied.

Definition 3.

A stochastic process ${\left\{X_t\right\}}_{t\in T}$is called a homogeneous process [35] if for any$s,t\in T$,$s<t$x$,y\in S$the property

$$P\left(X_t<y|X_s=x\right)=P\left(X_{t-s}<y|X_0=x\right)$$

(2)

is satisfied.

The condition (1.1) is called the Markov property. In the paper, we assume that the set of moments in which we observe the realizations of the stochastic process is a subset of the set of natural numbers, $T\subset N$, while the set of realizations of random variables $X_t$, $t\in N,$ is a finite set $S=\left\{S_1,S_2,\dots ,S_m\right\}$, $m\in N$. The set $S$ is called the set of states of the stochastic process. A Markov process with discrete time is called a Markov chain.

Definition 4.

A sequence of random variables${\left\{X_t\right\}}_{t\in N}$with the values in the set of states$S$is called a Markov chain if, and only if, for any$n\in N$and$x_t\in S$,$t=0,1,\dots ,n,$the property [10,39]

$$P\left(X_n=x_n|X_0=x_0,X_1=x_1,X_2=x_2,\dots ,X_{n-1}=x_{n-1}\right)=P\left(X_n=x_n|X_{n-1}=x_{n-1}\right)$$

(3)

is satisfied.

For a homogeneous Markov chain ${\left\{X_t\right\}}_{t\in N,}$, the distribution of transition probabilities between the states is determined by the transition probability matrix

$$P={\left[p_{ij}\right]}_{1\le i,j\le m},$$

(4)

where $p_{ij}\ge 0$ for $1 \le i,j \le \mathrm{m}$ and ${{\displaystyle \sum }}_{j=1}^m{p}_{ij} = 1$ for $1\le i\le m$, and $p_{ij}\stackrel{def}{=}{p}_{S_i{S}_j}=P\left(X_{n+1}=S_j|X_n=S_i\right)$ is the probability of transition from state $S_i$ at time $n$ to state $S_j$ at time $n+1$ for any $n\in N$.

For a homogeneous Markov chain, the probability matrix for $k$ steps is

$$P^{\left(k\right)}={\left[p_{ij}^{\left(k\right)}\right]}_{1\le i,j\le m},$$

(5)

where $p_{ij}^{\left(k\right)}=P\left(X_{n+k}=S_j|X_n=S_i\right)$ is the probability of transition from state $S_i$ at time $n$ to state $S_j$ at time $n+k$ for any $n\in N,$ and

$$P^{\left(k\right)}=P^{\left(k-l\right)}{P}^{\left(l\right)}$$

(6)

for $1\le l<k$. Equation (6) is called the Chapman–Kolmogorov equation [1,39,40].

Let $\pi =\left({\pi }_1,{\pi }_2,\dots ,{\pi }_m\right)$, ${\pi }_j=P\left(X_0=S_j\right)$ for $S_j\in S$ be the initial distribution for a homogeneous Markov chain ${\left\{X_t\right\}}_{t\in N}$ with a one-step transition matrix equal to $P$ and ${{\displaystyle \sum }}_{j=1}^m{\pi }_j=1$.

Definition 5.

The initial distribution$\pi =\left({\pi }_1,{\pi }_2,\dots ,{\pi }_m\right)$of a homogeneous Markov chain${\left\{X_t\right\}}_{t\in N}$with transition matrix$P$is stationary if the condition

$$\pi =\pi P$$

(7)

is satisfied.

A homogeneous Markov chain with a stationary initial distribution is called a stationary Markov chain [1,39,40].

4.2. Hidden Markov Models

Hidden Markov model (HMM) is a model consisting of two parts (elements, “layers”): hidden and observed (inner and outer) [11,40,41,42]. The hidden (inner) part is subordinated to the Markov process, of which the states are not observed (that is why they are called hidden), while we observe the external part of the model where the observed values depend on the realization of the hidden part. The identification of hidden states and the relationship between the hidden and the observed part consists in the analysis of a sequence of events (sequence of states) that has been recorded (observed).

Below, we consider a hidden Markov model with discrete outputs and assume that the hidden part is described by a homogeneous Markov chain:

• $S=\left\{S_1,\dots ,S_m\right\}—$a set of hidden states;
• $V=\left\{V_1,\dots ,V_h\right\}—$a set of outputs (set of observed states);
• $\pi =\left({\pi }_1,{\pi }_2,\dots ,{\pi }_m\right)—$initial distribution, where ${\pi }_i\stackrel{def}{=}{\pi }_{S_i}=P\left(X_0=S_i\right)\ge 0$;
• $P={\left[p_{ij}\right]}_{1\le i,j\le m}—$transition probability matrix between the hidden states, where for any $n\in N$ the value $p_i\stackrel{def}{=}{p}_{S_1{S}_j}=P\left(X_{n+}=S_j|X_n=S_i\right)$ denotes the probability of transition from state $S_i$ at time $n\in N$ to state $S_j$ at time $n+1$;
• $G={\left[g_{ij}\right]}_{1\le i\le m,1\le j\le h}—$output probability matrix, where for any $n\in N$ quantity $g_{ij}\stackrel{def}{=}{g}_{S_i{V}_j}=P\left(O_n=V_j|X_n=S_i\right)$ is the probability of observing the state $V_j$ at time $n$, provided that the hidden part of the system is in the state $S_j$.

The elements of the transition matrix and output probability matrix satisfy the condition

$${{\displaystyle \sum }}_{j=1}^m{p}_{ij}={{\displaystyle \sum }}_{j=1}^h{g}_{ih}=1 \mathrm{for}\mathrm{any}i\in \left\{1,2,\dots ,m\right\}$$

(8)

and the initial distribution $\pi$ satisfies the condition

$${\displaystyle \sum }_{j=1}^m{\pi }_j=1.$$

(9)

The hidden Markov model with discrete outputs is identified by the set $\lambda =\left(\pi ,P,G\right)$, where $\pi \in {\left[0,1\right]}^m$, $P\in {\left[0,1\right]}^{m\times m}$ and $G\in {\left[0,1\right]}^{m\times h}$, satisfying the conditions (8) and (9) [6,40,41].

4.3. The problem of Identifying Hidden Markov Models

The identification of hidden Markov models consists in estimating the parameters of the model, namely the initial distribution $\pi$, the transition probability matrix between the hidden states $P$ and the output probability matrix $G$, based on the observations $O=\left\{O_1,O_2,\dots ,O_T\right\}$, for $O_j\in V$, $1\le j\le T$. Determination of the structure of HMM $\lambda =\left(\pi ,P,G\right)$ consists in solving three problems [8,40].

Problem 1.

Estimation of the probability of the sequence of realizations$O=\left\{O_1,O_2,\dots ,O_T\right\}$for the hidden model of the Markov model$\lambda =\left(\pi ,P,G\right)$, which is the sum of all possible probabilities for the sequences of hidden states$Q=\left\{q_1,q_2,\dots ,q_T\right\}$,$q_i\in S$for$1\le i\le T$.

$$\begin{array}{cc}P\left(O|\lambda \right)& ={\displaystyle {\displaystyle \sum }_Q}P\left(O,Q|\lambda \right)={\displaystyle {\displaystyle \sum }_Q}P\left(O|Q,\lambda \right)P\left(Q|\lambda \right)\\ & ={\displaystyle {\displaystyle \sum }_Q}{\pi }_{q_1}{g}_{q_1{O}_1}{p}_{q_1{q}_2}{g}_{q_2{O}_2}{p}_{q_2{q}_3}{g}_{q_3{O}_3}\dots g_{q_T{O}_T}.\end{array}$$

(10)

Problem 2.

Determination of the sequence of hidden states$Q=\left\{q_1,q_2,\dots ,q_T\right\}$(Viterbi path), which is best explained for the sequence of observations$O=\left\{O_1,O_2,\dots ,O_T\right\}$, where$q_i\in S$and$O_j\in V$for$1\le j\le T$. Determining the sequence$Q$for the hidden model of the Markov model$\lambda =\left(\pi ,P,G\right)$consists in solving the problem

$$\underset{Q}{max}P\left(Q|O,\lambda \right)=\underset{Q}{max}\frac{P\left(Q,O|\lambda \right)}{P\left(O|\lambda \right)}.$$

(11)

Problem 3.

Determination of the parameters$\lambda =\left(\pi ,P,G\right)$for the hidden Markov model consists in solving the problem

$$\underset{\lambda }{max}P\left(O|\lambda \right).$$

(12)

We use the forward–backward algorithm to estimate the values defined by Equation (10). We use the Viterbi algorithm to determine the sequence of hidden states with the highest probability (11). The values $\lambda =\left(\pi ,P,G\right)$ are determined by maximizing the probability (12) of the occurrence of the series of observations $O$ using the Baum–Welch algorithm [6,40,41].

5. Hidden Markov Models in Forecasting the Supply Sequence

The works [43,44] underline that the effective planning of the supply process is the key importance in a company. For the presented company, the assessment of the probable level of supply in the production raw material (i.e., used tires) was analyzed. The company has many suppliers, which is why the tires come from many different sources, and due to the specificity of the assortment, it is often difficult to determine the expected weight of the delivery. The figure below, presenting the level of deliveries over almost three years (Figure 4), clearly shows how significantly they can vary.

The acceptance of this type of assortment requires the preparation of an appropriate storage place, securing the right number of people and means of transport. In addition, the knowledge of expected deliveries is crucial in planning production processes. Forecasting future deliveries was preceded by an analysis of historical data. First, the deliveries were divided into four types, depending on their weight. This way, the observed states, presented in

Table 1. Range of supply volumes assigned to the individual observed states.

Condition	Range of Supply Volumes [Ton]
O1	[0, 27]
O2	(27, 37]
O3	(37, 43]
O4	(43, 50]
O5	Over 50

, were distinguished.

Next, the four hidden states describing the dynamics of the hidden part of the system and using the Markov model for this purpose were determined. In order to describe the dynamics of hidden states, the probability matrix of transitions between the states was determined. These values are presented in the graph showing the relationships between hidden states (Figure 5).

The above figure shows that the highest values of transitions are for returns to the same state. The values of interstate transitions are much lower. The responses in the form of probabilities of transitions from the hidden states to the observed states are presented in

Table 2. Probability values of transitions from the observed states.

	O1	O2	O3	O4	O5
S1	0.906	0.070	0.000	0.000	0.024
S2	0.054	0.161	0.370	0.143	0.272
S3	0.139	0.410	0.311 0.141	0.000	S4
S4	0.042	0.000	0.104	0.129	0.725

.

Depending on the relations in the hidden layer, one can infer the probability of the size of the next delivery. Probability distributions of exits from the hidden states to the states observed are shown in a graphical form in Figure 6.

The description of the model with the use of Figure 4 and Figure 5 provides an overview of the behavior of raw material supplies in the supply chain. The probability that a given observed state will occur varies greatly depending on the hidden state. The highest probability values were obtained for the exit from the state S1 to O1 and from S4 to O5. In the event of the hidden state S1, a small supply of raw material (O1) will occur with high probability, while in the case of the hidden state S4 there will be a very large delivery of more than 50 tons.

6. Conclusions

In the article, the authors proposed a description of the supply of raw material to the enterprise using hidden Markov models, different from the time series models, and presented its implementation on the example of a manufacturing enterprise. Observing the order of deliveries, the sequence of the most probable hidden states was identified, the occurrence of which depends only on the state at the moment before. Then, the probabilities of exit from hidden states to the observed states were estimated. This made it possible to forecast the level of supply of raw material for production using hidden Markov models and thus provide the company with the valuable information (with a certain probability) which allows to prepare for the subsequent batches of goods. By identifying the hidden Markov model, we can define and predict changes in the transport schedule. As a result, this will reduce the number of journeys (number of transports), and thus reduce the costs of transporting waste. In addition, reducing the number of transports will have an impact on reducing emissions of harmful substances into the environment.

Additionally, the information obtained from the model allows for better scheduling of work and an even allocation of tasks between employees. If the workload is greater, it is possible to provide proportional salary as well as higher social benefits. Moreover, knowledge of planned deliveries also allows to shape the level of work safety, adjust the necessary equipment and equipment of employees adequately to the type of activities performed and the goods/assortment received.

The presented model is easy to apply directly in the analyzed enterprise. Moreover, the developed method is applicable especially in case of the observed features, phenomena or systems that are characterized by rapid changes, discontinuity or significant seasonality. It is the response to the volatility of the market and its randomness as well as a tool for forecasting that counteracts the negative impact of external factors on the quality of forecasting.

The next step of the research will be the development of models using other methods as well as the comparison and evaluation of the obtained results.