Advances in Artificial Neural Networks – Methodological Development and Application

Abstract

Artificial neural networks as a major soft-computing technology have been extensively studied and applied during the last three decades. Research on backpropagation training algorithms for multilayer perceptron networks has spurred development of other neural network training algorithms for other networks such as radial basis function, recurrent network, feedback network, and unsupervised Kohonen self-organizing network. These networks, especially the multilayer perceptron network with a backpropagation training algorithm, have gained recognition in research and applications in various scientific and engineering areas. In order to accelerate the training process and overcome data over-fitting, research has been conducted to improve the backpropagation algorithm. Further, artificial neural networks have been integrated with other advanced methods such as fuzzy logic and wavelet analysis, to enhance the ability of data interpretation and modeling and to avoid subjectivity in the operation of the training algorithm. In recent years, support vector machines have emerged as a set of high-performance supervised generalized linear classifiers in parallel with artificial neural networks. A review on development history of artificial neural networks is presented and the standard architectures and algorithms of artificial neural networks are described. Furthermore, advanced artificial neural networks will be introduced with support vector machines, and limitations of ANNs will be identified. The future of artificial neural network development in tandem with support vector machines will be discussed in conjunction with further applications to food science and engineering, soil and water relationship for crop management, and decision support for precision agriculture. Along with the network structures and training algorithms, the applications of artificial neural networks will be reviewed as well, especially in the fields of agricultural and biological engineering.

1. Introduction

Soft computing is a partnership of computing techniques. The partnership includes fuzzy logic, Artificial Neural Networks (ANNs), and genetic algorithms. Conventionally, a huge set of techniques are referred to as hard computing, such as stochastic and statistical methods bound by the concept called NP (verifiable in Nondeterministic Polynomial time)-complete. Unlike hard computing, soft computing techniques offer “inexact” solutions of very complex problems through modeling and analysis with a tolerance of imprecision, uncertainty, partial truth, and approximation. In fact, soft computing is an integration of biological structures and computing techniques. In the partnership, ANNs provides configurations made up of interconnecting artificial neurons that mimic the properties of biological neurons.

There are a variety of ANN architectures, such as Multilayer Feedforward Network (MFN), Radial Basis Function (RBF) network, recurrent network, feedback network, and Kohonen Self-Organizing Map (SOM) network. Each of these networks has a training algorithm to memorize what it “sees” and project what it does not “see” in the same population. Typical training algorithms include Back-Propagation (BP) for Multi-Layer Perceptron (MLP, which is a MFN model), data clustering and linear weight solution for RBF networks, and unsupervised SOM for Kohonen SOM networks. In 1986, Rumelhart et al. [1] reported the BP training algorithm by implementing the Delta rule, a gradient descent learning rule for updating the weights of the artificial neurons in the perceptron-type ANNs, for MFNs in the monograph of the Parallel Distributed Processing (PDP) research group [2]. This work introduced and enhanced recognition of the BP algorithm, gave a strong impulse to the study of the mechanisms and structure of the brain, and provided the catalyst for much of the subsequent research and application of ANNs.

There are many issues and problems in applying ANNs which have inspired studies and research for improving the standard methods to solve problems. Determination of network structure is an issue in identification of ANN models, such as before implementing a MFN, the optimal numbers of hidden layers and nodes need to be decided on [3,4,5], and network pruning for redundancy connection reduction to improve network generalization capacities [6]. The slow convergence speed and local minima sticking with suboptimal solutions of BP training for MLPs have been studied [7,8,9,10,11,12,13,14]. Research has been conducted to improve the standard BP algorithm to overcome data over-fitting [15,16,17,18,19,20,21].

With the development of research and applications, ANNs have been integrated or fused with other methods of soft computing and signal processing such as fuzzy logic [22,23,24,25,26,27] and wavelet analysis [28,29,30,31]. The fusion is to combine or cascade different computing methods with ANNs to improve system performance over an individual technique. In many cases, the problems can be solved more effectively by combining one or two other techniques rather than implementing ANNs exclusively. In this way, the fused methods complement each other to enhance the ability of data interpretation and modeling and to avoid subjectivity in the operation of the training algorithm with ANNs individually.

Support Vector Machines (SVMs) emerged as a method in parallel with artificial neural networks as a set of supervised generalized linear classifiers and often provide higher classification accuracies than MLP neural networks [32,33]. SVMs have a number of advantages over ANNs and have been attracting more and more interest in recent years.

In agricultural and biological engineering, although some early research and applications exist [34,35,36], the interest of ANNs has been growing greatly in the last fifteen years in studies of soil and water regimes related to crop growth, analysis of the operation of food processing, and support of decision-making in precision farming. In summary of papers and reports collected from various sources, particularly through searching in the technical library of ASABE (American Society of Agricultural and Biological Engineers; http://asae.frymulti.com/) and the National Agricultural Library of USDA (United States Department of Agriculture; http://www.nal.usda.gov/), it was found out that, from early 1990s to the present, there have been 348 related reports and papers (193 peer reviewed) on ANNs. It is interesting that of the 20 reports and papers (13 peer reviewed) on SVMs from 2003 to the present, seven (two peer reviewed) came out in 2008. This may signify more interest of SVMs in the next decade in agricultural and biological engineering.

The purpose of this paper is to overview ANN methodology, to determine the state of the art, to identify the limitations, and to project the future. Specifically, this paper will review the history of the development of ANNs. Then, the standard architectures and algorithms of ANNs will be described. Furthermore, a number of advanced ANN models will be introduced with SVMs, and the limitations of ANNs are identified. The future of research and applications of ANNs will be discussed with the development of SVMs. With the concepts, network structures, and training algorithms, the applications of ANNs will be reviewed with a focus on agricultural and biological engineering as well.

2. History of ANN Development

ANNs provide a method to characterize synthetic neurons to solve complex problems in the same manner as the human brain. For many years, especially since the middle of the last century, an interest in studying the brain’s mechanism and structure has been increasing. This growing research interest has led to the development of new computational models, connectionist systems or ANNs, based on the biological background, for solving complex problems like pattern recognition, and fast information processing and adaptation.

In the early 1940s, McCulloch and Pitts [37] studied the potential and capabilities of the interconnection of several basic components, based on the model of a neuron. Later on, others, like Hebb [38], were concerned with the adaptation laws involved in neural systems. In 1958, Rosenblatt [39] coined the name Perceptron and devised an architecture which has subsequently received much attention. In 1960, Widrow and his student, Hoff, [40] presented an important generalization of the perceptron training algorithm as the Least Mean Square (LMS) learning procedure, also known as the Delta rule. The learning rule was applied as the adaptive linear element in the ADALINE (ADAptive LInear Neuron) networks. Then, Minsky and Papert [41] introduced a rigorous analysis of the Perceptron, of which they proved many properties and pointed out limitations of several related models. In the 1970s, the work of Grossberg [42] came to prominence. His work, based on biological and psychological evidence, proposed several novel architectures of nonlinear dynamic systems. In 1982, Hopfield [43] applied a particular nonlinear dynamic structure to solve problems in optimization. All of them conducted pioneer studies on the theoretical aspect of ANNs, particularly starting in the 1950s and 1960s.

In 1986, the PDP research group published a series of algorithms and results [2]. This publication contains an article entitled “Learning Internal Representations by Error Propagation” [1]. This article made the recognition of the BP training algorithm although it was already described in 1974 [44]. The BP algorithm implemented with the general Delta rule [1,7], as the representative of supervised training algorithms, gave a strong impulse to the subsequent research and has resulted in the largest body of research and applications in ANNs although a number of other ANN architectures and training algorithms have been developed and applied at the same time. In 1982, Finnish Professor Teuvo Kohonen [45] first described a different ANN architecture which is trained using an unsupervised SOM learning procedure. This Kohonen SOM algorithm has been populated in many research and practical applications later on. In 1988, RBF neural networks were first used [46], although RBFs were introduced earlier in 1977 [47]. RBF networks have also been widely used with the strong capability of function approximation and, along with MLP perceptron, had the impact on the emergence of SVMs [32,33,48]. In the late 1980s, the standard MFNs were proven as universal approximators on a compact subset of Rⁿ [49,50]. The MFNs have a single hidden layer containing finite number of hidden neurons. The neurons use arbitrary activation function The theory assures that MFNs, including MLP networks, RBF networks and even SVMs under certain conditions [51] can handle problems which are highly complex and nonlinear.

So far, ANNs have been used in many industrial and commercial applications such as process modeling and control [52], character recognition [53], image recognition [54], credit evaluation [55], fraud detection [56,57], insurance [58], and stock forecasting [59]. In a later section, applications of ANNs in agricultural and biological engineering will be reviewed.

3. ANN Architectures and Training Algorithms

3.1. MLP and BP

Because the original perceptrons, single layer feedfoward networks, which were introduced by Rosenblatt [39], are limited to learning linearly separately patterns, nonlinear layers between the input and output are added to separate the data with enough training to model any well-defined function to arbitrary precision. This MFN model is known as a multilayer perceptron. The neural networks formed using the model are universal approximators [49,50]. Figure 1 shows the diagram of a one-hidden-layered MLP network structure. The MLP networks are typically trained with the BP algorithm. The BP algorithm is supervised, which is to map the process inputs to the desired outputs by minimizing the errors between the desired outputs and the calculated outputs driven from the inputs and network learning.

Figure 1. MLP network structure.

Figure 1. MLP network structure.

The standard BP algorithm [1] is a gradient descent algorithm to minimize the mean square error between the calculated outputs and the desired outputs of the MLP network. Assuming that there are n process inputs, x, and m desired outputs, d, with the network with a sample size of N: {x_it, d_jt | i = 1, 2, … n; j = 1, 2, …, m; t = 1, 2, …, N}, then the mean square error of the network outputs is:

$$\text{E}=\frac{\text{1}}{\text{2}}{\displaystyle \sum _{\text{t}=\text{1}}^{\text{N}}{\displaystyle \sum _{\text{j}=\text{1}}^{\text{m}}{{\text{(d}}_{\text{jt}}-{\text{y}}_{\text{jt}}\text{)}}^{\text{2}}}}$$

(1)

where y_jt is the calculated output.

Based on the gradient descent method, each of the network connection weights is updated as follows:

$$\begin{array}{ll}\text{Δw}& =-\text{η}\nabla \text{E(w)}\\ & =-\text{η}\frac{\partial \text{E}}{\partial \text{w}}\\ & ={\displaystyle \sum _{\text{t}=\text{1}}^{\text{l}}{\displaystyle \sum _{\text{j}=\text{1}}^{\text{m}}{\text{(d}}_{\text{jt}}-{\text{y}}_{\text{jt}}\text{)}\frac{\partial {\text{y}}_{\text{jt}}}{\partial \text{w}}}}\end{array}$$

(2)

where η is a small constant such as 0.1 as the learning rate in the training process of the algorithm. For the MLP in Figure 1, all the weights are the sets of {w_ik | i = 1, 2, …, n; k = 1, 2, …, h} and {w_kj | k= 1, 2, …, h; j = 1, 2, …, m} where h is the number hidden nodes. Each of the output is as follows:

$${\text{y}}_{\text{jt}}=\text{s(}{\displaystyle \sum _{\text{k}=\text{1}}^{\text{h}}{\text{w}}_{\text{kj}}\text{s(}{\displaystyle \sum _{\text{i}=\text{1}}^{\text{n}}{\text{w}}_{\text{ik}}{\text{x}}_{\text{it}}\text{))}}}$$

(3)

where s(z) is the activation function of each hidden node and output node. It typically is a sigmoid function as s(z) = 1/(1 + e^-z). Therefore,

$$\frac{\partial {\text{y}}_{\text{jt}}}{\partial {\text{w}}_{\text{kj}}}=\text{s'(}{\displaystyle \sum _{\text{k}=\text{1}}^{\text{h}}{\text{w}}_{\text{kj}}\text{s(}{\displaystyle \sum _{\text{i}=\text{1}}^{\text{n}}{\text{w}}_{\text{ik}}{\text{x}}_{\text{it}}\text{))}}}\text{[}{\displaystyle \sum _{\text{k}=\text{1}}^{\text{h}}\text{s(}{\displaystyle \sum _{\text{i}=\text{1}}^{\text{n}}{\text{w}}_{\text{ik}}{\text{x}}_{\text{it}}\text{)]}}}$$

(4)

$$\frac{\partial {\text{y}}_{\text{jt}}}{\partial {\text{w}}_{\text{ik}}}=\text{s'(}{\displaystyle \sum _{\text{k}=\text{1}}^{\text{h}}{\text{w}}_{\text{kj}}\text{s(}{\displaystyle \sum _{\text{i}=\text{1}}^{\text{n}}{\text{w}}_{\text{ik}}{\text{x}}_{\text{it}}\text{))}}}\text{[}{\displaystyle \sum _{\text{k}=\text{1}}^{\text{h}}\text{s(}{\displaystyle \sum _{\text{i}=\text{1}}^{\text{n}}{\text{w}}_{\text{ik}}{\text{x}}_{\text{it}}\text{)}+{\displaystyle \sum _{\text{k}=\text{1}}^{\text{h}}{\text{w}}_{\text{kj}}\text{s'(}{\displaystyle \sum _{\text{i}=\text{1}}^{\text{n}}{\text{w}}_{\text{ik}}{\text{x}}_{\text{it}}{\text{)x}}_{\text{it}}\text{]}}}}}$$

(5)

and s’(z) = s(z)(1 - s(z)) is the first derivative of s(z).

Earlier than Rumelhart et al. [1], Werbos [44] gave a different derivation of the BP algorithm. This derivation is more general and mathematically rigorous than the one given by Rumelhart et al. In Werbos’ derivation, the chain rule is expressed in a convenient form by ordered derivatives. Even so, most research and applications of the BP algorithm still referred to the results of Rumelhart et al. due to their milestone publication in 1986 with the PDP research group.

3.2. Radial Basis Function Network

Due to adding hidden nodes and layer(s) and the nonlinearity of the sigmoid activation function of each hidden nodes and/or output nodes in MLPs, the BP algorithm has the possibility to produce complex error surfaces which contain many minima. Since some minima are deeper than others, it is possible that the algorithm will not find a global minima. Instead, the network may fall into local minima, which represent suboptimal solutions. The RBF networks [46] were introduced with the centers of data clustering means, and then the linear solution for the connection weights to the network output directly to a unique local minimum, the global minimum, which shortens network training process greatly.

Figure 2 is a diagram of a one-hidden-layered RBF network structure. The RBF networks also are MFNs. With sufficient larger number of hidden nodes, they are universal approximators as well. The RBF network training also is supervised, involving determination of data clustering centers, c_k (k = 1, 2, …, h_c), where h_c also is the number of the centers to be determined, and network connection weights from the hidden layer to the network outputs, w_kj (k = 1, 2, …, h_c; j = 1, 2, …, m). Similar to BP training, the RBF training is to establish the map between process inputs and desired outputs by minimizing the errors between the desired outputs and the calculated outputs driven from the inputs and network learning. The objective function of error minimization is defined as:

$$\text{E}={\displaystyle \sum _{\text{t}=\text{1}}^{\text{N}}{\displaystyle \sum _{\text{j}=\text{1}}^{\text{m}}{{\text{(d}}_{\text{jt}}-{\text{y}}_{\text{jt}}\text{)}}^{\text{2}}}}$$

(6)

and:

$${\text{y}}_{\text{jt}}={\displaystyle \sum _{\text{k}=\text{1}}^{{\text{h}}_{\text{c}}}{\text{w}}_{\text{kj}}\text{R(}\Vert {\displaystyle \sum _{\text{i}=\text{1}}^{\text{n}}{\text{x}}_{\text{it}}}-{\text{c}}_{\text{k}}\Vert \text{)}}$$

(7)

where R( ) is a RBF function.

A RBF function is any function R that satisfies R(x) = R(||x||). It is a real-valued function that depends on the distance from the origin as R(x) = R(||x||) or from other point or center, c, as R(x - c) = R(||x - c||). The norm ||.|| is usually Euclidean distance.

The linear solution of output layer weights in equation (7) guarantees the objective function (6) to get the global minima and greatly speed up the training process.

RBF networks integrate nonlinear RBF activation functions in hidden layer and linear combination of weights in output layer. In input data clustering, different techniques can be used, such as k-means clustering [60] and fuzzy c-means [61]. There are choices of RBF functions, such as Gaussian RBF, multiquadric RBF, polyharmonic spline, and thin plate spline. Some of the RBFs will bring more parameters to determine in training process, such as Gaussian RBF, R(x - c) = exp(-β||x - c||²), which brings one more parameter, β, to determine.

Figure 2. RBF network structure.

Figure 2. RBF network structure.

3.3. Recurrent and Feedback Networks

A recurrent neural network is a type of ANN where network nodes may have a directed connection. A typical example of this is that either the network’s hidden node activation values or network output values are fed back into the network input nodes. These types of networks are good at characterizing process temporal dynamics. Examples of simple recurrent networks are Elman and Jordan networks [62,63].

MLP networks are good for mapping the static relationship of process input and output: x → y. They have been used for identification of dynamic systems [64,65]. However, they require a large number of input nodes which may result in long computing time and being affected by external noise [66]. Recurrent neural networks have been attracted in the field of dynamic system identification since they can avoid the problems MLP networks have [67,68,69].

In time series analysis, ANNs can establish AR (AutoRegressive) function map between process input and output: {y(t-i) | i = 1, 2, …, p} → y(t). In process modeling and control, ANNs can help establish ARX (AutoRegressive with eXogenous input) function map of process input and output: {y(t - i), x(t - j) | i = 1, 2, …, p; j = 1, 2, …, q} → y(t) where p and q are the orders of time lags of y and x, respectively. Figure 3 shows a network structure to map the dynamic process input/output relationship with the ARX function map. Internally, this network is identical to a MLP. Externally, it feeds back the network output back to the network input with a lagged time shift. This network is named External Recurrent Neural Network (ERNN) [70]. Obviously, training of this network should be different from standard BP. Around the network, y(t - i) is a time-delayed output from the network itself. This network model is determined as a specific form of time-lag recurrent networks [4]. For the time-lag recurrent networks with only one recurrent connection, i.e. i = 1, a number of training algorithms have been proposed [71,72,73]. Among the proposed algorithms, the BP Through Time (BPTT) is able to be modified for multiple recurrent connections, i.e. i > 1 using ordered derivatives from Werbos’s derivation of the BP algorithm [44]. When the external feedback signal y(t) replaces the input of the network, the change of weights will affect y(t + 1) and thus y(t + 1) all the way to y(N) where T is the length of the data. Thus, the gradient computation has to account for this chaining from t = 0 to t = N. The input layer at time t+1 can be considered as a part of the 3^rd layer of the network at time t. When calculating the error of the output layer at time t, it is necessary to calculate the error for the input nodes at time t + 1 up to t + p, all of which are connected to the corresponding output node at time t. In the regular BP algorithm, the error term for the network nodes propagates the required information all the way from t = N back to the current time instant. This is what the BPTT signifies. Based on Werbos’s work [44], a modified BPTT algorithm was derived for the ERNN [74] as described above.

This modified algorithm was used for training over the whole trajectory of the training data. The maximum number of prediction steps is equal to the total number of observations. In controller design, the prediction steps would be much smaller. In order to solve this problem, the network can be trained to do shorter prediction with the modified training algorithm [74]. The modified training algorithm is still BPTT, but at each time instant the error term is propagated from the specified prediction step t + L (L << N) back to the current time instant t. This work resulted in an ERNN-based multiple-step-ahead prediction for process controller design [70]:

$$\begin{array}{l}\text{y(t}+l\text{|t)}=\text{f(y(t}+l\text{-1|t),}\mathrm{...}\text{,y(t}+l\text{-p|t),x(t}+l\text{-1),}\mathrm{...}\text{,x(t}+l\text{-q),W)}\\ \text{y(t}+l\text{|t)}=\{\begin{array}{l}\text{y(t}+l\text{ ) when }l>\text{0}\\ \text{d(t}+l\text{ ) otherwise}\end{array}\\ \mathrm{ l}=\text{1,2,}\mathrm{...}\text{,L}\end{array}$$

(8)

where$\text{y(t}+l\text{|t)}$ is the l-step-ahead predictor of y(t) at the time instant t, f( ) is the approximation of the function map, and W is the set of weights and bias terms in the network.

Figure 3. ERNN network structure.

Figure 3. ERNN network structure.

3.4. Kohonen SOM Network and Unsupervised Training

Unlike the trainings of MLP, RBF and ERNN networks, Kohonen SOM networks are trained using an unsupervised algorithm, by which the input data are self-organized into clusters or classes, i.e. similar input data will activate the same network output node. The networks construct one-dimensional, two-dimensional, and even three-dimensional arrays of output nodes to form self-organized feature maps. Figure 4 is the structure of Kohonen SOM network with a two-dimensional array of output nodes used to form feature maps.

This is a two-layer network. The output nodes are orderly arranged in a two-dimensional grid. Each input is connected to all output nodes through a weight. The network training process has two stages. The first is to generate a coarse mapping, i.e. to create some form of topological ordering on the map of randomly oriented nodes. There may be large changes to the orientation of the nodes on the map, so the gain term or adaption rate, η, needs to be kept high to allow large weight modifications and settle into an approximate mapping quickly. Once a stable coarse map is found, the training process goes into the second stage. At this stage, the nodes within the localized regions of the map are fine-tuned to the network inputs. For this fine-tuning much smaller changes of weights must be made at each output node, so the adaptation rate, η, is reduced as training progresses. The weight updating in training is iteratively operated along with winning output node neighborhood:

$$\begin{array}{l}{\text{Δw}}_{\text{ijk}}=-{\text{η(w}}_{\text{ijk}}{\text{-x}}_{\text{k}}\text{)}\\ \text{ij}\in {\text{NE}}_{\text{ij*}}\\ \text{(i}=\text{1,2,}\mathrm{...}\text{,p;j}=\text{1,2,}\mathrm{...}\text{,q;k}=\text{1,2,}\mathrm{...}\text{,n)}\end{array}$$

(9)

where p and q the first and second dimension of the feature map array, respectively, n is the number of network inputs, and ij* is the location of optimal output node. The optimal node is determined by:

$$\begin{array}{l}\underset{{\text{ij}}^{\text{*}}}{\text{MIN}}{\text{(D}}_{\text{ij}}\text{)}\\ {\text{D}}_{\text{ij}}={\displaystyle \sum _{\text{k}=\text{1}}^{\text{n}}{{\text{(x}}_{\text{k}}-{\text{w}}_{\text{ijk}}\text{)}}^{\text{2}}}\end{array}$$

(10)

Figure 4. Kohonen SOM network with two-dimensional array of output nodes used to form feature maps.

Figure 4. Kohonen SOM network with two-dimensional array of output nodes used to form feature maps.

The neighborhood function NE_ij is set large at very beginning of training, and slowly decreases in size with the progress of the training (Figure 5). The output nodes of the network are activated using a winner-take-all method:

$${\text{y}}_{\text{ij}}=\{\begin{array}{l}\text{1 if ij}=\text{ij*}\\ \text{0 otherwise}\end{array}$$

(11)

That is that if (i,j) is the location of the optimal output node, the output node takes the value of 1; otherwise it takes 0. Accordingly, the weights connected to the neighborhood of the optimal node are updated.

Figure 5. Topological neighborhoods with the progress of training as feature maps are formed.

Figure 5. Topological neighborhoods with the progress of training as feature maps are formed.

4. Advanced Development of ANNs

With the progress of research and applications, ANN technology has been improved and advanced. Research has been done on modification of the standard BP algorithm to speed up algorithm convergence and avoid local minima in MLP network training. Much work has been devoted to preventing over-fitting in MLPs to improve the generalization ability of the networks. ANNs have been enhanced by fusing with other methods such as fuzzy logic and wavelet analysis. SVMs, emerging as a new soft computing technique, bring up a new wave of research and applications of ANNs.

4.1. Standard BP Enhancement

In standard BP training, it is critical to select a learning rate, η, to let the process converge to the true global minimum of the mean square error of the network outputs with a rather rapid speed. A BP training with a too small learning rate will have a noticeable slow progress. One with a too large learning rate will speed up significantly. However, this may result in oscillations around relatively poor solutions. Evidence shows that the use of a momentum term in the BP algorithm can be helpful in speeding the convergence and avoiding local minima (http://www.cs.bham.ac.uk/~pxt/NC/ASSIGNMENT/MICHAEL/momentum.html). The momentum term is defined as a fraction of the previous weight change αΔw^-, and added to equation (2) from the standard BP algorithm [7]:

$$\text{Δw}=-\text{η}\nabla \text{E(w)}+{\text{αΔw}}^{\text{-}}$$

(12)

where α is taken 0≤ α ≤ 0.9.

In this way, in BP training, the momentum is used to stabilize the weight change using a combination of the gradient decreasing term with a fraction of the previous weight change.

With the concept of momentum in BP training, the method has been further developed. Yu et al. [9] developed an adaptive momentum algorithm which can update the momentum coefficient automatically in every iteration step. The result of this adaptive process is equivalent to adding a momentum term to the standard BP algorithm. The momentum coefficient is updated automatically in every iteration. Numerical simulations show that the adaptive momentum algorithm can eliminate possible divergent oscillations during the initial training, and can also accelerate the learning process and result in a lower error when the final convergence is reached. Gerke and Hoyer [10] presented an analysis of fuzzy adaption of training parameters (learning rate and momentum) to accelerate BP learning in MFNs.

There are other studies on BP training acceleration and local minima. Huang et al. [75] investigated the efficiency of the training processes of MFNs using the gradient descent method in the standard BP algorithm and Levenberg-Marquardt method in backpropagation. It was found that in the case of low epoch training (below several thousand epochs) using the gradient descent algorithm, the Levenberg-Marquardt algorithm was less efficient, and in the case of high epoch training (above several thousand epochs) using the gradient descent algorithm, the Levenberg-Marquardt algorithm was more efficient. Jeenbekov and Sarybaeva [11] described properties of various parameters of sigmoid activation function mathematically with their influence on the speed of convergence of the BP training algorithm for MFNs. Wang et al. [12] proposed an improved BP algorithm intended to avoid the local minima problem caused by neuron saturation in the hidden layer. Each training pattern has its own activation functions of neurons in the hidden layer. When the network outputs have not received their desired signals, the activation functions are adapted so as to prevent neurons in the hidden layer from saturating. Bi et al. [13] proposed a modified error function with two terms. By adding one term to the conventional error function, the modified error function can harmonize the update of weights connected to the hidden layer and those connected to the output layer. Thus, it can avoid thelocal minima problem caused by update disharmony between weights connected to the hidden layer and the output layer. Simulations on some benchmark problems and a real classification task have been performed to test the validity of the modified error function. Otair and Salameh [14] proposed an algorithm used for training that depends on a multilayer neural network with a very small learning rate, especially when using a large training set size. It can be applied in a generic manner for any network size that uses a BP algorithm through an optical time (seen time).

4.2. Network Generalization

Although MFNs can be universal approximators with a sufficiently large number of hidden nodes, excessive number of nodes in the hidden layer may endanger the network to become memorized, which may lead to overfit the input variables [76,77,78]. The overfitted networks may fit training data points well but may not be able to well interpolate and extrapolate testing data points. The testing data points can be between the training points or out of the range of the training points. Overfitting is a very important problem in MLPs, and much work has been devoted to preventing overfitting to improve network generalization with techniques such as early stopping, weight decay, and pruning. Figure 6 shows typical error curves of MLP training and testing. With the increase of the hidden nodes, the training error is large at the beginning and then keeps decreasing with a gradual decline of the curve slope. Similarly, the testing error is also larger at the beginning and then keeps decreasing until it reaches a point, h*. Starting from this point, with further increase of the hidden nodes, the testing error increases gradually while the training error continues to decrease. During training, if the minimal point on the testing curve can be found, the overfitting problem may be solved. The training and testing curves with training steps have a similar behavior as shown in Figure 6, which can be used to investigate overfitting, as can happen with too many training steps.

Based on the explanation of Figure 6, during MLP training, the training error and testing error are checked at every step, and the training process terminates as long as the minimal point of testing error is reached. This is so called early stopping method [20]. Further, a three-set approach was proposed [15]. This approach is to divide a data set into three nonoverlapping subsets: one for training to update network weights, one for testing to terminate training to prevent overfitting, and one for validation to evaluate the trained network. This approach gives a reasonable estimate of the network’s generalization ability.

Adding a penalty term, E_p, to the mean square error of the network outputs in the standard BP algorithm, a new objective function is formulated for MLP training:

E_w = E + λE_p

(13)

where λ is a pre-set constant for the penalty term. The training algorithm using this objective function causes network weights to converge to smaller absolute values than they would in the standard case. This method is called weight decay [16,75].The generalization ability of a network depends on the adjustment of the decay constant, λ. With weight decay training, the network can avoid oscillation in the outputs caused by large weights.

Figure 6. MLP error curves vs. number of hidden nodes.

Figure 6. MLP error curves vs. number of hidden nodes.

Network pruning is another method used to prevent overfitting. This method begins with a fully connected network and the network is made smaller by iteratively eliminating least effective nodes in the hidden layer(s) or interconnections between nodes. The pruned network provides a structure that has a greater capacity for generalization [6,18,19].

4.3. Neuro-Fuzzy Systems

Fuzzy logic is a form of multi-valued logic derived from fuzzy set theory [79] to deal with reasoning that is approximate, rather than precise. In contrast to yes/no or 0/1 binary logic, Fuzzy logic provides a set of membership values inclusively between 0 and 1 to indicate the degree of truth. Fuzzy logic can be combined with ANNs to form a learning machine that is able to determine the parameters for a fuzzy system by ANN computing. A fuzzy system comprises of linguistic rules such as IF THEN from prior knowledge. Input and output variables of the system are described linguistically. If the knowledge is incomplete, wrong, or contradictory, the fuzzy system needs to be tuned. Fuzzy system is incapable of learning; it is a mechanism of straight interpretation and implementation. ANNs are capable of learning with a sufficient amount of observed examples for a problem. These observations are used to train the network model. The result can provide data for parameterization of fuzzy rules.

The fuzzy-neural systems have been studied and applied. Takagi and Hayashi [22] proposed a neural-fuzzy reasoning system that is capable of automatic determination of inference rules and adjustment according to the time-variant reasoning environment with the use of NN in fuzzy reasoning. Horikawa et al. [23] presented a fuzzy modeling method using fuzzy neural networks with the BP algorithm. The method can identify the fuzzy model of a nonlinear system automatically. Nie and Linkens [24] approximated reasoning through a BP Neural Network with the aid of fuzzy set theory. An example of multivariable fuzzy control of blood pressure was examined for the underlying principles. Simpson and Jahns [25] introduced the fuzzy min-max function approximation neural network. The function approximation network is realized by modifying the developed fuzzy min-max clustering network [80] to include an output layer that sums and thresholds the hidden layer membership functions. Jang [81] introduced the framework of ANFIS (Adaptive-Network-based Fuzzy Inference System). Jang and Sun [27] reviewed fundamental and advanced developments in neuro-fuzzy systems for modeling and control. They introduced design methods for ANFIS in modeling and control applications.

A fuzzy-neural system can be represented as a MFN. The first layer is for input variables. The middle or hidden layer(s) are for encoding fuzzy rules and fuzzy inference. The last layer is for output variables. The fuzzy inferences are converted as network connection weights. Figure 7 shows the structure of a fuzzy-neural system based on the idea of ANFIS [81].

Figure 7. Structure of a fuzzy-neural system.

Figure 7. Structure of a fuzzy-neural system.

For n input variables {x_i | i = 1, 2, …, n}, suppose that there are m fuzzy if-then rules:

Rule 1: If x₁ is M₁₁, x₂ is M₂₁, …, and x_n is M_n1, then f₁=a₁₁x₁+a₁₂x₂+…+a_1nx_n+r₁

Rule 2: If x₁ is M₁₂, x₂ is M₂₂, …, and x_n is M_n2, then f₂=a₂₁x₁+a₂₂x₂+…+a_2nx_n+r₂

(14)

… … …

Rule m: If x₁ is M_1m, x₂ is M_2m, …, and x_n is M_nm, then f_m=a_m1x₁+a_m2x₂+…+a_mnx_n+r_m

where M_ij is a linguistic label (small, medium, and large) of a membership function, μ_Mij(x), of x_i with the associated node j. The membership function can be in different forms with different parameter sets, such as the generalized bell-shaped function,

$${\text{μ}}_{{\text{M}}_{\text{ij}}}\text{(x)}=\frac{\text{1}}{\text{1}+{\text{[(}\frac{\text{x}-{\text{c}}_{\text{ij}}}{{\text{a}}_{\text{ij}}}{\text{)}}^{\text{2}}\text{]}}^{{\text{b}}_{\text{ij}}}}$$

or the Gaussian function, ${\text{μ}}_{{\text{M}}_{\text{ij}}}\text{(x)}=\text{exp[}-{\text{(}\frac{\text{x}-{\text{b}}_{\text{ij}}}{{\text{a}}_{\text{ij}}}\text{)}}^{\text{2}}\text{]}$. In the layer, each node labeled Π multiplies the incoming membership values and sends the product out to the next layer as connection weights:

$${\text{w}}_{\text{j}}={\text{μ}}_{{\text{M}}_{\text{1j}}}\text{(x)}\times {\text{μ}}_{{\text{M}}_{\text{2j}}}\text{(x)}\times \mathrm{...}\times {\text{μ}}_{{\text{M}}_{\text{nj}}}\text{(x)} \left(\text{j }=\text{ 1},\text{ 2},\dots ,\text{ m}\right)$$

(15)

In the next layer labeled N, the connection weights are normalized:

$${\tilde{\text{w}}}_{\text{j}}=\frac{{\text{w}}_{\text{j}}}{{\displaystyle \sum _{\text{i}=\text{1}}^{\text{m}}{\text{w}}_{\text{i}}}} \left(\text{j }=\text{ 1},\text{ 2},\dots ,\text{ m}\right)$$

(16)

In the next layer the outputs are:

$$\begin{array}{ll}{\text{y}}_{\text{j}}& ={\tilde{\text{w}}}_{\text{j}}{\text{f}}_{\text{j}}\\ & ={\tilde{\text{w}}}_{\text{j}}{\text{(a}}_{\text{j1}}{\text{x}}_{\text{1}}+{\text{a}}_{\text{j2}}{\text{x}}_{\text{2}}+\mathrm{...}+{\text{a}}_{\text{jn}}{\text{x}}_{\text{n}}+{\text{r}}_{\text{j}}\text{)} \left(\text{j }=\text{ 1},\text{ 2},\dots ,\text{ m}\right)\end{array}$$

(17)

In the last node, the node generates the overall output of the network by summing up from all incoming connections:

$$\begin{array}{ll}\text{Y}& ={\displaystyle \sum _{\text{i}=\text{1}}^{\text{m}}{\text{y}}_{\text{i}}}\\ & ={\displaystyle \sum _{\text{i}=\text{1}}^{\text{m}}{\tilde{\text{w}}}_{\text{j}}{\text{f}}_{\text{i}}}\\ & =\frac{{\displaystyle \sum _{\text{i}=\text{1}}^{\text{m}}{\text{w}}_{\text{i}}{\text{f}}_{\text{i}}}}{{\displaystyle \sum _{\text{i}=\text{1}}^{\text{m}}{\text{w}}_{\text{i}}}}\end{array}$$

(18)

The parameters included in the fuzzy and neural network equations above can be updated using the method of gradient descent in the standard BP algorithm. Other methods were suggested also to update the parameters [81].

4.4. Wavelet-Based Neural Networks

As discussed above, the standard BP algorithm may produce problems in training, for example, converge to local optima. Besides, the determination of the network structure in BP training, such as the number of hidden nodes, is still subjective. With the intelligent integration of wavelet transform, ANNs can be able to approximate data with non-linear and non-stationary properties and to conduct self-learning adaptive structure identification and parameter estimation in modeling process.

Wavelets are “small’’ waves that should integrate to zero around the x-axis. They localize functions well and a “mother’’ wavelet can be translated and dilated into a series of wavelet basis functions. The basic idea of wavelets can be traced back to very early in this century. However, the development of the construction of compactly supported orthonormal wavelets [82] and the wavelet-based multiresolution analysis [83] has resulted in extensive research and applications of wavelets in recent years. In comparison to the more widely known Fourier transform, wavelet basis functions are local both in frequency and time while Fourier basis functions are local only in frequency. Wavelet analysis has been widely used in a variety of fields, especially in signal and image processing.

Wavelet analysis is becoming a common tool for analyzing localized variations of power within a time series [84]. By decomposing a time series into time-frequency space, one is able to determine both the dominant modes of variability and how those modes vary in time. Wavelet-based ANNs (WANNs) have been developed and used for function approximation [28,29,30]. WANNs have been further developed for nonlinear time series forecasting [31]. WANNs have a strong ability of function approximation for optimal time series pattern recognition and their modeling algorithms are different from conventional BP neural network training algorithms. Therefore, WANNs have the potential to overcome the problems that conventional ANNs had.

WANN is a MFN model with wavelet analysis. It replaces the sigmoid function in the MLP with nonlinear wavelet basis function, $\varphi \text{(}\frac{\text{t}-\text{b}}{\text{a}}\text{)}$, where a and b are the scale factor and translation factor of the wavelet function, respectively. In this model, nonlinear time series is expressed as a linear combination of selected nonlinear wavelet basis so that a finite number of wavelet series items are used to approximate time series function, i.e.

$$\text{y(t)}={\displaystyle \sum _{\text{i}=\text{1}}^{\text{h}}{\text{w}}_{\text{i}}\varphi (\frac{t-b_i}{a_i})}$$

(19)

Figure 8 shows the structure of the WANN. This is a single hidden-layered feedforward network. It only has one input node and one output node. The objective of the modeling task is to determine the network parameters, w_i, a_i, b_i and h, to achieve an optimal fitting between the desired output series d(t) and the calculated output series y(t):

$${\text{J}}_h=\frac{1}{2}{\displaystyle \sum _{\text{t}=\text{1}}^{\text{N}}{\text{[d(t)-y(t)]}}^{\text{2}}}$$

(20)

Figure 8. Structure of WANN.

Figure 8. Structure of WANN.

Unlike in the standard BP algorithm, in WANN training determination of h is a step-wise process. Preset a toleranceε for nonlinear time series fitting error. Starting h from 1 and calculate J₁. If J₁ < ε, the optimal h, h* = 1; otherwise, set h = 2 and calculate J₂. If J₂<ε, h* = 2; otherwise, keep on going until h* is found with J_h* < ε.

Using Newton’s method it can be derived that the parameter updating equations can be as follows in training:

$$\mathrm{\Delta }{\text{w}}_{\text{i}}={\displaystyle \sum _{\text{t}=\text{1}}^{\text{N}}\text{[d(t)-y(t)]}\varphi \text{(}\frac{{\text{t-b}}_{\text{i}}}{{\text{a}}_{\text{i}}})}$$

$$\mathrm{\Delta }{\text{a}}_{\text{i}}={\displaystyle \sum _{\text{t}=\text{1}}^{\text{N}}\text{[d(t)-y(t)]}\frac{{\text{w}}_{\text{i}}}{{\text{a}}_{\text{i}}^{\text{2}}}(t-b_i)\varphi \text{'}\text{(}\frac{{\text{t-b}}_{\text{i}}}{{\text{a}}_{\text{i}}})}$$

(21)

$$\mathrm{\Delta }{\text{b}}_{\text{i}}=-{\displaystyle \sum _{\text{t}=\text{1}}^{\text{N}}\text{[d(t)-y(t)]}\frac{{\text{w}}_{\text{i}}}{{\text{a}}_{\text{i}}}\varphi \text{'}\text{(}\frac{{\text{t-b}}_{\text{i}}}{{\text{a}}_{\text{i}}})}$$

The forms of $\varphi \text{(t)}$ and $\varphi \text{'}\text{(t)}$depend on the selected wavelet function. For example, for Morlet mother wavelet, φ(t) = cos(1.75t)exp(–t²/2). Then its first derivative is φ′(t) = –1.75sin(1.75t)exp(–t²/2) –tcons(1.75t)exp(–t²/2).

4.5. SVMs

SVMs were proposed by Vapnik [85]. Based on the description of Burges [32] and Cristianini and Taylor [33], SVMs are a set of supervised generalized linear classifiers that have often been found to provide higher classification accuracies than widely used MLP classifiers. The growing interest in SVMs is shown by their successful application in many areas such as automobile [86], time series analysis [87,88], medicine [89], pattern recognition [90], geophysics [91,92], computational biology [93], and control engineering [94]. The interest resulted from SVMs’ intrinsic effectiveness with respect to traditional classifiers. This effectiveness results in high classification accuracies and very good generalization capabilities. In architecture design, SVMs requires limited effort to configure few control parameters compared with traditional classifiers such as MLPs. Also, based on the universal approximation capability of their standard MFN counterparts, the approximation capability of SVMs was investigated [51]. It was shown that an SVM with polynomial kernel of degree n-1, which is trained on a training set of size n can approximate the n training points up to any accuracy.

A SVM performs classification through mapping input vectors into a higher-dimensional space and constructing a hyperplane that optimally separates the data in the higher-dimensional space. A SVM model using sigmoid kernel function is equivalent to a two-layer perceptron neural network. However, the operation of the SVM is different. In training, a SVM implements a simple linear mapping or linear classifier together with a prior fixed nonlinear mapping in order to make data separable. This implementation allows training to not suffer from the problem of local minima, and to focus on function optimization with respect to its generalization ability. These advantages often lead to better results for SVMs compared to MFNs.

The goal of SVM modeling is to find the optimal hyperplane that separates clusters of vectors, a set of features, such that data points with one class of the variable are on one side of the plane and ones with the other class are on the other side of the plane. The vectors near the hyperplane are the support vectors. Let’s look at the 2-dimensional case as an example. Assume that for classification, the data are from a categorical variable with two classes, and there are two prediction variables, X and Y, with continuous values. The data points using the value of X on the X axis and Y on the Y axis may be plotted as Figure 9.

Figure 9. Margin between support vectors in two-dimension.

Figure 9. Margin between support vectors in two-dimension.

In the plot it can be seen what classes each (X,Y) point belongs to. In this example, the data points with one class are in the lower left corner of the plot and the data points with the other classes are in the upper right corner. The SVM analysis is find a one-dimensional hyperplane, i.e. a line, to separate the data points based on their target classes. Obviously, there are an infinite number of possible such lines. Two of them are shown in the plot. The task of SVM now is to determine which line is optimal. For this optimization two parallel (dashed) lines are constructed, one on each side of the separating line. The two lines are pushed up against the two sets of data points belonged to two classes. They label the distance between the separating line and the closest vectors to the line. The two lines are used to calculate the margin, which is defined as the distance between them. The SVM analysis is to find the line (hyperplane in general) that makes the margin between the support vectors maximized. In Figure 9, the line A is superior to the line B. In this sense, SVM classifiers are also known as maximum margin classifiers.

Mathematically, suppose having a training data pairs {(x_i, c_i) | i= 1, 2, …, N} with x_i ∈ Rⁿ and c_i∈{1, -1}, indicate what class the data point x_i belongs to. In a simple case, a hyperplane can be defined as F(x) = w^tx + w₀ where w is the adaptable weight vector and w₀ is the bias term, and T is the vector transpose operator. In training, the data points x_i are projected into the higher-dimensional space, and the classification is conducted through sign[f(x)] where f(x) is the estimate of F(x). The SVM algorithm searches for a hyperplane f(x) that maximizes the margin between the two sets of data points in class 1 and -1.

Further, the margin maximum problem can be solved in any high-dimensional space by introducing a kernel function [95,96]. With a nonlinear kernel function, the low-dimensional input space is nonlinearly transformed into a high-dimensional feature space such that the probability that the feature space is linear separable becomes higher. Theoretically, the kernel function is able to implicitly map the input space into an arbitrary high-dimensional feature space that can be linearly separable even if the input space may not be linearly separable. Some commonly used kernel functions are polynomial, Gaussian, Sigmoid, and RBF.

In general, SVMs’s training is to solve the following optimization problem:

$$\underset{{\text{w, w}}_{\text{0}}\text{,ξ}}{\text{MIN}}\text{(}\frac{\text{1}}{\text{2}}{\text{w}}^{\text{T}}\text{w}+\text{λ}{\displaystyle \sum _{\text{i}=\text{1}}^{\text{N}}{\text{ξ}}_{\text{i}}\text{)}}$$

(22)

$$\text{s.t.} {\text{c}}_{\text{i}}{\text{[w}}^{\text{T}}\phi {\text{(x}}_{\text{i}}\text{)}+{\text{w}}_{\text{0}}\text{]}\ge \text{1}-{\text{ξ}}_{\text{i}}$$

where λ is a user defined positive constant, a penalty parameter of the error term, ξ_i (≥0) is the slack variable to measure the degree of misclassification of x_i, and φ(x) is the function to map data points x_i from the input space to a higher-dimensional feature space.

With equation (22), data points xi are mapped into a higher-dimensional space by the function of φ(x). Then a linearly separable hyperplane (${\text{w}}^{\text{T}}\phi {\text{(x}}_{\text{i}}\text{)}+{\text{w}}_{\text{0}}$) is found with the maximal margin in the higher-dimensional space. Further, the mapping function φ(x) is determined with a specified kernel function K(x_i,x_j):

K(x_i,x_j) = φ(x_i) φ(x_j)

(23)

If RBF is chosen as the kernel function, then K(x_i, x_j) = exp(-γ||x_i - x_j||²) (γ > 0). The RBF kernel is good at handling nonlinear classification for SVMs.

5. Limitations of ANNs

As described above, ANNs are powerful computing techniques, which are designed to mimic human learning processes by establishing linkages between process input and output data. These techniques have been widely applied with advanced development with their unique advantages, such as no underlying assumption about the distribution of data, arbitrary decision boundary capabilities, universal approximation capabilities, easy adaptation to different types and structures of data, ability to fuzzy output values to enhance classification, and good generalization capabilities. However, ANNs have some disadvantages in common, which need to be considered in practical application:

• Black box
  ANNs are black box in nature. Therefore, if the problem is to find the output response to the input such as system identification [96], ANNs can be a good fit. However, if the problem is to specifically identify causal relationship between input and output, ANNs have only limited ability to do it compared with conventional statistical methods.
• Long computing time
  ANN training needs to iteratively determine network structure and update connection weights. This is a time-consuming process. With a typical personal computer or work station, the BP algorithm will take a lot of memory and may take hours, days and even longer before the network converges to the optimal point with minimum mean square error. Conventional statistical regression with the same set of data, on the contrary, may generate results in seconds using the same computer.
• Overfitting
  With too much training time, too many hidden nodes, or too large training data set, the network will overfit the data and have a poor generalization, i.e. high accuracy for training data set but poor interpolation of testing data. This is an important issue being investigated in ANN research and applications as described above.

Also, SVMs were originally developed to solve binary classification problems. How to effectively extend it for multiclass classification is still an ongoing research issue. Typically, a multiple class classifier can be constructed by combining several binary classifiers. Further, all classes can be considered at once. Hsu and Lin [98] gave decomposition implementations for two such "all-together" methods. They compared the performance with three methods based on binary classifications: one-against-all, one-against-one, and Directed Acyclic Graph SVM (DAGSVM). The experiments indicated that the one-against-one and DAG methods are more suitable for practical use than the other methods.

6. ANN Applications in Agricultural and Biological Engineering

ANNs have the largest body of applications in agricultural and biological engineering compared to other soft computing techniques such as fuzzy logic and genetic algorithms. In summary of related 348 papers and reports, ANNs have been applied in solving problems in food quality and safety (35.3%), crop (22.7%), soil and water (14.4%), precision agriculture (6.6%), animal management (5.2%), post harvest (2.6%), food processing (2.3%), greenhouse control (2%), agricultural vehicle control (1.2%), agricultural machinery (1.2%), agricultural pollution (1.2%), agricultural biology (1.2%), ecology and natural resources (1.4%), agricultural robotics (0.3%), chemical application (0.3%), and others (2.3%) such as bioenergy and agricultural facilities (Figure 10). These ANN applications have been created mainly through classification (45.1%), modeling and prediction (44%), control (4%), and simulation (2.6%), parameter estimation (2%), detection (1.2%), data clustering (0.6%), optimization (0.3%) and data fusion (0.3%) as well (Figure 11).

Early applications include modeling sensory color quality of tomato and peach [99], investigation of combined effects of CO₂ and sucrose on the growth of alfalfa cuttings using a Kalman filter neural network [100], classification of apple surface features [101], corn kernel breakage classification [102], and machine vision inspection of potatoes [103].

Recently, the BP ANN method was used in regression for modeling the correlation between crop yield and 10 yield components of chickpea [104]. A BP ANN was employed and trained with extracted features from the data collected by electronic nose to identify the wheat age [105]. BP neural network and generalized regression neural network techniques were developed and compared to model source gas and PM10 concentration and emission rate (GPCER) generated and emitted from swine deep-pit finishing buildings as affected by time of day, season, ventilation rates, animal growth cycles, in-house manure storage levels, and weather conditions [106]. MLP models were developed and used to predict or classify water stress in samples of Sunagoke moss using the texture features extracted from color co-occurrence matrix (CCM) and gray-level co-occurrence matrix (GLCM) methods [107]. A BP ANN was developed for predicting fungal infections using input from principal components extracted from sample visible and near-infrared (NIR) reflectance spectroscopy for early detection of Botrytis cinerea on eggplant leaves [108].

Figure 10. Application area distribution of ANNs in agricultural and biological engineering.

Figure 10. Application area distribution of ANNs in agricultural and biological engineering.

Figure 11. Technical area distribution of ANNs in agricultural and biological engineering.

Figure 11. Technical area distribution of ANNs in agricultural and biological engineering.

Similar to applications in general engineering fields, most of the work in agricultural and biological engineering has been accomplished by multilayer feedforward ANN trained by the famous BP algorithm, which was inspired by the work of Rumelhart et al. in 1986. Among the 348 collected papers and reports, besides 83 of them in which the ANNs’ structure and training algorithm were not explicitly stated, 210 of them (60%) were based on MFNs trained by the BP algorithm. Twelve of them (3.5%) were based on PNN (Probabilistic Neural Network). Eleven of them (3%) were based on MLP using different training algorithms such as Levenberg-Marquardt optimization procedure, Broyden-Fletcher-Goldfarb-Shanno (BFGS) optimization procedure, and genetic algorithm. Seven of them (2%) were based on Kohonen SOM. Six of them (2%) were based upon other unsupervised training algorithms such as fuzzy art, ART2 and Auto- Associative network. Four of them (1%) were based on RBF networks. Three of them (0.8%) were based on LVQ (Learning Vector Quantization). Two of them (0.6%) were based on GRNN (Generalized Regression Neural Network). Ten of them (3%) were based on other network structures such as Counter-Propagation (CP) and Adaptive Logic Network (ALN). Figure 12 shows the ANN method distribution of applications in agricultural and biological engineering.

Fuzzy-neural systems have been developed and used in agricultural and biological engineering. Table 1 listed papers and reports on fuzzy-neural methods and systems in applications in agricultural and biological engineering. Linko et al. [109] applied neural network modeling for fuzzy food extrusion control. Kim and Cho [110] used the BP algorithm in training ANN models for prediction of volume, browning and bread temperatures. The outputs of the ANN models were used for fuzzy control simulation of the oven used in the baking process. Morimoto et al. [111] proposed and then applied a new fuzzy control technique, which efficiently selects optimal membership functions and control rules by using neural networks and genetic algorithms to the control of relative humidity in a fruit-storage house. The response of relative humidity, as affected by ventilation, was first identified using neural networks, and then optimal membership functions and control rules were sought through simulation of the identified model using GAs. Odhiambo et al. [112] developed a strategy consisting of fusing the fuzzy logic and ANN on a conceptual and structural basis for an easy and efficient means of tuning fuzzy ET (Evapotranspiration) models. The neural component provided supervised learning capabilities for optimizing the membership functions and extracting fuzzy rules from a set of input–output examples selected to cover the data hyperspace of the sites evaluated.

Figure 12. ANN method distribution of applications in agricultural and biological engineering.

Figure 12. ANN method distribution of applications in agricultural and biological engineering.

Table 1. Papers and reports on fusion of fuzzy logic and ANNs in agricultural and biological engineering.

Year	Author	Fusion Type	Application Area
1992	Linko et al. [109]	ANN modeling for fuzzy control	Extrusion control
1997	Kim and Cho [110]	ANN modeling plus fuzzy control simulation	Bread baking process control
1997	Morimoto et al. [111]	ANN modeling plus GA parameter optimization for fuzzy control	Fruit storage control
2001	Odhiambo et al. [112]	Conceptual and structural fusion of fuzzy logic and ANN	ET model optimization
2003	Andriyas et al. [118]	FCM clustering for RBF training	Prediction of the performance of vegetative filter strips
2003	Chtioui et al. [113]	SOM with FCM clustering	Color image segmentation of edible beans
2003	Lee et al. [119]	ANFIS modeling	Prediction of multiple soil properties
2003	Neto et al. [120]	ANFIS classification	Adaptive image segmentation for weed detection
2004	Odhiambo et al. [115]	Fuzzy-Neural Netwok unsupervised classification	Classification of soils
2004	Meyer et al. [114]	ANFIS classification	Classification of uniform plant, soil, and residue color images
2004	Goel et al. [121]	Fuzzy c-means clustering for RBF training	Prediction of sediment and phosphorous movement through vegetative filter strips
2006	Hancock and Zhang [115]	ANFIS classification	Hydraulic vane pump health classification
2007	Xiang and Tian [117]	ANN modeling plus ANFIS training of fuzzy logic controller	Outdoor automatic camera parameter control

Table 1. Papers and reports on fusion of fuzzy logic and ANNs in agricultural and biological engineering.

Year	Author	Fusion Type	Application Area
1992	Linko et al. [109]	ANN modeling for fuzzy control	Extrusion control
1997	Kim and Cho [110]	ANN modeling plus fuzzy control simulation	Bread baking process control
1997	Morimoto et al. [111]	ANN modeling plus GA parameter optimization for fuzzy control	Fruit storage control
2001	Odhiambo et al. [112]	Conceptual and structural fusion of fuzzy logic and ANN	ET model optimization
2003	Andriyas et al. [118]	FCM clustering for RBF training	Prediction of the performance of vegetative filter strips
2003	Chtioui et al. [113]	SOM with FCM clustering	Color image segmentation of edible beans
2003	Lee et al. [119]	ANFIS modeling	Prediction of multiple soil properties
2003	Neto et al. [120]	ANFIS classification	Adaptive image segmentation for weed detection
2004	Odhiambo et al. [115]	Fuzzy-Neural Netwok unsupervised classification	Classification of soils
2004	Meyer et al. [114]	ANFIS classification	Classification of uniform plant, soil, and residue color images
2004	Goel et al. [121]	Fuzzy c-means clustering for RBF training	Prediction of sediment and phosphorous movement through vegetative filter strips
2006	Hancock and Zhang [115]	ANFIS classification	Hydraulic vane pump health classification
2007	Xiang and Tian [117]	ANN modeling plus ANFIS training of fuzzy logic controller	Outdoor automatic camera parameter control

Chtioui et al. [113] developed a novel segmentation approach that partitions color images into two uniform regions is described. This unsupervised procedure is based on a SOM neural network and fuzzy c–means clustering (FCM). The SOM allows the mapping of a color image related to edible beans into a consistent two–dimensional table through a non–linear projection. Fuzzy clustering is then applied to the Kohonen map to determine the two cluster centers. Meyer et al. [114] conducted a digital camera operation study for classifying uniform images of grass, bare soil, corn stalks residue, wheat straw residue, and a barium sulfate reference panel, based on color. The classifications were conducted with fuzzy inference systems, built with subtractive clustering, an ANFIS. Odhiambo et al. [115] applied a fuzzy-neural network (F-NN) classifier for unsupervised clustering and classification of soil profiles using ground-penetrating radar (GPR) imagery. Hancock and Zhang [116] developed an on-line hydraulic vane pump fault detection system. This fault detection system decomposed vertical pump vibration signals using wavelet packet analysis. Packets containing signal features distinguishing normal and failed pump operation were entered into an ANFIS for pump health classification. Xiang and Tian [117] developed an artificially intelligent controller based on an ANN and an ANFIS for implementing controller to automatically adjust multispectral camera parameter to compensate for changing natural lighting conditions and to acquire white-balanced images.

The interest of SVMs in agricultural and biological engineering is steadily growing (Figure 13). The figure indicates that although the number of peer reviewed publications on this topic did not increase steadily over the past few years, a trend exists for total number of papers and report to increase in the past few years and to be able to project this trend in the next few years. The successful implementations of SVMs in agricultural and biological engineering are listed in Table 2, which includes classification of intact and cracked eggs [122], classification of forest data cover types [123], classification of meat with small data set [124], black walnut shell and meat classification using hyperspectral fluorescence imaging [125], classification to differentiate individual fungal infected and healthy wheat kernels [126], classification for weed and nitrogen stress detection in corn [127], discrimination for screening of compound feeds using NIR hyperspectral data [128], identification of tea varieties by computer vision [129], discrimination of wheat classes with NIR spectroscopy [130], detection of underdeveloped hazenuts from fully developed nuts by impact acoustics [131], classification of electronic nose data [132], classification of modified starches [133], classification of milk with an electronic nose [134], and recognition of plant disease [135]. In these applications, Zhang et al. [126] fused SVM with kernel of RBF neural network. Pardo and Sberveglieri [132] implemented SVM with RBF kernel. Brudzewski et al. [134] applied SVM neural network with one hidden layer of non-linear neurons, one-output linear neuron and specialized learning procedure leading to the global minimum of the error function and excellent generalization ability of the trained network.

Figure 13. Publications on applications of SVMs in agricultural and biological engineering.

Figure 13. Publications on applications of SVMs in agricultural and biological engineering.

Table 2. Papers and reports on applications of SVMs in agricultural and biological engineering.

Year	Author	Application Method	Application Area
2003	Fletcher and Kong [136]	SVM classification	Classifying feature vectors and decide whether each pixel in hyperspectral fluorescence images of poultry carcasses falls in normal or skin tumor categories
2004	Brudzewski et al. [134]	SVM neural network classification	Classification of milk by an electronic nose
2004	Tian et al. [135]	SVM classification	Classification for recognition of plant disease
2005	Pardo and Sberveglieri [132]	SVM with RBF kernel of RBF	Classification of electronic nose data
2005	Pierna et al. [133]	SVM classification	Classification of modified starches by Fourier transform infrared spectroscopy
2006	Chen et al. [129]	SVM classification	Identification of tea varieties by computer vision
2006	Karimi et al. [127]	SVM classification	Classification for weed and nitrogen stress detection in corn
2006	Onaran et al. [131]	SVM classification	Detection of underdeveloped hazenuts from fully developed nuts by impact acoustics
2006	Pierna et al. [128]	SVM classification	Discrimination of screening of compound feeds using NIR hyperspectral data
2006	Wang and Paliwal [130]	Least-Squares SVM classification	Discrimination of wheat classes with NIR spectroscopy
2007	Jiang et al. [125]	Gaussian kernel based SVM classification	Black walnut shell and meat classification using hyperspectral fluorescence imaging
2007	Oommen et al. [137]	SVM modeling and prediction	Simulation of daily, weekly, and monthly runoff and sediment yield fron a watershed
2007	Zhang et al. [126]	Multi-class SVM with kernel of RBF neural network	Classification to differentiate individual fungal infected and healthy wheat kernels.
2008	Fu et al. [138]	Least-Squares SVM modeling and prediction	Quantification of vitamin C content in kiwifruit using NIR spectroscopy
2008	Khot et al. [124]	SVM classification	Classification of meat with small data set
2008	Kovacs et al. [139]	SVM modeling and prediction	Prediction of different concentration classes of instant coffee with electronic tongue measurements
2008	Peng and Wang [140]	Least-Squares SVM modeling and prediction	Prediction of pork meat total viable bacteria count with hyperspectral imaging
2008	Sun et al. [106]	SVM modeling and prediction	On-line assessing internal quality of pears using visible/NIR transmission
2008	Trebar and Steele [123]	SVM classification	Classification of forest data cover types
2008	Yu et al. [9]	Least-Squares SVM modeling and prediction	Rice wine composition prediction by visible/NIR spectroscopy
2009	Deng et al. [122]	SVM classification	classification of intact and cracked eggs

Table 2. Papers and reports on applications of SVMs in agricultural and biological engineering.

Year	Author	Application Method	Application Area
2003	Fletcher and Kong [136]	SVM classification	Classifying feature vectors and decide whether each pixel in hyperspectral fluorescence images of poultry carcasses falls in normal or skin tumor categories
2004	Brudzewski et al. [134]	SVM neural network classification	Classification of milk by an electronic nose
2004	Tian et al. [135]	SVM classification	Classification for recognition of plant disease
2005	Pardo and Sberveglieri [132]	SVM with RBF kernel of RBF	Classification of electronic nose data
2005	Pierna et al. [133]	SVM classification	Classification of modified starches by Fourier transform infrared spectroscopy
2006	Chen et al. [129]	SVM classification	Identification of tea varieties by computer vision
2006	Karimi et al. [127]	SVM classification	Classification for weed and nitrogen stress detection in corn
2006	Onaran et al. [131]	SVM classification	Detection of underdeveloped hazenuts from fully developed nuts by impact acoustics
2006	Pierna et al. [128]	SVM classification	Discrimination of screening of compound feeds using NIR hyperspectral data
2006	Wang and Paliwal [130]	Least-Squares SVM classification	Discrimination of wheat classes with NIR spectroscopy
2007	Jiang et al. [125]	Gaussian kernel based SVM classification	Black walnut shell and meat classification using hyperspectral fluorescence imaging
2007	Oommen et al. [137]	SVM modeling and prediction	Simulation of daily, weekly, and monthly runoff and sediment yield fron a watershed
2007	Zhang et al. [126]	Multi-class SVM with kernel of RBF neural network	Classification to differentiate individual fungal infected and healthy wheat kernels.
2008	Fu et al. [138]	Least-Squares SVM modeling and prediction	Quantification of vitamin C content in kiwifruit using NIR spectroscopy
2008	Khot et al. [124]	SVM classification	Classification of meat with small data set
2008	Kovacs et al. [139]	SVM modeling and prediction	Prediction of different concentration classes of instant coffee with electronic tongue measurements
2008	Peng and Wang [140]	Least-Squares SVM modeling and prediction	Prediction of pork meat total viable bacteria count with hyperspectral imaging
2008	Sun et al. [106]	SVM modeling and prediction	On-line assessing internal quality of pears using visible/NIR transmission
2008	Trebar and Steele [123]	SVM classification	Classification of forest data cover types
2008	Yu et al. [9]	Least-Squares SVM modeling and prediction	Rice wine composition prediction by visible/NIR spectroscopy
2009	Deng et al. [122]	SVM classification	classification of intact and cracked eggs

7. Conclusion and Future Directions

As described above, ANNs have their advantages and have been used as a powerful tool in solving problems in scientific research and engineering applications. ANNs have their own limitations to restrict them as a substitute of traditional methods such as statistical regression, pattern recognition, and time series analysis. In the next decade with advanced development of computer power, ANNs will continue to develop new applications in various fields, including agricultural and biological engineering. As a powerful alternative to conventional methods, ANNs will be studied more to develop approaches to overcoming problems of ANNs in general or in a specific research area. For effective research and development, the guidelines may be generated for predetermining best ANN structure and training algorithms for a given problem. For process statistical modeling, additional research topics may be to establish generic procedures to include significant variables to and exclude non-significant ones from ANN models and to add confidence limits on the output predictions and parameter estimations. These works will let ANNs inherit advantages from conventional statistics.

The fusion of ANNs with other advanced computing methods will continue. These methods will include but are not limited to fuzzy logic, genetic algorithms, decision tree, and wavelet analysis. The fusion will allow the system to inherit the advantages of both of the paradigms and avoid the drawbacks.

SVMs are an alternative learning method for polynomial, RBF and MLP classifiers in which the weights of the network are found by solving a quadratic programming problem with linear constraints, rather than by solving a non-convex, unconstrained minimization problem as in standard neural network training. With the advantages of SVMs over ANNs and the growing interests of SVMs, it can be expected that in the next decade SVMs will be more actively used in various fields, including agricultural and biological engineering, although the results were not already as expected [130].

In food science and engineering, soil and water relationship for crop management, and decision support for precision agriculture, more applications of ANNs and SVMs will be expected. These techniques will be applied standalone or fusion with other soft and hard techniques. Areas of study can involve classification and prediction of food quality and safety, classification for agricultural soil spatial distribution, water resource optimization for irrigation planning, detection and classification of crop stress and pests (weeds, insects and diseases) detection, analysis of remote sensing imagery, study of crop and yield, and field prescriptions for variable rate chemical application.