Application of Neural Networks in Distribution of the Load in Cluster-Based Web Systems

Abstract

Cloud computing systems revolutionized the Internet, and web systems in particular. Quality of service is the basis of resource configuration management in the cloud. Load balancing mechanisms are implemented in order to reduce costs and increase the quality of service. The usage of those methods with adaptive intelligent algorithms can deliver the highest quality of service. In this article, the method of load distribution using neural networks to estimate service times is presented. The discussed and conducted research and experiments include many approaches, among others, application of a single artificial neuron, different structures of the neural networks, and different inputs for the networks. The results of the experiments let us choose a solution that enables effective load distribution in the cloud. The best solution is also compared with other intelligent approaches and distribution methods often used in production systems.

1. Introduction

The Internet is the primary medium that provides information; business services; telecommunication services; entertainment; and, during the COVID-19 pandemic, also other services, such as education and group meetings. To operate, internet services use servers based on web technologies.

In the past, companies providing web services built and maintained their own server rooms. Cooperating servers that formed web clusters were used to maintain more demanding services. Nowadays, more sophisticated approaches are used, namely, cloud computing. It is a new computing paradigm, according to which companies rent computer system resources on demand, without direct active management of it. Cloud computing systems enable access to a shared pool of resources such as servers, storage, and applications [1,2]. Organization of resources improves the performance, utilization of resources, and energy consumption management [3]. The popularity of the cloud attracts a variety of providers that offer a wide range of cloud-based services to users.

Cloud system belonging to one company is physically divided into regions (server rooms) placed in different geographical locations. The regions themselves are also divided into availability zones—physically separated parts of the regions with independent infrastructure and resources [4]. Availability zones contain different resources, including servers with efficient processors.

Quality of service (QoS) is the basis of cloud service and resource configuration management [5]. Load balancing mechanisms are implemented in order to reduce costs and increase quality of service. Such mechanisms are widely used in cluster-based systems. Effective distribution of load is still an open problem in cloud computing. New algorithms, methods, and architectures meeting clients’ demands still need to be developed.

In our previous work, we proposed many methods and solutions for load sharing and load distribution in cloud-based web systems [6,7,8,9]. Most of the solutions use a neuro-fuzzy approach to estimate the response time on different resources in the cloud. In this article, we present a new method of load distribution using artificial neurons and neural networks to estimate service times in the web system. The proposed method enables us to minimize the response time for every single HTTP request. While conducting the research, we tried many approaches including the application of a single artificial neuron, different structures of the neural networks, and different inputs for the networks. The experiments made it possible to choose an adequate solution for load distribution in the cloud. This enabled us to make a new design of load balancer designated for cluster-based web systems. The results obtained in the simulation experiments indicate that it is justified to continue the research in the environment of the real cloud computing system.

The rest of the article is composed as follows. In Section 2, the related work is presented with the motivation and problem to solve. Section 3 introduces a description of load balancer construction. In Section 4, the testbed is given. Section 5 contains a description of different neural networks used in the load balancer and the results obtained. Section 6 includes conclusions and directions for further research.

2. Related Work and Motivations

Cloud computing, like many other computing technologies, uses distributed systems in order to meet a very high demand for computing power. The load balancing methods [10] are used to distribute tasks and loads between nodes in the system. These methods help to improve resource utilization, shorten response times, and enable flexible scalability, which is an essential part of cloud computing [11,12].

The load balancing mechanisms used in internet clouds can be divided into three main categories [10,13,14,15,16]: static, dynamic, and adaptive. In static load balancing, tasks are performed with the usage of deterministic or probabilistic algorithms that do not take into account the current state of the system. Simple strategies have been popular since the beginning of cluster-based internet systems and they are still being refined. An example of such a strategy is Round Robin, which assigns incoming HTTP requests to successive servers. Xu Zongyu and Wang Xingxuan created a slightly more intelligent version that takes into account the load on the web server [17]. Stochastic strategies are also still being developed, and they are becoming more and more sophisticated [18,19,20].

In the dynamic approach, decisions are made on the basis of the current state of the system. The most popular dynamic load balancing algorithm in AWS (Amazon Web Services) [21] is Least Load, which assigns HTTP requests to nodes with the lowest value of the selected load measure.

The adaptive approach is the most complex. Decisions are made not only on the basis of the state of the system. As the environment changes, the operating strategy may also change. [22]. Most adaptation strategies use technics commonly called “intelligent approaches”. Many specialists claim that only this type of strategy can effectively ensure an acceptable access time to WWW content under the conditions of typical WWW traffic, characterized by self-similarity and burstiness [23,24,25,26]. Among many artificial intelligence techniques, the Ant Colony Optimization (ACO) algorithm [27], proposed by Kumar Nishant et al., can be distinguished. In the algorithm, the generated ants move along the width and length of the cloud network in such a way that they know the location of both underloaded and overloaded nodes. Another interesting method of distributing HTTP requests using strategies based on natural phenomena is the artificial bee colony (ABC) [25]. It uses a decision mechanism that mimics the behavior of a bee family. Another solution based on heuristic algorithms is the particle swarm optimization (PSO) strategy for scheduling requests to individual cloud components [28].

Artificial neural networks have also been used in adaptive load distribution systems [29,30,31]. A good example of an energy-sensitive solution is provided by Sallami et al. [32].

Over the past few years, there have been many publications on machine learning as applied to fair distribution of tasks in parallel and distributed systems. For example, Xu et al. [33] used reinforcement learning to maximize computational performance in distributed memory systems during particle motion simulation. Oikawa et al. [34] used machine learning for load balancing in distributed scientific computing. Chen et al. [35] used a multilayer perceptron perceptron to balance Linux workloads in accessing lower-level hardware resources. Load balancing in the cloud is also a subject of ongoing research—artificial neural networks including deep neural networks are often used in them [36,37]. The solution we present differs from the others in that the load balancing task has been decomposed into many parts—for each server and for each type of request. In each of these parts, the neural network does only one thing—it predicts the server’s response time.

Another group of intelligent adaptive approaches using fuzzy neural models was proposed by Zatwarnicki. These are solutions enabling global distribution between server rooms located in different geographic locations, e.g., GARD [38] and GARDIB [7], and also local approaches such as FARD [39] and FNRD [6] strategies. All systems presented in the solutions work in a way that minimizes the response time for each individual HTTP request. Our latest work [8,9,40,41] proposed a two-layer cloud architecture.

The results obtained showed significantly better performance for intelligent switches than for nonintelligent strategies.

Figure 1 shows a typical architecture of cluster-based web system with load balancer and many web servers. This architecture was used while conducting the research in this article.

Taking into account the results of the latest research [40,41], we propose the design of a new load balancer that distributes the HTTP request among web servers placed in one location (one region) in this article. Furthermore, several solutions for request response time prediction using artificial neural networks were tested and applied in Load Balancer.

The detailed description of the proposed approach is presented in the subsequent sections.

3. Load Balancer

The primary purpose of an intelligent load balancer is to redirect incoming HTTP requests $r_i, i=1,\dots , I$ to web servers that are able to service requests in the shortest time. It can be expressed as in Equation (1):

$$z_i=\underset{w}{\min }\left\{tp\begin{array}{c}w\\ i\end{array}:w\in \left\{1,2,\dots ,W\right\}\right\}$$

(1)

where

$z\begin{array}{c} \\ i\end{array}$—the chosen executor out of $W$ executors;

$tp\begin{array}{c}w\\ i\end{array}$—an estimated response time to request $r_i$ for the $w$-th server;

$i$—the index of the request.

The load balancer (Figure 2) consists of the following modules: classification module ($CM$), measurement module ($MM$), decision module ($DM$), redirection module ($RM$), and executor models ($E\begin{array}{c}w\\ \end{array}$), where $w=1,\dots ,W$. At first, an incoming HTTP request $r_i$ is assigned to the class $k_i$ and it is determined as to whether it is a static or dynamic ($d_i$) request in CM. The extracted information is transferred to $E\begin{array}{c}w\\ \end{array}$ modules.

The classification of requests is needed in order to better estimate the response time of incoming requests. Each request can have the status “dynamic” or “static” ($d_i$) depending on whether it refers to a script executed on the server (e.g., PHP, Python, Java) or whether a static file is downloaded (e.g., an image file). Additionally, the request is also assigned to one of the classes, depending on the type of an object requested and its size. Requests belonging to the same class have similar request response times. More detailed classification is presented in Section 5.5.

The $MM$ measures the time of servicing requests on servers and their load, and then passes this information to $E\begin{array}{c}w\\ \end{array}$ modules.

The executor model estimates the $tp\begin{array}{c}w\\ i\end{array}$ response time of the corresponding server, taking into account its load $M\begin{array}{c}w\\ i\end{array}=\left[f\begin{array}{c}w\\ i\end{array},g\begin{array}{c}w\\ i\end{array}\right]$ where:

$f\begin{array}{c}w\\ i\end{array}$—the number of static requests processed on the $S\begin{array}{c}w\\ \end{array}$ at the time of arrival of the request $r_i$;

$g\begin{array}{c}w\\ i\end{array}$—the number of dynamic requests processed on the $S\begin{array}{c}w\\ \end{array}$ at the time of arrival of the request $r_i$.

The executor models $E\begin{array}{c}w\\ \end{array}$ can adapt to the environment using previous response time $tr\begin{array}{c}w\\ i-1\end{array}$.

The $DM$ makes a decision according to Equation (1) using information from $MM$ and $CM$ modules. The $RM$ module forwards the incoming request $r_i$ to the selected server.

4. Test Bed Used in the Experiments

In the further part of the article, various approaches to the construction and structure of the executor model are presented. In order to be able to assess the solutions adopted, we needed to develop a research environment, which is presented in this chapter.

Machine learning often uses a simulation environment at an earlier stage in the research (e.g., reinforcement learning). Creating such an environment involves preparing a policy and setting parameters that will imitate the real environment, and then adding an agent to the environment that will adapt to the given environment. For the initial training of an agent, establishing a policy to adapt to is more important than accurately reflecting the parameters of the real environment. An agent that can adapt well to the simulation environment will probably also be able to adapt to the real environment (although it has slightly different parameters). However, the measurements made show that the differences in the average response times obtained in the simulation and real environment do not exceed 20%, which according to the literature [42] is a sufficient value. After the initial selection of the algorithms used in the simulation environment, it is possible to proceed to the testing phase in a real distributed environment.

The simulation environment was created with the use of OMNeT ++ package. This system provides appropriate libraries and environment for simulation, and it is one of the most popular packages for evaluating network systems [43].

The simulation environment consisted of several modules that map the behavior of various elements of the real system, namely, HTTP request generator, load balancer, web servers, and database server. The simulator diagram is shown in Figure 3.

In order to determine the value of the real parameters of the elements of the simulation environment, we carried out preliminary experiments in a manner similar to that in [44]. The experiments were conducted on servers equipped with an Intel Core i7 7800X processor, a Samsung SSD 850 EVO driver, and 32 GB of RAM. The Apache server was running WordPress, the most commonly used CMS in the world, which is present on more than 25% of the world’s websites [45]. Thanks to this, it was possible to reproduce the behavior of many real business websites. Computers with higher performance are often used for servers, but only part of the resources (cores) is used for one virtual machine. The hardware whose parameters we used in the simulation usually do not differ in its performance from the average virtual machine used in general-purpose computing clouds, which are most often chosen for medium-sized business solutions.

The request generator module contained multiple clients, each simulating the behavior of a web browser. To download a website, we downloaded the content of the HTML file first, and then up to 6 TCP connections were opened to download the rest of the page, such as css, js, or images. The number of websites downloaded during one session was modeled using the inverse Gaussian distribution (μ = 3.86, λ = 9.46), according to human behavior. Regarding the time between the opening of subsequent pages (user think time), the Pareto distribution (α = 1.4, k = 1) was used [46]. Traffic in the network is generated synthetically on the basis of the observed behavior.

Each client simulated the download of a website whose parameters (type and size of HTML and nested objects) were the same as in the popular website https://www.sonymusic.com (accessed on 9 March 2019) [47] that also works through the WordPress CMS.

In the simulation environment, the web balancer was able to distribute HTTP requests with the usage of popular algorithms in Amazon AWS [21] and with the use of intelligent strategies, namely,

• Round Robin (RR);
• Least load (LL)—assigns HTTP requests to the servers with the lowest number of serviced HTTP requests (this strategy works very well in practical solutions and it is a kind of reference line);
• Fuzzy-neural request distribution (FNRD)—intelligent strategy using fuzzy-neural approach [6];
• Neural network (NN)—a new solution presented here, using neural networks.

The load balancer was modeled as a single queue in the simulation. Each of the web server modules contained separate CPU and SSD modeling queues. RAM acted as a cache for the file system. The database server has been modeled as a single queue. The response times for the above elements were obtained by measuring them on a real server with an Intel Xeon E5-2640 v3 processor.

The experiments were carried out for a cloud system with 20 WWW servers and 1 database server. The number of clients simulating different loads on the web system ranged from 100 to 2500.

In each experiment, 40 million HTTP requests were processed, the warming phase took about 10 million requests, and the response time for 30 million was measured.

5. Design of Executor Model

As it was previously described, the load balancer distributes HTTP requests to different web servers $S\begin{array}{c}w\\ \end{array}:w\in \left\{1,2,\dots ,W\right\}$. There is one executor model $E\begin{array}{c}w\\ \end{array}$ associated with each server $S\begin{array}{c}w\\ \end{array}$. The module $E\begin{array}{c}w\\ \end{array}$ estimates the time needed to respond to the $r_i$ request. On the basis of the results of the estimation for all servers, the server with the expected shortest response time is selected.

The description and research on the modules $E\begin{array}{c}w\\ \end{array}$ is given below.

Assuming that each of the requests $r_i$ can be serviced by one of the many servers $S\begin{array}{c}w\\ \end{array}$, we found that it essential to choose the right one. Since each $S\begin{array}{c}w\\ \end{array}$ has one executor model $E\begin{array}{c}w\\ \end{array}$ associated with it, it is possible to specify different operating parameters $Q\begin{array}{c}w\\ \end{array}$ for each server.

At the input to the executor, the parameters $k_i$ and $d_i$ are given, which characterize the incoming request. The executor $E\begin{array}{c}w\\ \end{array}$ input also includes information about the number of currently processed static requests $f\begin{array}{c}w\\ i\end{array}$ and the number of processed dynamic requests $g\begin{array}{c}w\\ i\end{array}$ on the server $S\begin{array}{c}w\\ \end{array}$. The model also receives feedback on the real time of servicing the last request $tr\begin{array}{c}w\\ i-1\end{array}$ by the server $S\begin{array}{c}w\\ \end{array}$ (the $i-1$ index is arbitrary here since at the time of the next request, the previous one may not have been processed yet, and therefore its response time may also be unknown). The general diagram of the executor module is presented in Figure 4.

The primary function of the executor model is to estimate the time $tp\begin{array}{c}w\\ i\end{array}$. Since further considerations are made for a single executor model $E\begin{array}{c}w\\ \end{array}$, there is no superscript $w$ in the markings and formulas.

The time $t{p}_i$is calculated as follows:

$$t{p}_i=p(f_i,g_i,k_i,d_i,t{r}_{i-1})$$

(2)

where

$p$—function that depends on the current operation parameters $Q$ at the time of the request $r_i$

This module can operate in two modes, learning or prediction. Both modes take into account the current state of the server $f_i,g_i$, the class of the incoming request $k_i$, and the fact whether it is a dynamic request $d_i$.

After servicing request $r_i$ by the server, the executor module $E\begin{array}{c} \\ \end{array}$ learning step follows. The information needed in the prediction mode $f_i,g_i,k_i$, and $d_i$ is also processed in this step. Additionally, the real response time $t{r}_{i-1}$ of the last request on the server is taken into account. This allows the operating parameters $Q$ to be modified in a way that enables predicted $t{p}_i$value to be closer to the real $t{r}_i$ value.

The following subsections describe the various possible approaches to the executor model structure and parameterization. They also include the evaluation of these approaches.

Since requests are classified and requests belonging to the same class have a similar response time, a separate structure $N{M}^k$ ($k=1,\dots ,K$), having different operating parameters $Q\begin{array}{c}k\\ \end{array}$, was extracted for each class, estimating the time. Each NM module contains an artificial neural network. Figure 5 presents the structure of the executor model. The inputs to the ${\mathit{NM}}^k$ module are $f_i,g_i$ and $d_i$, while $k_i$ decides which of the ${\mathit{NM}}^k$ elements will be used in the estimation of the response time $t{p}_i$.

Load balancing is a relatively complex task, but in this article, it was decomposed into many components (for each server and for each type of request). In each of these parts, there is a single simple neural network that should do one thing—predict the response time depending on several parameters. Only the entire system, composed of many simple elements, is the server load balancing system. In the preliminary research for the article, the influence of various activation functions, the additional bias parameter, the narrowing of the range of possible weight values, the use of multilayer networks, and the use of information obtained from the neural network in the previous steps were investigated. Some of these activities did not improve the system operation. The article describes some of them, as well as the results obtained.

Neural networks with different structures and operating parameters will be used to implement the $N{M}^k$ element. At the beginning, a single neuron with four inputs is used; then, it is expanded by adding a fifth input that will provide feedback from the last operation of the neural network. Finally, the application of multilayer networks is tested.

5.1. Single Neuron

At the beginning, the simplest element—a single neuron—is used as the $N{M}^k$ element (Figure 6). Thus, we provide the following inputs of the neuron: $f_i$, $g_i$, $d_i$, and $1-d_i$. The output of the neuron is the value $tp\begin{array}{c}k\\ i\end{array}$ computed as the weighted sum of the inputs, where operating parameters $Q\begin{array}{c}k\\ \end{array}=\left[q1\begin{array}{c}k\\ i\end{array},\dots ,q4\begin{array}{c}k\\ i\end{array}\right]$ are neuron weights.

If the weights assume real values, it is possible to obtain an estimated time value as a negative value. Obviously, this is an error that can be corrected during model training, but it would have a negative impact on the speed at which satisfactory results are obtained. The research carried out shows that better results are obtained when only nonnegative weights are used in the learning process $\left(\begin{array}{c}\bigwedge \\ i,k\end{array}q1\begin{array}{c}k\\ i\end{array},\dots ,q4\begin{array}{c}k\\ i\end{array}\in R\begin{array}{c}0\\ +\end{array}\right)$. The equation that describes this neural network is shown in Equation (3).

$$tp\begin{array}{c}k\\ i\end{array}=f_i·q1\begin{array}{c}k\\ i\end{array}+g_i·q2\begin{array}{c}k\\ i\end{array}+d_i·q3\begin{array}{c}k\\ i\end{array}+(1-d_i)·q4\begin{array}{c}k\\ i\end{array}$$

(3)

where

$d_i$—takes the value 1 if the request is dynamic, and 0 if the request is static.

5.2. Model Training

For multilayer networks without recursion, the steepest gradient descent methods (most often the backpropagation method) are used for training. However, these methods often have a problem with falling into local minima and learning for a specific dataset, which reduces their usefulness when data from outside the set appear.

Model training consists in adjusting the parameters $q1\begin{array}{c}k\\ i \end{array},\dots ,q4\begin{array}{c}k\\ i\end{array}$ of all neural networks so that when the request $r_i$ arrives, the estimated time $t{p}_i$ will be close to the real time $t{r}_i$ needed for the server to service the request.

The method in which weights are randomly changed was adopted as the learning method. This method has the ability to break out of any local minimum and reach the global optimization minimum. In addition, this method does not have to be based only on one-way networks (recursions or any other connections are possible).

The method is as follows: each weight of the neural network is treated as a separate, independent variable, regardless of its place in the network. The weights are randomly modified with random values, but smaller changes are more likely (Gaussian distribution has been proposed). If the approximation error after the weight modification has decreased, then the new weights are adopted; otherwise, they are rejected. With some probability, the algorithm can generate any combination of changed weights, making it possible to leave the local minimum and search all available space.

The Box–Muller [48] transformation was used to generate values in the Gaussian distribution (4):

$$\begin{array}{cc}{R}^2=-2\ln U_1,& \theta =2\pi U_2\\ Z_1=R\cos \theta ,& Z_2=R\sin \theta \end{array}$$

(4)

where:

$U_1$ i $U_2$—independent samples chosen from the uniform distribution on the unit interval $\left(0, 1\right)$;

$Z_1$ i $Z_2$—independent random variables with a standard normal distribution and with standard deviation $sd=1$

5.3. Error Evaluation

When training the model, it is important to evaluate the error made by the module $E$ before and after modifying the parameters. The absolute percentage error ($APE$) was taken as the measure of the error:

$$AP{E}_i=100·\left|\frac{t{r}_i-t{p}_i}{t{r}_i}\right|$$

(5)

It is also possible to evaluate the mean absolute percentage error ($MAPE$) of the model, taking into account the last $n$ requests (6):

$$MAP{E}_{i,n }=\frac{100}{n}{\displaystyle \sum }_{t=i-n+1}^i\left|\frac{t{r}_t-t{p}_t}{t{r}_t}\right|$$

(6)

However, evaluating the $MAPE$ for too high $n$ may result in excessive computation time that should be performed in real time (without causing significant system delays). Too high n may also cause the system to become unreliable and unable to respond quickly enough to changes taking place in the server and in the characteristics of incoming requests.

5.4. Determining Optimal Parameters for Adaptation

Due to the Gaussian distribution, any change of the weight values is possible. However, the selection of the appropriate standard deviation (sd) may improve the results of the neural network adaptation.

To select the best standard deviation value, we performed experiments for subsequent values of this parameter ($sd=0.05, 0.02, 0.01, 0.005, 0.002, 0.001$). For each $sd$ value, the average response times of requests in the system were examined. The changes in the request handling times were not too large. Therefore, the diagram (Figure 7) narrowed the view only to measurements for high system load. Another modification of the algorithm was concerned with the fact that for each request, during learning, several attempts are made to change the weights with different values of standard deviations ($sd=0.005, 0.002, 0.001, 0.0005, 0.0002, 0.0001, 0.00005$). The last method (tentatively named $NN1$) turned out to be slightly more effective for the highest workloads of the system. This method was used in subsequent studies.

5.5. Request Class Analysis

The next study was concerned with the influence of the division of requests into classes. The division into classes can also affect the time needed to response the requests. Requests differ from each other depending on what objects are expected, and it is especially important whether it is a dynamic object (script executed on the server) or a static object (downloaded file). The size of the requested files is also significant. The average time to handle such a script or to download a file of a certain size is crucial in the classification of the resource. Some objects may have similar response times. Therefore, the combination of request types into smaller or larger groups was considered, hereinafter referred to as request classes. Then, a separate neural network was built for each of these request classes.

In the beginning, preliminary tests were performed. The tests were conducted for a very low load on the servers (small number of clients). Only requests made without waiting in the queue on the server were taken into account for the statistics. For each type of request, the number of requests during the simulation and their total execution time were counted. On this basis, we also calculated the average request execution time for each type. Then, the request types were sorted according to average execution time. The next step was to create request classes by combining these request types with similar average response times.

The first method of division into classes was used so that the number of requests for these classes was balanced. This means that, during simulation, some types of requests were more frequent, and some less frequent. If for a certain class of requests, only a few requests were generated during the simulation, it was not enough for the neural network serving this request to learn correctly. Therefore, the grouping of requests into classes was aimed at ensuring that each network had a similar amount of data that needed to be processed. The division was made into 1, 5, 10, 20, 35, 70, and 140 groups. The simulation results for each division are presented in Figure 8.

The lowest value of the average response time for the load of 2500 clients was obtained for the division into 70 groups (tentatively named $NN2$), which was 0.56 s.

In another split method, it was assumed that requests in each group are served in a similar total time. This means that classes that serve “small” requests process more requests, while classes with higher average request response times process fewer requests. The results of such a division are shown in Figure 9.

The lowest value of the average response time for the load of 2500 clients was obtained for the division into 140 groups, and it was as much as 0.92 s.

The division into 70 groups, in such a way that balances the number of requests for each group, turned out to be the most advantageous (marked as $NN2$ in the chart in Figure 8). This method of division was adopted for further research.

5.6. Additional Input

Then, we proposed adding a fifth input, which would implement a state memory of the module $E$. An additional input was used for each neural network in several variants (Equations (7)–(9)):

(1)

the last predicted request ($t{p}_{i-1}$) response time was given to the input;

$$tp\begin{array}{c}k\\ i\end{array}=f_i·q1\begin{array}{c}k\\ i\end{array}+g_i·q2\begin{array}{c}k\\ i\end{array}+d_i·q3\begin{array}{c}k\\ i\end{array}+(1-d_i)·q4\begin{array}{c}k\\ i\end{array}+t{p}_{i-1}·q5\begin{array}{c}k\\ i\end{array}$$

(7)

(2)

the last request response time ($t{r}_{i-1}$) was given to the input;

$$tp\begin{array}{c}k\\ i\end{array}=f_i·q1\begin{array}{c}k\\ i\end{array}+g_i·q2\begin{array}{c}k\\ i\end{array}+d_i·q3\begin{array}{c}k\\ i\end{array}+(1-d_i)·q4\begin{array}{c}k\\ i\end{array}+t{r}_{i-1}·q5\begin{array}{c}k\\ i\end{array}$$

(8)

(3)

the difference between the response time ($t{r}_{i-1}$) of the last request and its estimated value ($t{p}_{i-1}$) was given to the input.

$$tp\begin{array}{c}k\\ i\end{array}=f_i·q1\begin{array}{c}k\\ i\end{array}+g_i·q2\begin{array}{c}k\\ i\end{array}+d_i·q3\begin{array}{c}k\\ i\end{array}+(1-d_i)·q4\begin{array}{c}k\\ i\end{array}+(t{r}_{i-1}-t{p}_{i-1})·q5\begin{array}{c}k\\ i\end{array}$$

(9)

The diagrams of proposed neural networks are shown in Figure 10, while the simulation results are presented in Figure 11.

The lowest value of the average response time for the load of 2500 clients was obtained for the fifth input with the last estimated request response time, and it was 0.50 s (marked as $NN$ in the chart). This structure of neural networks was used for further research.

5.7. Multilayer Networks

The structure of neural networks tested thus far was the basic structure (one neuron, no hidden layers). The next step in the research was to expand the network by one or two hidden layers. The complexity of the neural network should be adapted to the complexity of the problem to be solved. Too simple and too complex structures will yield worse results. The structures of the networks studied are shown in Figure 12.

Each of the tested networks had five inputs in the same configuration as the best solution from the previous study $NN$. The inputs are as follows:

• the number of static tasks currently serviced on the server;
• the number of dynamic tasks currently serviced on the server;
• is the current task dynamic (value 1—dynamic task, 0—static);
• is the current task static (value 1—static task, 0—dynamic);
• value of the last predicted request response time.

The networks differed in the presence of neurons in hidden layers. The version without hidden layers (earlier solution), 1, 2, 3, and 4 neurons in one hidden layer (Figure 9) and two neurons in two hidden layers (Figure 9) were used. Networks have a linear activation function.

The results obtained from the experiments conducted for neural networks are presented in Figure 13. The chart shows that the best results for this task were obtained by neural networks with a simple structure (marked as NN in the graph), without hidden layers.

5.8. Comparison with Other Algorithms

The results obtained were compared with the results in the following experiment steps ($NN1$ and $NN2$) and with other popular load balancing algorithms—$FNRD$ (fuzzy-neural request distribution), $LL$ (request handling by the server loaded with the least number of requests), and $RR$ (request handling in circular order). The results of the comparison of mean response time are shown in Figure 14.

Table 1. Results of the comparison of mean response time in tabular form.

Number of Clients	FNRD	LL	RR	NN1	NN2	NN
100	0.0200	0.0402	0.0543	0.0529	0.0275	0.0281
300	0.0550	0.1231	0.1533	0.1563	0.0742	0.0736
500	0.1031	0.2042	0.2520	0.2560	0.1152	0.1165
700	0.1426	0.2850	0.3503	0.3465	0.1647	0.1571
900	0.1957	0.3650	0.4499	0.4444	0.2017	0.1977
1100	0.2520	0.4438	0.5481	0.5387	0.2415	0.2519
1300	0.3181	0.5274	0.6457	0.6306	0.2842	0.2880
1500	0.3686	0.6077	0.7461	0.7273	0.3365	0.3161
1700	0.4418	0.6889	0.8448	0.8157	0.3896	0.3535
1900	0.5921	0.7701	0.9454	0.9103	0.4317	0.3984
2100	0.6660	0.8602	1.0412	1.0123	0.4896	0.4531
2300	0.7235	0.9399	1.1366	1.0947	0.5403	0.4788
2500	0.8572	1.0199	1.2225	1.1997	0.5620	0.5022

presents the data in tabular form.

Another metric by which the algorithms can be compared is the 95 percentile of response time. The results of the comparison of 95 percentile of response time are shown in Figure 15.

Table 2. Results of the comparison of 95 percentile of response time in tabular form.

Number of Clients	FNRD	LL	RR	NN1	NN2	NN
100	0.0437	0.1355	0.1812	0.1746	0.0670	0.0606
300	0.0643	0.3567	0.4566	0.4553	0.1730	0.1260
500	0.1057	0.5469	0.7352	0.6941	0.2057	0.1975
700	0.1638	0.8114	0.9988	0.9174	0.3130	0.2354
900	0.2296	1.0402	1.2668	1.1964	0.3594	0.2835
1100	0.3036	1.4115	1.5321	1.4390	0.3887	0.3702
1300	0.4656	1.7626	1.8227	1.7040	0.4520	0.4310
1500	0.6029	1.7951	2.0926	1.9555	0.6059	0.4475
1700	0.7820	2.2917	2.4390	2.1821	0.7531	0.4372
1900	1.4504	2.5644	2.5980	2.4590	0.7591	0.5234
2100	1.8845	2.7068	2.9808	2.7366	0.9055	0.7019
2300	2.3684	2.7236	3.1788	2.9363	0.9480	0.6451
2500	3.6745	3.0614	4.0762	3.2663	1.0931	0.6620

presents the data in tabular form.

The results presented show that the proposed method provides shorter response time results compared to other popular methods and it is also stable for high loads.

5.9. Conclusions and Further Research

The developed method consists in the decomposition of a relatively complex task, which is load balancing in web servers, into many small tasks. For each of these subtasks, a neural network is designated that estimates the response time to requests. The best results were obtained when separate elements predicting times for different servers and request categories were made (i.e., one neural network provides only service times for one server and one type of request). Both the types of requests and the load on the servers change in the simulated environment. Each neural network specializes in a small portion of the tasks reaching the load balancer—this helps it adapt more accurately.

In future studies, the authors intend to perform tests in a real distributed environment. An additional research direction will be an attempt to use neural networks that will adapt not only by changing weights but also by changing the structure of connections.

6. Summary

Quality of service includes many aspects such as customer satisfaction and cost reduction. The authors of this article focused on one of them, i.e., the request response time. In this article, we presented a decision-making algorithm and method applied in load balancer working in the cluster-based web system. The decision mechanism uses the neural approach to predict response times of processing HTTP requests in web servers. Several different configurations were tested. Among others, we used single-neuron and multilayer networks, different inputs, and parameters affecting the adaptation process.

A simulation environment was designed and implemented to evaluate the proposed method. The simulator correctly imitated the behavior of web clients, as well as the operation of the web switch, web servers, and database. For all workloads, the proposed distribution method outperformed the non-intelligent strategies in the simulation environment. For high server loads, the neural approach also achieved shorter times than the other intelligent strategies.

It should be noted that all tests were carried out with the use of a simulator. Another goal of the authors is to obtain research results in a real distributed environment. The results of the research indicate that the presented new solution introduces an improvement in the simulated environment, and it is worth conducting further research on neural decision-making systems.