Intelligent Load Balancing Techniques in Software Defined Networks: A Survey

Abstract

In the current technology driven era, the use of devices that connect to the internet has increased significantly. Consequently, there has been a significant increase in internet traffic. Some of the challenges that arise from the increased traffic include, but are not limited to, multiple clients on a single server (which can result in denial of service (DoS)), difficulty in network scalability, and poor service availability. One of the solutions proposed in literature, to mitigate these, is the use of multiple servers with a load balancer. Despite their common use, load balancers, have shown to have some disadvantages, like being vendor specific and non-programmable. To address these disadvantages and improve internet traffic, there has been a paradigm shift which resulted in the introduction of software defined networking (SDN). SDN allows for load balancers that are programmable and provides the flexibility for one to design and implement own load balancing strategies. In this survey, we highlight the key elements of SDN and OpenFlow technology and their effect on load balancing. We provide an overview of the various load balancing schemes in SDN. The overview is based on research challenges, existing solutions, and we give possible future research directions. A summary of emulators/mathematical tools commonly used in the design of intelligent load balancing SDN algorithms is provided. Finally, we outline the performance metrics used to evaluate the algorithms.

1. Introduction

Technological advancement has had great impact on the amount of traffic on the internet. As a consequence, there has been a need to improve the intelligence, efficiency and reliability of the underlying network. With traditional network architecture, depicted in Figure 1, it is difficult to cope with the growing and evolving needs of the internet [1]. This failure to cope with the rapid growth stems from the fact that traditional networks are not designed to be robust enough to accommodate the fast growth [2]. Traditional networks suffer from network management issues because network hardware devices, such as switches, routers, and load balancers, are all vendor specific. Therefore, such devices have tightly coupled data and control planes. Their functionality cannot be changed easily without the use of vendor specific network management systems [2].

To mitigate the short comings of traditional networks, researchers have proposed solution such as software defined networking (SDN) [2,3,4,5,6], and network functions virtualization (NFV) [7,8,9,10]. Both SDN and NFV offer the ability for the network design and infrastructure to be abstracted in software and then implemented across different hardware and devices. It is this abstraction in software that enables the two technologies to cope with the failings of traditional networks. SDN further allows for the configuration and running of the network to be defined and modified via programming, thus adding programmability to the network [11,12,13,14,15].

SDN abstracts physical network devices and moves all the decision-making to the control plane, see Figure 2. It is in the control plane where all the network intelligence takes place including packet forwarding, and network management policies. It is this structure of SDN that allows for robust administration, and scalability of the network [5,14,16,17]. SDN controller enables the entire network to be controlled from a centralised point. Therefore, simplifying network design, and the network control becomes vendor-independent, which results in the utilization of simpler network devices.

An important part of designing and managing a network includes the process of effectively distributing traffic among network devices, i.e., load balancing [5,18,19]. An effective load balancer is one that optimizes network parameters such as latency, resource utilization, throughput, and fault tolerance with minimal power consumption. In traditional networks, load balancing is usually carried out by a dedicated server, see Figure 1. To cope with the ever changing network demands, dynamical load balancing servers have been implemented [18]. For load balancing in VFNs, we refer the reader to [9] and references within. In this paper, we focus on analysis and review of different load balancing techniques on SDNs. In order to have a comprehensive survey, we carry out the following:

• We highlight the key elements of SDN and OpenFlow.
• Review of effective load balancing technique in SDN.
• We highlight research challenges, existing solutions, and research directions for load balancing in SDN.
• Finally, we give a summary of the emulators/mathematical tools commonly used in the design and test of load balancing algorithms in SDN. We also outline the performance metrics used to evaluate the algorithms.

The rest of the paper is organized as follows: A brief overview of SDN architecture is covered in Section 2. In Section 3, we provide a comprehensive review of load balancing routing. Related works on intelligent load balancing in SDN are reviewed in Section 4, where we cover brief summaries of the existing methods. In Section 5, we thoroughly discuss and analyze results and provide future research prospects on SDN based load balancing techniques. We then finally conclude the paper in Section 6.

2. The SDN and OpenFlow Architectures

In order to set a foundation for a comprehensive review of load balancing in SDN, and for the completeness of this paper, we do an overview of SDN and OpenFlow architecture. Part of the overview is to provide a clear separation between SDN and OpenFlow, as the two have been mistakenly regarded as synonyms in some instances.

2.1. SDN Architecture

We review SDN main technical features, and we refer the reader to [14,20,21,22,23,24] for comprehensive details on SDN architecture. The commonly adopted SDN architecture was proposed by the Software-Defined Networking Research Group (SDNRG) [25], depicted in Figure 2. This architecture comprises three different planes, as follows.

Application plane: This layer is comprised of the business process and SDN applications, and provides for user interfaces.

Control plane: This is where the intelligence of SDN resides. The control plane comprises of SND controller, which is responsible for the running of the network. For example, all policies, relevant decisions regarding packet forwarding, and rules for the infrastructure are processed by the controller. Communication between the application and control planes is via the northbound application programming interfaces (APIs) [14]. These APIs are usually open sources which enables flexibility, since one can always modify them to suit their needs.

Data plane: This plane is also known as the infrastructure plane, and it consists of all network forwarding devices, i.e., routers, switches and bridges, etc. Communication between the data and control planes is via the southbound APIs and the most commonly adopted API is OpenFlow, see [26].

It is this SDN structure that provides for easiness and flexibility for coming up with new services and applications, such as load balancing, routing, policy enforcement, or any customized application that may be required for optimized network resource services [14].

2.2. OpenFlow Technology

OpenFlow facilitates for routing decisions, forwarding of rules like load-balancing, security and quality of service (QoS) from the SND controller to be pushed onto devices in the data plane. It is the commonly adopted SDN protocol for communication between the network devices and the SDN controller, i.e., southbound API [26]. OpenFlow was first standardized by the open networking foundation (ONF), and it has since been adopted by information and communication technology (ICT) companies such as Cisco, HP, and Google [25,27]. OpenFlow protocol enables direct access and configurations of network devices such as routers and switches. As an example, OpenFlow enables access to the flow table and passing of instructions to the switches on how to forward network flows. This, along with programmability of the SDN controller, ensures that the network changes can be implemented with minimal delay.

There are two main types of OpenFlow-based switches: OpenFlow-hybrid and OpenFlow-only [28]. OpenFlow-hybrid switches make it possible to execute both OpenFlow operation and normal ethernet switching operation like traditional L2 ethernet switching, virtual local area network (VLAN) isolation, L3 routing (IPv4 routing, IPv6 routing), access control list (ACL), and QoS processing. On the contrary, OpenFlow-only switches allow for OpenFlow operations only, meaning that all packets are processed by the OpenFlow pipeline, and cannot be processed otherwise.

2.3. OpenFlow Architecture

OpenFlow controller, OpenFlow switch, and OpenFlow protocol constitute the main elements of an OpenFlow network architecture, Figure 3. In addition, how a software controller and switches share information is depicted in Figure 3. This sharing of information has abstraction of network elements and enabled by the OpenFlow protocol. OpenFlow architectures also provide a concentrated management of the global network; hence, the controller has total comprehension of the entire network, which makes it easier to implement topology recovery in case there are some eminent changes, e.g., during link failure. In case of load balancing, the architecture allows for a global view of the network and constant assessing of the load of several links. This allows for easy modification of forwarding rules such that the equality of the load is maintained. OpenFlow networks are based on the ability to conceptualize all traffic as flows, meaning that different flows will have different entries in a flow table and different rules can be stated for each entry.

The OpenFlow architecture, Figure 3, is not to be confused with SDN architecture, Figure 2, as SDN is an abstraction of the network in software. SND allows for the network to be programmable, hence giving network engineers the flexibility to develop and deploy network management tools. On the other hand, OpenFlow is an API which enables communication between two SDN planes, being the control and data planes. To be more precise, OpenFlow protocol makes the sharing of information between the SDN controller and network devices in the data plane easier.

3. Intelligent Load Balancing Routing

As already mentioned, intelligent load balancing has become a necessity in our networks because of continuous traffic volume increase, user demand growth, and applications’ complexity. Most organizations need to build adaptable and robust networks that can withstand new challenges without compromising performance, security or service, hence the need for intelligent load balancers. Intelligent load balancing techniques split the network traffic into multiple flow paths to minimize congestion on any of the flow path [29]. Moreover, the intelligent load balancing technique allows for minimized response time, optimized traffic, and maximized throughput. Therefore, when using software intelligent load balancers, network managers can enjoy benefits like predictive analytics that make it possible to discover traffic bottlenecks before they even occur. This simply means that the software intelligent load balancers are also used to provide actionable insights to an organization. In this section, we look at how routing is done in SDN to facilitate for intelligent load balancing. In order to appreciate the power of intelligent load balancing routing in SDN and its advantages, we first cover a summary of load balancing routing in IP networks.

3.1. Load Balancing Routing in IP Networks

There is not much flexibility in routing when using traditional IP networks. Normally, IP network routers use common link state routing protocols like open shortest opath first (OSPF) [30,31] and intermediate system to intermediate system (IS-IS) [32] to acquire link metrics and the network topology. Algorithms that utilize the shortest paths between the source and destination node pairs is used in IP networks, and this can lead to congestion at some bottleneck links due to the static routing path. Even though one can argue that changing the link weight will lead to change in routes, this can not be the best approach for intelligent load balancing in IP networks—the reason being that routers will have different receiving times when a link state change message is shared. This leads to a situation whereby the network states would become inconsistent when viewed from different routers. Whenever there are link metric updates, chances are that some transient traffic loops would be formed because there are no precise synchronization among the routers [33,34].

Therefore, it is very difficult to ensure that the load-balancing performance in the IP network is optimal [1], mainly due to the fact that there is no global view of the network. This lack of global network routing view also means that routing tables in IP networks contain destination network information only, hence routers can only know the next-hop network and the interface it must use.

Some of the well-known load balancing techniques that are IP based are discussed in [35,36,37,38,39,40]. Some works focused on the relative load on the different network instances [41], while some early works were based on round robin domain name system (RR-DNS) to allocate inbound connections towards a group of servers [42].

3.2. Intelligent Load Balancing Routing in SDN

Traditional load balancers are not programmable due to the fact that they are vendor locked by design, and this means that network managers and administrators are constricted from coming up with their own load balancing algorithms. The main contrast of SDN load balancers is that network managers and administrators are provided with a platform to design and build their own intelligent load balancing techniques, thus SDN load balancers are more flexible and programmable. Moreover, there is no separate hardware device needed in SDNs to act as a load balancer. In SDN, the network topology is used to make intelligent routing decisions between edge devices, which takes into account protection and bandwidth requirements for every service that is provisioned across the network. The outcome of the routing algorithm is a set of flow table entries that are installed in OpenFlow-capable switches to enable forwarding of packets from end to end.

The SDN architecture allows for the removal the routing protocols at the hardware level, thus resulting in an improved network management and diagnosis. The SDN controller, working together with the intelligent load balancing application via the OpenFlow protocol, makes data path control decisions without having to rely on algorithms defined by traditional network equipment. An SDN-based load balancer saves running time by having control over an entire network of application and web servers. Its main objective is to reduce overloads as optimally as possible, utilize network resources, and obtain minimum response time by distributing the network workload. Intelligent load balancing is also a basic problem in many practical systems in daily life; hence, various techniques based on SDN architecture to be discussed in Section 4 and references within have emerged.

The SDN controller is able to intelligently access and use the information of the state of the network at lower layers together with knowledge of the upper networking layers to make forwarding decisions. This means that information regarding the status of the network infrastructure, the applications running on them, and network traffic congestion all have effects on intelligent load balancing decision-making process. The SDN controller has the benefit of the global view of the network at the beginning of network construction, hence it can discover all possible flow paths between each source-destination node pairs by updating topology information of the global network. This global network views ability by the SDN and enables the load condition of every global path to be evaluated, resulting in a more optimized intelligent load balancing decision-making. This is demonstrated by the SDN network architecture shown in Figure 4.

4. Review of Load Balancing in SDN

Effective load balancers intelligently determine which device within the network farm is best to process an incoming data packet. Doing so requires algorithms that are geared towards distributing loads in a specific way. Algorithms vary widely, depending on whether a load is distributed on the network or application layer. Algorithm selection impacts the effectiveness of load distribution mechanisms and, consequently, performance and business continuity.

Optimizing network performance in SDN, like any other network, necessitates the need to find the best load balancing strategies. Various techniques have been proposed in literature to optimize network flow and to achieve intelligent load balancing among different links, servers, cloud, and controllers.

4.1. Controller Load Balancing

In order to increase capability of the control plane, sometimes it is essential to decentralize the SDN controller. The decentralization normally results in the control plane comprised of more than one controller. In the following, we review load balancing in the case where the control plane consists of several SDN controllers.

The reliable and load balance-aware multi-controller deployment (RLMD) scheme is proposed in [43]. According to the authors, the main objective of RLMD is to resolve the controller traffic load disparity and control plane accuracy issues that arise when the control and data planes split-up in the multi-controller SDN network. RLMD as a progressive scheme ensures that equitable allocation of controller traffic loads and effective dependable deployments of controllers are in place. The authors also came up with a multiple domain partition (MDP) algorithm that is able to match controllers and switches based on the load balancing rate of the controller and the node attractability. When tested against other typical strategies, their algorithm managed to balance the allocation of the controller traffic loads much better, and also enhanced the reliability of the control plane.

To solve the well-documented problem of load balancing in the SDN control plane, the authors of [44,45,46,47] focused on distributed controllers architecture for SDN control plane which can dynamically balance its traffic load. The researchers of these works were able to provide solutions to the problem of handling multiple controllers, hence providing a solution to open research on how to match each controller to a switch and where to place the controllers. These works effectively split the switch requests among different controllers in order to avoid congestion.

A switch migration based strategy called ElastiCon is proposed in [48]. ElastiCon periodically balances the load of controllers in a pool, and ensures good network performance at all times irrespective of the traffic dynamics. The downside of ElastiCon is that in some cases it leads to a high network overhead. Some similar works that enhance efficiency in the SDN controller were presented in [48,49,50]. The focus of these research works were to select a switch to migrate as a process of load balancing. This process reduces overloaded controllers, hence avoiding degradation of the network performance.

The authors of [51] came up with a load balancing scheme based on switch groups of multiple controller architecture to improve time efficiency of the load balancing process. The proposed scheme uses a switch selection algorithm and target controllers selection algorithm to solve the load oscillation problem among controllers.

In summary, the main idea across the articles cited above is to implement a controller load balancing architecture that can utilize one of these controllers to partition control traffic and distribute it across different available controllers.

4.2. Server Load Balancing

We now consider sever based load balancing, in the case where a network has multiple servers.

In [52], the authors propose a load balancing strategy for distributing clients requests across multiple servers. The strategy ensures that the next request is forwarded to the least-loaded server. The authors evaluated their strategy with random-based and round-robin, and they also analyzed their effects on the overall network performance.

A genetic algorithm (GA) is introduced in [53] for server load balancing in the OpenFlow based system. GA is based on the biological process of natural selection whereby the favorable genetic traits are passed to the offspring. The algorithm has three types of operators, which are selection, crossover, and mutation. The researchers make an assumption that multiple clients make multiple requests to a load balancing pool of servers with varying loads. Their application of GA uses a roulette wheel as the selection method, single point crossover and single point mutation. Its fitness function is calculated based on the coefficient of variation given K servers with varying workload. The performance objective was to minimize the average for the coefficient of variance. The GA was evaluated against load based, random, and round robin algorithms and it gave better results.

To optimize the distribution of cloud consumer application requests, [54] proposed a partial swarm optimization (PSO) based load balancing technique. This scheme uses swarm intelligence to minimize average application response time, maximized throughput, and for better resource utilization. This proposed scheme actively configures and comes up with traffic flow paths needed, according to the network traffic load.

In [55], the authors propose a solution for mitigating against congestion in a pool of servers. The researchers used bandwidth and round-robin metrics to measure the performance of their proposed system. From the analysis of this work, the bandwidth strategy was more effective in terms of throughput and response time when compared to allocating servers based on round-robin strategy. Researchers in [56,57,58,59,60,61] studied similar approaches for allocating requests to servers.

The SDN based server load balancing strategies are mainly concerned with ensuring that requests are distributed efficiently. The common argument of adopting SDN technology among researchers is that it does not need any dedicated hardware when compared to the traditional load balancers. Moreover, OpenFlow protocol enables solutions that are more effective as servers are pressured to handle lot of access requests in a short time. The above studied literature demonstrated how SDN technology can be utilized to intelligently solve the congestion problem on server clusters due to the ever increasing number of users in our networks.

4.3. Load Balancing in Wireless Links

IEEE 802.11 does not come with an automotive load balancing in WLAN setup, hence load balancing in the existing wireless LANs can be a troubling issue [62]. Algorithms deployed on SDN controller can be utilized to dynamically balance access points (APs) load by choosing the less loaded ones from the sharing locality for clients association.

A two-tier architecture for managing load in access points (APs) is proposed in [63]. In this scheme, the SDN controller gets information from the APs and decides on the maximum allowed load level an AP can accept association requests without consulting the SDN controller. Their solution strikes a fair balance as it adjusts automatically the maximum load level an AP can act on its own. The proposed solution is also applicable to any wireless devices that support OpenFlow.

In [64], authors studied an adaptive load balancing technique for Wi-Fi APs. The proposed strategy is made up of two sections: arrival event detection and adaptive load balancing. APs send their traffic load and connection situation in real time to the SDN controller. Then, an algorithm is used to derive the optimal association relationship between APs utilizing the global vision of SDN controller. A similar strategy is also proposed in [65,66,67].

Variance analysis for massive mobile users based on an efficient SDN load balancing scheme was proposed in [68]. The proposed scheme has good scalability to support massive mobile users, and can effectively monitor dynamic workload in mobile communications. The authors also argued that their work has better load balancing when compared to existing solutions. The SDN technology simplifies the network, and it was utilized to enable intelligent traffic management from a global point of view for quick reaction to dynamic network conditions. The proposed scheme showed better throughput, better response time, and better overhead, but was faced with issues of low availability, high packet losses, and high energy consumption.

In [69], the authors proposed a software defined wireless sensor network (SDWSN) based load balancing technique. This work formulated a mathematical model using multi-path routing technology to reduce the maximum link utilization in the wireless sensor network. The proposed technique utilizes the advantage of the global view of the wireless sensor network provided by software defined network to implement an effective and flexible traffic scheduling. The focus was on how to intelligently manage the wireless sensor networks infrastructure in a smart city. In [70], an SDN technology was utilized to provide a solution to the problem of load balancing on heterogeneous wireless access networks. They designed an algorithm to consider both the network load and the user experience for effective load balancing on the radio access network (RAN). Their simulation results indicate that users can enjoy good connection experience and, at the same time, the RAN traffic load remain balanced.

4.4. Communication Path Selection Load Balancing

Effective load balancing of a network traffic across multiple links has been a common challenge faced by network engineers as networks increase in speed and size [71]. SDN enables schemes that can easily divide the traffic flow among similar paths in order to enhance the performance of the network in a more scalable and dynamic manner [72]. In data center networks that are designed for multi-path routing, proposed load balancing schemes should distribute the traffic effectively by ensuring that neither path is under-utilized nor over-utilized. Networks like data center are dynamic and traffic congestion can happen any time because of the nature of traffic variation.

In [73], the authors proposed a dynamic load-balanced path optimization (DLPO) scheme, which is applicable in various OpenFlow-based data center network topologies. The technique proposed enables for change of paths during flow transmissions. This ensures that effective traffic load balancing through various communication paths, and avoid congestion challenges in SDN data center networks. A technique called LoAd-BalancEd routing wIth OpenFlow (LABERIO) [1] was one of the earlier SDN based research works to propose a dynamic path optimization scheme to balance network traffic during flow transmissions. In this scheme, a traffic flow could be diverted to a new path during the transmission if the link loads of a network are unbalanced. The drawback of this proposed technique is that, since its path optimization algorithm is dependent on network topology, the technique would have to be modified whenever there is a change in the network topology.

A load balancing routing algorithm for fat-tree networks that distributes traffic across multiple paths equally is introduced in [74]. This algorithm has delay drawback as it takes a long time to be executed. This drawback is addressed in [75], where the authors designed a dynamic load balancing algorithm with congestion control. They ensured that they looked into the weights of edges and nodes in order to find the shortest path between each nodes pair. The algorithm proposed in [75] also has its own drawback, however, as it is only suitable for small networks.

Strategies that optimize network load and at the same time reduce execution time, overheads, latency, and improve bandwidth utilization are covered in [76,77,78]. Even though these proposed techniques had some success in terms of improving the overall load balancing, they also suffered from packet loss and high energy consumption.

4.5. Artificial Intelligence Based Load Balancing

Researchers in [79,80,81,82,83,84], have utilized the advantages of SDN paradigm and integrated artificial neural networks (ANNs) to propose new load balancing techniques. ANNs are described as flexible mathematical structures that are capable of pin-pointing complex nonlinear connection between input and output data sets [85]. The model is found to be efficient and useful, especially in situations where the characteristics of the processes are not easy to explain using physical equations. ANNs are useful because they have no limits on input vectors, and due to the uncertainty of the traffic flow in network, a large amount of data can be collected to form a data-set. There is a high possibility that this formed complex traffic flow data-set follows no certain probability distribution, hence ANNs are most suitable for handling this uncertain network traffic data. The structure of the ANNs is depicted in Figure 5.

In [82], ANNs are used to overcome the challenge of overloaded or underloaded virtual machines (VMs) in cloud computing. The authors argued that the performance of applications eventually degrade due to overloaded and underloaded VMs resulting in poor resource utilization. To overcome this problem, they came up with an algorithm that can raise the processing speed, identify heavily loaded VMs, and optimize the resource utilization. The proposed algorithm performs better than the other two algorithms that it was compared with in terms of data migration time.

Reinforcement learning and deep neural networks are used to enhance both load balancing policies and routing in [86]. The scheme leverage based on the fact that, with the global view of the network provided by the SDN controller, it becomes easier to get enough information on network flow states like traffic, links, bandwidth, and hence the ability to establish an appropriate policy. This scheme was compared to the shortest path, round robin, and ANN load balancing algorithms. Its performance was found to be better in terms of enhancing network utilization and minimizing delay.

Another load balancing strategy is covered in [87]. This strategy uses reinforcement learning to design a switch migration algorithm. The aim was to effectively select a set of switch migration queues, so that eventually the system can attain a global optimum from some local one.

As a way of ensuring that the least congested path is selected, one needs to confirm the path traffic load condition by intelligently analyzing the integrated path information. Traffic flow estimation and assessment is an important task that needs to be performed in order to ensure network quality of service and to enhance network performance. When ANNs are deployed on SDN, it ensures rationality and precision in terms of traffic flow path load condition, hence giving the controller an opportunity to make a more informed load balancing decisions.

4.6. A Summary of the Reviews

To conclude this section, a summary of widely adopted SDN load balancing algorithms is covered in

Table 1. A summary of various intelligent load balancing techniques in the software defined networking (SDN).

Algorithm/Strategy	Performance Objective	Selection Criteria and Mechanism	Summary
Deep neural networks and Q-learning (DNQ) [12,86]	Reduction of packet loss rate with different load	Centralized intelligent center	In DNQ, intelligent techniques were used to carry out these three functions; path selection, the important nodes and flow forecasting.
Least connection [92,93]	Resource utilizations optimisation	Centralized and cooperative approach	A server with the least number of active connections is allocated more connection to balance the traffic.
Software-defined sensor network load balancing (SDSNLB) algorithm [69]	Throughput, link load jitter	Centralized	SDSNLB main objective is to allocate network traffic to different flow paths as a way of enhancing functionality. The strategy is to have balanced multipath allocation of bandwidth for optimal and productive network resource use.
Dynamic agent-based load balancing (DA-LB) [54,82]	Efficient and adaptive resource management.	Centralized intelligent center	The global visibility of SDN is used to efficiently migrate virtual machines in data center network.
Reliable and load balance-aware multi- controller deployment (RLMD) [43]	Node efficiency, node attractability, path quality, controller load balancing rate	Centralized and cooperative approach	RLMD proposed a scheme that managed to achieve effective deployment of controllers and can also balance loads successively among the controllers.
vSDN [94]	Average load, average delay, bandwidth utilization and throughput	Centralized intelligent approach	The strategy here is to utilize virtual SDN controller as a VNF to enhance load balancing. Under vSDN, a standby vSDN controller can be used to share the traffic load if the flow increases or become uneven.
Fuzzy synthetic evaluation mechanism (FSEM) [95]	Dynamically select the optimal path	Centralized	Network flow is sent to those flow paths under the OpenFlow switches, so that the SDN controller can utilizes its global view of the network and install the flow-handling.
Switch migration based decision-making (SMDM) [76]	Response time, load distribution and migration cost	Centralized approach	SMDM’s primary objective is the process of choosing a master controller that can enhance load balancing factor. Based on low cost, a switch will be chosen for migration.
Adaptive load balancing Scheme [96]	Throughput and loss rate	Centralized and cooperative approach	This is a new adaptive technique used in data centers that leverage on SDN for load balancing.
Self-adaptive load balancing (SALB) [97]	Throughput testing, load balancing time, bandwidth utilization and loss rate	Centralized approach	The objective of SALB scheme is to ensure that at the same time load balancing and distance between devices are considered effective.
Double deep Q network based VNF placement algorithm (DDQN- VNFPA) [98]	Path delay, running time of VNFIs, the number of VNFIs and utilization ratio of VNFIs	Centralized	This is a customized algorithm designed using gathered information to ensure that the network performance is optimized. This algorithm does enhance network performance when it comes to metrics like delay, throughput and load balancing.
Load variance- based synchro- nization (LVS) [99]	Loop-free forwarding, synchro- nization overhead	Centralized	LVS enhances load-balancing execution because it reduces synchronization frequency and also removes forwarding loops.

. Here, we look at the performance objectives that the authors wanted to achieve as well as the selection criteria and mechanism used. Most of the literature works used centralized criteria as these works were based on SDN. Other works combined SDN and the traditional IP network into a hybrid network. This is because currently not many networks are fully SDN based. Hybrid setup can be the best way for moving into a fully SDN based network slowly, due to cost implications. We also give some summary of what the strategy used by the authors entails.

Furthermore, in

Table 2. A summary of intelligent load balancing techniques in the SDN based on research challenges, existing solutions and research directions.

Ref.	Research Challenge	Existing Solution	Research Direction
[100]	-Mission-critical network failure. -Aviation management problem. -Management and monitoring of energy consumption.	-Migration of switches using active and dynamic traffic load re-balancing on SDN. -Using SDN to ensure that when a certain traffic load limit is exceeded by a controller, other controllers can help to share that traffic load.	-Network visualization through SDN. -Designing of SDN based architecture for critical missions. -Energy load balancing strategies based of OpenFlow technology.
[101]	-The issue of degradation of the controller performance. -Optimization problem based on minimum cost bipartite task. -The challenge of discovering a many-to-one matching between the sets regarding the cost function. -Addressing the optimization problem of switch-to-controller assignment	-Dynamic control devolution and dynamic switch-to-controller association. -WAN dynamic controller provisioning. -Addressing the SDN controller placement problem utilizing the idea of network partitioning. -Optimizing powerful decision-making based on game theory.	-Managing a trade-off between the current loads on a controller and the switch-to- controller round trip time (RTT). -Understanding the idea of load-driven penalty. -Designing of environments that can handle multi-controller.
[97]	-Poor system performance due to poor traffic load management in the controller. -The problem of nmanaged heterogeneous traffic in the network.	-Enhancing overall system performance and saving energy and time in mobile devices using a distributed algorithm. -Path load-balancing using the mechanism of fuzzy synthetic evaluation. -Utilization of server response time to balance traffic load.	-Algorithm design for optimal switch-to- controller round trip time. -Load-driven distributed SDN controllers. -Robust and scalable multi-controller environments.
[69]	-Global resource balanced utilization problem in wireless sensor networks. -Requirements of wireless sensor network infrastructure for smart cities, based on high flexibility and adaptability. -Effects of traffic load imbalances on the network service quality	-Designing of a load-balancing OpenFlow based mobile network architecture for achieving centralized control of energy-aware on demand. -Designing of an SDN enabled routing protocol for wireless multi-hop network. -Designing of an SDN-IoT integrated control system for smart cities.	-Intelligent load balancing scheme for software defined wireless sensor network (SDWSN). -Intelligent and flexible network flow scheduling for SDN. -Traffic optimization routing on network links for smart and saver city sensor network.
[102]	-End devices failure to mitigate against congestion due to the fact that traditional networks cannot use packets drop as an indication of congestion -Unavailability of the global view of the network topology from a centralized point	-Hashing reroutes flows on multiple paths using ECMP. -Utilizing reinforcement learning technique for the adoption of a QoS-aware adaptive routing (QAR). -Using reinforcement learning technique to select the best traffic flow paths.	-Utilizing SDN controller to design some intelligent methods for congestion control. -Reinforcement learning based rerouting algorithm based on SDN. -Rigorous evaluation of instant congestion control using fuzzy logic.
[103]	-Failure to dynamically optimize the customized strategies by intelligently changing the reward function. -Intelligent decision-making by software defined network controller.	-Utilizing a multi-machine learning strategy based on a route pre-design scheme. -The implementation of quality of service adaptive routing in OpenFlow protocol sing QoS-aware reward functions. -Replacement of Q-table in conventional Q-learning strategy using artificial neural network.	-Utilizing deep reinforcement learning on OpenFlow technology. -Optimized mechanisms to enhance intelligent SDN packet flow. -Adoption of the most effective black-box in continuous time
[104]	-Routing optimization problem. -How to get near real-time optimal execution via a dynamic traffic flow scheme.	-Enhancing media stream communication on end-to end HTTP through a Q-learning architecture. -Enhancing the global data traffic volume using quality of service information.	-Deployment of a deep reinforcement learning strategy that can utilize recurrent neural network. -Deep reinforcement learning based on software defined networking.

, we outline possible future improvements and research direction on existing SDN based load balancing techniques. To do this, we outline the research challenge, the existing solution, and finally how the solution may be improved or the research challenge be addressed. For more on SDN based intelligent load balancing techniques, we refer the reader to [88,89,90,91].

5. Results and Comparisons

In this section, we discuss some open issues and observations made, as well as presenting a comparative analysis of different sampled intelligent load balancing metrics on SDN.

5.1. Performance Metrics and Implementation Tools

For this discussion, we selected 15 articles (

Table 3. Qualitative comparison between different intelligent load balancing techniques in the SDN schemes based on the mathematical tools/emulators and the performance metrics adopted.

Ref.	Tools/Emulators	Performance Metrics
		Throu-Ghput	Fairnes	QoS	Complexity	Congestion Control	Interference Mitigation	Delay
[105]	Mininet [106,107]	√	×	√	×	√	×	√
[103]	OMNET++ [108]	√	×	√	√	√	×	√
[49]	Mininet	√	√	√	√	√	×	√
[109]	Mininet, Iperf [110]	×	√	√	√	×	×	√
[100]	Mininet	√	×	√	√	√	√	√
[15]	Mininet	√	×	√	√	√	×	√
[102]	Mininet, Iperf	√	×	√	√	√	×	√
[50]	Mininet	√	√	√	×	√	×	√
[69]	Matlab	√	√	√	√	√	×	×
[111]	OMNET++, Python	√	×	√	×	√	√	√
[44]	.Net	√	×	√	×	√	×	√
[68]	Mininet	×	√	√	√	×	√	√
[112]	Mininet, Iperf	√	×	√	√	√	×	√
[113]	Mininet, MaxiNet [114]	√	√	√	√	√	×	√
[115]	Mininet	√	×	√	√	×	√	√

) for qualitative metrics used to measure intelligent load balancing techniques in SDN. The selection is based on the fact that the articles cover some of the most used load balancing methods. From the selected articles, we discuss seven metrics used and seven different mathematical tools or emulator software adopted, as presented in Table 3.

Using Figure 6, one can deduce that, from the selected articles, $87\%$ used throughput, $93\%$ used delay, $40\%$ used fairness, $100\%$ used QoS, $73\%$ used complexity, $80\%$ used congestion control, and $27\%$ used interference mitigation as qualitative metrics to measure the effectiveness of their proposed techniques. Some of these metrics are very critical when it comes to load balancing because they are attributes of good intelligent load balancing techniques, for example, QoS and delay. It is not surprising that these two metrics have been used by all or most of the researchers. Fairness and interference mitigation are the least used metrics by researchers from the selected articles. In most cases, a technique is designed to intelligently balance the network traffic with respect to a chosen metric. Moreover, such a technique would be regarded as a success even if it does not mitigate against interference and fairness, if it can effectively balance the the traffic flow. In our opinion, if more attention is not given to interference mitigation, this can eventually lead to degradation in performance of the technique. This is because interferences can crop up at anytime in the network. Other metrics used in the literature, not covered under this subsection, include utilization, latency, and run time. It is also worth noting that metrics adopted are based on the objective, as well as the techniques that are used to test the effectiveness of the proposed solution.

We now look at the most used tools for the implementation of intelligent load balancing techniques in the SDN environment. As illustrated in Figure 7, $55\%$ of the sampled researchers use Mininet as an emulator to test their proposed techniques. This is because Mininet is a powerful tool in terms of interactive development, demonstrations, and testing when using OpenFlow and SDN [106]. Iperf is used $15\%$ of the time to generate network traffic, while OMNET++ is used $10\%$, and, finally, other tools like Matlab, Python, .Net, and MaxiNet are used a total of $20\%$ of the time, with each tool contributing $5\%$. Some of the tools like Iperf and MaxiNet were used concurrently with Mininet to accomplish the needed SDN network simulations. OMNET++, Python were also used together in [111]. This was because the authors combined deep learning and reinforcement learning techniques, as they only learn from rewards by trial-and-error. The use of various tools demonstrates the fact that SDN gives the network administrator the flexibility to develop network administration methods that are not limited by the tools or by equipment vendors.

5.2. Open Issues

This subsection addresses some issues that have been left open in SDN intelligent load balancing. Some of the reviewed techniques did not pay much attention to energy saving. As lots of devices connect to the internet, management and optimization of energy becomes paramount. When designing algorithms that manage traffic flow in the SDN network, the issue of energy conservation is of paramount importance and still remains open for future research.

Intelligent SDN load balancing techniques should also consider the issue of link or node failures. Incorporating failure recoveries management on some of the proposed intelligent load balancing techniques would be very promising for the future works.

Another open issue includes adaptation of the proposed SDN intelligent load balancing strategies in 5G networks. This is because 5G is regarded as a new revolutionary technology for mobile networks. Moreover, 5G networks are expected to be more efficient, achieve higher convenience, and handle a high number of users, applications like smart vehicles, video surveillance, smart cities and mobile industrial, and automation [116,117,118,119,120]—hence opening opportunities for the development of complex connections of powerful heterogeneous infrastructures with high traffic capacity. To cope with this level of complexity, 5G networks must be augmented with some level of intelligence load balancing, so as to ensure a successful deployment. SDN and NFV in particular would effectively offer such intelligence. Even though some work has been done in SDN load balancing for 5G, see [121,122], we believe more work still needs to be done, precisely in the development and testing of intelligent load balancers for 5G software defined networks. The potential works could focus on measures like congestion control, security, and QoS. This is because of the anticipated high traffic and number of devices in 5G networks.

6. Conclusions

In this paper, we presented a brief summary of SDN network, its architecture, and how the SDN has effectively improved communication in the fast and growing technology world. We discussed the OpenFlow technology, its architecture, and the role it plays in the effectiveness of SDN networks. The importance of SDN and NFV in terms of facilitating intelligent load balancing was also discussed. SDN and NFV enable network administrators to provide automatized network management and efficient resource allocation and provision. We conducted a review of the literature on intelligent SDN based load balancing by analyzing different techniques. The reviewed work was categorized according to the objective of the authors.

A summary of common intelligent load balancing techniques, their main performance objectives, and their selection criteria was presented in Table 1. Table 2 outlined some current research challenges, existing solutions, and future directions for load balancing techniques.

The performance metrics of some selected techniques were summarized in Table 3, and we also cover the tools used in the implementation and testing of these techniques. Mininet was found to be the most commonly used emulator by researchers because of its ability to simulate SDN based devices. In some cases, the Mininet emulator was used with other traffic generating tools like Iperf. From the sampled reviewed work in Table 3, most authors mainly tested their solutions against QoS, delay, and throughput metrics. Almost all effective solutions for intelligent load balancing are expected to perform exceptionally against these metrics.