Software-Defined Wide Area Networks (SD-WANs): A Survey

Abstract

SD-WANs are an innovative software-defined network (SDN) technology used to reinvent networks, services, and applications in wide area network (WANs). The development of SD-WANs ranges from network optimization in the past to service provision platforms at present and distributed computing systems in the future. The existing surveys on SD-WANs are fragmented, covering specific problems only, and are not comprehensive with detailed research directions. This paper seeks to provide a systematic survey on SD-WANs by introducing major research directions and stating specific problems. Therefore, four major research directions related to traffic engineering, network optimization and systems, service orchestration, and the security issues of SD-WANs are sequentially introduced, along with detailed statements relating to specific problems and the classification of state-of-the-art research. Finally, the trends and challenges regarding SD-WANs are summarized and our future work is described.

1. Introduction

SD-WANs were first proposed in Google’s B4 data centers to achieve high utilization, load balancing, and elastic computing of valuable interconnected links between geographically distributed clouds [1]. Over the last decade, SD-WANs have always been recognized as an innovative SDN technology that can be used to reinvent networks, services, and applications in WANs, ranging from remodeling the architecture of internet service provider (ISP) [2] to facilitating the ubiquitous connectivity of smart cities [3], enabling the heterogeneous interconnection in fog computing [4], and establishing the Internet of Everything in Internet of Things (IoT) [5].

The development of SD-WANs can be divided into three stages, namely SD-WAN 1.0, SD-WAN 2.0, and SD-WAN 3.0, representing the past, present, and future of SD-WAN, respectively. SD-WAN 1.0 provided low-cost deployment, high-efficiency management, and a high utilization bandwidth of network optimization, while SD-WAN 2.0 has evolved into an abundant service provision platform, providing services such as network interconnection, secure remote access, quality of service (QoS) strategies, cybersecurity policies, and other advanced on-demand services. SD-WAN 3.0 is considered as a promising distributed computing system combined with cutting-edge technologies of artificial intelligence, fog computing, edge computing, network security, and IoT.

To further investigate research directions related to SD-WANs, seven surveys on SD-WANs [6,7,8,9,10,11,12] can be found, although six of them are conference papers or short papers. The other study [6], presented a review on a specific control failure problem in SD-WANs, and does not comprehensively cover the vast majority of SD-WAN research directions. Therefore, this paper seeks to provide a systematic survey on SD-WANs by introducing major research directions, specific problems, and the classification of state-of-the-art research.

This paper reviews 79 studies on SD-WANs, and then provides a systematic and comprehensive survey by revisiting 7 existing surveys, introducing 4 research directions and classifying the state-of-the-art research according to specific research problems. As shown in Figure 1, the structure and content of this survey can be divided into five parts: surveys, traffic engineering, network optimization and systems, service orchestration, and the security issues of SD-WANs.

The rest of this paper is organized as follows. Section 2 provides the related work. Section 3 to Section 6 introduce traffic engineering, network optimization and systems, service orchestration, and the security issues of SD-WANs, respectively. Section 7 discusses the trends and challenges relating to SD-WANs. Section 8 concludes this paper.

2. Related Work

Seven surveys on SD-WANs can be found, although six of them are conference papers or short papers. Dou and Guo [6] presented a review of a specific control failure problem by stating its impacts on path programmability and network performance, classifying solutions of state-of-the-art network recovery under controller failures, evaluating limitations of the existing technologies, and pointing out future challenges and potential directions. Yalda et al. [7] reviewed SD-WANs from both physical and logical perspectives to clarify differences and provided a comparison and classification of existing technologies, ranging from definitions, fundamentals, capabilities, and advantages to architectures of the state-of-the-art SD-WANs. Rose Varuna and Vadivel [8] conducted a survey to discuss characteristics of SD-WANs in orchestration and automation, the capabilities of self-learning and failover, the applications for end-to-end secure communication and cloud environments, and solutions for network attacks and security issues. Ujan et al. [9] reviewed the latency-oriented controller placement problem for SD-WANs to prove that other functional objectives should be complemented for multi-objective optimization so that the placement of the controller can achieve a well-rounded outcome for various scenarios. Rajagopalan [10] carried out a brief overview for SD-WANs and solutions of load balancing through SD-WANs. Yang et al. [11] revisited the status and challenges of legacy WANs, then introduced the architecture of an SD-WAN and its representative advances, finally an SD-WAN-based multi-objective network and its applications were mentioned. Michel and Keller [12] provided an overview of SDNs in WANs, focusing on the evolution of SDNs, along with research and future directions relating to SD-WANs.

To summarize, the existing surveys on SD-WANs are fragmented, covering specific problems only, and are not comprehensive with detailed research directions. This paper seeks to provide a systematic survey on SD-WANs by introducing major research directions and stating specific problems.

3. Traffic Engineering of SD-WAN

The traffic engineering of SD-WANs was first mentioned in [1,13] for network deployment, traffic management, and edge controllability of interconnected data centers to achieve the high utilization and elastic provision of cloud-based services. As SD-WANs are gradually applied in the interconnection of enterprise branches, fog computing, edge computing and IoT, subsequent research was conducted on traffic measurement [14,15,16,17,18,19,20], traffic scheduling [1,13,21,22,23,24,25,26,27,28,29,30,31,32], and failover and recovery [33,34,35,36,37,38,39,40,41,42,43], as classified in

Table 1. Research contents, specific problems, and work related to SD-WAN traffic engineering.

Research Contents	Specific Problems	Related Work
Traffic measurement	Measurement of data transmission	[14,15,16,17]
	Measurement of routing and forwarding	[18,19,20]
Traffic scheduling	Flow-level traffic scheduling	[1,13,21,22,23,24,25,26]
	Application-level traffic scheduling	[27,28,29,30,31,32]
Failover and recovery	Node failures and recovery	[33,34,35,36,37]
	Link failures and recovery	[38,39,40,41,42,43]

.

3.1. Traffic Measurement

Research on traffic measurement is mainly based on testbeds or commercial platforms to evaluate the performance of data transmission [14,15,16,17] and routing and forwarding [18,19,20] in the SD-WAN data plane. Details of the research on traffic measurement are shown in

Table 2. Details of the research on traffic measurement.

Paper	Measurement Objects	Measurement Tools	Measurement Environment
			Testbeds	Real Network
[14]	Overlay	Open source tools	√
[15]	Overlay	Open source tools	√
[16]	Overlay	Commercial platform		√
[17]	Overlay	Commercial platform	√	√
[18]	Overlay	Open source tools	√
[19]	Overlay	Open source tools	√
[20]	Underlay/overlay	Open source tools	√

.

In relation to the measurement of data transmission, Scarpitta et al. [14] realized a high-performance user-space solution to monitor transmission delay for segment routing with IPv6(SRv6)-based SD-WAN services and integrated and evaluated this solution in an open source SD-WAN prototype called EveryWAN. Iddalagi and Mishra [15] analyzed the impacts of a bidirectional forwarding detection (BFD) protocol on the performance of data transmission, established an SD-WAN testbed based on open source tools, and proved that BFD traffic introduces additional delay and jittering and forms an overhead in the uplink streams, affecting mainstream traffic as well. Manova et al. [16] carried out a case study on an active WAN of an Indonesian company, which has three connections, namely an SD-WAN, traditional Multiprotocol Label Switching (MPLS), and an Ethernet over Internet Protocol (EoIP), respectively. Performance tests relating to QoS and data transmission on SD-WAN systems were conducted to measure network delay and the change in delay in different time periods. Troia et al. [17] built two SD-WAN testbeds with the open source software ONOS for a municipal network of Italian cities and simulation platforms in their laboratory. Then, a ONOS-based traffic measurement plugin was developed and the open source tool eBPF was used to monitor real-time network traffic. Finally, the SD-WAN was proved to be capable of network recovery and service failover.

Regarding the measurement of routing and forwarding, Emmanuel et al. [18] built an SD-WAN testbed with the open source project GNS3 to capture real-time data packets for traffic classification so that the results of classification were further utilized as guidelines for controllers to improve network performance and resource utilization. Fares et al. [19] studied the routing optimization problem in SD-WAN autonomous network systems and built an SD-WAN experimental platform based on open source projects including ONOS, Mininet, and Quagga for obtaining the statistics of network traffic, the construction of SD-WAN topologies, and the selection of routing algorithms. The experimental results showed that the increase in topology complexity and node size leads to poor routing performance. Zhao et al. [20] provided an accurate queueing system of packet forwarding in an SD-WAN and measured the average packet delay through an OpenFlow switch to achieve an optimization model of controller cluster deployments in WANs.

3.2. Traffic Scheduling

Considering the granularity of traffic scheduling, related work can be divided into flow-level traffic scheduling [21,22,23,24,25,26] and application-level traffic scheduling [27,28,29,30,31,32]. Detailed optimization objectives of the research on traffic scheduling are shown in

Table 3. Detailed optimization objectives of the research on traffic scheduling.

Paper	Optimization Objectives
	Latency	Bandwidth	Utilization Rate	Load Balancing	Resilience	System Overhead	Network Overhead	Time Overhead
[1]		√	√	√	√	√	√
[13]		√	√		√	√	√
[21]				√		√	√
[22]		√	√				√
[23]	√				√	√
[24]	√	√		√
[25]	√				√
[26]	√							√
[27]	√					√	√
[28]	√						√
[29]	√				√	√	√
[30]		√	√	√
[31]	√			√
[32]	√					√		√

.

In terms of flow-level traffic scheduling, Guo et al. [21] proposed a threshold-based critical flow routing method to maintain SD-WAN load balancing with less controller synchronization. Ma et al. [22] proposed a distributed storage mechanism of two-dimensional routing in combination with SRv6 and a corresponding SRv6 header compression method to realize the lightweight deployment of SD-WAN multipath routing. Borgianni et al. [23] explored using reinforcement learning algorithms to predict network performance degradation and to change routing information proactively for improving SD-WANs’ overall network performance. Xin and Wang [24] presented a load balancing method by partitioning traffic demands into multiple groups within a scalable SD-WAN framework and defined a link-based optimization formulation under constraints of both bandwidth and latency. Ouamri et al. [25] formulated a QoS-oriented problem of joint optimization for SD-WAN average request delay and survivability, and then proposed a multi-agent deep Q-Network algorithm to redefine its reward function with the optimization objectives. Ghaderi et al. [26] redefined a neural network-based traffic encoding matrix and designed a deep-reinforcement-learning-based traffic engineering framework for SD-WANs.

In terms of application-level traffic scheduling, Fan et al. [27] proposed a relay node selection and routing approach to minimize the number of relay nodes under transmission latency constraints in a cloud-native SD-WAN and solved this combinatorial optimization problem based on constraint programming. Botta et al. [28,29] proposed a control and orchestration plane for SD-WANs based on a cooperative version of multi-agent reinforcement learning for dynamic overlay selection and accommodating diverse network policies with varying QoS and cost objectives. Quang et al. [30] proposed a global QoS policy optimization model to dynamically adjust the rate limits of applications based on their requirements according to the evolution of network conditions. Ouamri et al. [31] studied component migration to balance load between headquarters and branch sites by proposing an MPLS-based SD-WAN and formulating a non-linear binary program to jointly optimize load balancing and running costs. Du et al. [32] proposed a federated learning method to minimize the total time of routing and forwarding through well-designed client selection and scheduling in an SD-WAN, which significantly reduced the upload time of each iteration, with slight impacts on the number of iterations.

3.3. Failover and Recovery

Network failures in SD-WANs can be classified as node failures and link failures, with the corresponding studies being [33,34,35,36,37] and [38,39,40,41,42,43], respectively. Details of the research on failover and recovery are shown in

Table 4. Details of the research on failover and recovery.

Paper	Optimization Objective	Method	Algorithm	Evaluation
[33,34]	Robustness and availability	Switch-controller mapping	/	Simulation
[35,36,37]	Programmability recovery	Flow-controller mapping	Heuristic	Simulation
[38]	Time overhead	Protocol optimization	/	Simulation
[39,40]	Availability and resilience	/	/	Testbed
[41]	Throughput and resilience	Protocol optimization	/	Simulation
[42]	System and network overhead	Multi-objective optimization	Heuristic	Testbed
[43]	QoS and time overhead	Routing optimization	Reinforcement learning	Testbed

.

To solve the controller failure problem in SD-WANs, Altheide et al. [33,34] proposed a fully distributed SD-WAN control plane and achieved a highly robust traffic engineering solution to implement fine-grained global policies while remaining responsive even in the event of device failures, network failures, or network partitioning. Guo et al. [35] proposed a switch-level programmability recovery scheme based on high-end commercial SDN switches. A mixed-integer programming of joint-flow-mode selection and switch mapping problem was defined and solved with an efficient heuristic algorithm, representing an improved method based on their previous research [36,37].

Botta et al. [38] studied the effectiveness of the BFD protocol to monitor SD-WAN links for rapid failure detection and seamless failover, and then proposed an automatic tuning mechanism for the parameters of the BFD protocol to enhance the accuracy of detection and reduce the time of failover across diverse WAN types. Troia et al. [39,40] established two SD-WAN testbeds in a municipal network and a campus network, respectively, to prove the ability of SD-WANs to guarantee service availability and resilience in cases of network failure. Zhang et al. [41] presented a WAN-aware MPTCP to aggregate multiple WAN links into a virtualized large pipe for better resilience, thus minimizing application performance degradation under WAN link failures. Shojaee et al. [42] formulated the SD-WAN failure recovery problem as a multi-objective MILP optimization problem for all possible single-link failures and designed a software-defined proactive recovery mechanism to improve bandwidth allocation and switch-memory usage. Golani et al. [43] proposed a fault-tolerant reactive routing system for SD-WANs to monitor various network parameters in real time and to recover failure links if necessary.

4. Network Optimization and Systems of SD-WAN

Research on the network optimization and systems of SD-WANs mainly includes the controller placement problem [44,45,46,47,48,49,50,51] and SD-WAN-based systems [52,53,54,55,56,57,58,59,60,61], which are introduced in Section 4.1 and Section 4.2, respectively.

4.1. Controller Placement Problem

The controller placement problem was first proposed in [62] to determine the number of controllers and their locations in an SDN topology. The placement of controllers directly determines the overall network performance of software-defined networking architectures, especially for SDNs in WANs. Therefore, related controller placement in SD-WANs can be classified by optimization objectives.

The vast majority of related work focuses on the optimization of control latency. Dou et al. [44] defined a controller placement with a switch-controller mapping solution and developed a programmability explorer by calculating the programmability of critical flows at switches to optimize the control latency of SD-WANs. Adebayo et al. [45] addressed a switch-to-controller allocation problem that considered switch-to-controller latency and heterogeneity of controller capacities and proposed two neighborhood centrality-based algorithms to implement ideal allocation and placement. Qi et al. [46] optimized SD-WAN control latency by rationally placing controllers, establishing switch-controller mapping and developing a heuristic algorithm to achieve the trade-off between network performance and time complexity. Adekoya and Aneiba [47] applied and improved an evolutionary algorithm called Non-Dominated Sorting Genetic Algorithm III of Mechanical Engineering to achieve high convergence and the diversification of controller placement. Chakraborty et al. [48] designed a distributed scheme on coalition formation game and social choice theory which is able to optimally place controllers and periodically assess the placement of controllers based on real-time network traffic. Sminesh et al. [49] proposed a modified-density peak clustering algorithm to determine multi-controller placement and proved its effectiveness by comparing it to the hierarchical k-means, modified affinity propagation, and basic DP clustering algorithms in selected networks from the Internet Topology Zoo.

Other complementary optimization objectives of the controller placement problem in SD-WANs include the capacity of controllers [45,50], the failure of nodes [51], and the transmission latency of the data plane [27].

4.2. SD-WAN Based Systems

Research on SD-WAN-based systems can be divided into the design of SD-WAN-based systems [52,53,54,55,56] and evaluations of SD-WAN solutions [57,58,59,60,61].

In terms of the design of SD-WAN-based systems, Menoni et al. [52] proposed an SD-WAN embedded white-label solution with low cost and low energy consumption which is suitable for commercial and academic use. Borgianni et al. [53] proposed an innovative architecture by integrating an SD-WAN and Satellite 6G to support applications in remote areas or in high-latency environments. Elizabeth et al. [54] established a dynamic multipoint VPN solution for SD-WANs by applying open source protocols, namely multipoint generic routing encapsulation, IPsec (Internet Protocol Security) encryption, and the next hop resolution protocol. Ushakov et al. [55] proposed an SD-WAN-based solution for the Internet of Vehicles by integrating an SD-WAN API controller with the edge points of overlay networks and solving the problem of bandwidth overload of overlay networks with encrypted traffic. Scarpitta et al. [56] introduced SD-WAN scenarios, illustrated the principles of SDNs and NFV, and designed an open source implementation called EveryWAN.

In terms of the evaluation of SD-WAN solutions, Tiana et al. [57] tested an SD-WAN platform of a telecommunications company in Indonesia and proved that the SD-WAN offers significant improvements over traditional WANs, including increased network efficiency, better management and control, flexibility in deployment and scalability, increased security, and lower operating costs. Troia et al. [58] presented the performance evaluation results measured in an SD-WAN testbed deployed in the municipal network to prove the capabilities of service availability and network protection of an SD-WAN. Hussain et al. [59] presented their case studies on Google’s deployment of SD-WANs in data center networks and TMNA’s deployment of SD-WANs and provided suggestions and best practices for deploying and managing SD-WANs. Soejantono et al. [60] carried out failover and recovery tests on an IPsec-based SD-WAN instance deployed in Indonesia and obtained results indicating that an SD-WAN works appropriately when one of the multiple links is down. Hong et al. [61] presented the five-year evolution of B4, Google’s private software-defined WAN from the perspectives of hierarchical network topologies, traffic engineering, and solutions to network failures.

5. Service Orchestration of SD-WAN

Research on SD-WAN service orchestration mainly focuses on service function chaining (SFC) [63,64,65] and service orchestration platforms [28,66,67,68].

Regarding research on SFC, Jiang et al. [63] formulated an SFC problem considering the heterogeneity of geographically distributed SD-WANs and designed heuristic algorithms to deploy SFC requests in batches. Leivadeas et al. [64] established collaboration and information exchange between enterprise branches and networks by configuring security, data privacy, and routing services of Amazon SD-WAN services, examining and evaluating the overall performance. Zhang et al. [65] proposed a service offloading method for jointly allocating communication and computation resources based on cloud-edge collaboration in SD-WANs.

Regarding research on service orchestration platforms, Botta et al. [28] designed an SD-WAN control and orchestration plane for network policy orchestration in order to implement dynamic overlay selection and to guarantee on-demand network performance. Perez et al. [66] set up a flexible, resilient and cloud-native SD-WAN orchestration solution for enterprise and academic networks purely based on open source tools. Kone and Kora [67] put forward a practical approach for management and orchestration based on open source platforms and evaluated their proposed testbed by orchestrating services of the Voice over Internet Protocol. Balachandran et al. [68] proposed a blockchain-based orchestration framework to allow the SDN clients and vendors to create, manage, and execute services through an auditable and zero-trust based solution.

6. Security Issues of SD-WAN

Research on security issues includes improvements to cybersecurity methods for SD-WANs [69,70,71,72,73] and innovations relating to SD-WAN security frameworks [68,74,75,76,77].

In terms of improvements to cybersecurity methods for SD-WANs, Ergawy et al. [69] introduced a game-based theoretic approach to model potential attack scenarios and to drive a proactive moving target defense method to prevent the potential exploitation of SD-WANs. Zhang et al. [70] proposed a machine learning-based anomalous traffic detection framework to extract representative features directly from the raw traffic and to make real-time adjustments based on an evolving isolation forest in complex environments. Lembke et al. [71] proposed a secure network update protocol for SD-WANs to preserve security by authenticating network events, to provide reliability by replicating the control plane, and to maintain resilience by using a distributed ledger for failure detection. Satheesh et al. [72] applied a machine learning framework to detect Distributed Denial of Service (DDoS) attacks in SD-WAN environments and improved the random forest algorithm to obtain better performance in traffic classification. Fan et al. [73] built blockchain-coordinating controllers (BCCs) to secure the control channel of SD-WANs formed by the distributed controllers spread across multiple domains, providing resilience against security threats in the control plane and guaranteeing secure control communications, even when the credentials of n controllers are compromised.

In terms of innovations relating to SD-WAN security frameworks, Yiliyaer and Kim [74] studied a secure access service edge (SASE) framework via the comparison of an MPLS VPN and an SD-WAN, the introduction of a zero-trust architecture, and the detailed implementation of secure web gateways and a cloud access security broker. Szymanski et al. [75] presented a cybersecurity via determinism paradigm for IoT by designing a forwarding sub-layer for deterministic SD-WANs, along with services of access control, rate control, and isolation control for improvements to cybersecurity. Bustamante and Avila-Pesantez [76] compared the cybersecurity of SD-WANs in commercial mechanisms versus an open source solution. It was found that the commercial solution provides better security mechanisms regarding confidentiality, integrity, and availability, while the open source solution offers tools for adaptability to future threats thanks to the efforts of the community. Balachandran et al. [68] presented a blockchain-based authentication and access control framework for a multi-stakeholder SD-WAN infrastructure that adheres to the zero-trust security model. Lopez-Millan et al. [77] proposed a solution to manage IPsec SAs with SD-WAN architectures to avoid manual configuration in the network resources and to enable the reduced involvement of network administrators.

7. Trends and Challenges of SD-WAN

SD-WANs were born as a combinatorial technique of WANs, SDNs, and network function virtualization (NFV) to reinvent networks, services, and applications in WANs. To leverage the controllability and programmability of SDNs, SD-WANs were originally used in data center networks and ISP networks to achieve high-efficiency management and a high utilization bandwidth. Therefore, SD-WAN 1.0 was considered as a network optimization technology to enhance the efficiency of network management, operations, and utilization.

Due to automatability and extensibility of NFV, SD-WANs are now applied to establish ubiquitous interconnections [78] in cloud computing, edge computing, and the Internet of Everything by constructing overlay networks, orchestrating user-defined services, and scheduling system-defined resources. As a result, SD-WAN 2.0 has evolved into a network service provision platform, providing services such as network interconnections, network acceleration, remote access, and cybersecurity solutions. Compared to the traditional WANs, SD-WAN 2.0 is superior in its zero-touch network deployment, high-performance service provision, and auto-constructed security solutions.

SD-WAN 3.0 is considered as a promising distributed computing system combined with cutting-edge technologies of artificial intelligence, fog computing, edge computing, network security, and IoT. Therefore, the challenges relating to SD-WANs can be divided into three aspects. First, the optimization of distributed resource management and scheduling is compulsory to guarantee the QoS of services and applications. Second, improvements to parallel service orchestration, placement, and deployment are challenging due to complex network structures, multi-dimensional network elements, and abundant network services. Last but not least, cybersecurity issues of SD-WAN are vital to consider, meaning that software-defined security and software-defined perimeters for hierarchical topologies and heterogeneous devices are worthy of further investigation.

8. Conclusions

The existing surveys on SD-WANs are fragmented, covering specific problems only, and are not comprehensive with detailed research directions. This paper seeks to provide a systematic survey on SD-WANs by introducing the major research directions, presenting the research contents, and stating specific problems. Traffic engineering, network optimization and systems, service orchestration, and security issues are sequentially introduced as the four major research directions in SD-WANs. Specific research contents or problems relating to traffic measurement, traffic scheduling, failover and recovery, and the placement of controllers are illustrated and classified according to the corresponding state-of-the-art research.

It is found that studies in the past decade have mainly focused on traffic engineering, network optimization, and SD-WAN systems. However, more attention has been paid toward service orchestration and security issues in the last 5 years. The trends in the research indicate the development of SD-WANs that migrate from network optimization to service provision, revealing that SD-WANs are far more than software-defined networking. Therefore, SD-WANs are regarded as a variety of software-defined features in wide area networks, including their services relating to network interconnections, network acceleration, remote access, and cybersecurity solutions. In the near future, SD-WANs can be considered as promising distributed computing systems combined with cutting-edge technologies of artificial intelligence, fog computing, edge computing, network security, and IoT. Therefore, challenges in the related research include distributed resource management, parallel service orchestration, software-defined security issues, and software-defined perimeter problems.