A Comprehensive Survey of In-Band Control in SDN: Challenges and Opportunities

Abstract

Software-Defined Networking (SDN) is a thriving networking architecture that has gained popularity in recent years, particularly as an enabling technology to foster paradigms like edge computing. SDN separates the control and data planes, which are later on synchronised via a control protocol such as OpenFlow. In-band control is a type of SDN control plane deployment in which the control and data planes share the same physical network. It poses several challenges, such as security vulnerabilities, network congestion, or data loss. Nevertheless, despite these challenges, in-band control also presents significant opportunities, including improved network flexibility and programmability, reduced costs, and increased reliability. Benefiting from the previous advantages, diverse in-band control designs exist in the literature, with the objective of improving the operation of SDN networks. This paper surveys the different approaches that have been proposed so far towards the advance in in-band SDN control, based on four main categories: automatic routing, fast failure recovery, network bootstrapping, and distributed control. Across these categories, detailed summary tables and comparisons are presented, followed by a discussion on current trends a challenges in the field. Our conclusion is that the use of in-band control in SDN networks is expected to drive innovation and growth in the networking industry, but efforts for holistic and full-fledged proposals are still needed.

1. Introduction

During the last decade, the flourishing of SDN [1] has led to a wide range of novel network services and applications, including inter-disciplinary use cases, such as those envisioned by the fifth generation of mobile technologies (5G) and beyond [2]. The cornerstone of SDN is the possibility to separate the control and data planes, hence providing multiple advantages in network management and operational costs, among others.

Uncoupling both planes involves the utilization of a control communication protocol that is in charge of synchronising both planes. Considering the SDN architecture, this protocol is part of the Southbound Interface (SBI), formally known as Data-Controller Plane Interface (D-CPI), as defined by the Open Networking Foundation (ONF) [3]. Although the control protocol that initiated the SDN paradigm is OpenFlow [4], others currently exist, such as P4Runtime [5]; legacy ones (e.g., Simple Network Management Protocol (SNMP)) are even considered potential control protocols in the generalised SDN architectural model.

When deploying the control channel, it is possible to follow either an out-of-band or in-band approach, as depicted in Figure 1. On the one hand, the channel is out-of-band when the physical links leveraged for communication are different for the control and data planes, as illustrated in Figure 1a. In this scenario, the network requires at least one extra physical link per SDN device, apart from the data plane topology, for the control channel. On the other hand, the channel is in-band when physical links are shared between the control and data planes and no additional links are required. Additionally, hybrid-band control is defined when some links are shared but, at the same time, dedicated links for control also exist [6], as in the example shown in Figure 2.

Each type of deployments has its own advantages [6], and the selected approach mainly depends on the network scenario [7,8,9]. Some of the disadvantages of out-of-band are advantages of in-band and vice versa, while hybrid-band could be considered a trade-off among both approaches. We summarised the main features of each model in

Table 1. Features of in-band and out-of-band control.

Feature	Out-of-Band Control	In-Band Control
SDN device configuration	Simple	Complex
Control channel security	Safe, isolated channel	Risky, shared channel
Network maintenance and deployment costs	High	Low
Scalability	Poor	Good
Resilience	Costly (re-wire)	Fast failure recovery (easy traffic re-routing)

:

The current trend shows that in-band seems to be prominently used in SDN deployments [10], particularly for large networks in which the cost of using out-of-band might be exorbitant. Additionally, in-band control allows a wide range of new applications, particularly for constrained and hybrid SDN environments [11,12], in which out-of-band control is complex (or even impossible) to deploy. Some of the use cases leveraging in-band control include 5G [13] and Non-Terrestrial Networks (NTNs) [14], Internet of Things (IoT) (e.g., underwater networks [15], energy saving [16] or resource-constrained environments [17]), emergency and disaster situations [18,19], and enhanced network security [20].

Nevertheless, despite of the multiple benefits of in-band control in SDN scenarios, few efforts have been made to design a common and holistic in-band control protocol. This design should consider standard and popular platforms (both for SDN controllers and switches/devices) for a complete integration in current SDN deployments. Therefore, the aim of this survey is to provide the current state of the art in this area, describing current proposals, technologies used, advantages and disadvantages, etc., so that it can serve as a comprehensive report for researchers and network managers interested in this topic. Therefore, our goal is to provide a comprehensive survey of the topic and a comprehensive overview for academia and industry. Accordingly, the main contributions of our survey are

• A comprehensive list of works about in-band SDN control proposals.
• A classification of the surveyed works in terms of designed features, defining four categories.
• A chronological table per category stating achieved functionality and maturity of each proposal.
• A summarised analysis for each proposal, describing its main characteristics, implementation details, and results, if applicable.
• A final discussion and comparison of all works, stating potential research trends and future challenges in the field.

After providing an overview about the motivation and objectives of this survey in Section 1, the remainder of the manuscript is structured as follows. In Section 2, we review the related work (including other SDN surveys, and technologies and platforms associated with in-band SDN control) and summarise the contributions of this survey. In Section 3, we describe the methodology followed to look for related articles and current statistics about the topic. The main survey is performed in Section 4, in which we cover all previously found articles, first providing a classification. Afterwards, Section 5 discusses current research trends and future challenges according to the performed analysis. Finally, Section 6 concludes the survey.

2. Related Work and Contribution of This Survey

After reviewing the literature, there are several surveys on SDN. To begin, there are various general surveys with a high number of citations [1,21,22,23,24,25,26,27,28,29]: all these surveys have in common that they were published at the same time as the SDN wave. They cover the challenges and new aspects that SDN brought and how it shattered the conventional networking paradigm.

As a technology becomes trendy in a field, challenges, opportunities and threats emerge. Thus, as SDN networks emerged, so did works [30,31,32,33] that addressed the security issues of such networks. One very controversial aspect is the centralisation of all network control in a single entity. In these works, they indicate lines of work, challenges, risks, threats, and works that propose countermeasures to deal with the most common threats.

As soon as the new technology stops being trendy and starts to be implemented in the industry, it is necessary to start studying how to make the transition from the old paradigm to the SDN paradigm. It is therefore necessary to study and analyse the necessary intermediate steps, as it is impossible to replace all legacy equipment in a single step. This is where hybrid SDN architectures and the corresponding surveys where they are studied in detail come in [11,34,35].

Another aspect that has been addressed [36,37] in the SDN field is distributed systems where more than one control entity come into play to manage a group of SDN nodes. These systems solve many of the performance problems, bottlenecks, and threats that arise from centralisation, but they also bring with them challenges in managing shared resources, such as race conditions, and atomicity in instantiating rules in SDN nodes, among others. Finally, another topic that has been deeply explored [38,39] is the integration of other trendy technologies such as IoT with SDN, what challenges arise in the integration, and what benefits the IoT environments might derive from the SDN paradigm. Additionally, other specialised surveys analyse SDN programming languages [40], link failure recovery [41], or Quality of Service (QoS) [42].

The aforementioned works represent only a small part of the total number of surveys on SDN. Our intention is to highlight some of the most referenced works and the topics they cover, while emphasising that an analysis of all existing surveys on SDN is beyond the scope of this article due to their large number. All the publications mentioned above have in common that they do not cover the in-band control paradigm in depth. For this reason, this review is of particular interest as it brings together all the advances in this area in a single work. Nevertheless, to date and to the best of our knowledge, there has been no survey of in-band control in SDN, which is remarkable given its importance for real SDN deployments.

Finally, we would like to clarify that in-band control is not the same (and not even directly related) to in-band network telemetry [43]. Although the latter also emerged with the growth of SDN and programmable networks, in-band network telemetry is a technique that uses the data plane to directly drive the network measurement process, while in-band control is a model to deploy control channels leveraging the existing data plane channels. In summary, in-band telemetry is related with data plane devices (hardware), while in-band control is related with data plane channels/link (logic).

3. Survey Methodology and Statistics

In order to comprehensively survey the topic of in-band control in SDN, we performed a manual search (in which all works were checked one by one) using Google Scholar. The core keywords used for this search were: in band, in-band, SDN, Software-Defined Networking, Software-Defined Network, softwarized network, softwarization, control, communication, and survey. In all performed searches, all works were manually inspected (mainly by reviewing title and abstract), from most to less relevant (according to the default Google Scholar resultant list), until the search provided around 10 consecutive non-related works: we considered that a specific search would not produce more related results. Only works in English and Spanish (these were the languages that the authors fully understand) were considered.

The first lookup was executed in August 2021 and periodically repeated approximately every 3 months until the last one was performed at the end of January 2023, when this survey was finally sent for revision. Additionally, the authors subscribed to the topics sdn survey "in band"-"intitle:telemetry” in Google Scholar to obtain alerts about newly published topics in the field. As it can be seen, the keyword telemetry was avoided in our searches because it is not related to our survey, as mentioned in Section 2.

Once the survey was finalised, in January 2023, we performed an additional search of keywords to have a quick overview of the current trends in the field. This search was automatic this time, looking for the keywords SDN in-band in Google Scholar once again and counting the amount of published articles. The result is depicted in Figure 3, in which the blue line (with circles) represents the amount of works found in Google Scholar for those specific years. As it can be observed, the trend rapidly grows from 2012 to 2018 and stays stable in the later years. However, it is important to highlight that this is an automatic search and, as such, it might include as well works for in-band telemetry in some cases (as they were not explicitly excluded). In any case, it provides a clear overview of the trend.

Additionally, based on the previously examined works, which were classified into four main topics (as detailed in Section 4, four additional searches were performed as well including those classification items: SDN in-band recovery, SDN in-band routing, SDN in-band distributed, and SDN in-band bootstrap. As illustrated in Figure 3, each of them covers almost half of the total articles in the automatic search.

Finally, we measured the importance of the topic as the number of references each article had obtained. More specifically, Figure 4 depicts how many papers have a number of citations in the ranges 0, 1–10, 11–20, etc. As it can be observed, the majority of works have received few citations, between 1 and 10, or none at all.

4. A Survey of In-Band SDN Control in SDN

In this section, we survey all works that were classified into four main categories, which can be found in Section 4.2, Section 4.3, Section 4.4, Section 4.5. Prior to this description, Section 4.1 provides a preliminary overview of the survey, describing those four classification items and a final subsection focusing on the technologies and tools commonly used in SDN. Finally, in Section 4.6, other works that do not directly implement in-band control, but are closely related, are examined.

4.1. Preliminary Survey Overview and Classification

Before studying each of the surveyed works, we decided first to provide an overview with patterns and similarities found in the articles.

Among the motivations for each design, we discovered the main following ones, which are illustrated in Figure 5 as four categories:

• Automatic routing;
• Fast failure recovery;
• Network bootstrapping;
• Distributed control.

The three first categories could be seen as three main features that in-band control should provide, while the fourth is an orthogonal aspect. More specifically, the need for automatic routing in in-band control seems quite straightforward: while out-of-band control is usually implemented with direct links towards the controller (although it should not be necessarily like that), in-band control requires calculation of routes between SDN devices and the controller/s and vice versa. Fast failure recovery could be considered one of the main advantages of in-band; by simply selecting a different route in the data plane, the control communication can be recovered after a link or switch failure. However, the procedure to build these new routes should ideally be defined in accordance with the routing mechanism. Regarding network bootstrapping, it refers to the ability of setting up all required parameters (such as the connectivity details for the controller–switch communication) prior to startup of the system. Network bootstrapping is also a desirable feature for out-of-band control, but it is almost a must for in-band, since routes might vary depending on each deployment and the routing method applied. Finally, if the network has distributed control (or controllers), it implies that two or more controllers are involved in the control decisions and, therefore, a coordination mechanism should be defined for routing, failure recovery, and bootstrapping. Additionally, in-band control can also serve as a potential means for internal communication among controllers.

The previous four aspects were the most commonly found. For that reason, they were selected as the four main classification items of our survey. Nevertheless, some other features were also found in the literature, such as topology discovery, the possibility of combining both out-of-band and in-band as a hybrid-band deployment, and other design parameters such as scalability or QoS. Although network topology discovery is a key element for a number of necessary network functionalities, it was decided to not include it as a key aspect to be analysed in the survey. This is because, in general, it is usually a process that is independent of the in-band protocols, as some solutions include a topology discovery system/function, while others rely on independent topology discovery protocols that are used to obtain the necessary paths to compute the in-band routes. To delve into these aspects, let us consider Figure 2 again. The network represented in the figure contains seven SDN switches and two SDN controllers. The controllers are distributed and control type is hybrid-band, as both types are combined. For example, Controller A is connected to switches 1 and 3 via out-of-band control, but also to 4 and 7 via in-band control. The case of Controller B is similar, with some switches connected via out-of-band control, other ones via in-band; there are even some combining both option, such as switches 1 and 2, which can connect to Controller B using both types of control.

In this scenario, fast failure recovery can be implemented via restoration or protection (as defined by Sharma et al. [44]). For instance, switch 1 could be connected to Controller A and Controller B at the same time via out-of-band control, while maintaining a third in-band control route to Controller B routed through switch 2 for enhanced resiliency. Furthermore, bootstrapping could help to deploy all initial parameters and default setup and this might require topology discovery for it. Although it might seem that routing and topology discovery are both closely related, not all routing proposals might need to obtain the full topology; this is especially true if it is based on tree routing. For example, in Figure 2, all switches have at least one route towards a controller, but link 2–4, 3–6, 4–6, and 6–7 might be completely unknown for the control plane if a topology discovery function is not implemented.

Common Technologies and Tools

It is also important to highlight the most common technologies and tools found in the surveyed works in order to provide an overview of the protocols, controllers, switches and simulation, and emulation tools currently used in in-band SDN environments.

First of all, almost all proposals in the D-CPI/SBI use OpenFlow [4] protocol, while some of the most recent ones consider Programming Protocol-independent Packet Processors (P4) [45,46] and/or P4Runtime [5]. OpenFlow exchanges messages between the controller and its legacy switches to communicate the control and data planes. The controller indicates forwarding or metric rules to the switches by matching header fields. However, the reduced number of predefined OpenFlow rules limits its flexibility. This has led to new solutions such as P4 and P4Runtime, which create a new programmable data plane paradigm through a universal high-level language. It also replaces control and data plane communication with message exchange using Google Remote Procedure Call (gRPC), providing a fast and flexible communication interface.

In the case of SDN controllers, the most commonly leveraged ones are Open Network Operating System (ONOS) [47], OpenDaylight (ODL) [48], Ryu [49], and Floodlight, among others [1]. ONOS and ODL are high-performance controllers used in real-world deployments, while Ryu and Floodlight are simpler and intended for prototyping or low-performance use. However, there are also several proposals that simply simulate the behaviour of the controller instead of using a real one.

Regarding SDN devices/switches, the most common by far is Open vSwitch (OVS) [50]; however, Basic OpenFlow Userspace Software Switch (BOFUSS) [51] is also used in a few works. The target, Behavioral Model version 2 (BMv2) [52], is also used, particularly when P4 is considered. In most cases, these switches were deployed using the Mininet [53] emulation platform. As for simulated environments, OMNeT++ [54,55] was predominant, followed by ns-3 [56,57]; many others exist as well [58].

Finally, for the evaluation, diverse types of topologies have been considered, including simple linear, star or mesh networks, data center networks like fat tree or Clos, or other modeled networks (adjusted by a set of parameters) like Waxman [59] or Barabási–Albert [60].

According to this preliminary study, we have organised all works in four main subsections, and for each of them, all surveyed items have been summarised in a table. This table contains the year the article was published, its key features, evaluation tools, if its code is publicly available, and an overall contribution based on three stars. The contribution was qualitatively assigned based on all previous characteristics together with its impact in number of citations.

4.2. Automatic Routing

To start, we describe the different articles found in the literature that propose designs for automatic routing in in-band control in SDN. Although the majority of surveyed works provide a routing approach, some of them only focused on routing (and failure recovery in some cases). Those are the ones classified in this initial section. To provide a quick overview,

Table 2. Summary table of in-band proposals that support automatic routing.

Article	Year	Key Features	Evaluation & Tools	Available Code?	Overall Contribution
Sharma et al. [44]	2016	Key ideas for in-band control with DHCP, queueing and failure recovery	Emulation: NOX/BOFUSS/Pan-European, ring, star networks	—
Goltsmann et al. [61]	2017	Routing and single-link failure recovery	Analytical model	—
Khakhalin et al. [62]	2017	Routing and link failure recovery	Analytical model	—
Raza et al. [63]	2017	Routing and link failure recovery	Analytical model/Random networks	—
Gonzalez et al. [64]	2018	Routing and link failure recovery	Emulation: Ryu/OVS/Mesh network	—
Mohan et al. [65]	2018	Routing and link failure recovery with a security focus	Analytical model/Internet topologies	—
Görkemli et al. [66]	2018	Routing and link failure recovery	Emulation: POX/OVS/Ring-tree topology	—
Holzmann et al. [67]	2019	Distributed routing protocol for in-band SDN control	Simulation: OMNeT++/WAN/Zoo topology	—
Raza et al. [68]	2019	Path selection algorithm for routing using MPTCP	Simulation: C++/BRITE networks	—
Fan et al. [69]	2020	Routing, in-band connection with QoS	Simulation: Python Networkx. Emulation: ONOS/OVS/Docker/Fat tree and random networks	—
Ningyuan et al. [70]	2021	Routing and link failure recovery in LEO networks	Simulation: Walker–Delta constellations	—
Kumazoe et al. [71]	2022	Routing in P4BMv2-based network	Real testbed: BMv2/Linear network	—

summarises them all. Afterwards, we individually examine each work.

Sharma et al. [44] present the basic ideas for the design of in-band control in OpenFlow networks. They analyse the extent to which diverse SDN switches could support this type of control and implement a prototype with NOX and TrafficLab1.3 (currently known as BOFUSS [51]), tested with diverse topologies. Additionally, they examine how failure recovery is a cornerstone in this type of control, presenting the restoration and protection methods to implement it. This article is particularly relevant in the field, as it defined the foundations for automatic and resilient bootstrapping for in-band control in SDN.

Goltsmann et al. [61] present the Intermediate System to Controller (ICS) protocol, focused on resilient connectivity, while keeping protocol overhead low, for in-band control in SDN. The design relies on the fact that communication only occurs between the controller and the rest of nodes—not among all pairs of nodes. Considering this aspect, they design a labelled spanning tree and its behavior when a single link failure occurs. However, it requires some convergence time and multiple failures are not considered.

Khakhalin et al. [62] formulate an algorithm for automatic in-band control routing which is resilient to multiple failures. However, it requires the assignment of a unique label to every switch. This becomes quite complex when multiple failures occurs because loops should be avoided without a global knowledge of paths. Moreover, no reference implementation with a SDN controller is provided.

Raza et al. [63] focus on the in-band channel availability and propose the use of MultiPath TCP (MPTCP) instead of simply Transmission Control Protocol (TCP). High availability is achieved using as many disjoint paths to a switch as possible by selecting a set of adjacent switches to the controller. The search of this adjacent switches might be time consuming, so they only propose and evaluate two simple heuristic algorithms.

Gonzalez et al. [64] design an in-band OpenFlow channel through MPTCP which leverages the multiple paths for enhanced resilience. However, it requires an out-of-band channel for the initial setup.

Mohan et al. [65] propose a control path routing approach that selects two node-disjoint control paths for every switch in the network. In that way, a malicious node can be detected based on the normal packet-in messages sent on the control paths. Then, the controller can send explicit test flows to determine and isolate the malicious switch. They derive an optimization formula to provide a routing solution that minimises the average number of intermediate nodes while satisfying the malicious switch detection and resilience constraints. They demonstrate the effectiveness of the proposed approach through numerical analysis. Their results show that the proposed approach enables faster malicious switch detection with less control overhead compared to a previous approach. However, the solution does not scale for large networks due to the high number of decision variables.

Görkemli et al. [66] define a dynamic control plane architecture to redistribute and scale controller load and re-route in-band control traffic using control flow tables depending on underlying the network topology and the requirements of the applications running on top. They test their dynamic paradigm on a virtualised network with an arbitrary number of controllers (running a modified version of POX) and OVS switches in hybrid ring-tree and balanced binary tree topologies. They achieve greater efficiency in the use of the controller’s resources than with a static approach.

Holzmann et al. [67] present Izzy, a distributed routing protocol for maintaining robust connectivity of in-band SDN control channels. Izzy is based on a combination of a spanning tree and topology-dependent temporary addresses called locators. The solution is divided into three modules; the first one, called TREE, is responsible for maintaining the spanning tree structure by assigning locators. The second module, known as UPDATER, is responsible for coordinating and updating the network’s forwarding tables. Finally, the Fast ReRoute (FRR) module is in charge of maintaining backup routes. The evaluation of Izzy has been carried out in OMNeT++ and shows recovery times under 100 ms in WAN topologies of 100 nodes.

Raza et al. [68] define an algorithm to create robust control paths when leveraging MPTCP for in-band communication in SDN. Their algorithm looks for the best set of switches to route through the shortest and disjoint paths in the network. They compute the sum of disjoint path lengths and computation time and compare their proposal with two baseline algorithms, proving better results. However, they only test it via simulation and leave the real implementation as future work.

Fan et al. [69] propose a centralised routing solution to improve the QoS of in-band control traffic in terms of bandwidth. They analyse the in-band control traffic, finding out that it is periodic and needs a very small bandwidth: around 120 kbits every 5 s. Hence, they place the in-band connection on the path with the most available bandwidth. They design a centralised routing algorithm in the SDN controller to compute the optimal path from the network topology and current consumed bandwidth at every link. However, their simulations show that it does not scale to generic large networks. They choose to stick to a Fat Tree Data Center (DC) network and a special network designed with higher connectivity and randomness, in which they clearly improve the Round-Trip Time (RTT) and the Packet Loss Ratio (PLR) compared with a L2 Spanning Tree Protocol (STP).

Ningyuan et al. [70] acknowledge the need for reliable in-band control in Low Earth Orbit (LEO) constellation systems and propose a dual-layer architecture where the upper-layer control plane provides reliable in-band control paths for the lower-SDN control plane. This plane, in turn, deploys virtual out-of-band control in the data plane, in spite of the constantly changing of network topology. In order to provide reliable control traffic transmission, a redundant path-planning algorithm is further proposed. They evaluate the framework with a simulation which proves much better satellites accessibility and has barely any inaccessible satellites under massive link faults.

Kumazoe et al. [71] design an in-band control channel communication scheme for P4, in which control messages are piggybacked onto data packets. The target switch is based on the BMv2, and their evaluation shows performance degradation in data forwarding, which they envision to be solved as future research work.

4.3. Fast Failure Recovery

This section covers a set of surveyed works exclusively focused on fast failure recovery methods for in-band control in SDN. These proposals might complement the rest: particularly, those that did not envision failure recovery in the first place.

Table 3. Summary table of in-band proposals that support fast failure recovery.

Article	Year	Key Features	Evaluation & Tools	Available Code?	Overall Contribution
Huang et al. [72]	2016	Protection method for link failure recovery	Simulation: Python	—
Park et al. [73]	2017	Restoration method for link failure recovery	Real testbed: ONOS/Raspberry Pi, OVS/Mesh network	—
Akhtar et al. [74]	2018	Mechanism to avoid retransmitions of in-band packets	Emulation: Ryu/Mininet/Linear network	—
An et al. [75]	2019	Protection method for interferences in in-band wireless networks	Real testbed: ONOS/Raspberry Pi/Mesh network (IEEE 802.11)	—
Alowa et al. [76]	2020	Protection method for link failure recovery	Simulation: C++/SNDlib networks	—
Ochoa-Aday et al. [77]	2020	Fast multiple-link failure recovery	Simulation: OMNeT++/Atlanta, Sun, Pioro networks	—

summarises all works in this section, which are described in detail in the remainder of this section.

Huang et al. [72] design a protection method for failures in the SDN control channel in Heterogeneous Networks (HetNets), usually implemented as in-band. Under a link failure, existing approaches rely on local rerouting that can bring congestion in the neighbouring links. They propose a weighted cost-minimization problem, where the traffic load balancing and path cost are jointly considered. This is an NP-hard problem and their solution is a near-optimal Markov approximation. After simulation in Python with diverse network types, number of controllers, and multiple failures, it proves to have better results (in resource utilisation and recovery time) than other standard routing algorithms.

Park et al. [73] define an algorithm for fast in-band control channel recovery after link failure. On receiving a port down message from the failure affected switch, the controller finds the shortest paths to the affected switch and searches the target switches (Tswitch) where the flow addition or modification is required. Then, the controller tests the shortest paths, transmitting 1000 flows on each paths. After that, it sends an special check message (request-reply) to each “Tswitch” switch to measure response time. Finally, it selects the path with the smallest delay. They compare it with other conventional detour approaches and it reduces both the recovery time and the detour traffic used for it.

Akhtar et al. [74] focus their proposal in mitigating the effects of delay and losses on the control plane traffic. They propose to use Network Coding (NC) on control flows to reduce both the number of transmissions in case of packet losses and the delay incurred. A NC switch combines (codes) packets to the same destination in a new packet before forwarding, which adds some redundancy, on a per link basis. In case of a packet loss the receiver can recover from it during the decoding process, eliminating end-to-end multiple re-transmissions. The drawback of NC is an additional delay to encode and decode packets and a slight packet size increase. They tested NC in Mininet and concluded that it overcomes packet losses satisfactorily and, at the same time, it is also able to reduce the delay.

An et al. [75] consider a protection method that is required for potential interferences in in-band wireless SDN deployments. For this reason, they design, implement, and evaluate a proposal using ONOS and Raspberry Pi boards in an IEEE 802.11 network. They test it with a six-node mesh network, but do not provide any code repository.

Alowa et al. [76] propose an in-band control protection system based on finding a set of ideal paths for the control channel. They simulate it in diverse SNDlib networks, but do not test it with real and multiple SDN controllers.

Ochoa-Aday et al. [77] define Self-Healing Protocol (SHP) as a protocol with minimum overhead and no controller intervention. The protocol is capable of recovering in-band control plane from multiple failures in multiple-controller environments, resulting in recovery times below 20 μs, which accomplishes the 50 ms requirement of carrier-grade networks [78].

4.4. Network Bootstrapping

This section examines works that include network bootstrapping in their design. Additionally, some of them might provide automatic routing and/or failure recovery as well. As a summary,

Table 4. Summary table of in-band proposals that support network boostrapping.

Article	Year	Key Features	Evaluation & Tools	Available Code?	Overall contribution
Sharma et al. [79]	2013	Bootstrapping and routing	Emulation: NOX/OVS/Linux namespaces/BT network, ring and mesh networks	—
Tu et al. [80]	2014	Bootstrapping, routing and link failure	Real testbed: Proprietary controller/1 GE and 10 GE generic switches/Spine-leaf network	—
Suo et al. [6]	2016	Bootstrapping, routing, link failure recovery and hybrid control mode	Emulation: Floodlight/OVS/Mininet/Fat tree network	—
Heise et al. [81]	2017	Bootstrapping, routing and link failure	Emulation: Ryu/OVS/Mininet/linear, ring and mesh networks	—
Su et al. [82]	2017	Bootstrapping, routing and link failure recovery	Emulation: ODL, Floodlight/OVS/Fat-tree network	—
Bentstuen et al. [83]	2018	Bootstrapping and routing	Emulation: ONOS/OVS/Linear network	—
Asadujjaman et al. [84]	2018	Bootstrapping, routing and link failure recovery	Simulation: OMNeT++	—
Lopez-Pajares et al. [85,86]	2020	Bootstrapping, routing and link failure recovery	Simulation: OMNeT++; Emulation: ONOS/ BOFUSS/Waxman, Barabasi-Albert networks	✓ (GitHub)
Sakic et al. [87]	2020	Bootstrapping, routing and link failure recovery	Emulation: ODL/OVS/line, star, ring, mesh networks	✓ (GitHub)
Silva-Freitas et al. [88]	2020	Bootstrapping and routing	Analytical model	—
Wu et al. [89]	2021	Bootstrapping	Emulation: Ryu/OVS, Mininet/Fat tree network	—
Li et al. [90]	2021	Bootstrapping and routing	Emulation: OVS	—
Álvarez-Horcajo et al. [91]	2022	Bootstrapping, routing and link failure recovery	Emulation: ONOS/BOFUSS/Waxman, Barabasi- Albert networks	✓ (GitHub)
Wong et al. [92]	2023	Bootstrapping, routing, link failure recovery, topology discovery and network monitoring	Emulation: BMv2/Testbed/P4Runtime	✓ (GitHub)

gathers all works, ordered by publication year.

Sharma et al. [79] devise an automatic bootstrapping method to set up the in-band control connection. To the best of our knowledge, they were the first to implement an in-band configuration channel for OpenFlow. They provide a very detailed and thorough explanation of the bootstrapping process. The switch relies on Dynamic Host Configuration Protocol (DHCP) to learn the IP address and Address Resolution Protocol (ARP) to learn the MAC address of the controller. The controller uses packet probes similar to Link Layer Discovery Protocol (LLDP) to acquire the full network topology. The method is tested on emulated networks of different shapes (linear, ring, star, and mesh topologies) and sizes, using the NOX controller. The tests show minimal bootstrapping times and good scalability response.

Tu et al. [80] apply an in-band SDN control plane in which they use standard Ethernet switches on a DC network for the communication of Virtual Machines (VMs). They place a special software agent in each server and a central SDN controller with a directory server and a route algorithm server to control all communications. Based on the in-band control plane, they could also implement load balancing and fast failure recovery.

Suo et al. [6] devise a new in-band control plane called ERIC, a hybrid-band scalable control designed for DC networks. They use pre-computed rules installed in the switches, both for the initial bootstrapping and in case of link failures or network congestion; however, they do not provide further details about it. They tested ERIC against the out-of-band approach on a Mininet simulator with the Floodlight controller and show similar RTT with a 9% throughput loss.

Heise et al. [81] propose a SDN in-band control plane to be deployed on standard avionic and industrial Ethernet networks to simplify network configuration and save additional cabling, which otherwise, means weight, complexity, and cost. Using the OpenFlow protocol in aeronautical use-cases requires: (1) the network has to be configurable via in-band, (2) the configuration must be done within a certain time, and (3) the configuration traffic must not interfere with data traffic. Two mechanisms are devised: (1) when a switch starts, it needs some preinstalled rules to connect to the controller, while at the same time, packet storm should be avoided; (2) a limitation to the in-band channel traffic based on traffic policing and admission-control techniques. They implement a proof-of-concept in Mininet and evaluate bootstrapping, fail-over times, and the influence of rate limitation in the overall configuration time under linear, ring, and mesh topologies.

Su et al. [82] define FASIC, which permits automatic bootstrapping of an OVS-based network by leveraging OVS Database (OVSDB), which allows the configuration of the SBI in the ODL controller. With the Floodlight controller, they monitor the congestion of the in-band connections. They use the ODL controller as a manager to move in-band connections from congested links, as through the OVSDB protocol, they send rules to OVS switches. They test it on a fat-tree network with ODL and Floodlight, proving recovery times that range from 0 to 5 s (faster than using stand-alone OVS, without OVSDB).

Bentstuen et al. [83] propose a technique to set up in-band control connections for dynamic environments like defense and emergency networks. They split the bootstrapping process into three steps: (1) to identify and register the new switches, (2) to set up flow rules on intermediate switches to enable a control channel, and (3) to discover and maintain the link topology database. Step 1 can be done using a registration protocol such as ARP or DHCP. In step 2, the connection is set up starting from the switch using either TCP or Transport Layer Security (TLS). The necessary flow rules on the intermediate switches must be preinstalled by the controller. Finally, for step 3, they propose to use a standard protocol like LLDP. They test their proposal using ONOS and OVS. However, they only check the case of a link failure but do not take into account the case of a node failure.

Asadujjaman et al. [84] propose a protocol for bootstrapping and routing in-band control connections, with a reactive approach for multiple link failure recovery. The only drawback is that controller-to-switch communication is source-routed, which increases packet overhead and should be improved for scalability reasons.

Lopez-Pajares et al. [85,86] present Amaru, an in-band protocol that provides bootstrapping and scalable shortest path routing for SDN networks, as well as multiple protection routes for link failures that guarantee very low fault recovery times with almost no overhead. The proposal leverages network exploration techniques to propagate a packet from the root switch (the one directly connected to the controller) to the rest of the network, in order to collect topological information, while installing the routes. In this way, each device in the network can reach the controller with the information contained in the exploration packets via various routes in order to establish the in-band control channel. However, it requires slight modifications in the SDN switches to be supported.

Sakic et al. [87] focus on the design of a full-fledged in-band control channel, considering it should implement features such as automatic bootstrapping, controller provisioning, multiple controller support, and fault resilience. It should also not be dependent on additional protocols or proprietary switch extensions. Based on these requirements, they design two approaches. The first option uses a pre-configuration to build a tree in sequential order by progressively adding new devices to the tree according to their distance from the controller. The second option relies on the Rapid Spanning Tree Protocol (RSTP) protocol to build a minimum-cost tree from the controller. Once the control channel is built using the RSTP tree, the tree is removed to avoid interference with the SDN network. Both options are very slow, providing convergence times of a few seconds, and they do not test failure recovery times.

Silva-Freitas et al. [88] propose a bootstrap protocol that provides network topology discovery and an automatic set of in-band control connections. Motivated by the high message number of OpenFlow Discovery Protocol (OFDP) and its security concerns, they propose ConForm, which is based on a spanning tree rooted at controller that spans towards every switch in the network. Initially, each switch is manually configured with a unique device ID and a cryptographic key, which only the controller can validate. When a SDN starts up, it does not forward any packet and, instead, it broadcasts a special message (reg_req) (to the controller) with its key. The controller initially only receives that message from the directly connected switches, authenticates them, and replies with their assigned ID. The process is repeated with the rest of the switches. There are additional steps to discover the topology than can be consulted in the paper. They provide an analytical model to calculate the number of exchanged messages, which is lower than other topology discovery protocols such as LLDP, OFDP, and OFDPv2. Additionally, ConForm has motivated the design of other configuration protocols, like the ETArch Switch Configuration Protocol (ETSCP) [93].

Wu et al. [89] devise a mechanism called rXstp to set up the in-band control connection with the controller. rXstp uses RSTP besides the basic layer 2 forwarding capacity of a switch to prevent loops. The node directly connected to the controller is the root node, so it is necessary to manually configure its switch priority to be the highest. When there are no more RSTP proposal messages in the network, it means that the network has converged (it is stable) and the in-band connections can be set up. The mechanism also uses a modified RSTP to allow the controller to discover the network topology. They implemented rXstp using OVS and the Ryu controller with Mininet. They compare rXstp with a SDN network that uses RSTP to set a topology and in-band control connections. They then use OFDP to discover the topology. The latter uses more messages and only discovers the tree links. This proposal only works with a single SDN controller.

Li et al. [90] propose One-Pass IBAB, a bootstrapping method for SDN in-band control, able to deploy control paths for 50 switches in 2 s. For this purpose, it builds a spanning tree rooted at the controller. If multiple controllers coexist, only the first messages exchanged are considered, so the switch connects to the first controller it is aware of. The authors envision fast failure recovery as future work.

Álvarez-Horcajo et al. [91] define an in-band control protocol with emphasis in hybrid networks (composed of both SDN and non-SDN) and also heterogeneous (both wired and wireless) devices. They evaluate their design with Waxman and Barabasi–Albert network topologies.

Wong et al. [92] propose in-band control plane called P4IBN for P4 networks with network bootstrapping, status monitoring, topology discovery, and fast failover. Among the challenges it manages to overcome is the fact that unlike OpenFlow switches, P4 switches are passive. The controller has to actively search for new P4 switches and initiate the control channel. Finally, they have been able to modify the restriction that a P4 switch can only have one control port to communicate with the controller. The evaluation was carried out using emulation and a Linux testbed. The results of the experiments indicate that the in-band control system proposed effectively carries out network bootstrapping and accomplishes rapid failover, even being able to handle multiple simultaneous failures.

4.5. Distributed Control

Finally, this section compiles surveyed works related to the use of distributed controllers (regardless of whether they provide automatic routing, failure recovery, or network bootstrapping). All articles are summarised in

Table 5. Summary table of in-band proposals that support distributed control.

Article	Year	Key Features	Evaluation & Tools	Available Code?	Overall contribution
Schiff et al. [94]	2015	Bootstrapping, routing and link failure recovery, and controller-to-controller connectivity	Simulation: Java/Fat-tree network	—
Schiff et al. [95]	2016	Distributed control plane that supports discovery topology and controller-to-controller connectivity	Simulation: Java	—
Schiff et al. [96]	2016	Shared memory OF-table system with atomic operations for distributed network control	Emulation: Mininet/OVS/ovs-ofctl Python Wrapper	✓ (GitHub)
Görkemli et al. [97]	2016	Distributed dynamic control plane architecture	Emulation: Floodlight/OVS/Mininet	—
Hark et al. [98]	2017	Control traffic isolation mechanisms and controller-to-controller connectivity	Emulation: Floodlight/OVS/Mininet	✓ (GitHub)
Canini et al. [99]	2017	Bootstrapping and topology discovery	Analytical model	—
Canini et al. [100]	2018	Restoration method for link-node-controller failure recovery in multi-controller networks	Emulation: Floodlight/OVS/Mininet	✓ (GitHub)
Kwan-Yee et al. [101]	2018	Fast failure recovery mechanism in multi-controller networks	Real testbed: ONOS/HP5900	—
Holzmann et al. [102]	2018	Routing for distributed controller clusters	Analytical model	—
Canini et al. [103]	2022	Bootstrapping, routing, and link failure recovery	Emulation: Floodlight/Mininet, OVS/B4, Clos, EBONE+ networks	✓ (GitHub)

and, later on, explained in detail in the rest of the section.

Schiff et al. [94] present Medieval, which allows the switches in the network to set up an in-band connection to a single controller and also to establish routes to support connectivity among controllers. Medieval sets up the in-band control plane by creating and maintaining two spanning trees for each controller: (1) A “per-region” spanning tree spans over the region owned by a controller. The region owned by a controller is a connected graph containing the controller and the switches it controls. (2) A “network-wide” spanning tree spans over the whole network to allow controller-to-controller connectivity. However, they do not detail how the controller-to-controller connectivity is achieved. The switches need a pre-configured controller IP and a set of a priori rules for the system to work. They provide a preliminary evaluation based on a Java emulator prototype.

Schiff et al. [95] propose a plug and play distributed control plane which supports automatic topology discovery and management, as well as flexible controller membership: controllers can be added and removed dynamically. This is achieved by managing two STP trees, one for membership and is a switch with controller, and the other, which provides connectivity between controllers. The implementation relies on OpenFlow and pre-configured rules at startup, so that the network is autonomously self-stabilising. Network resilience is achieved by applying timeout policies and low-priority rules that will only take effect in the event that a switch runs out of an associated controller. Additionally, it uses ARP for topology discovery.

Schiff et al. [96] propose an in-band synchronization framework for a distributed control plane based on atomic transactions. The synchronisation primitives and atomic operations have been designed so that they can be implemented exclusively with OpenFlow. The distributed control framework could be deployed without modifying any of the code of current software switches. It makes use of OF tables to generate a shared memory system to manage the distributed control plane of the network. In order to avoid inconsistencies, a series of synchronisation mechanisms have been proposed in addition to atomic operations. The validation has been carried out using Mininet and OVS switches to test the atomic operations described above. All three previous works seem to follow a progressive evolution of their preliminary idea for distributed SDN control.

Görkemli et al. [97] propose a dynamic control plane architecture for SDN networks that can adapt to changes in high controller load and new flow demands. The proposed architecture allows the activation and deactivation of controllers on-the-fly based on automatic measurement of network control traffic and changing service requirements. The authors describe a series of base scenarios and operations that can be performed with this dynamic data plane management architecture. The proposed architecture was evaluated using Mininet, OVS switches, and Floodlight as the controller. The results show that the architecture effectively addresses bottlenecks on switch–controller links and handles high CPU usage in controllers.

Hark et al. [98] propose a low-cost inter-controller communication service, which requires few resources in terms of state, communication overhead and computational complexity, and provides isolation between control and data plane traffic using shared in-band channels. The implementation has been carried out using the Floodlight controller. It leverages port-changed event messages to embed additional information that controllers can manage to decide the best route among them. Additionally, Virtual Local Area Networks (VLANs) are employed to isolate the control traffic. They study the messages generated for the convergence of the bi-directional communication channel between controllers and its subsequent activation. They observe that the number of messages grows linearly with the number of controllers in the network.

Canini et al. [99] present two procedures to ensure that each switch in the SDN network obtains routes to support connectivity, both between every switch and all controllers, and also among controllers. These procedures are classified according to the use or lack of usage of the timeouts available in the OpenFlow tables. In the methodology that requires timeouts, low-priority rules have been used together with pre-configured anycast addresses so that the switches periodically try to connect to a new controller when they are unmanaged. The second procedure is based on running a topology discovery algorithm on each controller and iteratively adding each switch to each controller $p_i$ on a first-come, first-served basis.

Canini et al. [100] present Renaissance as an evolution of their previously mentioned work, which is a group of algorithms for an in-band and distributed control plane for SDNs designed to overcome the shortcomings of Medieval [94]. It is not self-stabilising because its design depends on the presence of non-corrupted configuration data (e.g., the controller’s IP addresses). This solution ensures that, after the occurrence of an arbitrary combination of failures, every non-faulty SDN controller can eventually reach any switch in the network within a bounded communication delay and every switch is managed by at least one non-faulty controller. The validation is accomplished using Mininet, OVS, and Floodlight. The extensive, performed tests show start-up and recovery times for different topologies (Telstra, EBONE, and Exodus). Their prototype experiments exhibit promising results. Complementarily, M. Tran, in his PhD thesis [104], also implements Renaissance. Preliminary tests measure bootstrapping time, number of messages, and recovery time in Mininet using OVS under DC networks. These tests show promising results as well.

Kwan-Yee et al. [101] present a software solution scheme for fast failure detection and fast failure location identification in multi-controller SDN networks. Multiple monitoring probing cycles that cover all links and nodes in the network are performed. When a failure is detected in a monitoring cycle, the controller initiates failure location identification processes to identify the failure type and the failure location. In terms of validation, they set up a real network composed of two servers running two ONOS server instances each, HP5900 switches, and two other servers as host machines. The experimental results show that the average time to recover from any single device failure is less than 50 ms, even if the failure disconnects both working control and data channels at the same time.

Holzmann et al. [102] present Seedling, an algorithm that divides the controllers managing an SDN network into groups based on their proximity. This work is the evolution of another work by the same authors, in which they presented Izzy [67], another algorithm that provided robust connectivity between controllers and all managed switches by means of shortest-path spanning trees. In this article, they propose to use Seedling to save computational cost, memory in the forwarding tables, and all the overhead that it entails at the traffic level, instead of running instances of Izzy on each controller.

Canini et al. [103] present an enhanced and comprehensively evaluated version of Renaissance [100], an in-band and distributed control plane for SDN with a particular focus on network robustness and self-stabilisation. As already known, Renaissance is based on Medieval [94] (proposed by common authors), but the former is self-stabilised, as it recovers from failures in a bounded time. Renaissance’s paper provides a rigorous algorithm and an analytical proof of self-stabilisation. They comprehensively evaluate their algorithm in emulated scenarios with multiple types of networks and measuring various parameters (bootstrapping, failure recovery, etc.). Additionally, their code (based on the Floodlight controller) is publicly available.

4.6. Other Related Approaches

Although not directly implementing in-band control, Municio et al. [105] present Whisper, which provides end-to-end programmability for networks composed of IPv6 over the Time Synchronized Channel Hopping (TSCH) mode of IEEE 802.15.4e (6TiSCH) and wired devices. This is particularly interesting for IoT networks, in which some components might be constrained in energy or battery and out-of-band control is unfeasible. Therefore, Whisper aims to act as a middleware capable of applying directives from the SDN controllers by translating them into distributed routing messages understood by the IoT devices.

5. Discussion: Research Trends and Future Challenges

According to the performed survey, automatic routing and bootstrapping for in-band control in SDN is still an ongoing research field. Different proposals tackle both features using different techniques. Several proposals provide comprehensive implementations and evaluation, even with a publicly available code. Regarding failure recovery, some proposals exist, but just a few guarantee low recovery times with reduced overhead. From our point of view, the area could still be improved, particularly if combined with high-performance routing and automatic bootstrapping and if they are implemented in real testbeds (or at least emulated and deployed with SDN popular platforms in industry environments, like ONOS). Finally, distributed control planes still require additional research efforts. Even if several works are mentioned in this survey, most of them belong to a few authors.

Regarding research trends and future challenges, we envision potential research topics, including

• Considering IoT and Low-power and Lossy Networks (LLNs) as a key use case for leveraging in-band control, since network devices in these scenarios are usually constrained and cannot fully deploy out-of-band SDN control.
• Testing in-band approaches in big telecommunication networks (even if emulated) to check the feasibility of these approaches.
• Leveraging in-band as an East/Westbound Interface (EWBI) protocol to grant an effective and reliable communication among distributed controllers (which are the most common deployment option). This aspect is barely covered in the current literature [106].
• Evaluating potential security risks of in-band control and methods to tackle them, as just a few works mention it.
• Measuring Key Performance Indicators (KPIs) to compare in-band, out-of-band and hybrid-band approaches, as well as their implementation and compatibility with diverse SDN frameworks. Currently, few works tried to provide a holistic overview about it [7,107,108].
• Improving in-band connection routing with Artificial Intelligence (AI)/Machine Learning (ML). In computer science, AI is also known as machine intelligence. ML is a category of AI based on natural intelligence that can learn from data, make decisions, identify patterns, and perform various actions with less human intervention. ML can potentially be used to solve many problems in networking, including design, implementation, performance, and verification [109]. ML can be applied to improve in-band connectivity in many situations, such as finding the path with the highest QoS [69], finding disjoint paths for MPTCP [63,68], or improving network security by detecting a malicious node [65]. The use of ML for routing optimisation in SDN can be found in this survey [109]. A good example is the proposal by Chen [110]. They present a new routing algorithm that they implemented in a real SDN network with better results than other routing algorithms, such as OSPF and Least Loaded (LL). Ouamri et al [111] provides another example of using ML on a WAN SDN for load balancing optimisation, which could also be used for in-band backup links. ML should also be used to improve network resilience by providing some alternatives or backup paths for the in-band links.

Finally, we would like to highlight the importance of designing a full-fledged in-band protocol that covers all four aspects surveyed in this article and comprehensively integrates with an open source SDN controller like ONOS, ODL, or even TeraflowSDN [112], which was recently proposed as a SDN controller closely aligned with industry requirements and is a part of ETSI’s Network Function Virtualization (NFV) and Multi-access Edge Computing (MEC) ecosystems. Providing such a holistic design would facilitate the actual usage of automatic in-band control approaches. In the meantime, while partial solutions are published, industry might still manually deploy in-band control in SDN. Nevertheless, we are aware this is an ambitious objective and would imply collaboration among more than one research group together with standardisation bodies.

6. Conclusions

In this survey, we examined all existing works in relation to in-band control in SDN. We also showed that there was some tendency in the literature for the referenced keywords SDN and in-band to grow from 2012 to 2018 and stay stable in the later years.

According to the analysis, we classified them in four main categories: (1) automatic routing: in-band control requires calculation of routes between SDN devices and the controller/s and vice versa, (2) fast failure recovery: by selecting a different route in the data plane, the control communication can be recovered after a link or switch failure, (3) network bootstrapping: refers to the ability of setting up all required parameters before starting the system, and (4) distributed control: which implies that two or more controllers are involved in the control decisions and, therefore, a coordination mechanism should be defined for routing, failure recovery, and bootstrapping.

In order to provide a comprehensive overview, we have summarised them in one table per category, including a qualitative score based on their overall contribution. These tables show the year of publication of the article, its main characteristics, the evaluation tools, whether its code is publicly available, and an overall contribution based on three stars, which was qualitatively assigned on the basis of all the previous characteristics, together with its impact in terms of number of citations.

According to our investigation, research efforts are still needed for scalable fast failure recovery and the use of in-band in distributed control scenarios. Another line of research to be promoted is the application of ML to improve in-band routing links, network security, and failure recovery.

As future research trends and challenges, we envision the importance of IoT in which some components might be constrained in energy or battery and out-of-band control is unfeasible, leveraging in-band as an EWBI protocol for controller clusters, and research on tackling potential security risks. Furthermore, most proposals cover partial aspects or are not fully integrated into real SDN platforms and devices. Therefore, there is still a need for a holistic in-band control design integrated into an industry-driven SDN framework.