Distributed Cross-Domain Optimization for Software Defined Industrial Internet of Things

Abstract

As a promising paradigm, the Industrial Internet of Things (IIoT) provides a wide range of intelligent services through the interconnection and interaction of heterogeneous networks. The quality of these services depends on how the bandwidth is shared among different flows. Hence, it is critical to design a flexible flow control strategy in multi-region management scenarios. In this paper, we establish a flow optimization model based on the IIoT networks managed by multiple Software-Defined Networking (SDN) controllers. Specifically, it jointly optimizes the real-time delivery, route selection, and constrained resource allocation to maximize the total utilities of domains. Since the topology and resources within each domain are kept secret, the problem model belongs to a multi-block problem with coupling constraints, which is difficult to be solved directly. To this end, we first decompose the problem into several intra-domain subproblems, which can be solved in parallel. By considering the inter-domain communication problem, we then introduce the slack variables to implement the interaction among domains. Finally, we design a distributed Proximal Symmetric Alternating Direction Method of Multipliers (Prox-SADMM) algorithm to solve the above joint optimization problem. Through numerical simulations, we investigate the impact of data timeliness, multi-path routing, and resource constraints on the rate utility. The performance analysis confirms that the Prox-SADMM algorithm can be well applied to large-scale networks and provides guidance to set appropriate parameter values according to the realistic requirements of IIoT networks.

1. Introduction

The Industrial Internet of Things (IIoT) is a key enabling technology for smart industry. By collecting and processing industry data, it connects all industrial assets, and provides a wide range of intelligent services such as process optimization, supply chain management and predictive maintenance in production processes [1,2]. It is expected that the IIoT will generate USD 14.2 trillion in global GDP growth by 2030 [3].

With the development of intelligent applications around the world, IIoT brings new challenges in terms of real-time, energy efficiency, flexibility, and privacy. On the one hand, the exponential growth and concurrent transmission of flows in intelligent industrial networks will inevitably cause network problems such as huge energy consumption and serious transient congestion [4,5]. Therefore, it is crucial to study how to cost-effectively achieve the green deployment of IIoT and rational allocation of network resources. In addition, industrial events require deterministic transmission times to keep industrial applications running properly [6]. Therefore, IIoT needs to provide real-time communication services to ensure the communication latency and control accuracy of intelligent industrial systems.

On the other hand, IIoT has a large number of heterogeneous wired and wireless communication systems. The communication protocol standards of these systems are different, so it is difficult to achieve efficient cross-domain coordination and unified management. In addition, inter-domain communication poses the risk of privacy leakage to industrial production [7]. Therefore, the issue of how to guarantee industrial individual privacy when collaborative scheduling among heterogeneous systems in IIoT has received industry attention. The development of Software-Defined Networking (SDN) [8] and the emergence of distributed algorithm in SDN have brought new ideas to solve the above problems.

Software Defined Industrial Internet of Things (SDIIoT) is a communication architecture [9], in which the logic of physical devices is centralized into a control plane to simplify the management of IIoT networks. The research works in [10,11] have shown that centralized SDIIoT can effectively reduce the implementation complexity, support various business requirements, and achieve fine-grained flow control. To obtain SDN-based green IIoT, optimization of sensing, processing, and communication is a common solution [12]. Furthermore, developing energy-efficient deployment schemes is another effective measure to address energy issues. For example, in [13], the researchers proposed an SDIoT architecture that divides the entire system into five layers: sense layer, gateway layer, SDN switch layer, control layer, and application layer. The purpose of such an architecture is to assign network devices to designated layers based on their usage, thus the traffic loads can be balanced, and thus energy efficiency can be improved.

However, the current SDN-based IIoT systems are limited by centralized architectures, which cannot well solve scalability issues in IIoT networks. Therefore, the SDN control plane needs to adopt a distributed architecture in large-scale complex network scenarios [14]. Each local controller only has complete information about its own domain, which is not disclosed to the root controller to protect data privacy and reduce network overhead. Due to the huge traffic load and the time-varying environment, distributed SDIIoT requires dynamic multi-path resource allocation to support massive IIoT devices. The main challenge in allocating resources in multi-domain networks is that no controller maintains the state of all domains. To improve resource utilization, the limited resources of each domain, such as bandwidth and initial energy, should be optimally allocated to IIoT services. In [15], the researchers proposed a cross-domain resource allocation scheme in a software-defined optical access network to meet the huge bandwidth supporting the requirement of new network services. Typically, a network flow corresponds to traffic aggregates of a service or a class of services. The traffic aggregates can be transmitted on a single path or can be assigned to different paths connecting a source-destination pair. If the source and destination of a path are not in the same domain, the path is the responsibility of the root controller and associated local controllers. In fact, compared with single-path traffic engineering, multi-paths can guarantee routing robustness and low latency for better network performance [16]. In addition, due to the stringent real-time requirements discussed earlier, distributed SDIIoT requires real-time data delivery to support real-time applications in IIoT networks. To meet these objectives, the joint optimization of the real-time delivery, route selection, and limited resource allocation in distributed SDIIoT is critical to design an optimal dynamic multi-path resource allocation with real-time traffic and multi-domain scenarios.

This paper proposes a distributed architectural design of a multi-domain SDN-based green IIoT in heterogeneous wired and wireless environments to significantly improve resource utilization and user satisfaction, exploiting the information interaction between SDN controllers and IIoT gateways to achieves multi-path resource allocation and real-time flow control. To meet the above objectives, we investigate the joint real-time data delivery, multi-path routing and limited resource allocation optimization problem to maximize the total utilities of network domains. However, there are three challenges in solving the joint optimization problem quickly and efficiently. First, for obvious scalability and resiliency reasons, it is necessary to design massively parallel algorithms for multi-domain SDN to obtain higher network scalability. Second, due to the large communication overhead and delay between controllers, distributed algorithms for multi-domain SDN require a small number of iterations to obtain near-optimal solutions. Third, since the network state of each domain is subject to abrupt changes, such as flow rate variation and flow arrival or departure, distributed algorithms need to be highly responsive and robust so that resource allocation is implementable for each iteration. Inspired by [17,18], we design a distributed Proximal Symmetric Alternating Direction Method of Multipliers (Prox-SADMM) algorithm to achieve the optimal solution. The distributed Prox-SADMM algorithm can run in real-time, thus enabling SDN-based IIoT to achieve optimal resource allocation quickly. Meanwhile, when the designed algorithm is solving the joint optimization problem, the computation of all subproblems can be performed independently by each local controller, and the root controller only needs to update the Lagrange multipliers and slack variables. Such a distributed interaction makes the topology and resources within each domain confidential, thus effectively protecting the privacy of industrial individuals.

The main contributions of this work are summarized as:

• A distributed SDIIoT architecture design based on multi-domain SDN is proposed, which supports cross-domain interconnection of large-scale heterogeneous networks. This architecture leverages SDN controllers and IIoT gateways to control the underlying network devices for efficient resource management. The industrial infrastructure is deployed in the lower three layers of the architecture, namely the terminal layer, the relay layer, and the access layer. Such a layered deployment reduces the energy consumption in communications, thus achieving the green IIoT.
• A joint optimization of real-time data delivery, multi-path routing and limited resource allocation to maximize the total utilities of domains, in a setting where several SDN domain controllers operate in parallel. The reason for joint optimization is to improve the satisfaction of IIoT users by optimizing resource allocation. The joint optimization problem can be formulated as a multi-block problem with coupling constraints. The problem is difficult to be solved directly due to the time-varying and confidential nature of network information and the large number of domains and flows.
• A distributed algorithm based on the proximal symmetric ADMM is designed to solve the above optimization problem while meeting user privacy expectations. At each time slot, the problem is decomposed into several intra-domain subproblems that can be solved in parallel. These subproblems are updated serially with the slack variables, thus enabling intra- and inter-domain communication in IIoT networks. Finally, numerical simulations show that the designed algorithm can be applied to large-scale networks with multiple requests. Moreover, through numerical simulations, we illustrate the impact of data timeliness, multi-path routing, and resource constraints on the rate utility.

The paper is structured as follows. In Section 2, we analyze the distributed SDIIoT architecture for green communication. In Section 3, we provide the system model. The problem formulation is presented in Section 4 followed by the distributed Prox-SADMM algorithm. Performance analysis and conclusions are presented in Section 5 and Section 6, respectively.

We use the following notations in this paper. Uppercase and lowercase boldface letters indicate matrices and vectors, respectively. ${\left(\mathbf{A}\right)}_{il}$ denotes the ith row and lth column element of the matrix and $a_i$ denotes the ith element of the vector $\mathbf{a}$. $\mathbf{I}$ represents the identity matrix. Sets are denoted by Euler letters, e.g., $\mathcal{A}$. Furthermore, $\left|\mathcal{A}\right|$ denotes the cardinality of the set $\mathcal{A}$. The operation $\left|\right|·{\left|\right|}_2$ denotes the Euclidean norm, ${(·)}^T$ denotes transpose.

2. Hierarchical Multi-Domain SD-IIoT Architecture

In the complex industrial IoT environment, various smart devices are widely adopted, and network services are no longer homogeneous. With the progress of information technology, IIoT puts forward higher requirements for energy efficiency, privacy, and resilience. Meanwhile, it pays more attention to the interaction of multi-source heterogeneous data. Therefore, in this section, we propose a hierarchical multi-domain architecture based on the advancement of SDN technology. As shown in Figure 1, the SD-IIoT green architecture, which supports cross-domain interconnection of large-scale heterogeneous networks, includes four layers from bottom to top, namely SD-Industrial Terminal Access Layer, Communication Network Layer, Distributed Control Layer and Application Layer.

2.1. SD-Industrial Terminal Access Layer

This layer incorporates a large number of heterogeneous intelligent devices, including sensors and actuators, personal digital assistant (PDA), automated guided vehicles (AGV), etc. These devices can access the Internet with combinations of wireless and wired connections through devices such as base stations, wireless access points, and IoT gateways.

In this layer, we introduce the idea of layering and partitioning. This effectively reduces the industrial infrastructure energy while balancing the network traffic loads. In the idea of layering, industrial nodes, relay nodes, and access nodes are deployed in a three-layer framework. The bottom layer is the industrial terminal layer, where intelligent devices such as sensors, RFID, and M2M terminals are placed. The middle layer is the industrial relay layer, which is formed by a collection of energy-constrained relay nodes. The top layer is the industrial access layer, which consists of multiple access nodes. In the idea of partitioning, all nodes and links of the terminal access layer are divided into different local control domains. The access nodes in each domain serve as hubs for data reception and distribution, and as domain managers for accessing client-host applications. In addition, to ensure robust and reliable IIoT communications, the access nodes running domain management algorithms work with controllers to process data, configure routes, and allocate bandwidth.

The terminal access layer based on layer and domain control is highly scalable and manageable. Its structure is conducive to reducing communication energy consumption, thereby meeting “green” requirements of the IIoT. To further realize the IIoT towards green, we develop a new communication strategy between nodes. Industrial nodes cannot directly communicate with each other, but they can reach another industrial node through relay nodes and access nodes. The relay nodes in each domain interconnect and form a relay network. They can relay data from the access layer to industrial nodes in the lower layer or from the terminal layer to access nodes in the upper layer. Due to the long distance between domains, neither relay nodes nor access nodes can connect across domains.

2.2. SD-Industrial Communication Network Layer

This layer consists of the interconnected SDN-enabled switches. They are divided into different local control domains and managed by the corresponding controllers.

2.3. SD-Industrial Distributed Control Layer

This layer has a set of software-based controllers and is responsible for network management decisions [19]. Considering the complexity and long-term scalability of IIoT networks, we adopt a multi-level distributed control strategy. A super controller manages multiple local controllers that directly manage the underlying network devices. To protect the privacy and security of industrial individuals, local controllers only know the network topology and resource information within their own domain. Each domain appears as a whole for the super controller, i.e., details within the domain are closed to the super controller. This reduces network overhead while lightening the load on the super controller.

2.4. SD-Industrial Application Layer

This layer enables the complete abstraction of underlying equipment via the northbound interface that embodies the characteristics of industrial applications. In this way, network services can be deployed in more dimensions and with finer granularity as needed, thereby accelerating the innovative development of green IIoT. However, it is a complex and urgent challenge to design various engines and functions at the application layer for system management. Therefore, this proposed work focuses on continuously optimizing resource allocation to form a greener IIoT while meeting cross-layer, cross-domain, real-time, secure, and flexible transmission requirements.

3. System Model

In this paper, based on proposed green architecture, we consider the SD-IIoT network as a directed graph $\mathcal{C}=\{\mathcal{N},\mathcal{L}\}$. $\mathcal{N}=\mathcal{S}\cup \mathcal{R}\cup \mathcal{B}\cup \mathcal{W}$ is the set of network nodes, where $\mathcal{S}$ denotes the set of industrial nodes, $\mathcal{R}$ denotes the set of relay nodes, $\mathcal{B}$ denotes the set of access nodes, and $\mathcal{W}$ denotes the set of switch nodes. $\mathcal{L}=\mathcal{U}\cup \mathcal{V}$ is the set of network links, where $\mathcal{U}$ denotes the set of wireless links and $\mathcal{V}$ denotes the set of wired links.

Assume that the entire network is divided into Y local control domains. Let $\mathcal{Y}$ denote the set of local control domains. The local controller of each domain exchanges information with the super controller to manage nodes, links, and source data flows within the domain.

We describe the nodes of the y-th domain by the set ${\mathcal{N}}_y={\mathcal{S}}_y\cup {\mathcal{R}}_y\cup {\mathcal{B}}_y\cup {\mathcal{W}}_y$, and the links of the yth domain by the set ${\mathcal{L}}_y={\mathcal{U}}_y\cup {\mathcal{V}}_y$. All data flows in networks are identified by source-destination pairs. The data flow whose source node is in the y-th domain is called the source data flow of the yth domain and is denoted by the set ${\mathcal{J}}_y$. We assume that the network information of each domain is time-varying and varies irregularly. Let $\mathcal{T}$ denote the system time set. At time slot $t\in \mathcal{T}$, the source data flow j has a non-negative flow rate, denoted by $f_j^t$.

3.1. Flow Conservation Model

In SD-IIoT networks, packet routing is handled by controllers and access nodes through efficient algorithms. Assume that packet routing follows the braided multipath model, i.e., feasible paths between the same source-destination pair are only locally disjoint.

When the source and destination nodes of a packet are in the same domain, the intra-domain routing of the packet is handled by access nodes. For the case of intra-domain delivery, the flow conservation for each domain can be expressed as:

$$\begin{array}{c}\hfill {\mathbf{A}}_y^t{\mathbf{x}}_{yj}^t={\mathbf{b}}_{yj}^t{f}_j^t,\forall j\in {\mathcal{J}}_y,y\in \mathcal{Y},t\in \mathcal{T}\end{array}$$

(1)

where ${\mathbf{A}}_y^t\in {\mathbb{R}}^{|{\mathcal{N}}_y|\times |{\mathcal{L}}_y|}$ is the node-link association matrix for the yth domain at time slot t. Its elements are given by,

$${\left({\mathbf{A}}_y^t\right)}_{nl}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}l\mathrm{is}\mathrm{the}\mathrm{input}\mathrm{link}\mathrm{of}\mathrm{node}n\hfill \\ -1\hfill & \mathrm{if}l\mathrm{is}\mathrm{the}\mathrm{output}\mathrm{link}\mathrm{of}\mathrm{node}n\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

${\mathbf{x}}_{yj}^t={\left[x_{jl}^t\right]}_{l\in {\mathcal{L}}_y}\in {\mathbb{R}}^{|{\mathcal{L}}_y|}$ denotes the flow rates of source data flow j on the yth domain’ links at time slot t. ${\mathbf{b}}_{yj}^t={[\cdots -1\cdots 1\cdots ]}^T\in {\mathbb{R}}^{|{\mathcal{N}}_y|}$ denotes the state of source data flow j in the yth domain at time slot t, where −1 is the source node location of data flow j and 1 is the destination node location of data flow j.

When the source and destination nodes of a packet are in different domains, the super controller and local controllers cooperate to handle the cross-domain routing of the packet. The super controller obtains the path sequences of packets according to the link relationships between local domains and the abstract paths given by local controllers. For the case of cross-domain delivery, the flow conservation for each domain can be expressed as:

$$\begin{array}{c}\hfill {\mathbf{A}}_v^t{\mathbf{x}}_{vj}^t={\mathbf{b}}_{vj}^t{f}_j^t,\forall j\in {\mathcal{J}}_y,v,y\in \mathcal{Y},v\ne y,t\in \mathcal{T}\end{array}$$

(2)

3.2. Link Capacity Model

The system links are divided into wired links and wireless links. Unlike wired links, the channel capacity of each wireless link is variable. Its value depends on time-varying channel conditions such as path loss and interference [20]. Let $c_l^t$ denote the capacity of the link l at time slot t.

When the network transmits packets, each link carries numerous packets from different source data flows. These data flows may not belong to the same domain as the links they traverse. Furthermore, at any time slot, the sum of intra- and cross-domain communication data flowing through link l cannot exceed the maximum link capacity. Therefore, the link capacity constraints can be expressed as:

$$\begin{array}{c}\hfill \sum _{y\in \mathcal{Y}}\sum _{j\in {\mathcal{J}}_y}{x}_{jl}^t\le c_l^t,\forall l\in {\mathcal{L}}_v,v\in \mathcal{Y},t\in \mathcal{T}\end{array}$$

(3)

3.3. Real-Time Delivery Constraint

The specificity of industrial applications, such as precise control of assembly lines and alarm notification of industrial facilities, makes application data particularly sensitive to transmission delays.

With extremely short time slots, the SD-Industrial platform can support IIoT users’ demands for real-time data delivery. Real-time delivery means not only fast but also deterministic, which corresponds to the quality of service in networks. To facilitate the following discussion, we introduce the concept of a delivery contract [21]. In the contract, the total amount of a particular source data flow j, over a given period $[t_1,t_2]$, must meet or exceed a specified minimum amount q,

$$\begin{array}{c}\hfill \sum _{t=t_1}^{t_2}{f}_j^t\ge q\end{array}$$

(4)

Each source data flow may have different delivery contracts for different periods. Let ${\mathcal{D}}_j$ denotes the set of delivery contracts for source data flow j. Then, we describe the states of all delivery contracts for source data flow j at time slot t using the vector ${\mathbf{z}}_j^t\in {\mathbb{R}}^{|{\mathcal{D}}_j|}$, defined as,

$${\left({\mathbf{z}}_j^t\right)}_n=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\eta \mathrm{th}\mathrm{contract}\mathrm{of}\mathrm{flow}j\mathrm{is}\mathrm{active}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

Here, ‘is active’ means that time slot t lies within the delivery contract period. The constraints that delivery contracts for each domain are met can be expressed as follows:

$$\begin{array}{c}\hfill \sum _{t\in \mathcal{T}}{\mathbf{z}}_j^t{f}_j^t\ge {\mathbf{q}}_j,\forall j\in {\mathcal{J}}_y,y\in \mathcal{Y}\end{array}$$

(5)

where ${\mathbf{q}}_j={\left[q_{j\eta }\right]}_{\eta \in {\mathcal{D}}_j}\in {\mathbb{R}}^{|{\mathcal{D}}_j|}$ is the contract indicator vector of source data flow j.

3.4. Energy Consumption Model

In fact, each network device consumes much more energy in sending and receiving data than it does in sensing and processing data. Therefore, we only consider the energy consumption of data communication in the energy consumption model [22].

For each radio device, the energy spent to transmit a $\kappa$-bit packet to distance d is given by,

$$E_{tx}=\left\{\begin{array}{cc}\kappa E_{elec}+\kappa {\epsilon }_{fs}{d}^2\hfill & \mathrm{if}d\le d_0\hfill \\ \kappa E_{elec}+\kappa {\epsilon }_{mp}{d}^4\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

where $E_{elec}$ is the energy overhead of the transmit or receive circuit on one-bit data; ${\epsilon }_{fs}$ and ${\epsilon }_{mp}$ are the amplifier energy of the free-space and ground reflection models, respectively; and $d_0=\sqrt{{\epsilon }_{fs}/{\epsilon }_{mp}}$ is the threshold distance. Furthermore, the energy spent to receive a $\kappa$-bit packet is given by,

$$\begin{array}{c}\hfill E_{rx}=\kappa E_{elec}\end{array}$$

(6)

Assume that the radio devices have short data transmission distances and employ the free-space model for communication. Each industrial wireless node can send its sensed data and receive data flows from different domains. Furthermore, each relay node forwards packets that may come from different data flows of other domains. Accordingly, for node $n\in {\mathcal{S}}_v\cup {\mathcal{R}}_v$ of the vth domain, its energy constraints can be expressed as:

$$\begin{array}{c}\hfill \sum _{y\in \mathcal{Y}}\sum _{j\in {\mathcal{J}}_y}\left\{\sum _{l\in {\mathcal{O}}_n}{x}_{jl}^t(E_{elec}+{\epsilon }_{fs}{d}_{np}^2)+\sum _{l\in {\mathcal{I}}_n}{x}_{jl}^t{E}_{elec}\right\}\le e_n^t,\forall v\in \mathcal{Y},t\in \mathcal{T}\end{array}$$

(7)

where $e_n^t=e_n/{\tau }_n$, $e_n$ is the initial energy of node n, and ${\tau }_n$ is the predetermined lifetime of node n. $d_{nl}$ denotes the distance from node n to another node on link l. ${\mathcal{O}}_n$ and ${\mathcal{I}}_n$ are the output link set and input link set for node n, respectively.

For each SDN switch, it has two states: active and sleep. Local controllers can consciously control the state of the switches, thereby reducing the energy consumption of the whole system. For switch node $n\in \mathcal{W}$, its state at time slot t is given by,

$${\gamma }_n^t=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{node}n\mathrm{is}\mathrm{active}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

In the sleep state, the switch consumes only constant energy $P_{sleep}$ per unit time. However, in the active state, apart from constant energy demand $P_{active}$, the switch ports consume additional energy based on varying traffic loads. Here, we define the energy consumption per unit time when the port is fully loaded as $P_{full}$. For link $l\in \mathcal{L}$ connected to the port, its utilization at time slot t is calculated by:

$$\begin{array}{c}\hfill {\beta }_l^t=\sum _{y\in \mathcal{Y}}\sum _{j\in {\mathcal{J}}_y}\frac{x_{jl}^t}{c_l^t}\end{array}$$

(8)

When network data is transmitted across domains, the switches process the data according to the flow rules issued by local controllers. If the switch is overloaded, it can seriously affect the processing capacity of communication networks. Therefore, we must limit the operating power of each switch to avoid frequent packet loss. For switch node $n\in {\mathcal{W}}_v$ of the vth domain, its energy constraint can be expressed as:

$$\begin{array}{c}\hfill P_{sleep}+{\gamma }_n^t(P_{active}-P_{sleep})+\sum _{l\in {\mathcal{Q}}_n}{\beta }_l^t{P}_{full}\le p_n^t,\forall v\in \mathcal{Y},t\in \mathcal{T}\end{array}$$

(9)

where $p_n^t$ denotes the maximum energy consumption of node n over time slot t, and ${\mathcal{Q}}_n={\mathcal{O}}_n\cup {\mathcal{I}}_n$ is the set of links connecting node n.

3.5. Utility Function

Most IIoT services are inelastic, with strict requirements on the transmission rate and delay. In inelastic communications, insufficient bandwidth can severely affect the quality of network services.

To ensure utility fairness among competing flows, we adopt a sigmoid function to represent the utility function of inelastic communication [23,24]. For source data flow j, its utility function at time slot t is ${\psi }_j^t\left(f_j^t\right):{\mathbb{R}}^{+}$→$\mathbb{R}$. The utility functions of different flows have different concave and convex properties. Hence, for each flow j with utility ${\psi }_j^t$ at time slot t, we redefine a pseudo-utility function:

$$\begin{array}{c}\hfill U_j^t\left(f_j^t\right)={\int }_{m_j^t}^{f_j^t}\frac{1}{{\psi }_j^t\left(\phi \right)}d\phi ,m_j^t\le f_j^t\le M_j^t\end{array}$$

(10)

where $M_j^t<+\infty$ and $m_j^t\ge 0$ denote the maximum and minimum allowable values given by the flow rate $f_j^t$, respectively.

4. Problem Formulation and Distributed Optimization

In this section, we formulate a joint optimization problem for real-time delivery, route selection, and constrained resource allocation. This problem is difficult to be solved directly in multi-domain networks since the time-varying information in each domain is kept confidential. Therefore, we design the distributed Prox-SADMM algorithm. To ensure the convergence of the algorithm, we add special proximal terms in it. In general, our algorithm maximizes the summation of domain utilities, while solving the joint optimization problem.

4.1. Joint Optimization Problem

Our objective is to maximize the total utilities of network domains while satisfying user requirements and resource constraints. Therefore, based on (10), the utility maximization optimization problem is formulated as:

$$\begin{array}{ccc}\hfill & \underset{\mathbf{f},\mathbf{x}}{max}& \sum _{y\in \mathcal{Y}}\sum _{t\in \mathcal{T}}\sum _{j\in {\mathcal{J}}_y}{U}_j^t\left(f_j^t\right)\hfill \\ & s.t.& \mathrm{constraints}\left(\right(1\left)\right(2\left)\right(3\left)\right(5\left)\right(7\left)\right(9\left)\right)\hfill \\ \hfill & & m_j^t\le f_j^t\le M_j^t,0\le x_{jl}^t,\forall j,l,t.\hfill \end{array}$$

(11)

Combining the strictly increasing property of ${\psi }_j^t\left(f_j^t\right)$, it follows that $U_j^{t^{\prime }}\left(f_j^t\right)=1/{\psi }_j^t\left(f_j^t\right)>0$, and $U_j^{t^{\prime }}\left(f_j^t\right)$ is strictly decreasing. Therefore, regardless of the concavity of the utility function ${\psi }_j^t\left(f_j^t\right)$, the pseudo-utility $U_j^t\left(f_j^t\right)$ is a strictly increasing concave function on the interval $[m_j^t,M_j^t]$, together with the convex feasible set, it follows that the problem (11) is a convex optimization problem. If a centralized approach is used to solve the system model, all the complete information of the industrial production process is required, which may lead to privacy issues. Therefore, we apply the Prox-SADMM algorithm to solve the optimization problem in a distributed manner. The general ADMM method is used for solving two-block separable convex optimization. For multi-block problems, some sophisticated extensions have been proposed: ADMM with Gaussian Back Substitution [25], and Proximal Jacobian ADMM [26]. Without strict assumptions [26], establish global convergence results and $O\left(\frac{1}{t}\right)$ convergence rates in the traversal sense, where t is the number of iterations. The property that the Jacobian type iteration can be preferred for distributed and parallel optimization, which is quite attractive in a wide range of applications. Therefore, this iterative approach is applied to solving network problems in a distributed manner.

4.2. Distributed Optimization via Prox-SADMM

This subsection proposes a distributed algorithm based on the Prox-SADMM to solve the optimization problem (11). First, slack variables ${\mathbf{s}}^t\in {\mathbb{R}}_{+}^{\left|\mathcal{L}\right|+\left|\mathcal{S}\right|+\left|\mathcal{R}\right|+\left|\mathcal{W}\right|},t\in \mathcal{T}$ are introduced for the link capacity constraint (3), radio node energy constraint (7) and switch node power constraint (9). Due to the introduction of slack variables ${\mathbf{s}}^t$, the inequality constraints (3), (7) and (9) are transformed into equality constraints, which makes the feasible region of the joint optimization problem expand to a larger space. To facilitate the description, the original optimization problem (11) can be rewritten equivalently as:

$$\begin{array}{ccc}\hfill & \underset{\mathbf{f},\mathbf{x}}{min}& \sum _{y\in \mathcal{Y}}\sum _{t\in \mathcal{T}}{\tilde{U}}_y^t\left({\mathbf{f}}_y^t\right)\hfill \\ \hfill & s.t.& \mathrm{constraints}\left(1\right)\left(2\right)\left(5\right)\hfill \\ & & \sum _{y\in \mathcal{Y}}{\mathbf{H}}^t{\mathbf{x}}_y^t+{\mathbf{s}}^t={\mathbf{h}}^t,t\in \mathcal{T}\hfill \\ \hfill & & {\mathbf{s}}^t\ge 0,t\in \mathcal{T}\hfill \\ \hfill \end{array}$$

(12)

where

$${\tilde{U}}_y^t\left({\mathbf{f}}_y^t\right)=-\sum _{j\in {\mathcal{J}}_y}{U}_j^t\left(f_j^t\right),{\mathbf{f}}_y^t={\left[f_j^t\right]}_{j\in {\mathcal{J}}_y}$$

$${\mathbf{x}}_y^t=[{\left({\mathbf{x}}_{1y}^t\right)}^T,\cdots ,{\left({\mathbf{x}}_{Yy}^t\right)}^T],{\mathbf{x}}_{vy}^t=\sum _{j\in {\mathcal{J}}_y}{\mathbf{x}}_{vj}^t,$$

$${\mathbf{h}}^t={[{\left({\mathbf{c}}_1^t\right)}^T,\cdots ,{\left({\mathbf{c}}_Y^t\right)}^T,{\left({\mathbf{e}}_1^t\right)}^T,\cdots ,{\left({\mathbf{e}}_Y^t\right)}^T,{\left({\mathbf{w}}_1^t\right)}^T,\cdots ,{\left({\mathbf{w}}_Y^t\right)}^T]}^T,$$

$${\mathbf{c}}_v^t={\left[c_l^t\right]}_{l\in {\mathcal{L}}_v}\in {\mathbb{R}}^{|{\mathcal{L}}_v|},{\mathbf{e}}_v^t={\left[e_n^t\right]}_{n\in {\mathcal{S}}_v\cup {\mathcal{R}}_v}\in {\mathbb{R}}^{|{\mathcal{S}}_v|+|{\mathcal{R}}_v|},$$

$${\mathbf{w}}_v^t={[p_n^t-P_{sleep}+{\gamma }_n^t(P_{sleep}-P_{active})]}_{n\in {\mathcal{W}}_v}\in {\mathbb{R}}^{|{\mathcal{W}}_v|},$$

$${\mathbf{H}}^t={[{\left({\mathbf{I}}^t\right)}^T,{\left({\mathbf{E}}^t\right)}^T,{\left({\mathbf{P}}^t\right)}^T]}^T,{\mathbf{I}}^t=diag\{{\mathbf{I}}_1^t,\cdots ,{\mathbf{I}}_Y^t\},$$

$${\mathbf{E}}^t=diag\{{\mathbf{E}}_1^t,\cdots ,{\mathbf{E}}_Y^t\},{\mathbf{P}}^t=diag\{{\mathbf{P}}_1^t,\cdots ,{\mathbf{P}}_Y^t\},$$

$${\left({\mathbf{E}}_v^t\right)}_{nl}=\left\{\begin{array}{cc}{E}_{elec}+{\epsilon }_{fs}{d}_{nl}^2\hfill & \mathrm{if}l\mathrm{is}\mathrm{the}\mathrm{output}\mathrm{link}\mathrm{of}\mathrm{radio}\mathrm{node}n\hfill \\ E_{elec}\hfill & \mathrm{if}l\mathrm{is}\mathrm{the}\mathrm{input}\mathrm{link}\mathrm{of}\mathrm{radio}\mathrm{node}n\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

$${\left({\mathbf{P}}_v^t\right)}_{nl}=\left\{\begin{array}{cc}\frac{P_{full}}{c_l^t}\hfill & \mathrm{if}\mathrm{link}l\mathrm{is}\mathrm{connected}\mathrm{to}\mathrm{switch}\mathrm{node}n\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

The augmented Lagrangian function of (12) can be expressed as:

$$\begin{array}{ccc}\hfill & L_{\rho }(\mathbf{f},\mathbf{x},\mathbf{\lambda },\mathbf{s})=& \sum _{y\in \mathcal{Y}}\sum _{t\in \mathcal{T}}{\tilde{U}}_y^t\left({\mathbf{f}}_y^t\right)-\sum _{t\in \mathcal{T}}{\left({\mathbf{\lambda }}^t\right)}^T(\sum _{y\in \mathcal{Y}}{\mathbf{H}}^t{\mathbf{x}}_y^t+{\mathbf{s}}^t-{\mathbf{h}}^t)+\frac{\rho }{2}\hfill \\ \hfill & & \left(\sum _{t\in \mathcal{T}}\right||\sum _{y\in \mathcal{Y}}{\mathbf{H}}^t{\mathbf{x}}_y^t+{\mathbf{s}}^t-{\mathbf{h}}^t|{|}^2)\hfill \end{array}$$

(13)

where ${\mathbf{\lambda }}^t\in {\mathbb{R}}^{\tau },t\in \mathcal{T}$ is the Lagrangian multiplier, and $\rho \in {\mathbb{R}}_{+}$ is penalty parameter for adjusting the convergence speed of ADMM.

The variables are regrouped into two groups, naturally divided into original variables $\mathbf{f},\mathbf{x}$ and the slack variables $\mathbf{s}$. Firstly each group of variables is updated in parallel in its own subproblem in the domain. Next, these two groups of variables are updated in a Gauss–Seidel scheme. Finally, Lagrange multiplier $\mathbf{\lambda }$ is updated once after each group of variables is updated twice separately. The information exchange between the super controller and multiple local controllers in the distributed Prox-SADMM algorithm is given in Figure 2.

Furthermore, to further ensure the convergence of ADMM, we additionally add partially proximal terms as in [27]. Therefore, $\mathbf{f},\mathbf{x},\mathbf{s},\mathbf{\lambda }$ can iteratively update as follows:

$$\begin{array}{cc}\hfill \{{\tilde{\mathbf{f}}}_y^t\left(k\right),{\tilde{\mathbf{x}}}_y^t\left(k\right)\}=& argmin\{\sum _{t\in \mathcal{T}}{\tilde{U}}_y^t\left({\mathbf{f}}_y^t\right)-\sum _{t\in \mathcal{T}}{\left({\mathbf{\lambda }}^t\left(k\right)\right)}^T{\mathbf{H}}^t{\mathbf{x}}_y^t+\frac{\rho }{2}(\sum _{t\in \mathcal{T}}\left|\right|\sum _{\genfrac{}{}{0pt}{}{i\in \mathcal{Y}}{i\ne y}}{\mathbf{H}}^t{\mathbf{x}}_i^t\left(k\right)\\ & +{\mathbf{H}}^t{\mathbf{x}}_y^t+{\mathbf{s}}^t\left(k\right)-{\mathbf{h}}^t{\left|\right|}^2)+\frac{\rho }{2}\sum _{t\in \mathcal{T}}\left|\right|{\mathbf{H}}^t({\mathbf{x}}_y^t-{\mathbf{x}}_y^t{\left(k\right)\left|\right|}^2\left)\right\}\\ \hfill & s.t.\mathrm{constraints}\left(2\right)\left(5\right),m_j^t\le f_j^t\le M_j^t,0\le x_{jl}^t,\forall j,l,t.\end{array}$$

(14)

$$\begin{array}{c}\hfill {\overline{\mathbf{\lambda }}}^t\left(k\right)={\mathbf{\lambda }}^t\left(k\right)-\rho (\sum _{y\in \mathcal{Y}}{\mathbf{H}}^t{\tilde{\mathbf{x}}}_y^t\left(k\right)+{\mathbf{s}}^t\left(k\right)-{\mathbf{h}}^t)\end{array}$$

(15)

$$\begin{array}{c}\hfill {\tilde{\mathbf{s}}}^t\left(k\right)=\underset{{\mathbf{s}}^t\ge 0}{argmin}\{-{\left({\overline{\mathbf{\lambda }}}^t\left(k\right)\right)}^T{\mathbf{s}}^t+\frac{\rho }{2}\left|\right|\sum _{y\in \mathcal{Y}}{\mathbf{H}}^t{\tilde{\mathbf{x}}}_y^t\left(k\right)+{\mathbf{s}}^t-{\mathbf{h}}^t{\left|\right|}^2\}\end{array}$$

(16)

$$\begin{array}{c}\hfill {\tilde{\mathbf{\lambda }}}^t\left(k\right)={\overline{\mathbf{\lambda }}}^t\left(k\right)-\rho (\sum _{y\in \mathcal{Y}}{\mathbf{H}}^t{\tilde{\mathbf{x}}}_y^t\left(k\right)+{\tilde{\mathbf{s}}}^t\left(k\right)-{\mathbf{h}}^t)\\ \hfill \end{array}$$

(17)

where k denotes the index of iteration. At the end of each iteration, the algorithm performs a correction step to ensure convergence. We introduce the following notation:

$${\mathbf{\xi }}^t\left(k\right)={[{\left({\mathbf{x}}_1^t\right)}^T\left(k\right),\cdots ,{\left({\mathbf{x}}_Y^t\right)}^T\left(k\right),{\left({\mathbf{s}}^t\right)}^T\left(k\right),{\left({\mathbf{\lambda }}^t\right)}^T\left(k\right)]}^T$$

In the Prox-SADMM algorithm, the special proximal term is only added to each subproblem with variables $\mathbf{f},\mathbf{x}$ of the first group, while the subproblems with slack variables $\mathbf{s}$ in the second group remain unchanged. At the end of each iteration, an extension step with a fixed step size is performed on all variables for correction, so the generated sequences may produce faster convergence. The results of numerical experiments show that this approach is effective. The Prox-SADMM algorithm is summarized in Algorithm 1.    

Algorithm1: The Prox-SADMM algorithm

5. Performance Analysis

In this section, by using the Matlab tool, we present the simulation results to evaluate the performance of the proposed joint optimization and Prox-SADMM algorithm. The simulation parameters are listed in

Table 1. Simulation Parameters.

Parameter name	Value
Number of time slots	10–50
Capacity of wireless links	5–10 Mbps
Capacity of wired links	5–50 Mbps
Initial energy of radio devices	2000–3000 J
Energy overhead	50 nJ/bit
Amplifier characteristic constant	10 pJ/bit/m${}^2$
Fixed power of switches in sleep state	130 W [28]
Fixed power of switches in active state	260 W
Power of ports at full load	4 W
Power limit of switches	280–295 W
Flow rate	0.1–5 Mbps

. Since there are elastic and inelastic communications in IIoT networks, we set the utility function as a sigmoid function: ${\psi }_j^t\left(f_j^t\right)=\frac{1}{1+e^{-10(f_j^t-\varpi )}}$, where $\varpi =\frac{M_j^t+m_j^t}{2}$.

To simulate multi-domain networks, we adopt Interoute topology [29] as the test topology, and randomly divide the network into several domains. Furthermore, in our experiments, the local controller of each domain executes its update rule under a specific thread at each iteration.

5.1. Impact of Network Constraints on the Rate Utility

In this subsection, we investigate the impact of data timeliness, multi-path routing, flow control, and energy constraints on rate utility. Furthermore, we depict the convergence behavior of proposed algorithm for different network parameters.

Figure 3 and Figure 4 show part of the network topology adopted in our implementation. They demonstrate optimal paths and flow control for intra- and cross-domain communication, respectively. It can be observed that source nodes give preference to paths with fewer hops to transmit data. However, they may also distribute a portion of the traffic to paths with more hops. This is because the SDIIoT system considers constrained resources and other data flows when assigning paths to source nodes. Therefore, by rational allocation of resources, the system can greatly satisfy the requirements of each user.

The impact of data timeliness on flow rates are shown in Figure 5, where the time slots in the ellipse box is within the delivery contract periods. We can observe that the proposed algorithm not only responds to traffic variations in real-time, but also provides fast convergence to the optimal solution. Moreover, it can be observed that the flow rates usually increase within their contract periods, as we would expect, and are usually lower outside contract periods. This is because the system sets aside resources for other source data flows with contracts. Therefore, when many heterogeneous flows are transmitted concurrently, the inelastic flows with tasks are assigned resources in priority. This not only achieves real-time delivery of flow data, but also improves resource utilization.

Furthermore, Figure 6 illustrates the impact of the energy limitation of equipment on the rate utility. It can be seen that the value of total utility solved by the proposed algorithm firstly increases along with the initial energy of radio nodes until $e_n=2600$ J, $n\in \mathcal{S}\cup \mathcal{R}$. The higher initial energy allows radio nodes to provide higher transmission rates to complete data delivery. Hence, IIoT users can obtain more efficient services, which increases the value of total utility. As the initial energy of the radio nodes exceed 2600(J), the value of total utility remains unchanged. The reason why the value of total utility no longer increases is the presence of other constrained resources. For example, the maximum power constraint of switches also impacts flow rates. Besides, as shown in Figure 6, when the initial energy of radio nodes is high, increasing the maximum power of switches will significantly increase the utility value of cross-domain flows, but will not change the utility value of intra-domain flows. This is because a large number of cross-domain flows pass through switches per unit time, while intra-domain flows do not. Therefore, it is necessary to carefully consider the maximum energy consumption per unit time for each radio device and switch. By doing so, we both improve the utilization of constrained resources, and reduce the equipment cost of IIoT networks.

5.2. Impact of the Network Scale on the Convergence

We now evaluate the applicability of proposed Prox-SADMM algorithm in practical networks. To begin with, we set up random SD-IIoT networks with $\left|\mathcal{Y}\right|$ domains and $\left|\mathcal{J}\right|$ source data flows. Then, we assume that the number of cross-domain flows is twice the number of intra-domain flows. In these networks, $\left|\mathcal{J}\right|$ source nodes and corresponding destination nodes are chosen randomly.

In order to verify whether the number of source data flows has a strong impact on the convergence rate of proposed algorithm, we set up networks in three scales: $\left|\mathcal{J}\right|=5,\left|\mathcal{Y}\right|=3$, with $\left|\mathcal{J}\right|=50,\left|\mathcal{Y}\right|=3$, and with $\left|\mathcal{J}\right|=500,\left|\mathcal{Y}\right|=3$. The corresponding simulation results are shown in Figure 7. Here, Optimality gap means the relative error between the optimal utility and the utility achieved by the proposed algorithm. In fact, the proposed algorithm can improve the optimization gap by continuously providing effective measures that satisfy user requirements. The results show that the increased numbers of data flows do not affect the convergence behavior of proposed algorithm.

In order to verify whether the number of network domains has a strong impact on the convergence rate of proposed algorithm, we set up networks in three scales: with $\left|\mathcal{Y}\right|=1,\left|\mathcal{J}\right|=300$, with $\left|\mathcal{Y}\right|=5,\left|\mathcal{J}\right|=300$, and with $\left|\mathcal{Y}\right|=10,\left|\mathcal{J}\right|=300$. Figure 8 indicates the optimality gap over time within a deadline of 10 min for networks with different number of domains. It can be observed that distributed parallel computing permits the proposed algorithm to converge faster. For example, to reach a gap below 1%, the network with 1 domain takes about 10 min, whereas the network with 10 domains takes only 5 min.

Based on the above analysis of algorithm performance, we can draw the conclusion: the proposed Prox-SADMM algorithm can be well applied to large-scale SD-IIoT networks with thousands or even millions of requests. As a massively parallel algorithm of SDN, it can solve network optimization problems at a faster speed, so as to achieve long-term scalability of IIoT networks.

6. Conclusions

In this paper, based on our proposed multi-domain green SD-IIoT architecture, we develop a joint optimization model for real-time delivery, route selection, and constrained resource allocation. The model is a multi-block problem with coupling constraints. Its complexity increases with the number of network domains and data flows. This problem is difficult to solve directly, because the information in each domain is time-varying and confidential. Therefore, we design the distributed Prox-SADMM algorithm by introducing slack variables. The designed algorithm not only contributes to the privacy protection of IIoT users, but also efficiently exploits the computational power of the distributed system. Specifically, at each moment, the problem is decomposed into several intra-domain subproblems that can be solved in parallel. These subproblems are updated serially with slack variables, thus enabling intra- and cross-domain communication in networks. Further, we add a special proximal term in each subproblem to ensure the convergence of the designed algorithm. The simulation results show that our algorithm quickly solve network optimization problems and achieve the long-term scalability of IIoT networks. Finally, the impact of data timeliness, multi-path routing, and resource constraints on the rate utility is investigated through numerical examples. Therefore, we can set appropriate parameter values to achieve a desired performance of large-scale IIoT networks according to actual requirements, such as load balancing and energy efficiency. In our future work, we will delve more deeply into the impact of performance metrics on the functioning of the multi-domain SDIIoT system. What is more, the main results of this paper can serve as a valuable reference for our future research points.