1. Introduction
The cyber-physical system (CPS) is the next-generation engineering system to realize a deep integration of computing, communication, and control technology. Its core technology is the close integration of information systems and physical units, which are considered prospects of a networked control system [
1]. Physical units act on the control process of automation in the cores of embedded devices, while information systems make full use of communication facilities and resources to acquire, transfer, and apply CPS instructions and information.
The main difficulty of CPS control lies in the co-scheduling of the physical unit and the information system [
2,
3]. The sampling and control processes of physical units are run on spatially distributed entities, including controllers, actuators, and sensors, and these are generally divided into periodic embedded task sets which have precedence constraints, different rate intervals, strict deadlines, and jitter requirements. Under non-preemptive communication protocol, it is necessary to efficiently co-schedule tasks of physical and information systems to maximize the utilization of network facilities and resources. Instead of a low speed and low bandwidth fieldbus, the communication process of an information system can be undertaken by switched real-time Ethernet (RTE) protocols governed by IEC 61850 [
4] and IEC 61158 [
5], which make co-scheduling problems more complicated.
Scheduling problems are generally considered as restricted resource allocation problems with specific dependencies and constraints, which can be addressed using a constraint programming approach. Two types of approaches are formally defined: constraint satisfaction problems (CSPs) and constraint optimization problems (COPs) [
6]. The purpose of CSPs is to achieve the satisfiability of a set of constraints, while COPs aim at optimizing an objective function on the basis of constraints satisfaction. In time-triggered CPS scheduling, the hard real-time constraints include the period, deadline, precedence, etc., which must be satisfied while some quality of service (QoS) parameters, such as the jitter, latency and load balance of the scheduling table, can be optimized by COP to improve the communication performance.
A multitude of optimal and heuristic techniques have been developed to solve COPs in time-triggered scheduling. The main problem is the complexity of non-deterministic polynomial (NP) complete, which limits the scale of networks and communication tasks. Although advanced methods and algorithms are proposed in past and recent references, it is still a hot topic in the field of optimization theory.
The scheduling optimization of non-strictly periodic traffic further aggravates the NP-complete problem. In recent research and articles, most of the studies are focused on strictly periodic scheduling, which only needs to schedule the first periodic frame instance of traffic flow. However, allowing non-strictly periodic frames can refine the granularity of the schedule table from flow level to frame level and thus heavily improve the solution space and flexibility.
2. Related Works
The survey article [
7] proposes a mathematical modeling framework that sets forward fundamental challenges in sensing, decentralized computation, robustness, energy efficiency, and hardware security in CPS. Many references have explained in detail how the design and scheduling of information systems affect the control process and stability of CPS. Derler [
8] and Poovendran [
9] analyze the real-time and synchronization characteristics of CPSs, which must be guaranteed. Karsai [
10] introduces a joint modeling method of an embedded system based on the dynamic integration of software and physics. Xiaofeng Yu [
2] focuses on the information transmission path on physical systems and finds an optimal path to meet the requirement of physical system stability. The influence of information system delay on physical process is studied. Bradley [
3] establishes control models for an information system and physical system, respectively. Taking the sampling time as the research object, an optimal control of computational–physical state co-regulation is studied in CPS. The results show that CPS needs industrial communication protocol and a real-time scheduling scheme to integrate information and physical systems.
According to the characteristics and services of a control system, the scheduling methods of tasks among multicores or multi-agents in CPSs are also very different. For physical processes that have high variability, non-Gaussianity, and higher order spatiotemporal correlations in personalized medicine, Bogdan defines a cyber-physical task graph as a time-dependent representation and proposes a goal-oriented self-optimization inspired solution for resource allocation on Network-on-Chip-based multicore platforms [
11]. In industrial applications, such as aircraft, high-speed trains, power grids, etc., real-time scheduling methods of fieldbus and distributed communication systems (Lonworks [
12], WorldFIP [
13], CAN [
14], FlexRay [
15], etc.) have been widely studied in industrial control applications. Moreover, using Ethernet and transmission control protocol/Internet protocol (TCP/IP) technologies to integrate a local area network (LAN) in CPSs allows utilities to be interconnected with automation networks, which has features such as high bandwidth, openness, wide availability, and low cost [
16,
17]. Although this work focuses on fully switched real-time Ethernet, it can also be used in fieldbus protocol or another switched real-time industrial network by extension. The inherently non-deterministic carrier-sense with collision detection (CSMA/CD) mechanism conditioned the adoption of Ethernet in time-critical applications [
18]. To overcome those shortcomings, several real-time protocols have been introduced to enforce time-critical communication over Ethernet [
19]. One cost-efficient way is the time-triggered scheduling, which plans transmission points of frames off-line.
Some articles, such as [
20,
21,
22] propose QoS-based design methods for fieldbus networks with the bandwidth allocation described as multi-objective optimization problems. Natale [
23] solves the problem with an objective to minimize the jitter of activities using simulated annealing. However, the time complexity of NP-complete will become more prominent with the increase of network and traffic scale. The algorithms in these references are only applicable to fieldbus scenarios and are difficult to work at full capacity in a multi-hop switched Ethernet.
Many related studies have done a lot of work to realize real-time Ethernet scheduling. A rate-monotonic algorithm has been used to accelerate scheduling time in [
24,
25]. The tabu search algorithm has been used to design the scheduling table off-line and improve the overall real-time performance by reducing the packet delay of rate-constrained frames in [
26,
27]. The pinwheel scheduling solution [
28] is regarded as a relaxation of the jittered-bounded scheduling concept, where each activity is required to be scheduled at least once during each predefined number of consecutive time units. Hanzalek in [
29] introduces a resource-constrained project scheduling with temporal constraints to minimize the Profinet IO IRT scheduling makespan. However, the jobs are aperiodic and executed only once in a macro period. A formal specification of scheduling constraints for time-triggered multi-hop networks and strictly periodic traffic while solving with the Satisfiability Modulo Theories (SMT) solver is introduced in [
30,
31,
32,
33]. The strictly periodic model leads to a relatively lower resource utilization and flexibility. Besides, frame constraints are added into the solver without considering the urgency and schedulability of frames. Therefore, the computation time of the schedule table increases rapidly with the number of constraints. In large-scale or non-periodic scheduling, it is hard to get a feasible solution within a time limit. A jitter satisfaction problem is solved by a three-level greedy algorithm in [
34]. A strictly periodic multi-cluster scheduling approach is proposed in [
35] by dividing flows into intra-cluster and inter-cluster. The off-line and on-line algorithms are presented to fast react to cluster-level changes in networks. A novel search-space pruning technique based on the response times is proposed in [
36] to generate a schedule for large distributed time-triggered systems within a reasonable amount of time. Due to the NP-complete of multi-hop scheduling, most of the researches are aimed at increasing the scheduling scale and reducing the scheduling time in strictly periodic scheduling while satisfying the constraints. The optimization of scheduling performance with non-strictly periodic traffic has not been fully studied.
The classic particle swarm optimization (PSO) is not a global optimization algorithm. Sun et al. [
37] introduces quantum theory into a PSO algorithm and proposes a quantum-behaved particle swarm optimization (QPSO). The contraction–expansion coefficient is the only parameter in QPSO that needs to be set manually besides the population size, particle dimension, and maximum iteration times. Its control strategy has a great influence on the performance of the algorithm. Sun proves that the contraction–expansion (CE) coefficient must be less than 1.781 in order to make the particles converge reliably. The dynamic non-linear decreasing control strategy of the CE coefficient is proposed in [
38]. According to the characteristics of different optimization problems, different control curves with concavity and convexity are adopted. However, fixed, linear or non-linear decreasing are all non-feedback control strategies that are obtained by simulation experiments or empirical formulas. This kind of coefficient control strategy has weak self-regulation ability. It cannot adjust the coefficient according to the change of search stage. Liu proposes a novel dynamic CE coefficient for QPSO involving two factors, namely, the evolution speed factor and aggregation degree factor in [
39], which can be modified on-line. The fuzzy logic control is one of the most popular control techniques, and it is often used to adjust the parameters of control systems or other algorithms adaptively. The fuzzy logic control is applied in numerous mixes with other algorithms such as the fuzzy logic for inertia weight particle swarm optimization in [
40], immune particle swarm optimization in [
41], neural networks in [
42], model-free adaptive control in [
43], and virtual reference feedback tuning in [
44]. At present, we have not seen the work of using fuzzy control logic to adaptively control the CE coefficient in QPSO.
This article starts with an introduction of the situation and difficulty of CPS periodic scheduling.
Section 2 discusses the related works. A multi-objective optimization problem of non-strictly periodic scheduling in CPS based on time-triggered multi-hop network is proposed in
Section 3 for first-level scheduling. A fuzzy-controlled quantum-behaved particle swarm optimization (FQPSO) algorithm for adaptive parameter selection is proposed in
Section 4.
Section 5 introduces an improved incremental scheduler based on the SMT solver and schedulability ranking of frames to accelerate computation. The experiments and performance discussion are presented in
Section 6. This article finishes with conclusion.
This work is different in that it is the first to propose a two-level scheduling approach, which includes optimization and satisfiability parts, to simplify NP-complete complexity. It is also the first to define the load balance of message distribution in the macro cycle in order to improve an incremental schedule of periodic time-triggered messages. Fuzzy logic is firstly used to adaptively adjust CE coefficients in QPSO. An innovation of schedulability ranking in an incremental SMT solver is proposed to solve the problem of excessive callbacks in previous work.
3. Scheduling Optimization
3.1. System Model
This article improves an Ethernet train communication network (TCN) topology in IEC61375-3-4 [
45] as an application platform. TCN is a typical real-time CPS, whose topology is hierarchical with one or more train backbone subnets and one or more Ethernet consist network (ECN) subnets. Train backbone nodes (TBN) and links along the train are used to connect active ECN subsets together, while the ECN interconnects end devices (ED) located in one consist and transmits data frames between EDs and between EDs and TBNs through consist network nodes (CNNs) and links. We assume links as full duplex connecting multiprocessor nodes to transmit messages between devices in both directions. Each ED contains a set of homogeneous cores on an electronic control unit (ECU) with output and input ports acting as communication end points that can send or receive only one single message at a time.
The work of this paper is based on time-triggered improved Ethernet and focus on non-strictly periodic process data in TCN. The communication management, routing protocol, communication layer model, and packet format of TCN in our work are regulated by IEC 61375-2-3 [
46], which defines a state-of-the-art protocol based on switched Ethernet named train real-time data protocol (TRDP). Our work focuses on the optimization of a static scheduling table that is not confined to specific protocol and is easily extensible to other distributed control system communication networks such as bus, mesh, wireless communication, and so on. Similarly, on the premise of network clock synchronization and determined message routing, the structure of network topology, such as a ring, a tree, or a star, or redundant structures as parallel and ladder structures in IEC 61375-3-4, does not affect the optimization model, analysis, and conclusion of this paper.
The proposed communication topology in this article is shown in
Figure 1. A ring topology is formed by three switches and five devices distributed in it. Formally, the physical topology of the information system in CPS can be depicted by a directed graph
G (
V,
E), where
V =
ED ∪
CNN contains all the EDs and CNNs, and
E is the set of physical links. One physical link consists of two dataflow links in opposite directions. A dataflow link
lk ∈
L represents a directed communication link between two adjacent nodes, and
L is the set of dataflow links in a network. A dataflow path
dpi is formed by an ordered sequence of
lk connecting one sender to one or several receivers. For example, the dataflow path
dp1 is depicted by a dotted line from ED1 to ED5, which can be also denoted as the ordered
lk sequence, i.e.,
l1→
l5→
l9 in
Figure 1.
All the source and sink ports of EDs in the fieldbus network share the common collision domain so that only one port can occupy the physical link for message transmission. As switched Ethernet is isolated from ports of switches to their respective collision domains, the ports of all EDs can send or receive messages at the same time. When there is no overlap of dataflow paths, i.e., dp1 ∩ dp2 ∩…∩ dpn = ∅, messages can be scheduled as several independent fieldbuses at the same time in the network. Otherwise, dp1 ∩ dp2 ∩…∩ dpn ≠ ∅, and overlapped links should be co-scheduled through the store and forward mechanism of Ethernet switches, as long as the end-to-end delay can meet the deadline requirements of messages, which enhances the flexibility and utilization of the network.
In
Figure 1, communication between source and sink EDs is distributed globally by dataflow paths and relayed through switches. Messages need to go through several dataflow links during the transmission process. Due to the collision domain isolation characteristics of switch ports, the transmission of messages can be divided into a series of frame instances.
The communication between CPS physical units is undertaken by messages. Assume that a periodic message is transmitted in the payload of a dedicated frame
fi. We denote the set of all frames by
F with
fi ∈
F. For each frame, the prior attributes are given as shown in Equation (1):
where
ri is the release time, which specifies the earliest time that a message can be sent from source ED.
di is the message deadline, which represents the absolute time from 0 moments to the moment when a message is completely received by its destination.
ei is the end-to-end deadline from the moment when a data frame is sent from source ED to the moment when it is completely received by its destination.
pi is the amount of time that the frame of the message occupies the dataflow link, and
Ti is the time period of the message.
N is the number of frames.
Under synchronization, the global network communication time is divided into macro periodic units denoted by TMp, which is equal to the least common multiple (LCM) of all the message periods in the network. In addition, each link has its own basic period denoted by Tbp, which is equal to the greatest common divisor (GCD) of the message periods on this link. TMp is an integral multiple of Tbp.
3.2. Objective Function
In order to ensure the real-time performance of time-triggered periodic messages as well as keep more free time slots in the basic periods, the optimization target concentrates on the frame jitter, transmission delay, and load balance of the macro period in the k-th link. In a multi-hop non-strictly periodic scheduling problem, it is usually difficult to satisfy the collision-free constraints through optimization. In this section, we firstly assign frame instances to basic periods, regardless of their overlapping issues.
s(
i,
j,
lk) and
s(
i,
j + 1,
lk) are the sending time of the
j-th and (
j + 1)-th instances of frame
fi on dataflow link
lk.
M is the number of frames on
lk. The sum of jitters on
lk can be formulated as shown in Equation (2).
The periodic frames should be distributed uniformly so as to prevent data overflow and improve the load more balanced for incremental scheduling. The standard deviation of basic period utilization in a macro period can be used to define the load balance and is given by Equation (3):
where
Nk is the number of basic periods in one macro period on
lk and equals to
TMp/
Tbp,k,
TMp is the macro period, and
Tbp,k is the basic period for
lk.
, if any frame instance of
fi is located in the
r-th basic period,
pi(
r) equals the link occupancy time of
fi. Otherwise,
pi(
r) = 0.
For every message flow, the sending time offset of frames should satisfy not only the constraints on the application level but also on the network level. The temporal constraints of the network are divided into eight types and analyzed in detail [
28]. In this article, we use a directed graph to describe them formally. In order to facilitate description, five messages are assumed with their prior attributes defined in Equation (1) shown in
Table 1, while the directed graph
G is formed as shown in
Figure 2.
The link occupancy time pi = len × bTD + TGAP, where len is the frame length in bits. bTD is the single bit transmission time, which is approximately set as 0.01 μs, and TGAP is the inter-message protection interval with the value of 1.12 μs in this case. For example, if the unicast M1–M4 length is 72 bytes long and multicast M5 is 134 bytes long, we have p1-4 = 6.88 μs, p5 = 11.84 μs.
In
Figure 2, there are 19 tasks in total marked by
Nx with
x from 0 to 18.
N0 marks the start time of the system, and the link occupancy time is 0. The positive weight of the directed edges between the task nodes in
Figure 2 is depicted as
representing the link delay
TLD on
lk, which consists of two parts: physical links and connected switches. The positive weight between
N0 and the first task node of message transmission represents the release time
ri. The negative weight between the last task node and
N0 is given by
and indicates the absolute deadline of the frame from 0 moments. The negative weight between the last and first task node is given by
and indicates the end-to-end deadline of each message.
The weight of the directed edges in
Figure 2 forms a matrix
W, which represents all the temporal constraints of frame instances during scheduling, as shown in Equation (5).
Nx and
Nx’ are tasks in
Figure 2 with
. The element ‘*’ in
W indicates that the tasks of
Nx and
Nx’ have no constraint relations. The temporal constraints can be formalized as:
Therefore, ∀
lk ∈ L, the final objective function can be expressed as shown in Equation (6).
where
F(
lk) is the objective function of
lk,
J(
lk) is the overall jitter value given by Equation (2), and
J(
lk)
max and
J(
lk)
min are the maximum and minimum value of
J(
lk), respectively.
B(
lk) is the standard deviation of basic period utilization in a macro period of
lk given by Equation (3), and
Bmax and
Bmin are the maximum and minimum value of
B(
lk), respectively.
a1 and
a2 are the normalized weight coefficients. The first constraint means that the basic period
Tbp,k should not overflow with frames. The second constraint is the temporal constraint given by Equation (4).
5. Incremental Scheduler Based on SMT Solver
After FQPSO, frame instances are assigned to basic periods and meet the requirement of the objective function and constraints. However, since the collision-free constraints are not considered in the above process, frame instances will overlap each other in the table. The purpose of this section is to adjust the offset of frames to satisfy collision-free and temporal constraints in every basic period based on an improved SMT solver.
The FQPSO part greatly reduces the scale of collision-free constraints, which account for the majority of total constraints for the SMT solver. Delay and precedence can be constrained by Equations (4) and (5). The collision-free constraint is formally defined as shown in Equation (17):
where
L is the set of dataflow links, the
j-th basic period on dataflow link
lk is denoted by
,
fv and
fu are frame instances in
,
and
are their sending offset, and
pv and
pu are their link occupancy times.
We propose an improved SMT solver to solve constraints (4), (5) and (17). This article defines the priority to decide the sequence of incremental frames scheduling. Not only the temporal urgency of frames, i.e., the end-to-end deadline in
Figure 2, but also the interference of scheduled messages with unscheduled ones needs to be considered. The period utilization (
PU) is defined as the bandwidth needed to transmit a frame
fu on the premise that FQPSO has already allocated more urgent frame instances to basic periods, as given by Equation (18).
means the bandwidth occupied by other frames except fu in the j-th basic period of dataflow link lk.
means the fu transmission requires a fraction of time equal to pu every Tu time units.
Thus, the priority of each frame fu is set as a vector of two components priority = rank(min(eu), max(PUu)), where eu is the end-to-end deadline of fu and is compared firstly by the most critical parameter. PUu is compared secondly for frames that have the same end-to-end deadline. This priority value defines the difficulty of frame instance scheduling. The frame instance with less free scheduling space should be scheduled earlier.
Figure 4 shows the incremental SMT scheduler based on the schedulability ranking in our approach.
Prior_Rank is the set of priority queues. Ready_to_Sch is the set of frames that are ready to schedule. Sch_table is the completed schedule table. UnSch is the set of unscheduling frames. Lines 5–9 are schedulability ranking approach, and these are used to compute the priority of frames and sort them to form the Prior_Rank queue. The lower priority is at the tail and the higher is at the head. In each iteration, the frame instance with the highest priority in the queue is added to Ready_to_Sch and scheduled by the SMT solver first.
Line 15 uses the frames in Ready_to_Sch, basic periods scheduling results from FQPSO, and offset of scheduled frames in Sch_table to generate SMT constraints and check whether the logic context is satisfiable (line 16). If so, the SMT solver gives a yices_model that satisfies the logic context (line 17) and returns the feasible solution into local array sch_offset from SMT pointer & smt_offset (line 18). The function place() in line 19 is used to explicitly add these time offset of frame instances to Sch_table. Then, the Prior_Rank is updated by deleting the successfully scheduled frame fi from the head of the queue.
If the current frame fi is not schedulable, the last successfully scheduled frame fj and its offsets are taken out of Sch_table and pushed back to Ready_to_Sch (lines 22–23). In the next iteration, fi and fj are scheduled together by lines 15–20 to find a feasible solution. If there is still no feasible solution when all the scheduled frames are popped out from Sch_table for backtracking, the frame with minimum priority fun is identified as a problematic frame and put into UnSch to prevent a scheduling crash (lines 25–31).
After all the frames in Prior_Rank are scheduled, the frames in UnSch are added to the SMT solver together and they check whether there is a feasible solution for them without considering the basic period limits from FQPSO. If so, the final schedule table Sch_table is returned. Otherwise, the Sch_table with currently scheduled frames and UnSch with unscheduled ones are returned.
After the two-level schedule is calculated, if there are a few messages newly adding into the system, the incremental SMT scheduler will be called to determine whether the idle time of the schedule table can meet the constraints of frames. When there is enough appropriate idle time, the scheduler will be updated with new frames online in several to hundreds of milliseconds. Otherwise, the schedule table needs to be recalculated off-line.