An Efficient Multi-Lane SpaceFibre Core for Spacecraft Data-Handling Networks

Abstract

As one of the key technologies in spacecraft data-handling networks mounting high-resolution payloads, the multi-lane native SpaceFibre standard is designed to meet the demands for the high-speed and reliable interconnections between satellite payloads. A few studies are about the implementations of a multi-lane SpaceFibre core. Even though they exhibit good performance, several limitations still, more or less, exist. We detail an efficient multi-lane SpaceFibre (E-ML-SpFi) core for high-rate and fault-tolerant data communication. The E-ML-SpFi core exploits a hierarchical method to decouple the complex control logic of the retransmission mechanism. The core employs a modified word re-ordering block to map an extended set of words on a real-time variable number of data-sending lanes. Meanwhile, the core introduces the architecture of a fast alignment subsystem to accelerate the alignment process. A hardware implementation on XC7Z100 FPGA shows that the E-ML-SpFi core has complete functionality stipulated by the standard and requires limited resources, with 6290 (2.26%) LUTs and 8252 (1.49%) FFs. The maximum operating frequency of the core is 200 MHz. Moreover, the recovery time is reduced up to 30.9% compared with previous work. Therefore, the core provides a valid solution for future interconnection networks onboard spacecraft.

1. Introduction

Spacecraft data-handling networks [1] are required to sustain multi-gigabit capabilities, due to the increasingly high data volume produced by high-rate payloads, such as hyperspectral imagers [2] and synthetic aperture radars [3]. The demand for high bandwidth leads to some limitations of single-lane standards [4,5,6,7,8,9], and some multi-lane standards [10] support only an even number of lane configurations. To enhance the data processing ability in a space environment, the European Space Agency developed the SpaceFibre [11] standard for interconnection networks designed for satellite payloads.

SpaceFibre supports an arbitrary number of lanes between 1 and 16, and each lane has a maximum data rate of 6.25 Gbps. To handle a lane failure when a cable is damaged during the launch of a spacecraft, SpaceFibre provides hot redundant lanes [12] and graceful degradation. The standard adopts built-in integrated quality of service (QoS) [13], allowing data from different virtual channels (VCs) to multiplex over a link. In SpaceFibre, a fault detection isolation and recovery (FDIR) mechanism [14] is responsible for resending corrupted data, which are generally caused by space radiation. The standard introduces a token-based flow control mechanism to avoid overrunning the recipient resources. Several implementations of single-lane SpaceFibre cores are proposed in the literature [15,16,17,18,19]. Nevertheless, a limited reference is provided by the implementations because the inclusion of a multi-lane attribute brings major adjustments in the architecture of a SpaceFibre core. For these reasons, implementations of a multi-lane SpaceFibre core, compliant with all features, face complex challenges.

Implementations of multi-lane SpaceFibre cores can be found in the publications [20,21,22,23]. A multi-lane hardware prototype was proposed in Reference [20] for the development of the SpaceFibre protocol. Regardless, the prototype still needs to be improved in terms of its resource consumption. Reference [21] reported a silicon implementation of a multi-lane SpaceFibre core, with two lanes for low-cost satellite electrical ground segment equipment. However, the data signaling rate of the core is limited by the maximum number of lanes. In Reference [22], a multi-lane core with good performance was presented. As a proprietary core, the work does not report any details of the core. Reference [23] detailed the architecture of a multi-lane core and introduced a test campaign to measure recovery time after a lane failure, whereas the resource overhead and recovery time consumed by the core need to be improved.

To solve the above problems, we propose the complete hardware architecture of an efficient multi-lane SpaceFibre (E-ML-SpFi) core, featuring easy expansion, low resource consumption, and high performance. The three main contributions of this work are given as follows.

The first is the powerful hierarchical control method, which is responsible for simplifying the complex control logic [24] of the retry mechanism. The hierarchical control is divided into three levels, namely the response control level, the storage control level, and the retry control level. As a result, the hardware resources are saved and the Data-Link layer recovery (DLRec) time is shortened.

The second contribution is the modified word re-ordering block. The modified block introduces an extended set of words compared with the original block [25]. Each type of word in the set corresponds to rules belonging to a ruleset specific to SpaceFibre. Based on the ruleset and the initial order of words, the block automatically splits the words on a variable number of data-sending lanes. Consequently, the core can accommodate the entire ruleset for SpaceFibre and can accomplish adaptive control.

The third one is the design of a fast alignment subsystem, which is adopted to speed up alignment operations. The subsystem combines three main techniques: advance extractions, characteristic bits, and parallel examinations. Therefore, the Multi-Lane layer recovery (MLRec) time is reduced.

Experimental results show that the E-ML-SpFi core supports the full-hardware protocol stack, that the core has low area, that the core allows a high operating clock frequency, and that the core has better performance in terms of recovery time.

The rest of this paper is organized as follows. Section 2 briefly introduces the SpaceFibre protocol. Section 3 details the architecture of the proposed E-ML-SpFi core. The results of hardware implementations are discussed in Section 4. Finally, Section 5 summarizes the main conclusions.

2. SpaceFibre Standard

The SpaceFibre protocol stack has a Network layer, a Data-Link layer, a Multi-Lane layer, a Lane layer, a Physical layer and a management information base (MIB). An MIB is responsible for configuring and monitoring the status of each layer.

The Data-Link layer is responsible for QoS, flow control and FDIR services. The QoS technology arbitrates requests for data transmission from VCs, based on bandwidth reservations, priority and scheduled QoS. The flow control service utilizes flow control token (FCT) control words to indicate the available room of the VC at the receiver side. The error recovery mechanism can be found later in the current section.

In the Data-Link layer, data segments from each VC are encapsulated in data frames. Aside from data frames, multi-purpose broadcast messages with two bytes are segmented into broadcast frames, which have a similar format to data frames. Figure 1 shows the format of a data frame for a multi-lane link. A data frame contains at most L × 64 data words, where $L$($L\in [1,16]$) is equal to the maximum number of data-sending lanes. Moreover, a data frame begins with a start-of-data frame (SDF) control word being added before the data segment and ends with an end-of-data frame (EDF) control word added afterwards. An EDF word is made up of the control character K28.0, data characters containing a 16-bit cyclic redundancy check (CRC) field, and a data character containing a sequence number.

The description of the error recovery mechanism starts with sequence numbers, which are used in EDF words, FCT words, idle frames, broadcast frames, acknowledgement (ACK) control words, negative acknowledgement (NACK) control words and FULL control words. The current value of a transmit sequence number (TSN) is incremented after a transfer of data frames, broadcast frames, and FCT words. The TSN is kept when a FULL control word or an idle frame is sent. For convenience of description, the frames causing the increment of a sequence number are defined as increment frames in this study. When increment frames are transmitted for the first time, they are stored by the Data-Link layer. The stored data are held until the destination node acknowledges the correct reception through an ACK word. Similarly, a receive sequence number (RSN) is maintained on the receiver side. The receiver side adds the RSN and one, once an increment frame is received correctly, which is checked by the CRC checksum, TSN, and RSN. Meanwhile, the receiver side sends an ACK word to the transmitter. However, if the far-end of the link sends an NACK word with the current value of the RSN and the reversed polarity flag, the near-end retransmits the corresponding increment frames, which were previously saved. Before the retransmission, the RETRY control word is exploited to inform the far end that an NACK has been correctly received.

Control words and frames are interleaved over a SpaceFibre link with the following precedence: RETRY > broadcast frame > ACK/NACK > FCT > FULL > data frame > idle frame. Furthermore, the precedence of retransmitted increment frames is higher than that of increment frames generated for the first time.

The Multi-Lane layer sustains multiple lanes over a SpaceFibre link to provide high throughput and high reliability. The bandwidth of a link is proportional to the number of lanes. The set of data and control words passed from the Data-Link layer, one to each data-sending lane, together form a data-sending row. Similarly, a data-receiving row is formed by words from data-receiving lanes.

There is a ruleset in the Multi-Lane layer for forming a data-sending row. Data words from a data frame are spread across the data-sending lanes. The first data word of the frame is distributed to the data-sending lane with the lowest number. Each control word of the Data-Link layer is replicated and goes over all data-sending lanes. However, EDF words make an exception because the CRC value of each lane is different. Moreover, PAD control words are inserted to a data frame if the number of words in a data frame is not a multiple of M, where M is the number of data-sending lanes. A data word from a broadcast frame or an idle frame is replicated and sent over all data-sending lanes. To simplify the description in this study, a replicated row (RR) is introduced, which is composed of replicated words (RWs) with the identical value.

An alignment process is employed by the Multi-Lane layer for data coherency. ACTIVE control words are sent by each end of a multi-lane link to exchange the information regarding active lanes. ALIGN control words are used to align a data-receiving row. An ACTIVE word and an ALIGN word are repeatedly sent if the far-end needs to align data-receiving lanes. The alignment process is controlled by the alignment finite-state machine (FSM), dictated by the SpaceFibre protocol.

The Lane layer provides character synchronization, clock compensation, 8B/10B encoding, and 8B/10B decoding. An 8B/10B decoder introduces disparity check and detects invalid codes to find bit-flip errors. Moreover, the layer is applied to lane initialization and error detection. Each lane of a multi-lane link contains its own lane layer. The Physical layer is responsible for serializing the symbols provided by the Lane layer and for transmitting the serial data signal over copper cables or optical fiber cables. Meanwhile, the physical layer receives the data signals from physical mediums and de-serializes the signals.

3. Architecture of the E-ML-SpFi Core

3.1. Overall Architecture

In this section, we fully illustrate the architecture of the E-ML-SpFi core. Figure 2 shows a block diagram of the E-ML-SpFi core. The architecture does not specify the number of VCs and number of lanes, which can be easily expanded. The proposed core integrates an MIB, a Data-Link layer, a Multi-Lane layer, L Lane layers, and L Physical layers. Each layer is made up of several interconnected subsystems.

The broadcast subsystem is constituted by an output broadcast (BC) buffer, a BC flow control block, a BC-framing block and an input BC buffer in the Data-Link layer.

The QoS subsystem comprises VC buffers, a media access controller (MAC), a data-framing block and a VC De-Multiplexer on the receiver side of the Data-Link layer. The Data-Link layer has $N$($N\in [1,32]$) pairs of input VC buffers and output VC buffers, which are employed to provide separate packet streams. The arbitration between VCs is realized by the MAC providing a high degree of QoS with priority, reserved bandwidth, and deterministic delivery. Data segments, which are read out the chosen output VC buffer, are collected into a data frame by the data-framing block.

The flow control subsystem is made up of an FCT-framing block, output flow control blocks, and input flow control blocks. An input VC flow control block sends a request for FCT control words to the FCT-framing block if a certain number of words are read out an input VC buffer. Each output flow control block at the local node monitors the space available in the corresponding input flow control block at the destination node by receiving FCT words.

Data frames are backed up in the error recovery subsystem, and so are broadcast frames and FCT control words. The error recovery subsystem is also employed to achieve the retransmission mechanism. Subsequently, control words and data words are interleaved with the precedence described in Section 2. The interleaved traffic is adjusted by the word re-ordering block for adaptive bandwidth. After that, the traffic goes through scramblers and CRC blocks. The word re-ordering block and the word concentration block on the receiver side constitute the switching subsystem.

On the receiver side of the Data-Link layer, data from the Multi-Lane layer is transmitted to the error detection subsystem after CRC and descrambling processes. The subsystem requests an ACK control word if an increment frame mentioned in Section 2 is correctly received. Similarly, an NACK control word is requested in case of an error.

The multi-lane monitor FSM monitors the real-time status of all lanes and promotes hot redundant lanes to data-sending lanes in the event of a lane failure. On the transmitter side of the Multi-Lane layer, words from the Data-Link layer are distributed to each data-sending lane. The multi-lane control words are inserted into the traffic for alignment, as depicted in Section 2. On the receiver side, the alignment subsystem is employed to provide aligned data-receiving rows.

The hardware implementations of the Lane layer and the Physical layer leave no space for interpretation [26]. However, the Data-Link layer and the Multi-Lane layer have significantly different hardware implementations due to the high-level description in the SpaceFibre protocol. The following section will concentrate on the key technologies in the Data-Link layer and the Multi-Lane layer, alongside the design choice made and the impacts of the choice.

3.2. Key Technologies

3.2.1. Hierarchical Error Recovery

SpaceFibre provides a mechanism of retransmission [27] to achieve fault tolerance. The data retry mechanism without any optimization has high complex logic [24] and area consumption. To simplify the design and implementation of the mechanism, we propose a hierarchical architecture of error recovery, which involves three levels. Details about the architecture are explained in the following paragraphs, focusing on a response control level, a storage control level, and a retry control level.

The response control level, which is the first level, employs the lightweight and reliable response control FSM to control the behaviors of other layers. The lightweight FSM shown in Figure 3 has only three states, and the reliability of the FSM is ensured by the storage of the information carried by an ACK or NACK word. If an ACK or NACK word is correctly received and the FSM is not in the Idle state, the last received information from the word will be kept in a register, which can avoid the omission of an ACK or NACK word and overflow in retry buffers. The FSM moves to the Idle state in case the value of the signal Done from the storage control level is hexadecimal 7. Once the FSM returns to the Idle state and an ACK control word is stored in the register, the FSM enters the ACK Processing state. Similarly, the FSM shall move to the NACK Processing state if an NACK control word is registered. In the ACK Processing state and the NACK Processing state, the signal AnsDeal is updated and transmitted to the storage control level.

The second level is the storage control level, as depicted in Figure 4, which utilizes resource-optimized storage architecture to save the increment frames not already acknowledged. The architecture employs a differentiation scheme to store increment frames instead of sharing a mass retry buffer. The scheme stores data frames, broadcast frames, and FCT words in a data retry buffer, a BC retry buffer, and an FCT retry buffer, respectively. In addition, only one word in the data-sending row is stored, which consists of broadcast frames or flow control words. Moreover, an FCT control word only makes a backup of the channel number field and the unsigned sequence number. If the state of the response control FSM in the response control level is the NACK Processing state, the storage control FSMs assert the requests for retransmission and send to the retry control level. After the enable signals are received from the retry control level, the retransmission of the backups in retry buffers starts.

The retry control level is responsible for controlling the reuse of framing blocks to reduce circuit consumption, which is the third level. The level is composed of retry control units (RCUs) for data frames, broadcast frames, and flow control words. In the following, we take the RCU for data frames as an example to describe the design of RCUs. When a data frame needs to be sent, the RCU for data frames requests the transmission of data frames, as shown in Figure 5. Once the request is answered, the RCU transfers the corresponding data sources to the data-framing block. The selection of sources is based on the signal ReReqData and the signal ReqData. When the ReReqData is de-asserted and the signal ReqData is true, data words from an output VC buffer are selected. If the signal ReReqData is asserted, the RCU sends the data words from the data retry buffer in the storage control level to the data-framing block.

The proposed hierarchical architecture is composed of three levels. The first level, called the response control level, controls the entire retransmission process through a lightweight and reliable FSM. The second level is the storage control level, which employs the resource-optimized storage architecture. The retry control level, which is the third level, reuses the framing blocks to improve the utilization of hardware resources. Consequently, the hierarchical architecture provides a top-down description of the retry mechanism and combines a variety of strategies to simplify the complex implementation logic required by the retry process.

3.2.2. Modified Word Re-Ordering Block

As presented in Section 2, L is equal to the maximum number of data-sending lanes. In addition, M is the real-time number of data-sending lanes. In case of lane failures, the bandwidth of a link decreases. For this reason, the core needs to distribute L consecutive words to M active lanes with the initial order. A block for general purpose, described in [25], has a capability of shaping the data words of a data frame from a fixed number to a real-time variable number. Nevertheless, this block appears to be lacking in some rules special for SpaceFibre. In the protocol, data words, EDF control words and RWs are interleaved over a link with the precedence, as mentioned in Section 2. Besides, PAD control words shall be introduced to a data frame, which is also mentioned in Section 2.

In view of this, a modified word re-ordering architecture is specifically designed for SpaceFibre networks, which assigns an extended set of words to M data-sending lanes in order of appearance. The extended set of words contains data words of data frames, EDF control words, PAD control words, and RWs. Each type of word has its own rule in the assignment to data-sending lanes, which is described in Section 2.

Figure 6 shows the architecture of the block. The block includes two re-ordering registers (RORs), a modified re-ordering FSM, and a reshape module. Words imported to the block are shifted from data_in to ROR_1 and from ROR_1 to ROR_0. The width of signal data_in is L words. Based on fixed L, real-time M, and the type of words, the FSM updates the value of the signal Index. According to the signal Index, the reshape module reads M words stored in RORs and changes, in real time, the width of a data-sending row. The detailed process of forming a data-sending row is described as follows.

If M is the same as L, the signal Index is always equal to zero and the words in ROR_0 are read out by the reshape module to constitute an L-word data-sending row. Additionally, the block does not affect the shift between data_in and RORs. When $M<L$, the following different situations are presented.

If the row of ROR_0 is made up of data words from a data frame and the class of the words in ROR_1 is an RR, mentioned in Section 2, the value of the signal Index shall be maintained. In this case, words in ROR_0 are maintained, and words in ROR_1 are read and are mapped on the data-sending lanes. Afterwards, the modified re-ordering FSM allows the shift between data_in and ROR_1.

When ROR_0 has data words from a data frame and the words in ROR_1 are EDF control words, the signal Index is maintained and the reshape module calculates the number of PAD control words to make the length of the data frame a multiple of M. Subsequently, the signal Index is cleared.

If the words in RORs are data words from data frames, the value of the signal Index varies depending on (1), where i is the current value of the signal Index and $i^{\prime }$ is the latest value.

$$i^{\prime }=(i+M)\mathrm{mod}L$$

(1)

Due to the updates in the signal Index, the modified word re-ordering architecture accomplishes adaptive bandwidth. By solving the problem of random insertion of RWs into data frames and filling data frames with PAD control words, the architecture is compatible with the whole ruleset of SpaceFibre, so that it can be better applied to various complex and changeable environments.

3.2.3. Fast Alignment

Due to differences in physical mediums, transmission distances, and interference factors, data row misalignment appears. The misalignment may cause the failure of data consistency. Therefore, it is necessary to ensure receiving-row alignment. However, the SpaceFibre protocol [11] and existing designs [20,21,22,23] only describe the alignment of FSM and do not illustrate the concrete architecture of the alignment process.

To overcome the limitation, we propose the architecture of a fast alignment subsystem. The architecture, shown in Figure 7, includes an alignment FSM, condition module and L lane-locking modules. Each lane-locking module contains a locking unit, a detection unit, an examination unit, a pre-detection unit, and a three-position FIFO buffer. Each lane has a separate FIFO buffer to store the words from a data-receiving lane. In the following, we focus on the lane-locking module, which plays an important role in speeding up the alignment process.

The pre-detection unit directly recognizes ACTIVE words in the incoming data traffic from a Lane layer, without having to extract ACTIVE words from the words read out of a FIFO buffer. At the same time, the unit extracts the information regarding active lanes at the far-end and transfers the information to the condition module. In the module, the information is utilized to form the signal flag_align and the signal flag_misalign, which are important conditions of the alignment FSM. If the value of the signal flag_misalign is 1, a misaligned condition occurs. By reducing the time used to generate conditions, the unit makes it possible to shorten the time required by alignment operations.

The detection unit divides the words from an alignment FIFO buffer into three groups based on characteristic bits, so that the unit balances the resource utilization and speed of the fast alignment subsystem. The groups are data words, ALIGN control words, and other control words. PAD control words in this unit are considered data words. Characteristic bits are the minimum number of bits distinguishing different groups of words. For instance, data words have only two characteristic bits: bit 33 and bit 32. If a word is not recognized as an ALIGN control word and the alignment FSM is in the Not Ready state, the word shall be read out of the buffer. Once an ALIGN word is found, the detection unit asserts the signal getAlign to notify the examination unit.

The examination unit checks the validity and correctness of an ALIGN control word at the same time, instead of a serial examination, for the fact that only the correct and valid ALIGN control words can be used for alignment. The results of parallel examination are sent to the condition module to generate the signal flag_align. By reducing the time spent on forming conditions, the unit makes the alignment process much faster.

The locking unit allows one to lock a valid ALIGN word, correctly received, to form a valid data-receiving row by de-asserting the signal en_ed of the buffer when the Alignment FSM is in the Not Ready state. An ALIGN word constituting a valid data-receiving row is identified by the examination results from the examination unit. Associated with the buffer forbidden to read, the data-receiving lane is regarded as an aligned lane. After all lanes are aligned, each locking unit reads the ALIGN control word simultaneously.

The basic idea of fast alignment consists of accelerating the generation of conditions. Both advance extractions and parallel examinations respect the idea. In addition, the usage of characteristic bits reaches a compromise between speed and resource consumption.

4. FPGA Implementation and Analysis

The proposed E-ML-SpFi IP core has been synthesized and implemented on the XC7Z100 FPGA. As shown in Figure 8, board-level verification platforms aim at evaluating the traffic between two SpaceFibre end nodes connected by optical fiber cables. Each verification platform is equipped with a data generator, a broadcast generator, a parameter configurator, a status recorder, an integrated logic analyzer (ILA) and an E-ML-SpFi IP core. The data generator and BC generator are responsible for continuously feeding two IP cores with new data. A parameter configurator is exploited to configurate parameters. A status recorder is responsible for monitoring status information in the E-ML-SpFi core. The core depicted in Figure 8 contains four VCs and four lanes. Each VC buffer is configured to store 256 36-bit data words. Each lane has one of the seven data signaling rates specified by the SpaceFibre protocol [11].

4.1. Board-Level Verification Results

A board-level platform can effectively verify the complete functionality of the E-ML-SpFi core. During the verification stage, we verify, one by one, the compliance of the core to concrete requirements in SpaceFibre. The following paragraphs show several main results, which involve full bandwidth communication, exploiting all active lanes without any loss of data, hot redundancy and the functions mentioned in Section 3.2.

The correctness of data is guaranteed at a packet level and a frame level. In terms of the packet level, each packet generated by a data generator has incremental data and a CRC checksum for content integrity, and so does each broadcast message. At the frame level, the correctness of data frames, broadcast frames, FCT control words, and other data-link control words is ensured by corresponding CRC fields and TSNs. The far end of a SpaceFibre link calculates a CRC checksum based on received data and compares the checksum with the CRC checksum contained in the received data. If checksums are equal, the received data is considered correctly received without any CRC errors. The received data is corrupted by a sequence error if the TSN inserted in the received data does not match the RSN, as described in Section 2. Meanwhile, the sequence error is recorded. Besides, the frame errors are also monitored, which are caused by incorrect frame length and unexpectable frame delimiters.

Figure 9 shows the ILA results of a test to verify full-duplex communication with full bandwidth. In the test, L is set as four so that the link does not have any hot redundant lanes. Each graph in Figure 9 includes the words transferred from the Multi-lane layer to M Lane layers and includes error records. Errors are grouped into three categories, described in the paragraph above: CRC errors, sequence errors, and frame errors. The error records are read every hour for twelve hours. From Figure 9, we can conclude that the traffic in the E-ML-SpFi core is directed over the active lanes and that the core is compatible with 1×, 2×, 3×, and 4× active lanes. Furthermore, error records are 0 under each lane configuration.

The ILA waveforms of hot redundancy are shown in Figure 10. The test case, making L equal to three, has one hot redundant lane, namely lane 3, which has a higher lane number than that of data-sending lanes. In Figure 10a, the hot redundant lane sends pseudo-random bit sequence (PRBS) words, and valid information is only passed to data-sending lanes: lane 0, lane 1, and lane 2. As depicted in Figure 10b, lane 3 is automatically promoted to a data-sending lane when lane 2 is disconnected. As a result, the E-ML-SpFi core achieves the function of hot redundancy.

Figure 11 provides the waveforms of a test to verify the function of error recovery captured by ILA. As shown in Figure 11a, the far end of a link starts with four data-sending lanes. After the failure of lane 3, the far-end indicates to near-end that the current value of RSN is hexadecimal 74 by sending an NACK word. Moreover, the receive polarity flag in Figure 11a is reversed. As described in Figure 11b, a retransmitted data frame, composed of words stored in a data retry buffer, is generated at the near end after the analysis of the received NACK. As shown in Figure 11a, a RETRY control word is received by the far end before the retransmitted frame. The far end then begins to add the current value of RSN and one, which indicates that the error recovery is completed.

Figure 12 shows the ILA waveform of the modified word re-ordering block with the real-time disconnection of lane 3. As shown in Figure 12, the modified block maps data words from a data frame on lane 2, lane 1 and lane 0. The first data word in the data frame goes over lane 0, the next over lane 1, up to lane 2. When EDF words appear, the data frame introduces two PAD control words. Therefore, the modified block can split L words belonging to an extended set on M data-sending lanes in order of appearance.

The ILA waveform of a test to verify the function of alignment operations is shown in Figure 13. In the test, lane 2 is disabled and active lanes at the near end are lane 3, lane 1, and lane 0. When the three active lanes are aligned and active lanes at the far end are the same as the near end, the condition module asserts the signal flag_align. Subsequently, the alignment FSM transfers from the Idle state to the Near-End Ready state, and the ALIGN control words are read out simultaneously. The FSM then enters the Both-End Ready state because signal flag_misalign is de-asserted and the data-receiving row is full of data words. Consequently, the alignment subsystem can handle the alignment process, so that data consistency on the receiver side is ensured.

4.2. Hardware Utilization

As a promising hardware-based IP core, a low cost is a required property of the E-ML-SpFi core.

Table 1. Evaluation of hardware utilization.

Design	FPGA	Available LUTs	Available DFFs	CR	MDR (Gbps)	VCs	Lanes	LUTs	DFFs	BRAMs (Kb)
[22]	KU060	331,680	663,360	Yes	6.25	2	4	6633.6 (2%)	9287.04 (1.4%)	-
[23]	KU060	331,680	663,360	No	2.5	2	4	19,771 (6.96%)	28,579 (4.31%)	549
						4	4	23,648 (7.13%)	31,250 (4.41%)	918
proposed	XC7Z100	277,400	554,800	Yes	6.25	4	4	6290 (2.26%)	8252 (1.49%)	612
	KU060	331,680	663,360					7088 (2.14%)	8529 (1.29%)
	XC7K325T	203,800	407,600					6277 (3.08%)	8115 (1.99%)
	XC6VLX240T	150,720	301,400		3.125			6135 (4.07%)	6133 (2.03%)

presents the hardware utilization of E-ML-SpFi core and the comparison of this work with other state-of-art designs in [22,23]. As shown in Table 1, the design in [22] and the E-ML-SpFi core sustain a maximum data rate (MDR) of 6.25 Gbps per lane and each lane has configurable rates (CR). The design in [22], which is equipped with two VCs and four lanes, uses 6633.6 look-up tables (LUTs), accounting for 2% and 9287.04 flip-flops (FFs), accounting for 1.4%. The LUTs required by the design in [23], featuring two VCs and four lanes, are nearly three-times more than the design in [22], and the number of FFs increases to three times. The designs in [22,23] have been implemented on the Xilinx Kintex UltraScale KU060, which contains 331,680 LUTs and 663,360 FFs. Xilinx KU060 is larger than Xilinx XC7Z100 in terms of LUTs and FFs. The E-ML-SpFi core, with four lanes and four VCs, is implemented on XC7Z100 FPGA, utilizing slice LUTs at 2.26% and FFs at 1.49%. We can conclude that the E-ML-SpFi core has similar hardware resources to the core described in [22], which supports only two VCs. Besides, the BRAMs used by the E-ML-SpFi core is reduced by 33%, compared with the design in [22], with four lanes and four VCs. As depicted in Table 1, the E-ML-SpFi core, equipped with four lanes and four VCs is also implemented on KU060, Kintex-7 XC7K325T and Virtex-6 XC6VLX240T. The E-ML-SpFi core on KU060 and XC7K325T FPGAs consumes similar LUTs and FFs with the core on XC7Z100. The core on XC6VLX240T occupies less LUTs and FFs compared with other FPGAs because the implementation of a Physical layer based on Virtex-6 is simpler than that of other boards. However, the simpler Physical layer leads to an MDR of 3.125 Gbps per lane. BRAMs used by the E-ML-SpFi core on different FPGAs are the same. Therefore, the E-ML-SpFi core is a more cost-efficient design, consuming fewer resources compared with other designs.

4.3. Performance Analysis

The previous analysis shows that the E-ML-SpFi core saves more resources compared with other designs, and this conclusion is also confirmed in terms of recovery time. We utilize recovery time to quantify the performance of multi-lane SpaceFibre cores. The recovery time after a lane failure consists of two contributions: the Multi-Lane layer recovery (MLRec) time and the Data-Link layer recovery (DLRec) time [23]. The MLRec time begins with the moment when the Lane layer starts to send IDLEs. The MLRec time ends when the last multi-lane control word is sent and the alignment FSM moves to the Both-End Ready state. The beginning of DLRec time is a RETRY control word transmitted by the near end, and the ending is an EDF control word sent by the same end.

Because the total recovery time of cases with two VCs appears to be higher than that of the cases featuring one VC [23],

Table 2. Evaluation of recovery time.

Case Num.	Design	Avg. MLRec Time (clk Cycles)	Avg. DLRec Time (clk Cycles)	Tot. Rec. Time
				(clk Cycles)	(μs)
1	[23]	81	244.3	323.2	5.17
	proposed	73.1	212	285.1	4.56
2	[23]	79	314.5	393.5	6.30
	proposed	61.9	210.2	272.1	4.36

provides the overall recovery time of cases with two VCs. There are two test cases in Table 2. One case has only one data-sending lane at the beginning, and the second lane is then reactivated (this process can be denoted as Case 1.) Another case starts with two data-sending lanes, and one of the lanes is then disconnected (using Case 2 to represent this process.) In Table 2, Each case is repeated 100 times and the results are averaged. The average DLRec time in Case 1 of both designs is approximately equal. In Case 1, the E-ML-SpFi core has the similar average MLRec time as the design in [23]. In Case 2, the average MLRec time of the E-ML-SpFi core is reduced by 21.6% compared with the design in [23]. The average DLRec time in Case 2 of the E-ML-SpFi core is decreased by 33.2%. Benefiting from the architecture of hierarchical error recovery and fast alignment, the E-ML-SpFi core can save an average 11.8% and 30.8% of total recovery time in the two cases, respectively. The retry process can be completely restored in the 4.36 μs to 4.56 μs range with the system frequency of 62.5 MHz. As a result, the E-ML-SpFi core can achieve an improvement in recovery time.

5. Conclusions

This paper proposed an E-ML-SpFi core based on the SpaceFibre protocol. The architecture of hierarchical error recovery reduces the implementation complexity and execution overhead of the retry mechanism. Benefiting from the well-designed modified word re-ordering block, the core is compatible with different classes of words and various lane configurations, representing a compromise between flexibility and hardware utilization. By using the parallel architecture of alignment operations, the core accomplishes low latency. Implementation results show that the E-ML-SpFi core is fully compliant with the protocol, equipped with a data signaling rate up to 6.25 Gbps. Additionally, the core occupies lightweight hardware resources. Furthermore, the total recovery time in case of a lane failure is reduced up to 30.9%. As a result, the core is a valid baseline to design the emerging spacecraft data-handling network. In future work, the E-ML-SpFi core will experience a long-term comprehensive test in a harsher environment before the core is applied to space missions.