Research on Clock Synchronization of Data Acquisition Based on NoC

Abstract

Data acquisition based on network-on-chip (NoC) technology is a high-sampling-rate data acquisition scheme using low-sampling-rate analog–digital conversion (ADC) chips. It has the characteristics of multi-task parallel communication, being global asynchronous, local synchronous clock distribution, high throughput, low transmission latency, and strong scalability. High-speed data acquisition is realized through the combination of an on-chip network and time-interleaved data acquisition technology. In the time-interleaved sampling technique, the precision of clock synchronization directly affects the precision of sampling. Based on the proposed NOC data acquisition scheme, an improved White Rabbit clock synchronization protocol is applied to high-speed data acquisition to achieve high-precision synchronization of multi-channel time-interleaved sampling clocks. Firstly, the offset of the master clock and slave clock is determined by the PTP protocol, and the offset is corrected to achieve rough synchronization between the master clock and slave clock. Secondly, a digital dual-mixer time difference (DDMTD) is used to measure the phases of the master and slave clocks. After that, the phase of the slave clock is corrected through the dynamic phase-shift function of the clock’s phase-locked loop (PLL). Finally, according to the simulation results in Modelsim, the average absolute error of a TI-ADC sampling clock can be less than 20 ps.

1. Introduction

The development of data acquisition techniques shows a trend towards high sampling rates and high resolution, which gives rise to time-interleaved analog–digital converters (TI-ADCs) [1,2]. TI-ADCs can break the limitations of the ADC sampling rate and resolution caused by semiconductor process technology [3,4]. TI-ADCs use many ADCs with the same parameters to implement parallel data acquisition, and thus, can improve the sampling rate while preserving the resolution. However, the traditional system-on-chip-based (SoC-based) TI-ADCs rely on a bus architecture, and thus, have various limitations on communication bandwidth, clock synchronization, and scalability, etc. [5]. Such problems can be solved by incorporating the network-on-chip (NoC) technology into TI-ADCs [6], so that the sampling rate can be further improved [7].

To ensure data correctness, the clock phases of adjacent TI-ADCs’ sampling channels should be strictly synchronized [8,9]. Higher sampling rates call for higher synchronization accuracy and lower phase errors between the sampling clocks. Since NoC-based high-speed data acquisition takes the router clock as the ADC sampling clock, high-precision synchronization of ADC sampling clocks can be achieved as long as the router clocks are precisely synchronized [7].

IEEE 1588 [10,11] and White Rabbit (WR) [12] are the most commonly used distributed clock synchronization protocols. An active distributed clock generator was proposed in [13] for a multi-core SoC, which can achieve a less than 38 ps phase error between adjacent oscillators at 700–840 Mhz and VDD = 1.1 V. A Bayesian estimation time synchronization (BETS) algorithm was introduced in [14], which uses synchronization error compensation to reduce message interaction in clock synchronization and satisfies the resource constraints of wireless sensor networks. A distributed clock synchronization protocol based on an intelligent clustering algorithm was presented in [15], which allocates different synchronization frequencies according to the established cluster, in order to avoid excessive network access contention. This protocol also uses outlier detection and a second-order regression model to predict potential malicious nodes. Its superiority over other synchronization protocols was proven by simulations on synchronization performance and fault node detection. In [16], with the basic WR network structure of single principal and single subordinate, the test synchronization accuracy could be limited within 200 ps, and even as low as 21 ps.

According to the characteristics of NoC, the improvement in the White Rabbit clock synchronization protocol is applied to the ADC sampling clock synchronization to solve the NoC data acquisition sampling clock synchronization problem. The simulation results show that the mean absolute error of the time-interleaved sampling clock is less than 20 ps. Our main contributions are as follows:

• Based on the NoC, it has the characteristics of multi-task parallel communication such as multi-resource nodes and multi-routes. In this paper, we propose a high-sampling-rate data acquisition scheme in an NoC system by using multiple low-sampling-rate ADC chips after time interweaving.
• The multiple clock domains of NoC need to processes clock synchronization. We improve the White Rabbit clock synchronization protocol and improve the synchronization precision of the time-interleaved channel sampling clock as well as guaranteeing the performance of high-speed data acquisition. The mean absolute error of the time-interleaved sampling clock is less than 20 ps.
• We improve the PTP protocol to determine the offset of the master clock and slave clock and realize coarse synchronization of the master clock and slave clock. DDMTD is designed to determine the phase of the master–slave clock. According to the phase value of PLL IP in the Xilinx FPGA dynamic phase shift, the phases of the master clock and slave clock are corrected to achieve their precise synchronization.

The organization of this paper is as follows: Section 2 presents NoC data acquisition and the principles of White Rabbit. Section 3 provides the proposed clock synchronization scheme. In Section 4, the implementation of the clock synchronization is presented. Section 5 presents verification of the clock synchronization of the NoC data acquisition. Section 6 presents concluding remarks.

2. The General Scenario

2.1. Data Acquisition Based on NoC

In NoC-based data acquisition, ADCs are mapped to the nodes on the network resources [17], while the router clock is used as the ADC sampling clock. Under the control of the routers, TI-ADCs perform parallel data acquisition with a sampling rate fs and resolution of m bits. Thus, the total sampling rate is increased to n × fs, while the resolution remains m bits to achieve the effect of a high sampling rate. Firstly, the data pass through the analog front end and into the ADC. Secondly, under the control of the ADC resource network interface, the data are encapsulated and transmitted to the on-chip network. Thirdly, under the router’s forwarding, the data are transmitted to the destination node’s high-speed transmission resource network interface. Finally, the data are output through the optical port.

In the proposed scheme, according to the global-asynchronous local-synchronous (GALS) characteristics of the clock distribution on the NoC, the asynchronous router clock is used as the ADC’s sampling clock in Figure 1. To ensure high-precision TI-ADCs, an improved White Rabbit protocol is adopted in this paper to implement high-precision synchronization of the ADC’s sampling clock.

2.2. The Principle of White Rabbit

The traditional White Rabbit clock synchronization protocol is a distributed timing technology that developed from the synchronous Ethernet [18] (Sync-E), precision time protocol (PTP/IEEE1588v2) [19] and digital dual-mixer time difference (DDMTD), which can achieve multi-node sub-nanosecond clock synchronization [20] over a long distance [21,22]. Sync-E ensures the same clock frequency throughout the system by passing the clock [23]. The clock is embedded in the data and sent to each node by Ethernet, and each child node recovers the clock from the data link and eliminates clock jitter generated in the recovery link through an internal phase-locked loop (PLL). The clock recovered from each child node serves as the system clock of the node and the reference clock of the next child node, and so on. The whole system works at the same clock frequency, so as to achieve the clock synchronization of the whole system.

The PTP protocol, based on the principle of delayed response mechanism, adopts the master–slave clock propagation mode and achieves clock synchronization of the whole system through the interaction of clock message information with timestamps. The synchronization accuracy can reach the level of nanoseconds [24]. Moreover, the PTP protocol provides three basic types of time nodes, named the ordinary clock (OC), boundary clock (BC), and transparent clock (TC). The OC and BC can be configured as master or slave clocks, while the TC does not keep synchronization with the other clock nodes. The TC forwards PTP protocol messages of master and slave nodes, and it also calculates and forwards their residence time. Master–slave clock synchronization is implemented through the interactive clock message between master and slave nodes. The synchronization principle is shown in Figure 2.

According to the synchronization principle in Figure 2, t1, t2, t3, and t4 are obtained from the slave clock node. Then, according to Equations (1) and (2), the clock offset and link delay between the master and slave nodes can be calculated.

$$offset=\frac{(t_2-t_1)-(t_4-t_3)}{2}$$

(1)

$$delay=\frac{(t_2-t_1)+(t_4-t_3)}{2}$$

(2)

Since the frequency of $f_{vco}$ is close to that of $cl{k}_a$ and $cl{k}_b$, the two D flip-flops output the low-frequency signals Q1 and Q2, respectively. The phase difference between Q1 and Q2 is the amplified phase difference of the original signal. Therefore, the phase difference between $cl{k}_a$ and $cl{k}_b$ can be calculated through pulse shaping and counting, as shown in Figure 3.

As shown in Figure 4, The frequency $f_{clk}$ of the auxiliary clock $f_{vco}$ and the measured signals $cl{k}_a$ and $cl{k}_b$ meets the relation of Equation (3), where M and N are both positive integers, and the frequency of $f_{vco}$ depends on the ratio between M and N [25,26]. The frequency $f_Q$ of output signal Q1 (or Q2) from the D flip-flop satisfies Equation (4). Assuming that the original phase difference between $cl{k}_a$ and $cl{k}_b$ is $\Delta {\mathsf{\Phi }}_{clk}$, and the phase difference between the output signals Q1 and Q2 is $\Delta {\mathsf{\Phi }}_Q$, they satisfy the relation in Equation (5), while the magnification factor A is given by Equation (6).

$$f_{vco}=f_{clk}\frac{N}{M}(M>N)$$

(3)

$$f_Q=f_{clk}-f_{vco}$$

(4)

$$\Delta {\mathsf{\Phi }}_Q=A\ast \Delta {\mathsf{\Phi }}_{clk}$$

(5)

$$A=\frac{N}{M-N}=\frac{1}{\frac{T_{vco}}{T_{clk}}-1}$$

(6)

The magnified phase difference can be measured by the system clock count or the time digital converter (TDC). However, the amplification factor A cannot be increased indefinitely, and it is limited by the performance of the D flip-flop. If the amplification factor A is too large, the D flip-flop will be in a metastable state. A larger A leads to a more serious metastable state. As a result, the low-frequency signals Q1 and Q2 output by the D flip-flop will be inaccurate. Therefore, the amplification factor A should be properly selected to ensure the desired phase difference resolution.

DDMTD amplifies the phase difference between the synchronous clocks, and thus, it can overcome the shortcoming of PTP. The clock synchronization precision can be better than one clock cycle, up to sub-nanosecond level.

3. The Proposed Design Clock Synchronization Scheme

3.1. Overall Design

The performance of data transmission relies heavily on the on-chip network architecture, which not only significantly affects network latency, throughput, area, and power consumption, but also plays a crucial role in the mapping from cores to network nodes. This paper applies on-chip networks to high-speed data acquisition, mapping analog-to-digital converter (ADC) chips and high-speed transmission interfaces to network on-chip resource nodes (IP cores). In the distribution of IP cores, each data acquisition routing node connects to one ADC resource node IP core. Each router has an independent asynchronous clock, using the router’s clock as the sampling clock for the ADC, thereby ensuring that the sampling clocks of ADCs are also independent of each other. Regarding data communication, ADC resource nodes only communicate with high-speed transmission interface resource nodes. That is, communication between IP cores is singular. An excessive number of routing nodes in the communication path between source and destination nodes will result in significant data transmission delays. Therefore, in the design, the transmission distance between ADC resource nodes and high-speed transmission interface resource nodes should be kept as short as possible to meet the low-latency requirements of data transmission. This paper adopts a time-interleaved sampling scheme with six ADCs, each with a sampling rate of 250 MHz and a resolution of 8 bits, resulting in an input bandwidth of 2 Gbps per ADC. The overall sampling rate of the system is 1.5 GSPS, with a total input bandwidth of 12 Gbps, as shown in Figure 5. Each high-speed interface has an output bandwidth of 6.6 Gbps, resulting in a total system output bandwidth of 19.8 Gbps. Taking into account some resource consumption for data encapsulation and packaging, each high-speed interface resource node can simultaneously receive and process data from two ADC resource nodes. Thus, it is possible to design the system in a grouped manner, with two ADC resource nodes and one high-speed interface resource node forming a group for expansion.

The ADC chips connected by routers in the left and right columns collect data in a time-interleaved manner. The collected data are then packaged and transmitted through the resource network interface to the connected router. Subsequently, the data are sent to the routers in the middle column, and finally, transmitted to the host computer via high-speed interfaces. This setup ensures that the distance between data collection and data transmission is only one hop, resulting in minimal transmission latency.

NoC has the characteristics of multi-task parallel communication and global asynchronous and local synchronous clock distribution [27]. The clock of each routing node is supplied by an external independent clock oscillator, which is output to the router after the PLL frequency doubling and the frequency difference is eliminated, and is used as the sampling clock of the ADC after the clock synchronization is completed.

The sampling clock of the TI-ADC channel needs high-precision synchronization to ensure the performance of data acquisition. The White Rabbit protocol can realize sub-nanosecond-level synchronization, so it is applied to the NoC in this paper, and it is also improved to implement clock synchronization between routers. A 2D mesh topology of nine routing nodes is taken as the example to design the corresponding router clock synchronization scheme, as shown in Figure 6.

Firstly, the clock synchronization system uses the improved PTP to determine the clock offset between the master and slave clocks, and implement the rough synchronization. Secondly, DDMTD is used to determine the phase relationship between master and slave clocks to achieve the fine synchronization. Finally, the phase of the slave clock is adjusted according to the TI-ADC sampling interval.

In addition, there is a link delay between the master clock node and slave clock node during rough synchronization. Clock messages exchanged between master clock node R5 and slave clock nodes R1, R3, R7, and R9 need to be forwarded through transparent clock node R2 or R8.

When solving the link delay and clock offset between the master and slave nodes, the message resident time of the forwarding node should be deducted, as shown in Figure 7. Therefore, the new link delay and clock offset between the master and slave nodes are shown in Equations (7) and (8) respectively.

$$delay=\frac{(T_2-t_2-T_1)+T_4-t_5-T_3}{2}$$

(7)

$$offset=\frac{(T_2-t_2-T_1)-T_4-t_5-T_3}{2}$$

(8)

After determining the clock offset and the link delay between the master and slave nodes, the slave clock is roughly synchronized by subtracting the offset from the time. Then, DDMTD technology is used to determine the phases of the master and slave clocks, and achieve the fine synchronization of the slave clock.

3.2. Improved White Rabbit Protocol

The traditional White Rabbit clock synchronization protocol is mainly used for clock synchronization of off-chip independent systems, and it transmits the reference clock and clock message information via Ethernet. Both the transmission frame format and the link delay is complex [28,29].

In this paper, the clock synchronization of multiple clock domains on the NoC is carried out, and the clock message information is transmitted inside the chip. Therefore, the White Rabbit protocol is improved and applied in the data collection based on the NoC.

Firstly, in the traditional White Rabbit protocol, Sync-E transmits the clock by embedding the clock into the data link. But in this paper, the clock is synchronized inside a chip, and the clock can be transmitted only through the internal physical circuit, so the Sync-E technology is omitted.

In this paper, since the clock message information is only transmitted within the chip, the PTP message can be simplified, as shown in Figure 8. The simplified PTP format is shown in

Table 1. Improved PTP protocol format.

	[33-31] bit	[30-8] bit	[7-4] bit	[3-0] bit
Sync	3’b001	23’b0	1111 (dest)	0101 (source)
Follow_up	3’b010	T1 (23 bit)	1111 (dest)	0101 (source)
Delay_req	3’b011	23’b0	0101 (dest)	Slave_addr (source)
Delay_resp	3’b100	T4 (23 bit)	Slave_addr (dest)	0101 (source)

.

The bit width of each message is 34 bits, in which the first 3 bits are the clock message type encoding, the last 8 bits are the address of the source node and destination node of the message, and the other 23 bits are the timestamp information. In addition, the destination node addresses of the sync and follow_up packets are set to 4’b1111, and messages are sent to each slave node simultaneously by broadcasting. In addition, various delay calculations in the Ethernet transmission link are reduced.

The DDMTD used in this paper is consistent with the traditional White Rabbit protocol and needs no improvements.

4. The Implementation of Clock Synchronization

4.1. The Implementation of PTP

(1)

The implementation of the master node

According to Figure 6, master clock node R5 includes master clock, master clock timestamp, and master clock PTP. The master clock PTP module is responsible for generating sync, follow_up, and delay_resp messages and sending them to each slave node, as well as receiving delay_req messages sent by each slave node. In this paper, a state machine is adopted to implement the receiving and sending of the PTP clock message, and its state transition diagram is shown in Figure 9.

After power-up reset, the state machine enters the IDLE state. In IDLE state, once the master clock is stable, a pulse signal sync_start is generated to enter the SYNC state. In SYNC state, the sync message is encapsulated according to the format of the sync message, and then, the request signal req_to_x (e/s/w/n) is sent to the east, south, west, and north directions. After the master node receives the feedback signal gran_from_x from each direction, it sends the sync message to the four ports, records the sending time as T1, and saves it, and then, generates the sync_end pulse signal to enter the FOLLOW state.

In the FOLLOW state, the follow_up message is encapsulated in the format of follow_up message, which carries the T1 timestamp. Then, the req_to_x (e/s/w/n) request signal is sent to the east, south, west, and north. After receiving the grant_from_x signal, the encapsulated follow_up message is sent to the four ports, and the follow_end pulse signal is raised to enter the DELAY_REQ state.

In the DELAY_REQ state, the request signal req_from_x (e/s/w/n) is received from four directions, and the response signal grant_to_x (e/s/w/n) is given back. Each delay_req message sent from the slave node is then received and the arrival time of delay_req message is recorded as T4_x (1/3/4/6/7/9) according to the address of the corresponding source node in the message. In addition, a node pulse signal T4_x_end is generated upon receipt of each delay_req message from the slave node, which is then encapsulated in the delay_resp message format as delay_resp_x (1/3/4/6/7/9). When the package is completed, it is sent to the slave node and the corresponding delay_resp_x_flag is raised until delay_req_end goes high. Then, the delay_resp_x_flag signal is changed to the low level, and the state machine returns to the IDLE state. So far, the master–slave node clock message interaction is completed.

(2)

The implementation of the transparent node

Transparent clock nodes are responsible for forwarding clock messages, such as node 2 and node 8 in Figure 4. They communicate with neighboring nodes in the NoC clock synchronization system and forward clock messages. For example, if clock messages are sent to each other from slave node 1 and master node 5, the clock messages need to be sent to node 2 after a successful handshake with transparent clock node 2, and then, node 2 will communicate with the corresponding node by handshake and forward clock messages to it. This communication mode makes the clock message stay on the forwarding node for only 1 clock cycle.

When the master clock node sends sync and follow_up messages to each slave clock node, since slave clock nodes 4 and 6 are directly connected to the master clock node, the message does not need to be forwarded, but slave clock nodes 1, 3, and 7, 9 need to be forwarded through transparent clock nodes 2 and 8, respectively. In the forwarding process, the transparent clock node forwards the clock message to the corresponding destination node by identifying the destination address in the message.

In the process of transmitting the message to the transparent clock node, when the address in the message is detected as 4’b1111 (sync and follow_up packets) request signals req_to_e and req_to_w are sent to the slave clock node in the east and west directions, respectively. Then, when the transparent clock node receives grant_from_e and grant_from_w signals from the slave clock node in the east and west, it sends the message to the corresponding node, as shown in Figure 10.

For example, when transparent clock node 8 selects the message output from the slave clock node (node 9) in the east, it receives the clock message after shaking hands with the east direction node. Then, transparent clock node 8 generates the request signal req_to_e_n and assigns the request signal req_to_n of the node, and then, sends it to master clock node 5. After successful handshake with master node 5, the message from the east slave node (node 9) will be output. When the message output of the slave node (node 7) in the west direction is selected, it follows the same steps, except that the generated req_to_w_n request signal is assigned to the request signal req_to_n of the node. When a destination address other than 1111 is identified, it will shake hands with the corresponding node and forward its clock message according to the specific address.

(3)

The implementation of the slave node

The slave clock PTP synchronization module is responsible for receiving sync, follow_up, and delay_resp messages sent by the master clock node, and sending delay_req messages to the master clock node. In the SYNC state of the master node, the slave clock node shakes hands with the transparent clock node or the master node, and then, receives the sync message and records the timestamp of T2 at the received time with the slave clock’s time axis. Next, in the FOLLOW state of the master node, the follow_up message is received again through a handshake, and the T1 timestamp in the message is read. Then, in the DELAY_REQ state of the master node, a delay_req message is sent to the master node by handshake, and the sending time T3 is recorded. Finally, the delay_resp message sent by the master node is received by handshake, and the T4 timestamp in the message is read. After obtaining the four timestamps of T1, T2, T3, and T4, Equations (1)–(8) are used to calculate the clock offset between the master and slave nodes, and then, the clock offset is subtracted from the slave clock to achieve the coarse synchronization.

4.2. The Implementation of DDMTD

(1)

Phase discrimination of master and slave clocks

After coarse synchronization of the master and slave clocks, the phase difference of the master and slave clocks is kept within one clock cycle. Some slave clocks are ahead of the master clock, while others lag behind the master clock. Therefore, it is necessary to determine the phases of the master and slave clocks after rough synchronization, so as to implement fine synchronization of the master and slave clocks.

In this paper, DDMTD technology is used to determine the phase difference of the master and slave clocks. After obtaining the phase difference, the Xilinx 7 series FPGA [30,31] is used for functional verification, and the clock PLL of the FPGA is used to shift the synchronized slave clock phase, so as to achieve high-precision synchronization of the sampling clock of the TI-ADC [32,33]. In this paper, the asynchronous router’s clock at 250 MHz is used for the sampling clock. The external PLL is a high-precision clock (248.88 MHz) close to the router’s clock frequency, and the difference between its clock cycle and the router’s clock cycle is the resolution of clock synchronization. The resolution of clock synchronization cannot be enhanced infinitely, since a higher resolution needs a larger amplification factor A for the phase difference, which may cause the metastable state of the D flip-flop [34]. According to the clock IP core MMCM of the Xilinx 7 series FPGA, the highest dynamic phase-shift precision is 1/56 of the internal VCO period. For example, when the clock crystal oscillator is 50 MHz, the internal VCO frequency is 1000 MHz and the highest phase-shift resolution is 18 ps. According to Equation (9), the frequency of the external PLL can be calculated as 248.88 MHz, and then, the external DAC and VCO are adopted to generate this frequency. In addition, when the resolution is known, the corresponding amplification factor A is 221.22 according to Equation (10), which can avoid the metastable state of the D flip-flop.

$$\Delta T=T_{pll}-T_{clk}$$

(9)

$$A=\frac{T_{clk}}{\Delta T}-1$$

(10)

A counter is used to count the falling edge of the output signals from the two D flip-flops, and the phase relationship between the master and slave clocks has three possible scenarios, shown in Figure 11.

They are also the cases of the signals from the two D flip-flops. If the D flip-flop with the master clock as its input turns to the high level first, it means that the phase of the master clock is in front while the phase of the slave clock is behind, so the phase_flag_ms signal will be pulled high, and its count value is the amplified phase difference value between the master and slave clocks. On the contrary, if the D flip-flop with the slave clock as input turns to the high level first, it indicates that the phase of the slave clock is in front and the phase of the master clock is behind, so the phase_flag_sm signal will be pulled high, and its count value is the amplified phase difference. If the two D flip-flops turn high at the same time, that means the master and slave clocks have the same phase, and the phase difference is 0.

After obtaining the amplified phase difference between the master and slave clocks, according to the total sampling rate of 1.5 GSPS in the data acquisition system, the sampling interval of TI-ADC is 0.667 ns, and the phases of the six sampling clocks (with respect to the original phase of the reference master clock) are 0.000 ns, 0.667 ns, 1.332 ns, 1.998 ns, 2.664 ns, and 3.333 ns, respectively. In this paper, samples are taken in the sequence of routing nodes 1, 3, 4, 6, 7, and 9. According to Equations (1)–(11), the amplified phase difference between each slave clock and the master clock can be calculated, while i denotes the sampling sequence (i = 1, 2, …, 6).

$$phas{e}_{sample}=\frac{(i-1)\ast A\ast 0.667}{T_{clk}}$$

(11)

(2)

Phase shift of slave clock

After obtaining the phase value adjust_value that the slave clock needs to be shifted by, the changes to the phase of the slave clock are made by using the CMT, a clock management unit in the above-mentioned series of FPGA. The clock management module includes the mixed-mode clock manager (MMCM) and the PLL [35]. MMCM can produce a clock with high precision and low jitter, and can also dynamically adjust the clock phase. It has the following two phase-shift modes:

(a)

Static phase-shift (SPS) mode

VCO can provide eight phase-shift clocks with a 45-degree phase interval. All these clocks are independent and programmable. The higher the frequency of VCO, the higher the accuracy of its phase-shift resolution, as shown in Equation (12).

In Equation (12), $f_{vco}$ represents the frequency of the VCO, $f_{in}$ represents the frequency of the input clock from the crystal oscillator, and period represents the period of this frequency. M and D are two multi-bit counters, as shown in Equation (13). The frequency of the VCO can be calculated by designing the ratio of M and D and knowing the frequency of the input clock $f_{in}$.

$$SPS=\frac{1}{8f_{vco}}\ast period=\frac{D}{8\ast m\ast f_{in}}\ast period$$

(12)

$$f_{vco}=f_{in}\ast \frac{M}{D}$$

(13)

(b)

IFPS mode

IFPS mode has linear phase shift, and its phase-shift resolution only depends on the VCO frequency. In this mode, the phase shift can be cycled by a linear increment in $\Delta {\mathsf{\Phi }}_{step}$. Its $\Delta {\mathsf{\Phi }}_{step}$ satisfies Equation (14):

$$\Delta {\mathsf{\Phi }}_{step}=\frac{1}{56f_{vco}}$$

(14)

Assuming the frequency of VCO is 1000 MHz, then the phase-shift resolution $\Delta {\mathsf{\Phi }}_{step}$ is 17.8 ps. The phase offset can be configured as a fixed value, or the phase can be dynamically incremented or decremented in the application after the configuration is complete. But in fixed phase-shift mode, dynamic phase-shift port cannot be used.

In addition, the dynamic phase shift of the module is controlled by psen, psincdec, psclk, and psdone. psen represents the enabling signal that controls the phase shift; psincdec represents a signal that controls an increasing or decreasing phase; psclk represents the driver clock used for phase-shift control signal; psdone represents the phase-shift completion marker signal. The clock output of the MMCM is dynamically increasing or decaying from the initial phase or the phase executed before. The psen, psincdec, psclk, and psdone signals are synchronized, and the signal timing is shown in Figure 12.

The workflow is as follows: if there is a psen pulse signal in a psclk cycle, the phase shift will be started. The phase moves forward (increasing). or backward (decreasing), when psincdec is at the low or high level, with each phase shift $\Delta {\mathsf{\Phi }}_{step}$ being 1/56 of the VCO frequency. When the phase shift is completed, psdone outputs a high pulse for one clock cycle, and then, produces the next phase shift enabling psen pulse. It takes 12 psclk cycles to complete a phase shift.

After receiving the adjust_value sent from the master–slave clock verifier, the IFPS mode of the MMCM is adopted to implement the phase shift of the slave clock. When the crystal oscillator clock of the router is 50 MHz, the frequency of the VCO is 1000 MHz, and the stepping length of the clock phase shift is 17.8 ps. The MSB of adjust_value is assigned to psincdec to indicate forward or backward movement of the phase, and the falling edge signals (neg_s or neg_m) of the mixed output signals are assigned to psen as phase-shift-enabling signals. Furthermore, the phase_flag_ms and phase_flag_sm signals determine which falling edge signals to assign to psen. The counter is incremented for each phase shift. When the value of the counter is equal to the adjust_value, the phase shift of the slave clock is completed, and the signal shift_comple_flag is pulled to the high level and sent to the master node.

5. Clock Synchronization Verification of NoC Data Acquisition

The data acquisition system operates as follows: Firstly, after the system is powered up and reset, the master–slave clocks are synchronized using Precision Time Protocol (PTP). The offset between the master and slave clocks is calculated, and the local clock of the slave is adjusted by subtracting this offset. Once coarse correction is completed, the offset_shift_finish signal is raised. Subsequently, this signal serves as the trigger for phase detection of the master–slave clocks within the slave node, using DDMTD. The phase difference is determined, and each slave clock’s required phase shift is calculated based on the interleaving sampling interval, then applied using Xilinx’s MMCM to achieve fine synchronization of the master–slave clocks. Once completed, the phase_shift_flag signal is raised. Finally, the synchronized clocks are used as the sampling clocks for the ADCs, allowing for interleaved sampling across multiple ADCs. The resource network interface receives the collected data from the ADCs, packages it into microchip format, and sends it to the data acquisition router. After handshaking communication between the data acquisition router and the data transmission router, the data are forwarded to the transmission router, and then, sent to the host computer via high-speed transmission interfaces.

This article utilizes Xilinx’s A7 series FPGA chip XC7A200T to implement the system’s logic functions, which features 215,360 logic cells, 13,140 Kb of RAM, 10 clock management tiles (CMTs), and 16 high-speed transceivers (GTP) with speeds of up to 6.6 Gbps. Vivado development tools are used for logic design, combined with ModelSim simulation tools to verify the correctness of the logic code. Logic code is employed to simulate the data generated by the ADC chip, hence the overall system simulation is as depicted in Figure 13, Figure 14 and Figure 15.

DDMTD is then used to determine the phase relationship between the master and slave clocks, and the phase of the slave clock is shifted by the dynamic phase shift function of the clock phase-locked loop. Once the phase shifts of all slave clocks are completed, the NoC-based TI-ADC will start, as shown in Figure 14.

In this paper, six sampling channels are interwoven in time, and the ideal interval between adjacent sampling channels at a 250 MHz sampling clock is 0.667 ns. The error in the adjacent sampling interval after clock synchronization is shown in Figure 15. According to the results, the average absolute error in sampling intervals is 7 ps.

$$T_{interval}=\frac{1}{N{f}_{sampling}}$$

(15)

$$T_{average}=\frac{{\sum }_{i=1}^n\left|T_i-T_{interval}\right|}{n}$$

(16)

Next, by changing the initial phases of the master and slave clocks, the tests are repeated 10 times. The average absolute errors of the adjacent sampling interval after clock synchronization are shown in Figure 16.

In [15], using a basic WR network structure of one master and one slave, the test synchronization accuracy is not more than 200 ps, and the synchronization accuracy reaches 21 ps. In this paper, the improved White Rabbit protocol is applied to an NoC’s clock synchronization for data acquisition, which can also achieve the same clock synchronization accuracy and solve the problem of clock synchronization of TI-ADC. The feasibility of time-interleaved data acquisition using six ADCs with a sampling rate of 250 MHz and a resolution of 8 bits in the NoC system was verified through simulation. Under the architecture of the NoC system designed in this paper, the total sampling rate of data acquisition with the same six sampling points is 1.5 Gbps, far exceeding the 150 Mbps in [7]. The higher the clock synchronization accuracy, the higher the achievable sampling rate.

6. Conclusions

In this paper, based on the proposed NoC data acquisition system, the White Rabbit clock synchronization protocol is applied to the clock synchronization of an NoC router, and it is improved to meet the high-precision synchronization requirements for TI-ADC’s sampling clock. The traditional White Rabbit protocol is based on Ethernet to transmit the clock and clock message. For the synchronization of multiple clock domains on the chip in this paper, the improved White Rabbit protocol no longer uses the Sync-E in the original White Rabbit protocol, and the clock message transmission format of PTP protocol is also simplified. The improved White Rabbit clock synchronization protocol is adopted to synchronize the router’s clock. Firstly, the improved PTP protocol is used to determine and correct the offset of the master and slave clocks. Then, DDMTD is used to determine the phases of the master and slave clocks. Finally, the phase-shift function of the PLL is used to correct the phase of the slave clock. A Modelsim simulation shows that the average absolute error in the TI-ADC’s sampling clock after clock synchronization can be kept within 20 ps, achieving a higher sampling rate.

The architecture scale validated in this paper comprises only nine routing nodes. Subsequently, the data acquisition system’s scale can be expanded and validated to improve system performance. Furthermore, the router’s clock can be used as the sampling clock for the ADCs. When outputting, consideration should be given to the path delay from the FPGA chip to the ADC chip’s clock pins to enhance clock synchronization accuracy and eliminate associated errors.