Performance Testing of the Triple Modular Redundancy Mitigation Circuit Test Environment Implementation in Field Programmable Gate Array Structures

Abstract

The logic structures implemented in Field Programmable Gate Arrays (FPGAs) are often critical and their correct operation is vital. FPGA devices are often used in areas where there is increased ionising radiation (space, medical diagnostics, aviation or nuclear power). There is therefore a need for mechanisms to correct radiation-induced errors. A common approach is the redundant implementation of particularly critical parts of the logic structure. By triplicating selected fragments, it is possible not only to detect potential errors but also to correct them. Such an approach is called triple modular redundancy (TMR), and its essence lies in the use of specialised voting circuits called voters, which allow the erroneous results of individual subcircuits to be eliminated by voting. The triplicate circuit under consideration, together with the voter, constitutes the mitigation structure. It becomes necessary to develop a test environment to assess the correct operation of these circuits. Also key is the efficiency of the implementation of these structures, which can be related to the occupation of logical resources or the power consumption of a given implementation. This paper demonstrates the essence of implementing a test environment to test the correctness of the mitigation of logic structures using TMR voters. An error injector mechanism using the Pseudo-Random Bit Sequence (PRBS) register is proposed, which introduces an element of randomness into the testing process. The aim of this research is to determine the implementation efficiency of the proposed test environment. In the experimental part, the implementation costs of the proposed solution were examined. The results indicate that between 66 and 109 LUT blocks were required to implement the error injector, corresponding to a relatively small increase in dynamic power consumption: by 22% for combinational circuits and by 37% for sequential circuits.

1. Introduction

Designing complex digital systems implemented in FPGA (Field Programmable Gate Array) devices is a challenging task. In most cases, there are a number of constraints to consider. As a rule, these constraints may relate to power consumption [1,2,3,4,5], the availability of logic resources [6,7,8] or speed of operation [9]. In many cases, it is also necessary to consider in the design process the specific characteristics of the environment in which the device will operate. It is often the case that an FPGA device is exposed to ionising radiation, which can lead to a number of errors in the operation of the implemented functionality. Application areas where increased levels of radiation can be expected include aerospace, medical diagnostics, nuclear power plants and most importantly, space applications.

FPGA devices are often used in space missions [10,11]. A deeper analysis of this environment therefore becomes necessary. As shown in [12], three main sources of radiation for space missions within the solar system can be distinguished, and these are galactic cosmic rays, solar radiation and radiation belts. Galactic cosmic rays are high-energy protons from interstellar space that reach the solar system. Solar radiation is a stream of plasma emitted in all directions by the sun. This emission can be quite violent during Coronal Mass Ejections (CMEs). The last source of radiation is radiation belts. These are areas of increased radiation resulting from accumulations of energetic particles in the magnetic field of planets.

From a technological point of view, an FPGA device is a semiconductor device. Radiation effects impact semiconductors in the following ways: Single-Event Effects (SEEs) and Dose effects. Dose effects correspond to a permanent change in selected parameters due to prolonged exposure to radiation. SEEs are associated with phenomena resulting from the impact of a single particle on a logic structure. Such an event is random and instantaneous. It can cause a change in the value of individual bits. The work [13] divides SEEs into Soft Error (no physical damage) and Hard Error (physical damage) phenomena.

Soft Errors are upsets to the device operation. The most important types of Soft Errors are single-event transients (SETs), single-event upsets (SEUs) and single-event function interrupts (SEFIs). A SET event corresponds to the generation of additional pulses in combinational logic that can be propagated throughout the logic structure. SEU-type events correspond to a change in the state of a memory element due to the impact of a high-energy particle. SEFIs, on the other hand, correspond to a change in the normal functioning of the device (a change in the circuit functionality).

Also in the case of Hard Errors, several types of events can be distinguished. These can be distinguished as a single-event latch-up (SEL), which is a circuit latch-up induced by radiation; single-event burnout (SEB), which is short-circuiting caused when a high-energy ion impacts a transistor source; and single-event gate rupture (SEGR), which involves physical damage to the gate oxide insulation in a silicon structure.

The main contribution of this paper is a novel test circuit for mitigation circuits using TMR (triple modular redundancy) voters, based on the PRBS (Pseudo-Random Bit Sequence) register. The solution proposed in this paper will allow for more efficient testing of the mitigation capability of TMR circuits with a slight increase in the number of logic resources and power consumption required. This is of particular importance in space applications where such limitations are particularly relevant. Section 2 discusses the mitigation methods used in FPGA devices. Particular attention is paid here to triple redundancy. The design of different voter models is also discussed. Section 3 presents the concepts of a test environment for mitigation circuits using the PRBS. The focus is on the efficiency problem of implementing these circuits in FPGA devices. Section 4 addresses the practical implementation problems of the test environment in projects. Section 5 presents the experimental results obtained. This paper is concluded with a summary.

2. Theoretical Background

There are many mitigation methods to reduce the impact of radiation on the performance of a digital device [11,12,14]. Of course, technological methods, such as the appropriate doping of silicon, can be discussed to improve the mitigation properties [15]. Naturally, it is also crucial to ensure the adequate shielding of digital devices. Unfortunately, in many cases, these methods are insufficient and SEEs can still occur. In this situation, it is crucial that the digital system is properly designed to resist these phenomena.

In many space missions, key digital-system elements are implemented in FPGA devices (telemetry systems, onboard computers and AI systems) [16,17,18,19,20]. This raises the question of mitigation methods targeting FPGA devices. An extremely important component of FPGA devices is the configuration memory. This is most often SRAM (Static Random-Access Memory), which stores information on how to configure and interconnect the available resources of the FPGA device in the logical structure. As a result of SEUs, the values of selected configuration bits in the configuration memory can be swapped, which will affect the resulting logical structure. This problem is illustrated in Figure 1.

The primary mitigation method in this case is scrubbing, which involves ‘refreshing’ configuration memory from an external source. Scrubbing techniques are the subject of many research papers [21,22,23,24].

Radiation can also affect the logical structure itself. In this case, there are also mitigation techniques. The primary technique is the use of systems using Error Checking and Correction (ECC) [25]. Correction coding most commonly uses the Hamming code [26] or the Reed–Solomon code [27].

An extremely important method of mitigation is redundancy. The use of redundancy in mitigation processes involves replicating selected functional blocks or circuits several times in order to perform exactly the same task. In extreme cases, where the space mission is specifically exposed to significant radiation, entire units are duplicated. If the duplicated modules, at the same time, function in exactly the same way (the state of their outputs is the same), this means that there have been no SEE-induced errors in the operation of the modules. If, however, the signals on their outputs are different, this means that a fault must have occurred in one of the duplicated modules. Naturally, duplicating parts of a circuit, modules or entire devices several times is expensive. This raises the question of how many times a circuit fragment should be duplicated to ensure effective mitigation. It turns out that replicating the circuit twice only allows statements to be made that an SEE phenomenon has occurred but without the possibility of error correction. For this reason, triple redundancy (TMR, triple modular redundancy) is commonly used. In this approach, three identical and independent modules perform the same functionality. If one of them malfunctions due to an SEE, by comparing the state of the outputs of all three modules, the expected (correct) state of the module’s outputs can be unambiguously determined by a voting process. Of course, a single fault is assumed. Thus, in order to realise TMR, it is necessary to have a voter comparing the outputs of the individual modules and realising the voting. The essence of TMR is illustrated in Figure 2.

TMR is one of the basic mitigation techniques. It is the subject of many scientific papers [28,29,30,31,32]. It should be noted that one of the main drawbacks of this approach is the significant growth of the logical structure obtained. The use of TMR significantly increases the number of logical resources used. There are methods to reduce them [33,34]. Naturally, it is also extremely important to adopt a strategy for using TMR in a project [35].

As already mentioned, a key element of TMR is the voters. Voters in the voting process select the majority answer from triplicated modules. Most often, these are fairly simple combinatorial arrangements. There are a number of voter designs. These are summarised in the work [36]. Some of the more important constructions include the classical voter [37], the Kshirsagar voter [38], the Ban and Naviner voter [39], the Balasubramanian voter [40] or the Mux-based voter [36]. The structures of the individual voters are shown in Figure 3.

The classic voter is a design based on an AND–OR structure. This type of solution fit well into older PLD (Programmable Logic Device) circuit architectures such as CPLDs (Complex Programmable Logic Devices), where the core of the circuits were PAL (Programmable Array Logic) cells [41,42]. Of course, there is nothing to prevent these voters from also being used in FPGAs. In the case of the Kshirsagar voter or Naviner voter, the designs are based on XOR gates, leading to more reliable fault tolerance. A certain modification of the classical voter is the Balasubramanian voter, where the AND–OR structure is implemented in the form of a logic cell. It is also possible to implement the voter using only multiplexers (the Mux-based voter).

The work [36] is limited to a power and area analysis of transistor structures corresponding to individual voter designs. In the case of FPGA devices, voters are implemented either as IP (Intellectual Property) cores or in the form of HDL (Hardware Description Language) descriptions. Of course, the HDL description itself can take different forms: an array description, a structural description or a description in the form of equations. Often, FPGA device chip manufacturers provide their voter solutions [43,44]. Independent companies offering synthesis tools also support high-reliability designs [45]. In many cases, the inputs as well as the outputs from the voter are complex interfaces, e.g., AXIS (Advanced eXtensible Interface; Stream protocol) [43]. Taking all this into account, it is crucial to efficiently map the voter along with redundant modules (mitigation circuits) into LUT (Look-up Table) blocks, which form the basis of the FPGA chip architecture. LUT blocks are simple memories in which logic functions with a limited number of inputs are implemented. Of course, at the appropriate stages of logic synthesis, complex designs are split (function decomposition) and mapped into these resources [8].

Let us therefore consider the concepts of a test environment implemented in a real FPGA chip to test the correctness of error correction by TMR mitigation circuits.

3. Test Environment for TMR Mitigation Structures

Assessing the effectiveness of the implementation of mitigation mechanisms using TMR voters is a difficult task. It is possible to imagine a whole range of factors on which effectiveness should depend, such as the occupation of logical resources necessary for their implementation and power consumption.

From a logic-resource-occupation point of view, more efficient solutions are those that use less logic resources (a smaller silicon area). In the classical approach, the aim is to minimise transistors or gates. In the case of FPGAs, the problem is more complicated because the implemented logic functions are mapped in LUT blocks. These blocks implement simple logic functions with a small number of variables (e.g., 6). Logic functions in a larger number of variables have to be decomposed and then mapped into several suitably interconnected logic blocks [8]. Obviously, looking at the TMR voter, it is apparent that these structures are simple enough that, in theory, a single LUT block should be sufficient to implement them. The resource utilisation of the redundant modules attached to the voter inputs remains an open question. It turns out that synthesis tools can in some cases produce results far from optimal, which can result in a significant growth of the logic structure. In addition, it should be noted that TMR voter inputs are not necessarily single-bit. Each input can be an n-bit bus, making the voter itself much more complex. It is also possible to imagine much more complex voter designs, e.g., using a golden model (a priority model) or designs in which individual inputs are assigned appropriate weights. Theoretically, a voter can have more voting inputs (of course, this is no longer TMR), and the process of selecting the correct data is then more complex, involving a significantly higher number of necessary logic resources. From the point of view of FPGAs, it is also important how the voter and the circuits attached to its inputs are described in the HDL language. It turns out that a combinational circuit can be described in many ways (by arrays, equations and structural descriptions), which often leads to solutions using a different number of logic resources. Taking all this into account, evaluating the effectiveness of mitigation structures in terms of the use of logical resources is an extremely important issue.

Of considerable importance from the point of view of mitigation structures using TMR is their power consumption. The power consumed by a digital circuit is the sum of static and dynamic power. Static power depends on the technology in which the circuit is made. Mitigation circuits are not separate ICs, so a comparison in terms of static power makes no sense. The situation is different for dynamic power, which depends on the switching frequency of the digital circuit. The dynamic power $P_{dyn}$ of an n-vertex logic network can be expressed by (1):

$$P_{dyn}={\displaystyle \frac{1}{2}}{V}_{dd}^{2}f\sum _{i=1}^n{C}_{i}S{W}_i$$

(1)

where $V_{dd}$ is the supply voltage of the digital circuit and f is the frequency of the clock signal. Additionally, this power depends on the product of the connection capacitance between the nodes of the logic network $C_i$ and the switching activity $S{W}_i$. In the classical case, the vertices of the logic network are the individual gates, while in the case of FPGAs, they are the LUT blocks. The key parameter from the perspective of power reduction is the switching activity ($SW$). The switching activity $S{W}_i$ is related to the probability of ‘switching’ the considered ith node to the opposite state, i.e., the transition from 0 to 1 or the transition from 1 to 0. The switching activity can be expressed by (2):

$$S{W}_i=2P\left(x\right)\left(1-P\left(x\right)\right)$$

(2)

where $P\left(x\right)$ is the probability of the variable x taking the value one. In the case of mapping in logic gates, knowing the probability at the inputs $P\left(x\right)$, one can determine the probability of the value one at their output $P\left(y\right)$; for example, NOT: $P\left(y\right)=1-P\left(x\right)$, two-input AND: $P\left(y\right)=P\left(a\right)P\left(b\right)$, two-input OR: $P\left(y\right)=1-\left(1-P\left(a\right)\right)\left(1-P\left(b\right)\right)$. Of course, such an analysis can be applied to more complex gate structures such as TMR voters. This means that for different voter designs, different $SW$ values can be obtained depending on the logic structure, and similarly for the circuits connected to the voter inputs. This means that the dynamic power consumed for equivalent mitigation structures may differ. A circuit will be more efficient if it has a lower switching activity. It is naturally open to determine the value of $P\left(x\right)$ for the individual inputs of the implemented circuits. As a rule, for the analysis, it is assumed that the probability of 0 appearing is the same as the probability of 1 appearing at the input $P\left(1\right)=P\left(0\right)=P\left(x\right)=0.5$. In the case of mapping the logic network in LUT blocks, such an analysis in terms of the SW would be very difficult to carry out. As a rule, FPGA manufacturers provide suitable simulators to estimate dynamic power. There are also academic tools to determine the power of ABC [46] or PowerdekBDD [2,3]. In such an analysis, care should be taken to ensure that the probabilities of 0 and 1 are the same for the inputs to a given mitigation structure. In this situation, a good solution may be to attach a clock signal of a certain frequency to the individual inputs, as shown in Figure 4.

Extremely important from the perspective of studying mitigation circuits using TMR is to assess their error-correction capability. Naturally, under simulation conditions, a TMR voter will perform as expected. This may be slightly different in a real hardware implementation, where the performance of the voter may depend on the dynamic properties of the implemented circuits. In the case of FPGAs, this verification will involve implementing the mitigation circuit under a test in a hardware-test environment implemented in an FPGA device.

It is often the case that synthesis tools, in the process of optimising a circuit in terms of the number of blocks, realise the sharing of logic resources between selected logic functions. The common parts (the subfunction) of several selected functions are only realised once in a single shared subcircuit. Then, the result of this subfunction is delivered to the selected functions. From the point of view of reducing the number of logic blocks and power, this is an advantageous phenomenon (a given section of the circuit can be realised only once, rather than several times for the selected functions separately). Unfortunately, from the point of view of TMR mitigation, it is a very unfavourable phenomenon. The individual circuits attached to the voter should not have parts in common. Let us imagine a situation in which individual modules A, B and C have a part of the circuit in common, implementing a specific subfunction. If an error occurs in this part of the circuit, the error will be propagated through all three modules in exactly the same way. This means that the voter will not detect any differences in its inputs and will not correct the error. Such a case is shown in Figure 5. Thus, it can be seen that optimising the implementation in terms of the number of resources can, in some cases, lead to a degradation of the mitigation properties of the circuit.

A key component of the hardware-test environment is the error injector. Under normal operating conditions, the same signal is present on all inputs of the voter. In order to test the voter’s ability to correct an error, it is necessary to replace one of the input signals with an erroneous one (an error injection into the voter). In the general case, the error injector consists of two blocks: an error-sequence generator and an error selector, as shown in Figure 6.

In the simplest case, the selector can be implemented based on OR, AND or XOR gates. In the case of the OR function, the values for which the injector has generated a high state are blocked in the high state; in the case of the AND function, those for which the injector has generated a low state will be blocked in the low state. In the case of an XOR function, those values for which the injector has generated a high state are negated.

In the case of the error-sequence generator, two concepts can be distinguished: a concept in which the error is triggered at fixed cyclic and specified intervals, and a concept in which the error is triggered at random. The concept with randomness seems to better reflect SEU phenomena, which is why the authors focus on it later in this article.

There are various forms and possibilities for building such an error-sequence generator. One solution may be to use a module that generates a Pseudo-Random Bit Sequence (PRBS). To generate the PRBS, a shift register with feedback implemented by an XOR-type logic gate is usually used. Each shift register is associated with some polynomial over a finite body. The maximum length is for shifting registers associated with primitive (non-decomposable) polynomials over the body. Such registers include the polynomial $X^{15}+X^{14}+1$, which is associated with the sequence denoted PRBS15. It uses a 15-bit ring register and involves $2^{15}-1$ combinations (32,767 combinations). Since the polynomial is prime, the number of states includes all but zero. The micro-architecture of the error injector using the XOR function and the PRBS15 module is shown in Figure 7. When PRBS[15] returns a high state, the data bits are negated and thus an error is injected into the data stream.

An injection circuit built in this way provides a very high error rate because the distribution of zeros and ones is practically comparable in the output stream of the PRBS block (all possible states from the 15-bit space without the 0..00 state).

It is very common that a single voter input (the input channel) corresponds to an n-bit vector (bus) rather than a single bit. In this situation, the error injector should provide randomness when injecting an error with respect to both the channel and the bus bit. This type of operation is provided by the circuit shown in Figure 8. An input demultiplexer redirects a single PRBS15 stream to one of the channels, while a second layer of demultiplexers ensures that this stream is redirected to a specific bit on the k-bit bus. XOR gates ensure that the specified bit is negated, thus infecting the channel at that point in time. Such an architecture ensures that an error is injected into a randomly selected bit of a randomly selected channel. In this case, only one PRBS15 module is implemented, from whose output the stream for error injection is taken. Naturally, the demultiplexers are also controlled from individual registers (PRBS[1]-PRBS[15]). The error injector presented in Figure 8 ensures that a single error is injected into a randomly selected bit of a randomly selected channel.

The error injector is a key component of the hardware-test environment and can be used in the process of testing the correctness of the mitigation of a design using TMR, implemented in FPGAs. This concept is discussed in Section 4.

4. The Essence of Using a Hardware-Test Environment Based on an Error Injector to Test the Correctness of Design Mitigation

From a mitigation point of view, it is crucial to identify the circuit locations where TMR voters should be placed. It is extremely important to secure the memory elements appropriately. In the most elementary case, the individual registers can be triplicated as in Figure 9a. However, it must be remembered that the voter is then built into each set of registers. Although this kind of protection seems best, the disadvantage is the overhead resulting from the use of multiple voters. Therefore, in practical applications, triple modular redundancy is considered for the entire module performing a specific function, as shown in Figure 9b. This approach leads to a significant reduction in the number of voters required.

However, the question remains open as to how to test such a circuit, or more precisely, where to inject the errors. Well, attaching an error injector to the output of a module performing a specific function will have a different purpose than attaching an error injector to the input of such a circuit. In the former, it will be possible to analyse SEU-type error corrections, and in the latter also SEFI-type errors. This is because a change in the input state in the module can lead to a malfunction of this circuit (for example, the sequential automaton will go to the wrong state). At the same time, the individual modules of a given block subject to TMR may have different numbers of input and output ports. The error injector circuit will then be attached to each input and output, as shown in Figure 10.

The hardware-test environment with fault injection blocks shown in Figure 10 demonstrates the need for a flexible approach to testing. The FNCT block contains three modules (MODULE A, B and C). Each has a slightly different configuration of input and output ports: MODULE A has one input and two output ports, MODULE B has two input and one output and MODULE C has one input and one output. In the general case, the ports are multibit and the use of an error injector as in Figure 8 is appropriate here, with the injector module shown here broken down into three FNCT blocks. For example, MODULE A IN ERR_INJ at the input of MODULE A in FNCT block #0, FNCT #1 and FNCT #2 are one and the same injector enabling the generation of errors at the inputs of the individual modules. Of course, testing the correctness of the mitigation in the TMR circuit shown in Figure 10 only makes sense for injecting errors into one channel, i.e., into one of the three inputs. Secondly, as errors are injected at the input of a given module, errors are not injected at their output, in line with the very idea of TMR, which says that two misrepresentations are not corrected. In contrast, the testing of MODULE A, MODULE B and MODULE C can be carried out in parallel if there is no data flow between these modules, i.e., they perform functions independent of each other.

The error injector proposed by the authors was used in a mitigation study of the Quasar project carried out by KP Labs for the ESA (European Space Agency) [47]. This shows that the proposed solutions are scalable. This project uses an AMD Kintex UltraSacle+ FPGA device (XCKU035-2SFVA784I). The use of a device with such significant resources is necessary due to the need to implement a complex design (a radio channel for telemetry modules, telecommand, encryption, etc.). In general, in space missions, the implemented designs are most often so large that it is necessary to use FPGA devices with significant logical resources. Naturally, a TMR-based mitigation process is therefore often used for large-scale projects. The experimental section presents the results of mitigation in this project.

5. Experimental Results

A key element in assessing the effectiveness of a test-environment implementation is to evaluate that implementation in terms of the occupation of logic resources and the dynamic power consumption of the corresponding logic structures. The method of determining both the number of logic resources required and the dynamic power consumption boils down to synthesising appropriately described circuits in HDL languages (Verilog HDL) in selected synthesis tools. In this paper, both commercial tools (Vivado from AMD) and the academic ABC [46] were used for synthesis.

In order to carry out the experiments, it is necessary to identify the circuits that will be mitigated. A popular set of benchmarks [48] was chosen, which contains both combinational and sequential circuits. The individual benchmarks contain a description of the function in Verilog, which was subjected to an initial minimisation targeting multioutput function.

Before proceeding to the actual experiments, it is extremely important to determine the occupancy of the logical resources and to determine the dynamic power of a single voter. Various voter designs were considered (described in Section 2). After synthesis in the Vivado tool targeting seven-series FPGAs, as expected, each voter can be realised on a single LUT block regardless of the solutions presented in Section 2. Similarly, the dynamic power in each case was 0.086 W. Taking this into account, a classical voter design was used in further experiments in the mitigation circuits (Figure 3a).

The first series of experiments was carried out in the ABC academic system. The synthesis was targeted at LUT blocks with four inputs. We decided to choose blocks with as few inputs as possible in order to highlight the differences between the different series of experiments as clearly as possible. The approaches without mitigation (Bench. column) and with mitigation (Bench.+vot. column) were compared. The results for the combinatorial systems are summarised in

Table 1. Results of the synthesis of combinatorial circuits in the ABC system.

			Bench.		Bench.+vot.
Name	IN	OUT	nd	SW	nd	SW
9sym	9	1	118	48.47	118	46.99
alu2	10	8	43	18.62	43	19.59
alu4	14	8	756	251.86	759	253
b12	15	9	26	12.18	26	11.65
clip	9	5	156	57.39	162	59.56
cm162a	14	5	13	6.79	13	6.57
cm163a	16	5	13	6.31	13	5.98
cm85a	11	3	14	6.85	14	6.55
con1	7	2	5	2.97	5	2.91
ex1010	10	10	748	219.01	751	227.31
f51m	8	8	55	25.38	55	24.85
rd53	5	3	21	10.81	21	10.49
rd73	7	3	67	27	76	28.72
rd84	8	4	272	93.02	268	89.72
sao2	10	4	80	29.59	82	29.35
sqrt8	8	4	26	11.24	22	10.39
t481	16	1	72	23.52	72	22.6
x2	10	7	18	7.57	18	7.65
xor5	5	1	2	1.28	2	1.2
Sum			2505	859.86	2520	865.08

and for the sequential systems in

Table 2. Results of the synthesis of sequential circuits in the ABC system.

				Bench.		Bench.+vot.
Name	IN	OUT	ST	nd	SW	nd	SW
bbtas	2	2	6	6	3.22	15	8.47
beecount	3	4	7	18	6.48	53	19.77
dk14	3	5	7	40	28.23	118	78.04
dk15	3	5	4	26	15.3	77	46.67
keyb	7	2	10	124	41.01	313	96.9
lion	2	1	4	3	1.13	10	3.79
lion9	2	1	9	5	1.77	16	6.29
mc	3	5	4	8	4.54	23	14.03
train4	2	1	11	3	1.46	8	4.41
train11	2	1	4	17	4.76	50	14.39
Sum				250	107.90	683	292.76

.

In both tables, the first columns describe each benchmark: the name of the benchmark (Name), the number of inputs (IN), the number of outputs (OUT) and, in the case of sequential circuits, the number of internal FSM states (ST). For each series, the number of vertices of the logic network (nd) is specified. This number directly corresponds to the number of LUT blocks with four inputs. In addition, the switching activity value ($SW$), which is directly related to the dynamic power, was specified. The last row is the sum of the individual values, as shown as graphs in Figure 11.

Looking at the graphs in Figure 11a,b, it can be seen that the results obtained are practically the same regardless of whether triple redundancy was implemented or not. The expected result is a situation where the number of blocks and the dynamic power will be about three times higher in the approach where there is TMR (Bench.+Vot.) than for the approach without TMR (Bench.). Such a relationship can be observed for sequential systems. For the number of blocks (Figure 11c), the growth of the logic structure due to the implementation of TMR is 2.73 times. In the case of power, on the other hand, it is 2.71 times. Looking at combinational circuits, it can be seen that the expected redundancy was removed in the optimisation process during the synthesis process. In this situation, the question of methods for describing mitigating systems remains open. In the case of sequential circuits, the relationship between the number of blocks in the cases is less than three, indicating that the synthesis process was carried out in such a way that there must be resource sharing between the individual modules attached to the voter inputs. Given the nature of TMR, it is an open question whether this is a beneficial phenomenon. The $SW$ relationship in Figure 11d directly reflects the relationship of the number of resources used, so it is difficult to speak of power optimisation.

Having analysed the effects of the implementation of the mitigation circuits, it is possible to move on to an analysis of the implementation of the test environment, of which the error injector (EI) is a major component. A comparison was made between the benchmarks given the mitigation (Bench.+Vot.) and the benchmarks to which the test environment containing the error injector (EI) was additionally added. Due to the complex description of the EI, the synthesis for both series was realised in the Vivado system for the logic blocks contained in the seven-series FPGAs. The results obtained are shown in

Table 3. Results of the synthesis of combinatorial circuits in the Vivado system.

			Bench.		Bench.+vot.		Bench.+vot.+El
Name	IN	OUT	LUT	POWER	LUT	POWER	LUT	POWER
9sym	9	1	6	0.441	6	0.442	73	0.851
alu2	10	8	14	2.703	14	2.702	103	3.26
alu4	14	8	286	5.978	286	5.978	373	6.493
b12	15	9	14	2.29	13	2.305	110	2.817
clip	9	5	20	2.924	27	3.083	110	3.547
cm162a	14	5	7	1.435	7	1.433	91	1.944
cm163a	16	5	7	1.351	7	1.349	93	1.85
cm85a	11	3	6	0.793	6	0.793	81	1.29
con1	7	2	2	0.799	2	0.799	74	1.236
ex1010	10	10	170	8.11	170	8.121	279	8.95
f51m	8	8	10	3.925	10	3.923	99	4.487
inc	7	9	10	2.963	10	2.963	105	3.533
rd53	5	3	2	1.491	2	1.491	77	1.946
rd73	7	3	6	1.953	6	1.953	81	2.423
rd84	8	4	11	2.11	11	2.115	94	2.578
sao2	10	4	30	1.182	34	1.251	110	1.667
sqrt8	8	4	6	1.536	6	1.535	87	2.025
t481	16	1	5	0.721	5	0.721	72	1.135
x2	10	7	8	1.671	8	1.639	94	2.146
xor5	5	1	1	0.776	1	0.776	68	1.191
Sum			621	45.152	631	45.372	2274	55.369

for the combinational circuits and

Table 4. Results of the synthesis of sequential circuits in the Vivado system.

				Bench.		Bench.+vot.		Bench.+vot.+El
Name	IN	OUT	ST	LUT	POWER	LUT	POWER	LUT	POWER
bbtas	2	2	6	3	0.473	3	0.467	74	0.878
beecount	3	4	7	7	1.074	7	1.072	88	1.546
dk14	3	5	7	7	2.885	7	2.89	91	3.383
dk15	3	5	4	4	2.475	4	2.475	88	2.985
keyb	7	2	10	48	0.774	48	0.774	121	1.207
lion	2	1	4	2	0.245	2	0.245	69	0.636
lion9	2	1	9	2	0.265	2	0.265	69	0.636
mc	3	5	4	4	2.481	4	2.481	86	2.775
train11	2	1	11	4	0.265	4	0.263	71	0.662
train4	2	1	4	2	0.183	2	0.183	68	0.567
Sum				83	11.12	83	11.115	825	15.275

for the sequential circuits.

In both tables, the number of LUT blocks (LUT) was determined for each series. In addition, the dynamic power (Power) value is specified. The last row is the sum of the individual values, as shown as graphs in Figure 12.

Analysing the graphs in terms of the number of blocks used, it can be seen that for the considered set of benchmarks, the implementation of a test environment containing an error injector significantly increases the number of LUT blocks used (3.6 times for combinational circuits and almost 10 times for sequential circuits). However, it should be noted here that the benchmarks considered describe very small test circuits, so the relationships described are not entirely reliable for describing the implementation efficiency. It is worth noting that between 66 and 109 LUT blocks are required to implement the error injector, so the more complex the circuit, the lower the potential cost of EI implementation.

It is very interesting to compare the dynamic power in both cases (Figure 12b,d). Despite the significant growth of the logic structure in the case with the EI, the dynamic power increases only slightly (by 22% for combinational circuits and by 37% for sequential circuits). This indicates a good implementation efficiency of the test environment, even for small circuits.

Comparing the power consumption results between Bench. and Bench.+vot., it can be seen that the results obtained are practically the same. Minor differences are due to the nature of the synthesis tool in the placement and routing stages.

As written in the previous chapter, the proposed solutions were used in a project of considerable complexity (the Quasar project [47]). The synthesis targeting the XCKU035-2SFVA784I chip was realised in the Vivado tool (due to the support of the FPGA chip manufacturer, AMD). In this vendor’s devices, LUT blocks are grouped into more advanced CLB (Configurable Logic Block) cells, which can operate in well-defined configurations. In the first instance, synthesis was carried out without any mitigation. This led to the use of 113,347 LUT blocks, representing 55.80% of all blocks (corresponding to 26,027 complex CLB blocks (85.90%)). The described TMR mitigation was then carried out in the report of the encryption modules. This led to the use of 137761 LUT blocks, representing 67.82% of the total blocks (corresponding to 30,300 complex CLB blocks (99.49%)). It should be noted here that the system had to perform resynthesis several times for the synthesis to be successful. The bitstream was then generated and the FPGA device was programmed and tested using prototype DPU. This shows that the solutions proposed in this paper can be applied to practical applications of considerable complexity, which is crucial when designing devices aimed at space missions.

6. Conclusions

The aim of this paper was to demonstrate the efficiency of the implementation (measured by the number of logic resources used and dynamic power consumption) of the TMR mitigation circuit test environment. A novel element was the introduction of an ‘element of randomness’ into this environment through the use of the PRBS register in the error injectors. The testing performed revealed a number of observations.

The first observation is the issue of the synthesis of mitigation circuits. It is clear that the synthesis algorithms are focused on circuit optimisation, which, in extreme cases, can limit the robustness of the circuit to radiation errors (e.g., sharing logic resources). The second observation concerns the number of resources used by the error injector. It can be seen that in this case, it is inefficient to apply mitigation to small circuits. The last observation concerns dynamic power and shows that the implementation of the test environment does not significantly affect its consumption. All these observations lead to the conclusion that it is worthwhile to implement this environment provided that we have sufficient logic resources in the FPGA.

The efficient synthesis of mitigation circuits has proven to be of great importance. It is becoming necessary to develop algorithms that, on the one hand, provide suitably efficient logic structures and, on the other hand, do not degrade the mitigation properties of circuits. The authors plan to address the problem of mitigation-oriented synthesis in the future.