Global Sensitivity Analysis of Structural Reliability Using Cliff Delta

Abstract

This paper introduces innovative sensitivity indices based on Cliff’s Delta for the global sensitivity analysis of structural reliability. These indices build on the Sobol’ method, using binary outcomes (success or failure), but avoid the need to calculate failure probability P_f and the associated distributional assumptions of resistance R and load F. Cliff’s Delta, originally used for ordinal data, evaluates the dominance of resistance over load without specific assumptions. The mathematical formulations for computing Cliff’s Delta between R and F quantify structural reliability by assessing the random realizations of R > F using a double-nested-loop approach. The derived sensitivity indices, based on the squared value of Cliff’s Delta ${\delta }_C^2$, exhibit properties analogous to those in the Sobol’ sensitivity analysis, including first-order, second-order, and higher-order indices. This provides a framework for evaluating the contributions of input variables on structural reliability. The results demonstrate that the Cliff’s Delta method provides a more accurate estimate of P_f. In one case study, the Cliff’s Delta approach reduces the standard deviation of P_f estimates across various Monte Carlo run counts. This method is particularly significant for FEM applications, where repeated simulations of R or F are computationally intensive. The double-nested-loop algorithm of Cliff’s Delta maximizes the extraction of information about structural reliability from these simulations. However, the high computational demand of Cliff’s Delta is a disadvantage. Future research should optimize computational demands, especially for small values of P_f.

1. Introduction

Global sensitivity analysis (GSA) focuses on attributing the uncertainties of model outputs, or related performance indicators, to their inputs, thereby assessing the impact of input uncertainties on outputs or performance indicators [1,2]. Various GSA methods have been developed for this purpose [3,4]. These methods include the screening method [5,6], variance-based methods [7,8], moment-independent methods [9,10], and derivative-based methods [11,12]. Among these methods, variance-based sensitivity indices, also known as Sobol’ indices [7,8], are particularly notable for their mathematical elegance in measuring the individual, interaction, and total contributions of each input to the model output uncertainty, see, e.g., [13,14].

In the limit state method, probabilistic reliability analysis is based on the estimation of the failure probability [15]. Sobol’ indices have been widely used in structural reliability analysis to pinpoint variables that significantly influence failure probability [16,17]. These sensitivity indices, which focus on failure probability P_f, are derived from the variance decomposition of a binary function representing failure and success [16,17]. Fort et al. [18] expanded on Sobol’ sensitivity indices by introducing a contrast function in place of variance, allowing the indices to be oriented towards variance, probability, and quantile. This approach maintains the non-negative property of the indices and ensures that their sum is equal to one, as in the traditional Sobol’ sensitivity analysis. Extending GSA to P_f and design quantile represented a significant advancement in civil engineering, as these quantities are crucial in structural reliability assessments [19,20].

The development of GSA methods focused on reliability requires the precise estimation of P_f [21]. However, the complexity of mathematical models and the difficulty of uncertainty propagation using sampling-based methods, such as Monte Carlo (MC) simulations, present significant challenges, due to the large number of runs required [22]. Nonlinear finite element models are particularly demanding on CPU time [23], which further complicates the process of structural reliability estimation, see, e.g., [24,25]. To ensure the most accurate estimate of P_f, it is essential to develop methods that provide precise P_f estimates, while minimizing the computational costs associated with repeated calls to the computational model [26,27].

The computational burden can be reduced by using metamodels, also known as surrogate models, which approximate the behavior of complex models with simpler ones [28,29]. Methods, such as the polynomial response surface [30,31], the response surface-based multi-fidelity model [32], polynomial chaos expansion (PCE) [33,34], the Gaussian process [35,36], Kriging [37,38], and neural networks [39,40,41] are used to create such metamodels.

However, while the use of metamodels significantly reduces the computational burden, there are some critical drawbacks to this approach [42,43,44]. Metamodel-based approaches lack reliable error measures and may not accurately represent the limit-state functions, leading to potential inaccuracies in reliability analysis [45,46]. Models with discontinuities or sharp changes in behavior (such as buckling) present particular challenges for metamodels, which may struggle to fit such complexities accurately [47].

Consequently, despite the advancements and widespread use of metamodels, the traditional (quasi-) Monte Carlo methods have not been significantly displaced and remain an integral part of reliability and sensitivity analysis methods. Their enduring use is attributed to their flexibility, unbiased estimations, and effective integration with variance reduction techniques [3,4].

The existing research frequently explores the rate of advancements across various Monte Carlo-based reliability applications [48,49], but there is a notable shortage of studies focused on the implications of these advancements for enhancing the efficiency of reliability-oriented global sensitivity analyses (GSAs). Ensuring the accurate estimation of failure probability P_f with the available number of simulations is critical because it directly influences reliability assessments and decision-making processes, see, e.g., [50,51,52,53]. A more accurate P_f estimation is advantageous when using both the original model and the metamodel, depending on the available computational resources.

The solution proposed in this article involves adopting an alternative measure of structural reliability based on Cliff’s Delta [54], which can be calculated using double-nested-loop simulations. This approach enhances the precision of P_f estimation, albeit with higher computational demands for calculating Cliff’s Delta, but it more effectively utilizes existing model outputs compared to P_f estimation using the Monte Carlo method. Given the substantial cost of obtaining data using FEM models, this approach is justified. By maximizing the utility of existing simulations, this method ensures more accurate reliability analyses.

2. Cliff’s Delta

Cliff’s delta, denoted as δ_C, was initially devised by Norman Cliff, primarily for the handling of ordinal data [54]. It serves as a metric to assess the frequency with which values from one distribution exceed those in another distribution. A key feature of δ_C is that it does not necessitate any specific assumptions regarding the distributions’ form or variability.

The formula for computing the sample estimate of δ_C is expressed as follows:

$${\delta }_C=\frac{{\displaystyle {\sum }_{i=1}^m{\displaystyle {\sum }_{j=1}^n\left[y_i>y_j\right]-\left[y_i<y_j\right]}}}{m·n}, \mathrm{where} {\delta }_C\in [-1, 1].$$

(1)

In this equation, the two distributions are characterized by sizes n and m, with respective elements y_i and y_j. Here, the notation [⋅] refers to the Iverson bracket notation, resulting in 1 if the condition within the brackets holds, and 0 otherwise. Formally, the Iverson bracket notation can be defined as follows:

$$\left[y_i>y_j\right]=\left\{\begin{array}{l}1if{y}_i>y_j;\\ 0otherwise\end{array}\right.,\left[y_i<y_j\right]=\left\{\begin{array}{l}1if{y}_i<y_j;\\ 0otherwise\end{array}\right..$$

(2)

This statistical approach allows for an intuitive comparison of two distributions by quantifying the dominance of one distribution over the other.

Building upon the foundational description of δ_C, this measure is specifically applied to assess the relationship between resistance R and load F within the framework of structural reliability. In the limit state, a structure is reliable if R ≥ F; otherwise, failure occurs [19]. Let the difference between R and F be denoted as follows:

Z = R − F,

(3)

where R and F are statistically independent random variables. In Monte Carlo simulations, these variables are represented as arrays R and F, corresponding to resistance and load, respectively. Cliff’s delta, δ_C, is utilized to quantitatively evaluate the extent to which values of the resistance exceed or fall below those of the load, thereby providing fundamental insights into system reliability.

Assuming arrays R and F are equal in size, with n entries each, the formula for calculating δ_C is simplified to:

$${\delta }_C=\frac{{\displaystyle {\sum }_{i=1}^n{\displaystyle {\sum }_{j=1}^n\left[R_i>F_j\right]-\left[R_i<F_j\right]}}}{n^2}, \mathrm{where} {\delta }_C\in [-1, 1].$$

(4)

In this expression, R_i and F_j represent the i-th and j-th entries in the R and F arrays, respectively, denoting random realizations of resistance and load. The Iverson brackets, [⋅], return 1 when the enclosed condition is true, and 0 otherwise. This formulation facilitates a direct comparison between resistance and load across all sampled scenarios.

The estimation of δ_C between the measurements of resistance and force provides a metric quantifying the frequency with which the values of resistance surpass those of load. Employed as a statistical tool, this measure assesses how frequently resistance can withstand or exceed the applied loads.

3. Sensitivity Measures of Cliff’s Delta

Although Cliff’s Delta has been applied in numerous studies, see, e.g., [55,56,57,58,59], its utilization in the global sensitivity analysis of reliability is absent. This chapter demonstrates that the sensitivity measures based on the squared value of Cliff’s Delta, ${\delta }_C^2$, exhibit properties similar to the variance in the Sobol’ sensitivity analysis [7,8], oriented to reliability [16,17].

3.1. Approximation of Failure Probability with Cliff’s Delta in Sensitivity Analysis

In the limit state, a structure is considered reliable if R ≥ F; otherwise, failure occurs [15]. The probability of failure P_f can be defined as the overload probability that F > R (i.e., Z < 0). The failure probability can be expressed as the mean value of the binary reliability function of the Bernoulli distribution, where 1 occurs if Z < 0, and 0 otherwise [21]:

$$P_f=P\left(Z<0\right)=E\left(1_{Z<0}\right),$$

(5)

where

$$1_{Z<0}=\frac{\left|Z\right|-Z}{2\left|Z\right|}.$$

(6)

The conventional measure of reliability is 1 − P_f. The variance of the Bernoulli distribution of the random variable 1_Z < 0 can be written as follows:

$$V\left(1_{Z<0}\right)=P_f\left(1-P_f\right).$$

(7)

The second moment is a function of P_f, which is useful for the formulation of the Sobol’ sensitivity indices, quantifiable through the estimation of the conditional realizations of P_f [21]. If the variance V(1_Z < 0) is used in the decomposition within the Sobol’ sensitivity analysis, the first-order sensitivity index of the variance function 1_Z < 0 can be expressed as follows:

$$S_i=\frac{V\left(E\left(1_{Z<0}\left|X_i\right.\right)\right)}{V\left(1_{Z<0}\right)}=\frac{V\left(1_{Z<0}\right)-E\left(V\left(1_{Z<0}\left|X_i\right.\right)\right)}{V\left(1_{Z<0}\right)}=\frac{P_f\left(1-P_f\right)-E\left(P_f\left|X_i\right.\left(1-P_f\left|X_i\right.\right)\right)}{P_f\left(1-P_f\right)}.$$

(8)

The concept of sensitivity analysis based on Cliff’s Delta is predicated on the assumption that Cliff’s Delta can be expressed as follows:

$${\delta }_C=P\left(Z>0\right)-P\left(Z<0\right)=\left(1-P_f\right)-P_f=1-2P_f.$$

(9)

The failure probability P_f can be calculated using Cliff’s Delta, as follows:

$$P_f=\frac{1-{\delta }_C}{2}.$$

(10)

Similarly, the second moment can be approximated and written as a function of Cliff’s Delta, as follows:

$$V\left(1_{Z<0}\right)=P_f\left(1-P_f\right)=\left(\frac{1-{\delta }_C}{2}\right)\left(1-\frac{1-{\delta }_C}{2}\right)=\frac{1-{\delta }_C^2}{4}.$$

(11)

Substituting this into Equation (8), the Sobol’ sensitivity index can be expressed using δ_C, as follows:

$$S_i=\frac{\frac{1-{\delta }_C^2}{4}-E\left(\frac{1-{\delta }_C^2\left|X_i\right.}{4}\right)}{\frac{1-{\delta }_C^2}{4}}=\frac{E\left({\delta }_C^2\left|X_i\right.\right)-{\delta }_C^2}{1-{\delta }_C^2}.$$

(12)

This formulation provides an alternative method for calculating the sensitivity index, which carries all the advantages and disadvantages associated with the estimation of Cliff’s Delta compared to failure probability.

3.2. Sensitivity Indices Based on Cliff Delta

In the development of sensitivity indices for the evaluation of structural reliability, the use of the squared measure of Cliff’s delta, ${\delta }_C^2$, has been proposed. This chapter delineates the formal definitions of these indices, categorized from first to higher orders. The sensitivity indices are computed as ratios of differences normalized by the constant, 1 − C₀, where C₀ is defined as the square of Cliff’s delta, ${\delta }_C^2$.

$$C_0={\delta }_C^2,$$

(13)

where ${\delta }_C^2$ represents the measure calculated when all input random variables; X₁, X₂, …, X_M, of R and F are random and statistically independent.

The first-order sensitivity index, S_i, is defined to quantify the effect of a single variable X_i on the change observed in the squared Cliff’s delta, ${\delta }_C^2$. It is calculated as follows:

$$S_i=\frac{E\left({\delta }_C^2\left|X_i\right.\right)-C_0}{1-C_0}.$$

(14)

In Equation (14), having frozen one potential source of variation (X_i), the resulting $E\left({\delta }_C^2\left|X_i\right.\right)$ will be higher than the corresponding total or unconditional Cliff’s Delta, where C₀ = ${\delta }_C^2$. For example, if X_i were the sole source of change in the distance between R and F, fixing it to $x_i^{*}$ would result in $\left({\delta }_C^2\left|X_i\right.=x_i^{*}\right)$ = 1.

The second-order sensitivity index, S_ij, extends the analysis to pairs of variables, evaluating the joint effect of X_i and X_j on the change of ${\delta }_C^2$. This index is expressed as follows:

$$S_{ij}=\frac{E\left({\delta }_C^2\left|X_i\right.,X_j\right)-C_0}{1-C_0}-S_i-S_j.$$

(15)

Similarly, the third-order index, S_ijk, considers the combined influence of three variables X_i, X_j, and X_k. It is calculated as follows:

$$S_{ijk}=\frac{E\left({\delta }_C^2\left|X_i\right.,X_j,X_k\right)-C_0}{1-C_0}-S_i-S_j-S_k-S_{ij}-S_{ik}-S_{jk}.$$

(16)

Other Cliff’s sensitivity indices, which quantify higher-order interaction effects, are defined analogously. The sensitivity index of the last order can be expressed as follows:

$$S_{1,2,\dots ,M}=\frac{E\left({\delta }_C^2\left|X_1\right.,X_2,\dots ,X_M\right)-C_0}{1-C_0}-{\displaystyle {\sum }_{p=1}^{M-1}{\displaystyle {\sum }_{1\le i_1<\dots <i_p\le M}{S}_{i_1,i_2,\dots ,i_p}}}=1-{\displaystyle {\sum }_{p=1}^{M-1}{\displaystyle {\sum }_{1\le i_1<\dots <i_p\le M}{S}_{i_1,i_2,\dots ,i_p}}},$$

(17)

where E(${\delta }_C^2$|X₁, X₂, …, X_M) = 1 is ensured due to the nature of Cliff’s Delta, which assumes a value of either 1 or −1 when all input random variables are fixed. Consequently, each element in the array R adopts a consistent identical value denoted as v₁, and similarly, each element in the array F maintains another consistent identical value, denoted as v₂. It should be noted that the constants v₁ and v₂ are generally different.

The sum of all indices is equal to one. This characteristic is guaranteed by the computation of the last-order sensitivity index, as shown in Equation (17), which is derived from the difference between 1 and the sum of all lower-order sensitivity indices.

$${\displaystyle \sum _i{S}_i}+{\displaystyle \sum _i{\displaystyle \sum _{j>i}{S}_{ij}}}+{\displaystyle \sum _i{\displaystyle \sum _{j>i}{\displaystyle \sum _{k>j}{S}_{ijk}}}}+\dots +S_{123\dots M}=1.$$

(18)

The non-negativity of sensitivity indices is proven by their association with variance—see Equation (11)—and the Sobol’ decomposition of variance [7,8]. The fixing of multiple input variables typically leads to a higher value of ${\delta }_C^2$ (with a limit of one in the last-order sensitivity index) compared to the constant C₀. The properties of sensitivity indices based on Cliff’s delta and their comparison with the classical Sobol’ sensitivity analysis will be further explored in later chapters.

3.3. Sensitivity Indices Based on Failure Probability

The impact of input variables on the failure probability P_f can be analyzed using sensitivity analysis based on contrast functions [18]. Unlike the Sobol’ sensitivity analysis, contrast-oriented sensitivity analysis employs a contrast function, whose minimizer is of primary interest [60,61]. Fort [18] demonstrated that employing the quadratic contrast function, which calculates the mean of the squared deviations from the average, results in the well-known Sobol’ sensitivity indices [7,8]. In reliability-oriented sensitivity analysis, the variance from the Sobol’ sensitivity analysis is considered to be the variance of the binary reliability function V(1_Z<0) = P_f (1 − P_f).

The first-order sensitivity index, C_i, quantifies the main effect of a single variable X_i on the variance of the binary reliability function, as follows:

$$C_i=\frac{V\left(E\left(1_{Z<0}\left|X_i\right.\right)\right)}{V\left(1_{Z<0}\right)}=\frac{P_f\left(1-P_f\right)-E\left(P_f\left|X_i\right.\left(1-P_f\left|X_i\right.\right)\right)}{P_f\left(1-P_f\right)}.$$

(19)

In this equation, freezing the source of variation, X_i, affects P_f. The sensitivity index C_i indicates the extent to which one could reduce, on average, the output variance of the binary function 1_Z<0, if X_i could be fixed. Hence, it is a measure of the main effect.

The second-order sensitivity index, C_ij, measures the pair effects of X_i and X_j on the variance of the binary function 1_Z<0. This index can be written as follows:

$$C_{ij}=\frac{V\left(E\left(1_{Z<0}\left|X_i\right.,X_j\right)\right)}{V\left(1_{Z<0}\right)}-S_i-S_j.$$

(20)

Similarly, the third-order index, C_ijk, considers the combined influence of three variables X_i, X_j, and X_k, calculated as follows:

$$C_{ij}=\frac{V\left(E\left(1_{Z<0}\left|X_i\right.,X_j,X_k\right)\right)}{V\left(1_{Z<0}\right)}-S_i-S_j-S_k-S_{ij}-S_{ik}-S_{jk}.$$

(21)

Higher-order contrast sensitivity indices, which quantify higher-order interaction effects, are defined analogously. The sum of all sensitivity indices is equal to one.

$${\displaystyle \sum _i{C}_i}+{\displaystyle \sum _i{\displaystyle \sum _{j>i}{C}_{ij}}}+{\displaystyle \sum _i{\displaystyle \sum _{j>i}{\displaystyle \sum _{k>j}{C}_{ijk}}}}+\dots +C_{123\dots M}=1.$$

(22)

The non-negativity of sensitivity indices is guaranteed by their derivation from the Sobol’ sensitivity analysis [6,7,8], see also [19,21].

4. The Case Study

In the case study, the new sensitivity indices are compared with the results of the Sobol’ sensitivity analysis using a simple example. In Equation (3), the resistance can be considered as follows:

R = X₁·X₂ + X₂·X₃ + K,

(23)

where K is constant. The load is considered as follows:

F = X₄·X₅.

(24)

All input random variables, X₁, X₂, …, X₅, follow a Gaussian probability density function with a mean value of zero and a standard deviation of one. The input random variables are statistically independent.

Table 1. Average values of ${\delta }_C^2$ for sequentially fixed input variables.

Conditional Mean of ${\mathit{\delta }}_{\mathit{C}}^2$	K = 0	K = 1	K = 2	K = 3	K = 4
${\mathit{C}}_0={\mathit{\delta }}_{\mathit{C}}^2$	0	0.3150	0.6504	0.8323	0.9221
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1)	0	0.3223	0.6556	0.8355	0.9240
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2)	0	0.3508	0.6745	0.8430	0.9266
E$({\mathit{\delta }}_{\mathit{C}}^2$|X3)	0	0.3223	0.6556	0.8355	0.9240
E$({\mathit{\delta }}_{\mathit{C}}^2$|X4)	0	0.3284	0.6576	0.8371	0.9239
E$({\mathit{\delta }}_{\mathit{C}}^2$|X5)	0	0.3284	0.6576	0.8371	0.9239
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2)	0.1803	0.4633	0.7270	0.8664	0.9367
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X3)	0	0.3509	0.6746	0.8432	0.9267
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X4)	0	0.3362	0.6628	0.8413	0.9265
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X5)	0	0.3362	0.6628	0.8413	0.9265
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2, X3)	0.1803	0.4633	0.7270	0.8664	0.9367
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2, X4)	0	0.3704	0.6825	0.8478	0.9286
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2, X5)	0	0.3704	0.6825	0.8478	0.9286
E$({\mathit{\delta }}_{\mathit{C}}^2$|X3, X4)	0	0.3370	0.6628	0.8392	0.9252
E$({\mathit{\delta }}_{\mathit{C}}^2$|X3, X5)	0	0.3370	0.6628	0.8392	0.9252
E$({\mathit{\delta }}_{\mathit{C}}^2$|X4, X5)	0.2581	0.4820	0.7238	0.8616	0.9326
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2, X3)	0.4145	0.6274	0.8169	0.9133	0.9604
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2, X4)	0.1918	0.4852	0.7357	0.8713	0.9385
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2, X5)	0.1918	0.4852	0.7357	0.8713	0.9385
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X3, X4)	0	0.3702	0.6827	0.8481	0.9287
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X3, X5)	0	0.3702	0.6827	0.8481	0.9287
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X4, X5)	0.2644	0.4923	0.7303	0.8664	0.9356
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2, X3, X4)	0.1925	0.4844	0.7363	0.8722	0.9395
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2, X3, X5)	0.1925	0.4844	0.7363	0.8722	0.9395
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2, X4, X5)	0.3151	0.5425	0.7561	0.8760	0.9393
E$({\mathit{\delta }}_{\mathit{C}}^2$|X3, X4, X5)	0.2649	0.4928	0.7298	0.8644	0.9343
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2, X3, X4)	0.4630	0.6677	0.8346	0.9231	0.9648
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2, X3, X5)	0.4630	0.6677	0.8346	0.9231	0.9648
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2, X4, X5)	0.5361	0.6802	0.8236	0.9083	0.9539
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X3, X4, X5)	0.3123	0.5449	0.7569	0.8768	0.9397
E$({\mathit{\delta }}_{\mathit{C}}^2$|X2, X3, X4, X5)	0.5361	0.6802	0.8236	0.9083	0.9539
E$({\mathit{\delta }}_{\mathit{C}}^2$|X1, X2, X3, X4, X5)	1	1	1	1	1

Table 1 presents the average values of ${\delta }_C^2$ for various fixed input variables, computed using the LHS method with n = 12,000 runs. The conditional values of ${\delta }_C^2$ are estimated using a double-nested-loop computation.

presents the estimated values of C₀ = ${\delta }_C^2$, where ${\delta }_C$ is obtained using n = 12,000 runs of the Latin Hypercube Sampling (LHS) method [62,63]. The conditional values of ${\delta }_C$ are estimated using double-nested-loop computation of the LHS method. When a single input variable, X_i, is fixed, it is sampled using n = 12,000 runs, and for each realization, ${\delta }_C$ is calculated again using n = 12,000 runs of the LHS method. This numerical procedure is analogous to that of Sobol’ [7,8], but the sensitivity measure is not based on variance but on Cliff’s Delta. When computing S_i, the computational complexity of E(${\delta }_C^2$|X_i) is n² = 144,000,000.

In Table 1, the data in the last column represent a set of values that are highly consistent, with low variance. Most values range between 0.924 and 0.940, with a few exceptions approaching 1.000. In Table 1, the value 1 is always present in the last row, ensuring that the sum of all indices is equal to one.

The column with K = 0 exhibits values close to zero, indicating the absence of dominance of R over F in the observations of ${\delta }_C^2$. Conversely, values far from zero suggest a dominance of R over F in the observation of ${\delta }_C^2$.

In general, the strong influence of an input variable or variables occurs when the mean value of the fixed realizations of ${\delta }_C^2$ is significantly different from C₀—see Equations (14)–(16). It can be noted that the accuracy of the estimation of the sensitivity indices is lowest for K = 4, where the conditioned realizations of ${\delta }_C^2$ in the last column of Table 1 are very consistent, with low variance, and are minimally different from C₀.

The influences of the input variables and their groups, as expressed using sensitivity indices, are displayed in Figure 1, Figure 2 and Figure 3. The color legend in Figure 1 is applied to all subsequent pie charts in this article.

In Figure 1, the absence of red in the pie chart demonstrates that all first-order sensitivity indices (represented by the red square in the legend) are zero. The second-order sensitivity indices S₁₂ = 0.18 and S₂₃ = 0.18 have the same value, which is due to the nature of Equation 10 and the same characteristics of the input random variables. The dominant influence is the interaction effect of variables X₄ and X₅, as indicated by the value S₄₅ = 0.26.

Increasing the value of the constant K distances the random realizations of resistance R from load action F and enhances the value of Cliff’s Delta. The interaction effects indicated by sensitivity indices S₁₂, S₂₃, and S₄₅ decrease with increasing K, while the proportion of third-, fourth-, and fifth-order sensitivity indices increases—see Figure 2 and Figure 3.

In the sensitivity analysis based on Cliff’s Delta, the impact of input variables on the observed changes in ${\delta }_C^2$ is quantified using sensitivity indices of the first order and higher orders. To derive meaningful conclusions and to categorize input variables into influential, less influential, and non-influential groups, it is essential to assign the effects of each input variable without the complexity of interpreting numerous sensitivity indices.

To achieve this, the concept of the total effect index is employed. This index captures the comprehensive contribution of a factor, X_i, to the changes observed in ${\delta }_C^2$. Specifically, it encompasses both the first-order effects and all higher-order effects resulting from interactions. The total effect index provides a robust measure of the impact that each input variable has on the ${\delta }_C^2$, accounting for all potential interactions.

For instance, in a five-factor model, the total effect of the factor X₁ is calculated by adding all terms in Equation (18) where the factor X₁ is included.

$$S_{T1}=S_1+S_{12}+S_{13}+S_{14}+S_{15}+S_{123}+S_{124}+S_{125}+S_{134}+S_{135}+S_{145}+S_{1234}+S_{1235}+S_{1245}+S_{1345}+S_{12345}.$$

(25)

This sum accounts for X₁ direct influence on ${\delta }_C^2$, as well as its synergistic effects with other factors. The total effect measure provides an educated answer to the following question: which factor can be fixed anywhere over its range of variability without affecting the ${\delta }_C^2$? The total effect index reflects both the main and the interaction influences of X₁ on the outcome, providing a comprehensive view of its relative importance in the system’s reliability, measured by the distance from F to R. The total effects for the case study are displayed in Figure 4.

The sensitivity analysis results show the total sensitivity indices for each input variable (X₁, X₂, X₃, X₄, and X₅) across five distinct constant values (K = 0, 1, 2, 3, and 4), using Cliff’s delta as the sensitivity measure.

As K increases from 0 to 4, a general trend of increasing total sensitivity indices is observed for X₁ and X₂, as depicted in Figure 4. For X₃, the total sensitivity indices exhibit a slightly convex pattern. An increasing trend for X₃ is observed from K = 1 to K = 4. This suggests that these variables become more influential with higher values of K, as measured using Cliff’s delta. Conversely, the sensitivity indices for X₄ and X₅ decrease with increasing K.

The variable X₂ consistently exhibits the highest total sensitivity index across all tested values of K, indicating its dominant influence on Cliff’s delta. This effect is attributable to X₂’s involvement in both additive terms of the resistance function, as shown in Equation (23). The variables X₁ and X₃ also demonstrate an increasing influence, although they remain slightly less dominant than X₂, but are notably more influential than X₄ and X₅ as K increases. The influence of X₄ and X₅, which are involved in the load force equation, decreases especially as K surpasses 1, highlighting their reduced significance in affecting Cliff’s Delta.

The sensitivity analysis outcomes reflect a decreasing probability P(R < F), which diminishes as K increases. The Cliff Delta-based sensitivity analysis exhibits characteristics of a reliability-oriented sensitivity analysis [19], describing the change in the influence of each variable on Cliff’s Delta due to K. The influence on the results of the sensitivity indices according to the value of the deterministic quantity K is the main difference compared to the Sobol’ sensitivity analysis.

The results of the classical Sobol’ sensitivity analysis can be obtained analytically. Non-zero values of the Sobol’ sensitivity indices were obtained only for second-order sensitivity indices $S_{12}^{Sob}$ = 1/3, $S_{23}^{Sob}$ = 1/3, and $S_{45}^{Sob}$ = 1/3; other Sobol’ indices are zero. The total effect Sobol’ sensitivity indices are $S_{T1}^{Sob}$ = 1/3, $S_{T2}^{Sob}$ = 2/3, $S_{T3}^{Sob}$ = 1/3, $S_{T4}^{Sob}$ = 1/3, and $S_{T5}^{Sob}$ = 1/3. The dominant influence of the input variable X₂ confirms the most important conclusions of the newly introduced sensitivity analysis based on Cliff’s Delta; however, there are differences. The results of the Sobol’ sensitivity analysis are independent of the value of the constant K, because the Sobol’ indices are based on the decomposition of variance, which the deterministic variable K does not affect.

Although the classic Sobol’ sensitivity analysis of model output is empathetic to the results of reliability-oriented types of sensitivity analyses, it is not directly oriented towards reliability, as the Sobol’ sensitivity indices do not reflect changes in P_f [19].

The sensitivity indices results based on Cliff’s Delta closely resemble those obtained from the sensitivity analysis based on P_f. The pie chart on the left side of Figure 5 is practically identical to the chart in Figure 1. Moreover, the pie chart on the right side of Figure 5 is very similar to the chart on the right side of Figure 3. The results of the sensitivity analysis are the same, but the sensitivity scale is different. Overall, this demonstrates a high degree of similarity between Cliff’s Delta and P_f, from which useful conclusions can be drawn.

The sensitivity indices based on Cliff’s Delta are derived from the calculation of P_f using Cliff’s Delta. While it may seem that estimating Cliff’s Delta is more demanding compared to P_f, using double-loop simulations to estimate Cliff’s Delta leads to a more accurate numerical estimate of P_f and extracts more information from the available simulations. It can be noted that a similar double-loop simulation is used for estimating P_f based on the numerical integration of the distributions of the random variables R and F, see, e.g., [64].

The advantage of estimating Cliff’s Delta is that it does not require knowledge of the distributions or approximation approaches of the random variables R and F. This flexibility allows for a more robust analysis in practical scenarios where precise distribution functions may not be available or easily determined.

5. Comparative Analysis of P_f Estimations Using Cliff’s Delta and Basic Definition

It can be shown that the estimation of P_f according to Equation (10) is more accurate compared to the basic definition in Equation (5) when Cliff’s Delta is calculated using a double-nested-loop simulation according to Equation (4).

Let P_f denote the failure probability estimated from a binomial distribution, and let n represent the number of simulations or experiments.

$$P_f=\frac{1}{n}{\displaystyle \sum _{i=1}^n{1}_{Z<0}}.$$

(26)

The standard error SE for the failure probability estimation is provided by the binomial distribution

$$S{E}_1\left(P_f\right)=\sqrt{\frac{P_f\left(1-P_f\right)}{n}}.$$

(27)

The estimate of the failure probability P_f using Cliff’s Delta in Equation (10) is more accurate because it utilizes an increased number of simulations through a double-nested-loop process. However, the standard error SE₂(P_f) for this estimate cannot be straightforwardly determined using an analytical formula similar to SE₁(P_f). Instead, empirical observations indicate that the standard error for P_f estimated with Cliff’s Delta is smaller. This improved accuracy is attributed to the increased statistical information gained from the double-nested-loop simulations. The accuracy of both estimates can be demonstrated in the following case study using the Monte Carlo method.

The limit state of a rod made of elastic material, subjected to axial tension, is studied. Let resistance R have a Gaussian probability density function with a mean value μ_R = 9 kN and standard deviation σ_R = 0.9 kN, and let F have a Gaussian probability density function with a mean value μ_F = 4 kN and standard deviation σ_F = 1.218887 kN. Under these assumptions, Z is a random variable with a Gaussian probability density function with a mean value μ_Z = 9 − 4 = 5 kN and standard deviation σ_Z = (0.9² + 1.218887²)^0.5 = 1.515 kN. The failure probability P_f = 0.000483477 is evaluated using the numerical integration of the Gaussian probability density function of Z from negative infinity to zero. According to EN1990 [15], a structure designed with this failure probability is classified in reliability class RC1 for reference periods of 1 and 50 years.

The case study aim to compare the accuracy of P_f estimation according to the basic definition in Equation (5) and the alternative formula in Equation (10). The procedure is as follows: In the first step, the random variables R and F are simulated for 10,000 runs using the Monte Carlo (MC) method. The estimation of P_f from Equation (5) is plotted in Figure 6 as the first blue point from the left. The estimation of P_f from Equation (10) is plotted in Figure 7 as the first blue point from the left. Next, another 10,000 runs of MC are generated, and the procedure is repeated. In total, 100 estimates of P_f according to Equation (5) are plotted in the left quarter of Figure 6, and 100 estimates of P_f according to Equation (10) are plotted in the left quarter of Figure 7. This procedure is further repeated for 100,000, 1 million, and 10 million runs of the Monte Carlo (MC) method, as shown in the subsequent quarters in Figure 6 and Figure 7.

The numerical study revealed that the failure probability P_f estimation using Cliff’s Delta δ_C, see Equation (10), is more accurate compared to the basic definition—see Equation (5). As illustrated in Figure 6 and Figure 7, the alternative method resulted in noticeably lower standard deviations σ_Pf (approximately three times smaller) across varying Monte Carlo run counts, indicating improved consistency and precision. In contrast, the basic definition exhibited higher variability, especially with fewer Monte Carlo runs. These findings underscore the efficacy of the alternative formula for more reliable P_f estimations in structural reliability assessments.

6. Convergence Study of P_f Estimations Using Cliff’s Delta and Varying Runs of R and F

In structural engineering, complex mathematical models make uncertainty propagation via classical MC simulations difficult due to numerous numerical evaluations. This chapter shows that P_f estimation can be improved by combining a small number of computationally expensive (costly) simulations of R with a large number of computationally inexpensive (cheap) simulations of F.

Consider a case study where CPU time is limited, and repeated runs of a computational model are expensive (e.g., a finite element model, FEM). Only m = 1000 simulation runs of such a model, whose output is R, are available. Conversely, the load F can be simulated inexpensively using a very high number n of Monte Carlo simulations. The aim is to examine the accuracy of estimating P_f using Cliff’s delta and an alternative definition in Equation (10).

Assuming the arrays R and F have sizes m and n, respectively, the formula for calculating δ_C is given as follows:

$${\delta }_C=\frac{{\displaystyle {\sum }_{i=1}^m{\displaystyle {\sum }_{j=1}^n\left[R_i>F_j\right]-\left[R_i<F_j\right]}}}{m·n}, \mathrm{where} {\delta }_C\in [-1, 1].$$

(28)

In the case study, the computational model is considered in a closed form, where the resistance R is derived as the product of three input random variables f_y, b, t, each with Gaussian probability density functions. It can be noted that R itself does not follow a Gaussian probability density function.

$$R=f_y·b·t,$$

(29)

where R is the resistance of a tensile steel bar, f_y is the yield strength of S355 steel, b is the width of the rectangular cross-section, and t is its thickness. The input variables are characterized statistically based on experimental research, with μ_fy = 393.8 MPa, σ_fy = 22 MPa, μ_b = 10 mm, σ_b = 0.1 mm, μ_t = 4 mm, and σ_t = 0.2 mm [65,66]. The load F follows a Gaussian probability density function with a mean value μ_F = 8 kN and standard deviation σ_F = 2 kN.

The failure probability P_f estimated using Equation (26) and a large number of 20 million Monte Carlo runs is P_f = 0.0003954.

The case study aims to compare the accuracy of the P_f estimate according to the alternative formula in Equation (28) using different numbers of m and n runs for R and F. The procedure is as follows: The input random variables f_y, b, t, and their product R are simulated using m = 1000 of the Latin Hypercube Sampling (LHS) method. The random variable F is simulated using n = 1000 runs of the MC method. Cliff’s Delta is calculated using Equation (28), and P_f is calculated from Equation (10) and is plotted in Figure 8 as the first blue point from the left. Subsequently, an additional m = 1000 runs of LHS for f_y, b, and t are generated, R is recalculated for these 1000 runs, and another n = 1000 runs of F are generated. The P_f estimation procedure is repeated and another blue point is plotted. In total, 100 estimates of P_f according to Equation (10) are plotted in the left part of Figure 8.

This procedure is further repeated for n = 10⁴, 10⁵, 10⁶, and 10⁷ runs of the Monte Carlo method, as illustrated in the following four-fifths of Figure 8. Figure 9 illustrates the results of the same study, but using m = 100 runs.

The case study demonstrates that a higher number of simulations for the load F (compared to R) improves the accuracy of the estimate of P_f using the alternative definition in Equation (10), where δ_C is calculated according to Equation (28). For both m = 1000 and m = 100, convergence to an accurate solution was achieved when the number of simulations for F reached approximately 10⁵—see Figure 8 and Figure 9. Further increasing the number of simulations for F has a negligible effect on reducing the standard deviation of P_f.

7. Discussion

In civil engineering, estimating resistance R using nonlinear FEM calculations is time-consuming, and studies are limited by the computational cost of the number of Monte Carlo or LHS method runs required to simulate real structural behavior, see, e.g., [67,68,69,70].

Recent advancements in nonlinear FEM have introduced techniques that significantly reduce computational costs [71,72]. One such technique is using the Woodbury formula with interval analysis [72], which efficiently handles low-rank perturbations by avoiding the direct inversion of large stiffness matrices. This is achieved by keeping the global stiffness matrix unchanged and focusing computational efforts on a small Schur complement matrix representing local nonlinear behavior. Integrating interval analysis with the Sherman–Morrison–Woodbury formula [73] is a significant advancement in reducing computational costs for nonlinear FEM analysis [71]. To further enhance efficiency, parallel computation techniques like OpenMP can be employed [74]. Additionally, mathematical models that offer quick analytical solutions remain attractive due to their efficiency in providing rapid responses [75,76]. Overall, these advanced computational models allow for better integration with advanced numerical and simulation methods, enabling more efficient and comprehensive probabilistic reliability and sensitivity analysis.

The EN1990 [15] standard provides several options for reliability assessment. One common method is the FORM (First Order Reliability Method), where Z is approximated using a Gaussian probability density function. This method estimates reliability by comparing the reliability index β = μ_Z/σ_Z with the target reliability index β_t [15,19,25]. However, departure from the Gaussian distribution can lead to inaccuracies, and β cannot be used in reliability-oriented global sensitivity analysis based on Sobol’. In contrast, P_f, as a component of both the first and second moments of the binary reliability function, is generally a more suitable metric for both reliability and sensitivity analyses.

In the present article, Equation (10) offers an efficient alternative for estimating P_f with high utilization of information from the realized runs of the computational model. Although estimating Cliff’s Delta is more numerically demanding than the direct estimation of failure probability using Equation (5), the same number of model runs (the same number of random realizations of R and F) ensures a more accurate estimation of failure probability P_f—see Figure 6 compared to Figure 7. These advantages highlight the usefulness of Cliff’s Delta in both reliability and sensitivity analysis, where the estimation of P_f is applied.

According to the EN1990 [15] standard, common building structures are designed with a P_f around 7.2 · 10⁻⁵. For estimating the standard error of P_f using Equation (26), one needs to perform 1,388,889 Monte Carlo simulation runs to achieve a coefficient of variation of 0.1. This results in a standard error of SE₁ = 7.2⋅10⁻⁶. In practice, this requires conducting Monte Carlo runs until 100 failure events are observed, i.e., 100 runs where Z < 0. As described in previous chapters, the use of Cliff’s Delta and employing Equation (10) instead of Equation (5) increases the accuracy of the failure probability, P_f, estimate.

Furthermore, the computational complexity of estimating Cliff’s Delta can be reduced by optimizing the algorithm. In Equation (28), if a random realization of r_i is higher than the maximum random realization of all F, then the inner loop does not need to be executed for this r_i. The algorithm efficiently reduces the double-loop computational burden, especially for Cliff’s Delta estimates close to one (very small P_f values), where the vast majority of random realizations of R are much higher than F.

Conducting sensitivity analyses based on Cliff’s Delta offers functionalities similar to sensitivity analysis based on failure probability focused on reliability. Cliff’s Delta provides a robust measure for evaluating the extent to which one distribution dominates another without making specific assumptions about the distributions’ form or variability. This flexibility is particularly valuable in reliability studies where input variables may not follow normal distributions or exhibit significant variability. By focusing on the frequency with which resistance values exceed load values, Cliff’s Delta directly aligns with the fundamental concepts of reliability engineering.

8. Conclusions

In this article, an alternative measure of structural reliability based on Cliff’s Delta was adopted to ensure a more accurate estimation of failure probability P_f with the available number of simulations. Using Cliff’s Delta increases the accuracy of the P_f estimate by making better use of the statistical information from the data. This approach replaces the traditional failure probability calculations (such as Monte Carlo Simulation and First-Order Reliability Method—FORM) and provides enhanced flexibility and robustness in applications of structural reliability.

The case study illustrates that the use of Cliff’s Delta improves the accuracy of P_f estimations. For example, when using 10⁴ Monte Carlo runs, the standard deviation of P_f estimates using Cliff’s Delta was approximately 0.000077, compared to 0.000228 with the basic method in Equation (26), showing a reduction by a factor of nearly 3. Similar reductions in variability were observed across larger numbers of simulations, demonstrating the consistently higher precision of P_f estimates with Cliff’s Delta.

The adaptation of Cliff’s Delta, initially developed for ordinal data, has been successfully applied to evaluate the sensitivity of structural reliability. This adaptation allows for the evaluation of the dominance of resistance over load without specific distributional assumptions, making it particularly suitable for structural reliability analysis. The mathematical formulations for computing Cliff’s Delta between resistance R and load action F were presented. This formulation effectively quantifies the reliability of a structure by evaluating the probability that the resistance exceeds load.

Cliff’s Delta can be effectively evaluated using a small number of simulations for resistance R and a large number of simulations for load F. The case study demonstrated that increasing the number of simulations for F relative to R enhances the accuracy of the Cliff’s Delta estimate and the resulting P_f. This approach ensures precise reliability assessments even with limited computational resources for expensive simulations of R.

Sensitivity indices based on the squared value of Cliff’s Delta ${\delta }_C^2$ were derived, demonstrating properties analogous to those used in the Sobol’ sensitivity analysis. This includes first-order, second-order, and higher-order sensitivity indices, which offer a comprehensive framework for evaluating the contributions of individual variables and their interactions to the overall reliability of the system. These indices provide a nuanced understanding of the factors influencing structural reliability.

The application of Cliff’s Delta in reliability-oriented sensitivity analysis allows for a more computationally efficient evaluation of structural reliability. This is particularly beneficial in engineering applications where finite element method (FEM) calculations and repeated simulations of R or F are computationally expensive.

While the current study provides a foundation, future work could focus on optimizing the computational algorithms for estimating Cliff’s Delta, particularly for the global sensitivity analysis of reliability.