Towards Overcoming the Curse of Dimensionality: The Third-Order Adjoint Method for Sensitivity Analysis of Response-Coupled Linear Forward/Adjoint Systems, with Applications to Uncertainty Quantification and Predictive Modeling

Abstract

This work presents the Third-Order Adjoint Sensitivity Analysis Methodology (3rd-ASAM) for response-coupled forward and adjoint linear systems. The 3rd-ASAM enables the efficient computation of the exact expressions of the 3rd-order functional derivatives (“sensitivities”) of a general system response, which depends on both the forward and adjoint state functions, with respect to all of the parameters underlying the respective forward and adjoint systems. Such responses are often encountered when representing mathematically detector responses and reaction rates in reactor physics problems. The 3rd-ASAM extends the 2nd-ASAM in the quest to overcome the “curse of dimensionality” in sensitivity analysis, uncertainty quantification and predictive modeling. This work also presents new formulas that incorporate the contributions of the 3rd-order sensitivities into the expressions of the first four cumulants of the response distribution in the phase-space of model parameters. Using these newly developed formulas, this work also presents a new mathematical formalism, called the 2nd/3rd-BERRU-PM “Second/Third-Order Best-Estimated Results with Reduced Uncertainties Predictive Modeling”) formalism, which combines experimental and computational information in the joint phase-space of responses and model parameters, including not only the 1st-order response sensitivities, but also the complete hessian matrix of 2nd-order second-sensitivities and also the 3rd-order sensitivities, all computed using the 3rd-ASAM. The 2nd/3rd-BERRU-PM uses the maximum entropy principle to eliminate the need for introducing and “minimizing” a user-chosen “cost functional quantifying the discrepancies between measurements and computations,” thus yielding results that are free of subjective user-interferences while generalizing and significantly extending the 4D-VAR data assimilation procedures. Incorporating correlations, including those between the imprecisely known model parameters and computed model responses, the 2nd/3rd-BERRU-PM also provides a quantitative metric, constructed from sensitivity and covariance matrices, for determining the degree of agreement among the various computational and experimental data while eliminating discrepant information. The mathematical framework of the 2nd/3rd-BERRU-PM formalism requires the inversion of a single matrix of size N_r × N_r, where N_r denotes the number of considered responses. In the overwhelming majority of practical situations, the number of responses is much less than the number of model parameters. Thus, the 2nd-BERRU-PM methodology overcomes the curse of dimensionality which affects the inversion of hessian matrices in the parameter space.

1. Introduction

The functional derivatives (also called “sensitivities”) of results (also called “responses”) are needed for many purposes, including: (i) understanding the model by ranking the importance of the various parameters; (ii) performing “reduced-order modeling” by eliminating unimportant parameters and/or processes; (iii) quantifying the uncertainties induced in a model response due to model parameter uncertainties; (iv) performing “model validation,” by comparing computations to experiments to address the question “does the model represent reality?” (v) prioritizing improvements in the model; (vi) performing data assimilation and model calibration as part of forward “predictive modeling” to obtain best-estimate predicted results with reduced predicted uncertainties; (vii) performing inverse “predictive modeling”; (viii) designing and optimizing the system. As is well known, even the approximate determination of the first-order sensitivities $\partial R/\partial {\mathbf{\alpha }}_i$, i = 1, …, N_α of a model response R to N_α parameters α_i using conventional finite-difference methods would already require N_α large-scale computations with altered parameter values, which is unfeasible for large-scale models comprising many parameters. The computation of higher-order sensitivities by conventional methods is limited in practice by the so-called “curse of dimensionality,” since the number of such large-scale computations increases exponentially with the order of the response sensitivities. For the exact computation of the first- and second-order response sensitivities to parameters, the “curse of dimensionality” has been overcome by the Second-Order Adjoint Sensitivity Analysis Methodology (2nd-ASAM) conceived and developed by Cacuci [1,2,3]. The unique capability of the 2nd-ASAM to compute comprehensively and efficiently the exact first-and second-order sensitivities of a response to parameters in a large-scale physical system has been demonstrated [4,5,6] by an application to a reactor physics system which comprises 21,976 first-order sensitivities and 482,944,576 second-order sensitivities.

Section 2 and Section 3 of this work present the Third-Order Adjoint Sensitivity Analysis Methodology (3rd-ASAM) for coupled forward and adjoint linear systems, which evidently extends and generalizes the 2nd-ASAM. The 3rd-ASAM aims at the efficient computation of the exact expressions of the 3rd-order functional derivatives (“sensitivities”) of a system response that depends on both the forward and adjoint state functions with respect to all of the parameter underlying the respective forward and adjoint systems. Such responses are often encountered when representing mathematically detector responses and reaction rates in reactor physics problems. The 3rd-ASAM will be applied to the reactor physics system analyzed in [4,5,6,7] to compute the exact magnitude of the 3rd-order sensitivities to the model parameters that were found in [4,5,6,7] to have unexpectedly large 2nd-order sensitivities. Furthermore, the 3rd-order sensitivities computed using the 3rd-ASAM are incorporated in the new formulas, presented in Section 4, for computing to 3rd-order the first four cumulants of the response distribution in the phase-space of model parameters. Section 5 presents a new mathematical formalism, which will be called the “Second/Third-Order Best-Estimated Results with Reduced Uncertainties Predictive Modeling (2nd/3rd-BERRU-PM).” Set in the joint phase-space of responses and parameters, the 2nd/3rd-BERRU-PM incorporates experimental and computational information, including the complete (as opposed to partial) vector of 1st-order response sensitivities, the complete hessian matrix of 2nd-order second-sensitivities and also the 3rd-order sensitivities, all computed using the 3rd-ASAM presented in Section 3. Thus, the 2nd/3rd-BERRU-PM extends the “BERRU Predictive Modeling” [7], thereby generalizing and including, as particular cases, similar formulas used in other fields e.g., [8,9,10]. The 2nd/3rd-BERRU-PM uses the maximum entropy (MaxEnt) principle [11] to eliminate the need for introducing and “minimizing” a user-chosen “cost functional quantifying the discrepancies between measurements and computations.” Incorporating correlations, including those between the imprecisely known model parameters and computed model responses, the 2nd/3rd-BERRU-PM also provides a quantitative metric, constructed from sensitivity and covariance matrices, for determining the degree of agreement among the various computational and experimental data and helping eliminate discrepant information. Conclusions regarding the significance of this work’s novel results in the quest to overcome the curse of dimensionality in sensitivity analysis, uncertainty quantification and predictive modeling are presented in Section 6.

2. Mathematical Description of the Physical System

A linear physical system is generally represented by means of N_u coupled operator equations of the form ${\displaystyle \sum _{j=1}^{N_u}{L}_{ij}\left(\mathbf{\alpha }\right){\phi }_j\left(x\right)}=Q_i\left(\mathbf{\alpha }\right),i=1,\dots ,N_u,\hspace{1em}x\in {\Omega }_x,$ in which the operators $L_{ij}\left(\mathbf{\alpha }\right)$ act linearly on the state functions ${\phi }_j\left(x\right)$. The above system of equations can be written in matrix form as follows:

$$L\left[\mathbf{\alpha }\left(x\right)\right]\mathsf{\phi }\left(x\right)=Q\left[\mathbf{\alpha }\left(x\right)\right],x\in {\Omega }_x$$

(1)

Matrices and vectors will be denoted using bold letters. Since the right-side of Equation (1) may contain distributions, the equality in this equation is considered to hold in the weak (“distributional”) sense. Similarly, all of the equalities that involve differential equations in this work will be considered to hold in the weak/distributional sense. All vectors in this work are considered to be column vectors, and transposition will be indicated by a dagger $(†)$ superscript. The vectors, matrices and operators appearing in Equation (1) are defined as follows:

• $\mathbf{\alpha }\triangleq {\left({\mathsf{\alpha }}_1,\dots ,{\mathsf{\alpha }}_{N_{\mathsf{\alpha }}}\right)}^{†}$ denotes a $N_{\mathsf{\alpha }}$-dimensional column vector whose components are the physical system’s imprecisely known parameters, which are subject to uncertainties; $\mathbf{\alpha }\in {\mathrm{E}}_{\mathsf{\alpha }}\subset {ℝ}^{N_{\mathsf{\alpha }}}$, where ${\mathrm{E}}_{\mathsf{\alpha }}$ denotes a subset of the $N_{\mathsf{\alpha }}$-dimensional real vector space ${ℝ}^{N_{\mathsf{\alpha }}}$. The symbol “$\triangleq$” will be used to denote “is defined as” or “is by definition equal to.” The vector $\mathbf{\alpha }\in {\mathrm{E}}_{\mathsf{\alpha }}\subset {ℝ}^{N_{\mathsf{\alpha }}}$ is considered to include any imprecisely known model parameters that may enter into defining the system’s boundary in the phase space of independent variables.
• $x\triangleq {\left(x_1,\dots ,x_{N_x}\right)}^{†}\in {ℝ}^{N_x}$ denotes the $N_x$-dimensional phase-space position vector, defined on a phase-space domain denoted as ${\Omega }_x$.
• $\mathsf{\phi }\left(x\right)\triangleq {\left[{\phi }_1\left(x\right),\dots ,{\phi }_{N_u}\left(x\right)\right]}^{†}$ denotes a $N_u$-dimensional column vector whose components represent the system’s dependent variables (also called “state functions”). In virtually all of the physical systems represented by Equation (1), the components ${\phi }_i\left(x\right),i=1,\dots ,N_u$ are square-integrable functions and $\mathsf{\phi }\left(x\right)\in {\mathrm{H}}_{\phi }$, where ${\mathrm{H}}_{\phi }$ is a Hilbert space endowed with an inner product that will be denoted as ${\langle \mathsf{\phi }\left(x\right),\mathsf{\psi }\left(x\right)\rangle }_{\phi }$, $\mathsf{\psi }\left(x\right)\in {\mathrm{H}}_{\phi }$, and which is defined as follows:

  $${\langle \mathsf{\phi }\left(x\right),\mathsf{\psi }\left(x\right)\rangle }_{\phi }\triangleq {\displaystyle \underset{{\Omega }_x}{\int }\mathsf{\phi }\left(x\right)·\mathsf{\psi }\left(x\right)d}x\triangleq {\displaystyle \sum _{i=1}^{N_u}{\displaystyle \underset{{\Omega }_x}{\int }{\phi }_i\left(x\right){\psi }_i\left(x\right)d}x}.$$

  (2)
• $Q\left(\mathbf{\alpha }\right)\triangleq {\left[Q_1\left(\mathbf{\alpha }\right),\dots ,Q_{N_u}\left(\mathbf{\alpha }\right)\right]}^{†}$ denotes a $N_u$-dimensional column vector. The components of $Q$ are operators acting (in general nonlinearly) on $\mathbf{\alpha }$ and $x$;
• $L\left(\mathbf{\alpha }\right)\triangleq {\left[L_1\left(\mathbf{\alpha }\right),\dots ,L_{N_u}\left(\mathbf{\alpha }\right)\right]}^{†}$ denotes a $N_u$-component column vector. The components of $L\left(\mathbf{\alpha }\right)$ are operators acting linearly on $\mathsf{\phi }$ and nonlinearly on $\mathbf{\alpha }$. When $L\left(\mathbf{\alpha }\right)$ contains differential operators, a set of boundary and/or initial conditions which define the domain of $L\left(\mathbf{\alpha }\right)$ must also be given. Since $L\left(\mathbf{\alpha }\right)$ is considered to act linearly on $\mathsf{\phi }\left(x\right)$, the accompanying boundary and/or initial conditions must also be linear in $\mathsf{\phi }\left(x\right)$. Such linear boundary and/or initial conditions are represented in the following operator form:

  $$B_F\left(\mathbf{\alpha }\right)\mathsf{\phi }\left(x\right)-C_F\left(\mathbf{\alpha }\right)=0,x\in \partial {\Omega }_x.$$

  (3)

In Equation (3), the operator $B_F\left(\mathbf{\alpha }\right)\triangleq \left[B_{ij}\left(\mathbf{\alpha }\right)\right];i=1,\dots ,N_B;j=1,\dots ,N_u$ is a matrix comprising, as components, operators that act linearly on $\mathsf{\phi }\left(x\right)$ and nonlinearly on $\mathbf{\alpha }$; the quantity $N_B$ denotes the total number of boundary and initial conditions. The operator $C_F\left(\mathbf{\alpha }\right)\triangleq {\left[C_1\left(\mathbf{\alpha }\right),\dots ,C_{N_B}\left(\mathbf{\alpha }\right)\right]}^{†}$ is a $N_B$-dimensional vector comprising components that are operators acting, in general, nonlinearly on $\mathbf{\alpha }$. The subscript “F” in Equation (3) indicates boundary conditions associated with the “forward” system of equations.

In most practical situations the Hilbert space ${\mathrm{H}}_{\phi }$ is self-dual. The operator $L\left(\mathbf{\alpha }\right)$ admits an adjoint (operator), which will be denoted as $L^{+}\left(\mathbf{\alpha }\right)$, and which is defined through the following relation for an arbitrary vector $\mathsf{\psi }\left(x\right)\in {\mathrm{H}}_{\phi }$:

$${\langle \mathsf{\psi }\left(x\right),L\left(\mathbf{\alpha }\right)\mathsf{\phi }\left(x\right)\rangle }_{\phi }={\langle L^{+}\left(\mathbf{\alpha }\right)\mathsf{\psi }\left(x\right),\mathsf{\phi }\left(x\right)\rangle }_{\phi }$$

(4)

In Equation (4), the formal adjoint operator $L^{+}\left(\mathbf{\alpha }\right)$ is the $N_u\times N_u$ matrix:

$$L^{+}\left(\mathbf{\alpha }\right)\triangleq \left[L_{ji}^{+}\left(\mathbf{\alpha }\right)\right],i,j=1,\dots ,N_u,$$

(5)

comprising elements $L_{ji}^{+}\left(\mathbf{\alpha }\right)$ obtained by transposing the formal adjoints of the operators $L_{ij}\left(\mathbf{\alpha }\right)$. Thus, the system adjoint to Equations (1) and (3) has the following general representation in operator form:

$$L^{+}\left(\mathbf{\alpha }\right)\mathsf{\psi }\left(x\right)=Q^{+}\left(\mathbf{\alpha }\right),x\in {\Omega }_x,$$

(6)

$$B_A\left(\mathbf{\alpha }\right)\mathsf{\psi }\left(x\right)-C_A\left(\mathbf{\alpha }\right)=0,x\in \partial {\Omega }_x.$$

(7)

The domain of $L^{+}\left(\mathbf{\alpha }\right)$ is determined by selecting the adjoint boundary and/or initial conditions represented in operator form in Equation (7), where the letter “A” indicates “adjoint” and the letter “B” indicates “boundary and/or initial conditions.” These adjoint boundary and/or initial conditions are selected so as to ensure that the boundary terms that arise when going from the left-side to the right side of Equation (4) vanish, in conjunction with the forward boundary conditions given in Equation (3). The source term $Q^{+}\left[\mathbf{\alpha }\left(x\right)\right]$ in Equation (6) is associated with the system’s response which, in this work, is considered to be a scalar-valued nonlinear functional of the adjoint and forward fluxes, which will be denoted as $R\left[\mathsf{\phi }\left(x\right),\mathsf{\psi }\left(x\right);\mathbf{\alpha }\right]$. Such responses are often encountered when representing mathematically detector responses and reaction rates in reactor physics problems, and can be generally represented in the following inner-product form:

$$R\left[\mathsf{\phi }\left(x\right);\mathsf{\psi }\left(x\right);\mathbf{\alpha }\right]\triangleq {\langle S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right),1\rangle }_{\phi }$$

(8)

where $S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)$ denotes a suitably differentiable function of its arguments. The nominal solution of Equations (1) and (3) is denoted as ${\mathsf{\phi }}^0\left(x\right)$, and is obtained by solving these equations at the nominal parameter values ${\mathbf{\alpha }}^0$. The superscript “zero” will henceforth be used to denote “nominal” or “expected” or “mean” values. Thus, the vectors ${\mathsf{\phi }}^0\left(x\right)$ and ${\mathbf{\alpha }}^0$ satisfy the following equations:

$$L\left({\mathbf{\alpha }}^0\right){\mathsf{\phi }}^0\left(x\right)=Q\left[{\mathbf{\alpha }}^0\left(x\right)\right],x\in {\Omega }_x,$$

(9)

$$B_F\left({\mathbf{\alpha }}^0\right){\mathsf{\phi }}^0\left(x\right)-C_F\left({\mathbf{\alpha }}^0\right)=0,x\in \partial {\Omega }_x.$$

(10)

Equations (9) and (10) represent the “base-case” or nominal state of the forward physical system. Similarly, the “base-case” or nominal state of the adjoint physical system is given by the following equations:

$$L^{+}\left({\mathbf{\alpha }}^0\right){\mathsf{\psi }}^0\left(x\right)=Q^{+}\left({\mathbf{\alpha }}^0\right),x\in {\Omega }_x,$$

(11)

$$B_A\left({\mathbf{\alpha }}^0\right){\mathsf{\psi }}^0\left(x\right)-C_A\left({\mathbf{\alpha }}^0\right)=0,x\in \partial {\Omega }_x.$$

(12)

The nominal value of the response, $R\left[{\mathsf{\phi }}^0\left(x\right),{\mathsf{\psi }}^0\left(x\right);{\mathbf{\alpha }}^0\right]$, is determined by using the nominal parameter values ${\mathbf{\alpha }}^0$, the nominal value of the forward function ${\mathsf{\phi }}^0\left(x\right)$ obtained by solving Equations (9) and (10), and the nominal value of the adjoint function obtained by solving Equations (11) and (12).

3. The Third-Order Adjoint Sensitivity Analysis Methodology (3rd-ASAM) for Coupled Linear Forward and Adjoint Systems: Another Step Towards Overcoming the Curse of Dimensionality in the Exact Computation of High-Order Response Sensitivities

The model parameters ${\mathsf{\alpha }}_i$ are imprecisely known quantities, so their actual values may differ from their nominal values by quantities denoted as $\mathsf{\delta }{\mathsf{\alpha }}_i\triangleq {\mathsf{\alpha }}_i-{\mathsf{\alpha }}_i^0,i=1,\dots ,N_{\mathsf{\alpha }}.$ Since the model parameters $\mathbf{\alpha }$ and the state functions are related to each other through the forward and adjoint systems, it follows that variations $\mathsf{\delta }\mathbf{\alpha }\triangleq \left(\mathsf{\delta }{\mathsf{\alpha }}_1,\dots ,\mathsf{\delta }{\mathsf{\alpha }}_{N_{\mathsf{\alpha }}}\right)$ in the model parameters will cause corresponding variations $\mathsf{\delta }\mathsf{\phi }\triangleq \left(\mathsf{\delta }{\mathsf{\phi }}_1,\dots ,\mathsf{\delta }{\mathsf{\phi }}_{N_u}\right),$$\mathsf{\delta }{\mathsf{\phi }}_i\triangleq {\mathsf{\phi }}_i-{\mathsf{\phi }}_i^0,$$i=1,\dots ,N_u$ and $\mathsf{\delta }\mathsf{\psi }\triangleq \left(\mathsf{\delta }{\mathsf{\psi }}_1,\dots ,\mathsf{\delta }{\mathsf{\psi }}_{N_u}\right)$,$\mathsf{\delta }{\mathsf{\psi }}_i\triangleq {\mathsf{\psi }}_i-{\mathsf{\psi }}_i^0,$$i=1,\dots ,N_u$ in the forward and adjoint state functions. In turn, the variations $\mathsf{\delta }\mathbf{\alpha }$, $\mathsf{\delta }\mathsf{\phi }$, and $\mathsf{\delta }\mathsf{\psi }$ cause a response variation $R\left({\mathsf{\phi }}^0+\mathsf{\delta }\mathsf{\phi };{\mathsf{\psi }}^0+\mathsf{\delta }\mathsf{\psi };{\mathbf{\alpha }}^0+\mathsf{\delta }\mathbf{\alpha }\right)$ around the nominal response value $R\left[{\mathsf{\phi }}^0\left(x\right),{\mathsf{\psi }}^0\left(x\right);{\mathbf{\alpha }}^0\right]$.

3.1. The First-Level Adjoint System (1st-LASS) for Computing Exactly and Efficiently the First-Order Model Response Sensitivities to Parameters

The total first-order sensitivity of the response $R\left[\mathsf{\phi }\left(x\right),\mathsf{\psi }\left(x\right);\mathbf{\alpha }\right]$ to variations $\delta \mathbf{\alpha }\triangleq {\left[\delta {\mathsf{\alpha }}_1,\dots ,\delta {\alpha }_{N_{\alpha }}\right]}^{†}$ in the model parameters is given by the definition of the Gateaux (G-) differential, denoted as $\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathsf{\phi };\delta \mathsf{\psi };\delta \mathsf{\alpha }\right)$ of $R\left[\mathsf{\phi }\left(x\right),\mathsf{\psi }\left(x\right);\mathbf{\alpha }\right]$ around the nominal values $\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0\right)$. By definition, this G-differential is:

$$\begin{array}{l}\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathsf{\phi };\delta \mathsf{\psi };\delta \mathsf{\alpha }\right)\triangleq \frac{d}{d\epsilon }{\left\{R\left({\mathsf{\phi }}^0+\epsilon \delta \mathsf{\phi };{\mathsf{\psi }}^0+\epsilon \delta \mathsf{\psi };{\mathbf{\alpha }}^0+\epsilon \delta \mathbf{\alpha }\right)\right\}}_{\epsilon =0}\\ ={\left\{\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathbf{\alpha }\right)\right\}}_{dir}+{\left\{\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathsf{\phi };\delta \mathsf{\psi }\right)\right\}}_{ind},\end{array}$$

(13)

where the “direct-effect term” ${\left\{\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathbf{\alpha }\right)\right\}}_{dir}$ depends solely on the parameter variations $\delta \mathbf{\alpha }$ and is generally defined as follows:

$${\left\{\delta R\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\right\}}_{dir}\triangleq {\langle \left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathbf{\alpha }\right]\delta \mathsf{\alpha },1\rangle }_{\phi }$$

(14)

while the “indirect-effect term” ${\left\{\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathsf{\phi };\delta \mathsf{\psi }\right)\right\}}_{ind}$ depends solely on the variations $\delta \mathsf{\phi }$ and $\delta \mathsf{\psi }$ in the forward and, respectively, adjoint functions, and is generally defined as follows:

$${\left\{\delta R\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi }\right)\right\}}_{ind}\triangleq {\langle \left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]\delta \mathsf{\phi },1\rangle }_{\phi }+{\langle \left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]\delta \mathsf{\psi },1\rangle }_{\phi }.$$

(15)

Since the nominal values of the forward and adjoint functions are known after having solved Equations (9) through (12), it follows that the direct-effect term ${\left\{\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathbf{\alpha }\right)\right\}}_{dir}$ can already be computed at this stage. The indirect-effect term ${\left\{\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathsf{\phi };\delta \mathsf{\psi }\right)\right\}}_{ind}$ can be computed only after having determined the functions $\delta \mathsf{\phi }$ and $\delta \mathsf{\psi }$. These functions are obtained by solving the 1st-Level Forward Sensitivity System (1st-LFSS), which is obtained by G-differentiating the original forward and adjoint transport equations and respective boundary conditions given in Equations (1), (3), (6) and (7). Performing these differentiations yields the following 1st-LFSS:

$${\left\{\left(\begin{array}{cc}L\left(\mathbf{\alpha }\right)& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\right\}}_{\left({\mathbf{\alpha }}^0\right)}\left(\begin{array}{c}\delta \mathsf{\phi }\left(x\right)\\ \delta \mathsf{\psi }\left(x\right)\end{array}\right)={\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\end{array}\right)}_{\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0\right)},$$

(16)

together with the following boundary conditions:

$$\begin{array}{l}{\left\{B_F\left(\mathbf{\alpha }\right)\delta \mathsf{\phi }\left(x\right)\right\}}_{\left({\mathbf{\alpha }}^0\right)}={\left\{\partial \left[C_F\left(\mathbf{\alpha }\right)-B_F\left(\mathbf{\alpha }\right)\mathsf{\phi }\left(x\right)\right]/\partial \mathbf{\alpha }\right\}}_{\left({\mathsf{\phi }}^0;{\mathbf{\alpha }}^0\right)}\delta \mathsf{\alpha },x\in \partial {\Omega }_x,\\ {\left\{B_A\left(\mathbf{\alpha }\right)\delta \mathsf{\psi }\left(x\right)\right\}}_{\left({\mathbf{\alpha }}^0\right)}={\left\{\partial \left[C_A\left(\mathbf{\alpha }\right)-B_A\left(\mathbf{\alpha }\right)\mathsf{\psi }\left(x\right)\right]/\partial \mathbf{\alpha }\right\}}_{\left({\mathsf{\psi }}^0;{\mathbf{\alpha }}^0\right)}\delta \mathsf{\alpha },x\in \partial {\Omega }_x.\end{array}$$

(17)

The source-terms $Q_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathsf{\alpha }\right)$ and $Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathsf{\alpha }\right)$ in Equation (16) are defined as follows:

$$\begin{array}{l}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\mathbf{\delta }\mathbf{\alpha }\right)\triangleq \left\{\partial \left[Q\left[\mathbf{\alpha }\left(x\right)\right]-L\left(\mathbf{\alpha }\right)\mathsf{\phi }\left(x\right)\right]/\partial \mathbf{\alpha }\right\}\mathbf{\delta }\mathbf{\alpha }\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\mathbf{\delta }\mathbf{\alpha }\right)\triangleq \left\{\partial \left[Q^{+}\left[\mathbf{\alpha }\left(x\right)\right]-L^{+}\left(\mathbf{\alpha }\right)\mathsf{\psi }\left(x\right)\right]/\partial \mathbf{\alpha }\right\}\mathbf{\delta }\mathbf{\alpha },\end{array}$$

(18)

Solving the 1st-LFSS defined by Equations (16) and (17) is computationally expensive, since the 1st-LFSS would need to be solved anew for every variation $\delta {\alpha }_i,i=1,\dots ,N_{\alpha }$ in the model parameters, as each such variation would affect the source term on the right-side of Equation (16). The computationally expensive evaluation of the indirect-effect term by using the 1st-LFSS can be avoided by expressing this indirect-effect term ${\left\{\delta R\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0;\delta \mathsf{\phi };\delta \mathsf{\psi }\right)\right\}}_{ind}$ in terms of the solution of the 1st-Level Adjoint Sensitivity System (1st-LASS), which is constructed by implementing the following sequence of steps:

(i)

Consider two vector-valued functions $u^{\left(1\right)}\left(x\right)\triangleq {\left[u_1^{\left(1\right)}\left(x\right),u_2^{\left(1\right)}\left(x\right)\right]}^{†}$ and $v^{\left(1\right)}\left(x\right)\triangleq {\left[v_1^{\left(1\right)}\left(x\right),v_2^{\left(1\right)}\left(x\right)\right]}^{†}$, each having two $N_u$-dimensional vector-components defined as follows: $u_i^{\left(1\right)}\left(x\right)\triangleq {\left[u_{i,1}^{\left(1\right)}\left(x\right),\dots ,u_{i,j}^{\left(1\right)}\left(x\right),\dots ,u_{i,N_u}^{\left(1\right)}\left(x\right)\right]}^{†},i=1,2,$ and $v_i^{\left(1\right)}\left(x\right)\triangleq {\left[v_{i,1}^{\left(1\right)}\left(x\right),\dots ,v_{i,j}^{\left(1\right)}\left(x\right),\dots ,v_{i,N_u}^{\left(1\right)}\left(x\right)\right]}^{†},i=1,2,$. The components of these vectors are assumed to be square-integrable functions.

(ii)

Introduce a Hilbert space, denoted as ${\mathrm{H}}_{\left(1\right)}$, endowed with the following inner product, denoted as ${\langle u^{\left(1\right)}\left(x\right),v^{\left(1\right)}\left(x\right)\rangle }_{\left(1\right)}$, $u^{\left(1\right)}\left(x\right)\in {\mathrm{H}}_{\left(1\right)}$, $v^{\left(1\right)}\left(x\right)\in {\mathrm{H}}_{\left(1\right)}$, between the two functions defined in item (i), above:

$${\langle u^{\left(1\right)}\left(x\right),v^{\left(1\right)}\left(x\right)\rangle }_{\left(1\right)}\triangleq {\displaystyle \sum _{i=1}^2{\langle u_i^{\left(1\right)}\left(x\right),v_i^{\left(1\right)}\left(x\right)\rangle }_{\phi }}={\displaystyle \sum _{i=1}^2{\displaystyle \sum _{j=1}^{N_u}{\displaystyle \underset{{\Omega }_x}{\int }\left[u_{i,j}^{\left(1\right)}\left(x\right)\right]\left[v_{i,j}^{\left(1\right)}\left(x\right)\right]d}x}}$$

(19)

(iii)

In the Hilbert space ${\mathrm{H}}_{\left(1\right)}$, form the inner product of Equation (16) with a yet undefined vector-valued function ${\mathsf{\psi }}^{\left(1\right)}\left(x\right)\triangleq {\left[{\mathsf{\psi }}_1^{\left(1\right)}\left(x\right),{\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\right]}^{†}\in {\mathrm{H}}_{\left(1\right)}$ to obtain the following relation, evaluated at $\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0\right)$, in which the superscript “zero” is omitted to simplify the notation:

$${\langle {\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\\ {\mathsf{\psi }}_2^{\left(1\right)}\end{array}\right)}^{†},\left(\begin{array}{cc}L\left(\mathbf{\alpha }\right)& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\\ \delta \mathsf{\psi }\end{array}\right)\rangle }_{\left(1\right)}={\langle {\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\\ {\mathsf{\psi }}_2^{\left(1\right)}\end{array}\right)}^{†},\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\end{array}\right)\rangle }_{\left(1\right)}$$

(20)

(iv)

Use the definition of the adjoint operator in the Hilbert space ${\mathrm{H}}_{\left(1\right)}$ to recast the left-side of Equation (20) as follows:

$${\langle {\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\\ {\mathsf{\psi }}_2^{\left(1\right)}\end{array}\right)}^{†},\left(\begin{array}{cc}L\left(\mathbf{\alpha }\right)& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\\ \delta \mathsf{\psi }\end{array}\right)\rangle }_{\left(1\right)}={\langle {\left(\begin{array}{c}\delta \mathsf{\phi }\\ \delta \mathsf{\psi }\end{array}\right)}^{†},\left(\begin{array}{cc}{L}^{+}\left(\mathbf{\alpha }\right)& 0\\ 0& L\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\\ {\mathsf{\psi }}_2^{\left(1\right)}\end{array}\right)\rangle }_{\left(1\right)}+P^{\left(1\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)}\right]$$

(21)

where the bilinear concomitant $P^{\left(1\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)}\right]$ is defined on the phase-space boundary $x\in \partial {\Omega }_x$. The superscript “zero” denoting nominal values for the quantities $\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0\right)$ was also omitted in Equations (20) and (21), in order to simplify the notation. Omitting henceforth the superscript “zero” denoting nominal values for the quantities $\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathbf{\alpha }}^0\right)$ should not cause any loss of clarity, since all quantities are to be evaluated/computed using the respective nominal values of the model parameters and using the nominal values of the forward and adjoint functions evaluated at preceding stages/steps at nominal parameter values.

(v)

Identify the term on the left-side of Equation (21) with the indirect effect term defined in Equation (15), i.e., require that

$$\begin{array}{l}{\langle {\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\\ {\mathsf{\psi }}_2^{\left(1\right)}\end{array}\right)}^{†},\left(\begin{array}{cc}L\left(\mathbf{\alpha }\right)& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\\ \delta \mathsf{\psi }\end{array}\right)\rangle }_{\left(1\right)}={\langle \left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]\delta \mathsf{\phi },1\rangle }_{\phi }+{\langle \left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]\delta \mathsf{\psi },1\rangle }_{\phi }\\ ={\langle {\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\\ {\mathsf{\psi }}_2^{\left(1\right)}\end{array}\right)}^{†},\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\end{array}\right)\rangle }_{\left(1\right)}\end{array}$$

and use Equation (21) in conjunction with the boundary conditions given in Equation (17) on the above relation to construct the following 1st-Level Adjoint Sensitivity System (1st-LASS):

$$\left(\begin{array}{cc}{L}^{+}\left(\mathbf{\alpha }\right)& 0\\ 0& L\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\\ {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\end{array}\right)=\left(\begin{array}{c}{\left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†}\\ {\left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†}\end{array}\right).$$

(22)

(vi)

The boundary conditions given in Equation (17) are now implemented in Equation (21), thereby reducing by half the number of unknown boundary-values in the bilinear concomitant $P^{\left(1\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)}\right]$. The boundary conditions for the adjoint functions ${\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)$ and ${\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)$ are chosen next so as to eliminate the remaining unknown boundary-values of the functions $\delta \mathsf{\phi }$ and $\delta \mathsf{\psi }$ while ensuring that Equation (22) is well posed. The boundary conditions thus chosen for the adjoint functions ${\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)$ and ${\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)$ can be represented in operator form as follows:

$$B_A^{\left(1\right)}\left[\mathsf{\phi }\left(x\right);\mathsf{\psi }\left(x\right);{\mathsf{\psi }}_1^{\left(1\right)}\left(x\right),{\mathsf{\psi }}_2^{\left(1\right)}\left(x\right);\mathbf{\alpha }\right]=0,x\in \partial {\Omega }_x.$$

(23)

(vii)

In most cases, the above choice of boundary conditions for the 1st-level adjoint function ${\mathsf{\psi }}^{\left(1\right)}\left(x\right)$ will cause the bilinear concomitant $P^{\left(1\right)}\left[\delta \mathsf{\phi },\delta \mathsf{\psi },{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)}\right]$ in Equation (21) to vanish. When the boundary conditions for the original system are non-homogeneous, however, the bilinear concomitant $P^{\left(1\right)}\left[\delta \mathsf{\phi },\delta \mathsf{\psi },{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)}\right]$ may not vanish. Even when it does not vanish, however, this bilinear concomitant will be reduced to a quantity, denoted here as ${\widehat{P}}^{\left(1\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right]$, which will contain only known values of its arguments.

(viii)

Use the 1st-LASS defined by Equations (22) and (23) together with Equations (20) and (21) to obtain the following expression for the indirect-effect term defined in Equation (15), in terms of the adjoint functions ${\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)$ and ${\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)$:

$${\left\{\delta R\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\right\}}_{ind}={\langle {\left(\begin{array}{c}{\mathsf{\psi }}_1^{\left(1\right)}\\ {\mathsf{\psi }}_2^{\left(1\right)}\end{array}\right)}^{†},\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\end{array}\right)\rangle }_{\left(1\right)}-{\widehat{P}}^{\left(1\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right].$$

(24)

As indicated in Equation (22), the function ${\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)$ is computed by solving the original forward equation with the source $\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\psi }$ while the function ${\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)$ is obtained by solving the original adjoint equation with the source ${\left[\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†}$. Thus, after the 1st-LASS is solved to determine these two adjoint functions, the “indirect-effect term” is computed efficiently and exactly by simply performing the integrations (“quadratures”) indicated by the inner-product on the right-side of Equation (24).

Replacing Equations (24) and (14) in Equation (13) eliminates the appearance of the functions $\delta \mathsf{\phi }\left(x\right)$, $\delta \mathsf{\psi }\left(x\right)$ in the resulting expression. Consequently, the total 1st-order response sensitivity can be expressed in terms of the adjoint functions ${\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)$ and ${\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)$ as follows:

$$\begin{array}{l}\delta R\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)={\langle \frac{\partial S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathbf{\alpha }}\delta \mathbf{\alpha },1\rangle }_{\phi }+{\langle {\mathsf{\psi }}_1^{\left(1\right)},Q_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\rangle }_{\phi }\\ +{\langle {\mathsf{\psi }}_2^{\left(1\right)},Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\rangle }_{\phi }-{\widehat{P}}^{\left(1\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right]\triangleq {\displaystyle \sum _{i=1}^{N_{\mathbf{\alpha }}}\left\{\frac{\partial R\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i}\right\}\delta {\alpha }_i}.\end{array}$$

(25)

All of the quantities shown in Equation (25) are to be evaluated at the nominal values $\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathsf{\psi }}_1^{\left(1\right),0};{\mathsf{\psi }}_2^{\left(1\right),0};{\mathbf{\alpha }}^0\right)$. The partial 1st-order response sensitivities, denoted in Equation (25) as $\partial R\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\alpha }}_i$, $i=1,\dots ,N_{\mathsf{\alpha }},$ of the response $R\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)$ to a generic parameter ${\mathsf{\alpha }}_i$ are obtained by identifying the quantities that multiply the various parameter variations $\delta {\mathsf{\alpha }}_i$ in Equation (25).

3.2. The Second-Level Adjoint System (2nd-LASS) for Computing Exactly and Efficiently the Second-Order Model Response Sensitivities to Parameters

The second order sensitivities of the response $R\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)$ with respect to the parameters ${\mathsf{\alpha }}_i,i=1,\dots ,N_{\alpha }$, are obtained by determining the first-order G-differentials of the 1st-order sensitivities $\partial R\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\alpha }}_i$. For this purpose, it is convenient to use the notation $R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)\triangleq \partial R\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\alpha }}_i$. The first-order G-differential of $R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)$ is obtained, by definition, as follows:

$$\begin{array}{l}{\left\{\delta R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta \mathbf{\alpha }\right)\right\}}_{\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathsf{\psi }}_1^{\left(1\right),0};{\mathsf{\psi }}_2^{\left(1\right),0};{\mathbf{\alpha }}^0\right)}\\ \triangleq \frac{d}{d\epsilon }{\left\{R_i^{\left(1\right)}\left({\mathsf{\phi }}^0+\epsilon \delta \mathsf{\phi };{\mathsf{\psi }}^0+\epsilon \delta \mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right),0}+\epsilon \delta {\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right),0}+\epsilon \delta {\mathsf{\psi }}_2^{\left(1\right)};{\mathbf{\alpha }}^0+\epsilon \delta \mathbf{\alpha }\right)\right\}}_{\epsilon =0}\\ ={\left\{\frac{\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial \mathbf{\alpha }}\right\}}_{\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathsf{\psi }}^{\left(1\right),0};{\mathbf{\alpha }}^0\right)}\delta \mathsf{\alpha }+{\left\{\delta R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)}\right)\right\}}_{ind},\end{array}$$

(26)

where the “indirect-effect term” ${\left\{\delta R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)}\right)\right\}}_{ind}$ depends solely on the variations $\delta \mathsf{\phi }$, $\delta \mathsf{\psi }$, $\delta {\mathsf{\psi }}_1^{\left(1\right)}$ and $\delta {\mathsf{\psi }}_2^{\left(1\right)}$ in the forward and, respectively, adjoint functions and is defined as follows:

$$\begin{array}{l}{\left\{\delta R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)}\right)\right\}}_{ind}\\ \triangleq \{{\langle {\left[\frac{\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial \mathsf{\phi }}\right]}^{†},\delta \mathsf{\phi }\rangle }_{\phi }+{\langle {\left[\frac{\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial \mathsf{\psi }}\right]}^{†},\delta \mathsf{\psi }\rangle }_{\phi }\\ {+{\langle {\left[\frac{\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial {\mathsf{\psi }}_1^{\left(1\right)}}\right]}^{†},\delta {\mathsf{\psi }}_1^{\left(1\right)}\rangle }_{\phi }+{\langle {\left[\frac{\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial {\mathsf{\psi }}_2^{\left(1\right)}}\right]}^{†},\delta {\mathsf{\psi }}_2^{\left(1\right)}\rangle }_{\phi }\}}_{\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathsf{\psi }}_1^{\left(1\right),0};{\mathsf{\psi }}_2^{\left(1\right),0};{\mathbf{\alpha }}^0\right)}.\end{array}$$

(27)

The indirect-effect term defined in Equation (27) can be computed only after having obtained both the solution the functions $\delta \mathsf{\phi }$, $\delta \mathsf{\psi }$, $\delta {\mathsf{\psi }}_1^{\left(1\right)}$ and $\delta {\mathsf{\psi }}_2^{\left(1\right)}$. The functions $\delta \mathsf{\phi }$ and $\delta \mathsf{\psi }$ are the solutions of the 1st-LFSS defined by Equations (16). Furthermore, the functions $\delta {\mathsf{\psi }}_1^{\left(1\right)}$ and $\delta {\mathsf{\psi }}_2^{\left(1\right)}$ are the solutions of the following system of equations obtained by G-differentiating the 1st-LASS:

$$\begin{gathered} \begin{array}{c}{L}^{+}\left(\mathbf{\alpha }\right)\delta {\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }}\delta \mathsf{\phi }\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }}\delta \mathsf{\psi }\left(x\right)=Q_1^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ \\ L\left(\mathbf{\alpha }\right)\delta {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }}\delta \mathsf{\phi }\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }}\delta \mathsf{\psi }\left(x\right)=Q_2^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\end{array}\} \end{gathered}$$

(28)

$$\begin{array}{l}\delta B_A^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)\triangleq \frac{\partial B_A^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial \mathsf{\phi }}\delta \mathsf{\phi }+\frac{\partial B_A^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial \mathsf{\psi }}\delta \mathsf{\psi }\\ +\frac{\partial B_A^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial {\mathsf{\psi }}_1^{\left(1\right)}}\delta {\mathsf{\psi }}_1^{\left(1\right)}+\frac{\partial B_A^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial {\mathsf{\psi }}_2^{\left(1\right)}}\delta {\mathsf{\psi }}_2^{\left(1\right)}\\ +\frac{\partial B_A^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial \mathbf{\alpha }}\delta \mathbf{\alpha }=0,x\in \partial {\Omega }_x.\end{array}$$

(29)

where:

$$Q_1^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\triangleq \left\{\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathbf{\alpha }}-\frac{\partial \left[L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\right]}{\partial \mathbf{\alpha }}\right\}\delta \mathbf{\alpha },$$

(30)

$$Q_2^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\triangleq \left\{\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathbf{\alpha }}-\frac{\partial \left[L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\right]}{\partial \mathbf{\alpha }}\right\}\delta \mathbf{\alpha }.$$

(31)

The system comprising Equations (16) and (28), together with the boundary conditions provided in Equations (17) and (29) constitutes the 2nd-Level Forward Sensitivity System (2nd-LFSS). Since the source-terms of the 2nd-LFSS depend on the parameters variations $\delta {\mathsf{\alpha }}_i$, it follows that the determination of the functions $\delta {\mathsf{\psi }}_1^{\left(1\right)}$ and $\delta {\mathsf{\psi }}_2^{\left(1\right)}$ is at least as expensive computationally as determining the functions $\delta \mathsf{\phi }$ and $\delta \mathsf{\psi }$ by solving the 1st-LFSS. To avoid the need for solving the 2nd-LFSS, the indirect-effect term defined in Equation (27) will be expressed in terms of a 2nd-Level Adjoint Sensitivity System (2nd-LASS), which will be constructed by following the general principles introduced by Cacuci [1,2,3], comprising the following sequence of steps:

(i) Define a Hilbert space, denoted as ${\mathrm{H}}_{\left(2\right)}$, having vector-valued elements of the form the $u^{\left(2\right)}\left(x\right)\triangleq {\left[u_1^{\left(2\right)}\left(x\right),u_2^{\left(2\right)}\left(x\right),u_3^{\left(2\right)}\left(x\right),u_4^{\left(2\right)}\left(x\right)\right]}^{†}\in {\mathrm{H}}_{\left(2\right)}$, with components that are $N_u$-dimensional vectors of the form $u_i^{\left(2\right)}\left(x\right)\triangleq {\left[u_{i,1}^{\left(2\right)}\left(x\right),\dots ,u_{i,j}^{\left(2\right)}\left(x\right),\dots ,u_{i,N_u}^{\left(2\right)}\left(x\right)\right]}^{†},i=1,\dots ,4,$ with square-integrable components $u_{i,j}^{\left(2\right)}\left(x\right)$. In ${\mathrm{H}}_{\left(2\right)}$, define the inner-product, denoted as ${\langle u^{\left(2\right)}\left(x\right),v^{\left(2\right)}\left(x\right)\rangle }_{\left(2\right)}$, of two functions $u^{\left(2\right)}\left(x\right)\in {\mathrm{H}}_{\left(2\right)}$ and $v^{\left(2\right)}\left(x\right)\in {\mathrm{H}}_{\left(2\right)}$ as follows:

$${\langle u^{\left(2\right)}\left(x\right),v^{\left(2\right)}\left(x\right)\rangle }_{\left(2\right)}\triangleq {\displaystyle \sum _{i=1}^4{\langle u_i^{\left(2\right)}\left(x\right),v_i^{\left(2\right)}\left(x\right)\rangle }_{\phi }}={\displaystyle \sum _{i=1}^4{\displaystyle \sum _{j=1}^{N_u}{\displaystyle \underset{{\Omega }_x}{\int }{u}_{i,j}^{\left(2\right)}\left(x\right)v_{i,j}^{\left(2\right)}\left(x\right)dx}}}.$$

(32)

Using the definition provided in Equation (32), construct the inner product of a vector ${\mathsf{\psi }}_i^{\left(2\right)}\left(x\right)\triangleq {\left[{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\right]}^{†}\in {\mathrm{H}}_{\left(2\right)}$ with Equations (16) and (28) to obtain the following relation:

$$\begin{array}{l}{\langle {\left(\begin{array}{c}{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\end{array}\right)}^{†},\left(\begin{array}{cccc}L\left(\mathbf{\alpha }\right)& 0& 0& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)& 0& 0\\ -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }}& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }}& L^{+}\left(\mathbf{\alpha }\right)& 0\\ -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }}& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }}& 0& L\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\left(x\right)\\ \delta \mathsf{\psi }\left(x\right)\\ \delta {\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\end{array}\right)\rangle }_{\left(2\right)}=\\ ={\langle {\left(\begin{array}{c}{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\end{array}\right)}^{†},\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_1^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\end{array}\right)\rangle }_{\left(2\right)}.\end{array}$$

(33)

(ii)

Use the definition of the adjoint operator in the Hilbert space ${\mathrm{H}}_{\left(2\right)}$ to recast the left-side of Equation (33) as follows:

$$\begin{array}{l}{\langle {\left(\begin{array}{c}{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\end{array}\right)}^{†},\left(\begin{array}{cccc}L\left(\mathbf{\alpha }\right)& 0& 0& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)& 0& 0\\ -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }}& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }}& L^{+}\left(\mathbf{\alpha }\right)& 0\\ -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }}& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }}& 0& L\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\left(x\right)\\ \delta \mathsf{\psi }\left(x\right)\\ \delta {\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\end{array}\right)\rangle }_{\left(2\right)}=\\ ={\langle {\left(\begin{array}{c}\delta \mathsf{\phi }\left(x\right)\\ \delta \mathsf{\psi }\left(x\right)\\ \delta {\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\end{array}\right)}^{†},\left(\begin{array}{cccc}{L}^{+}\left(\mathbf{\alpha }\right)& 0& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }}& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }}\\ 0& L\left(\mathbf{\alpha }\right)& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }}& -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }}\\ 0& 0& L\left(\mathbf{\alpha }\right)& 0\\ 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\end{array}\right)\rangle }_{\left(2\right)}\\ +P^{\left(2\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\right],\end{array}$$

(34)

where the bilinear concomitant $P^{\left(2\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\right]$ is defined on the phase-space boundary $x\in \partial {\Omega }_x$.

(iii)

Identify the first term on the right-side of Equation (34) with the indirect-effect term defined in Equation (27) by requiring that the following system of equations be satisfied for $i=1,\dots N_{\mathsf{\alpha }}$:

$$\begin{array}{l}{L}^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)={\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†}\\ L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)={\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†}\\ L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)={\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†}\\ L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)={\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†}\end{array}$$

(35)

(iv)

The boundary conditions given in Equations (17) and (29) are now implemented in Equation (34), thereby reducing by half the number of unknown boundary-values in the bilinear concomitant $P^{\left(2\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\right]$. The boundary conditions for the 2nd-level adjoint functions ${\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)$ are now chosen so as to eliminate the remaining unknown boundary-values of the functions $\delta \mathsf{\phi }$, $\delta \mathsf{\psi }$, $\delta {\mathsf{\psi }}_1^{\left(1\right)}$ and $\delta {\mathsf{\psi }}_2^{\left(1\right)}$ while ensuring that Equation (35) is well posed. The boundary conditions thus chosen for the adjoint functions ${\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)$ and ${\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)$ can be represented in operator form as follows:

$$B_A^{\left(2\right)}\left[\mathsf{\phi }\left(x\right);\mathsf{\psi }\left(x\right);{\mathsf{\psi }}_1^{\left(1\right)}\left(x\right);{\mathsf{\psi }}_2^{\left(1\right)}\left(x\right);{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right);{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right);{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right);{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]=0,x\in \partial {\Omega }_x.$$

(36)

In most cases, the above choice of boundary conditions for the 1st-level adjoint function ${\mathsf{\psi }}^{\left(1\right)}\left(x\right)$ will cause the bilinear concomitant $P^{\left(2\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\right]$ in Equation (34) to vanish. Even when it does not vanish, however, this bilinear concomitant will be reduced to a quantity, denoted here as ${\widehat{P}}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha };\delta \mathbf{\alpha }\right]$, which will contain only known values of its arguments.

The system of equations comprising Equations (35) and (36) will be called the 2nd-Level Adjoint Sensitivity System (2nd-LASS) for the 2nd-level adjoint functions ${\mathsf{\psi }}_{1,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,j}^{\left(2\right)}\left(x\right)$, $i=1,\dots N_{\mathsf{\alpha }}$. These 2nd-level adjoint functions are obtained by solving the 2nd-LASS successively by using two “forward” and two “adjoint” computations, for each of the imprecisely known scalar model parameters.

(v)  Use the 2nd-LASS defined by Equations (35) and (36) together with Equations (33)–(35) to obtain the following expression for the indirect-effect term defined in Equation (27), in terms of the 2nd-level adjoint functions ${\mathsf{\psi }}_{1,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,j}^{\left(2\right)}\left(x\right)$

$$\begin{array}{l}{\left\{\delta R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha };\delta \mathbf{\alpha }\right)\right\}}_{ind}\\ ={\langle {\left(\begin{array}{c}{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\\ {\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\end{array}\right)}^{†},\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_1^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\end{array}\right)\rangle }_{\left(2\right)}\\ -{\widehat{P}}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha };\delta \mathbf{\alpha }\right].\end{array}$$

(37)

As Equation (37) indicates, the indirect-effect term can be computed speedily by quadratures once the 2nd-level adjoint functions ${\mathsf{\psi }}_{1,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,j}^{\left(2\right)}\left(x\right)$ become available.

(vi)  Replace Equation (37) in Equation (26) to obtain the following expression for the total 2nd-order response sensitivity to model parameters:

$$\begin{array}{l}\delta R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta \mathbf{\alpha }\right)=\\ \triangleq {\displaystyle \sum _{j=1}^{N_{\mathsf{\alpha }}}{R}_{ij}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]\delta {\alpha }_j},i=1,\dots ,N_{\alpha }.\end{array}$$

(38)

where $R_{ij}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]\equiv {\partial }^{2}R/\partial {\alpha }_i\partial {\alpha }_j;i,j=1,\dots ,N_{\alpha }$ denotes the 2nd-order partial sensitivity of the response to the model parameters and is defined as follows:

$$\begin{array}{l}{R}_{ij}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]\triangleq \frac{\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)}{\partial {\alpha }_j}\\ +{\langle {\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),\partial \left[Q\left[\mathbf{\alpha }\left(x\right)\right]-L\left(\mathbf{\alpha }\right)\mathsf{\phi }\left(x\right)\right]/\partial {\alpha }_j\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),\partial \left[Q^{+}\left[\mathbf{\alpha }\left(x\right)\right]-L^{+}\left(\mathbf{\alpha }\right)\mathsf{\psi }\left(x\right)\right]/\partial {\alpha }_j\rangle }_{\phi }\\ +{\langle {\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial {\alpha }_j}-\frac{\partial \left[L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\right]}{\partial {\alpha }_j}\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right),\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial {\alpha }_j}-\frac{\partial \left[L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\right]}{\partial {\alpha }_j}\rangle }_{\phi }\\ -\frac{\partial {\widehat{P}}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]}{\partial {\alpha }_j},i=1,\dots ,N_{\mathbf{\alpha }};j=1,\dots ,i.\end{array}$$

(39)

Note that the 2nd-LASS is independent of parameter variations $\delta \mathbf{\alpha }$. Thus, the exact computation of all of the partial second-order sensitivities, $R_{ij}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]$, $i,j=1,\dots ,N_{\alpha },$ requires at most $N_{\alpha }$ large-scale (adjoint) computations using the 2nd-LASS, rather than $O\left(N_{\mathbf{\alpha }}^2\right)$ large-scale computations as would be required by forward methods. It is also important to note that by solving the 2nd-LASS $N_{\mathsf{\alpha }}$-times, the “off-diagonal” 2nd-order mixed sensitivities $R_{ij}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]$ will be computed twice, in two different ways (using distinct 2nd-level adjoint functions), thereby providing an independent intrinsic (numerical) verification that the 1st- and 2nd-order sensitivities are computed accurately. In practice, it is useful to prioritize the computation of the 2nd-order sensitivities by using the rankings of the relative magnitudes of the 1st-order sensitivities as a “priority indicator”: the larger the magnitude of the relative 1st-order sensitivity, the higher the priority for computing the corresponding 2nd-order sensitivities. Also, since vanishing 1st-order sensitivities may indicate critical points of the response in the phase-space of model parameters, it is also of interest to compute the 2nd-order sensitivities that correspond to vanishing 1st-order sensitivities. Thus, only the 2nd-order partial sensitivities of the response $R\left[\mathsf{\phi }\left(x\right),\mathsf{\psi }\left(x\right);\mathbf{\alpha }\right]$ which are deemed important will need to be computed. Information provided by the 1st-order sensitivities might indicate which 2nd-order sensitivities could be neglected.

3.3. The Third-Level Adjoint System (3rd-LASS) for Computing Exactly and Efficiently the Third-Order Model Response Sensitivities to Parameters

The third-order sensitivities of the response $R\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)$ with respect to the model parameters ${\mathsf{\alpha }}_i,i=1,\dots ,N_{\mathsf{\alpha }}$, are obtained by determining the first-order G-differential, $\delta R_{ij}^{\left(2\right)}\left[{\mathsf{\phi }}^0;{\mathsf{\psi }}^0;{\mathsf{\psi }}_1^{\left(1\right),0};{\mathsf{\psi }}_2^{\left(1\right),0};{\mathsf{\psi }}_{1,i}^{\left(2\right),0}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right),0}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right),0}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right),0}\left(x\right);{\mathbf{\alpha }}^0\right]$, of the 2nd-order sensitivities $R_{ij}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);\mathbf{\alpha }\right]$, computed in Section 3.2, which is given by the following expression:

$$\begin{array}{l}\delta R_{ij}^{\left(2\right)}[{\mathsf{\phi }}^0\left(x\right);{\mathsf{\psi }}^0\left(x\right);{\mathsf{\psi }}_1^{\left(1\right),0}\left(x\right);{\mathsf{\psi }}_2^{\left(1\right),0}\left(x\right);{\mathsf{\psi }}_{1,i}^{\left(2\right),0}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(2\right),0}\left(x\right),{\mathsf{\psi }}_{3,i}^{\left(2\right),0}\left(x\right),{\mathsf{\psi }}_{4,i}^{\left(2\right),0}\left(x\right);{\mathbf{\alpha }}^0;\\ \delta \mathsf{\phi }\left(x\right);\delta \mathsf{\psi }\left(x\right);\delta {\mathsf{\psi }}_1^{\left(1\right)}\left(x\right);\delta {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right);\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),\delta {\mathsf{\psi }}_{2,i}^{\left(1\right)},\delta {\mathsf{\psi }}_{3,i}^{\left(1\right),0},\delta {\mathsf{\psi }}_{4,i}^{\left(1\right),0};\delta \mathbf{\alpha }]\\ \triangleq \frac{d}{d\epsilon }\{R_{ij}^{\left(2\right)}[{\mathsf{\phi }}^0\left(x\right)+\epsilon \delta \mathsf{\phi }\left(x\right);{\mathsf{\psi }}^0\left(x\right)+\epsilon \delta \mathsf{\psi }\left(x\right);{\mathsf{\psi }}_1^{\left(1\right),0}\left(x\right)+\epsilon \delta {\mathsf{\psi }}_1^{\left(1\right)}\left(x\right);{\mathsf{\psi }}_2^{\left(1\right),0}\left(x\right)+\epsilon \delta {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right);\\ {{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)+\epsilon \delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,i}^{\left(1\right),0}\left(x\right)+\epsilon \delta {\mathsf{\psi }}_{2,i}^{\left(1\right)},{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)+\epsilon \delta {\mathsf{\psi }}_{3,i}^{\left(1\right),0},{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)+\epsilon \delta {\mathsf{\psi }}_{4,i}^{\left(1\right),0};{\mathbf{\alpha }}^0+\epsilon \delta \mathbf{\alpha }]\}}_{\epsilon =0}\\ ={\left\{\frac{\partial R_{ij}^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };\dots ;\mathbf{\alpha }\right)}{\partial \mathbf{\alpha }}\right\}}_{\left({\mathsf{\phi }}^0;{\mathsf{\psi }}^0;\dots ;{\mathbf{\alpha }}^0\right)}\delta \mathbf{\alpha }\\ +{\left\{\delta R_{ij}^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\right)\right\}}_{ind},i=1,\dots ,N_{\mathbf{\alpha }};j=1,\dots ,i.\end{array}$$

(40)

The quantity ${\left\{\delta R_{ij}^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\right)\right\}}_{ind}$ denotes the “indirect-effect term”, which depends solely on the variations $\left(\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\right)$ in the forward and adjoint functions, respectively, and is defined as follows:

$$\begin{array}{l}{\left\{\delta R_{ij}^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};\mathbf{\alpha };\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)};\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\right)\right\}}_{ind}\\ \triangleq \{{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\mathsf{\phi }\dots \mathbf{\alpha }\right)}{\partial \mathsf{\phi }}\right]}^{†},\delta \mathsf{\phi }\rangle }_{\phi }+{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi }\dots \mathbf{\alpha }\right)}{\partial \mathsf{\psi }}\right]}^{†},\delta \mathsf{\psi }\rangle }_{\phi }+{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\dots ;{\mathsf{\psi }}_1^{\left(1\right)}\dots \mathbf{\alpha }\right)}{\partial {\mathsf{\psi }}_1^{\left(1\right)}}\right]}^{†},\delta {\mathsf{\psi }}_1^{\left(1\right)}\rangle }_{\phi }\\ +{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\dots ;{\mathsf{\psi }}_2^{\left(1\right)}\dots \right)}{\partial {\mathsf{\psi }}_2^{\left(1\right)}}\right]}^{†},\delta {\mathsf{\psi }}_2^{\left(1\right)}\rangle }_{\phi }+{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\dots ;{\mathsf{\psi }}_{1,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{1,i}^{\left(2\right)}}\right]}^{†},\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\rangle }_{\phi }+{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\dots ;{\mathsf{\psi }}_{2,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{1,i}^{\left(2\right)}}\right]}^{†},\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\rangle }_{\phi }\\ {+{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\dots ;{\mathsf{\psi }}_{3,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{3,i}^{\left(2\right)}}\right]}^{†},\delta {\mathsf{\psi }}_{3,i}^{\left(2\right)}\rangle }_{\phi }+{\langle {\left[\frac{\partial R_{i,j}^{\left(2\right)}\left(\dots ;{\mathsf{\psi }}_{4,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{4,i}^{\left(2\right)}}\right]}^{†},\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\rangle }_{\phi }\}}_{\left({\mathsf{\phi }}^0;\dots ;{\mathbf{\alpha }}^0\right)},i=1,\dots ,N_{\mathbf{\alpha }};j=1,\dots ,i.\end{array}$$

(41)

The indirect-effect term defined in Equation (41) can be computed only after having computed the functions $\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{2,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{3,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)},$ in addition to the functions $\delta \mathsf{\phi }$, $\delta \mathsf{\psi }$, $\delta {\mathsf{\psi }}_1^{\left(1\right)}$ and $\delta {\mathsf{\psi }}_2^{\left(1\right)}$. Altogether, the functions $\delta \mathsf{\phi }$, $\delta \mathsf{\psi }$, $\delta {\mathsf{\psi }}_1^{\left(1\right)}$, $\delta {\mathsf{\psi }}_2^{\left(1\right)}$ $\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{2,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{3,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)},$ are the solutions of Equation (28), augmented by the solutions of the system of equations obtained by G-differentiating the 2nd-LASS, which can be written in matrix form as follows:

$$\left(\begin{array}{cccccccc}L\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0& 0& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0& 0\\ F_{31}^{\left(2\right)}& F_{32}^{\left(2\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0\\ F_{41}^{\left(2\right)}& F_{42}^{\left(2\right)}& 0& L\left(\mathbf{\alpha }\right)& 0& 0& 0& 0\\ F_{51}^{\left(3\right)}& F_{52}^{\left(3\right)}& F_{53}^{\left(3\right)}& F_{54}^{\left(3\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0& F_{57}^{\left(3\right)}& F_{58}^{\left(3\right)}\\ F_{61}^{\left(3\right)}& F_{62}^{\left(3\right)}& F_{63}^{\left(3\right)}& F_{64}^{\left(3\right)}& 0& L\left(\mathbf{\alpha }\right)& F_{67}^{\left(3\right)}& F_{68}^{\left(3\right)}\\ F_{71}^{\left(3\right)}& F_{72}^{\left(3\right)}& F_{73}^{\left(3\right)}& F_{74}^{\left(3\right)}& 0& 0& L\left(\mathbf{\alpha }\right)& 0\\ F_{81}^{\left(3\right)}& F_{82}^{\left(3\right)}& F_{83}^{\left(3\right)}& F_{84}^{\left(3\right)}& 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\left(x\right)\\ \delta \mathsf{\psi }\left(x\right)\\ \delta {\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\\ \delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\end{array}\right)=\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_1^{\left(2\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_2^{\left(2\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{1,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{2,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{3,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{4,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\end{array}\right),$$

(42)

where:

$$F_{31}^{\left(2\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }},F_{32}^{\left(2\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }};F_{41}^{\left(2\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }};F_{42}^{\left(2\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }};$$

(43)

$$F_{51}^{\left(3\right)}\triangleq -\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)-\frac{\partial }{\partial \mathsf{\phi }}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†};$$

(44)

$$F_{52}^{\left(3\right)}\triangleq -\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)-\frac{\partial {\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†}}{\partial \mathsf{\psi }};$$

(45)

$$F_{53}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_1^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†};F_{54}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_2^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†};$$

(46)

$$F_{55}^{\left(3\right)}\triangleq L^{+}\left(\mathbf{\alpha }\right);F_{56}^{\left(3\right)}\equiv 0;F_{57}^{\left(3\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }};F_{58}^{\left(3\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }};$$

(47)

$$F_{61}^{\left(3\right)}\triangleq -\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)-\frac{\partial }{\partial \mathsf{\phi }}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†};$$

(48)

$$F_{62}^{\left(3\right)}\triangleq -\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{3}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)-\frac{\partial }{\partial \mathsf{\psi }}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†};$$

(49)

$$F_{63}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_1^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†};F_{64}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_2^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†};$$

(50)

$$F_{65}^{\left(3\right)}\equiv 0;F_{66}^{\left(3\right)}\triangleq L\left(\mathbf{\alpha }\right);F_{67}^{\left(3\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }};F_{68}^{\left(3\right)}\triangleq -\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }};$$

(51)

$$F_{71}^{\left(3\right)}\triangleq -\frac{\partial }{\partial \mathsf{\phi }}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†};F_{72}^{\left(3\right)}\triangleq -\frac{\partial }{\partial \mathsf{\psi }}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†};$$

(52)

$$F_{73}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_1^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†};F_{73}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_2^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†};$$

(53)

$$F_{75}^{\left(3\right)}\equiv 0;F_{76}^{\left(3\right)}\equiv 0;F_{77}^{\left(3\right)}\triangleq L\left(\mathbf{\alpha }\right);F_{78}^{\left(3\right)}\equiv 0;$$

(54)

$$F_{81}^{\left(3\right)}\triangleq -\frac{\partial }{\partial \mathsf{\phi }}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†};F_{82}^{\left(3\right)}\triangleq -\frac{\partial }{\partial \mathsf{\psi }}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†};$$

(55)

$$F_{83}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_1^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†};F_{84}^{\left(3\right)}\triangleq -\frac{\partial }{\partial {\mathsf{\psi }}_2^{\left(1\right)}}{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†};$$

(56)

$$F_{85}^{\left(3\right)}\equiv 0;F_{66}^{\left(3\right)}\equiv 0;F_{87}^{\left(3\right)}\equiv 0;F_{88}^{\left(3\right)}\triangleq L^{+}\left(\mathbf{\alpha }\right);$$

(57)

$$\begin{array}{l}{Q}_{1,i}^{\left(3\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)};\dots ;{\mathsf{\psi }}_{4,i}^{\left(2\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\triangleq \frac{\partial }{\partial \mathbf{\alpha }}{\{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†}\\ -L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\}\delta \mathbf{\alpha },\end{array}$$

(58)

$$\begin{array}{l}{Q}_{2,i}^{\left(3\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)};\dots ;{\mathsf{\psi }}_{4,i}^{\left(2\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\triangleq \frac{\partial }{\partial \mathbf{\alpha }}{\{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†}\\ -L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\}\delta \mathbf{\alpha },\end{array}$$

(59)

$$\begin{array}{l}{Q}_{3,i}^{\left(3\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)};\dots ;{\mathsf{\psi }}_{4,i}^{\left(2\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ \triangleq \frac{\partial }{\partial \mathbf{\alpha }}\left\{{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†}-L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\right\}\delta \mathbf{\alpha },\end{array}$$

(60)

$$\begin{array}{l}{Q}_{4,i}^{\left(3\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)};\dots ;{\mathsf{\psi }}_{4,i}^{\left(2\right)};\mathbf{\alpha };\delta \mathsf{\alpha }\right)\\ \triangleq \frac{\partial }{\partial \mathbf{\alpha }}\left\{{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†}-L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\right\}\delta \mathsf{\alpha }.\end{array}$$

(61)

The boundary conditions for the functions $\delta \mathsf{\phi }$, $\delta \mathsf{\psi }$, $\delta {\mathsf{\psi }}_1^{\left(1\right)}$, $\delta {\mathsf{\psi }}_2^{\left(1\right)}$ $\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{2,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{3,i}^{\left(2\right)},\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)},$ are those provided in Equations (17) and (29), augmented by the boundary conditions obtained by G-differentiating Equation (36), i.e.:

$$\begin{array}{l}\delta B_A^{\left(2\right)}\left(\mathsf{\phi }\dots \mathbf{\alpha }\right)\triangleq \frac{\partial B_A^{\left(2\right)}\left(\mathsf{\phi }\dots \right)}{\partial \mathsf{\phi }}\delta \mathsf{\phi }+\frac{\partial B_A^{\left(2\right)}\left(\dots \mathsf{\psi }\dots \right)}{\partial \mathsf{\psi }}\delta \mathsf{\psi }+\frac{\partial B_A^{\left(2\right)}\left(\dots {\mathsf{\psi }}_1^{\left(1\right)}\dots \right)}{\partial {\mathsf{\psi }}_1^{\left(1\right)}}\delta {\mathsf{\psi }}_1^{\left(1\right)}\\ +\frac{\partial B_A^{\left(2\right)}\left(\dots {\mathsf{\psi }}_2^{\left(1\right)}\dots \right)}{\partial {\mathsf{\psi }}_2^{\left(1\right)}}\delta {\mathsf{\psi }}_2^{\left(1\right)}+\frac{\partial B_A^{\left(2\right)}\left(\dots {\mathsf{\psi }}_{1,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{1,i}^{\left(2\right)}}\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}+\frac{\partial B_A^{\left(2\right)}\left(\dots {\mathsf{\psi }}_{2,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{2,i}^{\left(2\right)}}\delta {\mathsf{\psi }}_{2,i}^{\left(2\right)}\\ +\frac{\partial B_A^{\left(2\right)}\left(\dots {\mathsf{\psi }}_{3,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{3,i}^{\left(2\right)}}\delta {\mathsf{\psi }}_{3,i}^{\left(2\right)}+\frac{\partial B_A^{\left(2\right)}\left(\dots {\mathsf{\psi }}_{4,i}^{\left(2\right)}\dots \right)}{\partial {\mathsf{\psi }}_{4,i}^{\left(2\right)}}\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}+\frac{\partial B_A^{\left(1\right)}\left(\dots \mathbf{\alpha }\right)}{\partial \mathbf{\alpha }}\delta \mathsf{\alpha }=0,x\in \partial {\Omega }_x.\end{array}$$

(62)

The system comprising the 2nd-LFSS together with Equation (42) and the boundary conditions provided in Equation (62) is called the 3rd-Level Forward Sensitivity System (3rd-LFSS). Since the source-terms of the 3rd-LFSS depend on the parameters variations $\delta {\mathbf{\alpha }}_i$, it follows that solving this system of equations is prohibitively expensive computationally. To avoid the need for solving the 3rd-LFSS, the indirect-effect term defined in Equation (41) will be expressed in terms of a 3rd-Level Adjoint Sensitivity System (3rd-LASS), which will be constructed by implementing the following sequence of steps:

(i) Define a Hilbert space, denoted as ${\mathrm{H}}_{\left(3\right)}$, having vector-valued elements of the form the $u^{\left(3\right)}\left(x\right)\triangleq {\left[u_1^{\left(3\right)}\left(x\right),\dots ,u_8^{\left(3\right)}\left(x\right)\right]}^{†}\in {\mathrm{H}}_{\left(3\right)}$, with components that are $N_{\mathsf{\alpha }}$-dimensional vectors of the form $u_i^{\left(3\right)}\left(x\right)\triangleq {\left[u_{i,1}^{\left(3\right)}\left(x\right),\dots ,u_{i,j}^{\left(3\right)}\left(x\right),\dots ,u_{i,N_u}^{\left(3\right)}\left(x\right)\right]}^{†},i=1,\dots ,8,$ with square-integrable components $u_{i,j}^{\left(3\right)}\left(x\right)$. In ${\mathrm{H}}_{\left(3\right)}$, define the inner-product, denoted as ${\langle u^{\left(3\right)}\left(x\right),v^{\left(3\right)}\left(x\right)\rangle }_{\left(3\right)}$, of two functions $u^{\left(3\right)}\left(x\right)\in {\mathrm{H}}_{\left(3\right)}$ and $v^{\left(3\right)}\left(x\right)\in {\mathrm{H}}_{\left(3\right)}$ as follows:

$${\langle u^{\left(3\right)}\left(x\right),v^{\left(3\right)}\left(x\right)\rangle }_{\left(3\right)}\triangleq {\displaystyle \sum _{i=1}^8{\langle u_i^{\left(3\right)}\left(x\right),v_i^{\left(3\right)}\left(x\right)\rangle }_{\phi }}={\displaystyle \sum _{i=1}^8{\displaystyle \sum _{j=1}^{N_u}{\displaystyle \underset{{\Omega }_x}{\int }{u}_{i,j}^{\left(3\right)}\left(x\right)v_{i,j}^{\left(3\right)}\left(x\right)dx}}}.$$

(63)

Using the definition provided in Equation (63), construct the inner product of a vector ${\mathsf{\psi }}_{ij}^{\left(3\right)}\left(x\right)\triangleq {\left[{\mathsf{\psi }}_{1,ij}^{\left(3\right)}\left(x\right),\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}\left(x\right)\right]}^{†}\in {\mathrm{H}}_{\left(3\right)}$ with Equations (16) and (28) to obtain the following relation:

$$\begin{array}{l}{\langle {\left(\begin{array}{c}{\mathsf{\psi }}_{1,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{2,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{3,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{4,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{5,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{6,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{7,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{8,ij}^{\left(3\right)}\end{array}\right)}^{†},\left(\begin{array}{cccccccc}L\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0& 0& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0& 0\\ F_{31}^{\left(2\right)}& F_{32}^{\left(2\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0\\ F_{41}^{\left(2\right)}& F_{42}^{\left(2\right)}& 0& L\left(\mathbf{\alpha }\right)& 0& 0& 0& 0\\ F_{51}^{\left(3\right)}& F_{52}^{\left(3\right)}& F_{53}^{\left(3\right)}& F_{54}^{\left(3\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0& F_{57}^{\left(3\right)}& F_{58}^{\left(3\right)}\\ F_{61}^{\left(3\right)}& F_{62}^{\left(3\right)}& F_{63}^{\left(3\right)}& F_{64}^{\left(3\right)}& 0& L\left(\mathbf{\alpha }\right)& F_{67}^{\left(3\right)}& F_{68}^{\left(3\right)}\\ F_{71}^{\left(3\right)}& F_{72}^{\left(3\right)}& F_{73}^{\left(3\right)}& F_{74}^{\left(3\right)}& 0& 0& L\left(\mathbf{\alpha }\right)& 0\\ F_{81}^{\left(3\right)}& F_{82}^{\left(3\right)}& F_{83}^{\left(3\right)}& F_{84}^{\left(3\right)}& 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\\ \delta \mathsf{\psi }\\ \delta {\mathsf{\psi }}_1^{\left(1\right)}\\ \delta {\mathsf{\psi }}_2^{\left(1\right)}\\ \delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{2,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{3,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\end{array}\right)\rangle }_{\left(3\right)}=\\ ={\langle {\left(\begin{array}{c}{\mathsf{\psi }}_{1,ij}^{\left(3\right)}\left(x\right)\\ {\mathsf{\psi }}_{2,ij}^{\left(3\right)}\left(x\right)\\ {\mathsf{\psi }}_{3,ij}^{\left(3\right)}\left(x\right)\\ {\mathsf{\psi }}_{4,ij}^{\left(3\right)}\left(x\right)\\ {\mathsf{\psi }}_{5,ij}^{\left(3\right)}\left(x\right)\\ {\mathsf{\psi }}_{6,ij}^{\left(3\right)}\left(x\right)\\ {\mathsf{\psi }}_{7,ij}^{\left(3\right)}\left(x\right)\\ {\mathsf{\psi }}_{8,ij}^{\left(3\right)}\left(x\right)\end{array}\right)}^{†},\left(\begin{array}{c}{Q}_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\\ Q_1^{\left(2\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_2^{\left(2\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{1,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{2,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{3,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\\ Q_{4,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\end{array}\right)\rangle }_{\left(3\right)}.\end{array}$$

(64)

(ii)

Use the definition of the adjoint operator in the Hilbert space ${\mathrm{H}}_{\left(3\right)}$ to recast the left-side of Equation (64) as follows:

$$\begin{array}{l}{\langle {\left(\begin{array}{c}{\mathsf{\psi }}_{1,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{2,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{3,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{4,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{5,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{6,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{7,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{8,ij}^{\left(3\right)}\end{array}\right)}^{†},\left(\begin{array}{cccccccc}L\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0& 0& 0\\ 0& L^{+}\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0& 0\\ F_{31}^{\left(2\right)}& F_{32}^{\left(2\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0& 0& 0& 0& 0\\ F_{41}^{\left(2\right)}& F_{42}^{\left(2\right)}& 0& L\left(\mathbf{\alpha }\right)& 0& 0& 0& 0\\ F_{51}^{\left(3\right)}& F_{52}^{\left(3\right)}& F_{53}^{\left(3\right)}& F_{54}^{\left(3\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0& F_{57}^{\left(3\right)}& F_{58}^{\left(3\right)}\\ F_{61}^{\left(3\right)}& F_{62}^{\left(3\right)}& F_{63}^{\left(3\right)}& F_{64}^{\left(3\right)}& 0& L\left(\mathbf{\alpha }\right)& F_{67}^{\left(3\right)}& F_{68}^{\left(3\right)}\\ F_{71}^{\left(3\right)}& F_{72}^{\left(3\right)}& F_{73}^{\left(3\right)}& F_{74}^{\left(3\right)}& 0& 0& L\left(\mathbf{\alpha }\right)& 0\\ F_{81}^{\left(3\right)}& F_{82}^{\left(3\right)}& F_{83}^{\left(3\right)}& F_{84}^{\left(3\right)}& 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}\delta \mathsf{\phi }\\ \delta \mathsf{\psi }\\ \delta {\mathsf{\psi }}_1^{\left(1\right)}\\ \delta {\mathsf{\psi }}_2^{\left(1\right)}\\ \delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{2,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{3,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\end{array}\right)\rangle }_{\left(3\right)}\\ ={\langle {\left(\begin{array}{c}\delta \mathsf{\phi }\\ \delta \mathsf{\psi }\\ \delta {\mathsf{\psi }}_1^{\left(1\right)}\\ \delta {\mathsf{\psi }}_2^{\left(1\right)}\\ \delta {\mathsf{\psi }}_{1,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{2,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{3,i}^{\left(2\right)}\\ \delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}\end{array}\right)}^{†},\left(\begin{array}{cccccccc}{L}^{+}\left(\mathbf{\alpha }\right)& 0& F_{31}^{\left(2\right)}& F_{41}^{\left(2\right)}& F_{51}^{\left(3\right)}& F_{61}^{\left(3\right)}& F_{71}^{\left(3\right)}& F_{81}^{\left(3\right)}\\ 0& L\left(\mathbf{\alpha }\right)& F_{32}^{\left(2\right)}& F_{42}^{\left(2\right)}& F_{52}^{\left(3\right)}& F_{62}^{\left(3\right)}& F_{72}^{\left(3\right)}& F_{82}^{\left(3\right)}\\ 0& 0& L\left(\mathbf{\alpha }\right)& 0& F_{53}^{\left(3\right)}& F_{63}^{\left(3\right)}& F_{73}^{\left(3\right)}& F_{83}^{\left(3\right)}\\ 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)& F_{54}^{\left(3\right)}& F_{64}^{\left(3\right)}& F_{74}^{\left(3\right)}& F_{84}^{\left(3\right)}\\ 0& 0& 0& 0& L\left(\mathbf{\alpha }\right)& 0& 0& 0\\ 0& 0& 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)& 0& 0\\ 0& 0& 0& 0& F_{57}^{\left(3\right)}& F_{67}^{\left(3\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0\\ 0& 0& 0& 0& F_{58}^{\left(3\right)}& F_{68}^{\left(3\right)}& 0& L\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}{\mathsf{\psi }}_{1,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{2,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{3,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{4,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{5,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{6,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{7,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{8,ij}^{\left(3\right)}\end{array}\right)\rangle }_{\left(3\right)}\\ +P^{\left(3\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}\right],\end{array}$$

(65)

where the bilinear concomitant $P^{\left(3\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}\right]$ is defined on the phase-space boundary $x\in \partial {\Omega }_x$.

(iii)

Identify the first term on the right-side of Equation (65) with the indirect-effect term defined in Equation (41) by requiring that:

$$\begin{array}{l}\left(\begin{array}{cccccccc}{L}^{+}\left(\mathbf{\alpha }\right)& 0& F_{31}^{\left(2\right)}& F_{41}^{\left(2\right)}& F_{51}^{\left(3\right)}& F_{61}^{\left(3\right)}& F_{71}^{\left(3\right)}& F_{81}^{\left(3\right)}\\ 0& L\left(\mathbf{\alpha }\right)& F_{32}^{\left(2\right)}& F_{42}^{\left(2\right)}& F_{52}^{\left(3\right)}& F_{62}^{\left(3\right)}& F_{72}^{\left(3\right)}& F_{82}^{\left(3\right)}\\ 0& 0& L\left(\mathbf{\alpha }\right)& 0& F_{53}^{\left(3\right)}& F_{63}^{\left(3\right)}& F_{73}^{\left(3\right)}& F_{83}^{\left(3\right)}\\ 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)& F_{54}^{\left(3\right)}& F_{64}^{\left(3\right)}& F_{74}^{\left(3\right)}& F_{84}^{\left(3\right)}\\ 0& 0& 0& 0& L\left(\mathbf{\alpha }\right)& 0& 0& 0\\ 0& 0& 0& 0& 0& L^{+}\left(\mathbf{\alpha }\right)& 0& 0\\ 0& 0& 0& 0& F_{57}^{\left(3\right)}& F_{67}^{\left(3\right)}& L^{+}\left(\mathbf{\alpha }\right)& 0\\ 0& 0& 0& 0& F_{58}^{\left(3\right)}& F_{68}^{\left(3\right)}& 0& L\left(\mathbf{\alpha }\right)\end{array}\right)\left(\begin{array}{c}{\mathsf{\psi }}_{1,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{2,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{3,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{4,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{5,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{6,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{7,ij}^{\left(3\right)}\\ {\mathsf{\psi }}_{8,ij}^{\left(3\right)}\end{array}\right)=\left(\begin{array}{c}{\left[\partial R_{ij}^{\left(2\right)}/\partial \mathsf{\phi }\right]}^{†}\\ {\left[\partial R_{ij}^{\left(2\right)}/\partial \mathsf{\psi }\right]}^{†}\\ {\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†}\\ {\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†}\\ {\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{1,i}^{\left(2\right)}\right]}^{†}\\ {\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{2,i}^{\left(2\right)}\right]}^{†}\\ {\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{3,i}^{\left(2\right)}\right]}^{†}\\ {\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{4,i}^{\left(2\right)}\right]}^{†}\end{array}\right),\\ fori=1,\dots N_{\mathbf{\alpha }};j=1,\dots ,i.\end{array}$$

(66)

(iv)

The boundary conditions given for the 2nd-LFSS and those given in Equation (62) are now implemented in Equation (65), thereby reducing by half the number of unknown boundary-values in the bilinear concomitant $P^{\left(3\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}\right]$. The boundary conditions for the 3rd-level adjoint functions ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$ are chosen next so as to eliminate the remaining unknown boundary-values of the functions $\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)}$ while ensuring that Equation (66) is well posed. The boundary conditions thus chosen for the adjoint functions ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$ can be represented in operator form as follows:

$$B_A^{\left(3\right)}\left[\mathsf{\phi }\left(x\right);\mathsf{\psi }\left(x\right);{\mathsf{\psi }}_1^{\left(1\right)}\left(x\right);{\mathsf{\psi }}_2^{\left(1\right)}\left(x\right);{\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right),\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right);{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha }\right]=0,x\in \partial {\Omega }_x.$$

(67)

In most cases, the above choice of boundary conditions for the 3rd-level adjoint functions ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$ will cause the bilinear concomitant $P^{\left(3\right)}\left[\delta \mathsf{\phi };\delta \mathsf{\psi };\delta {\mathsf{\psi }}_1^{\left(1\right)},\delta {\mathsf{\psi }}_2^{\left(1\right)};\delta {\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,\delta {\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}\right]$ in Equation (65) to vanish. Even when it does not vanish, however, this bilinear concomitant will be reduced to a quantity, denoted here as ${\widehat{P}}^{\left(3\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right]$, which will contain only known values of its arguments. The system of equations comprising Equations (66) and (67) will be called the 3rd-Level Adjoint Sensitivity System (3rd-LASS) for the 3rd-level adjoint functions ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$, $i=1,\dots N_{\mathsf{\alpha }};j=1,\dots ,i.$

(v)  Use the 3rd-LASS together with Equations (65) and (64) to obtain the following expression for the indirect-effect term defined in Equation (41), in terms of the 3rd-level adjoint functions ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$:

$$\begin{array}{l}{\left\{\delta R_{ij}^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right)\right\}}_{ind}\\ ={\langle {\mathsf{\psi }}_{1,ij}^{\left(3\right)}\left(x\right),Q_1^{\left(1\right)}\left(\mathsf{\phi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{2,ij}^{\left(3\right)}\left(x\right),Q_2^{\left(1\right)}\left(\mathsf{\psi };\mathbf{\alpha };\delta \mathbf{\alpha }\right)\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{3,ij}^{\left(3\right)}\left(x\right),Q_1^{\left(2\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\rangle }_{\phi }\\ +{\langle {\mathsf{\psi }}_{4,ij}^{\left(3\right)}\left(x\right),Q_2^{\left(2\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{5,ij}^{\left(3\right)}\left(x\right),Q_{1,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{6,ij}^{\left(3\right)}\left(x\right),Q_{2,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\rangle }_{\phi }\\ +{\langle {\mathsf{\psi }}_{7,ij}^{\left(3\right)}\left(x\right),Q_{3,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{8,ij}^{\left(3\right)}\left(x\right),Q_{4,i}^{\left(3\right)}\left(\mathsf{\phi };\dots ;\delta \mathbf{\alpha }\right)\rangle }_{\phi }\\ -{\widehat{P}}^{\left(3\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right],i=1,\dots ,N_{\mathbf{\alpha }};j=1,\dots ,i.\end{array}$$

(68)

As Equation (68) indicates, the indirect-effect term can be computed speedily by quadratures once the 3rd-level adjoint functions ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$ become available.

(vi)

Replace Equation (68) in Equation (40) to obtain the following expression for the total 2nd-order response sensitivity to model parameters:

$$\begin{array}{l}\delta R_{ij}^{\left(2\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha };\delta \mathbf{\alpha }\right]\\ \triangleq {\displaystyle \sum _{k=1}^{N_{\mathbf{\alpha }}}{R}_{ijk}^{\left(3\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha }\right]\delta {\alpha }_k},i=1,\dots ,N_{\mathbf{\alpha }};j=1,\dots ,i,\end{array}$$

(69)

where the quantity $R_{ijk}^{\left(3\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha }\right]$ denotes the 3rd-order partial sensitivity of the response to the model parameters and is defined as follows:

$$\begin{array}{l}{R}_{ijk}^{\left(3\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha }\right]\triangleq \frac{\partial R_{ij}^{\left(2\right)}\left(\mathsf{\phi };\mathsf{\psi };\dots ;\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_k}\\ {\langle {\mathsf{\psi }}_{1,ij}^{\left(3\right)}\left(x\right),\partial \left[Q\left[\mathbf{\alpha }\left(x\right)\right]-L\left(\mathbf{\alpha }\right)\mathsf{\phi }\left(x\right)\right]/\partial {\alpha }_k\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{2,ij}^{\left(3\right)}\left(x\right),\partial \left[Q^{+}\left[\mathbf{\alpha }\left(x\right)\right]-L^{+}\left(\mathbf{\alpha }\right)\mathsf{\psi }\left(x\right)\right]/\partial {\alpha }_k\rangle }_{\phi }\\ +{\langle {\mathsf{\psi }}_{3,ij}^{\left(3\right)}\left(x\right),\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial {\mathbf{\alpha }}_k}-\frac{\partial \left[L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_1^{\left(1\right)}\left(x\right)\right]}{\partial {\mathbf{\alpha }}_k}\rangle }_{\phi }+{\langle {\mathsf{\psi }}_{4,ij}^{\left(3\right)}\left(x\right),\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial {\mathbf{\alpha }}_k}-\frac{\partial \left[L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_2^{\left(1\right)}\left(x\right)\right]}{\partial {\mathbf{\alpha }}_k}\rangle }_{\phi }\\ +\langle {\mathsf{\psi }}_{5,ij}^{\left(3\right)}\left(x\right),\frac{\partial }{\partial {\mathbf{\alpha }}_k}\{{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\phi }\right]}^{†}-L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{1,i}^{\left(2\right)}\left(x\right)\\ {-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\phi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\}\rangle }_{\phi }\\ +\langle {\mathsf{\psi }}_{6,ij}^{\left(3\right)}\left(x\right),\frac{\partial }{\partial {\mathbf{\alpha }}_k}\{{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial \mathsf{\psi }\right]}^{†}-L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{2,i}^{\left(2\right)}\left(x\right)\\ {-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\phi }}{\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)-\frac{{\partial }^{2}S\left(\mathsf{\phi };\mathsf{\psi };\mathbf{\alpha }\right)}{\partial \mathsf{\psi }\partial \mathsf{\psi }}{\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\}\rangle }_{\phi }\\ +{\langle {\mathsf{\psi }}_{7,ij}^{\left(3\right)}\left(x\right),\frac{\partial }{\partial {\mathbf{\alpha }}_k}\left\{{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†}-L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{3,i}^{\left(2\right)}\left(x\right)\right\}\rangle }_{\phi }\\ +{\langle {\mathsf{\psi }}_{8,ij}^{\left(3\right)}\left(x\right),\frac{\partial }{\partial {\mathbf{\alpha }}_k}\left\{{\left[\partial R_i^{\left(1\right)}\left(\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};\mathbf{\alpha }\right)/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†}-L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{4,i}^{\left(2\right)}\left(x\right)\right\}\rangle }_{\phi }\\ -\frac{\partial {\widehat{P}}^{\left(3\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)},{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha }\right]}{\partial {\mathsf{\alpha }}_k},i=1,\dots ,N_{\mathbf{\alpha }};j=1,\dots ,i;k=1,\dots ,j.\end{array}$$

(70)

(vii)

Note that the 2nd-LASS is independent of parameter variations $\delta \mathbf{\alpha }$. Thus, the exact computation of all of the partial third-order sensitivities, $R_{ijk}^{\left(3\right)}\left[\mathsf{\phi };\mathsf{\psi };{\mathsf{\psi }}_1^{\left(1\right)};{\mathsf{\psi }}_2^{\left(1\right)};{\mathsf{\psi }}_{1,i}^{\left(2\right)},\dots ,{\mathsf{\psi }}_{4,i}^{\left(2\right)};{\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)};\mathbf{\alpha }\right]$, $i=1,\dots ,N_{\alpha };j=1,\dots ,i;k=1,\dots ,j,$ requires at most $N_{\alpha }\left(N_{\alpha }+1\right)/2$ large-scale (adjoint) computations using the 3rd-LASS, rather than $N_{\alpha }\left(N_{\alpha }+1\right)\left(N_{\alpha }+2\right)/6$ large-scale computations as would be required by forward methods. In order to implement the practical computation of the 3rd-level adjoint functions, ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$ it is important to note that, in component form, Equation (66) has the following structure, for each $i=1,\dots N_{\alpha };j=1,\dots ,i$:

$$L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{5,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{1,i}^{\left(2\right)}\right]}^{†},$$

(71)

$$L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{6,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{2,i}^{\left(2\right)}\right]}^{†},$$

(72)

$$L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{7,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{3,i}^{\left(2\right)}\right]}^{†}-F_{57}^{\left(3\right)}{\mathsf{\psi }}_{5,ij}^{\left(3\right)}-F_{67}^{\left(3\right)}{\mathsf{\psi }}_{6,ij}^{\left(3\right)},$$

(73)

$$L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{8,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{4,i}^{\left(2\right)}\right]}^{†}-F_{58}^{\left(3\right)}{\mathsf{\psi }}_{5,ij}^{\left(3\right)}-F_{68}^{\left(3\right)}{\mathsf{\psi }}_{6,ij}^{\left(3\right)},$$

(74)

$$L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{4,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_2^{\left(1\right)}\right]}^{†}-F_{54}^{\left(3\right)}{\mathsf{\psi }}_{5,ij}^{\left(3\right)}-F_{64}^{\left(3\right)}{\mathsf{\psi }}_{6,ij}^{\left(3\right)}-F_{74}^{\left(3\right)}{\mathsf{\psi }}_{7,ij}^{\left(3\right)}-F_{84}^{\left(3\right)}{\mathsf{\psi }}_{8,ij}^{\left(3\right)},$$

(75)

$$L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{3,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_1^{\left(1\right)}\right]}^{†}-F_{53}^{\left(3\right)}{\mathsf{\psi }}_{5,ij}^{\left(3\right)}-F_{63}^{\left(3\right)}{\mathsf{\psi }}_{6,ij}^{\left(3\right)}-F_{73}^{\left(3\right)}{\mathsf{\psi }}_{7,ij}^{\left(3\right)}-F_{83}^{\left(3\right)}{\mathsf{\psi }}_{8,ij}^{\left(3\right)},$$

(76)

$$L\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{2,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial \mathsf{\psi }\right]}^{†}-F_{32}^{\left(2\right)}{\mathsf{\psi }}_{3,ij}^{\left(3\right)}-F_{42}^{\left(2\right)}{\mathsf{\psi }}_{4,ij}^{\left(3\right)}-F_{52}^{\left(3\right)}{\mathsf{\psi }}_{5,ij}^{\left(3\right)}-F_{62}^{\left(3\right)}{\mathsf{\psi }}_{6,ij}^{\left(3\right)}-F_{72}^{\left(3\right)}{\mathsf{\psi }}_{7,ij}^{\left(3\right)}-F_{82}^{\left(3\right)}{\mathsf{\psi }}_{8,ij}^{\left(3\right)},$$

(77)

$$L^{+}\left(\mathbf{\alpha }\right){\mathsf{\psi }}_{1,ij}^{\left(3\right)}={\left[\partial R_{ij}^{\left(2\right)}/\partial \mathsf{\phi }\right]}^{†}-F_{31}^{\left(2\right)}{\mathsf{\psi }}_{3,ij}^{\left(3\right)}-F_{41}^{\left(2\right)}{\mathsf{\psi }}_{4,ij}^{\left(3\right)}-F_{51}^{\left(3\right)}{\mathsf{\psi }}_{5,ij}^{\left(3\right)}-F_{61}^{\left(3\right)}{\mathsf{\psi }}_{6,ij}^{\left(3\right)}-F_{71}^{\left(3\right)}{\mathsf{\psi }}_{7,ij}^{\left(3\right)}-F_{81}^{\left(3\right)}{\mathsf{\psi }}_{8,ij}^{\left(3\right)}.$$

(78)

Thus, the 3rd-LASS it is to be solved successively, by first using Equations (71) and (72) to compute the 3rd-level adjoint functions ${\mathsf{\psi }}_{5,ij}^{\left(3\right)}$ and ${\mathsf{\psi }}_{6,ij}^{\left(3\right)}$. Note that solving using Equations (71) and (72) would be performed by using the same forward and adjoint solvers (i.e., computer codes) as used for solving the original forward and adjoint systems, namely Equations (1) and (6) subject to the corresponding boundary conditions, except that the right-sides of the respective solvers would have as “sources” the terms $\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{1,i}^{\left(2\right)}$ and $\partial R_{ij}^{\left(2\right)}/\partial {\mathsf{\psi }}_{2,i}^{\left(2\right)}$, respectively. After having obtained the 3rd-level adjoint functions ${\mathsf{\psi }}_{5,ij}^{\left(3\right)}$ and ${\mathsf{\psi }}_{6,ij}^{\left(3\right)}$, the next round of computations would be to solve Equations (73) and (74) in order to determine the 3rd-level adjoint functions ${\mathsf{\psi }}_{7,ij}^{\left(3\right)}$ and ${\mathsf{\psi }}_{8,ij}^{\left(3\right)}$, respectively. As before, solving using Equations (73) and (74) would be performed by using the same forward and adjoint solvers (i.e., computer codes) as used for solving the original forward and adjoint systems, namely Equations (1) and (6) subject to the corresponding boundary conditions, except that the right-sides of the respective solvers would have different “sources.” The next round of computations would require the use of the forward and adjoint solvers to solve Equations (75) and (76) in order to determine the 3rd-level adjoint functions ${\mathsf{\psi }}_{4,ij}^{\left(3\right)}$ and ${\mathsf{\psi }}_{3,ij}^{\left(3\right)}$, respectively. The final set of computations would require the use of the forward and adjoint solvers to solve Equations (77) and (78) in order to determine the 3rd-level adjoint functions ${\mathsf{\psi }}_{2,ij}^{\left(3\right)}$ and ${\mathsf{\psi }}_{1,ij}^{\left(3\right)}$, respectively. Thus, solving the 3rd-ASAM in order to determine the 3rd-level adjoint functions does not require any significant “code development,” since the original forward and adjoint solvers (codes) do not need to be modified; only the right-sides (i.e., “sources”) for these solvers/codes would need to be programmed accordingly.

Using the 3rd-LASS enables the specific computation of the 3rd-order sensitivities in the priority order set by the user, so that only the important 3rd-order partial sensitivities $R\left[\mathsf{\phi }\left(x\right),\mathsf{\psi }\left(x\right);\mathbf{\alpha }\right]$ would be computed. Information provided by the first- and second-order sensitivities might indicate which 3rd-order sensitivities could be neglected.

4. Third-Order Expressions for the Cumulants of the Response Distribution in Parameter Space

The 3rd-ASAM presented in Section 3, above, provides the most efficient way for computing exactly the first-, second- and third-order sensitivities of a response that couples the forward and adjoint systems that describe physical problems which are linear in the state-functions. The availability of these sensitivities enables the use of a third-order multivariate Taylor series-expansion (of the response around the known nominal parameter values) for quantifying the cumulants of the response distribution in the phase-space of model parameters. For a model’s computed-response, denoted as $r_{i_1}^c\left(\mathbf{\alpha }\right)$, where the superscript “c” denotes “computed” and the subscript $i_1=1,\dots ,N_r$ denotes one of a total of $N_r$ responses that would be of interest, the third-order Taylor-series of $r_a^c\left(\mathbf{\alpha }\right)$ around the model’s parameters’ mean (expected) values ${\mathbf{\alpha }}^0\triangleq \left({\mathsf{\alpha }}_1^0,\dots ,{\mathsf{\alpha }}_{N_{\mathsf{\alpha }}}^0\right)$ is:

$$\begin{array}{l}{r}_{i_1}^c\left(\mathbf{\alpha }\right)=r_{i_1}^c\left({\mathbf{\alpha }}^0\right)+{{\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}\left\{\frac{\partial r_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i}\right\}}}_{{\mathbf{\alpha }}^0}\left({\mathsf{\alpha }}_i-{\mathsf{\alpha }}_i^0\right)+\frac{1}{2}{\displaystyle \sum _{i,j=1}^{N_{\mathsf{\alpha }}}{\left\{\frac{{\partial }^2{r}_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i\partial {\mathsf{\alpha }}_j}\right\}}_{{\mathbf{\alpha }}^0}\left({\mathsf{\alpha }}_i-{\mathsf{\alpha }}_i^0\right)\left({\mathsf{\alpha }}_j-{\mathsf{\alpha }}_j^0\right)}\\ +\frac{1}{6}{\displaystyle \sum _{i,j,k=1}^{N_{\mathsf{\alpha }}}{\left\{\frac{{\partial }^3{r}_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i\partial {\mathsf{\alpha }}_j\partial {\mathsf{\alpha }}_k}\right\}}_{{\mathbf{\alpha }}^0}\left({\mathsf{\alpha }}_i-{\mathsf{\alpha }}_i^0\right)\left({\mathsf{\alpha }}_j-{\mathsf{\alpha }}_j^0\right)\left({\mathsf{\alpha }}_k-{\mathsf{\alpha }}_k^0\right)+\dots };a=1,\dots ,N_r.\end{array}$$

(79)

where $r_{i_1}^c\left({\mathbf{\alpha }}^0\right)$ denotes the nominal value of the response computed at the nominal (mean) parameter values ${\mathbf{\alpha }}^0\triangleq \left({\mathsf{\alpha }}_1^0,\dots ,{\mathsf{\alpha }}_{N_{\mathsf{\alpha }}}^0\right)$.

In practice, the values of the parameters ${\mathsf{\alpha }}_n$ are determined experimentally. Therefore, these parameters can be considered to be variates that behave stochastically, obeying a multivariate probability distribution function, denoted as $p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)$, which is seldom known in practice, particularly for large-scale systems involving many parameters. Considering that the multivariate distribution $p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)$ is formally defined on a domain $D_{\mathsf{\alpha }}$, the various moments (e.g., mean values, covariance and variances, etc.) of $p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)$ can be defined in a standard manner by using the following notation:

$${\langle u\left(\mathbf{\alpha }\right)\rangle }_{\mathsf{\alpha }}\triangleq {\displaystyle \underset{D_{\mathsf{\alpha }}}{\int }u\left(\mathbf{\alpha }\right)p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)d\mathbf{\alpha }}$$

(80)

where $u\left(\mathbf{\alpha }\right)$ is a continuous function of the parameters $\mathbf{\alpha }$. Using the notation defined in Equation (80), the expected (or mean) value of a model parameter ${\mathbf{\alpha }}_i$, denoted as ${\mathbf{\alpha }}_i^0$, is defined as follows:

$${\mathsf{\alpha }}_i^0\triangleq {\langle {\mathsf{\alpha }}_i\rangle }_{\mathbf{\alpha }}\triangleq {\displaystyle \underset{D_{\mathsf{\alpha }}}{\int }{\mathsf{\alpha }}_i{p}_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)d\mathbf{\alpha }},i=1,\dots ,N_{\mathsf{\alpha }}$$

(81)

The covariance, $cov\left({\mathsf{\alpha }}_i,{\mathsf{\alpha }}_j\right)$, of two parameters, ${\mathsf{\alpha }}_i$ and ${\mathsf{\alpha }}_j$, is defined as follows:

$${\mathsf{\mu }}_2^{ij}\left(\mathbf{\alpha }\right)\triangleq cov\left({\mathsf{\alpha }}_i,{\mathsf{\alpha }}_j\right)\triangleq {\langle \left({\mathsf{\alpha }}_i-{\mathsf{\alpha }}_i^0\right)\left({\mathsf{\alpha }}_j-{\mathsf{\alpha }}_j^0\right)\rangle }_{\mathsf{\alpha }},i,j=1,\dots ,N_{\mathsf{\alpha }}$$

(82)

The variance, $var\left({\mathsf{\alpha }}_i\right)$, of a parameter ${\mathsf{\alpha }}_i$, is defined as follows:

$$var\left({\mathsf{\alpha }}_i\right)\triangleq {\langle {\left({\mathsf{\alpha }}_i-{\mathsf{\alpha }}_i^0\right)}^2\rangle }_{\mathsf{\alpha }},i=1,\dots ,N_{\mathsf{\alpha }}$$

(83)

The standard deviation, ${\mathsf{\sigma }}_i$, of ${\mathsf{\alpha }}_i$, is defined as follows: ${\mathsf{\sigma }}_i\triangleq \sqrt{var({\mathsf{\alpha }}_i)}$;

The correlation, ${\mathsf{\rho }}_{ij}$, between two parameters ${\mathsf{\alpha }}_i$ and ${\mathsf{\alpha }}_j$, is defined as follows:

$${\mathsf{\rho }}_{ij}\triangleq cov\left({\mathsf{\alpha }}_i,{\mathsf{\alpha }}_j\right)/\left({\mathsf{\sigma }}_i{\mathsf{\sigma }}_j\right);i,j=1,\dots ,N_{\mathsf{\alpha }}$$

(84)

The 3rd-order moment, ${\mu }_3^{ijk}$, of the multivariate parameter distribution function $p\left(\mathbf{\alpha }\right)$, and the 3rd-order parameter correlation, $t_{ijk}$, respectively, are defined as follows:

$${\mu }_3^{ijk}\left(\mathbf{\alpha }\right)\triangleq {\displaystyle \underset{D_{\alpha }}{\int }\left({\alpha }_i-{\alpha }_i^0\right)\left({\alpha }_j-{\alpha }_j^0\right)\left({\alpha }_k-{\alpha }_k^0\right)p\left(\mathbf{\alpha }\right)d\mathbf{\alpha }}\triangleq t_{ijk}{\sigma }_i{\sigma }_j{\sigma }_k,i,j,k=1,\dots ,N_{\alpha };$$

(85)

The 4th-order moment, ${\mu }_4^{ijkl}$, of the multivariate parameter distribution function $p\left(\mathbf{\alpha }\right)$, and the 4th-order parameter correlation, $q_{ijkl}$, respectively, are defined as follows:

$${\mu }_4^{ijkl}\left(\mathbf{\alpha }\right)\triangleq {\displaystyle \underset{D_{\alpha }}{\int }\left({\alpha }_i-{\alpha }_i^0\right)\left({\alpha }_j-{\alpha }_j^0\right)\left({\alpha }_k-{\alpha }_k^0\right)\left({\alpha }_l-{\alpha }_l^0\right)p\left(\mathbf{\alpha }\right)d\mathbf{\alpha }}\triangleq q_{ijkl}{\sigma }_i{\sigma }_j{\sigma }_k{\sigma }_l;i,j,k,l=1,\dots ,N_{\mathsf{\alpha }}.$$

(86)

Using Equations (79) together with Equations (81) through (85) yields the following expression for the expected (mean) value, denoted as $E\left[r_{i_1}^c\left(\mathbf{\alpha }\right)\right]$, of a response $r_{i_1}^c\left(\mathbf{\alpha }\right)$:

$$E\left[r_{i_1}^c\left(\mathbf{\alpha }\right)\right]=r_{i_1}^c\left({\mathbf{\alpha }}^0\right)+\frac{1}{2}{\displaystyle \sum _{i,j=1}^{N_{\mathsf{\alpha }}}{\left\{\frac{{\partial }^2{r}_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i\partial {\mathsf{\alpha }}_j}\right\}}_{{\mathsf{\alpha }}^0}{\rho }_{ij}}{\sigma }_i{\sigma }_j+\frac{1}{6}{\displaystyle \sum _{i,j,k=1}^{N_{\mathsf{\alpha }}}{\left\{\frac{{\partial }^3{r}_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i\partial {\mathsf{\alpha }}_j\partial {\mathsf{\alpha }}_k}\right\}}_{{\mathsf{\alpha }}^0}{t}_{ijk}{\sigma }_i{\sigma }_j{\sigma }_k};i_1=1,\dots ,N_r.$$

(87)

Using Equations (79) together with Equations (81) through (86) yields the following expression for the covariance, denoted as $cov\left(r_{i_1}^c,r_{i_2}^c\right)$, of two responses, $r_{i_1}^c\left(\mathbf{\alpha }\right)$ and $r_{i_2}^c\left(\mathbf{\alpha }\right)$ $for{i}_1,i_2=1,\dots ,N_r$:

$$\begin{array}{c}cov\left(r_{i_1}^c,r_{i_2}^c\right)={\displaystyle \sum _{i=1}^{N_{\alpha }}{\displaystyle \sum _{j=1}^{N_{\alpha }}\left(\frac{\partial r_{i_1}^c}{\partial {\alpha }_i}\frac{\partial r_{i_2}^c}{\partial {\alpha }_j}\right){\rho }_{ij}{\sigma }_i{\sigma }_j}}+\frac{1}{2}{\displaystyle \sum _{i=1}^{N_{\alpha }}{\displaystyle \sum _{j=1}^{N_{\alpha }}{\displaystyle \sum _{\mu =1}^{N_{\alpha }}\left(\frac{{\partial }^2{r}_{i_1}^c}{\partial {\alpha }_i\partial {\alpha }_j}\frac{\partial r_{i_2}^c}{\partial {\mathbf{\alpha }}_{\mu }}+\frac{\partial r_{i_1}^c}{\partial {\alpha }_i}\frac{{\partial }^2{r}_{i_2}^c}{\partial {\alpha }_j\partial {\mathbf{\alpha }}_{\mu }}\right)t_{ij\mu }{\sigma }_i{\sigma }_j{\sigma }_{\mu }}}}\\ +\frac{1}{4}{\displaystyle \sum _{i=1}^{N_{\alpha }}{\displaystyle \sum _{j=1}^{N_{\alpha }}{\displaystyle \sum _{\mu =1}^{N_{\alpha }}{\displaystyle \sum _{\nu =1}^{N_{\mathbf{\alpha }}}\left(\frac{{\partial }^2{r}_{i_1}^c}{\partial {\mathbf{\alpha }}_i\partial {\mathbf{\alpha }}_j}\right)}\left(\frac{{\partial }^2{r}_{i_2}^c}{\partial {\mathbf{\alpha }}_{\mu }\partial {\mathbf{\alpha }}_{\nu }}\right)\left(q_{ij\mu \nu }-{\rho }_{ij}{\rho }_{\mu \nu }\right){\sigma }_i{\sigma }_j{\sigma }_{\mu }{\sigma }_{\nu }}}}\\ +\frac{1}{6}{\displaystyle \sum _{i=1}^{N_{\alpha }}{\displaystyle \sum _{j=1}^{N_{\alpha }}{\displaystyle \sum _{\mu =1}^{N_{\alpha }}{\displaystyle \sum _{\nu =1}^{N_{\mathbf{\alpha }}}\left(\frac{\partial r_{i_1}^c}{\partial {\mathbf{\alpha }}_i}\frac{{\partial }^3{r}_{i_2}^c}{\partial {\mathbf{\alpha }}_j\partial {\mathbf{\alpha }}_{\mu }\partial {\mathbf{\alpha }}_{\nu }}+\frac{\partial r_{i_2}^c}{\partial {\mathbf{\alpha }}_i}\frac{{\partial }^3{r}_{i_1}^c}{\partial {\mathbf{\alpha }}_j\partial {\mathbf{\alpha }}_{\mu }\partial {\mathbf{\alpha }}_{\nu }}\right)}{q}_{ij\mu \nu }{\sigma }_i{\sigma }_j{\sigma }_{\mu }{\sigma }_{\nu }}}}.\end{array}$$

(88)

In particular, the variance of a response $r_{i_1}^c\left(\mathbf{\alpha }\right)$ is obtained as by setting $i_1=i_2$ in Equation (88).

The covariance of a response, $r_{i_1}^c\left(\mathbf{\alpha }\right)$ and a parameter ${\mathsf{\alpha }}_{\ell }$, $i_1=1,\dots ,N_r$ and $\ell =1,\dots ,N_{\mathsf{\alpha }}$, which is denoted as $cov\left(r_{i_1}^c,{\mathsf{\alpha }}_{\ell }\right)$ and is given by the following expression:

$$\begin{array}{l}cov\left(r_{i_1}^c,{\mathsf{\alpha }}_{\ell }\right)={{\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}\left\{\frac{\partial r_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i}\right\}}}_{{\mathsf{\alpha }}^0}cov\left({\mathsf{\alpha }}_i,{\mathsf{\alpha }}_{\ell }\right)+\frac{1}{2}{\displaystyle \sum _{i,j=1}^{N_{\mathsf{\alpha }}}{\left\{\frac{{\partial }^2{r}_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i\partial {\mathsf{\alpha }}_j}\right\}}_{{\mathsf{\alpha }}^0}{t}_{ij\ell }{\sigma }_i{\sigma }_j{\sigma }_{\ell }}\\ +\frac{1}{6}{\displaystyle \sum _{i,j,k=1}^{N_{\mathsf{\alpha }}}{\left\{\frac{{\partial }^3{r}_{i_1}^c\left(\mathbf{\alpha }\right)}{\partial {\mathsf{\alpha }}_i\partial {\mathsf{\alpha }}_j\partial {\mathsf{\alpha }}_k}\right\}}_{{\mathsf{\alpha }}^0}{q}_{ij\mu \nu }{\sigma }_i{\sigma }_j{\sigma }_k{\sigma }_{\ell }};i_1=1,\dots ,N_r;\ell =1,\dots ,N_{\mathsf{\alpha }}.\end{array}$$

(89)

The third-order cumulant for three responses, $r_{i_1}^c\left(\mathbf{\alpha }\right)$, $r_{i_2}^c\left(\mathbf{\alpha }\right)$ and $r_{i_3}^c\left(\mathbf{\alpha }\right)$, which is denoted below as ${\mu }_3\left(r_{i_1}^c,r_{i_2}^c,r_{i_3}^c\right)$, $for{i}_1,i_2,i_3=1,\dots ,N_r$, is obtained similarly by using Equations (79) together with Equations (81) through (86), and has the following expression:

$$\begin{array}{l}{\mu }_3\left(r_{i_1}^c,r_{i_2}^c,r_{i_3}^c\right)\triangleq {\langle \left[r_{i_1}^c-E\left(r_{i_1}^c\right)\right]\left[r_{i_2}^c-E\left(r_{i_2}^c\right)\right]\left[r_{i_3}^c-E\left(r_{i_3}^c\right)\right]\rangle }_{{\alpha }}\\ ={\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{j=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\mu =1}^{N_{\mathsf{\alpha }}}\frac{\partial r_{i_1}^c}{\partial {\mathsf{\alpha }}_i}\frac{\partial r_{i_2}^c}{\partial {\mathsf{\alpha }}_j}}}}\frac{\partial r_{i_3}^c}{\partial {\mathsf{\alpha }}_{\mu }}{t}_{ij\mu }{\mathsf{\sigma }}_i{\mathsf{\sigma }}_j{\mathsf{\sigma }}_{\mu }+\frac{1}{2}{\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{j=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\mu =1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\nu =1}^{N_{\mathsf{\alpha }}}\frac{\partial r_{i_1}^c{}_k}{\partial {\mathsf{\alpha }}_i}\frac{\partial r_{i_2}^c}{\partial {\mathsf{\alpha }}_j}}}}\frac{{\partial }^2{r}_{i_3}^c}{\partial {\mathsf{\alpha }}_{\mu }\partial {\mathsf{\alpha }}_{\nu }}\left(q_{ij\mu \nu }-{\rho }_{ij}{\rho }_{\mu \nu }\right){\mathsf{\sigma }}_i{\mathsf{\sigma }}_j{\mathsf{\sigma }}_{\mu }{\mathsf{\sigma }}_{\nu }}\\ +\frac{1}{2}{\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{j=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\mu =1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\nu =1}^{N_{\mathsf{\alpha }}}\frac{\partial r_{i_1}^c}{\partial {\mathsf{\alpha }}_i}}}}\frac{{\partial }^2{r}_{i_2}^c}{\partial {\mathsf{\alpha }}_j\partial {\mathsf{\alpha }}_{\mu }}\frac{\partial r_{i_3}^c}{\partial {\mathsf{\alpha }}_{\nu }}\left(q_{ij\mu \nu }-{\rho }_{i\nu }{\rho }_{j\mu }\right){\mathsf{\sigma }}_i{\mathsf{\sigma }}_j{\mathsf{\sigma }}_{\mu }{\mathsf{\sigma }}_{\nu }}\\ +\frac{1}{2}{\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{j=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\mu =1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\nu =1}^{N_{\mathsf{\alpha }}}\frac{{\partial }^2{r}_{i_1}^c}{\partial {\mathsf{\alpha }}_i\partial {\mathsf{\alpha }}_j}\frac{\partial r_{i_2}^c}{\partial {\alpha }_{\mu }}\frac{\partial r_{i_3}^c}{\partial {\alpha }_{\nu }}}}}\left(q_{ij\mu \nu }-{\rho }_{ij}{\rho }_{\mu \nu }\right){\mathsf{\sigma }}_i{\mathsf{\sigma }}_j{\mathsf{\sigma }}_{\mu }{\mathsf{\sigma }}_{\nu }}.\end{array}$$

(90)

In particular, the skewness of a single response is customarily denoted as ${\gamma }_1\left(R\right)$, and is defined as follows: ${\gamma }_1\left(R\right)=\frac{{\mu }_3\left(R\right)}{{\left[var\left(R\right)\right]}^{\raisebox{1ex}{$3$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}},$ where ${\mu }_3\left(R\right)$ denotes the third central moment of the response distribution. For normally-distributed uncorrelated parameters and a single response, the expression in Equation (90) simplifies to yield ${\mu }_3\left(R\right)=3{\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}{\left(\frac{\partial R}{\partial {\mathsf{\alpha }}_i^{}}\right)}^2\frac{{\partial }^{2}R}{\partial {\mathsf{\alpha }}_i^2}{\mathsf{\sigma }}_i^4}$. As is well-known, the skewness provides a quantitative measure of the asymmetries in the respective distribution.

The first-order sensitivities contribute the leading terms to the second-, third-, and fourth-order moments of the response distribution, thus providing the leading contributions to the responses variance/covariances, skewness, and kurtosis. Obtaining the exact and complete set of first-order sensitivities of responses to model parameters is of paramount importance for any analysis of a computational model.

The second-order sensitivities contribute the leading correction terms to the response’s expected value (causing it to differ from the response’s computed value). The second-order sensitivities also contribute to the response variances and covariances. If the parameters follow a normal (Gaussian) multivariate distribution, the second-order sensitivities contribute the leading terms to the response’s third-order moment. Thus, neglecting the second-order response sensitivities to normally distributed parameters would nullify the third-order response correlations and hence would nullify the skewness of a response.

The fourth-order cumulant, denoted as ${\mu }_4\left(r_{i_1}^c,r_{i_2}^c,r_{i_3}^c,r_{i_4}^c\right)$ $for{i}_1,i_2,i_3,i_4=1,\dots ,N_r$, among four responses $r_{i_1}^c\left(\mathbf{\alpha }\right)$, $r_{i_2}^c\left(\mathbf{\alpha }\right)$, $r_{i_3}^c\left(\mathbf{\alpha }\right)$ and $r_{i_4}^c\left(\mathbf{\alpha }\right)$, is also obtained by using Equations (79). Up to fourth-order correlations in the model parameters, ${\mu }_4\left(r_{i_1}^c,r_{i_2}^c,r_{i_3}^c,r_{i_4}^c\right)$ has the following expression:

$$\begin{array}{l}{\mu }_4\left(r_{i_1}^c,r_{i_2}^c,r_{i_3}^c,r_{i_4}^c\right)\triangleq {\langle \left[r_{i_1}^c-E\left(r_{i_1}^c\right)\right]\left[r_{i_2}^c-E\left(r_{i_2}^c\right)\right]\left[r_{i_3}^c-E\left(r_{i_3}^c\right)\right]\left[r_{i_4}^c-E\left(r_{i_4}^c\right)\right]\rangle }_{\mathsf{\alpha }}\\ ={\displaystyle \sum _{i=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{j=1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\mu =1}^{N_{\mathsf{\alpha }}}{\displaystyle \sum _{\nu =1}^{N_{\mathbf{\alpha }}}\frac{\partial r_{i_1}^c{}_k}{\partial {\mathsf{\alpha }}_i}\frac{\partial r_{i_2}^c}{\partial {\mathsf{\alpha }}_j}}}}\frac{\partial r_{i_3}^c}{\partial {\mathsf{\alpha }}_{\mu }}\frac{\partial r_{i_4}^c}{\partial {\mathsf{\alpha }}_{\nu }}{q}_{ij\mu \nu }{\mathsf{\sigma }}_i{\mathsf{\sigma }}_j{\mathsf{\sigma }}_{\mu }{\mathsf{\sigma }}_{\nu }}.\end{array}$$

(91)

5. 2nd/3rd-Order Best-Estimated Results with Reduced Uncertainties Predictive Modeling (2nd/3rd-BERRU-PM) in the Joint Phase-Space of Responses and Parameters

Cacuci [7] has summarized the scope of “BERRU-PM” as follows: “BERRU-PM commences by identifying and characterizing the uncertainties involved in every step in the sequence of the numerical simulation processes that ultimately lead to a prediction. This includes characterizing: (a) errors and uncertainties in the data used in the simulation (e.g., input data, model parameters, initial conditions, boundary conditions, sources and forcing functions), (b) numerical discretization errors, and (c) uncertainties in (e.g., lack of knowledge of) the processes being modeled. Under ideal circumstances, the result of is a probabilistic description of possible future outcomes based on all recognized errors and uncertainties.”

Consider a vector-valued variate $x\triangleq {\left(x_1,\dots ,x_N\right)}^{†}$, the components of which obey an unknown multivariate distribution $p\left(x\right)$. Of course, the probability distribution function $p\left(x\right)$ would need to be properly normalized, i.e., it must also satisfy the constraint:

$${\displaystyle \int p\left(x\right)dx}=1.$$

(92)

Consider further that the moments of several known functions $F_k\left(x\right)$ over the unknown distribution $p\left(x\right)$, denoted as $\langle F_k\rangle$, and defined as:

$$\langle F_k\rangle ={\displaystyle \int p\left(x\right)F_k\left(x\right)}dx,k=1,2,\dots ,K$$

(93)

are also known. The problem of reconstructing a function from a finite number of its moments has been investigated for many decades in the mathematical and physical sciences. For the purposes of predictive modeling, the main goal is to determine a probability density function $p\left(x\right)$ which is consistent with the knowledge expressed by Equation (93) and introduces no unwarranted information. Such a probability density function, $p\left(x\right)$ , can be constructed using the method of maximum entropy (“MaxEnt”), which generates the most conservative estimate of a probability distribution with the given information and the most non-committal one with regard to missing information [11]. According to the MaxEnt principle, the unknown probability density function $p\left(x\right)$ must satisfy the constraints expressed by Equation (93) while having its Boltzmann–Shannon–Gibbs (BSG) entropy (also referred to as the “information entropy”) as large as possible. For a continuous distribution having a probability density function $p\left(x\right)$, the expression for its information/BSG-entropy is:

$$H\left(p\right)=-{\displaystyle \int dxp\left(x\right)ln}\frac{p\left(x\right)}{m\left(x\right)}$$

(94)

where $m\left(x\right)$ is a prior density function that ensures form invariance under change of variable. Intuitively, in a bounded domain, the most conservative distribution, i.e., the distribution of maximum entropy, is the one that assigns equal probability to all the accessible states. Hence, the method of maximum entropy can be thought of as choosing the most “uniform” distribution $p\left(x\right)$ that satisfies the given moment constraints expressed by Equation (93) and introduces no unwarranted information. Any probability density function $p\left(x\right)$ satisfying the constraints which has smaller entropy will contain more information (less uncertainty), and thus would predict something stronger than warranted by our knowledge and/or assumptions. The probability density function with maximum entropy, satisfying the constraints imposed, is the one which should be least surprising in terms of the predictions it makes.

Selecting the unique $p\left(x\right)$ which would maximize the information entropy defined by Equation (94) while simultaneously satisfying the known constraints given in Equations (93) and (92) is a variational problem that can be solved by the well-known method of using Lagrange multipliers, ${\lambda }_k,k=1,2,\dots ,K$, to construct the following Lagrangian functional:

$$L\left[p\left(x\right)\right]\triangleq -{\displaystyle \int dxp\left(x\right)ln}\frac{p\left(x\right)}{m\left(x\right)}-{\displaystyle \sum _{k=1}^K\left[{\lambda }_k{\displaystyle \int p\left(x\right)F_k\left(x\right)}dx-\langle F_k\rangle \right]}-{\lambda }_0\left[{\displaystyle \int p\left(x\right)dx}-1\right].$$

(95)

The critical point of $L\left[p\left(x\right)\right]$ is obtained by solving the equation that results from setting the first Gateaux-differential of $L\left[p\left(x\right)\right]$ to zero, namely:

$$\begin{array}{l}\mathsf{\delta }L\left[p;\delta p\right]\triangleq {\left\{\frac{d}{d\epsilon }L\left[p\left(x\right)+\epsilon \mathsf{\delta }p\left(x\right)\right]\right\}}_{\epsilon =0}=0\\ =-{\displaystyle \int ln}\frac{p\left(x\right)}{m\left(x\right)}\mathsf{\delta }p\left(x\right)dx-{\displaystyle \int \mathsf{\delta }p\left(x\right)}dx-{\displaystyle \sum _{k=1}^K{\lambda }_k{\displaystyle \int \mathsf{\delta }p\left(x\right)F_k\left(x\right)}dx}-{\lambda }_0{\displaystyle \int \mathsf{\delta }p\left(x\right)dx}.\end{array}$$

(96)

It follows from Equation (96) that:

$$p\left(x\right)=m\left(x\right)exp\left[-{\lambda }_0-1-{\displaystyle \sum _{k=1}^K{\lambda }_k{F}_k\left(x\right)}\right].$$

(97)

Replacing the results obtained in Equation (97) into Equation (92) and eliminating the Lagrange multiplier ${\mathsf{\lambda }}_0$ from the resulting expression leads to the following expression for the probability density function $p\left(x\right)$:

$$p\left(x\right)=\frac{1}{Z}m\left(x\right)exp\left[-{\displaystyle \sum _{k=1}^K{\mathsf{\lambda }}_k{F}_k\left(x\right)}\right]$$

(98)

where the normalization constant $Z$ in Equation (98) is defined as follows:

$$Z\triangleq {\displaystyle \int dx}m\left(x\right)exp\left[-{\displaystyle \sum _{k=1}^K{\mathsf{\lambda }}_k{F}_k\left(x\right)}\right]$$

(99)

In statistical mechanics, the normalization constant $Z$ is called the partition function (or sum over states), and carries all of the information available about the possible states of the system.

The expected integral data is obtained by differentiating $Z$ with respect to the Lagrange multiplier ${\mathsf{\lambda }}_k$, to obtain the following relationships:

$$\langle F_k\rangle =-\frac{\partial }{\partial {\mathsf{\lambda }}_k}lnZ,k=1,2,\dots ,K$$

(100)

When the integral data $\langle F_k\rangle$ are not yet known, the uniform distribution $m\left(x\right)=1$ is the most appropriate to consider. In this case, the maximum entropy (MaxEnt) algorithm yields the uniform distribution as would be required by the principle of insufficient reason. Thus, the MaxEnt principle generalizes the principle of insufficient reason. The MaxEnt can be applied to both discrete and to continuous distributions. The MaxEnt method has been shown [12] to be equivalent to constrained variational inference, thus establishing the link between MaxEnt and Bayesian approximations. The MaxEnt method has been used in many fields; enumerating these fields is beyond the scope of this work, which is limited to nuclear engineering applications. The pioneering application of the MaxEnt method to time-independent nuclear reactor physics problems was initiated in the 1970s [13,14,15,16,17]. The first application of the MaxEnt method to a time-dependent nuclear energy system was by Barhen et al. [18]. This work was subsequently extended by Cacuci and Ionescu-Bujor [19], which presented analytical formulas for predicted mean values and covariance matrices, for both predicted model parameters and responses, which have generalized the previous results presented in data assimilation procedures for geophysical sciences and linear Bayesian models. Other applications to nuclear energy systems are presented in [20,21,22,23,24,25,26,27].

To the author’s knowledge, none of the analytical results published thus far include comprehensively all of the second- and third-order response sensitivities to all of the model’s parameters. The end-results to be presented in this section will extend the results published thus far in the open literature by presenting analytical formulas for both predicted model responses and parameters by including all of the second-and third-order sensitivities of computed model responses to the model’s parameters.

5.1. 2nd/3rd-BERRU-PM: A Priori Information

This Subsection will present the mathematical form of the information that will be ultimately used for predicting the optimal, best-estimate mean values for both the model responses and model parameters, with reduced predicted uncertainties, in the combined parameter-response phase space.

5.1.1. Expected Values and Covariances of Measured Responses

Consider that $N_r$ quantities of interest, henceforth called “system responses” and denoted as $r_i^m$, $i=1,\dots ,N_r$, have been experimentally measured, yielding their expected values, as well as their corresponding covariances (i.e., standard deviations and correlations). For the mathematical derivations to follow, it is convenient to consider the responses $r_i^m$ to constitute the components of the $N_r$-dimensional column vector $r^m$ defined as follows:

$$r^m\triangleq {\left(r_1^m,\dots ,r_{N_r}^m\right)}^{†}$$

(101)

Since the experimentally measured responses cannot be measured exactly, they are usually considered to be variates that follow an unknown multivariate distribution function of the observations, denoted as $p_r\left(r\right)$, which is formally defined on a domain $D_r$. Methods for finding estimates of a measured quantity and indicators of the quality of the estimates depend on the assumed form of the unknown distribution function $p_r\left(r\right)$ of the observations. The moments of the distribution of measured responses can be conveniently denoted by introducing the following notation for the expectation (or mean value) of a function $w\left(r^m\right)$:

$$E\left[w\left(r^m\right)\right]\triangleq {\displaystyle \underset{D_r}{\int }w\left(r^m\right)p_r\left(r\right)dr}$$

(102)

When the distribution is discrete, the integral in Equation (102) denotes a sum over the respective discrete probabilities. Using the notation introduced in Equation (102), the expectation (or mean value) of the experimentally measured responses $r_i^m$ is formally defined as follows:

$$E\left(r_i^m\right)\triangleq {\displaystyle \underset{D_r}{\int }{r}_i^m{p}_r\left(r\right)d}r,i=1,\dots ,N_r,$$

(103)

The expected values of the measured responses will be considered to constitute the components of the vector $r^m$ defined as:

$$E\left(r^m\right)\triangleq {\left[E\left(r_1^m\right),\dots ,E\left(r_{N_r}^m\right)\right]}^{†}$$

(104)

The covariance matrix of measured responses will be denoted as $C_m$, and is defined as:

$$\begin{gathered} \begin{array}{l}{C}_m\triangleq {\langle \left[r^m-E\left(r^m\right)\right]{\left[r^m-E\left(r^m\right)\right]}^{†}\rangle }_r={\left[cov\left(r_i^m,r_j^m\right)\right]}_{N_r\times N_r},\\ \\ {\left[cov\left(r_i^m,r_j^m\right)\right]}_{N_r\times N_r}\triangleq {\langle \left[r_i^m-E\left(r_i^m\right)\right]\left[r_j^m-E\left(r_j^m\right)\right]\rangle }_r;i,j=1,\dots ,N_r.\end{array} \end{gathered}$$

(105)

5.1.2. Expectations and Covariances of Computed Responses Including Second- and Third-Order Sensitivities to Model Parameters

For subsequent computations, it is convenient to consider that the expected values ${\mathsf{\alpha }}_i^0$ of the components ${\mathsf{\alpha }}_i$ of the $N_{\mathsf{\alpha }}$-dimensional vector of model parameters $\mathbf{\alpha }\triangleq {\left({\mathsf{\alpha }}_1,\dots ,{\mathsf{\alpha }}_{N_{\mathsf{\alpha }}}\right)}^{†}$ are components of the following vector of mean (expected) values:

$${\mathbf{\alpha }}^0\triangleq {\left({\mathsf{\alpha }}_1^0,\dots ,{\mathsf{\alpha }}_{N_{\mathsf{\alpha }}}^0\right)}^{†}$$

(106)

The variances and covariances defined in Equations (82) and (83) are considered to constitute the elements of a symmetric, positive-definite parameter covariance matrix of dimension $N_{\mathsf{\alpha }}\times N_{\mathsf{\alpha }}$, denoted as $C_{\mathsf{\alpha }}$ and defined as follows:

$$C_{\mathsf{\alpha }}\triangleq {\langle \left(\mathbf{\alpha }-{\mathbf{\alpha }}^0\right){\left(\mathbf{\alpha }-{\mathbf{\alpha }}^0\right)}^{†}\rangle }_{\mathsf{\alpha }}.$$

(107)

Consider that the values of the $N_r$ experimentally measured responses $r_i^m$ can be computed using a multi-physics model that comprises $N_{\mathsf{\alpha }}$ model parameters ${\mathsf{\alpha }}_n,n=1,\dots ,N_{\mathsf{\alpha }}$ which are related to the model’s independent and dependent variables through the model’s underlying equations, correlations, tables, etc. Of course, the computed response values will not coincide with the measured ones, because, just like the experimentally measured responses, the model’s parameters and numerical solution of the underlying equations, and consequently the computed response values, are also subject to uncertainties. The computed responses $r_k^c\left(\mathbf{\alpha }\right),k=1,\dots ,N_r,$ are considered to be elements of an $N_r$-dimensional vector $r^c\left(\mathbf{\alpha }\right)$, defined as follows:

$$r^c\left(\mathbf{\alpha }\right)\triangleq {\left[r_1^c\left(\mathbf{\alpha }\right),\dots ,r_{N_r}^c\left(\mathbf{\alpha }\right)\right]}^{†}$$

(108)

The expectation values $E\left[r_k^c\left(\mathbf{\alpha }\right)\right]$ given by Equation (87) are considered to be the components of the following vector of “expected values of the computed response”:

$$E\left[r^c\left(\mathbf{\alpha }\right)\right]\triangleq {\left[E\left(r_1^c\right),\dots ,E\left(r_{N_r}^c\right)\right]}^{†}$$

(109)

The response covariances defined in Equation (88) are considered to be the components of a $\left(N_r\times N_r\right)$-dimensional matrix denoted as $C_r$ and defined as follows:

$$C_r\triangleq {\langle \left[r^c-E\left(r^c\right)\right]{\left[r^c-E\left(r^c\right)\right]}^{†}\rangle }_{\mathsf{\alpha }}.$$

(110)

The covariances between the computed responses and the model –parameters defined in Equation (89) are considered to be the components of an $\left(N_r\times N_{\mathsf{\alpha }}\right)$-dimensional matrix denoted as $C_{r\mathsf{\alpha }}$ and defined as follows:

$$C_{r\mathsf{\alpha }}\triangleq {\langle \left[r^c-E\left(r^c\right)\right]{\left(\mathbf{\alpha }-{\mathbf{\alpha }}^0\right)}^{†}\rangle }_{\mathsf{\alpha }}=C_{\mathsf{\alpha }r}^{†}$$

(111)

The joint covariance matrix, denoted as $C_M$, of the model parameters and model-computed responses is defined as follows:

$$C_M\triangleq \left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right),$$

(112)

5.2. 2nd/3rd-BERRU-PM: Analytical Expressions for Best-Estimate Results with Reduced Uncertainties for Responses and Parameters in the Joint Phase-Space of Responses and Parameters

Consider the joint probability function $p\left(\mathbf{\alpha },r\right)$ of the multi-variates $\mathbf{\alpha }$ and $r^m$, which is defined on the domain $D\triangleq D_{\mathsf{\alpha }}\times D_r$ and is properly normalized such that:

$${\displaystyle \underset{D}{\int }p\left(\mathbf{\alpha },r\right)d\mathbf{\alpha }dr}=1$$

(113)

The exact form of $p\left(\mathbf{\alpha },r\right)$ is unknown, of course. Since the (multi)variates $\mathbf{\alpha }$ and $r^m$ are statistically independent of each other, it follows that $p\left(\mathbf{\alpha },r\right)=p_r\left(r\right)p_{\mathbf{\alpha }}\left(\mathbf{\alpha }\right)$. Therefore, the expected value of a function $w\left(r^m\right)$ satisfies the following relations:

$$\begin{array}{l}\langle w\left(r^m\right)\rangle \triangleq {\displaystyle \underset{D}{\int }w\left(r^m\right)p\left(\mathbf{\alpha },r\right)d\mathbf{\alpha }dr}\\ ={\displaystyle \underset{D}{\int }w\left(r^m\right)p_r\left(r\right)p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)d\mathbf{\alpha }dr}=\left[{\displaystyle \underset{D_{\mathbf{\alpha }}}{\int }{p}_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)d\mathbf{\alpha }}\right]\left[{\displaystyle \underset{D_r}{\int }w\left(r^m\right)p_r\left(r\right)dr}\right]={\langle w\left(r^m\right)\rangle }_r\end{array}$$

(114)

while the expected value of a function $u\left(\mathbf{\alpha }\right)$ satisfies the following relations:

$$\begin{array}{l}\langle u\left(\mathbf{\alpha }\right)\rangle \triangleq {\displaystyle \underset{D}{\int }u\left(\mathbf{\alpha }\right)p\left(\mathbf{\alpha },r\right)d\mathbf{\alpha }dr}\\ ={\displaystyle \underset{D}{\int }u\left(\mathbf{\alpha }\right)p_r\left(r\right)p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)d\mathbf{\alpha }dr}=\left[{\displaystyle \underset{D_r}{\int }{p}_r\left(r\right)dr}\right]\left[{\displaystyle \underset{D_{\mathbf{\alpha }}}{\int }u\left(\mathbf{\alpha }\right)p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)d\mathbf{\alpha }}\right]={\langle u\left(\mathbf{\alpha }\right)\rangle }_{\mathbf{\alpha }}\end{array}$$

(115)

Therefore, the a priori information about the model parameters, computed and measured responses can be conveniently summarized by considering that the physical system under consideration is described mathematically by a multivariate vector:

$$x\triangleq {\left[\mathbf{\alpha },r^c\left(\mathbf{\alpha }\right),r^m\right]}^{†}$$

(116)

obeying an unknown joint multivariate distribution function:

$$p\left(\mathbf{\alpha },r\right)=p_r\left(r\right)p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)$$

(117)

but having a known vector of expected values denoted as:

$$x^0\triangleq {\left[{\mathbf{\alpha }}^0,E\left(r^c\right),E\left(r^m\right)\right]}^{†}$$

(118)

and a covariance matrix denoted as:

$$C\triangleq \left(\begin{array}{ccc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}& 0\\ C_{r\mathsf{\alpha }}& C_r& 0\\ 0& 0& C_m\end{array}\right)$$

(119)

Applying the MaxEnt principle as described in the Appendix A to the information provided in Equations (116) through (119) indicates that the MaxEnt form, denoted as $p_2\left(x|x^0,C\right)$, of the unknown distribution $p\left(\mathbf{\alpha },r\right)$ will have the following multivariate Gaussian form:

$$p_2\left(x|x^0,C\right)dx=\frac{exp\left[-\frac{1}{2}{\left(x-x^0\right)}^{†}{C}^{-1}\left(x-x^0\right)\right]dx,}{\sqrt{det\left(2\pi C\right)}},-\infty <x_j<\infty .$$

(120)

The MaxEnt-Gaussian form shown in Equation (120) can also be written in the equivalent form:

$$\begin{array}{l}{p}_2\left(x|x^0,C\right)=\frac{exp\left[Q\left(\mathbf{\alpha },r^c,r^m\right)\right]}{\sqrt{det\left(2\pi C_m\right)}\sqrt{det\left[2\pi \left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right)\right]}},\\ Q\left(\mathbf{\alpha },r^c,r^m\right)\triangleq -\frac{1}{2}{\left(\begin{array}{c}\mathbf{\alpha }-{\mathbf{\alpha }}^0\\ r^c-E\left(r^c\right)\end{array}\right)}^{†}{\left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right)}^{-1}\left(\begin{array}{c}\mathbf{\alpha }-{\mathbf{\alpha }}^0\\ r^c-E\left(r^c\right)\end{array}\right)-\frac{1}{2}\left[r^m-E\left(r^m\right)\right]C_m^{-1}{\left[r^m-E\left(r^m\right)\right]}^{†}.\end{array}$$

(121)

The expression provided in Equation (121) highlights the “Bayesian” construction underlying the expression of the a posteriori joint maximum-entropy provability distribution function $p_2\left(x|x^0,C\right)$ of the physical system’s responses and parameters. In order for the measured and computed responses to represent the same physical quantity, it is necessary that:

$$r^m=r^c\left(\mathbf{\alpha }\right)=r,$$

(122)

where the vector $r$ represents both the computed and measured responses. Determining the moments of $p_2\left(x|x^0,C\right)$ for subsequent predictions will require evaluations of integrals of the following form:

$$E\left[g\left(\mathbf{\alpha },r\right)\right]=\frac{{\displaystyle \underset{D}{\int }g\left(\mathbf{\alpha },r\right)exp\left[h\left(\mathbf{\alpha },r\right)\right]d\mathbf{\alpha }dr}}{{\displaystyle \underset{D}{\int }exp\left[h\left(\mathbf{\alpha },r\right)\right]d\mathbf{\alpha }dr}}$$

(123)

Expressions such as shown in Equation (123) can be evaluated to a high degree of accuracy (a priory controlled) by using the saddle-point method (also called Laplace approximation, or steepest descent method), which relies on evaluating the respective integrals at the so-called “saddle point(s).” For the integral in the denominator of Equation (123), the saddle point, denoted as $x_D\triangleq \left({\mathbf{\alpha }}_D,r_D\right)$, is the point at which the gradient of the function $h\left(x\right)$ vanishes, i.e., ${\nabla }_{x}h\left(x_D\right)=0$, so that $h\left(x\right)$ can be expanded in the following Taylor series:

$$h\left(x\right)=h\left(x_D\right)+SOT\left(x\right),$$

(124)

where the quantity $"SOT\left(x\right)"$ denotes terms of second- and higher-order in $x$. When the function $g\left(x\right)$ varies slowly, it is simply evaluated at the respective saddle point, and the resulting expression of $E\left[g\left(\mathbf{\alpha },r\right)\right]$ in Equation (123) becomes:

$$E\left[g\left(\mathbf{\alpha },r\right)\right]=\frac{g\left({\mathbf{\alpha }}_D,r_D\right)exp\left[h\left({\mathbf{\alpha }}_D,r_D\right)\right]{\displaystyle \underset{D}{\int }exp\left[SOT\left(\mathbf{\alpha },r\right)\right]d\mathbf{\alpha }dr}}{exp\left[h\left({\mathbf{\alpha }}_D,r_D\right)\right]{\displaystyle \underset{D}{\int }exp\left[SOT\left(\mathbf{\alpha },r\right)\right]d\mathbf{\alpha }dr}}\cong g\left({\mathbf{\alpha }}_D,r_D\right),$$

(125)

where the saddle-point $\left({\mathbf{\alpha }}_D,r_D\right)$ is defined as the point in phase-space where the gradients of $h\left(\mathbf{\alpha },r\right)$ vanish, i.e.,

$${\left\{{\nabla }_{\mathbf{\alpha }}h\left(\mathbf{\alpha },r\right)\right\}}_{\left({\mathbf{\alpha }}_D,r_D\right)}=0,{\left\{{\nabla }_{r}h\left(\mathbf{\alpha },r\right)\right\}}_{\left({\mathbf{\alpha }}_D,r_D\right)}=0$$

(126)

When the function $h\left(x\right)$ has a Taylor-series containing powers higher than second-order in $x$, the gradient ${\nabla }_{x}h\left(x\right)$ of the function $h\left(x\right)$ may vanish at multiple saddle points, in which case the contributions from all of these saddle points would need to be accounted for when evaluating the integrals in Equation (123).

5.2.1. Predicted Best-Estimate Expected Values for the Responses and Parameters in the Joint Phase-Space of Responses and Parameters

The MaxEnt Gaussian $p_2\left(x|x^0,C\right)$ has a quadratic form and hence possesses a single saddle point, which is determined by requiring the first-variation $\mathsf{\delta }Q\left({\mathbf{\alpha }}_D,r_D;\mathsf{\delta }\mathbf{\alpha },\mathsf{\delta }r\right)$ of the exponential term in Equation (121) to vanish at the saddle point $x_D\triangleq \left({\mathbf{\alpha }}_D,r_D\right)$, namely:

$$\mathsf{\delta }Q\left({\mathbf{\alpha }}_D,r_D;\mathsf{\delta }\mathbf{\alpha },\mathsf{\delta }r\right)\triangleq {\left\{\frac{d}{d\epsilon }Q\left({\mathbf{\alpha }}_D+\epsilon \mathsf{\delta }\mathbf{\alpha },r_D+\epsilon \mathsf{\delta }r\right)\right\}}_{\epsilon =0}=\frac{\partial Q}{\partial \mathbf{\alpha }}\mathsf{\delta }\mathbf{\alpha }+\frac{\partial Q}{\partial r}\mathsf{\delta }r=0.$$

(127)

Imposing the requirement indicated in Equation (127) while taking Equation (122) into account yields the following system of equations:

$$\frac{\partial }{\partial \mathbf{\alpha }}\left[-\frac{1}{2}{\left(\begin{array}{c}\mathbf{\alpha }-{\mathbf{\alpha }}^0\\ r-E\left(r^c\right)\end{array}\right)}^{†}{\left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right)}^{-1}\left(\begin{array}{c}\mathbf{\alpha }-{\mathbf{\alpha }}^0\\ r-E\left(r^c\right)\end{array}\right)-\frac{1}{2}{\left[r-E\left(r^m\right)\right]}^{†}{C}_m^{-1}\left[r-E\left(r^m\right)\right]\right]=0,$$

(128)

$$\frac{\partial }{\partial r}\left[-\frac{1}{2}{\left(\begin{array}{c}\mathbf{\alpha }-{\mathbf{\alpha }}^0\\ r-E\left(r^c\right)\end{array}\right)}^{†}{\left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right)}^{-1}\left(\begin{array}{c}\mathbf{\alpha }-{\mathbf{\alpha }}^0\\ r-E\left(r^c\right)\end{array}\right)-\frac{1}{2}{\left[r-E\left(r^m\right)\right]}^{†}{C}_m^{-1}\left[r-E\left(r^m\right)\right]\right]=0.$$

(129)

Recalling that:

$$\frac{\partial \left(x^{†}Ax\right)}{\partial x}=x^{†}\left(A+A^{†}\right)$$

(130)

and applying the result shown in Equation (130) to evaluate the expressions provided in Equations (128) and (129), respectively, yields the following system of equations at the saddle point, denoted as $\left({\mathbf{\alpha }}_D,r_D\right)$:

$${\left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right)}^{-1}\left(\begin{array}{c}{\mathbf{\alpha }}_D-{\mathbf{\alpha }}^0\\ r_D-E\left(r^c\right)\end{array}\right)+\left(\begin{array}{cc}0& 0\\ 0& C_m^{-1}\end{array}\right)\left(\begin{array}{c}{\mathbf{\alpha }}_D-{\mathbf{\alpha }}^0\\ r_D-E\left(r^c\right)\end{array}\right)+\left(\begin{array}{c}0\\ C_m^{-1}\left[E\left(r^c\right)-E\left(r^m\right)\right]\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right).$$

(131)

Solving Equation (131) yields the following intermediate expressions:

$$\left(\begin{array}{c}{\mathbf{\alpha }}_D-{\mathbf{\alpha }}^0+C_{\mathsf{\alpha }r}{C}_m^{-1}\left[r_D-E\left(r^m\right)\right]\\ r_D-E\left(r^c\right)+C_r{C}_m^{-1}\left[r_D-E\left(r^m\right)\right]\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right).$$

(132)

For computational purposes, the results obtained in Equation (132) are recast into the following forms:

$$r_D=E\left(r^m\right)+C_m{\left(C_m+C_r\right)}^{-1}\left[E\left(r^c\right)-E\left(r^m\right)\right],$$

(133)

$${\mathbf{\alpha }}_D={\mathbf{\alpha }}^0-C_{\mathsf{\alpha }r}{\left(C_m+C_r\right)}^{-1}\left[E\left(r^c\right)-E\left(r^m\right)\right].$$

(134)

In view of Equation (125), it follows that “best-estimate predicted” results for the posterior expectation values for the responses and parameters, which will be denoted as $\left(r^{be},{\mathbf{\alpha }}^{be}\right)$, will have the same expressions as shown in Equations (133) and (134), namely:

$$r^{be}\triangleq {\displaystyle \underset{D}{\int }r{p}_2\left(x|x^0,C\right)d\mathbf{\alpha }dr}=r_D=r^m+C_m{\left(C_m+C_r\right)}^{-1}\left[E\left(r^c\right)-E\left(r^m\right)\right],$$

(135)

$${\mathbf{\alpha }}^{be}\triangleq {\displaystyle \underset{D}{\int }\mathbf{\alpha }{p}_2\left(x|x^0,C\right)d\mathbf{\alpha }dr}={\mathbf{\alpha }}_D={\mathbf{\alpha }}^0-C_{\mathsf{\alpha }r}{\left(C_m+C_r\right)}^{-1}\left[E\left(r^c\right)-E\left(r^m\right)\right].$$

(136)

Since the components of the vector $E\left(r^c\right)$, and the components of the matrices $C_r$ and $C_{\mathsf{\alpha }r}$ contain 2nd-order and 3rd-order sensitivities, the formulas presented in Equations (135) and (136) generalize all of the previous formulas of this type found in data assimilation/assimilation procedures published to date (which contain at most first-order sensitivities).

5.2.2. Predicted Best-Estimate Covariances for the Responses and Parameters in the Joint Phase-Space of Responses and Parameters

The second-order moments of the posterior distribution $p\left(\mathbf{\alpha },r\right)=p_r\left(r\right)p_{\mathsf{\alpha }}\left(\mathbf{\alpha }\right)$ comprise the covariances between the best estimated response, which are denoted as $C_r^{be}$, the covariances between the best-estimate parameters, which are denoted as $C_{\mathbf{\alpha }}^{be}$, and the covariances between the best-estimate parameters and responses., which are denoted as $C_{\mathbf{\alpha }r}^{be}$. The expression of the “best-estimate” posterior parameter covariance matrix $C_r^{be}$ for the best-estimate responses $r^{be}$ is derived by using the results given in Equations (133) and (135) to obtain:

$$\begin{gathered} \begin{array}{l}{C}_r^{be}\triangleq {\displaystyle \underset{D}{\int }\left(r-r^{be}\right){\left(r-r^{be}\right)}^{†}p\left(\mathbf{\alpha },r\right)d\mathbf{\alpha }dr}=C_m-C_m{\left(C_m+C_r\right)}^{-1}{C}_m\\ \\ =C_m\left[I-{\left(C_m+C_r\right)}^{-1}{C}_m\right].\end{array} \end{gathered}$$

(137)

As indicated in Equation (137), the initial covariance matrix $C_m$ is multiplied by the matrix $\left[I-{\left(C_m+C_r\right)}^{-1}{C}_m\right]$, which means that the variances contained on the diagonal of the best-estimate matrix $C_r^{be}$ will be smaller than the experimentally measured variances contained in $C_m$. Hence, the addition of new experimental information has reduced the predicted best-estimate response variances in $C_r^{be}$ by comparison to the measured variances contained a priori in $C_m$. Since the components of the matrix $C_r$ contain 2nd-order and 3rd-order sensitivities, the formula presented in Equation (137) generalizes all of the previous formulas of this type found in data assimilation/assimilation procedures published to date (which contain at most first-order sensitivities).

The expression of the “best-estimate” posterior parameter covariance matrix $C_{\mathbf{\alpha }}^{be}$ for the best-estimate parameters ${\mathbf{\alpha }}^{be}$ is derived by using the result given in Equation (136) to obtain:

$$C_{\mathsf{\alpha }}^{be}\triangleq {\displaystyle \underset{D}{\int }\left(\mathbf{\alpha }-{\mathbf{\alpha }}^{be}\right){\left(\mathbf{\alpha }-{\mathbf{\alpha }}^{be}\right)}^{†}p\left(\mathbf{\alpha },r\right)d\mathbf{\alpha }dr}=C_{\mathsf{\alpha }}-C_{\mathsf{\alpha }r}{\left(C_m+C_r\right)}^{-1}{C}_{r\mathsf{\alpha }}.$$

(138)

Both matrices $C_{\mathsf{\alpha }}$ and $C_{\mathsf{\alpha }r}{\left(C_m+C_r\right)}^{-1}{C}_{r\mathsf{\alpha }}$ are symmetric and positive definite. Therefore, the subtraction indicated in Equation (138) implies that the components of the main diagonal of $C_{\mathsf{\alpha }}^{be}$ must have smaller values than the corresponding elements of the main diagonal of $C_{\mathsf{\alpha }}$. In this sense, the introduction of new computational and experimental information has reduced the best-estimate parameter variances on the diagonal of $C_{\mathsf{\alpha }}^{be}$. Since the components of the matrices $C_{\mathsf{\alpha }}$, $C_{\mathsf{\alpha }r}$, and $C_r$ contain 2nd-order and 3rd-order sensitivities, the formula presented in Equation (138) generalizes all of the previous formulas of this type found in data assimilation/assimilation procedures published to date (which contain at most first-order sensitivities).

The expression of the “best-estimate” posterior parameter covariance matrix $C_{\mathsf{\alpha }}^{be}$ for the best-estimate parameters ${\mathbf{\alpha }}^{be}$ and best-estimate responses $r^{be}$ is derived by using the results given in Equations (135) and (136) to obtain:

$$C_{\mathsf{\alpha }r}^{be}\triangleq {\displaystyle \underset{D}{\int }\left(\mathbf{\alpha }-{\mathbf{\alpha }}^{be}\right){\left(r-r^{be}\right)}^{†}p\left(\mathbf{\alpha },r\right)d\mathbf{\alpha }dr}=C_{\mathsf{\alpha }r}{\left(C_m+C_r\right)}^{-1}{C}_m.$$

(139)

As before, since the components of the matrices $C_{\mathsf{\alpha }r}$, and $C_r$ contain 2nd-order and 3rd-order sensitivities, the formula presented in Equation (139) generalizes all of the previous formulas of this type found in data assimilation/assimilation procedures published to date (which contain at most first-order sensitivities).

The expression of the best-estimate covariance matrix $C_{r\mathsf{\alpha }}^{be}$ is derived by performing a sequence of operations similar to that shown in Equation (139) to obtain:

$$C_{r\mathsf{\alpha }}^{be}\triangleq {\displaystyle \underset{D}{\int }\left(r-r^{be}\right){\left(\mathbf{\alpha }-{\mathbf{\alpha }}^{be}\right)}^{†}p\left(\mathbf{\alpha },r\right)d\mathbf{\alpha }dr}=C_m{\left(C_m+C_r\right)}^{-1}{C}_{r\mathsf{\alpha }}={\left(C_{\mathsf{\alpha }r}^{be}\right)}^{†}.$$

(140)

It is important to note from the results shown in Equations (135) through (140) that the computation of the best estimate parameter and response values, together with their corresponding best-estimate covariance matrices, only requires the computation of ${\left(C_m+C_r\right)}^{-1}$, which entails the inversion of a matrix of size $N_r\times N_r$. This is computationally very advantageous, since $N_r\ll N_{\mathsf{\alpha }}$ (i.e., the number of responses is much less than the number of model parameters) in the overwhelming majority of practical situations.

5.2.3. Data Consistency Indicator

As will be shown in the following, the minimum value, $Q_{\min }$, of the functional $Q\left({\mathbf{\alpha }}^{be},r^{be}\right)$ in the exponential in Equation (121), provides a “consistency indicator” which quantifies the mutual and joint consistency of the information available for model calibration. The minimum value, $Q_{min}$, of the functional $Q\left({\mathbf{\alpha }}^{be},r^{be}\right)$ in the exponential in Equation (121) has the following expression:

$$\begin{gathered} \begin{array}{l}{Q}_{\min }\triangleq Q\left({\mathbf{\alpha }}^{be},r^{be}\right)=-\frac{1}{2}{\left(\begin{array}{c}{\mathbf{\alpha }}^{be}-{\mathbf{\alpha }}^0\\ r^{be}-E\left(r^c\right)\end{array}\right)}^{†}{\left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right)}^{-1}\left(\begin{array}{c}{\mathbf{\alpha }}^{be}-{\mathbf{\alpha }}^0\\ r^{be}-E\left(r^c\right)\end{array}\right)\\ \\ -\frac{1}{2}\left[r^{be}-E\left(r^m\right)\right]C_m^{-1}{\left[r^{be}-E\left(r^m\right)\right]}^{†}.\end{array} \end{gathered}$$

(141)

For subsequent matrix algebra, it is convenient to use the following matrix:

$$\left(\begin{array}{cc}{B}_{11}& B_{12}\\ B_{21}& B_{22}\end{array}\right)\triangleq {\left(\begin{array}{cc}{C}_{\mathsf{\alpha }}& C_{\mathsf{\alpha }r}\\ C_{r\mathsf{\alpha }}& C_r\end{array}\right)}^{-1}.$$

(142)

Using Equations (142) and (132) in Equation (141) yields the following result:

$$\begin{gathered} \begin{array}{l}{Q}_{min}=-\frac{1}{2}{\left[E\left(r^c\right)-E\left(r^m\right)\right]}^{†}{\left(C_m+C_r\right)}^{-1}\\ \\ \times \left[C_{r\mathsf{\alpha }}\left(B_{11}{C}_{\mathsf{\alpha }r}+B_{12}{C}_r\right)+C_r\left(B_{21}{C}_{\mathsf{\alpha }r}+B_{22}{C}_r\right)+C_m\right]{\left(C_m+C_r\right)}^{-1}\left[E\left(r^c\right)-E\left(r^m\right)\right]\\ \\ =-\frac{1}{2}{d}^{†}{\left(C_m+C_r\right)}^{-1}d.\end{array} \end{gathered}$$

(143)

where

$$d\triangleq \left[E\left(r^c\right)-E\left(r^m\right)\right]$$

(144)

As the expression obtained in Equation (143) indicates, the quantity $Q_{min}$ represents the square of the length of the vector $d\triangleq \left[E\left(r^c\right)-E\left(r^m\right)\right]$, measuring (in the corresponding metric) the deviations between the experimental and nominally computed responses. The quantity $Q_{min}$ can be evaluated directly from the given data (i.e., given parameters and responses, together with their original uncertainties) after having inverted the covariance matrix $\left(C_m+C_r\right)$. It is also important to note that $Q_{min}$ is independent of calibrating (or adjusting) the original data. As the dimension of $\left[E\left(r^c\right)-E\left(r^m\right)\right]$ indicates, the number of degrees of freedom characteristic of the calibration under consideration is equal to the number $N_r$ of experimental responses. In the extreme case of absence of experimental responses, no actual calibration takes place. An actual calibration (adjustment) occurs only when including at least one experimental response.

It is noteworthy that the expressions presented in Equations (135) through (139) and (143) look superficially similar to the expressions presented in [7,19]. This superficial similarity stems from the fact that the 2nd/3rd-BERRU-PM formalism only includes means and covariances of the model parameters and computed and measured responses. However, contrary to previously published works, the 2nd/3rd-BERRU-PM formalism presented in in Equations (135) through (139) and (143) comprise all of the 2nd-order and 3rd-order sensitivities of the computed responses with respect to the model parameters. None of these 2nd- and 3rd-order sensitivities appear in the works previous to this work. Consequently, the 2nd/3rd-BERRU-PM formalism comprises, as particular cases, the previously published formulas used in Kalman filters, Bayesian linear statistics [10], data adjustment [13,14,15,16,17], data assimilation [8,9] and predictive modeling [7,18,19,20,21,22,23,24,25,26,27].

6. Conclusions

This work has presented the Third-Order Adjoint Sensitivity Analysis Methodology (3rd-ASAM), which enables the efficient computation of the exact expressions of the 3rd-order functional derivatives (“sensitivities”) of a general system response that depends on both the forward and adjoint state functions, with respect to all of the parameters underlying the respective forward and adjoint systems. Such responses are often encountered when representing mathematically detector responses and reaction rates in reactor physics problems. The 3rd-ASAM extends the 2nd-ASAM in the quest to overcome the “curse of dimensionality” in sensitivity analysis, uncertainty quantification and predictive modeling.

Very importantly, the computation of the 2nd-level adjoint functions ${\mathsf{\psi }}_{1,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{2,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{3,j}^{\left(2\right)}\left(x\right),{\mathsf{\psi }}_{4,j}^{\left(2\right)}\left(x\right)$, and of the 3rd-level adjoint functions, ${\mathsf{\psi }}_{1,ij}^{\left(3\right)},\dots ,{\mathsf{\psi }}_{8,ij}^{\left(3\right)}$, is performed by using the same forward and adjoint solvers (i.e., computer codes) as used for solving the original forward and adjoint systems, namely Equations (1) and (6) subject to the corresponding boundary conditions. Thus, solving the 2nd-LASS and 3rd-ASAM would not require any significant “code development,” since the original forward and adjoint solvers (codes) would not need to be modified; only the right-sides (i.e., “sources”) for these solvers/codes would need to be programmed accordingly. Of course, if the response depends only on the original forward or original adjoint function, than only half of the equations underlying the 2nd-ASAM and, correspondingly, the 3rd-ASAM will need to be solved.

This work also presents new formulas that incorporate the contributions of the 3rd-order sensitivities into the expressions of the first four cumulants of the response distribution in the phase-space of model parameters. Using these newly developed formulas, this work also presents a new mathematical formalism, called the 2nd/3rd-BERRU-PM (“Second/Third-Order Best-Estimated Results with Reduced Uncertainties Predictive Modeling”), which combines experimental and computational information in the joint phase-space of responses and model parameters, including not only the 1st-order response sensitivities, but also the complete hessian matrix of 2nd-order second-sensitivities and also the 3rd-order sensitivities, all computed using the 3rd-ASAM. The 2nd/3rd-BERRU-PM formalism uses the maximum entropy principle to eliminate the need for introducing and “minimizing” a user-chosen “cost functional quantifying the discrepancies between measurements and computations,” thus yielding results that are free of subjective user-interferences and generalizing and significantly extending the 4D-VAR data assimilation procedures. Incorporating correlations, including those between the imprecisely known model parameters and computed model responses, the 2nd/3rd-BERRU-PM also provides a quantitative metric, constructed from sensitivity and covariance matrices, for determining the degree of agreement among the various computational and experimental data while eliminating discrepant information. The mathematical framework of the 2nd/3rd-BERRU-PM formalism requires the inversion of a single matrix of size $N_r\times N_r$, where $N_r$ denotes the number of considered responses. In the overwhelming majority of practical situations, the number of responses is much less than the number of model parameters. Thus, the 2nd-BERRU-PM methodology overcomes the curse of dimensionality which affects the inversion of hessian matrices in the parameter space.