A Sample Average Approximation Approach for Stochastic Optimization of Flight Test Planning with Sorties Uncertainty

Abstract

In the context of flight test planning, numerous uncertainties exist, encompassing aircraft status, number of flights, and weather conditions, among others. These uncertainties ultimately manifest significantly in the actual number of flight sorties executed, rendering high significance to engineering problems related to the execution of flight test missions. However, there is a dearth of research in this specific aspect. To address this gap, this paper proposes an opportunity-constrained integer programming model tailored to the unique characteristics of the problem. To handle the uncertainties, Sample Average Approximation (SAA) is employed to perform oversampling of the uncertain parameters, followed by the Adaptive Large Neighborhood Search (ALNS) algorithm to solve for the optimal solution and objective function value. Results from numerical experiments conducted at varying scales and validated with diverse sampling distributions demonstrate the effectiveness and robustness of the proposed methodology. By decoding the generated execution sequences, comprehensive mission planning schemes can be derived. This approach yields sequences that exhibit commendable feasibility and robustness for the flight test planning problem with sorties uncertainty (FTPPSU), offering valuable support for the efficient execution of future flight test missions.

1. Introduction

1.1. Background and Motivation

Flight testing is a scientific experiment activity that systematically evaluates and verifies the test aircraft under real flight conditions. This phase is a crucial component of the research, development, improvement, and finalization of aviation equipment, involving comprehensive inspections of various parts such as the aircraft platform, engine, avionics systems, and weapon systems [1]. It plays an irreplaceable role in enhancing the quality and performance of aircraft and promoting innovative development in equipment technology. It serves as an important bridge between theoretical design and practical application, and it is a key aspect in elevating the level of China’s aviation industry. The Flight Test Planning Problem (FTPP) is a combinatorial optimization problem (COP), which aims to find the optimal planning scheme to allocate various subjects to suitable aircraft and determine their execution order and sortie generation. At the same time, it seeks to enhance subjects execution efficiency, reduce subject completion time, and optimize subjects resource allocation [2]. When planning, multiple constraints need to be considered, such as the logical relationship between subject executions and the technical status of the aircraft [3]. Currently, the field of flight test planning lacks a systematic and theoretical framework as well as mature methodologies as support, which directly leads to a high reliance on manual operations and past experience in key aspects such as demand analysis, planning formulation, optimization and adjustment, and management execution of flight test missions. This significantly constrains the scientific nature, accuracy, and overall efficiency of model flight-testing work [4].

During the actual execution process, due to the complexity and uniqueness of flight test missions, their accompanying test conditions often undergo various changes. Furthermore, there is uncertainty in the status of the test aircraft, which may lead to alterations in the sortie requirements for some subjects, and some test results may be invalid and require retesting. At this point, the actual number of sorties required for each test subject may change due to the aforementioned uncertainties. This question evolved into FTPPSU. In view of this, traditional planning methods have shown significant limitations and are difficult to adapt to the dynamic changes and high-efficiency requirements of flight test missions. Since the maximum number of sorties that each test aircraft can fly in different months is limited, the uncertainty in the number of sorties required for each test subject has a significant impact on the execution duration of the entire mission cycle and the subject allocation for each aircraft. Therefore, during the execution process, relying solely on a baseline schedule obtained from estimated conditions is most likely infeasible. To overcome this situation, a method is needed that can generate more robust flight test planning results, which can largely satisfy the changing sortie requirements of flight test missions and minimize the test mission cycle. However, there has been almost no research on this issue in previous studies, yet it is a problem that must be faced in practical engineering.

1.2. Advantages of the Research

The present study offers several distinct advantages that distinguish it from existing methodologies in the field of flight test mission planning. These advantages, outlined below, underscore the novelty, practicality, and robustness of the proposed framework.

First is the credibility and efficiency of the FTPPSU model with a heuristic solution approach. The introduction of the FTPPSU model, coupled with a heuristic solution method, represents a significant advancement in addressing the complexities of flight test planning. By incorporating uncertainty in sortie requirements, the model captures the inherent variability and unpredictability of flight test missions, thereby enhancing its credibility in real-world applications. Furthermore, the utilization of a heuristic approach, such as the adaptive large neighborhood search framework integrated with the Sample Average Approximation (SAA) method, ensures that the model can be solved efficiently, even for large-scale instances. This combination of credibility and efficiency renders the proposed approach highly suitable for practical implementation in the aviation industry.

Second is the suitability of the SAA method for discrete uncertain parameters. A key strength of the proposed methodology lies in its adept handling of discrete uncertain parameters, specifically the number of sorties required for each test subject. The SAA method, by repeatedly sampling from the probability distributions governing these uncertain parameters, effectively approximates the stochastic optimization problem. This approach is particularly well-suited for discrete uncertainties, as it allows for a more precise fitting of the probability distributions and, consequently, a closer approximation to the true optimal solution. The ability to accurately model and address discrete uncertainties is a crucial advantage, given the discrete and categorical nature of many parameters in flight test planning.

The last is the robustness and practical applicability of the results. The results obtained through the proposed approach exhibit a high degree of robustness and practical applicability. By considering the probability of satisfying sortie requirements and optimizing the mission planning under uncertain conditions, the FTPPSU model produces planning solutions that are not only optimal based on current estimates but also resilient to potential changes in sortie demands. This robustness ensures that the planning scheme remains adaptable and effective even in the face of unforeseen circumstances. Additionally, the strategy for constructing subject execution sequences, which takes into account logical dependencies, duration requirements, and aircraft compatibility constraints, provides valuable insights for organizing and prioritizing subjects. The resulting plans are thus not only theoretically sound but also practically feasible, making them highly useful for flight test program designers and decision-makers.

1.3. Contributions and Paper Organization

In this paper, a Mixed Integer Linear Programming (MILP) model [5,6] with chance constraints is proposed for modeling the FTPPSU, considering the uncertainty in the number of sorties required for test subjects due to the installation status of the test aircraft and actual flight conditions. This model aims to minimize the mission cycle under uncertain conditions and make the planning scheme as adaptable as possible to changes in the demand for sorties. Subsequently, the sample average approximation [7] method is applied to solve the flight test mission planning problem with sortie uncertainty. The FTTPSU integrates the variable nature of sortie requirements into the planning process, acknowledging that factors such as aircraft maintenance, weather conditions, and unforeseen technical issues can impact the number of sorties actually completed for each test subject. By modeling this uncertainty, the proposed model seeks to produce a planning solution that is not only optimal based on current estimates but also resilient to potential changes in sortie requirements. The SAA method is leveraged to handle the inherent stochasticity of the problem by repeatedly sampling from the probability distributions governing the uncertain parameters (in this case, the number of sorties required). Each sample results in a deterministic MILP instance, which is then solved to optimality. The resulting solutions are averaged or aggregated to obtain an approximate solution to the original stochastic optimization problem. This approach provides a trade-off between computational tractability and solution quality, enabling the development of robust planning solutions that can better navigate the uncertainties inherent in flight test missions. The main contributions of this paper are as follows.

• Establishment of a Chance Constrained Optimization Model for Flight Test Planning Problem with Sortie Uncertainty: This paper proposes a novel opportunity-constrained optimization model that incorporates the uncertainty in the number of sorties required for each test subject. This model acknowledges the dynamic and unpredictable nature of flight test missions and aims to optimize the mission planning under these uncertain conditions. By considering the probability of satisfying the sortie requirements, the model provides a more realistic and robust planning approach compared to traditional deterministic models;
• Integration of the SAA Method into an Adaptive Large Neighborhood Search Framework for Model Solution: This paper presents an innovative solution approach that combines the SAA method with an adaptive large neighborhood search framework. The SAA method is used to handle the stochastic nature of the problem by repeatedly sampling from the probability distributions of uncertain parameters. The resulting deterministic instances are then solved using an adaptive large neighborhood search algorithm, which is designed to explore the solution space efficiently and find high-quality solutions. This hybrid approach leverages the strengths of both methods, enabling the model to be solved effectively and efficiently;
• Obtained a Strategy for Constructing the Sequence of Subject Execution: This paper presents a strategy for constructing effective subject execution sequences, taking into account the logical dependencies, duration requirements, and aircraft compatibility constraints. This strategy provides valuable insights into how to organize and prioritize subjects in the face of uncertainty, ensuring that the test program is executed efficiently and effectively;
• Management Insights and References for Flight Test Program Design: The research findings and proposed methodology provide valuable management insights and references for flight test program design. The proposed model and solution approach can help decision-makers make more informed and data-driven decisions, ultimately leading to more efficient and effective flight test programs. Additionally, the study contributes to the broader field of project management and operations research by highlighting the challenges and opportunities in managing complex and uncertain projects such as flight test missions.

The remainder of the paper is organized as follows. Section 2 reviews the works on the related problems, and Section 3 states the formulation of the problem. The proposed solution algorithm is introduced in Section 4. A case study is accessed in Section 5 to show the results and advantages of the consideration of stochastic factors, followed by Section 6 as a conclusion of the study.

2. Literature Review

There is a scarcity of papers and reports directly related to FTPPSU. However, recently, some domestic and international scholars have published research on related topics such as deterministic problems and dynamic programming.

Yan et al. [8] addressed the global optimization problem of flight test missions by proposing a hybrid genetic algorithm that integrates human expertise with computational power. By leveraging both artificial experience and computational capabilities, they were able to generate optimized flight test schedules and further achieve global optimization of the subjects. Shen et al. [9] established a mathematical model for calculating transitional fuel consumption and time and investigated the optimal sequencing problem of flight test subjects by combining the EASTMAN method with the nearest neighbor heuristic approach. Through simulation, Shen et al. [10] found that fuel consumption and time consumption, cost reduction and efficiency improvement could be achieved by rationally arranging the sequence of test flight subjects. Liu [11] conducted research on the automatic generation of aircraft allocation schemes for flight test subjects under multiple constraints based on an improved genetic algorithm. Xu et al. [12], aiming at the problem of planning and scheduling of test points, established a multi-level optimization model of subject assignment and sequencing, proposed an improved genetic algorithm (FTTSOIGA) for subject sequences to solve it, and achieved good results on the tasks of 3 testing machines and 80 test points. Tian et al. [13] established a subject scheduling model with task timing relationship and duration based on the flight test scheduling problem and used an improved genetic algorithm and mixed integer linear programming method to solve the task at different scales. In the subsequent work, Tian et al. [14] proposed a prediction–response strategy based on deep reinforcement learning to solve the rescheduling problem in the flight test from the perspective of dynamic programming, aiming at the subject scheduling problem considering aircraft grounding. Zieja et al. [15] built an artificial neural network (ANN) to improve the management of aircraft flight test plans based on military aircraft test data and proposed an ANN-based aircraft prototype test management system model to improve the adaptability and accuracy of flight test plans. Kwon et al. [16] detailed factors such as flight test plan, test types, main flight test area, and overview of flight test results and then optimized the number of test points from 8500 to 7800, realizing the process of the South Korea Utility Helicopter (KUH) flight test optimization. Wiltshire et al. [17] studied the scheduling dynamics of NASA Exploration Flight Test 1 (EFT-1) activities, used the potential change score hybrid modeling and multi-level modeling methods to analyze the data, and extracted the attractor dynamics in the scheduling data. They found three major flight test mission scheduling modes and discussed how mission dependence affected the presentation of a given mode possibility of a formula.

To the best of our knowledge, although there are currently no research papers specifically addressing the FTPP with uncertainty, several typical COPs similar to our concern have been well-studied in the literature, such as location problems under uncertainty [18], supply chain problems [19], and routing problems [20]. Specifically, when tackling optimization problems with uncertainty, methods like stochastic optimization [21], robust optimization [22], and distributionally robust optimization [23] have been widely adopted in the literature, tailored to the unique contexts in which the problems arise.

Stochastic programming develops from deterministic programming due to the introduction of random variables. This problem was first proposed by Dantzig et al. [24] who described the two-stage stochastic programming problem. In the following year, taking the aircraft route assignment problem of linear programming under uncertain customer demand as an example, the research results of the stochastic programming problem were applied [25]. With the study of natural convexity in stochastic programming problems, Shapiro et al. [26] make a careful analysis of stochastic dual dynamic programming (SDDP) and extend it to the field of risk handling [27]. Powell et al. [28] review the canonical models of these communities and propose a universal modeling framework that encompasses all of these competing approaches.

SAA is a commonly used method to deal with uncertain programming problems. Since it was proposed by Kleywegt et al. [7], it has been applied to solve models, including chance constraints [29] and random dominance constraints [30]. Verweij et al. [31] conducted a detailed calculation study on the application of SAA to solve three types of random path problems, and the calculation results showed that the method could successfully solve more than 21694 scenarios with an optimal performance of less than 1%. Mancilla et al. [32] developed an algorithm for sorting and scheduling single-resource random appointments with waiting time, idle time, and overtime costs and expressed the problem as a random integer programming using the sample average approximation approach. They proposed a heuristic solution based on Benders decomposition, compared it with the exact method and the previously proposed method, and obtained good results. Schütz et al. [33] modeled the supply chain problem as a two-stage stochastic programming problem, solved the problem with the combination of SAA and dual decomposition, and provided the calculation results under different sample sizes and different data aggregation levels in the second stage.

3. Problem Statement and Formulation

In this section, an optimization model for FTPPSU allocation and sequencing is established. FTPPSU involves scheduling hundreds of subjects across multiple test aircraft, necessitating the consideration of numerous constraints. Firstly, the logical relationships among subjects must be satisfied, meaning that each subject can only be processed after the completion of its predecessor tasks. Otherwise, the test results would be invalid. It is noteworthy that subjects need to be executed in accordance with their given logical relationships, not necessarily in sequential order, and the duration of a subject is influenced by two factors: the number of sorties required for its execution and the maximum number of sorties the test aircraft can perform. For the aircraft level, due to variations in aircraft configurations, each subject can only be tested on compatible aircraft. Additionally, owing to manufacturing constraints, test aircraft cannot be simultaneously commissioned for use. Consequently, the arrival time of test aircraft, defined as the deployment time, must also be taken into consideration. Furthermore, in the context of FTPPSU, a critical aspect that significantly complicates the scheduling process is the inherent uncertainty associated with the number of sorties required for each subject. To address this uncertainty, FTPPSU must incorporate flexible planning strategies that can accommodate a range of sortie requirements for subjects. This necessitates the development of optimization models that can handle stochastic or probabilistic inputs, where the number of sorties for each subject is represented as a distribution or a range of possible values.

3.1. Notation

The notations used for parameters and variables are provided in

Table 1. Notations used in this paper.

	Symbols	Definition
Parameters	N	Total number of subjects, $\left|I\right|=N$
	H	Total duration of a subject period
	K	Set of all aircraft
	I	Set of all subjects, $\left|I\right|=N$
	B	A given positive number that is large enough, $B=1000$
	i	Subjects that need to be performed by aircraft, $i\in \mathbf{I}=\{1,2,\dots ,N\}$
	$m{s}_i$	Milestone of subject i
	$L_i$	Sorties of subject i need to be completed
	k	Type of aircraft, $k\in \mathbf{K}=\{1,2,\dots ,K\}$
	${\mathbf{S}}_{\mathbf{i}}$	Set of aircraft that is capable of performing subject i, ${\mathbf{S}}_{\mathbf{i}}\in \left\{k\right|k\in \mathbf{K},p_{ik}=1\}$
	$t_0\left(k\right)$	The deployment time of the aircraft of type k, $t\in [0,H]$
	$g_k\left(t\right)$	The maximum number of the sorties aircraft k can perform in month t
	t	Actual duration of H
	$r_{ik}$	Binary variable that is equal to 1 when subject i can be performed on the aircraft of type k, and 0 for otherwise, $p_{ik}\in \{0,1\}$
Decision variables	$x_{ikt}$	Binary variable that is equal to 1 when subject i performs on aircraft k in month t, and 0 for otherwise, $x_{ikt}\in \{0,1\}$
	$y_{ikt}$	The number of sorties that perform on aircraft k for subject i in month t, ${\sum }_t{\sum }_k{y}_{ikt}=L_i$
	$s_{it}$	Binary variable that is equal to 1 when subject i begin at month t, and 0 for otherwise, $s_{it}\in \{0,1\}$
	$e_{it}$	Binary variable that is equal to 1 when subject i end at month t, and 0 for otherwise, $e_{it}\in \{0,1\}$

as follows.

3.2. Problem Description and Assumptions

In FTPPSU, we provided a directed acyclic graph $\mathit{G}=(E,V)$ to illustrate the temporal relationships between subjects. Each vertex $i\in V$ represents a subject i with needed sorties $L_i$. On account of the uncertainty of sorties for each subject, this article uses $\widetilde{L_i}$ instead of $L_i$. In this case, if $i,l\in V,l\ne i$ and $(i,l)\in E$, it indicates that there is a timing relationship between i and l, and l cannot be executed until i has finished. The following assumptions are made in this study:

• This paper considers that all sorties performed by the aircraft are valid sorties, and there is no need to re-execute;
• All sorties of each subject must be completed;
• Under the premise of not exceeding the maximum number of sorties of each aircraft per month, different subjects can be executed on the same aircraft.
• The production schedule, technical status and availability of the aircraft can be determined in advance according to the test flight outline, so the maximum number of flights per aircraft in different months is known.

3.3. Objective Functions and Constraints Statement

As mentioned at the beginning of Section 3, the objective function in this paper is to optimize the total completion time $T\in [0,H]$ in the given planning time domain $[0,H]$. The total planning time T should be the time from the beginning of the first subject to the completion of the last subject.

The FTPPSU problem essentially involves the system uncertainty and the practical constraints on mission performance. The mathematical expressions of these constraints are as follows:

(1) Flight satisfaction constraint:

The number of flights required for each subject must be satisfied to a certain degree of confidence.

$${\widehat{y}}_{ikt}⩽x_{ikt}·B,\forall i\in \mathbf{I},\forall k\in \mathbf{K},\forall t\in [0,H]$$

(1)

$$\left\{\begin{array}{c}\mathrm{Pr}\{\mathit{A}\widehat{\mathit{y}}=\tilde{\mathit{L}}\}⩾1-\alpha \\ {\sum }_{t=0}^{t=H}{\sum }_{k=1}^{k=K}{\widehat{y}}_{ikt}=\widetilde{L_i},\forall i\in \mathbf{I}\end{array}\right.$$

(2)

where Formula (1) constrains the relationship between ${\widehat{y}}_{ikt}$ and $x_{ikt}$, Formula (2) indicates that all subjects must be satisfied with a confidence level of $\alpha$. The random variable $\widetilde{L_i}$ represents the actual number of sorties to be performed for each subject, and $\alpha \in (0,1)$ is a given confidence level.

(2) Configuration matching constraint:

Each subject can only be performed on an aircraft that matches its configuration.

$$x_{ikt}⩽r_{ik},\forall i\in \mathbf{I},\forall k\in \mathbf{K},\forall t\in [0,H]$$

(3)

(3) Aircraft manufacture constraint:

Subjects cannot be arranged before the selected experimental aircraft is manufactured.

$${\sum }_{t=0}^{t=t_0\left(k\right)}{\sum }_{i\in I}{x}_{ikt}=0,\forall k\in \mathbf{K}$$

(4)

(4) Flight intensity constraint:

In each month t, the flight intensity assigned to each aircraft shall not exceed the maximum sorties of the aircraft $g_k\left(t\right)$.

$${\sum }_{i=1}^{i=N}{\widehat{y}}_{ikt}⩽g_k\left(t\right),\forall k\in \mathbf{K},\forall t\in [0,H]$$

(5)

where $g_k\left(t\right)$ is a fixed value, and ${\widehat{y}}_{ikt}$ changes depending on the situation.

(5) Milestone constraint:

Some subjects need to be completed before a milestone.

$${\sum }_{t}t·e_{it}⩽m{s}_i,\forall i\in \mathbf{I}$$

(6)

(6) Temporal constraint:

Some subjects need to be carried out in accordance with certain logic or preconditions, so it is necessary to ensure that the subject with time sequence constraints can only be carried out after the completion of several previous subjects.

$${\sum }_{t^{\prime }=1}^t{s}_{l{t}^{\prime }}⩾{\sum }_{t^{\prime }=1}^t{e}_{i{t}^{\prime }},\forall t\in [0,H],(i,l)\in \mathit{G}$$

(7)

where i is the pre-subject of l.

(7) Other constraints:

To make the description of the problem more reasonable, this paper lists the added constraints as follows:

$$\begin{array}{}\mathrm{(8)}& & T⩾\max \{e_{it}·t\},\forall i\in \mathbf{I},t\in [0,H]\hfill \mathrm{(9)}& & {\sum }_{t^{\prime }=1}^t{s}_{i{t}^{\prime }}⩾{\sum }_{t^{\prime }=1}^t{e}_{i{t}^{\prime }},\forall i\in \mathbf{I},t\in [0,H]\hfill \mathrm{(10)}& & x_{ikt}⩽{\sum }_{t^{\prime }=1}^t{s}_{i{t}^{\prime }}-{\sum }_{t^{\prime }=1}^{t-1}{e}_{i{t}^{\prime }},\forall i\in \mathbf{I},k\in \mathbf{K},t\in [1,H]\hfill \mathrm{(11)}& & {\sum }_{t=1}^T{s}_{it}=1,\forall i\in \mathbf{I}\hfill \mathrm{(12)}& & {\sum }_{t=1}^T{e}_{it}=1,\forall i\in \mathbf{I}\hfill \mathrm{(13)}& & {\widehat{y}}_{ikt}⩾x_{ikt}\hfill \end{array}$$

Among them, constraint (8) means that the total execution time of the task should be after the completion of the last subject, constraint (9) means that the completion time of each subject should be later than the start time, and constraint (10) means that each subject can only be carried out within its start and end time. Constraints (11)–(12) indicate that each subject can have only one start and end time. Constraint (13) is to limit the specific sorties performed.

To sum up, the whole FTPPSU model can be expressed as:

$$\begin{array}{ccc}\hfill & & \min T\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& & \left(1\right)-\left(13\right)\hfill \end{array}$$

3.4. Complexity Analysis of the Problem

This paper demonstrates the complexity of the problem by relaxing the problem constraints and comparing them with the resource-constrained project scheduling problem (RCPSP).

The basic RCPSP model is as follows:

$$\begin{array}{ccc}\hfill & & \min f_N\hfill \end{array}$$

$$\begin{array}{ccc}& & \min f_N\hfill \\ \mathrm{(14)}& \hfill \mathrm{s}.\mathrm{t}.& & f_0=0\hfill \mathrm{(15)}& & & f_j-d_j⩾f_i,\forall (i,j)\in \mathbf{J},i=1,2,\dots ,N\hfill \mathrm{(16)}& & & \sum _{i=0}^t{r}_{ik}⩽R_k,t⩾0,k=1,2,\dots ,NR\hfill \end{array}$$

where $f_i$ indicates the completion time of the i activity, N indicates the total number of subjects contained in the project, $d_j$ means the duration of the j activity, $r_{ik}$ is the number of resource k required by each phase of task i. $NR$ and $R_k$ denote the total number of resource types and the supply of resource k, respectively. t is the total number of tasks at a certain time. At no time can a project demand for resource k be greater than $R_k$.

For the project with $\mathbf{N}$ subjects labeled $\left|I\right|=N,i=1,2,3,\dots ,I$, the sorties of subject i are donated as $L_i$. Given a graph $G=(V,E)$ with vertex set $V=\{v_1,v_2,\dots ,v_n\}$ and edge set E, this indicates subjects and temporal relationships, respectively. In FTTPSU, Formula (5) and the deterministic form of Formula (2) correspond to Formulas (14) and (15) in the RCPSP problem, and as an auxiliary constraint, Formula (13) can be treated as equivalent directly in the problem. The objective function of the two models also represents the minimum completion time of the whole task cycle, which is also equivalent.

If we relax the configuration matching constraint, the milestone constraint and aircraft manufacture constraint of the subjects in the model as below:

$$\begin{array}{}\mathrm{(17)}& & & r_{ik}⩾1,\forall i\in \mathbf{I},\forall k\in \mathbf{K}\hfill \mathrm{(18)}& & & m{s}_i⩾B,\forall i\in \mathbf{I}\hfill \mathrm{(19)}& & & t_0\left(k\right)\equiv 0,\forall k\in \mathbf{K}\hfill \end{array}$$

At the same time, since the tasks in the RCPSP problem are executed continuously, we add the auxiliary variable $v_{it}$ in this section to indicate whether the subject i is executed in time t and satisfies:

$$\begin{array}{}\mathrm{(20)}& & & v_{it}\in \{0,1\},\forall i\in \mathbf{I},\forall t\in [0,H],\hfill \mathrm{(21)}& & & v_{it}=\left\{\begin{array}{cc}1& \sum _{k\in \mathbf{K}}{x}_{ikt}>0,\forall i\in \mathbf{I},\forall t\in [0,H],\\ 0& \sum _{k\in \mathbf{K}}{x}_{ikt}⩽0,\forall i\in \mathbf{I},\forall t\in [0,H],\end{array}\right.\hfill \mathrm{(22)}& & & v_{it}={\sum }_{t^{\prime }=1}^t{s}_{i{t}^{\prime }}-{\sum }_{t^{\prime }=1}^{t-1}{e}_{i{t}^{\prime }},\forall i\in \mathbf{I},t\in [1,H]\hfill \end{array}$$

When Formulas (16)–(21) is satisfied, the FTPPSU problem is reduced to an RCPSP problem with task execution order, and the RCPSP problem has been proved to be an NP-hard problem [34], so the problem studied in this paper is also NP-hard.

4. Solution Procedure

The process of solving the problem is shown in Figure 1. Through the analysis of the data, the initial information about the subjects, aircraft, uncertainty of sorties and the constraints is obtained. The process of solving this problem mainly includes four parts: model building, the strategy of sequence generation, the SAA method and the Adaptive Large Neighborhood Search (ALNS) algorithm. The mathematical model is covered in Section 3.3, and the remaining three parts will be described in detail in this section.

4.1. Strategy of Sequence Generation

Since the number of sorties required by the subject will directly affect the execution time of the flight test mission (the optimization goal), this variable is uncertain in the FTPPSU problem. Therefore, the result given in this paper is an optimal sequence of subject execution, rather than a definite plan. The execution sequence can be decoded according to the information in the specific execution process, and the flight plan can be obtained in the current situation.

After obtaining the initial sequence, this paper optimizes the sequence using a heuristic algorithm, so that it can decode the execution plan with the confidence level of $\alpha$. In the process of decoding, in order to ensure the feasibility of the solution, it is necessary to verify and adjust the solution according to the constraints. According to the execution order in the sequence, the judgment and operation on the subjects are as follows:

Step1: Determine whether the solution candidate meets the executable condition, including constraints (3) and (6). If yes, go to Step2; otherwise, proceed to the next subject.

Step2: According to constraints (1), (2), and (4), select the aircraft for this subject, the time of execution, and the number of sorties.

Step3: Repeat Step1 and Step2 until all subjects are scheduled.

Step4: Verify the planning results.

After obtaining a correct sequence, decoding is necessary to derive the specific execution plan for each subject. During the decoding process, each subject is judged in order. If a subject has no pre-subject or its pre-subjects have been completed, and an aircraft capable of executing that subject is available, the subject can be assigned. Then, without exceeding the maximum number of flights per aircraft, it is scheduled to an appropriate available aircraft until the required sortie count is reached in chronological order. Finally, the next subject in the sequence is selected and the above process is repeated until all subjects have been assigned.

4.2. Sample Average Approximation

The SAA scheme [7] is a sampling based method for solving discrete stochastic optimization problems, and it can obtain high-quality solutions and provide statistical bounds. The method has been used to solve many stochastic problems successfully, as stated in Section 2. The usability of the SAA method is demonstrated in Appendix B.

The basic idea is that the desired target value of the random problem can be approximated by the corresponding value of the sampling problem. The sampling process is repeated several times to obtain enough information about the solution [35]. In each SAA replication, a random sample of size M is generated by realizing the random vector, i.e., ${\mathit{L}}^{\mathbf{1}},{\mathit{L}}^{\mathbf{2}},\dots ,{\mathit{L}}^{\mathit{M}}$, where each vector consists of the number of sorties that need to be performed for each subject (i.e., ${\mathit{L}}^{\mathit{M}}=\{L_1^M,L_2^M,\dots ,L_N^M\}$). Then, the left part of constraint (2) can be approximated as follows:

$$\begin{array}{cc}\hfill & Pr\{{\sum }_{t=0}^{t=H}{\sum }_{k=1}^{k=K}{\widehat{y}}_{ikt}=\widetilde{L_i}\}\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{I}\{{\mathbb{E}}_{\mathbb{P}}\left[\mathbb{I}\{{\sum }_{t=0}^{t=H}{\sum }_{k=1}^{k=K}{\widehat{y}}_{ikt}={\widehat{L}}_i\}\right]=N\}\right]\hfill \\ \hfill & =\frac{1}{M}{\sum }_{m\in M}\mathbb{I}\{{\sum }_{i\in I}\mathbb{I}\{{\sum }_{t=0}^{t=H}{\sum }_{k=1}^{k=K}{\widehat{y}}_{ikt}={\widehat{L}}_i^m\}=N\}\hfill \end{array}$$

(23)

where $\mathbb{I}(·)$ is the indicator function, $\mathbb{P}$ is the log of the variable and is distributed according to truncated normal distribution, and $\psi (·)$ stands for the truncated normal distribution function.

This paper conducted a total of Q independent samples, each time extracting M groups of results, and each result ${\mathit{L}}^{\mathit{M}}$ is an n-dimensional vector, representing the number of sorties required for each subject. By solving the associated SAA problem, the objective values $T_1^M,T_2^M,\dots ,T_Q^M$ and a set of candidate solutions ${\widehat{\mathit{\eta }}}_1^M,{\widehat{\mathit{\eta }}}_2^M,\dots ,{\widehat{\mathit{\eta }}}_Q^M$ are obtained. Each $\mathit{\eta }$ is a $1\ast n$ sequence that represents the priority of the subject’s execution and can be decoded to obtain the specific execution plan of the subject. Among all the Q sampling, we define the ${\overline{T}}_{Q,M}$ as the average value of the optimized objective function:

$${\overline{T}}_{Q,M}=\frac{1}{Q}{\sum }_{q=1}^Q{T}_q^M$$

(24)

It was proved in Theorem 1 of [36] that the ${\overline{T}}_{Q,M}$ is a statistical estimate for a lower bound of $T^{\ast }$, where $T^{\ast }$ is the optimal value of the objective function for the original problem.

Also, for any feasible solution, $\widehat{\mathit{\eta }}$ provides an upper bound ${\widehat{T}}_{\widehat{\mathit{\eta }},M}$ for $T^{\ast }$. This paper typically generated a problem with a larger sample size of $M^{\prime }$, where $M^{\prime }\gg M$ and $T_{\widehat{\mathit{\eta }},M^{\prime }}$ becomes the unbiased estimation of objective function, which means $\mathbb{E}\left[T_{\widehat{\mathit{\eta }},M^{\prime }}\right]>T^{\ast }$, also $\{{\xi }^1,{\xi }^2,\dots ,{\xi }^{M^{\prime }}\}$ is a sample of size $M^{\prime }$. The solution is then evaluated by the optimization gap ${ϵ}_{M,M^{\prime }}\left(\widehat{\mathit{\eta }}\right)$ and the variance of the optimization gap ${\sigma }_{ϵ}^2\left(\widehat{\mathit{\eta }}\right)$.

$${ϵ}_{M,M^{\prime }}\left(\widehat{\mathit{\eta }}\right)=T_{\widehat{\mathit{\eta }},M^{\prime }}-{\overline{T}}_{Q,M}$$

(25)

$${\sigma }^2\left({\overline{T}}_{Q,M}\right)=\frac{1}{Q(Q-1)}{\sum }_{q=1}^Q{(T_q^M-{\overline{T}}_{Q,M})}^2$$

(26)

$${\sigma }^2\left(T_{\widehat{\mathit{\eta }},M^{\prime }}\right)=\frac{1}{M^{\prime }(M^{\prime }-1)}{\sum }_{m=1}^{M^{\prime }}{({\widehat{T}}_{\widehat{\mathit{\eta }},M^{\prime }\left({\widehat{L}}_{{\xi }^m}\right)}-{\overline{T}}_{\widehat{\mathit{\eta }},M^{\prime }})}^2$$

(27)

$${\sigma }_{ϵ}^2\left(\widehat{\mathit{\eta }}\right)=({\sigma }^2\left({\overline{T}}_{Q,M}\right)+{\sigma }^2\left(T_{\widehat{\mathit{\eta }},M^{\prime }}\right))$$

(28)

where ${\widehat{T}}_{\widehat{\mathit{\eta }},M^{\prime }\left({\widehat{L}}_{{\xi }^m}\right)}$ represents the value of the objective function T when the total number of samples is $M^{\prime }$, the sortie data are the result ${\widehat{L}}_{{\xi }^m}$ of the $m^{th}$ sampling, and the solution is $\widehat{\eta }$.

When implementing the SAA algorithm specifically, this paper considers it effective if ${ϵ}_{M,M^{\prime }}\left(\widehat{\mathit{\eta }}\right)/T_{\widehat{\mathit{\eta }},M^{\prime }}⩽3\%$ and ${\sigma }_{ϵ}^2\left(\widehat{\mathit{\eta }}\right)/T_{\widehat{\mathit{\eta }},M^{\prime }}⩽5\%$ [37]. The process of SAA is shown in Algorithm 1.

Algorithm 1 Procedure of SAA algorithm

Input: Repetition times of SAA: Q; Sample sizes $M^{\prime }$ and M $(M^{\prime }\gg M)$. Output: Priority sequence of subject execution: $\mathit{\eta }$. 1: for $q=1,2,\dots ,Q$ do 2:     Generate M sampled n-dimensional arrays for each q: ${\mathit{L}}_q^1,{\mathit{L}}_q^2,\dots ,{\mathit{L}}_q^M$ 3:     Get the solution ${\widehat{\mathit{\eta }}}_q^M$ and the objective function ${\widehat{T}}_{Q,M}$ of SAA problem 4:     Calculate the statistical lower-bound ${\overline{T}}_{Q,M}$ 5:     Generate a sample of $M^{\prime }$ and get the upper-bound ${\widehat{T}}_{\widehat{\mathit{\eta }},M}$ 6:     Calculate the optimality gap ${ϵ}_{M,M^{\prime }}\left(\widehat{\mathit{\eta }}\right)$ and the variance of the gap ${\sigma }_{ϵ}^2\left(\widehat{\mathit{\eta }}\right)$ 7:     if ${ϵ}_{M,M^{\prime }}\left(\widehat{\mathit{\eta }}\right)/T_{\widehat{\mathit{\eta }},M^{\prime }}<3\%$ and ${\sigma }_{ϵ}^2\left(\widehat{\mathit{\eta }}\right)/T_{\widehat{\mathit{\eta }},M^{\prime }}<5\%$ then 8:         Invoke the ALNS algorithm (Section 4.3) to find the optimal sequence $\mathit{\eta }$. 9:     else 10:         Increase the sample size 11:     end if 12: end for

4.3. Adaptive Large Neighborhood Search

Given the complexity of the FTPPSU problem, addressing the SAA problem can be challenging for different sampling sizes M. Therefore, we design a heuristic algorithm based on the ALNS (Adaptive Large Neighborhood Search) framework to solve the SAA problem. This approach was initially proposed in article [38] and has achieved successful applications across multiple domains. In the algorithm described in this paper, an initial solution is first obtained using a greedy algorithm, followed by a destroy-and-repair process to generate a new set of solutions. Each destroy and repair operator is assigned a weight value, and the initial operator is selected through a roulette wheel selection method. When a new solution is generated that improves upon the original solution, the optimal solution and the weight of the operator selected in that step are updated. The above process is repeated iteratively until the stopping condition for backtracking is met, at which point the process terminates.

4.3.1. Solution Space and Representation

When applying heuristic algorithms to solve flight test planning problems, the large number of assigned subjects and the wide range of options for selecting available aircraft and flight sorties can lead to a significantly high cost of solving if directly approached based on the problem’s solution requirements. Furthermore, the inclusion of random factors in this problem significantly escalates the difficulty of finding a solution. In this paper, the representation of solutions has been restructured to simplify the solution of the problem as the execution sequence of subjects. Meanwhile, during the search process, each set of destroy and repair operators in ALNS can form a neighborhood action to adjust the subject sequence, and then specific subject execution plans are decoded based on constraints. This design significantly reduces the search space of solutions and improves the execution efficiency of the algorithm.

4.3.2. Design of Destroy Operators

The destroy operator selects a certain proportion of subjects and removes them from the current solution. If the total number of subjects is N, the number of subjects to be removed, h, is calculated as $h=\lfloor \alpha \times N\rfloor$, where $\alpha \in [0.05,0.15]$ represents the proportion of subjects to be removed, and $\lfloor ·\rfloor$ denotes the floor function (rounding down to the nearest integer). Common destroy operators include random point destroy, random segment destroy, and worst removal, among others. The destroy operators used in this article are shown below.

(1) Random removal:

Random destroy operators encompass two distinct approaches. The first randomly selects h subjects from the current solution and removes them. The second approach, given the temporal relationships that exist among many subjects, randomly selects a point and then removes a continuous sequence of h subjects starting from that point. This is done to ensure that the subsequently generated solutions are feasible and meet the requirements. The random destroy operator is relatively simple and easy to implement, and it significantly enhances the diversity of the algorithm’s search, expanding the solution search space. This, in turn, contributes to obtaining higher-quality solutions.

(2) Worst cost removal:

Here, the time consumed by subject i is defined as $t_{cost}\left(i\right)=t_N\left(i\right)-t_{N-1}\left(\overline{i}\right)$, where $t_N\left(i\right)$ represents the total time consumed by N subjects and $t_{N-1}\left(\overline{i}\right)$ represents the same $N-1$ subjects without i. It is necessary to calculate the time consumed by each subject, select the one that consumes the most time, and move it out of the current solution. This step is repeated until the number of removed subjects reaches h.

(3) Worst compatibility removal:

Since each subject can be performed on different aircraft, if a subject can be flexibly arranged on different aircraft, then the overall task completion time of the subject may be relatively smaller. In view of this situation, this paper defines the influence on the objective function as the compatibility of the subject and designs a special destroy operator and applies it to the algorithm. The compatibility of subjects is defined as $comp={\sum }_{k=1}^{S_i}{g}_k\left(t\right)/L_i$, where ${\sum }_{k=1}^{S_i}{g}_k\left(t\right)$ means the sorties that can be performed on different aircraft in month t.

4.3.3. Design of Repair Operators

The repair operator is used to put the result generated by the destruction operator back into the solution according to a certain strategy to form a new result and complete the entire neighborhood action. Then, the feasibility of the solution is judged and adjusted according to the constraints.

(1) Random insertion:

Randomly select a subject from the h subjects removed and insert it into any position in the solution to form a new complete solution. In this way, the search range of the solution can be expanded while the solution is constructed, and more initial solution sets can be formed, which can make the effect of the neighborhood search better.

(2) Greedy insertion:

Randomly select a subject from the h subjects that have been removed and calculate the increment of the objective function caused by inserting it at each position in the solution. In this problem, since the time unit is months, the insertion of different subjects may cause change in the month. This paper defines the characterization parameter ${\delta }_i=\Delta t+(T_{current}-\sum s_{it}/T_{current})$ of the object increment caused by subject i.

(3) Compatibility precedence insertion:

Through the analysis of the problem, we find that when the subjects with relatively low compatibility are executed first, the average execution time of all subjects is shorter, and the objective function is relatively better. Accordingly, we design a compatibility priority insertion operator, which randomly selects one of the q subjects removed, compares its compatibility with all subjects in the current solution, and puts the subject i into the current solution according to its compatibility.

4.4. Judgment and Acceptance Criteria

When a new solution is generated, it needs to be confirmed as to whether the solution meets the constraints.

First, for constraint (1), after every SAA sampling, a deterministic input is obtained, and the corresponding ALNS generates a solution. In each repetition, we sample the uncertain parameters M times. In order to conform to $Pr\{\mathit{A}\widehat{\mathit{y}}=\tilde{\mathit{L}}\}⩾1-\alpha$ in the constraint, we need to ensure that at least $(1-\alpha )M$ of these M times can fully satisfy constraint (1). Only in this way can the obtained solution be considered to satisfy the requirements.

During the decoding process, constraints (2)–(5) in Section 3.3 can be resolved. The algorithm needs to assign each selected subject to an aircraft that satisfies the constraints (2) and (3). Then, constraint (4) determines how many sorties can be configured on the corresponding aircraft. After decoding is complete, it can be confirmed whether the solution satisfies constraint (5), and if it does not, the subject that violates the constraint is adjusted forward in the sequence.

For the temporal relation in constraint (6), the traversal and adjustment method is not suitable because there are many subjects with temporal relation. Therefore, it is necessary to make the sequence of the generated solutions conform to the temporal constraint when the neighborhood action is performed on the encoded solution. The specific implementation process is as follows:

Step1: The reachable matrix ${\mathbb{A}}^{n\times n}$ is constructed according to the temporal relationship between the subjects, then element $a_{ij}=1$ indicates that subject i must be executed before subject j; otherwise, there is no strict temporal relationship between the two subjects.

Step2: According to the matrix $\mathbb{A}$ obtained in Step1, for each subject i, if there is a temporal relationship, find $\max \left\{m\right|a_{mi}=1,m<i\}$ and $\min \left\{m^{\prime }\right|a_{i{m}^{\prime }}=1,m^{\prime }>i\}$. At this time, a maneuverable range $[m,m^{\prime }]$ of subject i is formed, which means subject i can maneuver at different positions between m and $m^{\prime }$.

Step3: After forming an initial sequence of subjects, the ALNS algorithm is used to select operators for destruction and repair. When we use the repair operator, let each subject be repaired according to the repair strategy within the maneuver interval of subjects formed in Step2.

Through the above process, the generated solution can be guaranteed to be feasible to avoid the infeasible solution caused by the randomness of the repair point, and then the traversal adjustment will lead to a decrease in the solution efficiency.

In order to avoid the algorithm becoming trapped by a local optimum, the judgment criterion of the simulated annealing algorithm proposed by Adulyasak et al. [39] is used in this paper to judge whether the generated non-improved solution is accepted. Therefore, when a non-improved solution is generated during iteration, its probability of acceptance is:

$$p=e^{{\displaystyle -\frac{f\left({\eta }^{\prime }\right)-f\left(\eta \right)}{T}}}$$

(29)

where $\eta ,{\eta }^{\prime }$ are the current solution and the neighborhood solution, respectively, and T is the temperature corresponding to this iteration. In this paper, the initial temperature T is set. As the number of iterations increases, the temperature decreases as $T_n=a·T_{n-1}$, where a is the cooling rate.

5. Numerical Experiments

In this section, we conduct numerical experiments to evaluate the performance of the introduced SAA and ALNS framework on the generated instances. The experiments are divided into two parts: (1) First, this paper will verify the effectiveness of SAA sampling in dealing with uncertainties in this problem through the indicators given in Section 4.2. (2) In this part, the paper will verify the effectiveness and robustness of the FTPPSU method to ensure that the proposed method can better solve the flight test planning problem with sorties uncertainty.

5.1. Experimental Configuration

5.1.1. Data Construction

Due to the sensitivity of the data, actual project case data are not presented in this paper. Based on the characteristics of the overall mission planning of the flight test, random examples with scales of 2 aircraft 20 subjects, 3 aircraft 50 subjects, and 6 aircraft 100 subjects are generated in this paper to test the algorithm. Let the maximum number of executable flights for each test aircraft per month be 8. The attributes contained in the data set are: subject number, subject code, required number of flights, executable test aircraft, pre-requisite subjects, and milestone nodes. The subject number is the identification of each subject generated according to the timing logic of the subjects, in which the subject number with a preceding timing relationship must be less than the one with a later timing relationship. The required number of flights is the total number of flights required to complete the subject, which can be executed on test aircraft of different configurations in different months. In flight test activities, most subjects require between 4 and 8 flights, while a small number of subjects only require 1 flight or more than 10 flights. In this paper, when determining the flight requirements, a random number generator is used to generate random integers that conform to the truncated normal distribution $\mathbf{X}=\psi (6,4,1,11;x)$ as the number of flights for each subject, where x represents the value in distribution $\mathbf{X}$. The parameters in $\psi (·)$ from the first to fourth indicate the mean, variance, minimum and maximum values of the sample.

In the flight test planning problem, the uncertainty in the number of flight sorties primarily arises from factors that can lead to an increase in the required sorties. As indicated in Equation (13), the increment in the number of sorties follows a logarithmic normal distribution $\mathbb{P}$. When the initial number of sorties for subject i is $L_i$, we define the range of the increment to be within $[0,max(1,L_i/4)]$ and follow the distribution $\mathbb{P}$.

5.1.2. Parameters Setup

During the adaptive large neighborhood search, the destruction ratio of the destruction operator is set to 0.12, with both the initial weights and scores of the destruction operator and the repair operator being 1. The evaporation coefficient is set to 0.5, and the maximum number of iterations is 200. The initial temperature T of the simulated annealing criterion is set to 1000, with a cooling rate of 0.97. Among these parameters, the simulated annealing criterion serves solely as a criterion for determining whether a non-improving solution should be accepted. Additionally, to ensure the repeatability of the experiments and facilitate the adjustment and comparison of models, the random number seed for the experiments conducted in this paper is consistently set to 20.

The computational experiments for the problem described were conducted on a personal computer equipped with an Intel Core i9-13900 HX CPU operating at 2.20 GHz, 16.0 GB of RAM, and a 64-bit Windows 11 operating system.

5.2. Optimality Gap Estimation

Latin Hypercube Sampling (LHS) is a statistical method that is often used to generate near-random samples of parameter values from a multidimensional distribution. By ensuring that each dimension of the parameter space is divided into intervals and that each interval contains exactly one sample point, LHS can provide a more even coverage of the parameter space compared to simple random sampling. During the SAA sampling process, to reduce sampling errors, we used LHS instead of Monte Carlo Sampling [40]. Given that the number of sorties in the problem is a discrete integer, we rounded up the value of each sampling point to serve as the sampling result.

Here, we take the planning problem of 50 subjects across 2 aircraft as an example to estimate the optimality gap and present the results in

Table 2. Estimated optimality gaps.

M′	${\mathit{T}}_{\widehat{\mathit{\eta }},{\mathit{M}}^{\mathbf{\prime }}}$	Gap (%)	Variance
20	22.94	36.56	56.31
50	21.42	28.70	52.38
100	20.71	23.25	45.54
200	19.45	15.78	36.22
500	18.42	9.47	28.57
800	17.58	4.63	24.62
1000	17.03	1.76	16.75
2000	16.88	0.82	9.64

. The data in the table are the average of the results of Q times repeated tests. The results indicate that the estimated optimality gap decreases as the sample size increases. With the increase in the number of scenarios, the estimated optimality gap improves significantly. The optimality gap suggests that a satisfactory solution quality can be achieved when the number of samples is sufficiently large. Meanwhile, the variance gradually decreases with the increase in sampling, indicating that the sample results begin to stabilize as the number of samples increases. Selection and proof of the sample size can be obtained from Appendix A.

5.3. Effectiveness and Robustness Verification

In order to verify the effectiveness of the proposed method for solving the FTPPSU problem, we conducted tests on scale datasets of 20, 50 and 100 subjects, respectively, and compared the validity of the solution (subject sequence) on the deterministic algorithm, the robust optimization algorithm and the chance constraint model proposed in this paper. The specific results are shown in

Table 3. Numerical result of FTPPSU.

N	${\mathit{T}}_{\mathit{Ori}.}\left(\widehat{\mathit{\eta }}\right)$	M	${\mathit{T}}_{\mathit{Det}.}\left(\widehat{\mathit{\eta }}\right)$			${\mathit{T}}_{\mathit{Rob}.}\left(\widehat{\mathit{\eta }}\right)$			${\mathit{T}}_{\mathit{CC}}\left(\widehat{\mathit{\eta }}\right)$
			Mean	Variance		Mean	Variance		Mean	Variance
20 × 2	8	20	10.85	4.73		10.20	4.08		10.45	4.22
		50	9.82	3.06		9.66	2.58		9.74	2.96
		100	9.43	2.25		9.37	1.68		9.03	1.37
		500	8.94	0.63		8.76	0.42		8.72	0.24
		1000	8.58	0.13		8.42	0.04		8.46	0.06
50 × 3	16	20	18.60	11.52		18.40	8.74		18.35	9.85
		50	18.28	8.66		17.64	7.68		17.72	7.59
		100	17.93	5.39		17.43	6.17		17.20	5.17
		500	17.26	3.04		17.25	2.46		17.09	2.88
		1000	17.04	1.37		17.13	1.02		16.86	1.29
100 × 6	14	20	18.60	11.33		18.85	6.63		18.25	10.62
		50	18.28	9.68		18.24	4.83		17.48	7.94
		100	17.83	4.29		18.13	3.22		16.35	4.01
		500	17.19	2.87		17.92	1.45		16.14	2.64
		1000	16.23	1.36		17.64	0.94		15.87	1.12

The content in bold indicates better results.

, where the $T_{Ori.}$ represents the optimal value solved on the problem of determining the initial sorties, and its corresponding solution is $\widehat{\eta }$. $T_{Det.}$ represents the objective function result of the sampled result of the solution obtained through the original problem, while $T_{Rob.}$ and $T_{CC.}$ are the same as in the robust optimization and chance constraint model, respectively. The detailed data preview is shown in Appendix C.

In our rigorous verification process, we systematically derived the sequence of optimal values specific to each of the three methods under consideration. Subsequently, we implemented an extensive M-sampling scheme across the sortie scenarios within the current problem domain. Critically, we incorporated the variance of the mean planning period values, post-decoding according to the derived sequences, into a comparative table, highlighting the nuances of performance.

The experimental outcomes reveal a compelling trend: as the sampling size escalates, the mean and variance of the objective function exhibit a marked decline, underscoring the robustness of our analysis. Notably, the deterministic modeling approaches frequently falter when confronted with inherent uncertainties, emphasizing their limitations in tackling non-deterministic challenges.

When dealing with smaller sample sizes, robust optimization initially yields superior outcomes compared to the chance constraint model introduced herein. This advantage stems from the latter’s inability to capture the comprehensive variability patterns of random variables, leading to notable deviations. Nevertheless, as the sampling scale broadens, the proposed method shines, demonstrating its superiority in solution effectiveness. This trend intensifies with the augmentation of dataset size, underscoring the efficacy of our model and methodology in tackling the complex FTPPSU problem with remarkable precision and adaptability.

In order to verify the robustness of the algorithm, this paper takes a data set of 100 subjects as an example, adopts three methods of uniform distribution, normal distribution and Poisson distribution, respectively, during sampling, and decodes the obtained samples according to the sequence of subjects obtained by this method. The obtained objective function values are shown in Figure 2, where the curve CC represents the method obtained in this paper. The results show that the solution of the proposed method can obtain better objective function values in different distribution samples.

6. Conclusions

In this paper, we have investigated the uncertainty surrounding the required sorties in flight test planning, aiming to bridge a notable gap in the existing literature pertaining to managing uncertainty in such complex testing scenarios. By integrating the strengths of the Sample Average Approximation (SAA) and Adaptive Large Neighborhood Search (ALNS) algorithms within a chance-constrained integer programming framework, we have devised a novel method to tackle the Flight Test Planning Problem with Sorties Uncertainty (FTPPSU). This approach enables us to derive an optimal sequence of test subjects that not only accommodates the inherent uncertainties but also minimizes the overall execution time when subjects are allocated according to the proposed sequence.

Our method has been rigorously evaluated through a series of numerical experiments spanning various scales and sampling conditions, demonstrating its effectiveness in mitigating the uncertainty challenges inherent in flight test planning. These results underscore the robustness of our approach in adapting to diverse scenarios, ensuring its practical applicability in real-world settings.

While this study focuses primarily on addressing the uncertainty related to sorties, we acknowledge that there are numerous other factors, including aircraft arrival times, availability states, and subject execution efficiencies, which may also introduce uncertainties. These avenues represent promising directions for future research, where our framework can potentially be extended to encompass a broader spectrum of uncertainties, further enhancing its comprehensive problem-solving capabilities.

In summary, our work presents a significant contribution by introducing a novel and efficient methodology for flight test planning under uncertainty, which not only advances the state-of-the-art but also demonstrates tangible benefits in terms of reduced execution time and improved operational efficiency.