1. Introduction
The information obtained by satellites’s sensors account for an increasing proportion of the total information obtained by humans. Then, the sensorweb proposed by NASA, has created an interoperable environment for a diverse set of satellite sensors via the use of software and the Internet. The authors of [
1] point out that the development trend of sensorweb is integrating with other networks. The space information network is the high-level result of the further integration of sensorweb and space-based networks [
2]. As a result of the sensorweb‘s evolution, the space information network is a network system that acquires, transmits, and processes space information in real-time based on a space platform (such as a synchronous satellite, a medium or low orbit spacecraft, a near space vehicle, a drone, a manned aircraft, etc.) [
3]. With the construction and development of the space information network (SIN), several researches about space network models, network information theory, space information cognition, and space network security have been proposed [
4,
5]. However, a comprehensive demonstration and verification systems for these new results are still urgently required. In recent years, a consensus has been formed on the Tracking and Data Relay Satellite System (TDRSS) as the backbone network of SIN [
6]. As an integrated system in orbit, TDRSS is similar to the space information network to a certain extend regarding its structure, function, and service objects, which can provide a feasible method to the demonstration of space information network [
7]. The comprehensive integrated demonstration platform built with TDRSS as the core can be used as a basic network architecture for the current demonstration of space network integration demonstration. It has the advantages of low implementation cost, strong demonstration ability, high scalability and flexibility.
During the construction of a demonstration platform based on TDRSS, on the one hand, it is necessary to consider the evaluation of the demonstration platform itself. On the other hand, new technology test results need new evaluation models and method support. One or two indicators cannot meet the above needs. Therefore, comprehensive evaluation with multiple indicators for the demonstration platform needs to be studied and designed.
The comprehensive evaluation study will face many problems. First, due to the wide range of technical fields involved in the demonstration platform itself and the diversity of applications of the demonstration, the platform-related indicators are numerous and diverse. Second, the attributes of some indicators might overlap, have strong correlation or even repetition, leading to coupling redundancy. Then, as a typical tree structure, the indicator system may have too many lower-level indicators, causing difficulties in adjusting the number of layers. Finally, the subjective scoring by people caused evaluation errors, which often made the evaluation results unacceptable. Especially when there are too large differences among the subjective scores, how to aggregate the opinions of the experts is also a problem. In the field of decision-making evaluation, the first three problems usually appear in the process of constructing the indicator system, and the last problem is common in the application process of the indicator system.
The solution of the problem above in this paper is as follows. For the first three problems, we construct the index system layer by layer with the logical sequence. First, we use cluster analysis to achieve the classification and selection of indicators to determine the first level indicators (criteria layer). Then, we reconstruct the indicator system’s levels to determine the second level indicators (element layer). Finally, the third level indicators are decomposed and optimized to remove coupled redundant indicators, which completes the index system. Aiming at the last problem, the source of the error is analyzed and evaluated, and a combination of the eigenvector method and the geometric average method is adopted to reduce the generation and transmission of the error.
The specific contributions of this paper are as follows.
We built a comprehensive index system consisting of five criteria, 11 elements, and more than 30 indicators, which effectively satisfied the TDRSS-based demonstration platform’s evaluation needs. A universal construction method from the initial indicator set to the comprehensive indicator system is provided.
We proposed an analytic hierarchy process consistency enhancement strategy suitable for group decision-making/evaluation scenarios. Numerical experiments show that the initial value of the strategy is better and has a faster convergence speed, which can effectively improve the consistency of the comprehensive judgment matrix. In addition, in order to reduce the transmission of error in the aggregation process, a flexible method for determining the aggregation coefficient of group experts is given.
The rest of the paper is organized as follows.
Section 2 investigates the research status of the index system construction principles, methods and existing systems, and some new developments of the analytic hierarchy process.
Section 3 gives the index system construction’s key procedures of classification, layering, optimization and feedback. The comprehensive evaluation index system for the space information network demonstration platform is proposed based on the above operations.
Section 4 studies the subjective error caused by the multi-expert scoring in the classical evaluation method—Analytic Hierarchy Process—in group decision-making (AHP-GDM), and gives an improved consistency enhancement method.
Section 5 is a comprehensive evaluation of the effectiveness of the two-generation TDRSS to further verify the practical significance of the evaluation system. A logic diagram of each section is shown in
Figure 1.
3. Construction of Comprehensive Index System
Instead of a simple overlay of the indicators of the existing subsystems, the comprehensive index system is constructed step by step in a logical sequence in this paper. In this section, the complete process of constructing a unified comprehensive index system from the scattered initial indicator set is shown in
Figure 2.
The subsystem indicator sets are used as inputs to gradually implement classification, layering, and optimization. The process is as follows. First, similarities would be found in a large number of initial indicators. Cluster analysis is used to achieve classification and selection of indicators, obtain five types of initial indicator sets, and output first level indicators (criteria layer) of the indicator system. In the second step, on the current three-layer framework, reconstruct the level of the index system and output the second level indicators (element layer) of the index system. The third step is to decompose and optimize the remaining underlying indicators to remove coupling redundancy Other indicators and output third level indicators (indicator layer). After moderately adjusting the mapping relationship between the index layer and the element layer, the construction of the index system is completed.
3.1. Investigation, Screening, and Classification of Initial Indicator Sets
In this part, the screening and classification of the initial indicator set are introduced. Five major criteria of the index system are established, which are called 1st level indicators. These five criteria take into account the functional attributes of the SIN and the design requirements of the demonstration platform. First, the SIN-related systems are investigated, and the initial index set is prepared for the index system. Then, analyze the requirements of the SIN demonstration platform to establish classification rules. Finally, cluster analysis is used to classify and verify the correctness of the classification rules.
3.1.1. Investigation of the Initial Indicator Set
We acquire the important input for the index system construction, by investigating the indicators of SIN’s subsystem. Based on previous research, we investigated representative indicators of systems including mobile communication systems, terrestrial IP networks, space-based remote sensing, navigation, and measurement and control, etc. The results are shown in
Table 1.
3.1.2. Establishing Indicator Classification Criteria
After the initial indicators are obtained, they need to be classified according to the similarities between the indicators. This commonality is the core criterion of the comprehensive index system and provides classification criteria for subsequent cluster analysis. In this paper, we summarize the commonality from the development needs of the SIN, and then verify its rationality by cluster analysis.
By analyzing the typical application scenarios of relay satellite systems (including space test verification scenarios) and the development trend of SIN, the following requirements are obtained.
Real-time transmission and rapid response-described by time-domain related indicators.
High-efficiency and large-capacity-described by frequency-domain and data quality-related indicators.
Reliability and stability-described by system security-related indicators.
Airspace and service coverage-described by antenna coverage performance related indicators.
The measurement requirements required for the demonstration test-described by accurate measurement and simulation-fit related indicators.
With reference to the naming rules of the ADC model of the United States Weapons System Effectiveness Industry Advisory Committee (WSEIAC), the five categories of index criteria mentioned above are named coverage, timeliness, capability, accuracy, and dependability. They comprise the first level in the comprehensive index system.
3.1.3. Commonality Verification with Preliminary Mapping
The core goal of this step is to verify the foregoing classification criteria through cluster analysis. Based on the classification criteria, the indicators and criteria are initially mapped. After mapping, the index system has an initial three-tier structure. The processing steps are as follows.
Simple screening of hundreds of initial indicators obtained from the survey, then removing indicators that are obviously not related to SIN.
The selected indicators are then tabulated and distributed to experts for a single round of scoring. The specific score is the distance of a certain indicator compared to the five criteria.
Using calculation software with system cluster analysis function (such as SPSS) to divide the preliminary selection indicators into several categories. The initial mapping of indicators and criteria is then completed according to the tree diagram after classification.
3.2. Establishment and Optimization of Hierarchical Structure
The core work of this section is to follow the empirical principle to establish the appropriate number of structural levels of the index system, so that the “granularity” of the indicators at the same level is uniform. Meanwhile, the outdegree of the indicators with affiliation between the upper and lower levels should be moderate. And in the process, determine the element level indicators, i.e., second level indicators.
After preliminary classification, the current index system has a three-tier structure, which could be described as a three-layer tree. In the practical application, it is usually not desirable for a upper-level indicator to include too many lower-level ones. That means the outdegree of upper level in index system tree should be under-controlled. It is generally considered that an outdegree between 1 and 9 is suitable for the index system. However, aggregating more than forty indicators into five criteria could possibly lead to larger outdegree for certain criteria. In addition, there is also the problem that the range of attributes covered by one indicator, defined as “granularity”, could be largely different among every indicator. It is unreasonable to put indicators with different granularity in the same layer. Moreover, some indicators are considered redundant indicators which have a certain affiliation with other indicators, need to be deleted.
In order to solve the above problems of the current three-layer index system, we add an element layer between the criterion layer and the indicator layer, transforming the architecture to four layers. The method of creating the element layer is described as follows; from top to bottom, the evaluation attributes of the criterion, i.e., the physical meaning of the specific representation, are decomposed to obtain the main element layer elements. For example, the timeliness criterion can be further decomposed into two types—the delay of the network and the delay of the platform system. From bottom to top, the criteria of the index layer of the original system are determined as separable. If they are not separable, they will be included in the index layer of the new system. If they are separable, they will continue to be judged. If the system element layer is not an element, the sub-indexes after decomposition are classified into the new index system. The processing flow is shown in
Figure 3.
3.3. Decomposition and Optimization of Indicators
This section focuses on the problems of redundancy and low recognition in the indicator layer. Each indicator is optimized to meet the basic principles of indicator system construction: independence and testability. Finally, the content of the indicator layer (third level indicators) was determined.
After the criteria layer of the evaluation index system is clearly defined, the following problems still exist in the index layer.
The recognition of some indicators is relatively low (identification degree refers to the ability and effect of a statistical evaluation indicator in distinguishing the value characteristics of each evaluation unit in one aspect, so it is also called discrimination degree).
There are qualitative indicators that are difficult to characterize numerically.
There are redundant and overlapping indicators.
It could be seen that 1 and 2 violate the measurability principle of the indicator system, and 3 is against the principle of independence.
To ensure the measurability of the indicator, it is necessary to enhance the recognition of the indicator or directly remove the indicator with low recognition. A simple and direct method is to use the coefficient of variance of the measured value of the indicator (where the coefficient of variance = Standard deviation/mean) measure the degree of recognition of the index value. For some qualitative indicators that are difficult to quantify, a comprehensive quantitative decomposition method is employed. A single qualitative indicator is decomposed into a plurality of quantifiable and quantitative indicators, such as network handover performance indicators can be jointly characterized by indicators such as signaling overhead, packet loss rate, handover throughput, and handover delay.
In order to ensure the independence and simplification of indicators, redundant and coupled indicators need to be processed. Redundancy in this article refers to the degree of conceptual overlap between indicators. For example, the packet forwarding rate of a router is the same as the actual content of the router’s concept of port throughput. Therefore, one of them can be deleted when used simultaneously. The coupling degree mainly measures the degree of positive/negative correlation between indicators. Redundant indicators are reduced by adjusting the Comprehensive weight of each indicator in the system, where the comprehensive weight is used to characterize the redundancy and coupling between indicators. The input parameters are shown in
Table 2.
The correlation weight
is calculated by Equation (
1).
Comprehensive weight: is sorted according to the size of the comprehensive weight and filtered according to a certain threshold, such as 80%. Therefore, non-critical redundant indicators can be removed.
Finally, the optimized indicators are mapped to the corresponding elements, and the construction of the comprehensive indicator system is completed. According to the US Department of Defense’s definition of information operations: the indicators with dashed line is measures of effectiveness indicators (MOEI), and the indicators with solid line is measures of performance indicators (MOPI) [
30]. The final comprehensive evaluation index system is shown in
Figure 4.
4. Consistency Enhancement Strategy of Ahp in Group Decision-Making
After completing the construction of the comprehensive index system, determining the weight score of each index in the process of specific evaluation is the next key research content. We score the weight of each index based on the principle of AHP in group decision-making. The scoring matrix obtained by comparing the importance of each indicator pairwise is also called the judgment matrix. The authors of [
31] explain how the existing traditional methods would reduce the consistency of the judgment matrix:
The expert evaluation scoring input has a certain subjectivity, which could cause the unreliability of the initial input value of the judgment matrix.
The discrete integer constraint of the traditional scaling mechanism will further amplify and diffuse the initial input error.
For the first point, even if some experts have strong authority in a certain field, they still inevitably have knowledge limitations. Especially in the context of some interdisciplinary assessments, the subjective limitations of experts become more prominent. In addition, each expert may intentionally make a biased evaluation because of different preferences or representative interests.
Regarding the scaling problem in the second point, we introduce the concept of transitivity for the convenience of explanation [
32]. For any indicator a, b, or c, transitivity of importance between indicators can be described by
. Moreover, when
, if there are elements in the judgment matrix satisfying
, it indicates that there is a contradictory evaluation in the judgment matrix. In AHP, the detection of transitivity and the ranking of index importance are also a method to find the contradictory elements of the judgment matrix. In addition, this often means that it is difficult for the traditional scale to accurately describe the importance between indicators during the optimization process of the judgment matrix. For example,
. At this time, it is inappropriate to set both the b/d and c/d scales to 2 due to the [1–9] integer scale—the classic scale in the AHP [
33]. It is more suitable to set b to 2.5. Therefore, during the processing of the judgment matrix in this paper, the fractional scale will not be immediately reduced to the integer scale, thereby weakening the influence of the scale on the consistency.
Based on the above analysis, we decompose the AHP consistency problem in the group evaluation scenario into two sub-problems: (1) How to enhance the consistency of the judgment matrix that does not meet the consistency conditions; (2) how to effectively aggregate judgment matrices from multiple experts. For problem 1, we propose a consistency correction strategy with a better initial value and a faster decline. For problem 2, we give a method for generating the aggregation coefficient combining expert authority and initial deviation. The overall group evaluation strategy process is illustrated in
Figure 5.
4.1. A Consistency Correction Strategy Based on Improved Eigenvector Method
According to the foregoing description, the key solution of consistency enhancement of the judgment matrix is finding error elements and correcting them based on the existing information. The correction strategy proposed in [
31] is only a single correction that could not achieve iterative correction, so the correction effect is limited. Therefore, based on the feature vector method and the theorem of geometric mean induced bias matrix (GMIBM), this paper proposes a consistency correction strategy with better initial value and faster decline. The new strategy relies on the following theoretical foundations.
Perron–Frobenius theorem describes the eigenvalues and eigenvector properties of positive real number matrices. As a special kind of positive real matrix, the positive and negative reciprocal matrix is inherited. As the analytic hierarchy process is an evaluation method constructed based on the positive and negative matrix eigenvectors, the judgment matrix of AHP should also satisfy the PF theorem and related inferences. This theorem is also the theoretical basis for the establishment of the eigenvector method. The PF theorem is described as follows.
Theorem 1. For a given positive matrix A, the only positive vector x and only positive constant c that satisfy is a vector x that is a positive multiple of the Perron vector (principal eigenvector) of A, and the only such c is the Perron value (principal eigenvalue) of A.
The authors of [
21] give three properties and their demonstrations of a perfectly consistent positive reciprocal matrix. These properties provide a theoretical basis for the correction of the error elements of the judgment matrix, and give a new error matrix description method-GMIBM. Based on that, this paper introduces the valuation of GMIBM as the initial value of the revised strategy.
Based on the above theory, this paper proposes a consistency correction strategy based on eigenvector method with better initial elements. Compared with the previous confidence-based strategy in [
31], there are two major advantages: (1) faster descent speed—the speed of the modified matrix’s eigenvalues approaching perfect consistency eigenvalues is faster; (2) stronger adaptability for the judgment matrix with missing elements during rapid assessment. The core idea is to obtain the maximum deviation element through the search of the error matrix. The missing element is automatically determined as the maximum deviation/error element. Then the maximum error element is given a better initial value based on the geometric mean. Finally, the feature vector method iteration loop is enabled until the consistency ratio is met. The steps are shown in the Algorithm 1.
Algorithm 1: The algorithm pseudocode of improved method. |
1:
|
2:
|
3:
|
4:
|
5:
|
6:
|
4.2. Optimization of Aggregation Coefficient Based on Hadamard Convex Combination
Because SIN is a complex network that spans multiple fields, some evaluation objects, such as integration test results or technical verification schemes, often require scoring by multiple experts in different professional fields to achieve a scientific and comprehensive evaluation. Moreover, this process can be attributed to the category of group decision-making. In this paper, the aggregation sequence of judgment matrix with AIJ is applied for group AHP. That is to say, the judgment matrix is aggregated first, and then the eigenvector of the comprehensive evaluation matrix is obtained.
Common judgment matrix aggregation methods include additive aggregation based on arithmetic mean and multiplicative aggregation based on geometric mean. This paper chooses the latter as the aggregation method. Because, there is a serious flaw in the additive aggregation method: When the distance between elements of two near-coherent matrices at the same position is large, additive aggregation may increase the error of the matrix eigenvalue after aggregation, and the consistency of the aggregated matrix could not be guaranteed. Theorem 2 shows that multiplicative aggregation based on hadamard convex combination can neutralize eigenvalues better and inherit the consistency of each judgment matrix. The proving process can be found in [
24].
Theorem 2. If judgment matrices are consistent, their hadamard convex combination are consistent.
Hadamard convex combination defined as:
The scoring of the experts in the group decision-making process is usually independent. On the one hand, considering the existence of authoritative experts, their scores will have a higher weight. On the other hand, it is necessary to correlate the degree of deviation of the consistency of the initial matrix with the weight of the expert, considering that there is a correlation between the cognitive degree and the consistency when the expert gives the judgment matrix. Based on the above reasons, this paper proposes a new weighting coefficient for the hadamard convex combination (also called the hadamard product), which could be described with Equation (
2).
n is the number of experts, and
is the authority coefficient of the i-th expert, which is usually subjectively scored by the evaluator. This paper presents a generation method based on authoritative ranking, which is equivalent to increasing the authoritative coefficient by a given step size d (we could set
).
is the deviation coefficient of the i-th judgment matrix. The smaller the corresponding consistency index CI, the larger the deviation coefficient. At the same time, the distribution coefficients of
and
also provide strong flexibility. In particular, when
= 1 and
= 0, the hadamard coefficient is a single expert authority coefficient, while when
= 0 and
= 1, the hadamard coefficient is determined in full compliance with the initial consistency of each expert. Such an improvement factor not only reflects the authority of experts, but also takes into account the consistency of the comprehensive matrix.
6. Conclusions
In the process of evolution from sensorweb to space information network, the core technologies involved in remote sensing, navigation, communication, and other network systems urgently need to be demonstrated and verified by demonstration platform. To satisfy the evaluation needs of the SIN demonstration platform based on TDRSS, this paper proposes a comprehensive evaluation design for this demonstration platform. As the first partial work involving a comprehensive evaluation design, a comprehensive index system is built. A universal method for constructing the comprehensive index system from the initial indicator set to index system is provided, which has certain promotion value. As another important part work of comprehensive evaluation design, we propose an analytic hierarchy process consistency enhancement strategy suitable for group evaluation scenarios. Numerical experiments show that the strategy in this paper has a better initial value than the classical eigenvector method, and thus obtains a faster convergence rate. Through the evaluation example of two generation TDRSS, we verify the feasibility of this evaluation strategy.
The index system construction method and AHP improvement strategy proposed in this paper have certain promotion value and practical significance, and can be used for the comprehensive evaluation of other network systems (such as wireless sensor networks). The next work is to explore the construction of the SIN index system based on this paper’s work. We need to strengthen theoretical research about evaluation method, and try to remove dependence on feature vectors.