Are Brazilian Higher Education Institutions Efficient in Their Graduate Activities? A Two-Stage Dynamic Data-Envelopment-Analysis Cooperative Approach

Abstract

Higher education evaluation presents itself as a worldwide trend. It aims to improve performance due to its importance for economic and personal growth. Graduate activities are essential for Brazilian research and innovation systems. However, previous studies have disregarded the importance of this educational level and have evaluated efficiency by jointly considering teaching and research or only undergraduate courses. Therefore, this study contributes to Brazilian reality by proving a national graduate activities efficiency evaluation that considers them as a two-stage system (formative and scientific production stages). The study provides three main methodological contributions by presenting a new centralized two-stage dynamic network data envelopment analysis (DNDEA) model with shared resources. Besides measuring efficiency, an efficiency decomposition based on a leader–follower assumption shows managers how much efficiency can alter when one of the stages needs to be prioritized. Finally, a new framework based on modified virtual inputs and outputs provides a bi-dimensional representation of the efficiency frontier. Results indicate the usefulness of the approach for ranking universities, and the need to improve scientific production, highlighting the negative impacts of COVID-19 on the formative process efficiency and showing no significant regional discrepancies regarding performance.

1. Introduction

Universities represent a driving force of science and knowledge in countries, and they are crucial for providing a skilled and expert workforce in the job market [1]. Evaluating their results is a complex process due to the existence of different indicators to obtain an overview of system performance [2]. Understanding how to increase the universities’ performance is challenging for governments, leading operators, and funders [3,4]. The last decades have shown a worldwide trend for implementing exercises about evaluation and for a comparison of various estimation methods [5].

In Brazil, it is possible to verify the same trend. Law 10.861/2004 instituted the National Higher Education Assessment System (SINAES) in 2004. SINAES aims to improve the results of the Brazilian higher education system and consists of three main components: the performance evaluation of institutions, courses, and students. Higher education censuses occur annually to collect data about the three dimensions to obtain indicators used in SINAES to assess and accredit courses and institutions.

Brazilian graduate courses are responsible for a large part of the research and innovation generated in Brazil [6]. Regardless of its importance, the report on investments in research and development in the world carried out by the United Nations Educational, Scientific, and Cultural Organization (UNESCO) considering 2014 to 2018, shows that the budget reduction of the Ministry of Science, Technology and Innovation (MCTI) in the same period was around 50% [7]. Considering 2012 to 2021, the reduction corresponds to 84% (from BRL 11.5 billion to BRL 1.8 billion, in inflation-adjusted values). Despite the reduction, the report indicates the continuous growth in scientific production.

Governmental assessment of Brazilian HEIs’ graduate activities is crucial. However, the General Index of Courses (IGC) is the sole indicator that accounts for graduate accomplishments. Despite its importance, there are several criticisms regarding this indicator.

Efficiency estimates in complex systems with multiple inputs and outputs, such as education, are achievable using data envelopment analysis (DEA) [8]. DEA is an instructive tool in the educational context, and the ability to deal with multiples inputs and multiple outputs represents one of the reasons for the range of applications [8]. Surveys regarding educational applications and types of evaluations within the DEA field [8,9,10,11] show that analysis regarding cost efficiency, technical efficiency, research performance, administrative services evaluation, university rankings, assessing academics on teaching and research activities, and student performance were developed. Despite the extensive volume of research already carried out, discussions about the differential emphasis given to education or to research in each institution are still scarce [5].

In this paper, we propose an innovative way to measure the efficiency of graduate activities to provide a national view of this level of education in Brazilian HEIs. We developed a new centralized two-stage dynamic DEA model with shared resources and a bi-dimensional representation of its results. We contemplate the formative and scientific production processes. A leader–follower framework is also used to investigate the impacts on efficiency decomposition when one of the stages needs to be prioritized.

The choice of a network structure lies in the fact that simple black-box structures cannot accurately reflect the complex production process in real life, making it easy to overlook important information in production activities [12]. Although relevant, most investigations visualize universities as “black boxes” and do not consider internal processes. Several studies contemplated teaching or research activities, disregarding the existence of both processes at the graduate level. Following this reasoning, the use of multi-stage models is required in order to adequately portray such features.

Besides being multi-product and multi-process organizations, the educational process usually takes several years, and investigating productivity changes across time is necessary to comprehend whether universities have improved, stagnated, or regressed in their performance [13]. Due to this multi-period feature, suitable models are required to portray the situation adequately. Dynamic DEA (DDEA), and dynamic DEA models with network structure (DNDEA) represent DEA alternatives available to incorporate temporal aspects into efficiency measures. It is possible to identify works that have addressed university evaluation, considering DDEA [13,14,15] or DNDEA [16,17,18,19,20].

Clarity and simplifications can be very valuable in a world where data are increasingly abundant [20]. In Brazil, partial evaluations for 2021 indicate the existence of 27,711, 1054, 829, and 37 graduate programs in federal, state, private, and municipal institutions, respectively. Therefore, analyzing such dimensions requires significant effort from the committees and evaluation teams. It is also noteworthy that both in the Brazilian case and international assessments, the commissions are multidisciplinary, and not all members are always familiar with mathematical programming models.

Considering such particularities, we develop a bi-dimensional representation to visually display the efficiency frontier and the DMUs’ positions concerning the frontier. Since DNDEA models provide several efficiency levels, modified virtual inputs and outputs constitute the selected tool used to represent all the different efficiency scores obtained with the DNDEA model. The bi-dimensional representation summarizes the information in a simple and straightforward way. This tool allows direct efforts, helps persuade managers and policymakers about the validity of the results, and translates recommendations into actions [21].

This work presents four major contributions to the literature and governmental management actions. We develop a new DNDEA framework to reflect a vital but minimally explored level of education in Brazil, graduate activities. To the authors’ knowledge, we are the first to investigate the efficiency of the graduate level considering its internal processes (formative and scientific production) in a dynamic manner.

The proposed models open up a way to enrich DNDEA studies by considering the shared inputs among the stages of the network. To our knowledge, only few studies considered shared resources in the DNDEA framework. In addition to proposing a new model, we also discuss the impact on the efficiency decomposition of the stages when there is a need to prioritize one of them for managerial reasons. Although different DNDEA models have been used to evaluate university performance, optimizing resource allocation is most important for the Brazilian case, since approximately half of the Brazilian graduate courses are developed in public universities and Brazilian public research agencies finance scholarships for master’s and doctoral students in these institutions. Therefore, this analysis supports the best use of public resources.

We provide comprehensive efficiency analysis of teaching and research activities at graduate level. The decomposition of the overall efficiency for the entire horizon can better aid managers in finding the process that requires more attention to prioritize resources. Therefore, relevant insights are provided for the government and HEIs on improving performance. Lastly, we also developed a simple but effective way to present the results for managers and policymakers, aiding in better comprehension and reducing cognitive efforts in the decision-making process.

The following section details a literature review on the main topics relating to the work: the Brazilian evaluation system, DNDEA models and visual representation in the DEA context. Section 3 presents methodology and data. Section 4 presents the results, and Section 5 concludes the review.

2. Literature Review

2.1. The Brazilian Evaluation System

In Brazil, there are four groups of higher education institutions (HEIs): universities, university centers, faculties, and federal institutes. We can classify HEIs into four administrative categories: federal, state, municipal, and private. These institutions are evaluated annually by SINAES with the aid of micro data collected by the Anísio Teixeira National Institute for Educational Research and Studies (INEP) in the higher education census.

According to Normative Ordinance n. 550 [22], SINAES is composed of six quality metrics: Institutional Evaluation (AVALIES), Course Evaluation (ACG), General Index of Courses (IGC), Preliminary Concept of Courses (CPC), Indicator of the Difference between Observed and Expected Performances (IDD) and the National Student Performance Examination (ENADE). The last four converge in their results but they do not communicate much with the first two, and only these four have their results released annually: ENADE since 2004, CPC and IGC since 2007, and IDD since 2014.

The evaluation processes are coordinated and supervised by the National Commission for the Evaluation of Higher Education (CONAES), while the operation is the responsibility of INEP [23]. On the other hand, the evaluation of graduate programs is performed by the Coordination for the Improvement of Higher Education Personnel (CAPES), which gives a score ranging from one to seven.

Considering all the above, it is possible to affirm that SINAES is a complex process involving different time periods and multiple tools, and it also enables the production, dissemination and management of indicators and information for Brazilian HEIs [24].

The General Course Index (IGC) is the quality indicator used to rank and guide HEIs’ evaluation. It considers metrics of the quality of all undergraduate, master, and doctoral courses at an HEI, aggregating them all into one indicator. Despite the importance of this indicator for HEIs, it is possible to find several criticisms of its construction in the literature.

First, as shown in Figure 1, SINAES indicators directly impact each other. This implies that several composite indicators are applied in the construction of the IGC. Therefore, problems in these indicators can impact the final result of the IGC. It is also possible to verify problems regarding the weighting of the considered criteria.

Technical notes issued by the government do not justify the choice of weighting for each criteria, and minor weight variations can significantly change the results [25,26]. Another criticism relates to using the same criteria for courses in different areas, in different types of institutions, and for different regions of the country [27].

In addition to these issues, another point deserves attention. There are individual and in-depth assessments for undergraduate and graduate courses. However, the indicator relating to higher education institutions (IGC) aggregates information from all undergraduate and graduate courses at these HEIs without any distinction between these levels of education. This aggregation does not allow for providing targets or projections of how each educational level should improve to enhance the institution’s performance.

Due to the mentioned problems and to the fact that there is no indicator to aggregate and show a global overview of graduate activities, the current work proposes a method to limit this gap.

The implementation of graduate studies in Brazil took place with the creation of CAPES in 1951 and through the standards defined by Report CFE 977/65 of 1965 [28]. National discussions are taking place to reformulate the evaluation process of graduate programs in Brazil. CAPES evaluates graduate programs concerning the National Graduate Plan (PNPG) guidelines. Currently, the seventh PNPG is in effect, but we are using data for the period (2019–2020) contemplated by the sixth plan. Thus, our results can help in this discussion and foster the evaluation for the seventh plan, which is still ongoing.

For the sixth plan, we had political and economic crises (2011–2020). After 2015 and the impeachment of then-president Dilma Roussef, there was a reduction in federal government transfers to higher education, with budget cuts in science and technology. The scenario becomes even worse after 2019, with the contingency of part of the budget directed to discretionary spending by federal universities, including payment of academic grants and research inputs.

The scenario of scarce resources and high public investment in the sector has motivated research to measure efficiency in this field. Ref. [29] investigated the impact of information asymmetry on organizational efficiency using data about Brazilian undergraduate courses. Ref. [30] used DEA and SFA to investigate the efficiency of undergraduate business administration courses. Ref. [31] applied ordinary least squares and SFA to investigate differences in private and public Brazilian universities’ performance. Ref. [32] addresses the efficiency of public expenditure in federal universities, and their results indicate most federal universities analyzed are still inefficient in allocating public expenditures. Ref. [33] focused on Brazil’s Federal Institute of Education, Science and Technology. They created several efficiency measures based on DEA and TOPSIS and evaluated the correlation of such scores with performance indicators applied by the Ministry of Education between 2014 and 2017. The authors verified the fact that HEIs did not improve significantly in the considered time frame. Ref. [34] is the only study that considered graduate activities in their investigations. A multistage network DEA model is applied to investigate HEIs regarding their financial, undergraduate and graduate performance.

Although Ref. [34] consider graduate aspects, they focus on allocating public resources among undergraduate and graduate activities. Their discussions also disregard time effects and can present bias, since they used quality indicators as outputs. The referred indicators contemplate the same inputs used in their evaluation, therefore being redundant. It is also important to mention that none focused exclusively on graduate activities. Therefore, the present study is the first to propose a dynamic evaluation of graduate activities and consider their internal structure. Considering the network structure will facilitate the identification of aspects that need reinforcing to foster improvement and guide managers to use public resources better. In order to achieve such goals, we developed a new dynamic DEA model with network structure, and the following section discusses the characteristics of these models.

2.2. Dynamic DEA with Network Structure

The results of the survey in [35] indicate that network and dynamic models must be highlighted among the main research fronts in the DEA literature. This statement is corroborated by the research developed by [36,37,38,39], which states several areas of applications and the development of distinct propositions for both modelings.

When it comes to efficiency measurement, there is the quantification of the conversion of inputs into outputs of the unit in focus. In the literature, static models are predominant in which there is an assumption of consumption and production in the same period tempo [36]. Dynamic models measure the efficiency of several periods in an aggregated perspective, where a link variable interconnects the periods [37].

The distinction between dynamic models (DDEA) and classical DEA models is the existence of variables, called carry-over, to link two consecutive periods. Ref. [40] proposes categorizing carry-overs into four groups: (1) desirable (good), (2) undesirable (bad), (3) discretionary (free) and (4) discretionary (fixed). This inter-period temporal interdependence can be attributed to a combination of five factors associated with the dynamic aspects of production: (1) production delays; (2) inventories; (3) capital or quasi-fixed factors; (4) cost adjustments; and (5) incremental improvements and learning models [36].

Refs. [41,42,43] can be considered the first to address the interdependence between periods for efficiency measurement, while Refs. [40,42,43] served as a basis for other dynamic DEA formulations [37]. From there, theoretical models [44] and applied ones were proposed, among which were the areas of agriculture [45], education [46], energy [40,47,48] and forests [49].

On the other hand, network models (NDEA) consider that the overall system efficiency consists of combining the DMUs’ subdivision performance, whose pioneering spirit can be attributed to [50]. Considering the DMU internal structure is necessary to avoid misleading results, such as deeming systems efficient when they are not [51], and for identifying cases in which all processes of a DMU have lower performance than other DMUs. However, the overall system efficiency indicates superior results when compared to others [52].

It is also important to note that a network can be arranged in different ways. There are two basic structures, series and parallel [53]. Complex organizations formed by the combination of the basic ones are also found in the literature and can also portray specific situations such as shared inputs [54], shared inputs and outputs [55], and also shared intermediate measures [56].

These situations are analyzed by [57] in their consideration of multi-stage models and by [39], who make a unified classification of two-stage models. The last authors suggested four classes: (i) two independent stages; (ii) two connected stages considering the interaction between them; (iii) relational models, and (iv) game-theoretic models, considering cooperation and non-cooperation.

Regarding the DEA specifications, several models are also found in the literature, such as the slack-based model (SBM) [48], additive propositions [58], the inefficiency SBM measure [59], the relational model [53], models that combined relational and SBM aspects [60], and models that simultaneously consider multi-stage and multi-level aspects [61,62,63,64].

It is also important that the proposition of distinct network and dynamic models allows the investigation of different situations and areas of applications. This diversity can be verified in the broad range of applications of both models.

Most organizational structures can be characterized by processes structured in networks and related through multiple inputs and outputs over time. Under this scenario, multiple dynamic stages connected by network structure links in each period analysis are necessary to represent reality properly [65].

The combination of dynamic and network models enables the observation of the DMUs’ overall efficiency over the entire observed period, and also to conduct further analysis; that is, observing the dynamic change in the period efficiency and dynamic change in the DMUs’ divisional efficiency [66]. This framework enables considerations about the heterogeneous organizations of DMUs, in which the divisions are mutually connected by link-type variables and by the internal exchange of intermediate products [67,68]. In order to assess this broad range of analysis, the dynamic model with a network structure (DNDEA) considers a structure that consists of a finite number of static models’ interaction [69].

Despite recent development, distinct mathematical developments have been made to propose new DNDEA models. It is possible to find in the literature approaches to deal with input uncertainties [70], with non-homogeneous DMUs [71], super-efficiency models [72], and the use of common weights to measure efficiency [73], and they have been used to investigate distinct areas of application, such as energy [66,74], transportation [75,76], supply chain [77], banks [78], and insurance companies [79].

Regarding higher education, it is possible to identify some investigations using DNDEA models to measure efficiency. Ref. [19] investigated the efficiency of the knowledge production process for nanobiotechnology research in US universities. Ref. [20] considered the financial and academic divisions to measure the efficiency of Vietnamese public colleges. Ref. [17] extended the discussions of [20] to investigate the impacts of financial and academic divisions on overall efficiency with the aid of DNDEA. The authors also applied a regression analysis to verify the effects of contextual factors on the efficiency of the financial division. Ref. [18] focused on the Australian vocational education and its subprocesses (teaching and industry responsiveness) to measure the efficiency of the teaching–industry linkage. Ref. [16] applied the DNDEA model to investigate Chile’s higher education system. They aimed to compare the results of the three-stage system proposed (teaching, research and grant application) with the current one used to rank and accredit HEIs.

It is imperative at this point to differentiate dynamic network models from multi-level multi-stage models. In this last type of model, similar to network models, several internal stages to the network are considered. However, for same cases, instead of having multiple production stages and multiple levels, where the DMUs operating also exist, these could be geographical divisions (e.g., a DMU operating within a region, which in turn is part of a whole country) or functional (e.g., sub-units of an organization, divisions and subdivisions). In these cases, a hierarchical modeling seems appropriate [64]. Therefore, this type of modeling allows for observing the DMUs as a part of a larger system.

We highlight the works of [61,62,63,64] in this context, since they provided methodological advances and approached a situation that is somehow related to the university context, the innovation systems. Refs. [63,64] developed a multi-level multi-stage approach with a soft hierarchy to investigate the knowledge production process (KPP) and knowledge commercialization process (KCP). Refs. [61,62] also approach multi-level multi-stage models. Differently from Refs. [63,64], refs. [61,62] addressed the topics under the microeconomic theory. The new studies dealt with more stages [62] or the application of the Spence distortion principle to the hierarchy of a system [61].

By observing the previous studies, it is possible to verify that there are no propositions directed toward the investigations of graduate activities. In addition to the application, it is noteworthy that all applications used the model of [66] in their investigations. Therefore, in addition to proposing a new discussion, the current study develops a new model that allows simultaneously the consideration of shared resources to discuss resource allocation and the observation of the efficient decomposition of the stages.

Because of the relevance of DNDEA models and the large amount of information generated, the current study proposes a bi-dimensional representation of the DNDEA model proposed in the following section. The following section presents a brief overview of frontier representation alternatives and how our approach diverges from them.

2.3. Visual Representation in the DEA Field

Because of the relevance of DNDEA models and the large amount of information generated, the current study proposes a bi-dimensional representation of the DNDEA model proposed in the following section. The following section presents a brief overview of frontier representation alternatives and how our approach diverges from them. The original idea behind DEA was to provide a methodology whereby, within a set of comparable DMUs, those exhibiting best practice could be identified, and would form an efficient frontier, with this frontier allowing for the identification of benchmarks against which such inefficient units can be compared [57].

A significant part of the theoretical foundation of DEA comes from the proposition of [80]. Since this initial foundation, the graphic representation of the efficiency frontier has been of significant concern, because visual representation is a powerful tool for decision-makers, allowing them to ascertain how far the DMUs are from the efficient frontier or to look for concentrations of DMUs in some areas on the graph [81].

Ref. [80] presents different isoquants to discuss the efficient frontier when the production function is known and to estimate an efficient production function from observations of the inputs and outputs for some firms. In their seminal paper, ref. [82] considered two inputs and one output, transforming the input/unit of the output, plotting this information in a bi-dimensional graph. The same idea applies to the case of one input and two outputs. Each axis corresponds to one ratio of input/output or vice versa. However, this structure becomes unfeasible for cases with multiple variables. Since this, some discussions can be found in the literature to propose alternatives to represent the frontier.

Ref. [83] proposed an interactive visual DEA (VIDEA) consisting of an extension of the multiple criteria analysis model developed previously by the same authors. They employed a multiple-criteria hierarchical model to adapt the DEA model into an aggregate measure of input and output used to plot a two-dimensional graph. Ref. [84] proposed a set of two-dimensional charts to make the presentation to the managerial community more quickly. The authors compared efficiencies with individual factors, the impacts of virtual outputs, and the use of reference units to understand inefficient DMUs’ performance better.

Ref. [85] developed a combination of DEA and Sammon Mapping to visualize the efficiency and the reference relations. They highlighted the fact that several questions could be answered directly from observations of the two-dimensional images. For example, which DMUs are efficient and which are not, which DMUs exhibit influence on the efficiency scores of other DMUs, and how strong the influence of a specific reference unit on an inefficient DMU is.

Ref. [86] used Co-Plot, with the ratio of outputs to inputs rather than the actual DEA results, stating that efficient DMUs are around the ring sector. However, in their proposition, there must exist an efficient frontier. Ref. [87] proposed a bi-dimensional representation using one input and four outputs and considered normalization to adapt the CCR results and a defined efficiency frontier.

In the literature about software designed to represent DEA results visually, ref. [21] introduced the interactive data envelopment analysis laboratory (IDEAL) as a tool to plot 3D frontiers. Although this type of graph helps see the results, the software is limited to three variables, and visualization becomes more challenging with the increase in DMUs. Ref. [88] proposed the SmartDEA, combining DEA and data mining to develop a general decision support system (DSS) framework to analyze the results of basic DEA models.

Ref. [81] proposed a more general approach to a bi-dimensional representation of CCR and BCC models. The authors used weight normalization based on the development of [83] to obtain the modified virtual inputs and outputs. These metrics are then plotted on a graph for each DMU with an efficient frontier. The main advantage of this proposition lies in its simplicity: it does not require modifications to the original model, the frontier is defined and easily obtained, the distance of the DMUs is obtainable, and visualization is easy even with a large number of DMUs.

Ref. [89] developed an extension of [81] focused on the dynamic approach of [49]. They used virtual outputs and inputs to represent divisional efficiency and applied an average of virtual inputs and outputs to represent the global efficiency of DMUs in a two-dimensional approach. Ref. [90] extended the approach of [81] to the network DEA models of [53,91]. The authors developed modified virtual inputs and outputs to represent the overall efficiency and sub-process efficiency. The model can handle multiple inputs, outputs, and intermediate measures, but is limited to two stages.

The current study is related to the propositions of [81,89]. Differently from [89], we use different types of modified virtual outputs and inputs to represent each efficiency level provided by the DNDEA model. DNDEA models provide different levels of information, ranging from global efficiency to divisional efficiency, by period. Evidently, as the number of stages or periods increases, the volume of information increases significantly, making it challenging to understand the results. Thus, the proposition of a visual tool to understand all levels of the results provided is of paramount importance, since the similar nomenclature for the different types of efficiency can represent an obstacle for decision-makers to understand the results. This feature is another significant contribution of the current study.

The bi-dimensional representation summarizes this information in a simple and straightforward way. This can help decision-makers who need to make faster and more accurate decisions. To the authors’ knowledge, we are the first to propose a bi-dimensional representation of the frontier for DNDEA models. We are also the first to deepen the discussion of bi-dimensional representation for all efficiency types measured by this type of modeling, and this is particularly important in the educational field because national assessments contemplate voluminous amounts of information and reducing the cognitive effort in these processes helps significantly in the decision-making process.

3. Materials and Methods

The application of DEA in the educational context goes back to the beginning of applied studies using the technique. The discussions employing DEA includes analysis at distinct education levels and for distinct types of investigations. Ref. [92] indicates that there are two main paths when analyzing DEA development in the education field: higher education and basic education. In the context of higher education, ref. [10] details a broad range of topics covered with DEA studies, such as university efficiency, the efficiency of individual academic departments, programs within an institution, and the central administration or services across universities. Ref. [11] also highlights using student ratings to assess performance in tertiary education, while ref. [8] details a new range of investigations such as cost efficiency, technical efficiency, research performance, rankings, and personal and teaching evaluations in higher education with DEA. In this section, we present the developed approach in Section 3.1, Section 3.2 and Section 3.3, while the Brazilian context is presented along with its data in Section 3.4.

3.1. Two-Stage Dynamic DEA with Shared Inputs: A Centralized Approach

The developed model aims to investigate resource sharing in a two-stage network model and to measure efficiency in a dynamic manner. The framework considered to develop our model is displayed in Figure 2. We considered the presence of shared and specific inputs. However specific inputs are present only in the first stage. The following models are designed to deal with shared inputs among the two stages. Therefore, the shared input p is divided into parcel ${\alpha }_{pj}$ which is consumed by the first division, and parcel $\left(1-{\alpha }_{pj}\right),$ which is used by the second division. It is important to highlight that $\alpha$ is a decision variable, and it will be determined by the model. However, we proposed the use of upper and lower bounds for $\alpha$ because the stages share the resource and a parcel must be allocated to both of them.

No exogenous inputs are entering the second division. It is also considered that all intermediate measures produced by the first stage are consumed by the second. With these assumptions in mind, we proposed two distinct frameworks to investigate the referred context, a cooperative and a non-cooperative one.

The notations, summarized in

Table 1. Indexes, parameters and variables of the model.

Indexes
$j=1,\dots ,n$	Index for jth DMU;
$t=1,\dots ,T$	Index for tth period;
$k=1,\dots ,K$	Index for kth division;
$i=1,\dots ,m$	Index for ith specific input;
$p=1,\dots ,P$	Index for pth input shared between the divisions;
$r=1,\dots ,s$	Index for rth output;
$d=1,\dots ,D$	Index for dth link;
$l=1,\dots ,L$	Index for lth carry-over.
Parameters
$x_{ij}^{\left(t\right)}$	ith specific input of DMU j in division 1 in period t;
$x_{pj}^{\left(t\right)}$	pth shared input of DMU j between divisions 1 and 2 at period t;
$y_{rj}^{\left(t\right)}$	rth output of DMU j at division 2 at period t;
$z_{dj}^{\left(t\right)}$	dth link of DMU j leaving division 1 to division 2 at period t;
$c_{lj}^{\left(t,k\right)}$	lth carry-over at DMU j in division k that connects period t to the next one; (l = 1, …, lk,… L; j = 1, …, n; t = 1,…, T − 1, k = 1,…, K).
Variables
${\propto }_{pj}$	The proportion of the shared input of DMU j that will be used by division 1;
$v_i^{\ast },v_p^{\ast },u_r^{\ast }$ $w_l^{\ast },f_d^{\ast }$	The optimal weights attached to specific inputs, shared inputs, outputs, carry-overs and links, respectively.

, present the indexes, parameters, and variables considered in a centralized relational DNDEA model and a leader–follower form of the DNDEA model.

For the two-stage system illustrated in Figure 2, the divisions of an observed ${DMU}_0$ can be evaluated considering constant returns to scale by Model (1) and (2) in each period. In Model (1) and (2), the objective function portrays the efficiency of the DMU under evaluation, while the restrictions ensure that the efficiency scores do not exceed one.

$$\begin{array}{c}{E}_j^{(t,1)}=max \frac{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lo}^{\left(t,1\right)}+{\sum }_{d=1}^D{w}_d{z}_{do}^{(t)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lo}^{\left(t-\mathrm{1,1}\right)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{po}^{(t)}+{\sum }_{i=1}^m{v}_i{x}_{io}^{(t)} }\\ \mathrm{s}.\mathrm{t}. \frac{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lj}^{\left(t,1\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{(t)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{pj}^{(t)}+{\sum }_{i=1}^m{v}_i{x}_{ij}^{(t)} }\le 1 (j=1, \dots ,n)\\ L_{pj}^1\le {\alpha }_{pj}\le L_{pj}^2 \\ v_i,w_l, f_d,v_p,\ge \epsilon ; i=1,\dots ,m;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(1)

$$\begin{array}{c}{E}_j^{(t,2)}=max \frac{{\sum }_{r=1}^s{u}_r{y}_{ro}^{(t)}+{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lo}^{\left(t,2\right)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lo}^{\left(t-\mathrm{1,2}\right)}+{\sum }_{p=1}^P{{(1-\alpha }_{pj})v}_p{x}_{po}^{(t)}+{\sum }_{d=1}^D{w}_d{z}_{do}^{(t)} }\\ \mathrm{s}.\mathrm{t} \frac{{\sum }_{r=1}^s{u}_r{y}_{rj}^{(t)}+{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lj}^{\left(t,2\right)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}+{\sum }_{p=1}^P{{(1-\alpha }_{pj})v}_p{x}_{pj}^{(t)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{(t)} }\le 1 (j=1,\dots ,n)\\ L_{pj}^1\le {\alpha }_{pj}\le L_{pj}^2\\ v_i,u_{r, }{w}_l, f_d,v_p\ge \epsilon ; i=1,\dots ,m;r=1,\dots , s;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(2)

Therefore, similar to the [51] assumption of the centralized model, we considered the same weights for the variables in all periods. We proposed a weighted average of stages 1 and 2 for each period, as displayed in (3).

$$E_j^{(t,sys)}=w_1^t{E}_j^{\left(t,1\right)}+w_2^t{E}_j^{\left(t,2\right)}$$

(3)

In order to define $w_1^t$ and $w_2^t$, the consideration of [54] was selected. The authors argued that the proportion of total resources devoted to each stage presents one reasonable choice of weight to reflect the relative size of a stage. It is important to note that in dynamic models with network structures, carry-overs, and links play a dual role. Carry-overs represent both the output of one period and an input of the following one, while links consist of outputs from the first stage and inputs from the second. Therefore, we define $w_1^t$ and $w_2^t$ in (4) and (5).

$$w_1^t=\frac{{\sum }_{i=1}^m{v}_{i }{x}_{ij}^{(t)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{l=1}^{l_1}{f}_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}}{{\sum }_{i=1}^m{v}_{i }{x}_{ij}^{\left(t\right)}+{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{\left(t-1,k\right)}}$$

(4)

$$w_2^t=\frac{{\displaystyle {\sum }_{p=1}^P}{\left(1-{\alpha }_{pj}\right)v}_p{x}_{pj}^{\left(t\right)}+{\displaystyle {\sum }_{d=1}^D}{w}_d{z}_{dj}^{\left(t\right)}+{\displaystyle {\sum }_{l=1}^{l_2}}{f}_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}}{{\displaystyle {\sum }_{i=1}^m}{v}_{i }{x}_{ij}^{\left(t\right)}+{\displaystyle {\sum }_{p=1}^P}{v}_p{x}_{pj}^{\left(t\right)}+{\displaystyle {\sum }_{d=1}^D}{w}_d{z}_{dj}^{\left(t\right)}+{\displaystyle {\sum }_{k=1}^K}{\displaystyle {\sum }_{l=1}^L}{f}_l{c}_o^{\left(t-1,k\right)}}$$

(5)

In (4) and (5), ${\sum }_{i=1}^m{v}_i{x}_{ij}^{\left(t\right)}+{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_o^{\left(t-1,k\right)}$ represents the total amount of resources (inputs) used by the stages in a period t. On the other hand, ${\sum }_{i=1}^m{v}_i{x}_{ij}^{(t)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{l=1}^{l_1}{f}_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}$ and ${\sum }_{p=1}^P{\left(1-{\alpha }_{pj}\right)v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{l=1}^{l_2}{f}_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}$ indicates the resource size of stage 1 and 2, respectively. Therefore, the system efficiency in each period is detailed in (6).

$$E_j^{(t, sys)}=\frac{{\sum }_{r=1}^s{u}_r{y}_{rj}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{(t,k)}}{{\sum }_{i=1}^m{v}_{i }{x}_{ij}^{\left(t\right)}+{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{\left(t-1,k\right)}}$$

(6)

We also considered that the overall efficiency is a weighted average of the system efficiency in each period. The proportion of total resources devoted to each period presents the choice to reflect the relative size of the period. Therefore, we define $w^t$ in (7).

$$w^t=\frac{{\sum }_{i=1}^m{v}_i{x}_{ij}^{\left(t\right)}+{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{\left(t-1,k\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}}{{\sum }_{t=1}^T{\sum }_{i=1}^m{v}_i{x}_{ij}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{\left(t-1,k\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}}$$

(7)

In (7), ${\sum }_{t=1}^T{\sum }_{i=1}^m{v}_i{x}_{ij}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{\left(t-1,k\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}$ represents the total amount of resources (inputs) used in all time frames considered. On the other hand, it indicates the resource size of each period t. Therefore, the overall system efficiency is detailed in (8).

$$E_j^{(sys)}=\frac{{\sum }_{t=1}^T{\sum }_{r=1}^s{u}_r{y}_{rj}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{(t,k)}}{{\sum }_{t=1}^T{\sum }_{i=1}^m{v}_i{x}_{ij}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{\left(t-1,k\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}}$$

(8)

Thus, under CRS, the overall efficiency score can be evaluated by solving the following fractional program as presented in Model (9). In Model (9), the objective function corresponds to the overall system efficiency. The first constraint relates to the system efficiency in each period, the second one relates to the first stage, the third one relates to the second stage, and the fourth limits ${\alpha }_{pj}$ between the upper and lower bounds. The last ensures that the weights do not assume negative values.

$$\begin{array}{c}{\theta }_o^{\ast }=Max\frac{{\sum }_{t=1}^T{\sum }_{r=1}^s{u}_r{y}_{ro}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{do}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lo}^{(t,k)}}{{\sum }_{t=1}^T{\sum }_{i=1}^m{v}_i{x}_{io}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{p=1}^P{v}_p{x}_{po}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lo}^{\left(t-1,k\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{do}^{\left(t\right)}}\\ \mathrm{s}.\mathrm{t} \frac{{\sum }_{r=1}^s{u}_r{y}_{rj}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{(t,k)}}{{\sum }_{i=1}^m{v}_{i }{x}_{ij}^{\left(t\right)}+{\sum }_{p=1}^P{v}_p{x}_{pj}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lj}^{\left(t-1,k\right)}}\le 1 (j=1,\dots ,n;t=1,\dots , T)\\ \frac{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lj}^{\left(t,1\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{(t)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{pj}^{(t)}+{\sum }_{i=1}^m{v}_i{x}_{ij}^{(t)} }\le 1 (j=1, \dots ,n;t=1,\dots ,T)\\ \frac{{\sum }_{r=1}^s{u}_r{y}_{rj}^{(t)}+{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lj}^{\left(t,2\right)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}+{\sum }_{p=1}^P{{(1-\alpha }_{pj})v}_p{x}_{pj}^{(t)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{(t)} }\le 1 (j=1,\dots ,n;t=1,\dots ,T)\\ L_{pj}^1\le {\alpha }_{pj}\le L_{pj}^2\\ v_i,u_{r, }{w}_l, f_d,v_p\ge \epsilon ; i=1,\dots ,m;r=1,\dots , s;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(9)

With the aid of the Charnes–Cooper transformation, the fractional program proposed in Model (9) can be converted into Model (10).

$$\begin{array}{l}{\theta }_o^{\ast }=max{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}\\ {\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{po}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t-1,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}=1\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{rj}^{(t)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lj}^{(t,k)}-{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{ij}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{pj}^{\left(t\right)}-{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{\left(t\right)}-{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lj}^{\left(t-1,k\right)}\le 0 \left(j=1,\dots ,n;t=1,\dots ,T\right)\\ {\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{(t,1)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}-{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{ij}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}{\alpha }_{pj}{\nu }_p{x}_{pj}^{\left(t\right)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}\le 0 \left(j=1,\dots ,n;t=1,\dots ,T\right)\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{rj}^{(t)}+{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{(t,2)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}-{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}(1-{\alpha }_{pj}){\nu }_p{x}_{pj}^{(t)}\le 0 \left(j=1,\dots ,n;t=1,\dots ,T\right)\\ L_{pj}^1\le {\alpha }_{pj}\le L_{pj}^2\\ {{\nu }_i,\nu }_p, {\mu }_r,{\gamma }_l,{\mu }_d\ge \epsilon ; i=1,\dots ,m;r=1,\dots , s;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(10)

Model (10) is non-linear since ${\alpha }_{pj}{\nu }_p$ is present in the constraints related to stage efficiency. It is possible to obtain a linear model considering that ${\beta }_{pj}={\alpha }_{pj}{\nu }_p(p=1,\dots ,P,j=1,\dots ,n)$. After this substitution, Model (10) can be converted into Model (11).

$$\begin{array}{l}{\theta }_o^{\ast }=max{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}\\ {\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{po}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t-1,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}=1\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{rj}^{(t)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lj}^{(t,k)}-{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{ij}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{pj}^{\left(t\right)}-{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{\left(t\right)}-{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lj}^{\left(t-1,k\right)}\le 0 \left(j=1,\dots ,n;t=1,\dots ,T\right)\\ {\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{(t,1)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}-{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{ij}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}{\beta }_p{x}_{pj}^{\left(t\right)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}\le 0 \left(j=1,\dots ,n;t=1,\dots ,T\right)\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{rj}^{(t)}+{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{(t,2)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}-{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}({\nu }_p-{\beta }_{pj})x_{pj}^{(t)}\le 0 \left(j=1,\dots ,n;t=1,\dots ,T\right)\\ {\nu }_p{L}_{pj}^1\le {\beta }_{pj}\le {{\nu }_{p}L}_{pj}^2\\ {{\nu }_i,\nu }_p, {\mu }_r,{\gamma }_l,{\mu }_d\ge \epsilon ; i=1,\dots ,m;r=1,\dots , s;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(11)

3.2. Efficiency Decomposition

After solving Model (11), it is possible to obtain all efficiency scores discussed previously, namely, process efficiency, system efficiency and overall efficiency. Still, it is possible for Model (11) to present alternative optimal solutions. This multiplicity implies that the efficiency decomposition may not be unique. To investigate this, we adopted a leader–follower approach. This type of analysis has been employed in several DEA studies, such as [51,93,94].

We employed a similar framework to [51,54] in which the first division has its efficiency maximized while the overall and system efficiency is maintained at the level identified with the aid of Model (11). Let ${\nu }_i^{\ast },{\nu }_{p,}^{\ast },{\mu }_{r,}^{\ast }{\gamma }_l^{\ast },{\mu }_d^{\ast }$ be the optimal weights, while ${\theta }_o^{\ast }$, ${\theta }_o^{(t,sys)\ast }$, ${\theta }_o^{(t,1)\ast }$ and ${\theta }_o^{(2,sys)\ast }$ represents the optimal overall, the optimal system efficiency by period, and the division 1 and division 2 at period t optimal efficiency ${\theta }_o^{\ast }$ of an observed DMU_o. Suppose we focus on the maximization of the first stage: while maintaining the system by period and overall score, we have:

$$\begin{array}{c}{\theta }_o^{\left(t,1\right)}=max \frac{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lo}^{\left(t,1\right)}+{\sum }_{d=1}^D{w}_d{z}_{do}^{(t)}}{{\sum }_{i=1}^m{v}_i{x}_{io}^{(t)}+{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lo}^{\left(t-\mathrm{1,1}\right)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{po}^{(t)} }\\ \mathrm{s}.\mathrm{t}. \frac{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lj}^{\left(t,1\right)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{(t)}}{ {\sum }_{i=1}^m{v}_i{x}_{ij}^{(t)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{pj}^{(t)}+{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)} }\le 1 (j=1, \dots ,n)\\ \frac{{\sum }_{r=1}^s{u}_r{y}_{rj}^{(t)}+{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lj}^{\left(t,2\right)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^2}{f}_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}+{\sum }_{p=1}^P{{(1-\alpha }_{pj})v}_p{x}_{pj}^{(t)}+{\sum }_{d=1}^D{w}_d{z}_{dj}^{(t)} }\le 1 (j=1, \dots ,n)\\ \frac{{\sum }_{r=1}^s{u}_r{y}_{ro}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{do}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lo}^{(t,k)}}{{\sum }_{i=1}^m{v}_{i }{x}_{io}^{\left(t\right)}+{\sum }_{p=1}^P{v}_p{x}_{po}^{\left(t\right)}+{\sum }_{d=1}^D{w}_d{z}_{do}^{\left(t\right)}+{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lo}^{\left(t-1,k\right)}}={\theta }_o^{(t,sys)\ast }\\ \frac{{\sum }_{t=1}^T{\sum }_{r=1}^s{u}_r{y}_{ro}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{do}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lo}^{(t,k)}}{{\sum }_{t=1}^T{\sum }_{i=1}^m{v}_i{x}_{io}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{p=1}^P{v}_p{x}_{po}^{\left(t\right)}+{\sum }_{t=1}^T{\sum }_{k=1}^K{\sum }_{l=1}^L{f}_l{c}_{lo}^{\left(t-1,k\right)}+{\sum }_{t=1}^T{\sum }_{d=1}^D{w}_d{z}_{do}^{\left(t\right)}}={\theta }_o^{\ast }\\ w_1^{t\ast }\ast \frac{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lo}^{\left(t,1\right)}+{\sum }_{d=1}^D{w}_d{z}_{do}^{(t)}}{{\sum }_{\mathrm{l}\in {\mathrm{l}}^1}{f}_l{c}_{lo}^{\left(t-\mathrm{1,1}\right)}+{\sum }_{p=1}^P{{\alpha }_{pj}v}_p{x}_{po}^{(t)}+{\sum }_{i=1}^m{v}_i{x}_{io}^{(t)} }\le {\theta }_o^{(t,sys)\ast }\\ L_{pj}^1\le {\alpha }_{pj}\le L_{pj}^2 \\ v_i,u_{r, }{w}_l, f_d,v_p\ge \epsilon ; i=1,\dots ,m;r=1,\dots , s;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(12)

Model (12) can be converted into linear programming, as displayed in Model (13).

$$\begin{array}{l}{\theta }_o^{\left(t,1\right)\ast }=max{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{(t,1)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}\\ {\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{\left(t\right)}+{\displaystyle \sum _{p=1}^P}{\beta }_p{x}_{po}^{\left(t\right)}+{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lo}^{\left(t-\mathrm{1,1}\right)}=1\\ {\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{(t,1)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}-{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}{\beta }_p{x}_{pj}^{\left(t\right)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}\le 0 \left(j=1,\dots ,n\right)\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{rj}^{(t)}+{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{(t,2)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}-{\displaystyle \sum _{p=1}^P}({\nu }_p-{\beta }_{pj}){\nu }_p{x}_{pj}^{(t)}-{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{\left(t\right)}\le 0 \left(j=1,\dots ,n\right)\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t,k)}-{\theta }_o^{(t,sys)\ast }\left({\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{\left(t\right)}+{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{po}^{\left(t\right)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{\left(t\right)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{\left(t-1,k\right)}\right)\le 0\\ {\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}-{\theta }_o^{\ast }\left({\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{po}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t-1,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}\right) \le 0\\ w_1^{t\ast }\ast \left({\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{(t,1)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}\right)\le {\theta }_o^{(t,sys)\ast }\\ {\nu }_p{L}_{pj}^1\le {\beta }_{pj}\le {{\nu }_{p}L}_{pj}^2\\ {{\nu }_i,\nu }_p, {\mu }_r,{\gamma }_l,{\mu }_d\ge \epsilon ; i=1,\dots ,m;r=1,\dots , s;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(13)

As previously discussed, the system efficiency is a weighted average of the stages; therefore, is possible to obtain the efficiency of the second stage as ${\theta }_o^{\left(t,2\right)}=\frac{{\theta }_o^{(t,sys)\ast }-w_1^{t\ast }{\theta }_o^{\left(t,1\right)\ast }}{w_2^{t\ast }}$. It is important to highlight that ${\theta }_o^{(t,sys)\ast }$, $w_1^{t\ast }$ and $w_2^{t\ast }$ are obtained with the optimal solution of Model (11), and ${\theta }_o^{\left(t,1\right)\ast }$ indicates that the efficiency of Stage 1 was prioritized and optimized first. Based on this assumption, we maintained overall, system efficiency in each period and proportion of total resources devoted to each stage in each period unchanged. Therefore, it possible to proceed to the efficiency decomposition. The same hypotheses can be used to investigate Stage 2 efficiency, as shown in Model (14).

It is possible to obtain the efficiency of the first stage as ${\theta }_o^{\left(t,1\right)}=\frac{{\theta }_o^{(t,sys)\ast }-w_2^{t\ast }{\theta }_o^{\left(t,2\right)\ast }}{w_1^{t\ast }}$. It is important to mention that the proposed models and evaluation must be used for each period t under analysis. If ${\theta }_o^{\left(t,1\right)}={\theta }_o^{\left(t,1\right)\ast }$ or, ${\theta }_o^{\left(t,2\right)}={\theta }_o^{\left(t,2\right)\ast }$, there is a unique decomposition.

$$\begin{array}{l}{\theta }_o^{\left(t,2\right)\ast }=max{\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lo}^{(t,2)}\\ {\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lo}^{\left(t-\mathrm{1,2}\right)}-{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}({\nu }_p-{\beta }_{po}){\nu }_p{x}_{po}^{(t)}=1\\ {\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{(t,1)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}-{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{\left(t\right)}-{\displaystyle \sum _{p=1}^P}{\beta }_p{x}_{pj}^{\left(t\right)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^1}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,1}\right)}\le 0 \left(j=1,\dots ,n\right)\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{rj}^{(t)}+{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{(t,2)}-{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lj}^{\left(t-\mathrm{1,2}\right)}-{\displaystyle \sum _{p=1}^P}({\nu }_p-{\beta }_{pj}){\nu }_p{x}_{pj}^{(t)}-{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{\left(t\right)}\le 0 \left(j=1,\dots ,n\right)\\ {\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t,k)}-{\theta }_o^{(t,sys)\ast }\left({\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{\left(t\right)}+{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{po}^{\left(t\right)}+{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{\left(t\right)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{\left(t-1,k\right)}\right)\le 0\\ {\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{dj}^{(t)}-{\theta }_o^{\ast }\left({\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{i=1}^m}{\nu }_i{x}_{io}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{p=1}^P}{\nu }_p{x}_{po}^{(t)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_l{c}_{lo}^{(t-1,k)}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{d=1}^D}{\mu }_d{z}_{do}^{(t)}\right) \le 0\\ w_2^{t\ast }\ast \left({\displaystyle \sum _{r=1}^s}{\mu }_r{y}_{ro}^{(t)}+{\displaystyle \sum _{\mathrm{l}\in {\mathrm{l}}^2}}{\gamma }_l{c}_{lo}^{(t,2)}\right)\le {\theta }_o^{(t,sys)\ast }\\ {\nu }_p{L}_{pj}^1\le {\beta }_{pj}\le {{\nu }_{p}L}_{pj}^2\\ {{\nu }_i,\nu }_p, {\mu }_r,{\gamma }_l,{\mu }_d\ge \epsilon ; i=1,\dots ,m;r=1,\dots , s;l=1,\dots ,L;d=1,\dots ,D; p=1,\dots , P\end{array}$$

(14)

3.3. Bi-Dimensional Representation

We use virtual inputs and outputs to obtain the bi-dimensional representation. The main issue is the constraint that states that the virtual input or virtual output equals 1 (added to linearize the mathematical model). So, in a virtual-input or virtual-output plot, all DMUs would be located on the same vertical straight line, and such a graphical representation would be meaningless [81]. In the case of dynamic models, it is necessary to add a parcel related to the other variables that also play the role of system input. The authors introduced a constraint that limits the sum of input weights to be equal to 1. In order to bypass this limitation, we follow the proposition of [81].

In the case of DNDEA models, it is necessary to add parcels related to all variables presented in the constraint that is equal to one in Model (11) because they also represent the system’s input in input-oriented cases. Then, we must consider the total sum of the weights for all the variables in the constraint referred to, which is equal to 1.

A new model is developed by adding this constraint. However, with a simple mathematical operation, it is possible to apply the results of the Model (11) by dividing the resulting weights by the total sum of the weights of the DMU under observation.

Let $S_j$ be the total sum of the shared inputs, specific inputs, carry-overs and link weights of DMU j:

$$S_j={\sum }_{i=1}^m{v}_{ij}+{\sum }_{p=1}^P{v}_{pj}+{\sum }_{l=1}^L{\gamma }_{lj}+{\sum }_{d=1}^D{\mu }_{dj}$$

(15)

$v_{ij}$ is the weight of the specific input i of DMU j, $v_{pj}$ is the weight of the shared input p of DMU j, ${\gamma }_{lj}$ is the weight of carry-over l in DMU j and ${\mu }_{dj}$ is the weight of the link d in DMU j. To obtain the representation with virtual variables, let $v_{ij}^{\prime }$, ($v_{pj}^{\prime }$; ${\beta }_{pj}^{\prime }$), ${\gamma }_{lj}^{\prime }$, ${\mu }_{dj}^{\prime }$ and ${\mu }_{rj}^{\prime }$ be the modified weights of the specific input i, shared input p, carry-over l, link d and output r of DMU j, respectively:

$$v_{ij}^{\prime }=\frac{v_{ij}}{S_j};v_{pj}^{\prime }=\frac{v_{pj}}{S_j};{\beta }_{pj}^{\prime }=\frac{{\beta }_{pj}}{S_j}{\gamma }_{lj}^{\prime }=\frac{{\gamma }_{lj}}{S_j};{\mu }_{dj}^{\prime }=\frac{{\mu }_{dj}}{S_j};{\mu }_{rj}^{\prime }=\frac{{\mu }_{rj}}{S_j}$$

(16)

When DNDEA models are used, different efficiency results are obtained. We proposed a distinct set of modified virtual inputs and outputs to represent visually all levels of results. We start with overall system efficiency, following the system’s efficiency in each period, and to conclude, we present the process efficiency in each period. Appendix B presents proof that the efficiency values obtained with modified virtual inputs and outputs do not change the scores provided by the original DNDEA model.

3.3.1. Overall System Efficiency

Let $x_{ij}={\sum }_{t=1}^T{x}_{ij}^{(t)}$, $x_{pj}={\sum }_{t=1}^T{x}_{pj}^{(t)}$, $z_{dj}={\sum }_{t=1}^T{z}_{dj}^{(t)}$, and $y_{rj}={\sum }_{t=1}^T{y}_{rj}^{(t)}$. We shall consider $I_j^{\prime (sys)}$ the system’s virtual input and $O_j^{\prime (sys)}$ the system’s virtual output of DMU j.

$$I_j^{\prime (sys)}={\displaystyle \sum _{i=1}^m}{v}_{ij}^{\prime }{x}_{ij}+{\displaystyle \sum _{p=1}^P}{v}_{pj}^{\prime }{x}_{pj}+{\displaystyle \sum _{d=1}^D}{\mu }_{dj}^{\prime }{z}_{dj}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_{lj}^{\prime }{c}_{lj}^{(t-1,k)}$$

(17)

$$O_j^{\prime (sys)}={\displaystyle \sum _{r=1}^s}{\mu }_{rj}^{\prime }{y}_{rj}+{\displaystyle \sum _{d=1}^D}{\mu }_{dj}^{\prime }{z}_{dj}+{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_{lj}^{\prime }{c}_{lj}^{(t,k)}$$

(18)

We obtain the efficiency of ${DMU}_0$, $E_o^{(sys)}$, by dividing the virtual output by the virtual input. Hereafter, we refer to (17) as the system virtual input and we do the same for (18) in the case of outputs.

3.3.2. System Efficiency in Each Period

We shall also consider ${I\prime }_j^{(t,sys)}$ the virtual input of the system in period t in DMU j and ${O\prime }_j^{(t,sys)}$ the virtual output of the system in period t in DMU j. However, since there are differences in the role of the variables for the stage, two distinct virtual inputs and outputs are required.

$$I_j^{\prime (t,sys)}={\displaystyle \sum _{i=1}^m}{v}_{ij}^{\prime }{x}_{ij}^{(t)}+{\displaystyle \sum _{p=1}^P}{v}_{pj}^{\prime }{x}_{pj}^{(t)}+{\displaystyle \sum _{d=1}^D}{\mu }_{dj}^{\prime }{z}_{dj}^{(t)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_{lj}^{\prime }{c}_{lj}^{(t-1,k)}$$

(19)

$$O_j^{\prime (\mathrm{t},sys)}={\displaystyle \sum _{r=1}^s}{\mu }_{rj}^{\prime }{y}_{rj}^{(t)}+{\displaystyle \sum _{d=1}^D}{\mu }_{dj}^{\prime }{z}_{dj}^{(t)}+{\displaystyle \sum _{k=1}^K}{\displaystyle \sum _{l=1}^L}{\gamma }_{lj}^{\prime }{c}_{lj}^{(t,k)}$$

(20)

Hereafter, we refer to (19) and (20) as the virtual inputs of the modified system by period and the virtual outputs of the modified system by period, respectively. We obtain $E_O^{(t,sys)}$ by dividing the virtual output by the virtual input.

3.3.3. Process Efficiency in Each Period

Let ${I\prime }_j^{(t,k)}$ be the virtual input of division k in period t and ${O\prime }_j^{(t,k)}$ the virtual output of division k in period t:

$${I^{\prime }}_j^{\left(t,1\right)}={\displaystyle \sum _{i\in i^1}}{v}_{ij}^{\prime }{x}_{ij}^{\left(t\right)}+{\displaystyle \sum _{p=1}^P}{\beta }_{pj}^{\prime }{x}_{pj}^{\left(t\right)}+{\displaystyle \sum _{l\in l^1}}{\gamma }_{lj}^{\prime }{c}_{lj}^{\left(t-\mathrm{1,1}\right)}$$

(21)

$${I\prime }_j^{(t,2)}={\displaystyle \sum _{l\in l^2}}{\gamma }_{lj}^{\prime }{c}_{lj}^{\left(t-\mathrm{1,2}\right)}+{\displaystyle \sum _{d=1}^D}{\mu }_{dj}^{\prime }{z}_{dj}^{(t)}+{\displaystyle \sum _{p=1}^P}{\beta }_{pj}^{\prime }{x}_{pj}^{(t)}-{\displaystyle \sum _{p=1}^P}{v}_{pj}^{\prime }{x}_{pj}^{\left(t\right)}$$

(22)

$$O_j^{\prime (\mathrm{t},1)}={\displaystyle \sum _{l\in l^1}}{\gamma }_{lj}^{\prime }{c}_{lj}^{\left(t,1\right)}+{\displaystyle \sum _{d=1}^D}{\mu }_{dj}^{\prime }{z}_{dj}^{(t)}$$

(23)

$$O_j^{\prime (\mathrm{t},2)}={\displaystyle \sum _{r=1}^s}{\mu }_{rj}^{\prime }{y}_{rj}^{(t)}+{\displaystyle \sum _{l\in l^2}}{\gamma }_{lj}^{\prime }{c}_{lj}^{\left(t,2\right)}$$

(24)

Summarizing our approach in a step-by-step procedure, as in [81]:

• Run the input-oriented DNDEA Model (11) for each DMU j;
• Calculate $S_j$ for each DMU j;
• Calculate the modified variable weights $v_{ij}^{\prime };v_{pj}^{\prime };{\beta }_{pj}^{\prime };{\gamma }_{lj}^{\prime };{\mu }_{dj}^{\prime };{\mu }_{rj}^{\prime }$ according to Equation (16);
• Calculate the modified virtual input- and virtual output overall system using Equations (17) and (18) for each DMU j;
• Calculate the virtual input and output of the modified system in each period using Equations (19) and (20) for each DMU j;
• Calculate the process-modified virtual input and output using Equations (21)–(24) for each DMU j;
• Use the modified virtual input ${I\prime }_j^{(sys)}$ in the x-axis and the modified virtual output ${O\prime }_j^{(sys)}$ in the y-axis in a bi-dimensional graph for each DMU j for overall efficiency;
• Use the modified virtual input ${I\prime }_j^{(t,sys)}$ in the x-axis and the modified virtual output ${O\prime }_j^{(t,sys)}$ in the y-axis in a bi-dimensional graph for each DMU j for system efficiency for each period;
• Use the modified virtual input ${I\prime }_j^{(t,k)}$ in the x-axis and the modified virtual output ${O\prime }_j^{(t,k)}$ in the y-axis in a bi-dimensional graph for each DMU j for process efficiency in each period;
• Draw the 45° line representing the efficient frontier where efficient DMU presents ${I\prime }_j^{(t,k)}={O\prime }_j^{(t,k)}$, ${I\prime }_j^{(t,sys)}={O\prime }_j^{(t,sys)}$, and ${I\prime }_j^{(sys)}={O\prime }_j^{(sys)}$ for process efficiency, system efficiency in each period, and overall system efficiency.

3.4. Data

In the current discussion, we aim to evaluate the graduate activities of Brazilian HEIs with the aid of a DNDEA model. As previously mentioned, universities present a multi-activity framework, and contemplating the productivity changes of these institutions is of high importance. Our DNDEA model considers two stages: the formative process and the scientific production process. The proposed framework is displayed in Figure 3.

In the first one, a parcel of faculty and enrolled student workload represents the inputs. It is important to clarify that when considering these two inputs, we are not allocating part of the students and faculty to the formative process and another to the scientific production process. We consider that all students and all faculty divide their workload between these activities. The number of programs available in a university represents the carry-over variable, while master’s dissertations and Ph.D. theses correspond to the intermediate factor linking the stages. Variable dropout represents an undesirable output and reflects the reality that some students do not finish their master’s or Ph.D. training. Since it consists of an undesirable variable, it requires treatment to be properly used in the DEA framework. We subtracted values from a large number, ensuring the results were isotonic, as discussed by [95].

The second stage (the scientific production process) converts the other portion of faculty and enrolled students’ workload and the desirable products of the formative process into research products: dissertations and theses correspond to the research developed, representing the basis for generating papers and patents. The publications considered are in the SCOPUS database.

As previously mentioned, faculty and students divide their workloads between both processes. Therefore, they correspond to the shared inputs, and entirely allocating these inputs to the first stage would be inappropriate and penalize its efficiency. The analysis of these resource allocations responds to the question of whether they are being efficiently used or not, and this information can benefit HEIs’ performance.

These variables were selected due to their relevance to the national reality; they are already used for the individual evaluation of programs, and most of them are also used in the international literature and university rankings. We would also like to highlight the fact that for the Brazilian context, master’s dissertations should be considered as inputs to the scientific production process. Firstly, in Brazil, there are extremely rare cases in which students enter directly onto a doctorate. For the most part, students enroll and complete their master’s degree before applying for a doctorate position, with the master’s degree being a prerequisite for most universities in the Ph.D. application process.

Furthermore, due to changes that have occurred in recent years in the process of monitoring postgraduate courses, many programs have been applying publication requirements for students to obtain a master’s degree. Therefore, master’s dissertations have contributed to the Brazilian scientific production process. However, for other countries, this variable could be considered an output of the first stage, with the model being easily modified to adapt to such a situation.

We emphasize that the choice to use DEA aims to mitigate one of the main criticisms verified among the government’s already-used indicators. Brazil is a country with very different regions in socio-economic and demographic terms. This national characteristic is reflected in the universities’ very different missions and objectives. Therefore, the flexibility of the weights for weighing the criteria is essential, so that each university has the autonomy to reflect these characteristics and so that the final result is not questioned, with the claim that the weighting of the criteria benefited some to the detriment of others.

In this study, we focus on the graduate activities in federal universities, because (i) they are responsible for more than half of the country’s master’s and doctoral courses and students, and produce most of the national science [6]; (ii) they represent a set of more homogeneous institutions; and (iii) they use public funds to finance their activities.

This analysis is vital, given federal government spending. Approximately half of the Brazilian graduate courses are developed in public universities. Data from 2020 indicate that the federal government spent 23 billion in federal universities to finance personnel and charges in the same year. In addition, it is worth mentioning that Brazilian research agencies such as Coordination for the Improvement of Higher Education Personnel (CAPES) and the National Council for Scientific and Technological Development (CNPq) finance scholarships for master’s and doctoral students in these institutions. Therefore, this analysis helps with the best use of public resources.

According to data released by the 2020 Higher Education Census, there are 68 federal universities in Brazil. Reports generated by CAPES correspond to the data source used, since CAPES is responsible for evaluating and consolidating information regarding individual graduate activities in Brazil. The reduction in the number of universities analyzed was due to a lack of data on one or more variables, mainly in patents and Ph.D. theses. Consequently, our sample contains 32 universities, with data from 2019 to 2020.

The selected time frame aims to evaluate the most-recent available data and obtain a glimpse of the COVID-19 pandemic’s impact on graduate activities. The descriptive statistics of the sample are presented in

Table 2. Descriptive statistics of data.

Variable	Category	Average	SD	Minimum	Maximum
Formative process
Faculty (number)	Input	1033.42	700.22	235	2913
Enrollments (number)	Input	2833.43	2289.97	321	9163
Programs (number)	Carry-over	100.94	71.71	9	321
Dropouts (number)	Output	40.35	23.49	7	89
Ph.D. theses (number)	Link	41.95	23.53	9	91
Master’s dissertations (number)	Link	219.78	218.70	8	954
Scientific production process		584.60	408.58	84	1786
Publications (number)	Output	3019.22	2228.38	326	10,400
Patents (number)	Output	52.32	51.55	1	210

.

4. Discussion

The results are divided into three sections. First, we present the DNDEA efficiencies of the 32 federal universities. Second, with the aid of the bidimensional representation, we deepen the performance discussion. Then, the efficiency decomposition under the leader–follower assumption with the procedure detailed in Section 3.2 is presented.

4.1. DNDEA Efficiency Results

The framework and variables in the proposition for investigation of graduate activities are displayed in Figure 3. We applied the developed DNDEA model discussed in Section 3 to investigate graduate activities in Brazilian federal universities.

First, we applied Model (11), considering 0.40 and 0.70 as lower and upper bounds for both shared inputs, and

Table 3. Descriptive results of the efficiencies.

	${\mathit{E}}^{(\mathit{s}\mathit{y}\mathit{s})}$	${\mathit{E}}^{(1,\mathit{s}\mathit{y}\mathit{s})}$	${\mathit{E}}^{(2,\mathit{s}\mathit{y}\mathit{s})}$	${\mathit{E}}^{(\mathrm{1,1})}$	${\mathit{E}}^{(\mathrm{1,2})}$	${\mathit{E}}^{(\mathrm{2,1})}$	${\mathit{E}}^{(\mathrm{2,2})}$
Mean	80.97%	79.59%	82.75%	88.63%	65.55%	81.87%	82.14%
S.D	5.07%	7.13%	5.89%	7.74%	11.23%	7.68%	10.97%
Max	89.57%	92.31%	94.47%	100%	89.29%	97.91%	100%
Min	68.94%	66.40%	66.77%	71.62%	46.42%	64.45%	58.66%

displays the descriptive statistics for all the efficiency results. Table 3 shows, in the second column, the overall efficiency. Columns three and four report the system efficiency, while five to eight present the process efficiencies for 2019 and 2020.

The average overall efficiency of the considered period is 80.97%. When observing the periods, 2019 obtained an average result of 79.59%, while 2020 returned 82.75%. When analyzing the average division values, it is possible to verify that 2020 returned higher efficiency scores, and the increase in performance in scientific production can explain such results. The training process showed an efficiency decline of 6.76% (88.63% in 2019 to 81.87% in 2020). A total of 26 of the 32 DMUs showed reduced efficiency when comparing the periods. On the other hand, there was an increase of 16.59% in efficiency (65.55% in 2019 to 82.14% in 2020) in the scientific production process. A total of 30 of the 32 DMUs displayed increased performance.

Considering DNDEA scores, federal universities could increase their efficiency in a network structure of the formative process and scientific production by approximately 19.03%. The scores in Table 3 indicate that, on average, the training process had better results than the scientific production process before the COVID-19 pandemic. However, in 2020, the average values are closer (81.87% and 82.14%), but with better results for the scientific production process.

The number of publications explains the better performance of the scientific production process in 2020. When comparing 2019 with 2020, there is a reduction in thesis and dissertation numbers for more than 90% of the DMUs. However, the number of publications grew for all DMUs, and approximately 60% of DMUs also saw increased patent numbers.

The performance fluctuations in 2020 may also be related to the COVID-19 pandemic. Teaching activities were suspended for several periods in Brazilian HEIs, which corresponded to most of the year. During this interval, research activities and, consequently, publications derived from this research continued remotely. In addition, the significant impacts of the pandemic on the most diverse areas of knowledge and the need for quick responses stimulated the development of a high amount of research, as seen in special COVID-19 specific discussion sections at scientific events and special issues in various journals.

However, the verified impact on teaching activities was negative. Learning in remote teaching requires a learning curve for both students and teachers. It is also worth noting that, unfortunately, access to the internet with the minimum conditions necessary to participate in activities was a problem for some of the students, with classes being one of the activities most affected by these issues, directly impacting teaching and learning.

The scientific process plays an indispensable role in disseminating the research produced in the university to the academic community and society. It is important to note that in the period before the pandemic, the performance of this stage was significantly lower than the training process. These results indicate that the investigation of more recent data is necessary to verify whether the increase in performance remains or if the difficulties verified in 2019 persist, indicating a significant difficulty in disseminating the produced knowledge beyond the university.

The investigation of more recent data is indispensable because funds directed to graduate activities in Brazil have been reduced drastically over the last decade. As pointed out by the UNESCO report, the increase in publications over recent years indicates that Brazilian research is resilient. However, resilience also has its limits. Therefore, it is relevant to understand whether the lower performance in the scientific production verified in 2019 can be related to difficulties in research funding. This topic becomes even more critical in the context of the migration of several journals to the open-access format, consequently increasing publishing costs. The increasing costs in a scenario of successive cuts in public funds can negatively impact the number of publications in Brazilian public universities.

The correlation between formative process and scientific production is negative. This result supports the previous discussion of the possible difficulty of transforming knowledge into products. Given that the theses and dissertations consist of second-stage inputs, there must be an effort to increase them in order to obtain better results for this process. However, although most universities increased their performance in 2020 in this process, there is still room for improvement.

As previously mentioned, the lower limits of ${\alpha }_1$ and ${\alpha }_2$ were defined a priori. A sensitivity analysis was performed to verify the impact of this choice on efficiency values. We performed two types of investigations. First only one parameter was altered, and then we altered both, simultaneously.

Figure 4 displays the overall and system efficiency values when only ${\mathsf{\alpha }}_1$ or ${\mathsf{\alpha }}_2$ were changed. The graphs present the efficiency values for all evaluated DMUs. We performed a similar analysis for all the efficiency levels. It is possible to verify changes in some DMU scores.

However, these changes were not very significant. Although small, it is possible to notice that greater alterations occurred in ${\alpha }_2$ values. It is also observed that for ${\alpha }_1$ and ${\alpha }_2$, the greater alterations occurred for the smallest observed values, that is, for reductions in the lower limits.

We also observed that for ${\alpha }_2$ values, some variations related to the reductions in the upper limit, that is, for values between 0.5 and 0.7. These variations can be particularly observed for system efficiency in 2020. These observations initially indicate that a greater variation in the allocation of students’ workload would have a greater impact on efficiency than a variation in the allocation of teachers’ workload.

Appendix C presents additional graphs relating to sensitivity analyses for process efficiencies and for cases where the two alpha values are changed. The results of these tests converge to ensure that the model is robust. The variations observed in all levels of efficiency are minimal, occurring in many cases only in the fourth decimal place.

4.2. Bi-Dimensional Representation

Following the procedure described in Section 3.3, the first step requires running the DNDEA model. In this subsection, we present the graphs and the empirical findings. Appendix B details the mathematical proof that the efficiency values are maintained with the bi-dimensional representation.

Figure 5 displays the frontier for the system efficiency in 2019 and 2020. The green line leaving the origin (0, 0) corresponds to the efficiency frontier in our bi-dimensional representation. The different colors in the graphs relate to the five Brazilian macro-regions. The choice to highlight the macro-regions relates to the significant social and economic discrepancies among them. Also, previous literature findings indicate that the DMU location can impact on the efficiency score.

The graphics in Figure 5 indicate that for the system efficiencies per year, there are no significant discrepancies among the Brazilian macro-regions. It is also noteworthy that no DMU obtained maximum performance in 2019 and 2020. This fact is also true for overall efficiency values. The results presented so far show the greater power of discrimination of the proposed DNDEA model.

We further examine the bi-dimensional representation of process efficiencies. The results in Figure 6 show no significant discrepancies among the Brazilian macro-regions for both processes. The Kruskal–Wallis non-parametric test was used to investigate whether the differences among the macro-regions are significant.

Table 4. Kruskal–Wallis test results for overall and system efficiencies.

Efficiency	Degrees of Freedom	Chi-Square	p-Value
Overall	4	3.7004	0.4481
2019	4	7.2126	0.1251
2020	4	2.3201	0.6771

present the test results for the overall efficiency and system’s efficiency for all years, while

Table 5. Kruskal–Wallis test results for process efficiencies.

Efficiency	Degrees of Freedom	Chi-Square	p-Value
Formative Process in 2019	4	3.5366	0.4723
Formative Process in 2020	4	3.8443	0.4275
Scientific Production in 2019	4	4.7731	0.3114
Scientific Production in 2020	4	6.3686	0.1734

presents the test results for process efficiency. At a 5% and 10% significance level, it is possible to infer that there are no differences in the median of the Brazilian macro-regions for all efficiency levels.

However, unlike overall and system efficiencies for which no DMU obtained maximum performance in 2019 and 2020, four DMUs were considered efficient in the training process in 2019, and four were considered efficient in the scientific process in 2020. It is also important to mention the findings that the average performance of the formative process is superior to that of scientific production in 2019 and that the pattern reversed in 2020 is easily verifiable when observing that most DMUs are further away from the efficiency frontier in the respective graphics of Figure 6. The results presented so far show the greater power of discrimination of the proposed DNDEA model.

Table 2 details the same results that are presented in Figure 5 and Figure 6. However, it is simpler to identify patterns and obtain a quicker understanding of the results. It is also noteworthy that both in the Brazilian case and international assessments, the commissions responsible for evaluations are multidisciplinary, and not all the members involved are always familiar with mathematical programming models.

The visual analysis will also aid in faster identification of DMU performance patterns and faster identification of the best-performing units. It can also help to verify the existence of performance discrepancies between geographical regions, and these checks are faster than analyzing large data tables. Although it does not represent the main objective of the analysis, the value of the DMU’s efficiency is easily obtained by observing the graph. We can quickly obtain the value of the DMU’s efficiency by observing Figure 5: for the highlighted DMU (UFGD), its virtual output corresponds to 22.72, whereas the virtual input corresponds to 20.97, and the efficiency corresponds to 0.9230 (20.97/22.72).

Table 6. Results of the centralized model.

University	${\mathit{\theta }}^{\mathbf{*}}$	${\mathit{\theta }}^{(1,\mathit{s}\mathit{y}\mathit{s})\mathbf{*}}$	${\mathit{\theta }}^{(2,\mathit{s}\mathit{y}\mathit{s})\mathbf{*}}$	${\mathit{w}}^1$	${\mathit{w}}^2$	${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	Formative Process ${\mathit{\theta }}^{(\mathrm{1,1})\mathbf{*}}$	Scientific Production ${\mathit{\theta }}^{(\mathrm{1,2})\mathbf{*}}$	${\mathit{w}}_1^1$	${\mathit{w}}_2^1$	Formative Process ${\mathit{\theta }}^{(\mathrm{2,1})\mathbf{*}}$	Scientific Production ${\mathit{\theta }}^{(\mathrm{2,2})\mathbf{*}}$	${\mathit{w}}_1^2$	${\mathit{w}}_2^2$
UFSCPA	0.823	0.724	0.945	0.551	0.449	0.40	0.70	0.716	0.739	0.676	0.324	0.941	0.950	0.622	0.378
UFMS	0.896	0.885	0.906	0.484	0.516	0.70	0.40	0.928	0.853	0.426	0.574	0.785	1.000	0.439	0.561
UFRR	0.782	0.872	0.709	0.446	0.554	0.70	0.70	0.906	0.610	0.885	0.115	0.724	0.587	0.892	0.108
UFS	0.826	0.777	0.881	0.529	0.471	0.40	0.59	0.821	0.655	0.738	0.262	0.909	0.805	0.726	0.274
UNIPAMPA	0.821	0.842	0.801	0.490	0.510	0.70	0.70	0.949	0.631	0.665	0.335	0.766	0.896	0.731	0.269
UFPI	0.846	0.833	0.860	0.512	0.488	0.40	0.70	0.897	0.658	0.733	0.267	0.871	0.828	0.746	0.254
UNB	0.772	0.710	0.850	0.553	0.447	0.40	0.70	0.858	0.525	0.557	0.443	0.967	0.716	0.532	0.468
UFBA	0.711	0.664	0.767	0.548	0.452	0.69	0.70	0.736	0.491	0.705	0.295	0.778	0.741	0.714	0.286
UFGD	0.863	0.923	0.803	0.498	0.502	0.70	0.40	0.966	0.893	0.410	0.590	0.784	0.818	0.435	0.565
UFPB	0.793	0.773	0.814	0.522	0.478	0.43	0.70	0.887	0.555	0.657	0.343	0.866	0.705	0.674	0.326
UFAL	0.804	0.777	0.831	0.503	0.497	0.70	0.40	0.969	0.641	0.413	0.587	0.757	0.891	0.448	0.552
UNIFAL-MG	0.844	0.831	0.856	0.501	0.499	0.70	0.55	0.884	0.528	0.852	0.148	0.871	0.754	0.876	0.124
UFCG	0.821	0.758	0.889	0.516	0.484	0.40	0.40	0.881	0.672	0.410	0.590	0.746	1.000	0.437	0.563
UFG	0.762	0.732	0.792	0.504	0.496	0.40	0.70	0.849	0.619	0.493	0.507	0.782	0.803	0.514	0.486
UNIFEI	0.765	0.745	0.785	0.510	0.490	0.70	0.70	0.823	0.577	0.684	0.316	0.815	0.718	0.692	0.308
UFJF	0.769	0.708	0.846	0.559	0.441	0.40	0.70	0.821	0.464	0.684	0.316	0.903	0.718	0.690	0.310
UFLA	0.874	0.900	0.848	0.495	0.505	0.40	0.70	1.000	0.807	0.484	0.516	0.798	0.906	0.537	0.463
UFMT	0.811	0.753	0.879	0.537	0.463	0.70	0.40	0.790	0.664	0.706	0.294	0.959	0.716	0.672	0.328
UFMG	0.814	0.772	0.861	0.527	0.473	0.70	0.70	0.981	0.541	0.526	0.474	0.979	0.730	0.526	0.474
UFOP	0.760	0.758	0.761	0.497	0.503	0.70	0.70	0.837	0.592	0.679	0.321	0.762	0.759	0.704	0.296
UFPEL	0.862	0.829	0.897	0.510	0.490	0.40	0.40	0.852	0.811	0.448	0.552	0.778	1.000	0.464	0.536
UFPE	0.743	0.718	0.770	0.509	0.491	0.50	0.70	0.865	0.557	0.521	0.479	0.800	0.735	0.540	0.460
UNIR	0.852	0.904	0.805	0.477	0.523	0.40	0.40	1.000	0.602	0.760	0.240	0.771	1.000	0.851	0.149
UFSC	0.769	0.749	0.790	0.511	0.489	0.40	0.70	0.877	0.642	0.455	0.545	0.744	0.833	0.486	0.514
UFSM	0.840	0.813	0.870	0.512	0.488	0.40	0.70	0.946	0.704	0.448	0.552	0.796	0.939	0.485	0.515
UFSCAR	0.782	0.751	0.813	0.506	0.494	0.40	0.70	0.885	0.629	0.478	0.522	0.836	0.790	0.494	0.506
UFSJ	0.827	0.824	0.830	0.522	0.478	0.70	0.40	0.827	0.821	0.488	0.512	0.789	0.870	0.489	0.511
UNIFESP	0.855	0.897	0.815	0.483	0.517	0.70	0.58	1.000	0.643	0.711	0.289	0.826	0.781	0.765	0.235
UFU	0.770	0.755	0.786	0.504	0.496	0.62	0.70	0.871	0.593	0.583	0.417	0.825	0.727	0.601	0.399
UFV	0.892	0.869	0.916	0.512	0.488	0.40	0.70	1.000	0.764	0.445	0.555	0.864	0.962	0.476	0.524
UFABC	0.689	0.713	0.668	0.484	0.516	0.70	0.70	0.774	0.626	0.585	0.415	0.644	0.705	0.614	0.386
UFAC	0.871	0.909	0.837	0.469	0.531	0.70	0.40	0.963	0.871	0.414	0.586	0.759	0.902	0.453	0.547

also shows that the proposed method makes ranking universities based on efficiency values possible. In addition to these values, the results related to the proportion of the allocation of resources shared between the stages and the importance of the stages reflected by the proportion of inputs are presented.

4.3. Efficiency Decomposition

Table 7. Results with Stage 1 as leader.

University	2019				2020
	${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	Formative Process	Scientific Production	${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	Formative Process	Scientific Production
UFSCPA	0.4	0.7	0.7166	0.7764	0.4	0.7	0.9977	0.8574
UFMS	0.5568	0.7	0.9306	0.8258	0.7	0.7	0.7998	0.9886
UFRR	0.7	0.7	0.9094	0.6362	0.7	0.7	0.7246	0.5801
UFS	0.4	0.7	0.8657	0.6422	0.4	0.7	0.9577	0.6762
UNIPAMPA	0.7	0.7	0.9499	0.5514	0.4	0.7	0.7910	0.8292
UFPI	0.7	0.7	0.9198	0.5797	0.7	0.7	0.8995	0.7453
UNB	0.4	0.7	0.8717	0.5572	0.4	0.7	0.9974	0.6817
UFBA	0.4	0.7	0.7460	0.5461	0.6336	0.7	0.8010	0.6830
UFGD	0.7	0.7	0.9944	0.8355	0.4	0.7	0.8032	0.8027
UFPB	0.7	0.7	0.9228	0.5548	0.7	0.7	0.9023	0.6311
UFAL	0.6656	0.7	0.9706	0.5901	0.4	0.7	0.7573	0.8913
UNIFAL-MG	0.4	0.7	0.9480	0.1355	0.7	0.7	0.8724	0.7413
UFCG	0.7	0.7	0.8816	0.6751	0.7	0.7	0.7834	0.9708
UFG	0.4	0.7	0.8554	0.6269	0.7	0.7	0.7910	0.7938
UNIFEI	0.7	0.7	0.8366	0.5961	0.7	0.7	0.8161	0.7159
UFJF	0.4	0.7	0.9264	0.4152	0.4	0.7	0.9408	0.6337
UFLA	0.7	0.7	1.0000	0.7904	0.4	0.7	0.8531	0.8424
UFMT	0.7	0.7	0.8043	0.7364	0.4	0.7	1.0000	0.6308
UFMG	0.7	0.7	1.0000	0.5257	0.7	0.7	1.0000	0.7070
UFOP	0.6509	0.7	0.8807	0.5704	0.7	0.7	0.7628	0.7570
UFPEL	0.4	0.7	0.8794	0.7999	0.7	0.7	0.8049	0.9767
UFPE	0.5274	0.7	0.9008	0.5106	0.5134	0.7	0.8330	0.6958
UNIR	0.7	0.7	1.0000	0.4534	0.4	0.7	0.7727	0.9889
UFSC	0.4870	0.7	0.8994	0.6225	0.7	0.7	0.7860	0.7930
UFSM	0.4	0.7	0.9463	0.7150	0.7	0.7	0.8406	0.8968
UFSCAR	0.4	0.7	0.9112	0.6035	0.5090	0.7	0.8438	0.7832
UFSJ	0.7	0.7	0.8750	0.6844	0.4	0.7	0.8847	0.7779
UNIFESP	0.4333	0.7	1.0000	0.6712	0.7	0.7	0.8443	0.7208
UFU	0.7	0.7	0.9061	0.5626	0.7	0.7	0.8471	0.6934
UFV	0.4	0.7	1.0000	0.7490	0.7	0.7	0.9008	0.9294
UFABC	0.5236	0.7	0.7777	0.5634	0.7	0.7	0.6450	0.7038
UFAC	0.4	0.7	0.9816	0.8712	0.7	0.7	0.7862	0.8795

and

Table 8. Results with Stage 2 as leader.

University	2019				2020
	${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	Formative Process	Scientific Production	${\mathit{\alpha }}_1$	${\mathit{\alpha }}_2$	Formative Process	Scientific Production
UFSCPA	0.59	0.70	0.5912	1.0000	0.4	0.7	0.9176	0.9893
UFMS	0.70	0.40	0.7304	1.0000	0.4	0.7	0.7853	1.0000
UFRR	0.70	0.70	0.8977	0.6770	0.7	0.7	0.7202	0.6163
UFS	0.40	0.70	0.7322	0.9045	0.4	0.4	0.8908	0.8533
UNIPAMPA	0.40	0.65	0.8310	0.8645	0.7	0.7	0.7282	1.0000
UFPI	0.40	0.48	0.7981	0.9292	0.4	0.7	0.8402	0.9195
UNB	0.40	0.70	0.6281	0.8130	0.4	0.7	0.8682	0.8284
UFBA	0.68	0.70	0.6388	0.7242	0.7	0.7	0.7522	0.8049
UFGD	0.40	0.40	0.8125	1.0000	0.4	0.4	0.7197	0.8671
UFPB	0.40	0.70	0.7222	0.8717	0.4	0.7	0.8048	0.8324
UFAL	0.58	0.70	0.7489	0.7960	0.7	0.4	0.7122	0.9278
UNIFAL-MG	0.40	0.40	0.8322	0.8273	0.4	0.4	0.8652	0.7917
UFCG	0.40	0.70	0.4090	1.0000	0.7	0.4	0.7457	1.0000
UFG	0.46	0.70	0.6540	0.8084	0.4	0.7	0.7579	0.8288
UNIFEI	0.70	0.70	0.7453	0.7454	0.7	0.7	0.7653	0.8300
UFJF	0.40	0.41	0.6915	0.7451	0.4	0.7	0.8770	0.7760
UFLA	0.40	0.70	0.7940	1.0000	0.4	0.7	0.7926	0.9125
UFMT	0.40	0.70	0.7087	0.8601	0.4	0.4	0.9526	0.7281
UFMG	0.40	0.70	0.6752	0.8794	0.4	0.7	0.8633	0.8591
UFOP	0.56	0.40	0.7271	0.8240	0.4	0.4475	0.7266	0.8430
UFPEL	0.40	0.70	0.6185	1.0000	0.4	0.7	0.7780	1.0000
UFPE	0.51	0.70	0.6346	0.8076	0.4	0.7	0.7370	0.8086
UNIR	0.40	0.70	0.8743	1.0000	0.7	0.7	0.7708	1.0000
UFSC	0.48	0.70	0.6164	0.8595	0.4	0.7	0.6870	0.8866
UFSM	0.45	0.70	0.6562	0.9396	0.4	0.7	0.7501	0.9821
UFSCAR	0.46	0.70	0.7065	0.7923	0.4	0.4	0.8137	0.8126
UFSJ	0.70	0.40	0.6396	1.0000	0.7	0.4	0.7095	0.9457
UNIFESP	0.69	0.70	0.9070	0.8719	0.4	0.7	0.8199	0.8000
UFU	0.67	0.70	0.7159	0.8100	0.4	0.4	0.7775	0.7981
UFV	0.40	0.70	0.7055	1.0000	0.4	0.7	0.8229	1.0000
UFABC	0.70	0.70	0.6485	0.8026	0.7	0.7	0.6338	0.7217
UFAC	0.70	0.70	0.7803	1.0000	0.7	0.4	0.6407	1.0000

present the efficiency decomposition results. The first one portrays the case when the first stage is prioritized, while the second views Stage 2 as a leader. Besides efficiency values, these tables also present the optimal proportions of each shared input for all years under investigation.

When analyzing the efficiency decomposition of the processes, it is possible to verify that the decomposition was unique only when one of the stages was considered efficient. This is the case of UFLA, UNIR, UNIFESP, and UFV in the formative process in 2019. This situation was also observed in UFMS, UFCG, UFPEL, and UNIR in 2020 for the scientific production process.

In Table 7, it is possible to identify that the number of efficient DMUs remains the same in 2019. However, in 2020, two DMUs became efficient when the first stage was the leader. In contrast, nine and seven are deemed efficient in 2019 and 2020, respectively, when the second stage becomes the leader, as displayed in Table 8.

It is also relevant to observe that when the first stage is prioritized, the allocation of students is maintained or even enlarged in 2020 for the majority of the DMUs. However, the same pattern is not verified for professors. On the other hand, the pattern verified for the second stage is similar for both years. The majority of DMUs are inclined to maintain or reduce both students’ and professors’ workloads when compared to the initial DNDEA results.

Efficiency decomposition analysis allows universities to evaluate different scenarios and consider the impact of prioritizing the performance of one process over another. In addition, the model used provides individual answers for each university, as well as the proportion of resource allocation for the investigated cases.

5. Conclusions

Universities are essential for social and economic development. Public funds used in these institutions have stimulated the development of proposals for evaluation. DEA has stood out in the field of efficiency measurements in education, with the application of models in distinct areas, such as primary education, secondary schools, teachers, students, research, and teaching.

Educational processes usually span several consecutive periods. Therefore, it is adequate to use models considering the temporal effects on efficiency. We also consider that there is a network structure when analyzing the processes of graduate activities. Thus, in this paper, there is a proposition of a two-stage dynamic network model that considers shared inputs among the stages. First, we propose a centralized approach that maximizes the efficiency of the system, considering all periods and stages under investigation. The overall efficiency is obtained with a weighted sum of the period and process efficiency. In this initial view, the approach considers that all stages cooperate and act in unity to obtain the best possible results, considering the entire time frame evaluated.

Considering resource sharing between the stages makes it possible to represent the context of graduate activities more accurately. Nevertheless, the proposed DNDEA use is broader than the educational context and can be applied to others where the stages share common resources. Also, Appendix A points out that our approach can easily be adapted to cases without shared inputs, and considers exogenous inputs in the second division of the DMU. After this initial analysis, we investigated the efficiency uniqueness of the centralized DNDEA with a decomposition based on a leader–follower approach. In this framework, we investigated the cases where the first stage takes priority and the situations where the second stage is the leader.

This paper also presents a new framework for a bi-dimensional representation of the DNDEA efficiency frontier and the location of DMUs regarding the frontier when multiple inputs and outputs are present. Then, we present the step-by-step procedure developed to generate the graphs to present all the distinct levels of DNDEA efficiency results. Through a linearization of weights, we obtain a new set of weights—the modified weights- to obtain modified virtual inputs and outputs, allowing a two-dimensional representation.

The bi-dimensional framework provides intelligible graphs. The proposed approach’s advantages are related to the simplicity of the method. Also, dynamic models provide a more comprehensive range of information when compared with classical models. In this sense, graphical representation offers a critical and effective way to deliver all the information to the decision-maker.

Combining the models and their bi-dimensional representation indicates that the DNDEA model is more suitable for analyzing universities. We verify an increase in system efficiency from 2019 to 2020. Results indicate that the COVID pandemic impacted the formative and scientific production processes differently. We also evaluated whether there were significant performance differences when considering the five Brazilian macro-regions. No significant disparities were found when analyzing the bi-dimensional representation and the statistical tests.

The formative and scientific production process results inversed the patterns in 2019 and 2020. Before the pandemic, the formative process performed better, but the scientific production process obtained superior results in 2020. Correlation analyses between the efficiency scores highlight the fact that the scientific production process significantly impacts the system’s results. However, cuts in national budgets earmarked for education and research have been negatively impacting the performance of this activity. Furthermore, it is relevant to map and understand the main difficulties in the formative process because scientific production directly depends on the products generated by it.

The empirical results allow for ranking universities, aid in graduate activities’ improvements, and support the development of public policies to enhance Brazilian research results. Despite the relevant results, we must highlight a limitation of the study. A thorough analysis is necessary to investigate more data for both processes to verify whether the superior performance of the scientific production remains. The graduate activities have been resilient throughout a decade of successive budget cuts. However, it is essential to mention that this resilience is not unlimited.

Second, our study did not include quality metrics of graduate activity products. Therefore, more investigations are required to add variables that reflect quality. We can use the classification of publications considering the journal impact factor or quartile to segregate this variable and provide more thorough evaluations.

Third, although federal universities are highly relevant to Brazilian research, the investigation of private, state, and municipal institutions should also be considered to assess the performance of graduate activities. The absence of these HEIs represents the main limitation of this research.

Besides the empirical contributions to the Brazilian HEIs, this paper provides three main methodological contributions. The first relates to a new framework for investigating two-stage systems in a dynamic setting with shared resources between the stages. The second relates to the discussion of efficiency decomposition to verify the uniqueness of the efficiency scores provided by the DNDEA model. The last refers to a simple but effective way to present the results provided by the bi-dimensional representation.

We concluded that extensions of this work are also possible. Initially, the investigations did not consider undergraduate activities, and they represent a significant part of federal universities’ operating processes and expenses. It is also relevant to mention that no indicator evaluates undergraduate activities in an aggregate manner to rank the universities. Thus, this extension represents a relevant contribution due to the importance of federal universities to society.

From the methodological point of view, it is important to highlight that the current study evaluates the efficiency decomposition after a cooperative evaluation considering collaboration between the stages. However, analyzing this context from a non-cooperative perspective is interesting for assessing real cases in which cooperation cannot be guaranteed. Modifications of the current model using non-radial measures are extremely valuable in improving the applicability range of the model. Lastly, the model’s extension to a multiple-stage and multi-level framework is highly recommended. The extension to multiple stages will allow discussions such as the investigation of the Brazilian university triple helix: teaching, research and extension activities. The multi-level investig ation can aid in investigating how public universities are contributing to obtaining the Ministry of Education goals.