*3.2. Results*

## 3.2.1. Initial Screening

As presented in the previous section 387 papers were reviewed in the initial screening stage. The filtering of the reviewed sample according to the scope of the review under study, resulted in 38 peer-reviewed journal articles. This section presents the initial systematic review of the 38-paper sample with the use of descriptive statistics to gain further insight about the general information that derive from the reviewed sample. With respect to the general information, the majority of papers (21%) were issued in 2017, whereas only two papers (5%) fitting the review criteria ware published in 2012, 2011, and 2010 [27]. However, it is worth noting that 45% of the examined papers was issued during the last three years (2016–2018), indicating a boost in the scientific community's interest regarding integrated sustainability assessment (Figure 7).

Regarding the geographical origination, as presented in Figure 7, half of the assessments were performed in Europe (50%), whereas 16% were performed in Asia. Additionally, only three out of 38 assessments were performed in North America. With respect to the literature typology of the studies reviewed, as it was mentioned before only peer-reviewed journal articles were included in the reviewed sample.

**Figure 7.** (**a**) Year of publication (% percentage). (**b**) Geographical origination of publication (% percentage).

Regarding the method identification category, Table 3 presents all the methods and tools that were identified during the review process (the nomenclature is presented in Appendix A). All of the relevant methods will be presented in detail later. In the majority of the papers examined (66%), the methods or tools presented are also practically tested presenting the relevant examples (case studies). In 18% of the papers, an already existing methodology was applied and presented while 16% of papers presented a methodology without testing it in practice. Continuing with the level of assessment, in 79% of the works examined, the assessment was performed exclusively for the farm level, whereas for 21% of the works, the level of assessment was also broadened beyond the farm level by examining local, regional, or national sustainability. The most frequently examined crop is maize and wheat (examined in five cases each), followed by olive, spinach and rice (examined in two cases studies each). The other crops examined in the papers reviewed included legumes, lettuce, scallions, red radish, banana, soybean, grapes, cranberry, potato, and co ffee. Additionally, di fferent agronomic practices are examined as for example organic farms [28], greenhouse cultivations [29], and school gardens [30].


**Table 3.** Initial screening (38-paper collection).

#### *Sustainability* **2019**, *11*,5120



127

## *Sustainability* **2019**,*11*,5120

## 3.2.2. In-Depth Review

This section presents the systematic review results against the in-depth review criteria initializing the presentation with the scope criteria category (Tables A2 and A3 of Appendix A). Regarding the goal of the assessment, 61% of the examined studies are system describing, whereas the other 40% attempts to identify and evaluate policies and techniques that could be used to improve agricultural sustainability performance. Regarding the target users of the methodologies proposed, the majority of the examined works is aimed at decision-makers, farmers, and researchers. More specifically, 40% of the studies identify decision-makers as their target users, whereas 26% aim at farmers and 21% aim at researchers. Continuing, only three (3) works define a functional unit as a basis for the assessment. In particular, De Luca et al. (2018), when examining the sustainability of olive growing systems, proposed "One hectare (1ha) of cultivated surface" as a functional unit [31]. On the other hand, Theurl et al. (2017) and El Chami et al. (2015) preferred functional units related to the weight of the final product ("kg of un-/packed fresh product at the point of sale-POS" and "1 tn fresh weight standardized to 86% dry matter, respectively") [36,43].

Concerning the criterion of the time dimension, in several studies the assessment was performed for a single year period [25,28,29,33,36,41,51,60]. However, there are also studies that perform the assessment for a range of years. Snapp et al. (2018) performed a 3-year trial, and Vasileiadis et al. (2017) extracted their data during a 4-year experiment [32,38]. Sharma et al. (2011) collected data from three separate decades from 1950 to present [55], and Gomes et al. (2009) collected data from 1986 to 2002 [57]. From another point of view, De Luca et al. (2018) expanded their assessment to the life cycle of an olive tree orchard (50 years) [31], and el Chami et al. (2015) projected the assessment to 2050 [43].

Regarding the Impact Identification category, as described above, the research scope contains only studies that attempt to examine all the three dimensions of sustainability, namely, the environmental, economic, as well as social pillar, contributing towards an integrated sustainability assessment evaluation. During the extensive review, all of the individual impacts—expressed as indicators—that were examined within the reviewed studies were extracted and documented. However, further thorough classification and commenting on the individual indicators used goes beyond the limits of this analysis and has already been investigated in several review studies in the past [4,13,18,19,22].

With respect to the data calculation method category of criteria, for 82% of the papers examined a validation process is not mentioned. Only 18% of the papers describe a validation process for the proposed methodologies. On the other hand, 74% of the studies mention the use of an aggregation technique or methodology aiming at the simplification and the generalization of the results. Regarding the type of data used for the assessments performed (Figure 8), the majority uses experimental data (68%), whereas a small percentage of works (18.4%) employ only model data for the sustainability assessment. Accordingly, 58% are ex post assessments attempting to evaluate current practices; whereas, in 31.6% of the papers, the evaluation of prediction scenarios is attempted.

**Figure 8.** (**a**) Type of data (% percentage). (**b**) Accessibility of data (% percentage).

#### *3.3. Agricultural Sustainability Methods and Tools*

In the previous sections a descriptive qualitative analysis of the review criteria was presented. The aim was to examine the research trend of crop agricultural sustainability and specifically the trend of the criteria concerning the scope and the calculation methods used. In this section, the methodologies and tools, extracted as a result of the review conducted, are presented. Figure 9 demonstrates the methods and tools identified and the corresponding frequency of occurrence. These methods and tools were classified in five major categories based on the main scope of the assessment (as expressed by the authors), underlining the fact that the categories selected may overlap as part of the overall concept. A distinctive example is MCDA which is used to facilitate the assessment of multivariate problems that are expressed with indicators. Nevertheless, the scope of studies employing MCDA methods focus on the aggregation of the results while methods proposing indicator sets and indexes focus on determining the criteria of the assessment. Another example is the carbon footprint (CF) which is an indicator that is often met in Indicators sets and frameworks. Nevertheless, it is a very commonly used standalone methodology for environmental impact assessment.

To that end, LCA methods relate to the life cycle of the examined element. Environmental methods relate to the quantification of the environmental impact of the examined element, and economic methods refer to the use of financial methods in the impact assessment. Multicriteria methods are methods that employ multicriteria assessment for the evaluation of agricultural sustainability, and Indicator methods include indicator sets and frameworks for the assessment of agricultural sustainability. With respect to the individual methodologies that were identified, the term "indicators" refers to all those methodologies that were not given a specific name by their developers.

**Figure 9.** Crop agricultural sustainability at farm level (method categories and methods).

3.3.1. Life Cycle Assessment, Environmental, and Economic Methods and Tools

For 21% of the studies reviewed, methods belonging to LCA, environmental, or economic method categories are employed. El Chami et al. (2015) performed an integrated sustainability assessment comparing different irrigation scenarios of winter wheat production at the farm level by proposing a methodology that combined LCA, SCC, and CBA [43]. Following the concept of LCA, Theurl et al. (2017) assessed the environmental and socio-economic impacts of unheated soil-grown vegetables [36]. Theurl et al. performed a comparative assessment utilizing experimental field data and data collected from literature, calculating the GHG emissions with socio-economic indicators deriving from the Sustainability Assessment of Food and Agriculture Systems (SAFA) guidelines of the Food and Agriculture Organization (FAO) [36]. Theurl et al. combined methodologies from three of the five categories identified, namely the LCA, the environmental and the indicator methods.

From the most recent studies, De Luca et al. (2018) assess the sustainability of olive growing systems by focusing on scenarios differentiated in weeding [31]. For the assessment, authors combined a series of tools to evaluate the three pillars of sustainability, namely, LCA for the environmental pillar, LCC for the economic, and SLCA for the societal pillar. They integrated their results by employing the AHP method for multicriteria analysis [31]. From the economic methods category, Van Passel et al. (2009) proposed a methodological framework based on the sustainable value approach (SVA) to assess the sustainability on farm production level [58]. Van Passel et al. employed the SVA method attempting to correlate farm performance in respect to consumption of resources. The work represents a benchmarking approach since it does not focus on the evaluation of sustainability in absolute terms, but it assesses the performance compared to standards [58]. Van Passel et al. (2011) stated that to perform multilevel and multi-user assessments, a combination of methodologies can offer more advantages than integrated methodologies [53]. To that end, the SVA method was combined with the MOTIFS indicator tool. According to Van Passel et al. (2011), MOTIFS is a visual monitoring tool

used for the aggregation of indicators of various themes, which creates benchmarks for the rescaling of the indicator values [53].

#### 3.3.2. Multicriteria Assessment Methods and Tools

Within the multicriteria assessment methods that are used for assessing agricultural sustainability, the works examined can be classified into groups that employ and develop the same methodological framework. Such groups are the studies that use the MASC decision model developed by Sadok et al. (2009), which was built as part of the decision support system DEXi [59]. The MASC model is a hierarchical multiattribute decision support model designed for the ex ante assessment of cropping systems to address the need of in-field alternative scenario evaluation. Such models allow for the simplification of the decision problem by downscaling it to smaller and less complex problems expressed by designated variables [59]. The DEX methodology performs aggregation of qualitative attributes and utility functions using "IF-THEN" aggregation rules [59]. Colomb et al. (2013) building upon Sadok's et al. (2009) model, proposed the MASC-OF model to assess the strong and weak points of organic cropping systems in a regional context [50]. Pelzer et al. (2012) also following Sadok et al. (2009) presented the DEXiPM model which was particularly developed for integrated pest managemen<sup>t</sup> systems (IPM). The DEXiPM model is built upon the MASC model and it is an ex ante methodology contributing towards the discussion around innovative systems. The model was implemented in winter crop and maize-based cropping systems and consists of seventy-five (75) basic and eighty-six (86) aggregated indicators [27].

Vasileiadis et al. (2013) used the DEXiPM model to compare the sustainability of innovative IPM-based systems [51]. Also, Vasileiadis et al. (2017) used Pelzer's et al. (2012) DEXiPM model, which was supplemented by Angevin et al. (2017), for the ex post assessment of the economic, environmental and social sustainability of conventional winter wheat and maize cropping systems [26]. The IPM-based systems were designed and tested in nine (9) locations in Europe [38]. They compared the sustainability of the examined systems, discussing the benefits or drawbacks of the IPM systems. Vasileiadis et al. (2017) also adopted methodologies from the environmental and economic categories. Economic data, with the use of a template, were collected from participants to perform cost–benefit analysis (CBA). Furthermore, an environmental risk assessment was performed by implementing the SYNOPS-WEB Tool [38]. Lastly, Chopin et al. (2017) adapted the MASC model in order to ex ante assess the sustainability in the area of local banana farming systems [37].

Multicriteria methods facilitate decision making while considering multiple variables, and such methods use weighting techniques in order to produce composite indices [24]. Among the studies examined, the most frequently used methods are the principal component analysis (PCA) and the data envelopment analysis (DEA). Specifically, Gomez-Limon et al. (2010) and Sottile et al. (2016) used the PCA method, whereas Gomes et al. (2009) and Reig-Martinez et al. (2011) used the DEA method to create a composite indicator [24,30,54,57]. Dong et al. (2015 and 2016) combined both methodologies attempting to construct a complex indicator composed of a large number of individual and interdepended variable [40,46].

Concluding with the multicriteria method category, Siciliano et al. (2009) used the social multicriteria evaluation (SMCE) framework, which was implemented through the NAIADE (novel approach to imprecise assessment and decision environments) software, to assess the sustainability of farming practices in a small rural area in Italy [60]. Egea et al. (2016) employed the analytic hierarchy process (AHP) in order to investigate the combination of protected destination of origin oil production system that leads to optimal sustainability [39]. Bockstaller et al. (2017) introduced the CONTRA tool, an innovative aggregation method that leads to the creation of decision trees using fuzzy sets [34]. Peano et al. (2014) proposed a multicriteria methodology to evaluate the effectiveness of the slow food presidia, which are organized structures aiming at the preservation of quality production at risk to extinction by following specific guidelines and protocols for each product category [48].

#### 3.3.3. Indicator Sets, Indexed and Frameworks

This category of methods and tools contains indicator sets, indexes, and frameworks that were used in the reviewed works to assess agricultural sustainability at the farm level. Walter et al. (2009a; 2009b) proposes a new indicator-based method to assess the unsustainability of a system rather than its sustainability [62]. Their method borrows elements of the LCA methodology and was implemented in two stages. The first stage includes the creation of an issue inventory and its contextualization, while the second stage includes the standardization and sustainability valuation process [61,62]. Rodriguez et al. (2010) proposed the APOIA-NovoRural framework, which comprises a collection of basic and composite indicators covering five dimensions of sustainability: landscape ecology, environmental quality, sociocultural values, economic values, and managemen<sup>t</sup> and administration [56]. Sharma et al. (2011) introduced a methodology based on questionnaires and surveys and composed an agricultural sustainability index (ASI) targeted to Bihar province (India) [55], and also calculated the sustainability parameters for a 60-year period.

Sami et al. (2013) selected six indicators that were considered appropriate to assess sustainability in a regional context. Additionally, in order to evaluate some of these indicators they used a selection of fuzzy submodels [52]. Van Asselt et al. (2014) propose a protocol for the collection and evaluation of indicators for the sustainability assessment of agri-food production systems [49]. Their proposed list covers a wide range of indicators related to the three pillars of sustainability, aiming at supporting policy makers in decision making by choosing the most relevant indicators. Yegbemey et al. (2014), proposed an innovative participatory approach that resulted in seventeen (17) indicators. All relevant data were collected through a household survey. The sustainability was evaluated with relative scores while the total sustainability level was based on the average scores of the individual indicators [47]. Peano et al. (2015), proposed the SAEMETH monitoring tool based on a set of qualitative indicators. The selection of the indicators was based on the criteria introduced by Meul et al. (2008) [63] and, for their evaluation, they set a minimum and maximum threshold based on reference values that was derived from best practices or through surveys [45]. Santiago-Brown et al. (2015) presented the process for selecting indicators to assess viticulture production sustainability. For the selection of the indicators, the adapted nominal group technique was used. The selected indicators were reduced according to their relevance [44] resulting in seventy-six (76) indicators hierarchized based on their importance.

Allahyari et al. (2016) selected five-hundred-and-eighty-eight (588) indicators through an extensive literature review. Following erasing duplicates and prioritizing the sample, it resulted in 62 indicators, which were used in an extensive survey among experts. The indicators were assessed based on their importance while the resulting data were assessed with the Minskowski fuzzy screening method [42]. Sajjad et al. (2016) examined the relevant agricultural sustainability at farm and regional scale using the sustainable livelihood security index (SLSI) [41]. Yang et al. (2016) assessed the sustainability of greenhouse vegetables using indicators. More specifically, to examine the greenhouse vegetable farming practices and the economic and social managemen<sup>t</sup> conditions, they used rapid and participatory rural appraisal (RRA/PRA) tools combined with data derived from in-field measurements and parallel surveys [29]. In 2016, de Olde et al. proposed the sustainability assessment tool named response-inducing sustainability evaluation (RISE), which was implemented for the evaluation of organic farms in Denmark. The tool contains indicators for a total of 10 themes and 51 subthemes. The indicators were normalized and aggregated and each theme was evaluated based on the average score of the relevant subthemes [28].

Goswami et al. (2017) integrated the sustainable livelihood (SL) and the drivers–pressures–state– impact–response (DPSIR) framework, proposing a small farm sustainability index (SFSI) that could address the complexity of small-holder family farms under a participatory approach [35]. The proposed framework assesses sustainability in multiple levels assigning the relevant weights and resulting in the creation of an aggregated index for the entire system. They indicate that the introduction ICT technologies in agriculture (web-based platforms, wireless sensors, etc.) can facilitate data sharing among stakeholders and provide the basis for assessing the sustainability of farming systems.

Recanati et al. (2017) proposed an indicator-based framework for the assessment of sustainability of small-scale farming systems in water-limited regions. They implemented the framework by modeling an "average" farm based on a survey among 30 farmers [33]. Gaviglio et al. (2017), attempting to integrate various analytical techniques, introduced the 4AGRO tool, which is an online self-assessment tool based on indicators. It consists of 42 subindicators that are divided in 15 complex indicators, five for each pillar of sustainability [25]. The tool was demonstrated in an agricultural park in Italy. Finally, Snapp et al. (2018) proposed a methodology based on indicators that derived through a participatory approach involving a steering committee with multidisciplinary participants from eight (8) institutions [32]. The indicators were normalized based on max possible values.
