**E**ff**ects of an Eco-Friendly Sanitizing Wash on Spinach Leaf Bacterial Community Structure and Diversity**

**Sangay Tenzin 1,\*, Abiodun D. Ogunniyi <sup>1</sup> , Sergio Ferro <sup>2</sup> , Permal Deo <sup>3</sup> and Darren J. Trott 1,\***


Received: 16 February 2020; Accepted: 21 April 2020; Published: 24 April 2020

**Abstract:** Ready-to-eat (RTE) spinach is considered a high-risk food, susceptible to colonization by foodborne pathogens; however, other microbial populations present on the vegetable surface may interact with foodborne pathogens by inhibiting/inactivating their growth. In addition, sanitizers applied to minimally processed salad leaves should not disrupt this autochthonous barrier and should be maintained throughout the shelf life of the product. This investigation aimed at comparing the effects of a pH neutral electrochemically activated solution (ECAS), a peroxyacetic acid (PAA)-based commercial sanitizer (Ecolab Tsunami® 100), and tap water wash on the minimally processed spinach leaf microbiome profile for 10 days after washing. The bacterial microbiota composition on spinach samples was assessed by 16S rRNA pyrosequencing and downstream analyses. Predominant phyla observed in decreasing order of abundance were Proteobacteria, Bacteroidetes, Actinobacteria and Firmicutes corresponding with the dominant families *Micrococcaceae*, *Clostridiales Family XII*, *Flavobacteriaceae*, *Pseudomonadaceae*, and *Burkholderiaceae*. Bacterial species richness and evenness (alpha diversity) and bacterial community composition among all wash types were not significantly different. However, a significant difference was apparent between sampling days, corresponding to a loss of overall heterogeneity over time. Analysis of composition of microbiome (ANCOM) did not identify any amplicon sequence variants (ASVs) or families having significantly different abundance in wash types; however, differences (17 ASVs and five families) were found depending on sampling day. This was the first bacterial microbiome composition study focused on ECAS and PAA-based wash solutions. These wash alternatives do not significantly alter microbial community composition of RTE spinach leaves; however, storage at refrigerated temperature reduces bacterial species heterogeneity.

**Keywords:** *Spinacia oleracea* microbiota; electrochemically activated solution; peroxyacetic acid; sanitization; 16s rRNA pyrosequencing; amplicon sequence variants; alpha diversity; bacterial community composition

#### **1. Introduction**

A wide range of microbes, with distinct phylogenetic structure, is associated with the aerial organs (phyllosphere) of plants through parasitic or symbiotic interactions; in particular, bacteria are the most common microorganisms colonizing plant phyllosphere in comparison to fungi and archaea. The bacterial communities associated with edible leafy vegetables are less diversified than those of farm soil and coastal seawater habitats [1]. Actinobacteria, Bacteroidetes, Firmicutes, and Proteobacteria are the predominant bacterial phyla present in ready-to-eat (RTE) leafy vegetables (which are consumed raw, either treated or minimally processed) [2–6]. The core bacterial genera identified in most studies are *Pseudomonas*, *Sphingomonas*, *Methylobacterium*, *Bacillus*, *Massilia*, *Arthrobacter*, and *Pantoea* [2,3]. Human pathogens mostly associated with RTE leafy vegetables include *Escherichia coli* O157:H7, *Listeria monocytogenes*, and *Salmonella* spp. [7,8], but these are greatly affected by the vegetable type and bacterial community structure [9,10].

Lettuce and spinach are minimally processed RTE vegetables highly susceptible to colonization by foodborne pathogens [11]; therefore, various post-harvest sanitizing washing strategies are generally implemented to reduce spoilage and eliminate human pathogens. Today, the effectiveness of a post-harvest sanitizer is assessed based on its effect on the overall microbial populations, in addition to its propensity to reduce the microbial load and eliminate foodborne pathogens [12]. The composition of the microbiome community is assessed because the microbiome present on fresh produce is not only responsible for spoilage but rather acts as a natural biological barrier against spoilage organisms and pathogens, which constitute a smaller subset of the whole soil microbial population [13–15]. Furthermore, the bacterial microbiota on the surface of the plant inhibits or inactivates the growth of bacterial pathogen by producing acidic antimicrobial peptides and other secondary metabolites [16–18] that adversely affect the survival of the pathogen [19].

Bacterial population on RTE spinach is generally assessed using traditional culture-based techniques or specific polymerase chain reaction (PCR) to detect pathogens known for public health risk and quantify the population of indicator bacteria. Molecular techniques such as denaturing gradient gel electrophoresis and terminal restriction length polymorphism have been used for the analysis of 16S ribosomal RNA (rRNA) gene to understand the bacterial community of the phyllosphere on spinach leaves [20–22]. Contemporary next-generation sequencing techniques are now widely used for comprehensive analysis of the composition of bacterial community due to the increase in the depth of sequence readings and improved easier to use bioinformatics pipelines [23,24]. This method, in addition to providing information on the community structure, provides insights into the association of bacterial phyllosphere diversity with environmental factors [6,23], use of biocidal agents [6,23], and pesticides [6,25]. It also provides the interaction dynamics of the composition of the bacterial community with the various stages of plant growth, post-harvest, during processing and storage [3,23,26,27].

For leafy vegetable processing, chlorine- or peroxyacetic acid (PAA)-based sanitizers are commonly used. Chlorine is used for its effectiveness and low cost, whereas PAA for its activity over a wide pH range and limited reaction with organic matter. Electrochemically activated solution (ECAS) with an approximately neutral pH (6.5–7.5) has been suggested as a promising alternative washing solution with disinfection capability comparable to that of other commonly used disinfection chemicals such as chlorine and PAA [28–32]. Izumi [28] reported that neutral ECAS containing 50 mg/L of free available chlorine (FAC), completely inactivated the total bacteria on leaf surface. Guentzel et al. [31] reported a reduction of 4.0–5.0 Log<sup>10</sup> CFU/mL of *E. coli*, *S. typhimurium*, *S. aureus*, *L. monocytogenes*, and *E. faecalis* inoculated on spinach leaves, working with 100 mg/L and 200 mg/L of FAC.

The sanitizers used in washing RTE vegetables have a different influence on bacterial microbiota. Some sections of the bacteria composition of plants affect the survival of pathogens through competition for limited nutrients or production of growth inhibitors [16,19,33], and others facilitate the growth of pathogens through the metabolism of different carbon sources [24]. Chlorine-based washing has previously been reported to reduce the number of microbes that inhibit the growth of pathogens in lettuce and spinach [18]. Gu et al. [25] observed changes in the bacteria community in spinach leaves washed with chlorine. Tatsika et al. [34] reported a reduction in the richness of the bacterial community of RTE spinach without affecting bacterial diversity after washing the spinach leaves with vinegar. However, the effect of washing with ECAS on the composition of the microbiome of RTE spinach leaves compared to that of PAA sanitizer has not previously been assessed.

This study evaluated the effect of an ECAS at neutral pH with proven efficacy against foodborne pathogens and in reducing the overall bacterial load in RTE spinach [28,35–37] focusing on the structure of the bacterial community present on RTE spinach leaves. We compared the changes in the profile of the bacterial microbiome in minimally processed fresh spinach leaves washed with tap water, PAA (50 mg/L), and ECAS (50 mg/L and 85 mg/L of FAC) on days 0, 5, and 10 after the sanitizing wash and storage at 4 ± 1 ◦C. Furthermore, a comparative analysis of the bacterial composition was performed through an analysis of the composition of microbiomes among all the treatment types and sampling days.

#### **2. Materials and Methods**

#### *2.1. Sanitizers Treatment of Spinach Leaves*

Freshly cut Tasmanian baby spinach leaves, grown in soil, stored and shipped at 4 ± 1 ◦C, were used within 24–48 h of receipt. ECAS (produced by Ecas4 Australia Pty Ltd., Mile End South, Adelaide, South Australia, Australia) was also stored at 4 ± 1 ◦C and used within one week of production, diluted in Milli-Q water (Milli-Q academic A10 deionizer, Millipore Corporation, Molsheim, France) to 50 mg/L and 85 mg/L of FAC. Peroxyacetic acid (Ecolab Tsunami® 100, which nominally contains 30–60% acetic acid, 10–30% peroxyacetic acid and 10–30% H2O2), commonly used as a post-harvest sanitization of fresh agriculture produce, was used at 50 mg/L of PAA. The temperature, pH, and oxidation-reduction potential (ORP) of ECAS, Tsunami® 100, and tap water were measured using a portable MC-80 m (TPS Pty Ltd., Brendale, Queensland, Australia). The quantities of free and total chlorine in ECAS were measured using a Free Chlorine Checker® HC-HI701 and a Total Chlorine Checker HC-HI711, both from Hanna Instruments (Keysborough, Victoria, Australia). The amount of PAA in Tsunami® 100 was measured using specific test strips (Hydrion PAA160 Peroxyacetic Acid (PAA) Sanitizer Test Strips, Brooklyn, New York, USA).

Three samples of spinach leaves (200 g each) were washed with 800 mL of either tap water (control, pH 7.4 ± 0.1) or sanitizers (52 ± 2 mg/L of PAA, ORP of 492 ± 15 mV, pH 3.6 ± 0.1; ECAS with 48 ± 4 mg/L of FAC, ORP of 833 ± 13, pH 7.1 ± 0.2; and ECAS with 82 ± 4 mg/L of FAC, ORP of 864 ± 13, pH 7.0 ± 0.2) at 4 ± 1 ◦C for 60 s, and the excess liquid removed using a salad spinner at 70 rpm for 30 s. Samples (3 × 25 g) from each treatment were homogenized in 225 mL of sterile 0.1% peptone water for 60 s in a stomacher (BA 6021 Stomacher, Seward Ltd., Worthing, UK) immediately after treatment (day 0) and stored at −20 ◦C. Spinach samples from each treatment were stored at 4 ± 1 ◦C and further processed on day 5 and day 10, as described by Ogunniyi et al. [37].

#### *2.2. Samples Preparation for Variable V3-V4 Region Sequencing*

Samples stored at −20 ◦C were thawed in a shaking incubator kept at 20 ◦C for about 45 min. Samples from each type of treatment and for the various sampling days were centrifuged at 15,000× *g* for 15 min; the supernatants were discarded, and the pellets were frozen at −20 ◦C for DNA extraction. The DNA from the samples was then isolated and purified using the Qiagen QIAamp DNA Mini Kit (Cat. #51304, Germantown, MD, USA) as per the manufacturer's instructions. DNA concentrations were measured using the multi-mode microplate reader (CLARIOstar Plus).

The amplicon-sequence PCR was performed using the 16S DNA V3-V4 region primers from Klindworth et al. [38] and following the guidelines provided in "16S Metagenomic Sequencing Library Preparation" (Part #15044223 Rev. B) [39]. PCR products were confirmed to produce a single amplicon size of ~460 bp after electrophoresis on a 2.0% agarose gel. Aliquots (25 µL) of all samples were subjected to clean-up PCR, index PCR, second clean-up PCR and MiSeq 16S metagenomic sequencing at the South Australian Health and Medical Research Institute (SAHMRI), Adelaide, South Australia. The data analyzed were based on Illumina Miseq sequences of 300 bp paired amplicon sequences from the V3 and V4 region of 16S rRNA gene from baby spinach leaf samples with and without sanitizing treatments. The profile of the demultiplexed fastq paired-end reads was assessed using FastQC [40] for

sequence quality scores and adapter contents. First, the forward reads were truncated at position 260 and the reverse at position 220 to remove low quality reads (<26 Phred). Trimming was set up for the first 20 nucleotides for forward reads and 10 nucleotides for reverse reads to remove primer sequences and low-quality reads. The trim and filter parameters were performed jointly on the paired-end read by setting a maximum of two errors expected per read [41], so that both paired-end reads passes the filter for the pair to pass. Downstream analysis to infer the amplicon sequence variants (ASVs) was performed in R version 3.5.3 [42] using the DADA2 workflow that resolves variants that differ by a single nucleotide [43]. Taxonomic assignments were made for the sequence variants data implementing the naïve Bayesian classifier method [44] using the SILVA reference data set (version 132) [45] formatted for DADA2 [46]. The DECIPHER R package [47] was used for the alignment of multiple sequences, and a phylogenetic tree was built using the phanghorn R package [48]. The phyloseq R package [49] was used to synthesize sample data, phylogeny and taxonomic assignment objects into a single phyloseq object. Further downstream analyses and graphical visualization of the microbiome data were performed in phyloseq [49] and Shiny-phyloseq [50] R packages.

#### *2.3. Statistical Analysis*

Calculations of alpha diversity indexes were performed in R versions 3.5.3 [42] with the phyloseq R package [49]. The Shannon and inverse Simpson indexes were compared among the variables since these indexes consider the richness and evenness that are powerful in providing insights into the structure of the microbial community [51,52]. In addition, the number of ASVs (species) was estimated using the observed richness and Chao1 richness estimator. The alpha diversities among the groups of samples were statistically tested using the analysis of variance (ANOVA) test to evaluate any differences in the microbial composition among treatment types and sampling days, as both variables (treatment type and sampling day) had more than two levels and the data distribution was normal according to the Shapiro–Wilk normality test. Tukey's honest significance test as a post hoc test was performed on the ANOVA results to compare within-group alpha diversity.

Measurements of samples similarity (beta diversity) with the R phyloseq and vegan packages [53] were also performed at ASV level based on non-metric multidimensional scaling (NMDS) Bray–Curtis dissimilarity [54] and Unifrac distances [55], which include abundance and phylogenetic information respectively, in addition to taxon counts. Statistical significance testing among the groups, such as the type of sanitization and the days post sanitizing treatment, was performed using permutational multivariate analysis of variance (PERMANOVA) [56] using the adonis function in the R package vegan. The community pattern of microbial composition among the groups using taxon dissimilarity information was visualized by NMDS Bray–Curtis and Unifrac ordination methods. In addition, microbiota heterogeneity, a measure of dissimilarity of the beta diversity (Bray–Curtis) of each sample with respect to the group, was compared between the various types of treatment (sanitizing and control washes) and days of sampling to evaluate the differences in homogeneity of each treatment group and homogeneity of sampling day using the R package microbiome [57]. Statistical tests for multiple variables within the type of treatment and sampling days were performed by the betadisper function on distance matrix (Bray–Curtis), and an ANOVA was performed to compare the variances between pairs of groups using the permutest function by setting the pairwise variable to true and the number of permutations to 1000 on R package vegan [56].

Analysis of differentially abundant taxa among the types of sanitization and days 0, 5, and 10 post treatment, at ASV and family level, were performed using analysis of composition of microbiomes (ANCOM) [58] plugin in QIIME 2 [59], at ASV and genus levels. For ANCOM analysis, ASVs present in less than three samples and ASV frequencies below fifty were removed before the analysis.

#### *2.4. Data Submission*

The access number for raw reads submitted to GenBank-SRA is PRJNA576552.

#### **3. Results**

We characterized the overall bacterial composition of minimally processed spinach leaves using high-throughput amplicon sequences from the V3–V4 region of the 16S rRNA gene. Moreover, changes in bacterial composition at phyla and families levels were compared for the washed samples and the control (unwashed) on day 0, day 5 and day 10 post sanitization, and between the types of washing (ECAS, Tsunami® 100, and Tap Water). In addition, the differences in bacterial diversity associated with the days post-treatment and the types of sanitizer were evaluated.

#### *3.1. Composition of Spinach Bacterial Community*

Overall, a total of 1,093,364 ASVs were observed, with a maximum of 113,737 and minimum of 39,474 reads. After removing uncharacterized phyla and contaminants and normalizing the data to the lowest number of reads (1000), the total number of ASVs reduced to 383,290 with a minimum of 1044 (observed for samples washed with ECAS at 85 mg/L of FAC on day 0) and a maximum of 50,871 (observed for samples washed with ECAS at 50 mg/L of FAC on day 5). The above reads were assigned to 12 distinct phyla, with the majority identified as Proteobacteria (2949 distinct ASVs), followed by Bacteroidetes (1876 ASVs), Actinobacteria (756 ASVs) and Firmicutes (396 ASVs). All other phyla had ≤8 ASVs (Table 1) and were excluded from further analysis [27] as the percentage abundance of these phyla were approximately 0.1% which would not affect the biological interpretation. All ASVs were assigned to one of 65 bacterial family identified and 84% of reads were further assigned to different bacterial genus with 158 genera identified. The five most abundant families identified were *Micrococcaceae* (28.2%), *Clostridiales Family XII* (19.7%), *Flavobacteriaceae* (17.9%), *Pseudomonadaceae* (12.8%), and *Burkholderiaceae* (10.1%). The five most abundant genera identified were *Exiguobacterium* (19.7%), *Flavobacterium* (17.7%), *Arthobacter* (15.4%), *Pseudomonas* (12.6%), and *Paeniglutamicibacter* (10.3%) (Table S1).


**Table 1.** Abundance and percentage abundances of phyla present in spinach leaf samples identified from 16S rRNA gene sequences analyzed using DADA2 package in R and taxonomic assignment performed according to the SILVA rRNA database.

The overall relative abundances (RA) of phyla observed for all types of sanitization wash are presented in Figure 1a, and the relative abundances for the samples collected immediately after treatment (day 0) as well as on day 5 and day 10 after storage at 4 ◦C are presented in Figure 1b. On day 0, the phyla Proteobacteria had the highest RA (0.36 ± 0.07), while the phyla Actinobacteria had the lowest RA (0.18 ± 0.05). On day 5, Proteobacteria was still the most abundant phyla (0.35 ± 0.08), whereas phyla Bacteroidetes was the least abundant (0.09 ± 0.01). However, on day 10, Actinobacteria was the most abundant phyla (0.34 ± 0.05) and Firmicutes was the least abundant phyla (0.10 ± 0.05). The relative abundances of bacterial taxonomy at order level for sanitization wash types and sampling days are presented in Supplementary Figure S1.

**Figure 1.** Relative abundance of phyla (Proteobacteria, Bacteroidetes, Firmicutes, Actinobacteria) for samples collected (**a**) after the sanitizing wash (no wash, tap water, peroxyacetic acid (PAA) at 50 mg/L, ECAS at 50 mg/L and 85 mg/L of free available chlorine (FAC)), and (**b**) immediately after the treatment (day 0) and on days 5 and 10 post sanitizing wash.

#### *3.2. Alpha Diversity*

– The alpha diversity metrics (Figure S2) and the Shapiro–Wilk tests for normality showed that the data were normally distributed. The alpha diversity measures for all samples are presented in Table 2. The mean ratio between observed to expected (Chao1) richness was >0.99 for all samples. The lowest Shannon (3.4), Inverse Simpson (17.6), and richness (Chao1 = 60) indexes were recorded for samples that were not washed (control) on day 0. The highest Shannon (5.6) and richness (771) indexes were recorded for the samples that were washed in ECAS at 85 mg/L of FAC on day 0, while the highest Inverse Simpson index (134.5) was observed for the no-wash control on day 5. Species richness (Shannon diversity and Inverse Simpson indexes) and species evenness (Chao1 and abundance-based coverage estimator, ACE) measures of the bacterial community structure were assessed for the four types of treatment plus control and the three sampling days. For all samples washed with sanitizers, the Shannon and Inverse Simpson diversity measures were higher than those found for the no-wash and tap water wash, but these measures were not significantly different (ANOVA and Tukey's honestly significant difference (HSD)). Similar results on species richness (Chao1 and ACE) were observed, with no significant differences between all types of washing (Kruskal–Wallis and pairwise Wilcox (FDR corrected) (Table S2).


**Table 2.** Alpha diversity metrics of species richness (Shannon and Inverse Simpson & Fisher) and evenness (Chao1 and abundance-based coverage estimator, ACE) for all samples.

#### *3.3. Bacterial Diversity Associated with Treatment Type and Sampling Day*

The PERMANOVA analyses of microbial communities for the different types of treatment were not significantly different among all the variables tested (*p* = 0.053). PERMANOVA analysis of Bray–Curtis distances for sampling days determined that the microbial communities were significantly different on sampling days (*p* = 0.006) (Table 3a). Moreover, non-metric multidimensional scaling (NMDS) cluster analysis showed that the microbial communities for different treatment groups did not cluster into distinct treatment groups (Figure 2a); however, the bacterial communities on day 5 and day 10 assembled distinctly, with a divergent microbial community observed for day 0 (Figure 2b). Also the quantification of the group divergence between the treatment types (ECAS at 50 and 85 mg/L of FAC, tap water and PAA washing) plotted as a box and whisker diagram showed that the group homogeneity among treatment types did not differ (Figure 2c). However, it shows that the microbiota of the ECAS and PAA wash treatments were more homogenous, whereas the tap water wash and the no wash (control) samples were more divergent (Figure 2c). The group divergence measurement for the sampling days shows that samples on day 0 had a higher value (>0.7), indicating that the composition was more heterogeneous. On the contrary, samples on day 5 and day 10 had lower divergence values (>0.3 and >0.2, respectively), indicating homogenous microbiota (Figure 2d).

The statistical homogeneity test of the multivariate dispersion of microbial composition among the types of treatment and the sampling days showed that the variances between the different washing treatments were not significantly different. In the case of the sampling days, the composition changes between day 5 and day 10 were not significantly different, but changes between day 0 and day 5 and between day 0 and day 10 were significantly different (*p* < 0.05) (Table 3b).

**Table 3.** (**a**) Permutational multivariate analysis of variance (PERMANOVA) results based on Bray-Curtis dissimilarities using abundance data for treatment types and sampling days. (**b**) Analysis of variance (ANOVA) pairwise comparison tests of dispersion of microbial composition among sampling days (significant if *p* value < 0.05).


(**a**) Df—degrees of freedom; Sum Sq—sum of squares; F.Model—F value by permutation. R <sup>2</sup>—the effect size. Boldface indicates statistical significance with *p* < 0.05 based on 1000 permutations.


Boldface indicates statistical significance with *p* < 0.05 based on 1000 permutations.

– **Figure 2.** Microbial community cluster analysis of assembled sequence variants (ASV) non-metric multidimensional scaling (NMDS) based on Bray-Curtis dissimilarity index for (**a**) electrochemically activated solution (ECAS), tap water, and PAA washing, and (**b**) sampling days. Dispersion of the beta diversity group based on Bray–Curtis dissimilarity index for (**c**) ECAS, tap water, and PAA washing, and (**d**) sampling days.

— — — —

– – –

#### *3.4. Taxa Di*ff*erences Among the Di*ff*erent Sampling Days*

ANCOM performed with a false discovery rate (FDR) of 0.05 identified 17 ASVs and four families with significantly different abundances (ANCOM W ≥ 5) among the different sampling days (Table 4). No significantly different ASVs and families were identified for the various washing treatments. The relative abundance ratios of ASVs and taxa rank family were calculated using day 0 as the basis for displaying the relative abundance in Figure 3a,b, respectively. Out of 17 ASVs identified as significantly different, 4 ASVs on day 5, and 8 ASVs on day 10 had an increase in relative abundance. An ASV identified as belonging to the *Pseudomonadaceae* family (unclassified genus) had the highest relative abundance (RA ratio of 8.82), followed by an ASV belonging to the *Moraxellaceae* family (*Alkanindiges illinoisensis*—RA ratio of 6.41) on day 5. ASVs identified as belonging to the *Flavobacteriaceae* (unclassified genus) and *Pseudomonadaceae* (unclassified genus) families had RAs of 4.3 and 3.2, respectively. ANCOM family-level analysis revealed that *Pseudomonadaceae* had the highest relative abundance (2.9) on day 5. The relative abundance of three additional families (*Spingobacteriaceae*, *Flavobacteriaceae* and *Xanthomonadaceae*) on day 5 and day 10 were lower than on day 0 (Table 4).

**Table 4.** Taxa (17 ASVs) and genera (5 genera) identified as significantly different in abundance on sampling days 0, 5 and 10 by analysis of composition of microbiomes (ANCOM) analysis at a false discovery rate (FDR) of 0.05. The higher the W value, the more significant are the differences in abundance levels between the sampling days.


\* Amplicon sequence variants, <sup>a</sup> Taxa are identified from Greengenes database. \*\* Indicate rejected null hypothesis. o-order, f-family, g-genus, s-species.

**Figure 3.** The relative abundance (RA) ratio of (**a**) ASVs (17 ASVs with taxa identification in Table 4) and (**b**) taxa rank family (4) identified as significantly different in abundance on sampling days 5 and 10 by ANCOM analysis at a false discovery rate (FDR) of 0.05. The RA ratio is calculated as RA on day 5 or day 10 divided by RA on day 0. f-family.

#### **4. Discussion**

This study investigated the microbiome profiles of RTE spinach leaves washed with different sanitizers (ECAS, PAA) and compared with leaves washed with tap water and not washed at all (control), at three time points over 10 days (day 0, day 5, and day 10). Although a higher proportion of ASVs was found compared to previous studies [26,27], their richness and evenness (alpha diversity) did not significantly differ among the types of sanitizer and the sampling points. We also found that the types of sanitizing washing, apart from a reduced heterogeneity over time, did not significantly influence the community structure of the bacteria (beta diversity). ANCOM analyses identified that the composition of ASVs and families changed significantly over the sampling days.

The number of ASVs identified (>2000) in the present study was much higher than that observed in the spinach leaf microbiome profiling studies by Gu et al. [26] and Söderqvist et al. [27], who identified 673 and 190 operational taxonomic units respectively. In addition, 12 phyla were identified in this study compared to the four phyla observed by Söderqvist et al. [27] and 14 phyla detected by Gu et al. [26]; however, the number of predominant phyla (*n* = 4) and their relative proportions are similar in all three studies. In agreement with previous observations, the phylum Proteobacteria showed the highest total abundance on day 0, followed by phyla Bacteroidetes, Firmicutes, and Actinobacteria [3,6,26,34]. The basal bacterial microbiome of RTE spinach leaves is therefore very similar to that of other minimally processed fruits and vegetables [3,6,27,34,60].

Our analyses also showed that the Shannon and Inverse Simpson diversity indexes and richness (ACE and Chao1) measures did not differ significantly among all spinach samples. Furthermore, the community composition of bacteria (beta diversity) for all types of washing did not differ significantly, indicating that ECAS treatments did not affect bacterial microbial diversity. This could be seen as a good outcome, since it has been suggested that the microbiome on fresh produce is not responsible for spoilage but acts as a natural biological barrier against spoilage organisms and pathogens [13–15]. On the other hand, a significant grouping of spinach microbial community structures was observed for sampling days and reduction over time of the heterogeneity of bacterial composition. The reduction in heterogeneity can be attributed to the reduction in the relative abundance of phylum Proteobacteria on day 5 and day 10, in accordance with the reduction observed by Gu et al. [26] in RTE spinach leaves washed with chlorine and stored at 4 ◦C for a week. Moreover, the microbiome community on day 10 clustered distinctly due to a significant increase in the relative abundance of Bacteroidetes, similar to that observed by Gu et al. [26] when the spinach leaves were stored at 4 ◦C for a week.

ANCOM is a method based on compositional log-ratios to detect differences in relative abundance and has been used to detect taxa abundance in the spinach microbiome at ASV and family level. Taxa at ASV and family level for the different types of treatment were not significantly different, but differences in ASVs and family-related abundances were identified at different sampling days. ASVs identified as *Pseudomonadaceae* and *Moraxellaceae* families, and the order Bacillales (unclassified family) had a high relative abundance on day 5. The increase in the relative abundance of these families of bacteria (*Pseudomonadaceae* and *Moraxellaceae*) has been correlated strongly with the spoilage of leafy vegetables at cold storage temperatures [34]. Increases in the relative abundance of the order Bacillales were also observed by Söderqvist et al. [27] and have been positively correlated with the increase in the viable counts of bacteria causing food safety concerns (*Yersinia enterocolitica*, *Listeria monocytogenes* and *E. coli* O157:H7) [27]. A significant increase was observed for four ASVs (classified as *Pseudomonadaceae*, *Flavobacteriaceae*, *Micrococcaceae* and *Oxalobacteraceae*) on day 10 and it is interesting to note that the relative abundance of the order Flavobacteriales was negatively correlated with foodborne pathogens in a previous study [27]. The predominance of *Micrococcaceae* and *Oxalobacteraceae* may be explained by their ability to grow at extremely low temperatures [61]; they are considered putative protectors against *Rhizoctonia* (fungal) rot of root crops [62]. The abundance of the family *Xanthomonadaceae* was significantly reduced on day 10, as observed by Lopez-Velasco et al. [3] and Schwartz et al. [60]. Similarly, the abundance of *Spingobacteriaceae* was significantly reduced, and the order Sphingobacteriales was correlated positively to *Escherichia coli* O157:H7 counts and negatively to *L. monocytogenes* and *Y. enterocolitica* counts [27].

#### **5. Conclusions**

To our knowledge, this study represents the first documented profile of the bacterial microbiome present on minimally processed RTE Australian spinach treated with ECAS. We have shown that washing with a neutral ECAS did not significantly change the composition of the bacterial communities compared to washing with PAA (Tsunami® 100) and tap water. In addition, complete changes over time in the community composition of bacterial species have been documented during storage at refrigeration temperature (4 ± 1 ◦C) on day 5 and day 10 after washing treatments, compared to day 0. The information that ECAS does not change the structure of the bacterial community could help select an environmentally friendly biocidal agent capable of meeting the aesthetic needs of current consumers and production industries.

**Supplementary Materials:** The following are available online at http://www.mdpi.com/2076-3417/10/8/2986/s1, Figure S1. Relative Abundance ratio of bacterial taxonomy level order for sanitization wash types and sampling days.; Figure S2. Visualization of Shannon and Inverse–Simpson diversity (alpha-diversity) and Chao and ACE richness metrics of all samples.; Table S1. Total abundances and percentage abundances of most abundant taxa at family and genus level.; Table S2. Probability values of analysis of variance (ANOVA), Tukey's HSD test on ANOVA of Shannon diversity index, Kruskal–Wallis H test, and Wilcoxon pairwise rank-sum test of Chao1 richness comparing alpha diversity metrics among the types of sanitizing treatment (Treatment Types) and day post-sanitation treatment (Day Sampled). Alpha diversity was not significantly different among the types of treatment and the sampling days, as determined by ANOVA and Tukey's HSD test.

**Author Contributions:** Conceptualization, S.T. and A.D.O.; Data curation, S.T.; Formal analysis, S.T.; Methodology, S.T. and A.D.O.; Supervision, D.J.T. and P.D.; Visualization, S.T.; Writing, original draft, S.T.; Writing, review and editing, S.T., A.D.O, S.F., P.D., and D.J.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** Ecas4 Australia Pty Ltd. funded the study, including the 16s rRNA pyrosequencing.

**Acknowledgments:** S.T. was supported by the Endeavour Postgraduate Scholarship. ANCOM analysis was performed with supercomputer resources provided by the Phoenix HPC service of the University of Adelaide. We would like to acknowledge Mark Van der Hoek (David Gunn Genomics Facility, South Australian Health and Medical Research Institute, Australia) for his generous technical guidance on preparing samples for genomic analysis.

**Conflicts of Interest:** Page: 12, Ecas4 Australia Pty Ltd. played no role in the study design, data collection and interpretation, and decision to submit the article for publication. The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

#### *Article*

## **Minimizing the Environmental Impact of Industrial Production: Evidence from South Korean Waste Treatment Investment Projects**

## **Olga A. Shvetsova and Jang Hee Lee \***

School of Industrial Management, Korea University of Technology and Education, 1600, Chungjeol-ro, Byeongcheon-myeon, Dongnam-gu, Cheonan-si, Chungcheongnam-do, Cheonan City 31253, Korea; shvetsova@koreatech.ac.kr

**\*** Correspondence: janghlee@koreatech.ac.kr

Received: 4 April 2020; Accepted: 14 May 2020; Published: 18 May 2020

**Abstract:** This research deals with the theoretical and practical issues of investment support activities for industrial waste management in developed countries, based on the example of South Korea. The main goal of this research is the evaluation of waste treatment investment projects and understanding their impact on the development of environmental policies. The problems of forming the sustainable systems for controlling the disposal of industrial wastes are being studied. The authors discuss the practical application of environmental policies and modern technologies of South Korean companies in the field of industrial waste processing. The approaches of waste investment project's evaluation are applied and multi-criteria decision making (MCDM) methods were discussed for various cases and applications. Using MCDM methods, the authors study the effectiveness of investment projects in waste treatment activities in Korea. The analyses of MCDM methods are implemented in this research to provide some instructions on how to effectively apply these methods in waste treatment investment project analyses. Furthermore, the authors propose a combination of multi-criterial selection and interval preferences to evaluate waste treatment projects. The proposed approach improves the method of calculating economic efficiency based on a one-dimensional criterion and sensitivity analysis. The main results of this research perform the investment impact and risk-analysis on the environmental policies development.

**Keywords:** environmental impact; waste management; industrial production; investment project; risk evaluation; multicriterial approach

#### **1. Introduction**

The rapid industrialization of South Korea has three characteristics: authoritarian state control over the industrial sphere, high economic growth rates due to export heavy industry, and rapid capital accumulation [1,2]. In particular, heavy industry has had a significant impact on the environment. The problems of forming sustainable systems for controlling the disposal of industrial wastes are become more and more apparent in South Korea. Business and government organizations discuss the practical application of environmental policies and modern technologies of Korean companies in the field of industrial waste processing. That is why it is necessary to discover all of the industrial factors which have a strong impact on the environment, and to develop effective models of waste treatment processing. Large industrial complexes involved in heavy industry have strongly polluted the earth, water and air [3]. In the areas that neighbor industrial complexes, sulfur was found on rice leaves, and some elementary school students fainted on the way to school due to toxic gas emitted from factory stacks. The economic development of South Korea was achieved at the expense of the environment. The same process has been followed by most European countries [4].

Solid waste was not considered an environmental issue in Korea for a long time. Bekun et al. in 2019, and Paramati in 2018 [5–7], discuss that there was no concern about how much solid waste was generated, and any waste collected from households was dumped in open landfills without regard for environmental hazards. At the same time, the government charged only a fixed amount for household waste disposal services, regardless of how much waste was disposed of. The Korean economic boom over the past few decades has led to a significant increase in solid waste generation. The Korea Institute of Environmental Technology Development in 1996 announced that, in just 20 years, the total amount of municipal solid waste generated per day increased from about 12,000 tons in 1970 to 84,000 tons in 1990 [8]. As the amount of waste increased rapidly, several problems related to waste disposal developed in Korea.

The Republic of Korea officially recycles more than 85 percent of all waste, as stated in a 2017 government study [8]. On the other hand, illegally dumped waste can be found in rural areas of the country. Despite this criticism, the Korean waste management system is quite effective.

According to the 2018 Korea Environmental Review (ECOREA), 10.3 percent of the country's waste (including household and business waste) was disposed of, 6.3 percent was burnt, 82.4 percent was recycled, and 1.0 percent was dumped into the sea. As of 2012, 97.3 percent of construction waste and 76.5 percent of commercial waste was recycled, 14.9 percent of them was buried in landfills, 6.5 percent was burnt, and 2.1 percent dumped into the sea. Moreover, 54.4 percent of designated waste (the term refers to industrial waste) was processed, 16.4 percent was burnt, 23.0 percent was buried and 6.2 percent was treated with other measures (storage) [9]. Even though the percentage of waste processed during incineration or processing is increasing annually, the rate of processing waste generated in landfills and discharges into the ocean is reduced. The reduction in waste discharged into the sea is the result of a ban on the discharge of wastewater, wastewater from food waste, and cattle wastewater into the ocean in 2012–2013; treatment methods for these types of waste have been replaced by incineration or recycling (Figure 1).

**Figure 1.** Trends in Municipal Solid Waste treatment in South Korea, 2000–2018. Source: Korea's Environmental Review, 2018, Ministry of Environment, ECOREA, http://eng.me.go.kr [9].

Korea has been one of the fastest growing OECD economies over the past decades, driven by a large export-oriented manufacturing sector [10]. However, growth has come with high pollution and resource consumption. With increasing energy demand, greenhouse gas (GHG) emissions have risen significantly and air pollution remains a major health concern. Despite significant improvements in wastewater treatment, diffuse pollution increasingly affects scarce water resources. Urbanization and industrialization also put considerable pressure on biodiversity. The environmental challenges are exacerbated by Korea's population density—the highest in the OECD [11]. Access to environmental goods and services, and exposure to environmental risk vary significantly by region.

To tackle these challenges, Korea has invested considerable effort into improving environmental management, for example, by introducing strategic environmental assessment, reforming the environmental permitting system and strengthening air and water quality standards. Korea introduced the world's second largest emission trading scheme, and remains one of the most innovative countries in terms of climate change mitigation technology. However, coal is set to remain integral to energy production, and road transport continues to be supported as the dominant form of mobility. Current energy prices and taxes do not reflect the environmental costs of energy production and use. The OECD Review Report in 2017 emphasizes that Korea needs to align its energy and climate policies to reduce GHG emissions to 37% below business-as-usual levels by 2030 [12].

There are many investment projects currently being supported in Korea. Korea's transition to a low-carbon economy is vital for its future prosperity. This is a core message of the OECD report in 2018, which provides 45 recommendations to help Korea pursue the implementation of green growth and strengthen environmental performance [13]. To attain this end, the Korean Government incorporates foreign investment projects in its development of a National Green Strategy. The problem refers to considerations about risk identification for such investment projects, and the further development of waste treatment technologies and methods.

#### **2. Environmental Policies and Waste Management Methods: Case Study of South Korea**

Starting from 1978, the Korean Government actively enacted comprehensive environmental protection legislation and policies. Since that time, approximately every 10 years, the Korean Environmental Ministry and other government institutions—together with companies—work together to improve environmental policies and develop new technologies for waste treatment. In this section, we discuss the successful waste management methods and regulations in Korea.

#### *2.1. Waste Management Methods and Environmental Policy in Industrial Production*

Industrialized countries face the challenge of quickly and safely disposing of waste. Non-biodegradable and toxic wastes, such as radioactive residues, can cause irreparable damage to the environment and human health if they are not disposed of strategically.

Although waste management has been a concern for several decades, the main problem has been the enormous proportions of generated waste due to population growth and industrialization—the two main factors contributing to waste generation. Although some successes have been achieved in waste disposal methods, they are still insufficient. The challenge is to discover new and dangerous methods of waste disposal and use them.

Several global companies and local Korean companies (Samsung, CMG, Ovarro, Presona AB, Amandus Kahl GmbH & Co, Ion Science Ltd., Tana Oy, AMETEK Brookfield and others)—together with leading scientists—have developed a unique technology for processing solid waste [14,15]. The latest international developments in the field were combined into a single complex, including the unique technology of catalytic pyrolysis. The uniqueness of this technology is that the garbage is not burned whatsoever. Newly patented in 2013, catalytic pyrolysis makes it possible to completely recycle solid waste without careful sorting measures. It turns waste into valuable commercial products of a high quality and secondary raw materials. Based around the method, a recycling complex of a new type was created. However, the problems of risk-management and cost-analysis of these methods' implementation and operation quickly arose. Nowadays, these companies try to invent new risk-analysis methods within the waste management process to reduce operational costs. The managers understand that project-management concepts help them understand certain basic requirements and cooperative management strategies.

Innovative technology deserves attention. The complex processing of solid industrial waste in a vortex air-mineral flow—which is based on in-depth study of the chemical, mineral composition and technological properties of the secondary raw materials, and is an individual solution for each type of industrial solid waste with maximum processing efficiency—eliminates the use of water and chemical reagents.

Industrial waste, as it turned out, can be used as a secondary resource in various industries, especially in the production of building materials, products and structures. The main tasks in the field of processing industrial waste should be aimed at improving existing processes, creating and implementing the latest technology. It is necessary to develop and implement measures aimed at implementing waste directly in the manufacturing process.

There are six effective waste disposal methods in South Korea which foreign and domestic companies prefer to venture into:


There is a three-tier approach to assessing the risk associated with air and water emissions from waste management facilities. With this approach, an acceptable level of protection is provided at all levels, but with each progressive level, the level of uncertainty in the risk analysis decreases [19–22]. Reducing the level of uncertainty in risk analysis can reduce the level of control required by the waste management unit (if necessary for the site), while maintaining an acceptable level of protection. An enterprise performing a risk assessment incurs higher costs associated with a more complex risk assessment in exchange for greater confidence and potentially lower construction and operating costs. The advantages and relative costs of each tier are outlined below in Table 1.


**Table 1.** Three-tiered approach for assessing the risk of waste treatment projects.

Source: adopted from Tonmoy F. N., Rissik D. and Palutikof J. P. A three-tier risk assessment process for climate change adaptation at a local scale. Climatic Change, 153, 2019, pp. 539–557 [22].

We try to solve these practical problems associated with conducting an assessment of the risk of waste management in an organization, and offer a multi-level structure that supports the training path necessary for adaptation, allowing organizations to optimize their resources for adaptation, systematically increase the knowledge base on waste management, and develop targeted interaction strategies.

#### *2.2. Waste Management Regulations and Environmental Policies in South Korea*

Currently, the regulatory regime for environmental protection in South Korea consists of legislative acts, executive decrees, ministerial decrees and regulatory acts related to the general environment, including:


The basic framework is laid in the Framework Act on Environmental Policy (FAEP), which contains the main objectives of environmental policy, including pollution prevention and natural resource management for sustainable use. To create such a sustainable waste management system, Korea went through many regulatory improvements and discoveries, which are presented below [23–27].

In 1995, a volume-based waste disposal system was introduced, which is a waste containment system that uses the principle of payment for emissions. This system, which is representative of market incentives in Korea, represents a shift from the old system in which a fixed fee was charged regardless of the volume of waste disposed, to a system in which a proportional charge is applied to the volume discharged. As reported, this provides an incentive to reduce discharge and increase processing. In addition, a Promotion of Installation of Waste Facilities and Assistance Law was issued in 1995 to related areas in order to prevent the NIMBY phenomenon (not in my backyard) due to the installation of incinerators, and also to help resolve social conflicts through means such as relief projects for affected communities [28].

In the 2000s, the foundation was created for a society of resource recycling. According to this master plan, the waste was not just recycled, but recycled as a resource. Korea is currently pursuing a "Zero Waste" policy that seeks to use waste as a source of resource, in addition to minimizing waste generation, which was emphasized by the authors of [29].

Since the late 2000s, a reduction in greenhouse gases has been necessary due to the rapid increase in prices for resources and energy, as well as global warming. Therefore, the government emphasized the need to restore resources and energy from waste. In September 2011, the Korean Ministry of the Environment developed the First Framework Plan (2011–2015) for the management of resources, to form the basis for waste management and thereby promote green growth aimed to at encouraging the effective societal dealing with resources (without waste) [30]. In addition, the measure to facilitate the transition to a resource circulation society (2013) facilitates the collection and transportation of recyclable resources through the free collection of large household electronic waste, the consolidation of the sorting system, and the expansion of equipment installation. Waste energy utilization facilities and other similar complexes are the foundations of a waste recycling society [31]. The creation of a market for processed products and support for their industries was also announced.

Currently, intensive advertising and educational programs are beginning to educate the population on how to classify recyclables and use designated garbage bags, and, more importantly, to educate the public on the purpose and significance of waste reduction [31–33]. Due to difficulties in monitoring the levels of flies in rural areas, residents are not required to take out the trash in specially designed trash bags. Instead, public trash and recycle bins were installed in rural areas, and the fees are shared among all residents of the community. Local officials noted that low-income people were offered appropriate assistance to alleviate the burden of paying for MSW. It is interesting to compare waste quantities in different Asian countries; many of these countries employ similar approaches to waste treatment (Table 2).

The Framework Act on Resource Circulation (FARC) was adopted in 2016 to form the basis for implementing this policy; the Korean government has been applying it since 2018 [33]. The country intends to transform the economic structure, focused on mass production and mass waste, into a much more stable and efficient, resource-oriented structure at a fundamental level. The provisions of this framework can be divided into three categories, each of which creates a framework for the circulation of resources, stimulates the circulation of resources and supports the processing industry. It introduces new waste management related programs such as the Recyclable Resource Recognition Program (RRRP), Resource Circulation Performance Management Program (RCPMP), Waste Disposal Tariffs, etc. [33,34]. The government expects to obtain economic, environmental and social benefits

by preventing environmental pollution, but is also aware of the fact that the country needs to make further efforts to change the paradigm of its waste management policy.


**Table 2.** Comparison of waste quantities among countries of Asia.

Source: Environmental Protection Department of Hong Kong; Environmental protection authority of Taiwan; Ministry of Environment of South Korea; National Environment Agency of Singapore [32].

#### *2.3. Waste Management System in Samsung Electronics Corporation*

To protect the environment in times of crisis and make more efficient use of resources, Samsung Electronics is working to focus on a circular economy. Going beyond the normal practice of single-use resources and throwing them away, the Samsung Corporation is working to ensure that resources are reused through recovery, reuse, and disposal at the end of the product's life. By minimizing the type of materials used and optimizing the assembly method, the company has developed production methods that minimize the use of resources [35]. By collecting products that have expired, they acquire valuable materials. Through this circular economy, the Samsung Corporation is reducing the amount of natural resources it requires, reducing greenhouse gas and pollutant emissions from waste incineration, and preventing soil and groundwater pollution from landfills.

Samsung responsibly has to collect and recycle old, unwanted or nonworking electronic products in the U.S.; therefore, the company has to develop new and adaptive programs to manage the recycling needs of its business partners. This is not always easy to manage—but the reasons behind it are twofold:


The best way to conserve resources is to make quality, durable products. The Samsung Corporation extends the life of products by making an additional contribution to the circular economy and resources by increasing the longevity of its products prior to release, carrying out a series of rigorous reliability tests, and providing convenient repair services through their global service centers, including ongoing software updates.

The amount of waste generated during the product development and manufacturing processes is significant. For complex electronic devices with numerous components, even the packaging is thrown away for every part. At the end of the project, many of the pilot products, which are used to improve the product, are lost. If all of them are burned or buried, the environment will be polluted and resources depleted. By processing waste through an environmentally responsible recycling company, the Samsung Corporation has increased the amount of recyclable materials it utilizes, and achieved its overall waste recycling goal of 95% in 2016—four years earlier than was planned (Figure 2).

**Figure 2.** Waste treatment process in Samsung Electronics [36].

Since 2016, Samsung Electronics has been continuously implementing the Galaxy Upcycling Project, which turns old Galaxy smartphones that are no longer in use into new IoT devices. Using such IoT devices, for example, pet feeders and doorbells, etc., the company was able to improve the quality of life of its consumers and, at the same time, protect the environment.

In 2018, the company completed a project to develop a low-level ophthalmoscope using Galaxy Upcycling technology as an appropriate technology [36]. In collaboration with the Yonsei University Health System Design Specification—which Samsung Electronics supports as part of Tomorrow's Decision Program (based on competition events)—ophthalmoscopes have been developed that can be used in developing countries where people cannot receive proper medical treatment because of the difficulty in distributing expensive diagnostic devices. As a result, many people in developing countries are expected to develop blindness. In addition, the company has a plan to expand the use of technology for cervical diagnostic devices—among other things—and promote better health in developing countries. The Galaxy Upcycling plans offer a variety of concepts that go beyond the health sector to reduce resource waste in collaboration with institutions that seek to achieve sustainable development with limited resources.

So, the Samsung Corporation has put forward many proposals to invest in the Korean system of waste management. However, these proposals should have business planning directions and risk-management support. Even if Samsung Engineering offers a full range of EPC services for a variety of industries, from hydrocarbon offshore facilities to wastewater treatment, from initial financing through operation and management, many consulting companies will try to develop an effective financial and risk-preventive model for it. EPC stands for "Engineering, Procurement, Construction", and is a prominent form of contracting agreement in the construction industry. In this case, we want to present the waste treatment investment project's evaluation approach.

#### **3. Methods**

All national and foreign enterprises in Korea, to one degree or another, are engaged in investment activities in waste processing; in addition, the decision-making associated with such investment in waste processing involves various complex factors, including limited financial resources, the type of investment, and the possible losses that an enterprise may suffer if the project subsequently turns out to

(i)

(ii) (iii)

(iv)

be less profitable or if it fails completely for unforeseen circumstances [37–40]. Thus, risk management allows us to confirm the viability of decisions for the project and reduce the likelihood of adopting an ineffective or unprofitable project.

In light of the discussions held so far, we consider the following research questions (RQ) in our study:

RQ1: What is the relevance of investment projects in South Korea for reducing the environmental impact of industrial production?

RQ2: What are the positive effects and shortcuts of the application of multi-criteria approaches for the evaluation of waste processing investment projects with uncertainty consideration?

RQ3: What are the risks and disadvantages of multi-criteria approaches for the assessment of waste treatment investment projects?

The remainder of this paper is organized as follows: The current trend of waste management in South Korea is discussed in Introduction Section 1. Environmental policies and waste management methods are discussed in Section 2. Section 3 describes the method, within which we consider the analysis of the feasibility and management an optimal investment project in a risk environment based on the Pareto model. Section 4 presents the results and discussion, which present various possible calculations within the framework of the discussed model. These calculations prove various possible situations for applying this method, as well as the possibility of using other performance indicators with our method. In addition, various approbations of this method in future research projects are discussed. In conclusion, Section 5 presents the main results of testing the methodology for evaluating investment projects in the field of waste management, describes our findings and predicts possible future discussions in this area.

In this research, we continuously develop this approach and take into account the risk of multivariate estimation. Vedernikov and Mogilenko in 2011 [41] suggest that uncertainty has a number of factors that affect the results of actions. Actions in this case cannot be clearly defined, and questions about how to determine the degree of possible influence of these factors on the results are accumulated. Therefore, when determining methods for industrial waste management, and methods of investing in the development of processing technologies are considered, it is important to identify the possible risks and damages from the effects of poorly predicted external factors in detail. In doing this, it is possible to carry out a scenario analysis of market risks, as well as assess the possible effects of new approaches in industrial waste management systems. This approach allows us to take the assessment of the aggregated scenario of factors into account, which enables us to represent various types of risks as components of the analysis [42–45]. This is shown in Figure 3.

**Figure 3.** Research methodology.

In 2013–2017, Rodionova et al. began to analyze an integrated approach to making investment decisions, including the calculation of NPV, DPP and IRR for each alternative [42–44]. In addition, this approach is specific, since it takes the uncertainty of the external environment into account [45–48]. For this, expert assessments of the probability of damage from the implementation of the project and the intervals of fluctuations of the three criteria are used to assess the effectiveness of the investment project.

IRR measures the effectiveness of capital investments; thus, this indicator partially allows a comparison between investment projects with different capital investments and terms of implementation. The typical methodological recommendations for calculating the effectiveness of investment projects solve the problem of selecting from alternative projects by using the NPV indicator for risk evaluation. This method is useful for certain cases, such as efficiency comparison within existing external circumstances. This recommendation helps avoid conflict of interests in terms of which indicators to use. We suggest to include them all, as they each reflect different aspects (e.g., uncertainty, market situation, project capacity, etc.) of the economic system. Each of these aspects is important for the formation of criteria in the economic system.

There are four primary reasons that justify the use of multicriterial (MCA) methods; these are listed as follows:


Because Cost Benefit Analysis (CBA) is dependent on the time at which it is being performed, it is more appropriate as an ex-ante instrument; in contrast, the multi-criteria approach can be adopted for both ex-ante and ex-post assessments, which is an advantage of the MCA. Considering the dimensions of the project or the policy to be evaluated, the characteristics (evaluative standpoint, decision-relevance, comparability, verifiability, accountability, and scientific progression) of CBA and MCA render the dimensions of the project useful. In particular, on a large scale—i.e., when public and private costs are consistent—the CBA approach is necessary, whereas MCA appears to be useful on a small-scale, where all the stakeholders can be considered individually and can be consulted or express informed opinions on their priorities.

Based on the discussion thus far, it is necessary to use methods for the evaluation of the effectiveness of alternative investment projects that are based on multi-criteria selection. However, the known methods for multi-criteria selection are still not considered in the commonly used methods that can solve the problem of selecting the optimal investment solution (Roy, 1976). In particular, the selection of an effective investment project involves the best combination of values based on the analysis of disparate indicators characterizing the investment project.

We assess the variety of values for all components of the model, taking into account the risk associated with alternative waste treatment investment projects. Intervals are determined by both the absolute values of indicators and by estimates [49–52].

To evaluate the effectiveness of alternatives and to select the most preferable, our method is based on the built-in interval preference ratio (IPR).

Let us suggest that *I* = {*I*α, α = 1 . . . *n*} is a pool of types of waste treatment investment projects; *Ki* (*I*α) = [*A<sup>i</sup>* (*I*α); *B<sup>i</sup>* (*I*α)] represents the standards for assessing the effectiveness of every investment project within the interval type; *i* = 1 . . . *r*, *r* is the total range of evaluation criteria; *A<sup>i</sup>* (*I*α) and *B<sup>i</sup>* (*I*α) are the area units of the lower and higher bounds of the interval analysis; *K* (*I*α) = {*K*<sup>1</sup> (*I*α), *K*2(*I*α), . . . *K<sup>r</sup>* (*I*α)} = {[*A*1(*I*α); *B*1(*I*α)], and [*A*2(*I*α); *B*2(*I*α)], . . . [*A<sup>r</sup>* (*I*α); *B<sup>r</sup>* (*I*α)]} are the direction indicators of every

waste treatment investment project's effective results. We tend to introduce the notation *II* for the set of Pareto-optimal information processing *IP* (*II*⊂*I*), with the amount of parts γ ≤ *n* meeting the main condition *IIm*<sup>1</sup> > *IIm*<sup>2</sup> > . . . *IImy*, *m<sup>j</sup>* = 1 . . . *y*. Then, the matter may be developed as follows to construct the economic expert tuple of thought of the variants of waste treatment investment flows, the parts of which satisfy one in all the conditions: *K<sup>i</sup>* (*Iyj*) = *min*[*K<sup>i</sup>* (*I*α)], *Iyj* ∈ *II* or *K<sup>i</sup>* (*Iyj*) = *max*[*K<sup>i</sup>* (*I*α)], *Iyj* ∈ *II*.

We also consider that if the exponent is a scalar amount, it may be mentioned as a degenerate interval with coincident ends *A<sup>i</sup>* (*I*α) = *B<sup>i</sup>* (*I*α). This concept was presented by Orlovsky in 1981, Serguieva and Olson in 2014, and Stoyanova in 2006 [53–56].

There is an ambiguity in the choice of standards and the form of factors that the unit considers, due to the quality of the question of assessing the effectiveness of waste treatment investments. It must be assumed that the person responsible for making decisions (usually the project manager) does not have a transparent opinion about the preferences of the analyzed variants. Within the scope of the indicators of victimization, the values of the intervals and the qualitative difference of the measured values—which are expressed in the fact that the difference in the units of assessment—are built, it is convenient to check the options supported by the IPR. This concept was presented by Minakova L. V. and Anikanov P. V. in 2013 [57].

Let *m<sup>i</sup>* be the breadth of the estimates' interval for the *i*-th criterion. Consistent with fuzzy methods—which were discussed by Parrino et al. in 2014, Roy in 1976, and Saaty in 1990—the interval relation of preference *R* on the set *I*<sup>α</sup> is the set of the Cartesian product *I<sup>k</sup>* × *I<sup>l</sup>* , (*k* = 1, . . . *n*, *l* = 1,.*..n*, *k* , *l*) [58–60]. For the characteristic of the set of the Cartesian product, we should take the interval membership operation µ*K<sup>i</sup>* (*Ik* ,*Il* ): *I<sup>k</sup>* × *Il*→[−1;1] into account.

$$
\mu^i \mathcal{K}\_i(I\_{k'} I\_l) = m\_i^{-1} (\mathcal{K}\_i(I\_k) - \mathcal{K}\_i(I\_l)) \tag{1}
$$

Each valuable measure of the membership function µ*K<sup>i</sup>* (*Ik* ,*Il* ) estimates the degree of injury and gain in recognizing position of *I<sup>k</sup>* as the dominant variant *I<sup>l</sup>* supported by the criteria *K<sup>i</sup> .*

The degree of privilege of the choice *I<sup>k</sup>* over the choice *I<sup>l</sup>* , supported by the interval criterion *Ki* , is diagrammatically presented by the membership function µ*DK<sup>i</sup>* (*Ik* ,*PI* ), which determines the quantitative relation of the strict interval preference.

$$
\mu\_{\rm D} \mu\_{\rm I}(\mathbf{I}\_{k}, \mathbf{I}\_{l}) = \mu\_{\rm u} \mathcal{K}\_{\rm i}(\mathbf{I}\_{k}, \mathbf{I}\_{l}) - \mu\_{\rm u} \mathcal{K}\_{\rm i}(\mathbf{I}\_{l}, \mathbf{I}\_{k}) \tag{2}
$$

For comparison, it is vital to establish that the alternative *I<sup>k</sup>* is not undermined compared with the *I<sup>l</sup>* alternative, which would be a mistreatment of the membership operation.

$$
\mu\_{\rm ND} \mathbb{K}\_{\rm i}(I\_{\mathbf{k}\prime} I\_{\mathbf{l}}) = 1 - \mathbf{x}\_{\prime} \mathbf{x} \ge \mathbf{0}; \mathbf{x} = \mu\_{\rm D}^{\mu}(\mathbb{K}\_{\mathbf{i}}(I\_{\mathbf{K}\prime} I\_{\mathbf{l}}) \tag{3}
$$

In this case, for the criterion of the *i*-th interval criteria, the approximation of the alternative *I<sup>k</sup>* to the Pareto optimal variant is determined by the index of the membership function for the set of non-privileged alternatives [61,62].

$$
\mu\_{\rm D}^\* \mathbb{K}\_i(I\_k) = \min \mu\_{\rm ND} \mathbb{K}\_i(I\_{k'} I\_l) \tag{4}
$$

Wang et al. in 2009, and Zare et al. in 2016 [63,64] suggested that the indicator NPV is based on the quantity of cash flows at a certain time and the discount rate *r*:

$$NPV = \mathbb{C}\_1(1+r)^{-t1} + \dots + \mathbb{C}\_n(1+r)^{-tn} \tag{5}$$

The discount rate usually uses the risk-free interest rate or interest rate for investment projects with a similar degree of risk, as well as the market and industry coefficient of efficiency for capital investments. This criterion underlies the choice of an environmental management project with a maximum value, or with the same value of *r*. It is known that NPV is entirely dependent on the discount rate; therefore, a poor-quality and unverified forecast of the discount rate definitely leads to risky management decisions. For example, a good project with high-quality technologies, but with high costs, can be rejected, and a project with lower costs but low-quality technologies can be accepted for discussion and subsequent implementation. The refinement of the values of the NPV interval allows us to determine that the maximum possible determining factor for the NPV criterion is the maximum value.

In addition, DPP is represented as a time interval; the optimal condition for this criterion should correspond to its minimum value. Furthermore, IRR is presented as a percentage and is defined as the value of the interval; in accordance with this criterion, the waste treatment investment project that matches the maximum value is selected.

#### **4. Results**

Risk assessment is predicated on the interval values in the estimates. The presumption is that the rate of interest r may be a variable, and for that the likelihood of a random event may be found, *NPV* (*r*, *t*) > 0, *P* (*NPV* (*r*, *t*) > 0) = *P* (*r* < *IRR*) = *F* (*IRR*). Here, *F* (*x*) = *P* (*r* < *x*) is the distribution operation of *r*; IRR is the internal rate of pay back, that is obtained as an answer of the equation *NPV* (*t*, *r*) = 0. For various *r*, it is possible to ascertain the likelihood that the waste treatment project will not pay off at time *t*; then the estimates of victimization are obtained in the analysis procedure. Here, we tend to conduct a risk assessment for the project supported by the explained methodology for three doable and inevitable market conditions; those conditions are then evaluated by consultants, and an evaluation of the probability of every of them is enforced. It ought to be noted that the chance assessment criterion for a waste processing investment project needs to select the most effective possibility, supported by the minimum worth of the standards.

Given the well-known theoretical ideas, the values of *m<sup>i</sup>* are selected as the most allowable values for the considered option (standard). The initial knowledge necessary for the calculations and investment analysis are presented in Table 3. Three different investment projects are presented (*I*1—investment in landfill production; *I*2—investment in industrial waste recycling R&D; *I*3—investment in the implementation of green technologies).


**Table 3.** Data implication for different projects.

The risk evaluation process is presented using interval values in grade system. Assuming that the rate of interest *r* could be a variable quantity for which the chance of a random event is found, *NPV* (*r*, *t*) > 0, *P* (*NPV* (*r*, *t*) > 0) = *P* (*r* < *IRR*) = *F* (*IRR*). Here, *F* (*x*) = *P* (*r* < *x*) is the disseminative operation of *r*, *IRR* is the internal rate of pay back, which is suggested as an explanation of the formula *NPV* (*t*, *r*) = 0. For variety of *r*, it is important to determine the likelihood that the investment project will not be profitable at time *t*. Then the results will be determined through the evaluation analysis procedure. This study assesses the risk of an industrial waste management project using the aforementioned methodology for three defined and predicted market conditions, and an expert assessment of the likelihood of each of market condition is introduced. It is important to indicate that

the risk assessment criterion for a waste processing investment project requires the selection of the best option based on the minimum value of the criteria.

Based on known theoretical models, *m<sup>i</sup>* values are defined as the maximum allowable indicators for the criteria under consideration. The initial data necessary for calculations on the analysis of investment projects are presented in Table 3.

Taking in consideration Equation (1), we achieve the appraisal of the membership operation µ*K<sup>i</sup>* (*Ik* ,*Il* ) for each pair of variants for each criterion and calculate their estimated matrices. Thus, Equation (1) can be expanded as:

$$\mu^{\mu}K\_i(I\_k I\_l) = (\left[\min\{A\_i(I\_k) - A\_i(I\_l); B\_i(I\_k) - B\_i(I\_l)\};\right]$$

$$\max \left\{ A\_i(I\_k) - A\_i(I\_l); B\_i(I\_k) - B\_i(I\_l) \right\} \big| m\_i$$

and be denoted by

$$C\_i^{kl} = \min\{A\_i(I\_k) - A\_i(I\_l); B\_i(I\_k) - B\_i(I\_l)\} \mu\_{i \times l}$$

$$D\_i^{kl} = \max\{A\_i(I\_k) - A\_i(I\_l); B\_i(I\_k) - B\_i(I\_l)\} \mu\_i$$

Then,

$$
\mu^i \mathcal{K}\_i(I\_{k'} I\_l) = [\mathbb{C}\_i^{kl}; \mathcal{D}^{kl}\_{\ i}] \tag{6}
$$

Furthermore, the interval membership function for the *I<sup>l</sup>* , *I<sup>k</sup>* takes the following form:

$$
\mu^{\mu} \mathcal{K}\_i(I\_{k\prime} I\_l) = [-\mathcal{D}\_i^{kl}; -\mathcal{C}\_i^{kl}] \tag{7}
$$

Hence, if the relation |*C kl i* | = *Dkl i* is true, then the values µ*K<sup>i</sup>* (*Il* ,*Ik* ) µ*K<sup>i</sup>* (*Ik* ,*Il* ) coincide.

Based on Equation (2), we take into account the preference frequency for each pair of options for each indicator using the value of the membership operation µ*DK<sup>i</sup>* (*Ik* ,*Il* ), and place them in the evaluation matrices. Using Equations (6) and (7), we move the calculations into a simple method.

Thus, we evidently have

$$M\_D{}^{\mu}K\_i \text{ (}I\_{k\prime}I\_l\text{)}=[\mathbf{C}\_i^{kl}; D\_i{}^{kl}] - [-D\_i{}^{kl}; -\mathbf{C}\_i{}^{kl}] = [\mathbf{C}\_i{}^{kl} + D\_i{}^{kl}; \mathbf{C}\_i{}^{kl} + D\_i{}^{kl}]$$


*μ* From Equations (3) and (4), we achieve the valuable measures of the membership function µ*NDK<sup>i</sup>* (*Ik* ,*Il* ) for each pair of options for each criterion, and assemble the membership function valuable measures for the set of non-privilege options µ*DK<sup>i</sup>* (*Ik* ):

$$
\mu\_D \stackrel{\*}{\smile} K\_1(I\_k) = \langle 0.6, 1, 0.95 \rangle;$$

$$
\mu\_D \stackrel{\*}{\smile} K\_2(I\_k) = \langle 0.9, 1, 0.6 \rangle;$$

*α*

*μ α α*

*μ* <sup>−</sup>

≠ ≠

≠ ≠

*μ μ ≠*

≠ ≠ 0 1/

µ

*μ <sup>и</sup>*

*μ μ μ μ μ μ*

*μ*

µ

µ

 µ µ*<sup>D</sup> \*K*3(*I<sup>k</sup>* ) = {1, 0.9, 0.93}; µ*<sup>D</sup> \*K*4(*I<sup>k</sup>* ) = {1, 0.8, 1}; µ*<sup>D</sup> \*K*5(*I<sup>k</sup>* ) = {0.9, 0.75, 1}; µ*<sup>D</sup> \*K*6(*I<sup>k</sup>* ) = {0.95, 0.75, 1}.

After analyzing the values of µ*<sup>D</sup> \*Ki* (*Ik* ), we can conclude that the investment project *I*<sup>2</sup> is the best option based on the criteria *K*1(*I*α) and *K*2(*I*α), the investment project *I*<sup>1</sup> is the best option based on the criteria *K*3(*I*α) while it is possible to perform risk management in a pessimistic scenario of market development, and investment project *I*<sup>3</sup> is optimal in terms of risk, based on the considered set of options for waste processing investment projects [65–67].

Savchuk in 2007, and Syroezhin in 1980 [68,69] suggested that in order to determine the best preference in this set of waste management investment projects, it is necessary to determine the vector preference using some previous studies in this area (membership functions µ*<sup>D</sup> \*Ki* (*Ik* ) determine the degree of proximity of variant *I<sup>k</sup>* to the Pareto-optimal variant of an investment project using the *K<sup>i</sup>* criterion; this justifies the use of special criteria instead of traditional factors, indicating the importance of the indicator. The next step is to compare the variants *I<sup>k</sup>* and *I<sup>l</sup>* in pairs, calculate the values of µ*<sup>D</sup> \*Ki* (*Ik* ), and introduce the subsets *Ikl*+, *Ikl*−, and *Ikl*<sup>=</sup> for optimal, pessimistic and realistic values of µ*<sup>D</sup> \*Ki* (*Ik* ) and µ*<sup>D</sup> \*Ki* (*Ik* ), (where *i* = 1...4; *k*, *l* = 1,... 3, *k* , *l*) of these variants, respectively. The next step determines the elements of the estimation matrix **C** = k*C* µ *kl*<sup>k</sup> based on these conditions; this is shown in Table 4.

$$\mathbf{C}\_{kl}^{\mu} = (\sum\_{i=1}^{3} \mu\_{D}^{\*} \mathbf{K}\_{i}(\mathbf{I}\_{k})) \left(\sum\_{i=1}^{3} \mu\_{D}^{\*} \mathbf{K}\_{i}(\mathbf{I}\_{l})\right)^{-1} \tag{8}$$


**Table 4.** Evaluation matrix.

The matrix is created considering the risk criteria; therefore, it is necessary to pay attention to the possibility of various risk conditions being weighted evaluated options of the matrix component.

$$\begin{aligned} \mathbf{C}\_{kl}{}^{\mu} &= (\boldsymbol{\Sigma}^\* \boldsymbol{\Sigma}\_{\boldsymbol{i}=1} \boldsymbol{a}\_{\boldsymbol{i}} \boldsymbol{\mu}\_D \mathbf{\boldsymbol{\zeta}}^\star k\_{\boldsymbol{i}}(\boldsymbol{I}\_k)) (\boldsymbol{\Sigma}^\* \boldsymbol{\epsilon}\_{\boldsymbol{i}=1} \boldsymbol{a}\_{\boldsymbol{i}} \boldsymbol{\mu}\_D \mathbf{\boldsymbol{\zeta}}^\star k\_{\boldsymbol{i}}(\boldsymbol{I}\_l))^{-1} \boldsymbol{\lambda} \\ \left\{ \begin{array}{ll} \boldsymbol{a}\_{\boldsymbol{i}} = \mathbf{1}, \ \boldsymbol{i} = \mathbf{1}, \mathbf{2}, \mathbf{3} \\\ p\_{\boldsymbol{i}\nu} \boldsymbol{l} = \mathbf{4}, \mathbf{5}, \mathbf{6} \end{array} \right\} \end{aligned}$$

Then, we get the following matrix of preferences

$$\mathbf{C} = \begin{array}{|cc|cc|} \hline \cdot & 0.66 & 5.01 \\ \hline 1.51 & \cdot & 0.94 \\ 0.19 & 1.05 & \cdot \\ \hline \end{array}$$

Using well-known theoretical methods and developing a methodology for evaluating investment projects, we introduce the indicators *G* µ *<sup>l</sup>* and *H*<sup>µ</sup> *l* , which denote the set of elements of the *l*-th column in C, the value of which is less than one, but greater than zero and more than one, respectively, and the exponent *C* µ *kl* max, which is equal to the indicator of the maximum value of the *l*-th column. It can be argued that *H*<sup>µ</sup> *<sup>l</sup>* represents the number of investment project options that dominate the *l*-th column. Furthermore, *G* µ *<sup>l</sup>* shows the number of investment project options that dominate the *l*-th column,

and *C* µ *kl* max represents the maximum level of dominance of the *k*-th version of the investment project over the *l*-th column.

We include these indicators in the matrix, as shown in Table 5.


**Table 5.** Matrix of variations.

Source: made by authors.

According to the results of Table 5, investment project *I*<sup>2</sup> turned out to be the best variable alternative with a minimum value of *C* µ *kl* max. In this regard, the second version of investment design should be included in the Pareto tuple, but is excluded from the subsequent analysis. For this exception procedure, we delete the corresponding row and column in the preference matrix.

At the next stage, we analyze the other (above mentioned) options for investment design, and analyze them using the new matrix of indicators in a similar way.

As a result, the Pareto preference tuple can be denoted as *II* = {*I*2, *I*1, *I*3}. In this regard, the best alternative for the vector of the heterogeneous performance index is *K* (*I*α) = {*K*1 (*I*α), *K2* (*I*α), *K*3 (*I*α), *K*4 (*I*α), *K*5 (*I*α), *K*6 (*I*α)}. In the Pareto tuple of the three options considered, those criteria that characterize NPV—discounting for calculating DPP in the vector efficiency index—became preferable.

Based on Table 5, the best alternative to an investment project with a minimum value max is option *I*2. Therefore, the second version of the investment project is included in the Pareto tuple and excluded from further analyses by deleting the corresponding row and the column in the preference matrix.

The remaining options are analyzed using the new matrix of indicators in a similar manner.

Finally, the tuple of Pareto preferences can be obtained as *II* = {*I*2, *I*1, *I*3}. Therefore, the best alternative for the vector inhomogeneous efficiency index *K*(*I*α) = {*K*1(*I*α), *K*2(*I*α), *K*3(*I*α), *K*<sup>4</sup> (*I*α), *K*5(*I*α), *K*6(*I*α)} should be recognized as the second variant. In the Pareto tuple of the considered variants, preference was expressed for the criteria characterizing the NPV and discounting the calculation of the DPP in the vector efficiency index.

So, to answer the research questions which were submitted in Section 3, we consider these research points and describe them above to show the significance of this research:


#### **5. Discussion**

Since Korea's volume of waste is evidence of changing lifestyles in the midst of a trend toward a convenience-oriented life (single-use products, convenient goods, instant food, etc.) and an abundant capitalist socioeconomic environment (mass consumption/mass production), the current state of waste management in Korea is at a turning point, where a paradigm shift from a convenience-oriented society (single-use product society) to a society oriented toward resource conservation (resource-circulating society) is taking place [71,72]. Wastes have a close relationship with each country's life and cultural patterns, as well as with changes in society, patterns of waste generation and treatment, thereof, change. A summary of the evolution of Korea's system of legislations for waste management shows that this evolution has been taking place alongside the flow of developmental processes in Korea, and each developing country needs to introduce waste policies suitable for its current economic and social conditions. These policies should be supported with investment perspectives and based upon the development of technology and innovation [73–75].

According to the well-developed support policies of waste treatment investment projects and new environmental regulations, the Korean Government expects to observe some effects in terms of the economic, environmental and social impacts of the Framework Act on Resource Circulation (Table 6).


**Table 6.** Expected results from investment activities in Korean waste management systems.

Source: adopted by authors from Joint work of related departments and agencies, September 2011, the 1st Framework Plan for Resource Circulation (2011–2015) [76].

Waste treatment investment projects are needed to be classified according to the type of waste treatment and operational technology. Each investment project requires risk analysis, which should be evaluated. For further research, it is necessary to identify the correlation between risk type and used technology in waste treatment operations.

Risk-evaluation procedures are the importing starting points for waste management. A welldesigned risk evaluation system presents a structured mechanism for searching for potential problems and creating judgements on the consequences. Assessing the risks of waste processing investment underpins the "suitable for use" approach adopted by the Korean regulatory mechanism and supports planning policy [77]. The main goal is to decide whether there are any unacceptable risks to people or the wider environment—including industries. The risk evaluation process can be very detailed, particularly where risks are diversified. For the discussion of investment in waste recycling, there are a range of specific technical approaches for different contaminants and circumstances.

However, these all broadly fit within a general process that can be seen as a tiered approach. Each tier is applied to the case circumstances, with an increasing level of detail information required by the assessor in progressing through each tier [78].

There are three tiers, or steps:


During the management of investment projects in the field of industrial waste management, it is important to determine the possible influence of various external factors on the result (market factors, production factors, technological factors, social factors, etc.). In this case, the degree of occurrence of a particular risk is determined and an initial assessment of risk is carried out. Depending on the likelihood of a negative event in the external environment of the investment project and the degree of negative influence of factors on its successful implementation, managers can use only one approach to assess risks, or conduct a multi-stage risk analysis. Usually it depends on the properties (complexity and degree of influence) of pollutants—some of them can be evaluated using common criteria; others may need to develop specific indicators for a more detailed and comprehensive risk assessment.

#### **6. Conclusions**

Our proposed investment project selection algorithm discusses the associated risks and allows us to determine the degree of their influence on the result of improving the environmental safety systems of production facilities. Do not forget that this examines the various conditions and components of the economic and environmental systems, as well as identifies all possible risks in each of them. This is achieved by describing risk situations and introducing a multi-component presentation of the risk component as one of the decision criteria. In subsequent studies, it will be necessary to identify a correlation effect between environmental factors and elements of the ecological and economic systems.

The investment project appraisal approach proposed in this study opens up new possibilities for applying the multi-criteria selection method to the conditions of economic and environmental activities in the field of waste management.

For the successful selection and implementation of an investment project in the field of waste management, it is extremely important to carry out the following management activities:


This paper is a combination of three elements: (i) presentation of the South Korean policy for waste, (ii) case study of the Samsung Corporation waste management system, and (iii) a multidimensional approach to risk-analysis in investment projects. Let us summarize the correlation between these three parts:


Thus, our proposed method allows us to use multidimensional and specific information for the process of making comprehensive economic and environmental (managerial) decisions in a market system. Moreover, our algorithm can be used to make long-term strategic decisions in the field of investment risk management in environmental and economic systems.

**Author Contributions:** J.H.L. and O.A.S. conceptualized the study; J.H.L. designed the methodology; O.A.S. carried out the investigation; J.H.L. and O.A.S. did the analyses, validation, and data curation; O.A.S. wrote

the paper, and J.H.L. contributed to reviewing and editing all sections; J.H.L. supervised the work. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was done as a part of Research project 2020-0108 and was funded in 2020 by Korea University of Technology and Education (KOREATECH).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Applied Sciences* Editorial Office E-mail: applsci@mdpi.com www.mdpi.com/journal/applsci

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com ISBN 978-3-0365-1811-4