Analysis and Optimization of Coagulation Efficiency for Brackish Water Reverse Osmosis Brine Based on Ensemble Approach

Abstract

Reuse of wastewater through brackish water reverse osmosis presents a major challenge due to the generation of brine, which contains organic and inorganic compounds to be removed. This study focuses on analyzing and optimizing coagulation conditions for brackish reverse osmosis brine treatment by evaluating pollutant removal efficiencies under various scenarios and leveraging advanced modeling techniques. Jar tests were performed using polyaluminum chloride and ferric chloride, evaluating the removal of total organic carbon, turbidity, UV₅₂₄, and phosphorus. Models were developed using response surface methodology, support vector machines, and random forest. Although the same data sets were used, the characteristics of these models were found to be different: Response surface methodology delivered high-fidelity, smooth response surfaces (R² > 0.92), support vector machine pinpointed sharp threshold regions, and random forest defined robust operating plateaus. By overlaying model-specific optimum contours, the consensus regions were identified for reliable removal across total organic carbon, turbidity, and phosphate. This ensemble strategy enhanced predictive reliability and provided a comprehensive decision-support tool for multi-objective optimization. The findings underscore the potential of ensemble-based modeling to improve the design and control of brackish reverse osmosis brine treatment processes, offering a data-driven pathway for addressing one of the most critical bottlenecks in wastewater reuse systems.

1. Introduction

Water supply is challenged by limited water resources, increasing water demand, and climate change. The world’s potential renewable water resource accounts is 52,580 km³/year [1], which has remained almost constant over time. On the other hand, the water demand has increased significantly over time due to population growth and industrial development [2,3]. In addition, climate change has increased the variability of regional precipitation and the availability of water resources, and is expected to exacerbate this situation [4]. These facts suggest that there is an urgent need for the development of alternative water resources that can bridge the gap between the demand for water and the conventional supply of water. Among the available water resources, water reuse is identified as a primary strategy to ensure sufficient water in regions with water scarcity [5,6]. Unlike conventional water sources, water reuse provides a stable water supply that is hydrologically independent of the source [7]. Water reuse is being increasingly implemented around the world [8,9]. There are several different types of water reuse, including non-potable recreational, environmental, and residential uses; de facto water reuse; industrial reuse; indirect potable reuse (IPR); and direct potable reuse (DPR) [10,11]. Depending on the purpose of water reuse, different treatment options should be applied [12]. For example, non-potable recreational uses require relatively simple treatment steps, while either IPR or DPR require multi-barrier approaches to meet stringent requirements [10,13].

When treating municipal or industrial wastewater to high quality water, reverse osmosis (RO) plays an important role [14]. RO is used to remove ions and organic compounds from feed water, which may be secondary or tertiary wastewater, for either industrial or potable reuse [12,15]. RO is the most mature and widely implemented membrane technology, typically utilizing spiral wound modules (SWM) with feed spacers, which are important components designed to maintain channel thickness [16]. The advantages of RO for water reuse applications include: (1) ability to produce high quality product water; (2) small footprint; (3) modularity; (4) operational reliability; and (5) relatively low energy consumption compared to competing technologies such as multi-effect distillation (MED) or mechanical vapor compression (MVC) [14,17,18]. However, one of the critical challenges facing RO is brine management. Since RO brine contains four to seven times the amount of rejected pollutants, this can lead to potential environmental and human health problems [11,18]. RO brine from a water reuse facility is typically discharged to surface water, injected into deep wells, or sent to evaporation ponds [19,20]. In the first case, the impact on the environment is a major concern, and in the second and third cases, the cost of treatment is very high [14,21]. Therefore, it is necessary to find an affordable way to treat RO brine for a sustainable reuse of wastewater.

Considerable research has been devoted to RO brine treatment for water reuse applications [22,23,24]. Much of this work has focused on reducing the organic pollutant load in brine through destructive processes [25]. For instance, various advanced oxidation processes (AOPs), such as ozonation, Fenton oxidation, photocatalysis, sonolysis, and electrochemical oxidation, have been studied (primarily at laboratory scale) to break down organic contaminants in RO brine [26,27,28]. These AOPs can effectively degrade dissolved organic carbon and certain micropollutants, but they often come with high reagent or energy costs and may produce by-products that require further treatment [29,30]. Another approach to manage brine is further concentration using pressure-driven membranes (e.g., secondary RO or nanofiltration) to recover more water and minimize volume [31,32,33]. However, such processes face practical limitations due to the high osmotic pressure of RO brine, which drastically lowers water flux and increases membrane fouling and energy usage at extreme salinities [34]. Emerging membrane technologies like forward osmosis or membrane distillation have shown promise for high-salinity brine concentration [35,36], they must also address scaling and fouling issues at the pilot scale [37,38]. To date, no solution has been identified as an effective and comprehensive solution to address all facets of brine treatment. Accordingly, a combination of methods may be required to tackle different components of the RO brine.

Physical-chemical treatment is a conventional method and can be easily combined with other technologies for RO brine treatment [18]. In particular, coagulation can be applied to RO brine to target colloidal particles, organic matters, and phosphate ions that contribute to problems of treated water quality [39]. Recent studies suggest that coagulation can substantially reduce pollutants in RO brine under certain conditions [38]. The type of coagulant is a critical factor influencing treatment efficiency, not only for RO brine but also for other challenging wastewater streams. For instance, a study on coal mine wastewater demonstrated that treatment efficiency in an ultrafiltration–reverse osmosis process varied significantly with different coagulant types [40]. Similarly, ferric chloride (FeCl₃) was found to remove a broad range of dissolved organic fractions from a RO brine, achieving roughly double the TOC and UV-254 removal efficiency of an aluminum-based coagulant at equal molar doses [41]. Similarly, a study on inline coagulation–ultrafiltration pretreatment for RO brine recovery found that FeCl₃ outperformed polyaluminum chloride (PACl) and other aluminum coagulants in removing dissolved organic carbon (DOC) and mitigating fouling potential [39]. By lowering organic content, turbidity, and even nutrients (via co-precipitation of phosphorus, for instance), coagulation could make the concentrate less hazardous for discharge or more amenable to high-recovery processes [42,43]. However, the efficiency of coagulation can depend on multiple operational factors, including coagulant dose, solution pH, temperature, and the characteristics of the brine, which need to be carefully optimized for each situation.

To systematically evaluate and optimize coagulation conditions for RO brine treatment, it is important to apply robust experimental design and modeling techniques. Response Surface Methodology (RSM) is one such statistical approach that has been widely used in environmental engineering to model and optimize treatment processes involving several variables [44,45]. RSM employs designed experiments and quadratic polynomial modeling to explore the relationships between factors and performance outcomes [46]. This methodology can efficiently identify interaction effects and optimal operating conditions with relatively few experiments [47]. In coagulation processes, RSM has been successfully applied to determine optimal coagulant dosages, pH, and other conditions to maximize removal of turbidity and organic matter. For instance, a drinking water coagulation process was optimized using RSM, revealing the combined influence of coagulant dose, water temperature, and pH on removal of turbidity, TOC, and total nitrogen [48].

In recent years, data-driven modeling and machine learning (ML) techniques have also been introduced in water treatment optimization, offering powerful predictive capabilities beyond traditional empirical models [49]. Algorithms such as support vector machines (SVM) and random forest (RF) have been employed to model complex nonlinear relationships in treatment processes, including coagulation, often with high accuracy [50,51]. Researchers have reported successful applications of SVMs and RFs in predicting various water treatment outcomes, especially when provided with sufficient training data [49,52]. Integrating such computational tools with conventional brine treatment technologies offers promising pathways to enhance the efficiency, reliability and sustainability of RO concentrate management.

One challenge, however, is that single machine learning models can be prone to issues like overfitting or limited performance, particularly when the available dataset is relatively small, as is often the case with lab or pilot-scale water treatment studies [53]. In the context of coagulation studies on RO brine, data may indeed be limited to tens of experimental runs, making it difficult for ML models to analyze the outcomes reliably. To address the limitations of individual models, an ensemble modeling approach can be adopted. Ensemble modeling involves combining the predictions of multiple models to produce a more robust overall prediction [54]. This strategy uses the strengths of different modeling techniques while reducing their weaknesses [55]. Research has shown that ensemble models can achieve higher prediction accuracy and better generalization than any single model alone [56]. By combining multiple models, ensemble models can reduce the impact of noise and overfitting, leading to more stable and reliable performance even with smaller datasets [57]. In water treatment applications, ensemble methods (e.g., bagging, boosting, or stacking of models) have been used to improve the prediction of key parameters like effluent organic levels and to enhance control strategies [58,59]. The ensemble concept is particularly attractive for multi-objective optimization.

In this study, coagulation was investigated as a treatment strategy for brackish water RO (BWRO) brine in a wastewater reuse plant. Unlike many RO studies that emphasize fouling, scaling, or energy demand at the membrane stage, this work focuses on RO brine treatment, highlighting that optimizing coagulation efficiency offers a practical means to reduce organics, nutrients, and turbidity in the concentrate, thereby mitigating environmental impacts and supporting integration with other treatment processes. Jar-test experiments with aluminum- and iron-based coagulants were conducted to evaluate removal of dissolved organic carbon, UV-absorbing substances, turbidity, and phosphate from RO brine. Coagulation performance was analyzed using response surface methodology (RSM) and machine learning models (SVM, RF), which were then integrated into an ensemble framework. This approach harnesses the complementary strengths of statistical and ML models to identify optimal conditions more reliably and offers a practical decision-support tool for tuning coagulation in brine treatment, thereby improving the design and control of wastewater reuse processes

2. Materials and Methods

2.1. Experimental Conditions

2.1.1. Feed Water

The RO brine used in the coagulation experiments was provided by a wastewater reuse plant located at Gumi in the Republic of Korea. The reuse plant reprocesses wastewater effluents, which are previously treated at a municipal wastewater treatment plant. It primarily uses the combination of UF and BWRO to remove ions and dissolved organic matters and generates BWRO brine, which is currently returned back to the wastewater treatment plant.

Table 1. Summary of water quality in BWRO brine from wastewater reuse plant (average ± standard deviation, number of measurements = 3).

Parameters	Unit	Values (Average ± SD)
TOC	mg/L	20.0 ± 1.5
Turbidity	NTU	0.2 ± 0.02
Orthophosphate	mg/L	4.2 ± 0.3
UV-254	cm−1	0.539 ± 0.02

presents the water-quality parameters of the BWRO brine. The feed exhibits a moderately high organic load (TOC = 20.0 mg/L) and an aromatic character (UV-254 = 0.539 cm⁻¹), while its particulate content is minimal (turbidity = 0.2 NTU). Nutrient levels are notable, with orthophosphate measured at 4.2 mg/L. Additionally, the total dissolved solids (TDS) and the electric conductivity (EC) of the brine were approximately 3500 mg/L and 5500 mS/cm, respectively. These compositions underscore the challenge of removing colloidal and truly dissolved contaminants and justify the need for optimized coagulation strategies.

2.1.2. Experimental Setup

The coagulation efficiency of BWRO brine from the Gumi plant was evaluated using a jar tester (C-JT-H, Changshin science, Seoul, Republic of Korea). Prior to each test, 500 mL of brine was transferred into each beaker and set to the target temperature (10, 20, 30, or 40 °C) using a temperature-controlled water bath. The pH of each sample was adjusted to the desired value using 0.1 M HCl or NaOH. Coagulant stock solutions of polyaluminum chloride (PACl, commercial grade, Junsei Chemical Co., Tokyo, Japan) and ferric chloride (FeCl₃·6H₂O, analytical grade, Sigma-Aldrich, Burlington, MA, USA) were prepared freshly before each set of experiments by dissolving the required amount of salt in deionized water. These stock solutions were then dosed over the range 0.2–0.6 mM as Al³⁺ or Fe³⁺, respectively. Each run comprised a rapid mixing phase of 150 rpm for 3 min to ensure homogeneous dispersal of the coagulant, followed by a slow mixing phase of 80 rpm for 25 min to promote floc growth. After mixing, samples were allowed to settle quiescently for 30 min, and the clarified supernatant was withdrawn for analysis. TOC was determined using a TOC analyzer (Multi N/C 3300, Analytic-Jena, Jena, Germany) based on the high-temperature catalytic combustion method with non-dispersive infrared (NDIR) detection. Turbidity was measured using a turbidity meter (2100Q, HACH, Loveland, CO, USA), UV₂₅₄ absorbance was determined with a UV–Vis spectrophotometer (Aqualog-UV-800-C, Horiba, Kyoto, Japan) at 254 nm, and orthophosphate was measured by the ascorbic acid method (USEPA PhosVer 3 method). All conditions were tested in duplicate, and mean values are reported.

2.2. Modeling Approach

Three modeling methods were applied to analyze the coagulation efficiency of the BWRO brine, including RSM, SVM, and RF. Then, the coagulation conditions were optimized through an ensemble approach of these models.

2.2.1. Design of Experiment (DOE) and RSM

To develop coagulation models via RSM, a three-factor, three-level experimental design was employed as listed in

Table 2. Design of experiments for coagulation of BWRO brine.

Run	Coagulant Dose (mM)	Temperature (°C)	pH
1	0.2	17.5	5
2	0.4	25	7
3	0.6	17.5	9
4	0.4	25	7
5	0.4	25	3.6
6	0.6	32.5	5
7	0.4	37.6	7
8	0.4	25	7
9	0.2	32.5	9
10	0.4	25	10.4
11	0.4	12.4	7
12	0.063	25	7
13	0.4	25	7
14	0.6	17.5	5
15	0.74	25	7
16	0.4	25	7
17	0.6	32.5	9
18	0.2	17.5	9
19	0.2	32.5	5
20	0.4	25	7

. This experimental design was intended to encompass a coagulant concentration range of 0.2–0.6 mM, a temperature range of 17.5~32.5 °C, and a pH range of 5~9. A total of 20 runs were conducted to analyze both linear/quadratic effects and two-way interactions. These include twelve factorial points, six axial points at the extremes of each variable, and two center-point replicates at the mid-levels (dose = 0.4 mM, T = 25 °C, pH = 7). Based on this design, RSM models were developed using Design Expert Software 12 (Stat-Ease, Minneapolis, MN, USA), resulting in robust estimation of second-order polynomial coefficients for each response (TOC, turbidity, UV-254, and phosphate removal). Then, the RSM models were used to analyze the relative significance of each factor, defined optimal coagulation conditions, and produced contour plots illustrating the synergistic effects of input variables.

2.2.2. Support Vector Machine

Support vector machine (SVM) regression was applied to predict coagulation performance. The model estimates a function of the form:

$$f(x)=w·\varphi (x)+b$$

(1)

where $\varphi \left(x\right)$ is a nonlinear mapping of the input variables (coagulant dose, temperature, and pH) into a higher-dimensional feature space, w is the weight vector, and b is the bias term [60]. The goal is to find a function that fits the data within a specified error tolerance while minimizing model complexity. A radial basis function (RBF) kernel was used in this study to capture nonlinear relationships.

In this study, support vector machine (SVM) modelling was conducted using the CAMO Unscrambler^® X software 10.3 (AspenTech, Bedford, MA, USA), leveraging its built-in classification and regression engine to predict coagulation removal efficiencies [61]. The raw dataset was pretreated by autoscaling and mean-centering within Unscrambler’s automatic preprocessing workflow and then was divided into training and validation subsets, and a radial basis function (RBF) kernel was selected to capture potential nonlinear relationships between coagulant dose, temperature, and pH and each response variable. Hyperparameters (the penalty parameter, C, and kernel width, γ) were optimized via grid search combined with k-fold cross-validation (k = 5) to prevent overfitting and ensure model generalizability. The final SVM models demonstrated robust predictive performance, with the Unscrambler software reporting support vectors and margin plots for each response, thereby allowing clear visualization of decision boundaries and model diagnostics.

2.2.3. Random Forest (RF)

Random forest (RF) regression builds an ensemble of decision trees, and the final prediction is obtained as the average of all individual tree predictions:

$$\widehat{y}={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T{h}_t(x)}$$

(2)

where $T$ is the total number of trees and $h_t\left(x\right)$ is the prediction of the $t$-th regression tree [62]. Each tree is trained on a bootstrap sample of the data, and at each split, a random subset of predictors is considered, which introduces randomness and reduces overfitting. Feature importance can be quantified by evaluating the mean decrease in impurity (MDI) or mean squared error (MSE) across trees.

RF regression was implemented in Python 3.13 (using the scikit-learn application programming interface (API)) to predict UV₂₅₄ removal as a function of three key process variables (coagulant dose, temperature, and pH) [63]. A decision-tree forest of up to 200 trees was grown, each with a maximum depth of 50 splitting levels and a minimum of two samples required to split any internal node, ensuring that even subtle nonlinear effects could be captured without excessive overfitting. Continuous predictors were allowed up to 200 distinct split categories, and any missing values encountered were imputed by the median of that variable. All observations (20 rows × 7 columns, no missing data) were used with equal weights, and surrogate splitters were disabled to streamline computation. Model performance was assessed via out-of-bag (OOB) validation, which internally estimates prediction error without a separate hold-out set. Feature-importance scores extracted from the trained forest quantified the relative contributions, guiding both interpretations.

2.2.4. Ensemble Approach

Ensemble approach is a technique that combines the outputs of two or more individual models to produce a single, more accurate prediction than any model alone. Using multiple models helps to reduce the variability that may come from relying on a single approach, a process known as variance reduction. For this, each model was first run independently across the experimental range of input variables. The predicted optimal parameter combinations from each model were then normalized onto a common scale and aggregated, giving greater weight to regions where all three models converged. To make these consensus regions easy to interpret, contour plots from each model’s predictions were overlapped. This created combined visual maps that highlight parameter zones where high coagulation efficiency is consistently predicted.

3. Results and Discussion

3.1. Results of Coagulation Experiments

Table 3. Post-coagulation water quality (residual concentrations) for PACl and FeCl₃ under varied coagulant dose, temperature, and pH conditions.

Run	PACl				FeCl3
	TOC (mg/L)	Turbidity (NTU)	Phosphate (mg/L)	UV254 (cm−1)	TOC (mg/L)	Turbidity (NTU)	Phosphate (mg/L)	UV254 (cm−1)
1	14	1.78	4.1	0.487	19	1.99	4.0	0.508
2	12.3	4.60	0.0	0.289	12	1.76	0.0	0.395
3	16	3.25	0.0	0.385	19	4.06	4.0	0.520
4	9.7	3.49	0.0	0.355	7.0	1.66	0.0	0.390
5	15	0.28	3.1	0.491	8.0	0.16	0.0	0.527
6	5.0	1.92	0.0	0.233	4.0	1.02	0.0	0.209
7	6.0	1.79	0.0	0.245	4.0	0.95	0.0	0.264
8	10	5.30	0.0	0.349	8.0	1.05	0.0	0.314
9	13	0.82	4.2	0.486	17	1.77	3.7	0.531
10	9.0	0.47	3.9	0.423	16	1.71	3.8	0.497
11	14	8.19	3.7	0.504	20	3.88	4.1	0.538
12	19	0.19	4.1	0.501	14	0.19	3.4	0.483
13	9.0	5.30	1.7	0.336	7.0	1.35	0.0	0.313
14	13	6.31	4.2	0.468	19	2.75	3.9	0.509
15	7.0	5.90	0.0	0.268	9.0	2.51	0.0	0.329
16	10	5.70	0.0	0.347	6.0	1.02	1.8	0.278
17	7.0	1.71	0.0	0.299	5.0	1.22	0.0	0.261
18	15	2.26	3.9	0.490	20	2.81	4.1	0.539
19	12	1.15	4.4	0.456	15	0.63	0.0	0.445
20	9.0	3.83	0.0	0.347	10	1.67	0.0	0.361

summarizes the coagulation treatment results from all 20 jar-test runs for both PACl and FeCl₃. For each coagulant, the residual TOC, turbidity, phosphate, and UV₂₅₄ were measured after settling. PACl and FeCl₃ achieved similar TOC removal: PACl reduced TOC to between 5.0 and 19.0 mg/L, and FeCl₃ reduced TOC to between 4.0 and 20.0 mg/L. The lowest turbidity values, 0.28 NTU for PACl (Run 5) and 0.16 NTU for FeCl₃ (Run 5), highlight conditions where floc formation was most effective. In contrast, some runs (e.g., Run 8) resulted in higher turbidity (>5 NTU), indicating suboptimal mixing or dosing. Phosphate removal showed the maximum at 4.4 mg L⁻¹ with PACl (Run 19) and 4.1 mg L⁻¹ with FeCl₃ (Runs 11 and 18). UV₂₅₄ absorbance, which is an indicator of residual organic matters with aromatic functional groups, ranged from 0.233 to 0.504 cm⁻¹ for PACl and from 0.209 to 0.539 cm⁻¹ for FeCl₃. Overall, both coagulants demonstrated strong performance under certain temperature, pH, and dosage combinations, providing a robust dataset for our subsequent modeling efforts.

The differences in removal efficiency observed under varying conditions can be explained by the underlying coagulation mechanisms of aluminum- and iron-based coagulants. At acidic pH, limited hydrolysis of Al³⁺ and Fe³⁺ restricts floc formation, resulting in poorer removal. As pH increases toward neutral and slightly alkaline conditions, extensive hydrolysis produces polymeric and amorphous hydroxide species (e.g., Al(OH)₃(s), Fe(OH)₃(s)), which enhance charge neutralization and sweep flocculation, thereby improving turbidity and organic matter removal. At higher doses and alkaline pH, hydrolyzed Al³⁺ or Fe³⁺ species promote charge neutralization and sweep flocculation, enhancing removal of turbidity and organic matter [64,65]. Elevated temperature further accelerates hydrolysis reactions and promotes floc growth, increasing removal efficiency [66]. Phosphate elimination occurs primarily via co-precipitation with Fe/Al hydroxides and adsorption onto their surfaces [67,68], which explains the strong dependence of phosphate removal on both pH and coagulant dose. These mechanisms are consistent with prior studies reporting the importance of pH, temperature, and dose in controlling the speciation of hydrolyzed coagulants and their pollutant binding capacities.

3.2. RSM Models

3.2.1. Analysis of PACl Coagulation by RSM Model

Although RSM is a traditional method, it is still widely applied in environmental engineering due to its efficiency with small datasets and its ability to reveal factor interactions through interpretable response surfaces. In this study, RSM was used not only for optimization but also as a benchmark to compare with advanced machine learning models (SVM and RF).

Using the experimental data in Table 3, RSM models were developed for the analysis of coagulation efficiency under various conditions. Figure 1 illustrates the response surfaces for PACl coagulation at three temperatures (rows) across four key water-quality variables (columns): TOC, turbidity, phosphate, and UV₂₅₄. Residual TOC (Figure 1a) is almost insensitive to operating conditions with a slight increase at higher dose and acidic pH, indicating limited organic removal leverage at this temperature. Turbidity (Figure 1b) shows a convex response. It reaches a maximum at mid-range dose/pH and drops toward alkaline pH. Phosphate (Figure 1c) exhibits relatively a strong bowl-shaped trend. Residual P is minimized at higher dose and alkaline pH but rises sharply at low dose and acidic pH, indicating the importance of both variables for P removal. UV₂₅₄ (Figure 1d) decreases gradually toward alkaline pH and lower doses and increases toward acidic pH and high dose, suggesting that removal of UV-absorbing organics is favored under alkaline conditions and without overdosing.

At 25 °C (Figure 1e–h), residual TOC (Figure 1e) decreases monotonically with increasing coagulant dose and pH. On the other hand, turbidity (Figure 1f) displays a convex surface with the lowest value at low dose and alkaline pH. Phosphate (Figure 1g) shows strong curvature. Minimal residual P occurs at high dose and high pH, while low dose and acidic pH cause a sharp increase. UV₂₅₄ absorbance (Figure 1h) decreases toward higher dose and pH, indicating improved removal of UV-absorbing organics under alkaline, adequately dosed conditions. Compared with the cases at 17.5 °C, the removals of these compounds were improved, indicating the effect of temperature on the coagulation efficiency.

At the higher temperature, 32.5 °C (Figure 1i–l), all surfaces become steeper: TOC removal accelerates strongly with dose and pH, the turbidity removal exhibits low values at high does and neutral pH, phosphate removal increases markedly at high dose and pH, and UV₂₅₄ absorbance drops under the same conditions. These progressive changes confirm that increasing temperature enhances both the sensitivity and magnitude of PACl’s coagulation performance. Except for the turbidity, the optimal operating condition shifts toward higher doses and more alkaline pH.

3.2.2. Analysis of FeCl₃ Coagulation by RSM Model

Figure 2 presents the response surfaces for FeCl₃ coagulation across three temperatures, revealing distinct temperature-dependent behaviors for each water-quality metric. At 17.5 °C (Figure 2a–d), TOC removal (Figure 2a) increases modestly with both dose and pH, producing a gently sloping plane, while turbidity (Figure 2b) slightly increases with increasing dose and pH. Phosphate (Figure 2c) exhibits pronounced curvature even at this low temperature, with removal accelerating at higher doses (>0.4 mg/L) and acidic pH. UV₂₅₄ (Figure 2d) is essentially unaffected by operational variables, forming a flat plateau that signifies limited organic removal at 17.5 °C.

Raising the temperature to 25 °C (Figure 2e–h) amplifies all responses: the TOC plane (Figure 2e) tilts more steeply toward higher doses and pH, turbidity (Figure 2f) shows low values (~1 NTU) at low dose and pH. Phosphate (Figure 2g) decreases sharply once dose exceeds ~0.45 mg/L at acidic pH, whereas low dose and alkaline pH result in the poorest removal. UV₂₅₄ (Figure 2h) decreases linearly with dose and does not significantly depend on pH. At 32.5 °C (Figure 2i–l), these trends intensify, underscoring that high temperature and pH conditions coupled with sufficient FeCl₃ dosing are critical for optimal removal of both organic and particulate contaminants.

3.2.3. ANOVA of RSM Models

The significance of the RSM models and their terms was determined through analysis of variance (ANOVA). For PACl (

Table 4. Comparison of R² values for RSM, SVM, and RF models.

	TOC		Turbidity		Phosphate		UV254
	F-Value	p-Value	F-Value	p-Value	F-Value	p-Value	F-Value	p-Value
Model	15.80	<0.0001	15.04	<0.0001	6.94	0.0018	18.12	<0.0001
A (dose)	21.10	0.0003	18.72	0.0007	19.24	0.0007	39.82	<0.0001
B (Temperature)	21.10	0.0003	23.42	0.0003	5.49	0.0358	30.42	<0.0001
C (pH)	-	-	0.406	0.5341	0.4080	0.5341	0.5735	0.4614
AB	5.21	0.0365	-	-	-	-	6.51	0.0230
AC	-	-	-	-	-	-	-	-
BC	-	-	-	-	-	-	-	-
A2	-	-	5.54	0.0337	4.62	0.0510	-	-
B2	-	-	-	-	3.75	0.0749	-	-
C2	-	-	30.72	0.0001	13.67	0.0027	13.28	0.0027
Lack of fit	3.54	0.0868	1.72	0.2860	4.60	0.0530	3.55	0.0883

), the quadratic RSM model is highly significant for all four responses (model p < 0.0001), confirming that the chosen second-order polynomial adequately describes the effects of dose (A), temperature (B), and pH (C) on coagulation performance. Both the linear dose (A) and temperature (B) terms exhibit strong influence across TOC, turbidity, phosphate, and UV₂₅₄ removal (p < 0.001), while the pH linear term is not significant. Notably, the A × B interaction contributes significantly to TOC (p = 0.0365) and UV₂₅₄ (p = 0.0230) removal, indicating synergistic behavior between coagulant dose and temperature. Curvature is captured by the quadratic terms: B² is significant for turbidity (p = 0.0337) and phosphate (p = 0.0510), and C² is highly significant for turbidity, phosphate, and UV₂₅₄ (p < 0.003), underscoring nonlinear pH effects. In all cases, the lack-of-fit tests are non-significant (p > 0.05), confirming that the models neither overfit nor omit systematic variation.

For FeCl₃ (

Table 5. Comparison of R² values for RSM, SVM, and RF models.

	TOC		Turbidity		Phosphate		UV254
	F-Value	p-Value	F-Value	p-Value	F-Value	p-Value	F-Value	p-Value
Model	11.14	0.0008	20.28	<0.0001	7.86	0.0019	15.91	<0.0001
A (dose)	4.46	0.0499	11.79	0.0037	4.05	0.0614	17.11	0.0010
B (Temperature)	17.82	0.006	46.06	<0.0001	14.86	0.0014	34.30	<0.0001
C (pH)	-	-	12.08	0.0034	4.68	0.0461	0.3560	0.5693
AB	-	-	-	-	-	-	11.77	0.0041
AC	-	-	-	-	-	-	-	-
BC	-	-	-	-	-	-	-	-
A2	-	-	-	-	-	-	-	-
B2	-	-	11.18	0.0044	-	-	-	-
C2	-	-	-	-	-	-	16.01	0.0013
Lack of fit	4.12	0.0644	2.62	0.150	4.31	0.0596	1.20	0.4422

), the overall quadratic models also demonstrate high statistical significance (TOC p = 0.0008; turbidity and UV₂₅₄ p < 0.0001; phosphate p = 0.0019). Temperature (B) is the most influential factor, with strong linear effects on turbidity (p < 0.0001), UV₂₅₄ (p < 0.0001), and phosphate (p = 0.0014), while dose (A) significantly affects turbidity (p = 0.0037) and UV₂₅₄ (p = 0.0010) but has a marginal impact on TOC (p = 0.0499). The pH linear term shows significance for turbidity (p = 0.0034) and phosphate (p = 0.0461) but not for TOC or UV₂₅₄. A single interaction (A × B) is significant for UV₂₅₄ removal (p = 0.0041), and quadratic curvature appears through B² for turbidity (p = 0.0044) and C² for UV₂₅₄ (p = 0.0013). As with PACl, non-significant lack-of-fit values (p > 0.05) indicate that these second-order models reliably capture the system behavior without overparameterization.

3.3. SVM Models

3.3.1. Analysis of PACl Coagulation by SVM Model

In addition to RSM, SVM models were developed using the experimental data in Table 3. The SVM response surfaces for PACl are shown in Figure 3 to reveal distinct nonlinear interactions between dose and pH at different temperatures. At 17.5 °C (Figure 3a–d), TOC (Figure 3a) shows only a weak gradient, indicating limited sensitivity to operational changes, whereas turbidity (Figure 3b) displays a broad convex surface with a minimum toward alkaline pH at mid-range doses. Phosphate (Figure 3c) removal exhibits a non-linear behavior: residual P decreases rapidly only when dose and pH are simultaneously elevated. UV₂₅₄ (Figure 3d) shows a relatively modest slope at this temperature, suggesting that removal of UV-absorbing organics is limited under cool conditions.

At 25 °C, the response surfaces (Figure 3e–h) indicate that PACl performance improves as both dose and pH increase, with a distinct threshold behavior. Residual TOC (Figure 3e) declines almost linearly with increasing dose and pH. Turbidity (Figure 3f) exhibits a broad convex profile with a minimum at high doses and alkaline pH. Phosphate removal (Figure 3f) remains limited until both variables are high, after which residual P drops sharply (dose ≥ 0.45 mg/L, pH ≥ 8). UV₂₅₄ (Figure 3h) decreases nearly linearly toward the high-dose, high-pH conditions, reflecting enhanced elimination of UV-absorbing organics. At 32.5 °C, the response surfaces for PACl (Figure 3i–l) show stronger dependence on dose and pH than at 25 °C. The overall trends are similar to the cases at lower temperatures but exhibit lower values, indicating that the removal efficiency was improved with an increase in temperature.

Compared with the RSM surfaces (Figure 1), the SVM surfaces (Figure 3) show the same overall tendencies for PACl. However, the SVM responses are much sharper than the RSM responses. RSM yields smooth, quadratic planes and bowls that suggest broad operating regions, whereas SVM presents steeper gradients and discrete valleys/plateaus, indicating narrower windows of effectiveness. The difference becomes more evident at higher temperature. Accordingly, it is suggested to consider the differences between the models for more reliable optimization of the coagulation process for BWRO brine.

3.3.2. Analysis of FeCl₃ Coagulation by SVM Model

Figure 4 presents the SVM response surfaces for FeCl₃, which show the effects of FeCl₃ dose and pH on four responses across temperature. At 17.5 °C (Figure 4a–d), TOC shows a gentle decline with increasing dose and pH, whereas turbidity is weakly responsive, indicating that Fe(III) coagulation achieves stable clarification over a broad operating space. Residual phosphate is high at low dose/acidic pH but falls rapidly once both variables increase, consistent with Fe(III) hydrolysis and co-precipitation of Fe–P complexes. UV₂₅₄ decreases modestly with dose and pH, reflecting progressive removal of UV-absorbing organics by sweep flocculation and adsorption.

Increasing the temperature to 25 and 32.5 °C (Figure 4e–l) amplifies these trends and improves the overall removal efficiency. TOC and UV₂₅₄ decline more steeply toward the high-dose/alkaline corner, while turbidity remains low across much of the conditions, underscoring FeCl₃’s robustness for clarification. Residual Phosphate substantially drops when dose and pH are high, consistent with faster Fe(III) hydrolysis and enhanced floc growth at higher temperature. Overall, the SVM results indicate that FeCl₃ provides comparatively high coagulation efficiencies, with best simultaneous performance for all responses at moderate-to-high doses and alkaline pH, particularly at ≥25 °C. These results also qualitatively match with those by RSM models (Figure 2).

3.4. RF Models

3.4.1. Analysis of PACl Coagulation by RF Model

Along with the RSM and SVM models, RF models were also developed using the data in Table 3. Figure 5 presents response surfaces by RF models for PACl, mapping coagulant dose (0.20–0.60 mM) and pH (5.0–9.0) to four responses, including TOC, turbidity, phosphate, and UV₂₅₄, at 17.5, 25, and 32.5 °C. The tree-ensemble in RF models produces the expected piecewise (step-like) landscapes that partition the operating space into discrete regimes, with a consistent switch from poor to good performance. These patterns may also result from the insufficient data points used to develop these models.

At 17.5 °C (Figure 5a–d), TOC decreases with increasing dose, whereas turbidity remains elevated under acidic pH and under-dosing but collapses to a low-NTU plateau once the threshold is crossed; phosphate exhibits the sharpest transition from high residuals at low dose/low pH to near-minimum values in the high-dose/alkaline conditions. UV₂₅₄ shows a similar step change. With an increase in temperature to 25 (Figure 5e–h) and 32.5 °C (Figure 5i–l), these transitions become steeper, and the low-residual plateaus expand, indicating higher removal efficiency at higher temperature. Overall, the RF models show a practical operating region centered around ~0.5 mg L⁻¹ and pH 8–9 where turbidity, phosphate, and UV₂₅₄ are simultaneously minimized (with TOC only weakly sensitive), while acidic conditions and under-dosing lead to distinctly inferior outcomes.

Compared with the smooth and continuous varying surfaces from RSM and SVM for PACl, the RF surfaces for PACl (Figure 5) display step-like plateaus and abrupt switches. This contrast is methodological: RSM fits a single global quadratic, enforcing smooth trends and gradually varying interactions; SVM (with kernel and regularization) also produces continuous functions but can capture stronger curvature; RF partitions the dose–pH plane with many axis-aligned splits and averages leaf values, yielding piecewise-constant regions that look like “stairs.” The discrepancy is amplified by data scarcity: only 20 DOE runs were available to train the RF (with deep trees and small minimum split), which makes a high-variance model prone to overfitting local noise and positioning hard splits at specific doses/pH. Consequently, RF highlights narrow threshold islands whereas RSM/SVM show broader, smoother gradients toward similar optima.

3.4.2. Analysis of FeCl₃ Coagulation by RF Model

Figure 6 shows the RF-predicted surfaces for FeCl₃, characterized by step-like plateaus and abrupt transitions typical of tree ensembles. At 17.5 °C (Figure 6a–d), TOC and UV₂₅₄ vary weakly with operating conditions, turbidity remains low over most of the domain, and phosphate is high except near the high-dose/alkaline conditions. At 25 (Figure 6e–h) and 32.5 °C (Figure 6i–l), RF identifies a clear threshold (dose ≈ 0.45–0.50 mg L⁻¹ and pH ≥ 7.5–8.0) beyond which most responses shift to a stable low-residual plateau; turbidity becomes essentially invariant (~1 NTU), while TOC, UV₂₅₄, and especially phosphate drop sharply after the threshold. Overall, the RF model points to a robust operating window at moderate-to-high dose under alkaline conditions—broader at higher temperature—consistent with rule-based, high-confidence regions of effective Fe(III) coagulation.

For FeCl₃, the reasons the RF response surfaces differ from the RSM and SVM results are essentially the same as for PACl: RF’s tree-based, piecewise partitioning produces step-like plateaus and abrupt threshold behaviors, whereas RSM and SVM result in smooth, continuous trends. The very small training set (20 runs) amplifies this behavior, making RF prone to hard splits at specific dose—pH values and exaggerating transitions. Thus, as with PACl, the RF maps for FeCl₃ should be interpreted as indicating transition zones rather than precise boundaries. Adding the experimental dataset and tightening RF regularization would likely reduce the stair-step artifacts and align the RF surfaces more closely with the smoother RSM/SVM responses.

While the response surfaces for PACl and FeCl₃ showed similar trends in TOC and UV₂₅₄ removal, notable differences were observed in turbidity and phosphate removal. These differences can be explained by the distinct hydrolysis and complexation behaviors of aluminum and iron species. Fe³⁺ tends to form larger and denser hydroxide flocs, which enhance sweep flocculation and thereby improve turbidity reduction [69,70]. Moreover, the strong affinity of Fe³⁺ for phosphate ions facilitates co-precipitation and adsorption, resulting in higher P removal efficiency than with Al³⁺ [71]. By contrast, PACl tends to form more stable but less dense flocs, which are effective for organic removal but less effective for turbidity and phosphate removal [69].

3.5. Comparison of RSM, SVM, and RF Models

The coefficient of determination, R², was calculated for each model and compared to assess the model’s accuracy. For PACl (

Table 6. Comparison of R² values for RSM, SVM, and RF models.

Models	PACl				FeCl3
	TOC (mg/L)	Turbidity (NTU)	Phosphate (mg/L)	UV254 (cm−1)	TOC (mg/L)	Turbidity (NTU)	Phosphate (mg/L)	UV254 (cm−1)
RSM	0.9772	0.9477	0.8875	0.9937	0.9210	0.9580	0.7445	0.9901
SVM	0.9703	0.8170	0.7079	0.9796	0.8903	0.9200	0.6815	0.9773
RF	0.9051	0.8418	0.5858	0.9240	0.8965	0.9169	0.6324	0.9002

), the RSM models consistently achieved the highest goodness-of-fit across all four responses, with R² values of 0.9772 for TOC, 0.9477 for turbidity, 0.8875 for phosphate, and 0.9937 for UV₂₅₄. The SVM models followed closely for TOC (R² = 0.9703) and UV₂₅₄ (R² = 0.9796) but showed a noticeable drop in predictive power for turbidity (R² = 0.8170) and phosphate removal (R² = 0.7079). RF regression yielded the lowest R² values particularly for phosphate (R² = 0.5858) and TOC (R² = 0.9051), although it still provided reasonable fits for turbidity (R² = 0.8418) and UV₂₅₄ (R² = 0.9240). While all three models capture the main trends in the data, RSM seems to deliver the most accurate quantitative predictions in this case, with SVM offering a good compromise between flexibility and accuracy and RF trailing primarily due to underfitting the more complex response surfaces. As mentioned earlier, the inferior performance of RF is attributed to the insufficient data used for model development.

For FeCl₃ (Table 6), a similar ranking of model performance is observed, albeit with some shifts in relative strengths. RSM again leads with R² values of 0.9210 (TOC), 0.9580 (turbidity), 0.7445 (phosphate), and 0.9901 (UV₂₅₄), underscoring its robustness even when experimental conditions change. SVM maintains strong performance for turbidity (R² = 0.9200) and UV₂₅₄ (R² = 0.9773), though its fit for phosphate (R² = 0.6815) and TOC (R² = 0.8903) remains lower than RSM. RF shows a relatively poor phosphate prediction (R² = 0.6324) and achieves a solid turbidity fit (R² = 0.9169), yet it continues to underperform relative to the other methods for TOC (R² = 0.8965) and UV₂₅₄ (R² = 0.9002). Overall, these comparisons confirm that these three models show different results and have pros and cons. RSM and SVM offer consistently reliable models, there may be situations that these models fail to fit the data. Although RF requires further tuning or additional data to match the accuracy of the other techniques, it may be less sensitive to overfitting [72].

3.6. Optimization by Ensemble Approach

3.6.1. Optimum Conditions for PACl Coagulation

The optimization was conducted as a multi-objective process, where several responses (TOC, turbidity, UV₂₅₄, and orthophosphate removal) were considered simultaneously. Instead of focusing on a single objective, consensus regions were identified where all responses met acceptable thresholds, thereby ensuring robust and balanced operating conditions. Using the previous three models, the optimum coagulation conditions were proposed based on the ensemble approach. Figure 7 presents ensemble optimization map for PACl where the high-efficiency zones predicted by RSM (blue), SVM (red), and RF (yellow) models have been overlaid and shaded to indicate model agreement. In each model, the criteria for the shaded region are set to: TOC ≤ 10 mg/L (50% removal), turbidity ≤ 5 NTU, phosphate ≤ 1 mg/L (approximately 76% removal). In this analysis, UV₂₅₄ were not included since the organic compounds absorbing UV₂₅₄ are generally included in TOC. Lighter areas denote regions where only one model forecasts optimal removal, while progressively darker shading highlights parameter combinations where two or all three methods concur.

At 25 °C (Figure 7a), the darkest region, which occurs around pH 7.0–9.0 and pH 5.5–5.7 at 0.52–0.60 mg/L PACl. This identifies the consensus “sweet spot” for simultaneously minimizing TOC, turbidity, and phosphate. By focusing on this overlapping zone, one can select robust coagulation conditions that are supported by multiple modeling approaches, thereby reducing prediction uncertainty. The optimization maps for PACl at 17.5 °C could not be provided because there is no condition to satisfy the above criteria. At 32.5 °C (Figure 7b), the consensus region shifts slightly toward higher doses and broader pH values compared to 25 °C. While RSM still favors a mid/high-dose range (>0.35–0.45 mg L⁻¹), both SVM and RF extend the high-efficiency overlap up to ~0.60 mg L⁻¹ and down to pH 6.0. The darkest shading now spans 0.37–0.60 mg/L across pH 5.5–9.0, indicating enhanced agreement among all three models under high temperature conditions. This broadened region suggests that all three models predict higher removal performance over wider pH conditions.

In Figure 7a, the SVM model identified a broader high-efficiency region compared to RSM although SVM typically produces narrower operating windows than RSM because of its sharp decision boundaries. This discrepancy arises from the limited experimental dataset and the nature of the SVM kernel, which can extend threshold regions when data points cluster near removal cutoffs [73,74]. In contrast, RSM’s quadratic formulation tends to smooth and confine response surfaces, sometimes underestimating broader optimal regions. This highlights that differences between models should be interpreted cautiously, particularly when the number of experimental runs is limited.

3.6.2. Optimum Conditions for FeCl₃ Coagulation

Figure 8 shows the ensemble optimization for FeCl₃ by overlaying the high-efficiency regions predicted independently by RSM, SVM, and RF (left insets) and then mapping their consensus in the main figure. At 25 °C (Figure 8a), the three models jointly indicate the optimal conditions with high dose and low pH. The darkest shading marks the area where all models agree, which is relatively due to the results of RSM model. Regions where only two models concur form a surrounding light-blue buffer, reflecting minor differences in how each model shows the boundary. At 32.5 °C (b), the consensus region expands and shifts slightly toward lower dose (0.30–0.60 mM) and broader pH (5.0–9.0), indicating that higher temperature increases the efficiency of FeCl₃ coagulation. Overall, the ensemble balances model-specific biases, resulting in practical operating conditions that emphasize cross-model agreement rather than any single model’s prediction.

3.6.3. Practical Considerations for Industrial-Scale Application

A practical caution of this ensemble optimization is that a weak or poorly calibrated base model can distort the consensus region. Ensembles work best when base models are both accurate and exhibit complementary errors; otherwise one model can dominate or add noise to the aggregate decision region. In this study, RF models may cause such problems due to its low R² values. To mitigate this, it is recommended to implement performance-weighted ensembling rather than equal voting so that higher-performing models (here, RSM/SVM) contribute more than weaker ones (RF) [75].

Although incorporating multiple parameters and their cross-effects results in well-fitted models that describe laboratory-scale data with high accuracy, it is important to note that in industrial-scale applications, simpler modeling or rule-based approaches are often preferred. This preference reflects the need for operational simplicity, lower computational demands, and user-friendly decision support. Therefore, the ensemble strategy proposed here should be considered as a complementary tool that can inform design and optimization, while practical implementations may rely on simplified models adapted for real-time plant operation.

It is also important to note that the present models were developed based on a specific feed water composition (Table 1). Since wastewater reuse brines may exhibit significant variability in organic matter, salinity, and nutrient content, the predictive power of the models may be affected under different conditions. Therefore, prior to application in other facilities, site-specific validation or recalibration with representative data is recommended. This step will ensure that the optimization outcomes remain reliable and applicable across variable feed water qualities.

4. Conclusions

This study systematically evaluated the coagulation of BWRO brine using PACl and FeCl₃ across a wide range of dose, pH, and temperature conditions, and demonstrated the power of combining traditional statistical modeling (RSM) with machine learning techniques (SVM and RF) in an ensemble framework. Jar-test experiments showed that both coagulants can achieve substantial removal of TOC, turbidity, phosphate, and UV₂₅₄, with FeCl₃ generally outperforming PACl under comparable conditions. RSM provided highly accurate, smooth response surfaces that captured linear, quadratic, and interaction effects, while SVM highlighted sharp threshold behaviors and RF showed robust “all-or-nothing” operating zones.

By overlaying the individual model predictions, the ensemble approach utilized each model’s strengths into consensus region of dose–pH–temperature combinations that minimize prediction uncertainty. At 25 °C, the optimal conditions for PACl occurred on 0.52–0.60 mg/L PACl at acidic and alkaline pH conditions, whereas the optimum conditions for FeCl₃ shifted to 0.58–0.60 mM at pH 5.0–8.7. Increasing the temperature to 32.5 °C broadened the consensus region due to improved coagulation efficiency. These ensemble-derived guidelines provide a practical, data-driven foundation for designing and operating coagulation processes in BWRO brine treatment. More broadly, the results underscore the value of integrating complementary modeling techniques to enhance predictive reliability and support multiobjective optimization in water-treatment applications.

Considering both the accuracy of predictions and the practical requirements for wider application, response surface methodology (RSM) is recommended as a robust first choice. RSM demonstrated consistently high predictive power and produces smooth, interpretable response surfaces that can be easily adopted by practitioners in both academic and industrial settings. While support vector machines (SVM) and random forest (RF) provided additional insights, they may require more computational resources, expertise, and larger datasets for stable performance. Therefore, for a broader audience seeking a balance between reliability, simplicity, and accessibility, RSM represents the most practical modeling approach, with ensemble strategies offering added value where sufficient expertise and data are available.