Towards Generic Failure-Prediction Models in Large-Scale Distributed Computing Systems

Abstract

The increasing complexity of Distributed Computing (DC) systems requires advanced failure-prediction models to enhance reliability and efficiency. This study proposes a comprehensive methodology for developing generic machine learning (ML) models capable of cross-layer and cross-platform failure-prediction without requiring platform-specific retraining. Using the Grid5000 failure dataset from the Failure Trace Archive (FTA), we explored Linear and Logistic Regression, Random Forest, and XGBoost to predict three critical metrics: Time Between Failures (TBF), Time to Return/Repair (TTR), and Failing Node Identification (FNI). Our approach involved extensive exploratory data analysis (EDA), statistical examination of failure patterns, and model evaluation across the cluster, site, and system levels. The results demonstrate that XGBoost consistently outperforms the other models, achieving near-perfect 100% accuracy for TBF and FNI, with robust generalisability across diverse DC environments. In addition, we introduce a hierarchical DC architecture that integrates these failure-prediction models. In the form of a use case, we also demonstrate how service providers can use these prediction models to balance service reliability and cost.

1. Introduction

Accurate failure identification and predictive modelling using Machine Learning (ML) techniques have become essential components of managing modern Distributed Computing (DC) systems [1,2]. These techniques enable system administrators and resource management units to proactively detect potential failures, thereby minimising service disruptions, optimising resource allocation, and enhancing the system reliability [3]. However, the heterogeneous nature of DC systems, comprising multilayered hierarchical architectures of interconnected nodes, clusters, and sites (datacenters) (Figure 1) combined with the resource-intensive nature of ML methods, poses significant challenges. These include developing advanced and generalised failure-prediction models that enable cross-layer and cross-platform transferability without retraining, that is, models which can be trained on one layer or platform and seamlessly applied to other layers or entirely different DC platforms. This challenge arises from the resource and architectural heterogeneity of DC systems as well as from the fundamental trade-off between prediction accuracy, cost of prediction, and cost of repair in such complex environments [4].

Node-specific failure-prediction models can achieve high accuracy, but are expensive to implement at scale. For example, deploying individual models for thousands of compute nodes would be very expensive in terms of computational resources and management overheads. In contrast, more generic failure-prediction models at the cluster, site, or system level are more cost-effective and easier to manage; however, these broader models tend to sacrifice failure-prediction accuracy as they attempt to be generic across increasingly diverse components in the DC hierarchy. This inverse relationship between model generalisation and prediction accuracy presents a significant challenge for the development of effective ML-based failure-prediction solutions for large-scale DC platforms.

To address this challenge, this study proposes the development and evaluation of generic ML models for failure detection and prediction, which are designed to operate across multiple hierarchical levels of diverse DC environments without requiring platform-specific retraining. To achieve this, we explored various ML models, including Linear and logistic regression, Random Forest, and Extreme Gradient Boost (XGBoost) to perform both regression and classification tasks. These models were explicitly designed to predict three key metrics: (1) time between failures (TBF) to predict the time of the next failure, (2) time to return/repair (TTR) to predict the duration of the next failure, and (3) failing node identification (FNI) to predict the next nodes that are likely to fail. FNI prediction is treated as a classification problem, whereas TBF and TTR predictions are treated as regression problems. For model training and testing, we used several datasets from the Grid5000 failure benchmark obtained from the Failure Trace Archives [5]. The specific contributions of this study are as follows:

• We proposed a comprehensive methodology for developing and evaluating cross-layer and cross-platform failure-prediction models in DC environments.
• We conducted an in-depth exploratory data analysis (EDA) and statistically examined system failure characteristics, offering valuable insights for researchers seeking to understand complex failure patterns in DC systems.
• We developed, evaluated, and validated a range of ML-based failure-prediction models at multiple hierarchical levels (cluster and site) within a DC platform, as well as across different DC platforms.
• We proposed a hierarchical architecture and to integrate failure-prediction models to enhance DC system reliability and cost efficiency.

The remainder of this paper is organised as follows: Section 2 reviews the related work. Section 3 details the failure traces used in this study. Section 4 outlines the methodology employed, the system setup, and covers data analysis and model generation. Section 5 presents the results and findings, followed by the proposed system architecture in Section 6. Section 7 concludes the study and discusses future work.

2. Related Work

In this section, we examine the existing research on failure-prediction modelling in DC systems. We categorised the related literature into four main approaches: Statistics and Rule-based, Traditional Machine Learning (ML), Deep Learning, and Hybrid techniques. By examining these categories, we aim to clarify how our work contributes to and extends current state-of-the-art DC failure-prediction.

Statistics and Rule-based Methods: Earlier approaches for fault detection relied on rule-based systems and statistical methods. Hochenbaum et al. [6] introduced two methods for detecting anomalies in cloud infrastructure data, both focusing on long-term trends. The authors enhanced the generalised Extreme Studentized Deviate (ESD) test, using time-series decomposition and robust statistics, to improve accuracy while reducing false positives. These methods account for daily and weekly patterns in metrics, such as Tweets-Per-Second and CPU usage. The authors demonstrated the effectiveness of their methods using real production data and a randomised approach to introduce anomalies, both of which help to maintain high availability and performance in cloud environments. Kourtis et al. [7] presented a method for finding anomalies in Network Function Virtualization (NFV) environments using statistical techniques. The authors combined an open-source monitoring system with statistical methods to analyse multiple metrics from the NFV infrastructure and identify unusual behaviours in network functions. This approach detects performance problems and security threats early, thereby ensuring the quality and security of the NFV services. Most statistics and rule-based methods, such as those presented above, are straightforward and interpretable but often struggle with high-dimensional data and may not capture complex fault patterns in current DC platforms [8].

Traditional Machine Learning-based Methods: In recent years, machine learning (ML) techniques have gained significant attention for fault detection and prediction modelling. Mariani et al. [1] introduced PreMiSE, a method for predicting failures and locating faults in DC platforms. It combines anomaly detection using unsupervised ML to identify unusual behaviours with signature-based prediction, which employs supervised ML to recognise known failure patterns. The authors also used clustering to group similar behaviours and identify potential failures. By integrating these methods, PreMiSE was demonstrated to reduce false alarms and improve the accuracy of failure-prediction, thereby enhancing the reliability of multi-tier DC systems. ML approaches offer significant advantages, particularly in their capacity to manage large datasets and model the complex relationships between features. However, they rely heavily on the availability of the data, which is not always accessible [9]. Additionally, the performance of these models depends on the quality and representativeness of the training data, which can restrict their effectiveness in highly dynamic DC environments [10].

Deep Learning-based Methods: These methods are proposed based on the promising results from ML approaches, while addressing some of their shortcomings. For example, Long Short-Term Memory (LSTM) networks have emerged as powerful tools for fault prediction. Gao et al. [11] presented a multilayer Bidirectional Long Short Term Memory (Bi-LSTM) model designed to predict task and job failures in cloud data centres that often face high failure rates owing to hardware and software problems. The authors analysed previous system message logs to improve the prediction accuracy and achieved very high accuracy for tasks and job failures in their experiments. Shen et al. [12] presents a model using LSTM networks to predict hard disk drive (HDD) failures in mobile edge computing. The model analyses long-term data trends to improve the prediction accuracy and introduces a method for assessing HDD health by tracking current conditions over time. Tests on real-world datasets show that their proposed model achieves high accuracy with minimal resource usage, thereby enhancing device reliability in edge computing environments. Most of these approaches are based on LSTM’s learning patterns over time and are known to outperform traditional ML models in time-series fault prediction [13,14,15]. However, they face criticism for their lack of interpretability because their internal mechanisms are not easily understood [16]. Training deep learning models can also be computationally expensive and slow, posing additional challenges for deployment in resource-limited environments or real-time systems [17].

Hybrid Methods: These methods integrate multiple approaches to enhance the fault detection and prediction in DC platforms. Gaykar et al. [18] presented a hybrid ML model that combined various supervised learning techniques to detect and mitigate job failures in virtual machines (VMs) in DC environments. To avoid failures, the proposed model focuses on predicting underperforming nodes using metrics, such as CPU and memory loads, by integrating statistical methods with ML algorithms. The experimental results showed improved prediction accuracy while enhancing the overall system reliability. Ensemble learning techniques that aggregate the strengths of various models have also been employed to achieve superior results. Zhong et al. [19] proposed a framework (HELAD) to effectively detect network anomalies and avoid failures in DC systems. HELAD combines various ML algorithms, specifically the Damped Incremental Statistics algorithm, for feature extraction from network traffic, followed by training an Auto-encoder with a small amount of labelled data to mark abnormal scores. The output of the Auto-encoder is then used to train an LSTM network, which helps predict abnormal scores to recognise malicious activities in diverse environments. Although these hybrid approaches yield a better performance, they add complexity to the model training and deployment process, sharing concerns with the use of deep learning approaches [20].

Despite these advancements, noticeable gaps remain in the literature. First, there is a lack of models that are generalisable across diverse DC systems, both cross-layer and cross-platform. Second, real-time fault prediction remains underexplored, with high-accuracy offline models often untested in live scenarios, limiting their utility for rapid decision-making in modern DC systems. This study addresses these gaps by developing accurate and generalisable fault prediction models using a combination of ML techniques, with the aim of reducing downtime through proactive fault management in multilayered and multiple DC platforms.

3. Overview of Failure Traces

We used multiple failure traces from the Grid5000 testbed, which is a large-scale platform for experiment-driven research in parallel and distributed computing, including cloud computing, high-performance computing (HPC), Big Data, and AI [21]. The traces were sourced from the Failure Trace Archive (FTA), a public repository containing standardised tab-separated files from 26 diverse parallel and distributed computing systems [5].

Grid5000 comprises nine geographically distributed sites (sites 1 to 9), each operating as a local datacentre with its own clusters of nodes that can be independently reserved, configured, and operated. These sites are interconnected via RENATER, a dedicated 10 Gbps national backbone network [22]. The architecture is often described as a “cluster of clusters” or a “meta-datacenter,” exhibiting a hierarchical structure as illustrated in Figure 1, where each site contains multiple clusters, and each cluster consists of individual hosts or nodes. Specifically, site 1 included four clusters (c1–c4), sites 4–6 each had two clusters (c1–c2), and sites 2–3 and 7–9 each had one cluster (c1), totalling 1288 nodes.

This high degree of heterogeneity, geographical independence, and hierarchical structure among the computational resources means that the data collected from the Grid5000 system can be accurately described as originating from different DC systems. This makes the Grid5000 dataset a robust choice for developing targeted generic failure-prediction models.

4. Methodology

Figure 2 illustrates our methodology and highlights the various stages of this study. It is divided into three phases: (1) Data Preprocessing and Analysis, (2) Model Generation, and (3) Experiments and Analysis. Phase-1 focuses on data collection, preprocessing, and initial exploratory and statistical analyses. This ensures data quality, integrity, and preparation for subsequent modelling. Phase-2 involves the development and training of ML-based failure-prediction models. This includes the model selection, design, training and feature importance analysis. Phase-3 evaluates the performance of the ML models using comprehensive metrics and provides critical insights and interpretations.

4.1. Data Preprocessing and Analysis

This section corresponds to Phase-1 of the methodology (Figure 2). It involves pre-processing and analysing Grid5000 failure traces to create a dataset optimised for modelling. The original dataset, consisting of tab-separated files, is pre-processed by selecting and integrating relevant data, removing empty files (’event_state.tab’, ’node_perf.tab’) and eliminating features with no variability (e.g., component_id, platform_id). This resulted in a reduction to seven key features: node_id, node_name, event_id, event_type, event_start_time, event_stop_time, and node_location. Time-related data in epoch format are transformed into a standard data-time format, and new features, such as Time between Failures (TBF), Time to Return (TTR), fault frequency and event count, are engineered from event_start_time, event_stop_time, and event_type features to enhance predictive and analytical capabilities.

Exploratory Data Analysis (EDA) revealed significant variability in features: ‘node_id’, ‘node_location’ and ‘event_type’ across nine sites and 15 clusters (Figure 3). Features with ‘NULL’ or constant values (e.g., ‘num_procs’, ‘event_end_reason’) were excluded due to their lack of predictive utility. ‘event_type’ is binary in nature: ‘1’ for availability and ‘0’ for unavailability events (Figure 3a). ‘node_location’ showed notable variability, with site8/c1 (cluster1 of site8) having the highest node count and event frequency whereas site6/c2 the lowest (Figure 3b–d). This observation suggests that location-specific failure patterns are potentially influenced by node density or external anomalies such as power outages.

Statistical analysis of TBF and TTR at the system, site, and cluster levels (

Table 1. Site level statistical analysis of Grid5000 failure traces.

Site	#Nodes	#Failures	MTBF	SD(MTBF)	CV(MTBF)	MTTR	SD(MTTR)	CV(MTTR)	Availability	Maintainability
s1	259	69,234	28.77	73.62	2.56	13.36	64.93	4.86	68.29	31.71
s2	95	8161	41.31	109.66	2.65	14.11	72.63	5.15	74.57	25.43
s3	57	5760	102.71	196.58	1.91	5.85	89.97	15.39	94.62	5.38
s4	163	37,470	37.47	89.72	2.39	4.06	31.27	7.70	90.23	9.77
s5	125	7509	79.55	271.63	3.41	36.02	217.04	6.03	68.83	31.17
s6	127	14,115	89.73	207.11	2.31	6.21	74.45	11.98	93.52	6.48
s7	46	21,120	14.57	33.98	2.33	0.96	15.32	15.97	93.82	6.18
s8	342	123,467	22.86	46.82	2.05	4.29	39.18	9.12	84.19	15.81
s9	74	7309	27.62	72.48	2.62	6.34	39.73	6.26	81.32	18.68

and

Table 2. Cluster level statistical analysis of Grid5000 failure traces.

Cluster	#Nodes	#Failures	MTBF	SD(MTBF)	CV(MTBF)	MTTR	SD(MTTR)	CV(MTTR)	Availability	Maintainability
s1/c1	64	13,067	43.27	111.13	2.57	21.15	91.07	4.31	63.12	36.88
s1/c2	64	25,461	22.26	41.09	1.85	10.22	40.75	3.99	74.88	25.12
s1/c3	99	25,954	26.14	62.27	2.38	6.26	46.84	7.48	77.33	22.67
s1/c4	32	4752	38.23	117.71	3.08	47.50	126.78	2.67	34.47	65.53
s2/c1	95	8161	41.37	109.66	2.65	14.11	72.63	5.15	74.57	25.43
s3/c1	57	5760	102.71	196.58	1.91	5.85	89.97	15.39	94.62	5.38
s4/c1	106	34,541	34.40	85.00	2.47	4.31	32.24	7.48	88.87	11.13
s4/c2	57	2929	73.76	127.94	1.73	1.12	15.51	13.90	98.51	1.49
s5/c1	56	5348	101.23	316.88	3.13	29.11	167.98	5.77	77.67	22.33
s5/c2	69	2161	26.47	68.13	2.57	53.12	305.73	5.76	33.26	66.74
s6/c1	103	12,998	94.44	213.31	2.26	5.40	67.86	12.57	94.59	5.41
s6/c2	24	1117	35.93	99.69	2.77	15.69	127.97	8.16	69.61	30.39
s7/c1	46	21,120	14.57	33.98	2.33	0.96	15.32	15.97	93.82	6.18
s8/c1	342	123,467	22.86	46.82	2.05	4.29	39.18	9.12	84.19	15.81
s9/c1	74	7309	27.62	72.48	2.62	6.34	39.73	6.26	81.32	18.68

) further highlights the high variability in the failure patterns. The availability and unavailability were calculated as ${\displaystyle \frac{MTBF}{MTBF+MTTR}}$ and ${\displaystyle \frac{MTTR}{MTBF+MTTR}}$, respectively. System-level analysis of the entire Grid5000 infrastructure revealed a Mean Time between Failures (MTBF) of 32.41 h (SD = 94.24) and Mean Time to Return (MTTR) of 7.41 h (SD = 60.24), with an overall system availability of 81.38%. Similar to the system-level observations, the site-level analysis revealed diverse patterns, with site s3 having the highest MTBF (102.71 h) and availability (94.62%), whereas site s7 had the lowest MTBF (14.57 h) (Table 1). The cluster-level analysis indicated intra-site similarities (Table 2). For instance, c1–c2 of s1 exhibited a similar TBF behaviour. Based on the observed statistical patterns, we hypothesised that cluster-specific failure-prediction models, which benefit from reduced variability, may offer greater accuracy and robustness than site-level models, whereas site-level models are likely to outperform system-level models. These findings guided the selection of features: ‘node_id’, ‘node_location’, ‘event_types’, ‘event_start_time’, ‘event_stop_time’, ‘TBF’ and ‘TTR’ and the model development, evaluation and validation in Phase-2.

4.2. Model Generation

This section addresses Phase-2 of our methodology, as illustrated in Figure 2. It encompasses three components, namely Data Splitting, Model Selection, and Model Training and Testing. All steps were performed in Google Colab Pro with 52 GB RAM and a T4 GPU [23], using Python 3.10 [23]. The setup included NumPy 1.25.2 [24] for efficient array operations and numerical computing, Pandas 2.1.4 [25] for data table management, and Scikit-learn [26] for regression and classification algorithms.

4.2.1. Data Splitting

We applied an 80–20 split to partition the dataset into the training and testing sets. Approximately 80% of the data were allocated for training to enable robust model learning and 20% were reserved for independent evaluation. Stratified sampling was employed to ensure a balanced test set, supporting an unbiased assessment of the performance on unseen data. Additionally, cross-validation was used to enhance the stability of the results and reduce the dependence on a single split. This involved training the models over various subsets and validating the rest to improve the robustness and provide a reliable measure of generalisation.

4.2.2. Model Selection

The selection of models is guided by the nature of each problem statement, specifically whether it is a regression or classification problem, as well as by the unique characteristics of the dataset identified in Section 4.1. The TBF and TTR predictions are approached as regression problems, whereas the FNI is treated as a classification problem. We employed Linear Regression and Logistic Regression exclusively for regression and classification tasks, respectively. Random Forest and XGBoost (Extreme Gradient Boost) were employed for both regression and classification. We chose these models because of their ability to effectively address both types of problems, and manage high-dimensional and variable data [27].

Linear Regression: It is a fundamental supervised ML technique used for predictive modelling where the goal is to predict the continuous outcomes based on the input features. In linear regression, the models is expressed as $z=b+w_1{x}_1+w_2{x}_2+\dots +w_n{x}_n$ where z is the predicted continuous outcome, b is the bias term, and $w_i$ is the weight of the features $x_i$ identified in Section 4.1. Both b and $w_i$ are calculated and fine-tuned during the training phase by minimising the difference between the predicted values and the target values, usually through least squares optimization [28]. This technique is widely applied to failure-prediction scenarios, such as TBF and TTR [29,30].

Logistic Regression: It is a foundational supervised ML technique primarily used for classification tasks, especially to predict the probability that an instance belongs to one of two categories, which in our case is for FNI (i.e., whether a node is likely to fail or not) [31]. Essentially, logistic regression is similar to linear regression, as both begin with a linear combination of input features with corresponding weights, including a bias term that is iteratively optimized during training. However, logistic regression applies a sigmoid (logistic) function $y^{\prime }={\displaystyle \frac{1}{1+e^{-z}}}$ to the output of linear regression z, transforming it into a predicted probability $y^{\prime }$ used for classification tasks such as FNI.

Random Forest: It is an ensemble ML technique that enhances decision trees by combining multiple trees to boost prediction accuracy and robustness. Each tree in a Random Forest is trained on a random subset of the data and features, with the final prediction derived by aggregating the outputs of all trees, typically by averaging for regression tasks or majority voting for classification. For regression, the estimate of a new data point x can be expressed as $m_{M,n}(x,{\theta }_1,\dots ,{\theta }_M)={\displaystyle \frac{1}{M}}{\sum }_{j=1}^M{m}_n(x,{\theta }_j)$, where $m_n(x,{\theta }_j)$ is the prediction from j-th tree, M is the total number of trees, and ${\theta }_j$ represents random parameters (e.g., feature subsets chosen at each node) that introduce variability during the tree construction [32]. For classification, each tree casts a vote for a class label (i.e., node is likely to fail), and the Random Forest predicts the class with the most votes. This randomness, combined with the model’s ability to manage high-dimensional data and mitigate overfitting, makes Random Forest a powerful tool for many real applications such as failure-prediction.

XGBoost: This is another ensemble ML technique that constructs trees sequentially, unlike Random Forest that trains independent trees in parallel. Each tree is designed to correct the errors of its predecessors by minimising a loss function such as the mean squared error for regression or log loss for classification. This is achieved through gradient boosting, where trees are added iteratively, guided by the gradient of the loss with respect to previous predictions. For a new data point x, the prediction can be expressed as $f_M\left(x\right)={\sum }_{j=1}^M{h}_j\left(x\right)$, where $h_j\left(x\right)$ is the output of j-th tree and M is the total number of trees [33]. This iterative error-correcting process enables XGBoost to effectively capture rare events such as system failures, which are often imbalanced or buried in noisy, high-dimensional datasets. In addition, XGBoost incorporates regularisation techniques, such as L1 and L2 penalties, to prevent overfitting. These qualities render XGBoost a powerful tool for failure-prediction and reliability engineering.

4.2.3. Model Training and Testing

To build generic models with cross-layer and cross-platform transferability, we began with clusters (instead of nodes) that represent the smallest collective units in Grid5000 systems. Although node-level failure-prediction models can offer higher accuracy, deploying separate models for thousands of nodes is costly and complex, and has no generalisation. In contrast, cluster-level models are more cost-effective, easier to manage, and can achieve accuracy comparable to that of node-level models. In addition, they are better suited for capturing the generic behaviour of systems. Among the nine sites analysed in Table 2, four (s1, s4, s5, and s6) contained multiple clusters, whereas the remaining five operated with single-cluster configurations. From the multi-cluster sites, we selected cluster 1 of site 4 (s4/c1) to build the failure-prediction models. This cluster was chosen based on two key factors: (1) it has the highest number of documented failure events (34,541) and (2) it has the largest compute node count (106) among all clusters in multi-cluster sites. The models were then evaluated using a comprehensive set of metrics. For regression tasks, we used the Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and R-squared ($R^2$); for the classification models, we used the accuracy. Additionally, we performed a comprehensive feature importance analysis for each model to identify the most significant features and quantify their relative contributions to the predictions of TBF, TTR and FNI. This information can be used to recalibrate the models when they are deployed for production, as discussed in Section 6. To ensure reproducibility, the models and data have been made available in a public repository at [34].

TBF Prediction:

Table 3. Time Between Failures (TBF) prediction results for Cluster 1 of Site 4.

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
2,907,292.09	3,550,861	0.93	1,391,069.55	1,979,144.19	0.98	528,141.4	764,059.13	1	Training
2,909,707.78	3,549,290.19	0.93	1,425,669.2	2,008,907.7	0.98	547,655.48	797,043.5	1	Testing

presents the prediction results for TBF representing the occurrence of the next failure. The analysis revealed that XGBoost was the superior model, demonstrating the best performance with the lowest MAE of 528,141.4, and the highest $R^2$ score of 1 for both the training and testing datasets. The calculated $R^2$ value indicates that the model explains 100% of the variance in the TBF, suggesting a failure-prediction accuracy of 100%, which represents an exceptional level of performance compared with the results seen in related works. This strength is underpinned by XGBoost’s balanced feature importance profile (Figure 4c), where total_event_duration (32.2%), event_id (26.7%), avg_event_duration (15.4%) and event_count (13.2%) emerge as the most influential features. This highlights XGBoost’s ability to leverage both default and engineered features drawn from the Grid5000 dataset. The XGBoost model was configured with key hyper-parameters, including n_estimators=100, max_depth=5, learning_rate=0.1, and random_state=42, optimized using grid search to minimise the root mean squared error (RMSE) (

Table 4. Hyperparameters for Random Forest and XGBoost models for Time between Failures (TBF) prediction for Cluster 1 of Site 4.

Hyperparameter	Random Forest	XGBoost
Number of Trees (n_estimators)	10	100
Maximum Depth (max_depth)	5	5
Random Seed (random_state)	42	42
Feature Subsampling (max_features)	1.0	Not tuned (default: 1.0)
Minimum Samples per Split (min_samples_split)	2	Not tuned (default: 1)
Minimum Samples per Leaf (min_samples_leaf)	1	Not tuned (default: 1)
Learning Rate (learning_rate)	NA	0.1
Bootstrap Sampling (bootstrap)	True	Not tuned (default: 1.0)

). Random Forest showed moderate performance, with a TBF prediction accuracy of 98%, driven by a focus on features such as event_id (57.3%), total_event_duration (17.3%), and avg_event_duration (11.4%), indicating its dependence on event-based features (Figure 4b). For Random Forest, the hyper-parameters were set to n_estimators=10, max_depth=5, random_state=42, max_features=1.0, min_samples_split=2, and min_samples_leaf=1, also tuned using grid search. Linear Regression, the weakest model with a 93% accuracy, heavily depended on event_count (97.6%) and was negatively affected by total_event_duration (−0.4%) and fault_frequency (−100%) (Figure 4a). This suggests that the linear regression–based model did not leverage the available features effectively, which in turn limited its predictive accuracy. The model produced a learned intercept of $1.1439\times {10}^9$ and 12 coefficients with magnitudes ranging from $-4.125\times {10}^8$ to $4.026\times {10}^8$.

TTR Prediction:

Table 5. Time to Return/Repair (TTR) prediction results for Cluster 1 of Site 4.

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
82,799.85	241,987.31	0.06	25,152.81	85,626.94	0.88	72,240.72	180,713.41	0.48	Training
80,839.64	173,716.85	0.09	63,601.27	168,353.66	0.15	71,728.74	161,147.80	0.22	Testing

presents the prediction results for TTR, representing the duration of the next failure. Random Forest initially demonstrated strong performance during training, achieving a low MAE of 25,152.81 and $R^2$ f 0.88 with hyper-parameters set to n_estimators=5, max_depth=None, and random_state=42 (

Table 6. Hyperparameters for Random Forest, and XGBoost models for Time to Return/Repair (TTR) prediction for Cluster 1 of Site 4.

Hyperparameter	Random Forest	XGBoost
Number of Trees (n_estimators)	5	5
Maximum Depth (max_depth)	unlimited	unlimited
Random Seed (random_state)	42	42
Feature Subsampling (max_features)	1.0	Not tuned (default: 1.0)
Minimum Samples per Split (min_samples_split)	2	Not tuned (default: 1)
Minimum Samples per Leaf (min_samples_leaf)	1	Not tuned (default: 1)
Learning Rate (learning_rate)	NA	0.3
Bootstrap Sampling (bootstrap)	True	Not tuned (default: 1.0)

). However, it experienced a significant performance drop owing to overfitting, as reflected in the testing phase results (MAE: 63601.27, $R^2$: 0.15). In contrast, XGBoost showed moderate but consistent performance across both the training and testing phases, with $R^2$ values of 0.48 and 0.22, respectively, indicating that it explained 48% and 22% of the variance in TTR during these phases. The model was configured with n_estimators=5, max_depth=None, and learning_rate=0.3. Linear Regression performed poorly, with an $R^2$ of 0.06 during testing and a learned intercept of 71,594.74, and 13 coefficients (ranging from $-2.372\times {10}^{-9}$ to $1.239\times {10}^5$), indicating a minimal ability to explain the variability in failure duration. This poor performance corresponds with the feature importance results as well, where the model relies heavily on attributes related to time_to_repair and time_between_failures (1.000 and 0.621, respectively). Features like event_start_seconds (−0.012) and event_stop_seconds (0.012) have little impact, suggesting a linear focus on repair metrics that may oversimplify the data (Figure 5a). In contrast, Random Forest and XGBoost both rank time_to_repair (0.400) and time_between_failures (0.343) as their top features, and have negligible contributions from event_stop_seconds (0.094) and other features (Figure 5b,c). While none of the models achieved high prediction accuracy, XGBoost emerged as the most reliable method for predicting TTR owing to its consistency compared with Random Forest and Linear Regression.

FNI Prediction:

Table 7. Failing Node Identification (FNI) for Cluster 1 of Site 4 (s4/c1).

Logistic Regression	Random Forest	XGBoost
2.73%	97%	99%

presents the prediction results for FNI, representing the next node that is likely to fail. A significant contrast in the performance was observed among the three models. XGBoost, configured with n_estimators=100, learning_rate=0.3, and objective=multi:softprob (

Table 8. Hyperparameters for Logistic Regression, Random Forest, and XGBoost models for Failure Node Identification (FNI) for Cluster 1 of Site 4.

Hyperparameter	Logistic Regression	Random Forest	XGBoost
Number of Trees (n_estimators)	NA	10	100
Maximum Depth (max_depth)	unlimi ted	unlimited	unlimited
Random Seed (random_state)	42	42	42
Feature Subsampling (max_features)	NA	max_features: ‘sqrt’	1.0
Minimum Samples per Split (min_samples_split)	NA	2	1
Minimum Samples per Leaf (min_samples_leaf)	NA	1	1
Learning Rate (learning_rate)	NA	NA	0.3
Bootstrap Sampling (bootstrap)	NA	True	1.0
Objective	penalty: ‘l2’	criterion: ‘gini’	objective: ‘multi:softprob’
Regularization Strength (C)	1.0	NA	NA

), emerged as a superior classifier with an exceptional accuracy of 99%, followed by a Random Forest with an accuracy of 97%. In contrast, the Logistic Regression showed surprisingly poor performance, with an accuracy of 2.73%. This performance disparity aligns with the feature importance results. XGBoost highlights total_event_duration (0.322) and event_id (0.267) as its top predictors, supported by avg_event_duration (0.154) and event_count (0.132), reflecting its strong predictive ability (Figure 6c). Random Forest set with n_estimators=10, bootstrap sampling, and criterion:‘gini’, distributes the importance more evenly across avg_event_duration (0.238), event_count (0.217), and fault_frequency (0.214), showing its ensemble strength (Figure 6b). However, the Logistic Regression using L2 regularization (penalty:‘l2’) with regularization strength, C of 1.0 and random_state=42 relies heavily on total_event_duration (1.000) with negative contributions from event_start_time (−0.655) and event_stop_seconds (−0.163) (Figure 6a). This indicate its linear focus that likely explains its lower accuracy due to misalignment with the dataset’s complex patterns.

In summary, the outstanding performance of XGBoost in predicting TBF and FNI, along with its moderate reliability for TTR, highlights the potential of ensemble methods for failure-prediction in cross-layer and cross-platform DC systems. These findings are validated and discussed in the following section.

5. Model Validation and Discussion

In this section, the models trained in Section 4.2.3 were rigorously tested across various datacenters (represented as sites in the Grid5000 system) to assess their generalisability and robustness. Validation includes both intra- and inter-platform approaches. Intra-platform validation covers all clusters associated with Site 4, excluding Cluster 1, as well as the results for Site 4 itself. The inter-platform covers all clusters and sites, with the exception of Site 4. To validate the model’s generalisability, their performance was compared against the specialized baseline models trained and tested on inter or intra-platform domains. By using this approach, we performed a comprehensive evaluation of the proposed models, demonstrating that they achieves near-comparable prediction accuracy to baseline models without requiring domain-specific retraining.

5.1. Intra-Platform Validations

This section concentrates on Site 4, in which Cluster 1 resources are used to build candidate generic models. Validations were performed at the cluster level (within-layer) using Cluster 2 and at the site level (cross-layer) across the entire Site 4. This approach enables a comprehensive evaluation of model performance and generalisability within and across datacentre layers.

5.1.1. Within-Layer Prediction Results

At the intra-platform within-layer level (i.e., Cluster 2 of Site 4), the models from Section 4.2.3 demonstrated performance patterns consistent with those observed in their system of origin (i.e., Cluster 1 of Site 4). The XGBoost-based model delivered the highest prediction accuracy, achieving an $R^2$ score of 0.99, matching the specialized baseline model trained and tested on Cluster 2 of Site 4 (

Table 9. Time Between Failures (TBF) prediction for Cluster 2 of Site 4 (s4/c2).

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
1,516,222.63	1,878,392.72	0.8	711,677.83	938,202.09	0.95	249,227.43	340,042.93	0.99	Baseline
1,503,313.88	1,863,247.75	0.8	748,930.95	1,000,889.96	0.94	311,125.64	427,993.66	0.99	Generic

). Random Forest and Linear Regression-based models yielded $R^2$ values of 0.94 and 0.80, respectively for generic models, which is almost identical to baseline values of 0.95 and 0.80, respectively. This highlights the superior reliability of XGBoost for TBF predictions. For TTR prediction (

Table 10. Time to Return/Repair (TTR) prediction for Cluster 2 of Site 4 (s4/c2).

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
168,434.33	320,717.22	0.16	54,213.23	143,246.53	0.83	136,879.06	274,331.86	0.39	Baseline
170,774.77	335,822.34	0.16	135,691.16	329,574.96	0.19	149,090.82	313,381.94	0.27	Generic

), all models struggled, with Linear Regression and Random Forest producing low $R^2$ values of 0.16 and 0.19, respectively for generic model, compared to 0.16 and 0.83 for the baseline model. However, XGBoost reached a modest 0.27 for generic model in comparison to 0.39 for baseline model. This highlights the inherent difficulty in forecasting repair times. For FNI, XGBoost and Random Forest achieved accuracies of 90% and 89%, respectively, for the generic model, which was nearly equivalent to 90% and 91% for the baseline model (

Table 11. Failing Node Identification (FNI) for Cluster 2 of Site 4 (s4/c2).

Logistic Regression	Random Forest	XGBoost
5.76%	91%	90%	Baseline
4.01%	89%	90%	Generic

). These prediction results highlight the ability of models from Cluster 1 of Site 4 to perform comparably to specialized baseline models, demonstrating strong generalisation and transferability.

5.1.2. Cross-Layer Prediction Results

After validating the models for individual clusters of site 4, this section presents the cross-layer validation results, evaluating their performance and generalisability across the entirety of Site 4. Consistent with cluster-level trends, the generic XGBoost-based model delivered the strongest results for TBF prediction, with a perfect $R^2$ of 1, matching the specialized baseline model trained and tested on Site 4 (

Table 12. Time Between Failures (TBF) Prediction for Site 4 (s4).

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
2,929,578.49	3,639,185.1	0.93	1,504,051.79	2,236,507.97	0.97	587,965.81	843,305.69	1	Baseline
2,970,018.64	3,677,263.41	0.93	1,492,496.03	2,197,940.25	0.97	608,063.14	876,199.7	1	Generic

). Random Forest and Linear Regression produced $R^2$ values of 0.97 and 0.93 for the generic model, respectively, which are identical to their baseline values of 0.97 and 0.93, respectively. For TTR prediction, all models struggled, with XGBoost yielding the negative $R^2$ (−0.04) for the generic model, compared to 0.44 for baseline. However, Random Forest and Linear Regression produced $R^2$ values of −1.21 and 0.10 (generic) versus 0.86 and 0.07 (baseline), respectively, reflecting the challenge of modelling repair times at the site level as well (

Table 13. Time to Return/Repair (TTR) prediction for Site 4 (s4).

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
90,910.18	250,519.23	0.07	27,631.40	96,427.20	0.86	79,319.45	194,747.05	0.44	Baseline
89,362.67	190,010.69	0.10	71,980.13	297,582.00	−1.21	80,320.44	203,852.45	−0.04	Generic

). For FNI, XGBoost achieved 98% accuracy for both generic and baseline models, with Random Forest at 96% (generic) vs. 97% (baseline), whereas Logistic Regression trailed at 1.59% (generic) vs. 2.83% (baseline) (

Table 14. Failing Node Identification (FNI) for Site 4.

Logistic Regression	Random Forest	XGBoost
2.83%	97%	98%	Baseline
1.59%	96%	98%	Baseline

). These prediction results, particularly for TBF and FNI, shows that the model from Cluster 1 of Site 4 performed comparably to specialized models on Site 4, supporting their effectiveness and generalisability across datacentre layers.

These within- and cross-layer validation results for Site 4 confirm that the XGBoost- and Random Forest-based models, trained on Cluster 1 of Site 4, offer strong generalisability for the failure prediction by performing comparably to specialized baseline models. These models function as robust and generic tools that can be deployed across various layers of the resource hierarchy of datacenters without requiring domain-specific retraining. The next section evaluates the validity of these models across different datacenters.

5.2. Inter-Platform Validations

This section presents the inter-platform (cross-platform) validation results starting with cluster-level (within-layer) validation, followed by site-level (cross-layer) validation. The within-layer analysis covered all clusters except Clusters 1 and 2 of Site 4, while the cross-layer analysis included all sites except Site 4.

5.2.1. Within-Layer Prediction Results

Applying models to clusters, such as s1/c1–c4, s5/c1–c2, and s6/c1–c2, showed varying performances, highlighting the challenges of their inter-platform applicability. For TBF prediction (

Table 15. Time Between Failures (TBF) prediction for clusters.

Cluster	Linear Regression			Random Forest			XGBoost
	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
s1/c1	2,199,158.49	3,268,040.41	0.88	1,324,695.21	2,316,888.16	0.94	347,184.94	559,061.22	1	Baseline
	2,189,153.88	3,218,301.35	0.88	1,348,533.01	2,346,680.86	0.94	379,420.4	632,303.97	1	Generic
s1/c2	2,659,912.14	3,273,118.61	0.95	1,749,929.32	2,314,754.93	0.98	596,981.65	791,038.14	1	Baseline
	2,680,141.68	3,298,153.24	0.95	1,739,446.02	2,300,283.33	0.98	605,162.2	804,285.57	1	Generic
s1/c3	3,233,986.4	4,122,216.75	0.77	970,806.41	1,500,475.02	0.97	459,026.57	709,658.3	0.99	Baseline
	3,262,003.85	4,140,497.15	0.77	995,052.43	1,552,075.99	0.97	476,904.74	742,083.37	0.99	Generic
s1/c4	5,405,297.07	7,121,341.45	0.68	1,230,335.78	1,693,237.59	0.98	299,407.66	438,295.97	1	Baseline
	5,565,339.12	7,293,853.22	0.68	1,293,780.17	1,769,684.09	0.98	368,256.01	587,340.22	1	Generic
s5/c1	3,908,274.31	6,371,524.43	0.61	1,463,884.23	2,141,642.88	0.96	500,899.21	759,944.39	0.99	Baseline
	4,048,997.78	6,616,939.18	0.59	1,524,337.44	2,287,389.78	0.95	596,031.88	1,017,532.05	0.99	Generic
s5/c2	723,416.46	1,077,801.67	0.62	259,957.17	418,968.67	0.94	78,197.26	118,401.72	1	Baseline
	796,535.8	1,218,561.45	0.6	272,547.21	437,900.86	0.95	104,169.1	180,896.82	0.999	Generic
s6/c1	3,153,144.41	4,279,153.41	0.8	1,501,901.58	2,182,402.99	0.95	780,128.98	1,180,636.25	0.99	Baseline
	3,081,761.98	4,162,700.42	0.81	1,540,490.79	2,264,878.39	0.94	833,206.58	1,284,525.99	0.98	Generic
s6/c2	1,298,294.51	1,539,855.13	0.72	312,892.46	569,396.19	0.96	102,592.13	194,585.32	1	Baseline
	1,334,536.26	1,563,239.86	0.72	337,260.72	604,949.25	0.96	202,532.96	403,870.33	0.98	Generic

), the XGBoost-based model maintained high accuracy, with $R^2$ scores ranging from 0.98 to 1.0, closely matching the specialized baseline models trained and tested on each cluster and confirming its reliability across different clusters. Random Forest scored $R^2$ from 0.94 to 0.98, and Linear Regression varied from 0.59 to 0.95, both performing comparably to their respective baselines. TTR prediction (

Table 16. Time to Return/Repair (TTR) prediction for clusters.

Cluster	Linear Regression			Random Forest			XGBoost
	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
s1/c1	152,763.12	349,213.89	0.07	15,694.34	77,177.63	0.95	103,916.68	215,901.53	0.64	Baseline
	157,409.89	381,882.06	0.06	43,177.68	182,658.91	0.78	109,583.82	240,805.48	0.63	Generic
s1/c2	69,610.00	147,501.28	0.03	17,886.32	51,671.62	0.88	61,943.50	127,711.20	0.27	Baseline
	69,486.31	144,658.84	0.03	47,007.45	108,098.98	0.46	63,099.48	126,937.25	0.25	Generic
s1/c3	82,588.79	193,336.03	0.07	17,144.05	68,314.25	0.88	61,183.82	155,740.73	0.40	Baseline
	82,699.51	199,264.32	0.06	44,046.25	156,888.36	0.42	61,663.23	167,280.48	0.34	Generic
s1/c4	214,730.28	446,047.88	0.03	29,888.69	117,178.39	0.93	152,044.45	310,917.54	0.53	Baseline
	207,948.02	378,717.08	0.04	851,54.97	283,222.59	0.46	156,238.60	295,533.46	0.41	Generic
s5/c1	336,979.00	867,680.11	0.07	71,563.04	278,300.57	0.90	192,955.42	517,171.39	0.67	Baseline
	357,527.44	977,364.90	0.07	218,493.89	769,349.08	0.42	240,967.19	769,882.47	0.42	Generic
s5/c2	219,572.57	640,135.30	0.38	23,718.38	90,259.35	0.99	78,866.88	201,647.32	0.94	Baseline
	22,6461.26	663,247.17	0.13	67,366.99	299,789.34	0.82	86,299.34	284,238.21	0.84	Generic
s6/c1	255,238.28	558,095.17	0.10	63,722.97	222,363.16	0.86	179,200.94	432,513.96	0.46	Baseline
	252,873.31	581,151.54	0.09	167,258.92	521,851.38	0.27	187,279.23	505,451.88	0.31	Generic
s6/c2	159,384.58	419,298.69	0.04	52,639.95	187,566.56	0.81	102,106.25	286,142.70	0.55	Baseline
	151,952.16	346,023.51	0.03	146,792.12	412,390.40	−0.38	121,590.90	322,413.14	0.16	Generic

) proved difficult, with Linear Regression producing consistently low $R^2$ values (0.03 to 0.13) and Random Forest achieving moderate results ($R^2$ between 0.27 and 0.82). In relative terms, XGBoost achieved the highest TTR prediction accuracy with $R^2$ as high as 0.84, though generally trailing baseline models. This mirrors the pattern from the development and intra-platform tests, highlighting the challenging nature of TTR prediction. For FNI (

Table 17. Failing Node Identification (FNI) for clusters.

Cluster	Logistic Regression	Random Forest	XGBoost
s1/c1	2.01%	70%	72%	Baseline
	1.70%	68%	71%	Generic
s1/c2	3.78%	100%	100%	Baseline
	2.76%	100%	100%	Generic
s1/c3	4.52%	89%	92%	Baseline
	2.16%	86%	91%	Generic
s1/c4	9.63%	84%	87%	Baseline
	9.47%	83%	85%	Generic
s5/c1	7.11%	96%	97%	Baseline
	6.37%	96%	97%	Generic
s5/c2	6.47%	99%	100%	Baseline
	3.55%	98%	100%	Generic
s6/c1	3.57%	71%	77%	Baseline
	2.19%	66%	76%	Generic
s6/c2	8.29%	100%	100%	Baseline
	6.43%	100%	100%	Generic

), XGBoost and Random Forest achieved 71% to 100% and 66% to 100% accuracy, respectively, performing comparably to baseline models across most clusters. Overall, these results suggest that the models trained and tested on Cluster 1 of Site 4, specifically the XGBoost-based model, are sufficiently generalisable across platforms, without retraining.

5.2.2. Cross-Layer Prediction Results

This section presents the cross-layer site-level validation results for all sites except Site 4. Consistent with previous findings, the XGBoost-based model, trained on Cluster 1 of Site 4, achieved the best results for TBF prediction (

Table 18. Time between Failures (TBF) prediction for sites.

Site	Linear Regression			Random Forest			XGBoost
	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
s1	3,594,633.79	4,804,189.99	0.89	2,610,815.28	4,170,278.14	0.92	902,671.41	1,344,152.73	0.99	Baseline
	3,603,998.9	4,820,869.4	0.89	2,633,029.66	4,183,762.66	0.92	922,609.09	1,375,960.08	0.99	Generic
s2	1,875,998.15	2,654,180.88	0.89	1,298,825.62	1,832,759	0.95	388,923.52	542,769.17	1	Baseline
	1,864,392.5	2,632,621.32	0.89	1,297,918.93	1,862,982.86	0.94	435,750.29	623,398.86	0.99	Generic
s3	2,243,590.35	3,359,843.84	0.88	1,474,401.75	2,171,123.02	0.95	579,448.46	781,626.05	0.99	Baseline
	2,308,853.7	3,520,793.58	0.87	1,589,971.91	246,9831.48	0.94	667,541.89	936,021.59	0.99	Generic
s5	3,371,720.22	5,644,021.54	0.64	1,629,967.04	2,527,218.54	0.93	468,405.11	767,984.11	0.99	Baseline
	3,320,485.51	5,528,168.72	0.65	1,713,441.11	2,600,937.34	0.92	523,765.47	891,070.42	0.99	Generic
s6	3,015,870.24	4,134,077.2	0.82	1,489,560.71	2,233,483.46	0.95	788,098.41	1,201,042.1	0.98	Baseline
	3,013,126.08	4,107,923.41	0.81	1,486,652.87	2,231,897.67	0.94	846,030.15	1,297,938.08	0.98	Generic
s7	3,812,223.14	4,685,561.77	0.59	1,355,112.79	2,109,539.43	0.92	468,925.3	742,990.44	0.99	Baseline
	3,769,956.96	4,637,465.53	0.6	1,359,839.76	2,114,679.41	0.92	480,980.74	761,642.39	0.99	Generic
s8	2,934,946.18	3,850,773.55	0.93	2,717,223.55	3,712,231.89	0.94	1,192,225.37	1,733,187.45	0.99	Baseline
	2,942,925.57	3,865,924.17	0.93	2,738,509.14	3,748,577.23	0.93	1,207,040.17	1,751,338.91	0.99	Generic
s9	1,568,903.71	2,132,447.66	0.38	509,430.12	794,061.51	0.91	221,646.51	338,657.31	0.98	Baseline
	1,537,169.02	2,107,557.67	0.39	521,487.09	807,399.1	0.91	247,652.1	384,438.38	0.98	Generic

), maintaining $R^2$ scores of 0.98 to 0.99. Notably, a value of 0.99 was observed for sites 1–3, 5, 7, and 8, closely matching the performance of specialized baseline models trained and tested on each site. Random Forest delivered reliable $R^2$ values ranging from 0.91 to 0.94. These results align with cluster-level findings and confirm the strong cross-layer generalisability. For TTR prediction (

Table 19. Time to Return/Repair (TTR) prediction for sites.

Site	Linear Regression			Random Forest			XGBoost
	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
s1	98,084.42	243,160.00	0.03	19,472.36	74,867.44	0.91	86,326.38	204,099.35	0.32	Baseline
	99,591.32	261,988.05	0.03	50,707.36	164,248.80	0.62	87,906.56	218,229.07	0.33	Generic
s2	129,295.72	344,254.98	0.04	40,150.03	145,700.24	0.83	97,977.73	223,790.45	0.59	Baseline
	123,680.56	277,375.12	0.04	99,671.44	246,767.06	0.24	101,645.88	266,755.79	0.11	Generic
s3	256,350.49	516,752.08	0.20	80,455.05	233,111.39	0.84	190,387.69	414,668.29	0.48	Baseline
	269,241.48	544,391.12	0.13	208,308.36	496,903.93	0.27	210,672.04	476,608.63	0.33	Generic
s5	313,675.73	855,227.09	0.12	59,524.0	257,933.16	0.92	184,782.26	530,638.40	0.66	Baseline
	295,104.21	730,672.56	0.14	148,396.99	498,143.30	0.60	184,808.14	490,943.98	0.61	Generic
s6	244,538.51	539,270.31	0.09	62333.38	214,946.62	0.86	174,018.99	425,989.95	0.43	Baseline
	254,923.49	607,804.99	0.08	170,037.90	539,915.14	0.27	191,738.13	534,703.41	0.29	Generic
s7	42,818.57	95,503.49	0.08	11,834.98	44,314.83	0.80	32,363.12	85,351.86	0.27	Baseline
	40,743.34	85,885.05	0.10	27,689.87	85,674.92	0.10	31,179.39	79,177.38	0.23	Generic
s8	59,907.70	154,166.46	0.06	20,572.11	69,455.39	0.81	56,645.48	144,189.60	0.18	Baseline
	59,604.74	151,910.12	0.07	52,183.00	147,429.36	0.12	56,475.81	143,090.13	0.17	Generic
s9	88,555.11	208,842.04	0.08	23,839.88	82,151.00	0.86	62,367.80	154,173.24	0.50	Baseline
	84,642.17	193,272.17	0.07	60,390.07	174,247.89	0.24	64,478.85	163,684.96	0.33	Generic

), challenge persisted, with XGBoost reaching a peak $R^2$ of 0.61 on Site 5, which is approximately 26% lower than the corresponding value for Cluster 2 of the same site, and generally lower than the values of the baseline models. Random Forest underperformed with $R^2$ values ranging from −0.38 to 0.68, and Linear Regression yielded the lowest performance. This reflects the ongoing difficulty of repair-time prediction but highlights the relative strength of XGBoost. For FNI (

Table 20. Failing Node Identification (FNI) for sites.

Site	Logistic Regression	Random Forest	XGBoost
s1	2.15%	89%	74%	Baseline
	1.43%	88%	71%	Generic
s2	4.31%	99%	100%	Baseline
	3.79%	99%	99%	Generic
s3	5.21%	100%	100%	Baseline
	4.24%	100%	100%	Generic
s5	5.89%	98%	99%	Baseline
	5.56%	97%	98%	Generic
s6	3.16%	72%	82%	Baseline
	2.19%	69%	77%	Generic
s7	6.14%	96%	98%	Baseline
	5.79%	95%	97%	Generic
s8	1.48%	99%	100%	Baseline
	0.78%	99%	99%	Generic
s9	6.23%	100%	100%	Baseline
	4.06%	100%	100%	Generic

), the XGBoost-based model achieved 71% to 100% accuracy and Random Forest achieved 69% to 100%, showing almost similar performance to baseline models across most sites, mirroring the cluster-level trends and demonstrating robust performance across layers.

These cross-layer validation results across all sites, except Site 4, further substantiate the generalisability of the XGBoost and Random Forest-based models, trained on Cluster 1 of Site 4, for failure-prediction tasks. XGBoost consistently delivered robust and reliable performance, particularly for TBF and FNI predictions, where it closely matched its baseline model, while Random Forest also showed strong performance relative to its baseline model. The complexity of TTR prediction remained evident at both cluster and site levels. Overall, these findings confirm the suitability of XGBoost and Random Forest as effective generic models capable of sustaining accurate failure prediction across diverse DC layers and environments.

5.3. System Level Validations

The conclusions drawn from Section 5.1 on intra-platform validations, together with those from Section 5.2 on inter-platform validations, provide strong evidence of the model’s generalisability. Building on this confidence, this section evaluates their performance at the meta-datacentre (system) level across the entire Grid5000 system.

Table 21. Time between Failures (TBF) prediction for the system.

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$	MAE	RMSE	${\mathit{R}}^{\mathbf{2}}$
4,919,256.24	6,540,568.72	0.78	377,005.23	512,468.65	0.87	1,546,876.4	2,217,474.9	0.97	Baseline
4,917,627.14	6,556,851.1	0.78	3,764,432.5	5,113,083.8	0.87	1,554,655.7	1,554,655.7	0.97	Generic

shows that the XGBoost-based model, trained on Cluster 1 of Site 4, achieved the highest TBF prediction accuracy with an $R^2$ score of 0.97, aligning precisely with the specialized baseline model trained and tested on the entire system, with only a 2% drop from the lower levels in the system hierarchy, confirming its robust generalisability without retraining. Random Forest also showed reasonable performance and achieved an $R^2$ of 0.87, a 10% decline from lower levels. In contrast,

Table 22. Time to Return/Repair (TTR) prediction for the system.

Linear Regression			Random Forest			XGBoost
MAE	RMSE	${\mathit{R}}^2$	MAE	RMSE	${\mathit{R}}^2$	MAE	RMSE	${\mathit{R}}^2$
93,686.57	277,284.75	0.08	26,289.11	103,791.21	0.87	84,889.04	248,664.81	0.26	Baseline
94,398.16	274,163.97	0.07	67,538.24	224,107.79	0.39	86,020.85	251,870.95	0.22	Generic

reveals ongoing TTR prediction challenges, with XGBoost at 0.22 $R^2$ and Random Forest at 0.39, both lagging behind their baseline equivalents, indicating persistent difficulty across all models. For FNI (

Table 23. Failing Node Identification (FNI) for the system.

Logistic Regression	Random Forest	XGBoost
1.15%	4.03%	95%	Baseline
0.70%	2.95%	94.28%	Generic

), XGBoost achieved the highest accuracy of 94.28%, nearly identical to the baseline’s 95% with only a modest 5% drop compared with the lower levels, further reinforcing the evidence of its strong generalisability. However, the performance of Random Forest-based model dropped sharply to 2.95%, compared to the baseline’s 4.03%, representing an approximately 80% decline compared to the lower levels.

These intra-platform, inter-platform, and system-level validation results for the models developed in Cluster 1 of Site 4 demonstrate their strong generalisability and robustness by performing similarly to specialized baseline models trained and tested on each respective level. XGBoost consistently delivered high TBF and FNI prediction accuracies across the DC system layers, with only slight performance drops at higher levels. The Random Forest method performed well, except at the system level. TTR prediction remains a challenge across all levels. Overall, the XGBoost-based model proved to be a reliable, generic solution for failure prediction in diverse DC setups, and Random Forest suits specific layers. Their accuracy across levels without retraining supports cost-effective and proactive resource management in DC systems.

6. Proposed System Architecture

This section introduces the conceptual architecture of a reliable DC system that incorporates the failure-prediction models proposed in Section 5. In addition, a practical case is presented to show the applicability of the proposed architecture in a real-world scenario.

6.1. Reliable Distributed Computing System Architecture: A Conceptual Model

Figure 7 illustrates the hierarchical architecture of a DC system designed to integrate failure-prediction modules across multiple levels (system, site, and cluster) to enhance the reliability and optimise resource utilisation. The architecture comprises compute nodes organised into clusters that are grouped into geographically distributed sites, forming the overall DC system. Each level is equipped with dedicated failure-prediction modules (${\mathrm{PM}}_{\mathrm{CT}}$ for clusters, ${\mathrm{PM}}_{ST}$ for sites, and ${\mathrm{PM}}_{SS}$ for the system). These modules leverage the ML models from Section 5 to predict the key metrics TBF, TTR, and FNI. These modules continuously perform predictions by using the monitoring data collected in the corresponding data repositories (${\mathrm{DR}}_{\mathrm{CT}}$ for clusters, ${\mathrm{DR}}_{ST}$ for sites, and ${\mathrm{DR}}_{SS}$ for the system). These predictions are fed to the corresponding resource managers (${\mathrm{RM}}_{CT}$, ${\mathrm{RM}}_{ST}$ and ${\mathrm{RM}}_{SS}$). These resource managers (RM) are responsible for failure prevention and mitigation and employ various failure-aware resource provisioning and allocation policies to reliably compute resource assignments to service requests. They also implemented fault-tolerance mechanisms (e.g., check-pointing and migration) to minimise the impact of failures on hosted services and ensure service continuity [35]. Thus, all of these components work in a closed-loop manner, with the prediction modules closing the loop.

6.2. Balancing Reliability and Cost in Service Design

To demonstrate the practical applicability of the proposed architecture, we consider a cloud service provider managing a DC system, structured as shown in Figure 7. By leveraging the results from Section 5, the service provider can tailor services to align with the user’s cost and reliability priorities as follows:

• Cost-Optimized Service (No Explicit Reliability Requirements): For a client submitted a data analytics workload with strict cost constraints but no reliability requirements, the service provider can design a service by using system-level resource manager, $R{M}_{SS}$. It employs predictions from $P{M}_{SS}$ module, which achieves an $R^2$ of 0.97 for TBF with XGBoost (Table 21), enabling system-wide resource allocation with 81.38% system availability (Section 4.1) at a reduced computational cost. This balances affordability and baseline reliability of the service.
• High-Reliability Service (No Budget Constraints): For a client submitted a mission-critical medical imaging application demanding maximal uptime with no cost constraints, the service will be designed by using cluster-level managers ($R{M}_{CT}$) using predictions from $P{M}_{CT}$ modules, offering 100% FNI accuracy and an $R^2$ of 1.0 for TBF in clusters like s1/c2 (Table 9 and Table 11). This ensures a higher availability of 98.51% (Table 2) and a minimal downtime.
• Balanced Cost-Reliability Service: For users seeking a balance between reliability and cost, the service will be designed by using site-level resource managers ($R{M}_{ST}$) utilizing $P{M}_{ST}$ predictions, such as an $R^2$ of 0.99 TBF in site 4 (Table 21) with availability of 90.23% (Table 1). This strategy strategically allocates resources across sites to balance the cost and reliability of the service.

By integrating the hierarchical DC architecture with generalisable failure-prediction models, providers can dynamically address diverse user needs by maximising cost efficiency, reliability, or a hybrid of both while optimising infrastructure utilisation. However, the integration of ML models may influence failure patterns over time by adapting system operations, potentially requiring service providers to perform periodic model adjustments to maintain the predictive accuracy. This ongoing adaptability ensures the architecture remains responsive to changing DC conditions, enhancing its long-term effectiveness across diverse service requirements.

7. Conclusions and Future Work

This study presents a robust framework for developing generic machine-learning-based failure-prediction models tailored for large-scale distributed computing (DC) systems, addressing the need for cross-layer and cross-platform transferability. By leveraging the Grid5000 failure dataset, we designed and evaluated models based on Logistic Regression, Random Forest, and XGBoost to predict Time Between Failures (TBF), Time to Return/Repair (TTR), and Failing Node Identification (FNI). Our findings demonstrate that XGBoost consistently outperforms other models, achieving near-perfect TBF prediction ($R^2$ up to 1.0) and high FNI accuracy (up to 100%) across the cluster, site, and system levels, with minimal performance drops at higher hierarchical levels. Random Forest also showed strong performance, particularly for TBF and FNI, while TTR prediction remained challenging across all models owing to the inherent complexity of repair time estimation. The proposed models showed strong generalisability, enabling their application across diverse DC platforms without retraining, thereby reducing the computational costs and management overhead. Additionally, we proposed a hierarchical DC system architecture that integrates failure-prediction modules, supported by its applicability in balancing the cost and reliability for diverse service requirements. This research lays the foundation for proactive, cost-effective, and reliable resource management in modern DC systems.

While generic models are cost-effective and easier to manage, performing deep analysis to quantify their practical benefits (e.g., computational cost and training time) based on the characteristics of available data remains an open issue to be addressed in our future work. Based on our observations, we predict that XGBoost, which delivers top performance (e.g., an $R^2$ of 1.0 for TBF), likely requires more resources owing to its ensemble nature; Linear Regression, with lower accuracy (e.g., 93% FNI), probably uses fewer resources; and Random Forest, with balanced performance (e.g., 98% FNI accuracy) may have moderate training costs. In future work, we will propose models to predict/link their exact amounts based on the several characteristics (size, mean, standard-deviation, etc.) of the available data. Additionally, we will explore advanced time-series models, including autoregressive integrated moving average (ARIMA), to enhance the prediction accuracy, specifically for TTR. We will also extend our analysis to datasets from other DC environments, such as Google and Los Alamos National Laboratory (LANL), to provide a more comprehensive evaluation of our proposed models. Furthermore, we aim to simulate and implement the proposed hierarchical DC system architecture with integrated prediction modules and use cases, allowing rigorous performance evaluation under real-time dynamic scenarios.