MAS-DR: An ML-Based Aggregation and Segmentation Framework for Residential Consumption Users to Assist DR Programs

Abstract

The increasing complexity of energy grids, driven by rising demand and unpredictable residential consumption, highlights the need for efficient demand response (DR) strategies and data-driven services. This paper proposes a machine learning-based framework for DR that clusters users based on their consumption patterns and categorizes individual usage into distinct profiles using K-means, Hierarchical Agglomerative Clustering, Spectral Clustering, and DBSCAN. Key features such as statistical, temporal, and behavioral characteristics are extracted, and the novel Household Daily Load (HDL) approach is used to identify residential consumption groups. The framework also includes context analysis to detect daily variations and peak usage periods for individual users. High-impact users, identified by anomalies such as frequent consumption spikes or grid instability risks using IsolationForest and kNN, are flagged. Additionally, a classification service integrates new users into the segmented portfolio. Experiments on real-world datasets demonstrate the framework’s effectiveness in helping energy managers design tailored DR programs.

1. Introduction

In recent decades, a shift towards Renewable Energy Sources (RES), like solar and wind, has been observed. The intermittent and unpredictable nature of RES, due to its dependence on weather conditions, has introduced challenges in energy grid’s management [1]. Furthermore, forecasting residential energy consumption poses additional challenges due to the high variability and fluctuations of the household usage patterns [2,3,4]. Therefore, a need has arisen for effective strategies to counterbalance these aforementioned uncertainties and, thus, enhance grid stability and flexibility. For this purpose, the concept of Demand Response (DR) strategies was birthed. DR strategies enable consumers to adjust their usage during peak periods in response to signals [5,6]. These strategies offer opportunities to not only achieve energy supply and demand balance effectively by aligning demand management strategies to the distinct needs and behaviors of different groups, but also offer significant cost-saving benefits [7]. Based on the applied DR programs, consumers are able to receive personalized incentives that are tailored to their usage profiles [8].

In more recent years, the increase in the usage of smart meters in residential buildings has given a powerful tool in the hands of energy policy makers, regarding the availability of real-time energy consumption and production data [9,10]. In addition, the exponential increase in the usage of Artificial Intelligent (AI), and more specifically Machine Learning, in residential energy management systems enable utilities to process the available data in order to identify trends, predict consumption behaviors, forecast renewable energy units’ production accurately, and manage the storage system effectively [11,12,13,14,15]. Unsupervised machine learning, and more specifically clustering algorithms, can be very effective when used for the optimization of the DR programs. More specifically, it can provide useful insights for the energy consumers, both residential and industrial, while, additionally, grouping consumers with similar load profiles, assisting utility companies in creating more tailored and responsive DR programs that boost both efficiency and participation. Leveraging clustering algorithms such as K-means or Hierarchical Agglomerative Clustering, DR programs can optimize how policy makers engage with different consumer groups [16,17,18,19].

Despite the indisputable benefits of unsupervised learning in energy-related applications, there is a sizable gap in research that needs to be filled to further investigate clustering algorithms’ usefulness [20]. First and foremost, most approaches lack a systematic methodology to determine the most suitable clustering algorithms that align with specific objectives. Moreover, the inherent subjectivity and intuitive interpretation of segmentation results often undermine their reproducibility, emphasizing the critical absence of a universally accepted metric for evaluating cluster quality [21]. Adding to this, the dynamic and high-dimensional nature of energy consumption data demands a robust framework capable of handling real-world complexity while delivering actionable insights. This underscores the necessity for clearly defined clustering services to ensure consistency and reliability in applications. Furthermore, integrating clustering with advanced evaluation metrics could bridge the gap between theoretical models and practical deployment, ultimately optimizing residential energy consumption strategies and supporting tailored DR programs.

Within this context, this paper addresses a holistic approach that is based on a variety of commonly used clustering algorithms and metrics for aggregating and segmenting the consumption of residential users, in order to assist Demand Response (DR) programs with useful insights. The  framework can be separated into two major categories based on the data used. The first category analyzes the data from several different residential users, in order to segment them into groups, based on a wide variety of features, which are carefully selected, derived from their consumption and based on the underused concept of the Household Daily Load (HDL), which will be further presented in following sections. Furthermore, based on similar features, classification techniques are explored to classify new users into already existing groups, an approach that can benefit the decision makers by providing them with implicit information on the consumption patterns of said users without the need of difficult to acquire historical data. In addition, different services in analyzing the consumption patterns of a single residential user are presented. Specifically, a novel approach is presented to identify users suitable for DR strategies by analyzing a user’s daily variation, or by finding their peak consumption in different daily periods. Additionally, by utilizing anomaly detection techniques, among others, a methodology to identify users that could have high impact on DR strategies is proposed. Overall, this paper addresses gaps in systematic clustering methodologies and practical evaluation metrics, advancing the application of machine learning in energy systems introducing the following main novelties:

• Integration of Clustering and Classification for DR Programs: This paper introduces a machine learning-based framework (MAS-DR) that combines clustering and classification techniques to segment residential users and classify new users into pre-existing clusters, optimizing DR strategies and improving their applicability to a dynamic user base.
• Household Daily Load (HDL) Concept: This paper incorporates the novel Household Daily Load (HDL) approach, which analyzes seasonal and monthly consumption patterns. This allows for finer segmentation of user behavior based on temporal variations, supporting tailored interventions such as season-specific DR strategies.
• Feature Selection for Enhanced Clustering: This study applies advanced feature extraction and selection techniques, including Principal Component Analysis (PCA) and iterative clustering evaluation, to ensure that only the most relevant features influence segmentation, enhancing the accuracy and efficiency of clustering.
• Comprehensive Evaluation of Multiple Clustering Algorithms: The framework systematically evaluates several clustering algorithms (e.g., K-means, Hierarchical Clustering, Spectral Clustering, DBSCAN) and metrics (e.g., Silhouette Score, Davies–Bouldin Index, Calinski–Harabasz Index) to identify the optimal methods for segmenting residential energy consumption patterns.

The remainder of the paper is organized as follows. Section 1 offers an introduction on the concept of Demand Response and the value that Artificial Intelligence (AI) and more specifically clustering algorithms offer in building DR strategies. Section 2 offers a comprehensive literature review on the related works that explore clustering models in energy related applications. Section 3 presents in detail the methodology that was followed in the design of the proposed framework, together with the clustering algorithms and the evaluation metrics that were used. In Section 4, the case study that was followed in the demonstration of the framework is presented. Section 5 presents the results of the evaluation of the MAS-DR framework on real-life data. Finally, Section 7 offers a conclusive analysis of the results of the proposed framework.

2. Literature Review

As already mentioned, in this section, a literature review of recent advancements in the clustering techniques used for gathering insights on users’ consumption is presented. In most of the studied works, the clustering algorithms are used for segmenting the users into distinguished groups. The primary focus, in many approaches, is to determine the optimal number of clusters, in order to enhance the segmentation accuracy, making use of the Silhouette Score (SIL) evaluation metric [22,23]. In a similar manner, Abdulnassar and Nair in their approach in [24] tried to minimize the computation cost, without sacrificing accuracy, by optimizing the initial centroids. Furthermore, feature selection techniques can be used to maximize the efficiency of the clustering algorithms [25,26].

Another common approach is the utilization of different clustering models. More specifically, Gaussian mixture models [25], Self-Attention LSTM [27], and Ant Colony Clustering [28], together with commonly used clustering algorithms, such as K-means, Hierarchical Clustering [29,30], and DBSCAN [31], are only some of the algorithms used to segment smart energy data effectively. These studies highlight the growing trend of using hybrid approaches in gathering insights for the consumption of the residential users.

Another categorization of the available works is the clustering objectives. These vary from the most common segmentation of the users into clusters based on their consumption [19,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41] to identifying different consumption patterns on different users’ or a single user’s consumption time series [25,26,28,29,30,31,32,33,34,35,37], to identifying the daily variation of the consumption, in order to provide helpful insights [26,28,33,40], and, finally, to detecting anomalies in the consumption time series [28,33,41]. Each clustering services can offer valuable insights to decision makers regarding the energy consumption of users, whether residential or not, and their availability for applying DR solutions.

Finally, in [29], Satre-Meloy et al. made use of several different classification methods to predict the cluster membership of households based on occupant activity data during evening peak electricity usage hours, highlighting the importance of classification methods in different segmentation services. While prior works have explored demand-side management frameworks, highlighting their use of game-theoretic approaches and mixed-integer programming, for scheduling of Responsive Loads in Smart Distribution System, these primarily focus on optimization-based strategies for scheduling and load management [42,43]. In contrast, this work emphasizes a machine learning-based framework that combines clustering, classification, and advanced feature extraction techniques to provide a data-driven segmentation of residential users. The MAS-DR framework specifically addresses the gaps in clustering methodologies by introducing the novel Household Daily Load (HDL) concept and systematically evaluating clustering algorithms and metrics for tailored Demand Response (DR) strategies. This distinction lies in our emphasis on unsupervised learning for user behavior insights, which complements but differs fundamentally from optimization-based approaches.

In

Table 1. Summary of related works for clustering on energy-related data.

Citation	Resident. Cons.	Various Models	Various Metrics	Optimal Cluster No	Segment Users	Classify Users	Cons. Patterns	Daily Variation	Anomaly Detection	Feature Selection
[22]			✓	✓	✓
[19]	✓	✓	✓	✓	✓
[23]			✓	✓	✓
[24]		✓	✓
[25]	✓	✓	✓	✓	✓		✓		✓	✓
[27]		✓	✓	✓	✓
[28]	✓	✓	✓		✓		✓	✓
[32]	✓		✓	✓	✓		✓
[33]	✓	✓	✓	✓	✓		✓	✓
[34]	✓	✓	✓	✓	✓		✓
[29]	✓	✓	✓	✓	✓	✓	✓
[35]		✓	✓	✓	✓		✓
[30]	✓	✓	✓	✓	✓		✓
[36]			✓	✓	✓
[26]	✓	✓	✓	✓	✓		✓	✓		✓
[31]	✓	✓	✓	✓	✓		✓
[37]	✓		✓	✓	✓		✓
[38]	✓	✓	✓		✓
[39]	✓	✓			✓
[40]	✓	✓	✓	✓	✓			✓
[41]	✓		✓	✓	✓				✓
This Paper	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓

, an overview of the studied literature on the application of clustering and classification techniques for aggregating and segmenting energy users’ consumption is presented. This table highlights the gaps addressed by the proposed work, which aims to provide a holistic approach to the identified issue, offering valuable insights for DR specialists.

3. Methodology

This section presents the general methodology of the proposed framework, which is visualized in Figure 1. MAS-DR offers six different segmentation and aggregation services, which will be further discussed in the following subsections. The whole framework is designed to acquire and process raw consumption measurements from different residential houses and provide meaningful insights tailored to DR programs.

The raw data are processed through various levels of feature extraction and clustering techniques, tailored to a unique use case. The first use case explores the consumption of different users in order to divide them into groups (Service 1 and 2). The extracted features provide an overview of the users’ behavior, which is grouped and evaluated for targeted interventions using optimal clustering algorithms and feature importance analysis (Service 1.1). Based on classification methods, the cluster that a newly added user belongs to is also identified (Service 6).

Furthermore, advanced thresholds approaches are used to analyze a user’s daily variation (Service 3) or identify a user’s peak consumption (Service 4), providing insights on whether the particular user is suitable for DR strategies. Anomaly detection techniques are employed to identify users that could cause issues in DR processes (Service 5).

In this section, the different services that the MAS-DR framework offers to fulfill these objectives are analyzed in detailed. In addition, the different clustering algorithms and evaluation metrics are presented.

3.1. Aggregation and Segmentation Services

The MAS-DR framework introduces a suite of aggregation and segmentation services aimed at analyzing and grouping residential users based on their energy consumption patterns. These services are designed to provide actionable insights for DR strategies, allowing energy managers to optimize interventions for individual or clustered households. The aggregation and segmentation processes leverage machine learning techniques to extract meaningful features from raw consumption data, identify usage patterns, and classify users into distinct categories. The framework addresses various segmentation goals, including identifying daily consumption trends, peak load periods, and high-impact users who could influence grid stability. It integrates advanced clustering methods, combined with feature selection techniques while ensuring robust and efficient segmentation results. These services are critical in tailoring DR programs to user-specific behaviors while enhancing the grid’s operational resilience and energy efficiency. In the following subsections, each aggregation and segmentation service is elaborated in detail, highlighting its methodology, algorithms, and key objectives.

3.1.1. Service 1: Segment Users into Clusters After Feature Extraction

The first objective of MAS-DR framework is the segmentation of different residential users into users based on features derived from their energy consumption. As already mentioned, this is a highly common objective of recent studies. This work tries to dig deeper into the consumption time series of a household by extracting a wide variety of features, which are presented in

Table 2. Features used for energy consumption segmentation.

Category	Subcategory	Features	Details
Basic	Statistical	Average Consumption	The mean value of energy consumption.
		Variance Consumption	The variance of energy consumption.
		10th quantile	The consumption level above which 10% of the values lie.
		90th quantile	The consumption level below which 90% of the values lie.
Extended	Statistical	Median Consumption	The median value of energy consumption.
		Std Consumption	The standard deviation of energy consumption.
		Max Consumption	The highest recorder value of energy consumption.
		Min Consumption	The lowest recorder value of energy consumption.
		25th quantile	The consumption level above which 25% of the values lie.
		75th quantile	The consumption level below which 75% of the values lie.
	Temporal	Daily Average	The average consumption during a day.
		Weekly Average	The average consumption during a week.
		Weekend Average	The average consumption during weekends.
		Weekday Average	The average consumption during weekdays.
	Behavioral	Midnight average	The average consumption during late-night hours (00:00–05:00), typically representing idle usage.
		Morning average	The average consumption during morning hours (05:00–12:00), typically representing start-of-day activities.
		Afternoon average	The average consumption during midday hours (12:00–18:00), typically representing work-related activities.
		Evening average	The average consumption during evening hours (18:00–00:00), typically representing peak household activities.
	Trend	Trend Slope	The gradient of a linear trend line fitted to the data, showing whether energy usage is increasing, decreasing, or stable over time.
		Month-to-month Change	The percentage change in energy consumption from one month to the next, indicating seasonal or behavioral shifts.
		Average Daily Change	The average of day-to-day changes in consumption, revealing volatility in daily energy usage patterns.

. The features are separated into two main categories, basic and extended. The basic features, namely average consumption, variance of the consumption, and key quantiles (10th and 90th), are the most commonly used in different studies and provide an initial understanding of a household’s energy use and its variability.

Having the basic features as a foundation, the extended features offer a deeper analysis and can be divided into Statistical, Temporal, Behavioral, and Trend subcategories. Statistical metrics like median, standard deviation, and extreme values (max and min) represent a greater range of variability, whereas temporal features focus on specific patterns, such as workday versus weekend consumption. Behavioral features provide detailed insights regarding the type of the users that live in a household, as they can represent activities in different time of the day. Furthermore, trend analysis, using variables such as trend slope and month-to-month change, identifies long-term usage trends, whereas rate-of-change measurements reveal daily consumption volatility. These variables, combined with their hierarchical classification, provide a thorough picture of energy usage.

The next step of this service is to select the most important of the aforementioned features, which ensures that the clustering procedure is based on the most important and relevant variables. Two complementary methodologies are used for this analysis. First, Principal Component Analysis (PCA) [44] is performed to determine which features account for the majority of the variance in the dataset, giving a quantitative foundation for feature prioritizing. Second, the dataset is iteratively clustered, removing one feature at a time, and the influence on the cluster Silhouette Score [45] is monitored, which evaluates clustering quality.

3.1.2. Service 2: Segment Users into Clusters Based on Household Daily Load (HDL)

The term Household Daily Loads (HDL) was presented by Yilmaz et al. in [46] and it represents the average daily consumption of a household derived from smart meter data. The feature represents the average of hourly consumption data over a given period within a day and smooth out day-to-day variations while highlighting general consumption trends. The final result of this approach is one vector for each user that contains their average consumption for each hour of a day for a specific period, typically a month, a season, or a year.

In the proposed framework, Season-based HDL and Month-based HDL is integrated into the framework. Season-based HDL investigates differences in consumption habits between seasons (e.g., winter and summer). These profiles demonstrate how seasonal elements such as weather and daylight affect residential energy consumption. Clustering based on season-specific HDL allows energy operators to adjust interventions, such as heating-focused programs in the winter or cooling-focused initiatives in the summer. On the other hand, Month-based HDL catches monthly variations, such as increased energy consumption during the holidays or changes in daylight savings.

Enhancing the MAS-DR framework with HDL analysis offers a finer understanding on a households’ behavior and provides stakeholders, such as energy providers, with a tool to aggregate users that could benefit from similar DR strategies.

3.1.3. Service 3: Analyze User Daily Variation

The next objective that the proposed tool focuses on is the analysis of a user’s consumption variation in a single day. This service aims to find a way to quantify the variation of the user’s daily patterns by utilizing clustering methods on their consumption. The MAS-DR follows the methodology that is highlighted in Algorithm 1.

Algorithm 1: Analysis of users’ daily energy consumption variation.

function  AnalizeVariation(userList)                                                                                                                                                                                                                ▹ Define thresholds     clustering_number_threshold ← predefinedThreshold     sum_of_top_threshold ← topThreshold                                                                                                                                                                                        ▹ Step 1: Cluster user’s daily data     for user ∈ userList do          dailyData ← GroupByDay(user.consumptionData)          optimalClusters ← FindOptimalNumberClusters(dailyData)          dailyClusters ← ApplyClusteringAlgorithm(dailyData, optimalClusters)          user.clusterData ← dailyClusters          user.optimalClusterCount ← optimalClusters     end for                                                                                                                                                                                 ▹ Step 2: Analyze cluster distribution     for user ∈ userList do         if user.optimalClusterCount < clustering_number_threshold then                                                            ▹Condition 1: Consistent behavior               user.behavior ← “Consistent Behavior”         else                                                                                                                                                                       ▹Condition 2: Predictable pattern            topTwoCount ← CountTopTwoClusters(user.clusterData)            if topTwoCount > sum_of_top_threshold then                user.behavior ← “Predictable Pattern”            else                user.behavior ← “Diverse Behavior”            end if         end if     end for                                                                                                                                                                    ▹ Step 3: Assess suitability for DR programs     for user ∈ userList do         if user.behavior ∈ {“Consistent Behavior”, “Predictable Pattern”} then                user.DRSuitability ← True         else                user.DRSuitability ← False         end if     end for end function

After the definition of the thresholds, the first step of the methodology contains the grouping of the consumption data for each user by day to facilitate further analysis. Next, for each user, the optimal number of clusters is determined, representing the potential patterns the user’s consumption could fall into. A clustering algorithm is then applied to group these daily consumption profiles into meaningful clusters. In the second step, the cluster distribution is analyzed to identify users with significant representation in a limited number of clusters. This involves investigating two conditions. The first condition checks whether the optimal number of clusters is below a predefined threshold, which would indicate “Consistent Behavior”. If not, the second condition evaluates whether the data points in the most populated clusters exceed a specified threshold, suggesting that a significant portion of the user’s days follow a “Predictable Pattern”. In the third and final step of the algorithm, users are assessed for suitability in DR programs, with users that meet either of the conditions being characterized as either consistent or predictable, and in turn, marked as suitable for DR services.

Based on the aforementioned methodology, the MAS-DR framework can identify residential users that exhibit either consistent daily variation or are likely to have predictable consumption schedules, enabling them to respond effectively to price signals or load adjustment requests, and thus can be considered for DR programs.

3.1.4. Service 4: Finding the Consumption Peak in Daily Periods

The aim of this objective is to identify the peak consumption periods within a user’s daily electricity usage by segmenting the day into distinct time periods. This can assist in assessing whether the user can participate in DR programs. This notion is based on the hypothesis that when the peak consumption is pinpointed on evening periods, this is a typical residential household; additionally, evening peaks are of particular interest due to higher electricity prices during this period, offering an opportunity for cost savings and grid optimization. The methodology that this services follows is presented in Algorithm 2.

Algorithm 2: Finding the consumption peak in daily periods.

function  FindPeakDaily(userList)                                                                                                                                                                                        ▹ Step 1: Segmentation of the Day     midnightPeriod ← [00:00, 05:00]     morningPeriod ← [05:00, 12:00]     afternoonPeriod ← [12:00, 18:00]     eveningPeriod ← [18:00, 24:00]     for user ∈ userList do          dailyData ← GroupByTimePeriod(user.consumptionData, {midnightPeriod, morningPeriod, afternoonPeriod, eveningPeriod})          user.segmentedData ← dailyData     end for                                                                                                                                                                                 ▹ Step 2: Peak Consumption Analysis      for user ∈ userList do           peakPeriod ← FindPeakConsumptionPeriod(user.segmentedData)           user.peakPeriod ← peakPeriod                                                                                                            ▹ Condition: Peak occurs in the evening           if peakPeriod == eveningPeriod then                user.peakBehavior ← “Evening Peak”           else                user.peakBehavior ← “Other Peak”           end if     end for                                                                                                                                                                      ▹ Step 3: Assess Suitability for DR Programs     for user ∈ userList do         if user.peakBehavior == “Evening Peak” then              user.DRSuitability ← “Potentially Suitable for DR Programs”         else              user.DRSuitability ← “Not Suitable for DR Programs”         end if     end for end function

In the first step of the algorithm, the day is divided into four distinct periods: midnight (00:00–05:00), morning (05:00–12:00), afternoon (12:00–18:00), and evening (18:00–00:00). Each user’s electricity consumption data are grouped into these periods to create a segmented dataset. The second step involves the identification of the time period with the highest electricity consumption (peak period) for each user and determines whether this peak consistently occurs during the evening period. If the peak is in the evening, the users’ behavior is classified as “Evening Peak”. In the third step of the methodology, each user is assessed for suitability for DR programs. More specifically, users with an evening peak can be prioritized for DR strategies, such us load adjustments during high-demand hours, while users with with peaks in other periods may still be monitored.

3.1.5. Service 5: Identifying High-Impact Users

This service offered by the MAS-DR framework aims to identify users that have a high impact on the operation of the energy grid, and could potentially cause issues when DR strategies are applied to them. The service examines the consumption of each user in order to identify their peak consumption, the variance of their consumption, and the frequency of their peaks. In addition, the number and frequency of the potential anomalies that the users’ consumption presents is detected. Key stakeholders, using this framework, can establish a threshold for each of these finding, based on their knowledge on the domain and their needs. If the threshold is surpassed, valuable insights are derived from the segmentation process of the residential consumption. Algorithm 3 presents the methodology for Service 5.

Algorithm 3: Identifying High-Impact Users.

function IdentifyHighImpactUsers(userList)                                                                                                                                                                                                                ▹ Define thresholds     peak_consumption_threshold ← predefinedPeakThreshold     variability_threshold ← predefinedVariabilityThreshold     spike_frequency_threshold ← predefinedSpikeFrequencyThreshold     anomaly_threshold ← predefinedAnomalyThreshold                                                                                                                                                             ▹ Step 1: Feature Analysis for High-Impact Users     for user ∈ userList do           peakConsumption ← FindPeakConsumption(user.consumptionData)           if peakConsumption > peak_consumption_threshold then                 user.isHighPeakUser ← True           else                 user.isHighPeakUser ← False           end if           consumptionVariability ← CalculateVariability(user.consumptionData)           if consumptionVariability > variability_threshold then                 user.isHighVariabilityUser ← True           else                 user.isHighVariabilityUser ← False           end if           spikeFrequency ← CalculateSpikeFrequency(user.consumptionData)           if spikeFrequency > spike_frequency_threshold then                user.isFrequentPeakUser ← True           else                 user.isFrequentPeakUser ← False           end if           if user.isHighPeakUser and user.isHighVariabilityUser and user.isFrequentPeakUser then                 user.isHighImpactUser ← True           else                 user.isHighImpactUser ← False           end if       end for                                                                                                                                                                                               ▹ Step 2: Anomaly Detection       for user ∈ userList do           anomalies ← DetectAnomalies(user.consumptionData, method={“IsolationForest”, “kNN”})           anomalyCount ← CountAnomalies(anomalies)           if anomalyCount > anomaly_threshold then                user.isHighAnomalyUser ← True           else                 user.isHighAnomalyUser ← False           end if       end for                                                                                                                                                                                                 ▹ Step 3: User Segmentation       for user ∈ userList do           if user.isHighImpactUser and user.isHighAnomalyUser then               user.segment ← “Potential Issue User”           else if user.isHighImpactUser then               user.segment ← “High-Impact User”           else              user.segment ← “Regular User”           end if       end for end function

The algorithm for identifying high-impact users involves a three-step process. In the first step, the feature analysis phase, thresholds are predefined for peak consumption, consumption variability, and spike frequency. For each user, the algorithm calculates their peak consumption, variability, and frequency of spikes in their data. Users exceeding the respective thresholds are flagged as “high peak users”, “high variability users”, or “frequent peak users”. If a user meets all three criteria, they are classified as a “high-impact user”. The anomaly detection phase leverages techniques such as the Isolation Forest algorithm [47] and k-Nearest Neighbors (kNN) [48] to identify anomalies in user consumption data. Anomalies are counted, and users with anomaly counts exceeding the predefined threshold are marked as “high anomaly users”. In the segmentation phase, users are categorized based on their behaviors and anomaly counts. High-impact users who also exhibit a high anomaly count are labeled as “potential issue users”, indicating their potential risk to grid stability. Meanwhile, high-impact users without significant anomalies are simply tagged as “high-impact users”. This structured approach allows utility providers to identify and manage users whose behaviors have the most significant impact on energy systems.

3.1.6. Service 6: Classify New Users

The final service of the MAS-DR framework provides a solution that assigns new users into pre-existing clusters by exploiting classification-based approaches. In quite common scenarios where a new user joins a DR schema, their energy consumption patterns are often unknown or underrepresented due to limited data. This challenge is addressed through a classifier algorithm trained on historical data from existing users. The classifier assigns new users to one of the predefined clusters, which represent distinct patterns of energy consumption. These clusters are derived from earlier clustering analysis, using features identified during Service 1, with the cluster labels serving as target variables during training. A great advantage of this service is that, by classifying new users into clusters, the service ensures that DR strategies can be extended to new users without delay, while additionally, the framework can accommodate growing numbers of users by continuously updating classification models with new data. A visual representation of the algorithm that Service 6 follows is presented in Algorithm 4.

Algorithm 4: Classification framework for new residential energy users.

function  TrainClassificationModels(existingUsersData, clusterLabels)                                                                                                                                                                            ▹ Step 1: Prepare Training Data     trainingFeatures ← ExtractFeatures(existingUsersData)     trainingLabels ← clusterLabels                                                                                                                                                                      ▹ Step 2: Train and save Classifiers     classifiers ← TrainModels(trainingFeatures, trainingLabels)     SaveModels(classifiers) end function function  ClassifyNewUser(newUserData, pretrainedModels)                                                                                                                                                                                  ▹ Step 1: Feature Extraction     extractedFeatures ← ExtractFeatures(newUserData)                                                                                                                                                                      ▹ Step 2: Apply Pretrained Models     clusterPredictions ← ClassifyUser(extractedFeatures, pretrainedModels)     assignedCluster ← DetermineFinalCluster(clusterPredictions)                                                                                                                                                                                        ▹ Step 3: Assign Cluster     newUser.cluster ← assignedCluster end function

3.2. Machine Learning Algorithms

In this subsection, the clustering and classification methods used in this paper are presented. Six clustering methods, including K-Means, Hierarchical Clustering, Spectral Clustering, DBSCAN, Mean Shift Clustering, and Gaussian Mixture Model, are employed, along with a classification algorithm, Random Forest.

3.2.1. Clustering Algorithms

The proposed framework incorporates a wide variety of commonly used clustering algorithms to segment the electricity consumption patterns of the residential users. The algorithms were selected based on both their application on similar approaches, as already mentioned in Section 2, and their unique capabilities in uncovering distinct patterns in residential consumption data. The selection of the clustering algorithms is motivated by their complementary characteristics and demonstrated efficacy in similar applications. They offer a diverse exploration of segmentation capabilities, which will be further presented in each algorithm’s analysis that follows. More specifically, the algorithms are shown below.

K-Means is a centroid-based algorithm, which is particularly effective for datasets with a clear spherical cluster structure. It operates iteratively by partitioning n data points into k clusters, minimizing the within-cluster variance [49]. The algorithm partitions n data points into k clusters, minimizing within-cluster variance, starting with k random centroids and assigning each point to the nearest centroid. Centroids are recalculated as the mean of their points, repeating until assignments stabilize or variance reduction is minimal. The objective function of the algorithm is given by:

$$argmi{n}_c\sum _{i=1}^k\sum _{xϵC_i}\left|\right|x-{\mu }_i{\left|\right|}^2,$$

(1)

where $C_i$ is the $i_{th}$ cluster, $m{u}_i$ is the centroid of $C_i$, and x represents the data points.

Hierarchical clustering is a method that creates a hierarchy of clusters in a tree-like structure known as a dendrogram. There are two different types of the alogirthm, the agglomerative (bottom-up) and divisive (top-down) method [50]. In this paper, the agglomerative approach was used, in which each data point starts as an individual cluster, and pairs of clusters are merged iteratively based on a chosen linkage criterion including single, complete, or average linkage. The minimum distance between points in two clusters is calculated as follows:

$$d(C_i,C_j)=mi{n}_{xϵC_i,yϵC_j}\left|\right|x-y\left|\right|$$

(2)

Spectral clustering is a method based on graph theory that embeds data into a lower-dimensional space before clustering. The method converts datasets into graph representations, computes the graph Laplacian, and extracts eigenvectors corresponding to the smallest eigenvalues. These eigenvectors are then used in a simpler clustering method, such as K-means, to determine the clusters [51].

DBSCAN is a density-based algorithm that detects clusters based on the density of data points. The clusters are defined as regions with a minimum number of points within a specified radius ($ϵ$). A point is considered as a core point if at least minimum number of points are present in its neighborhood, whereas border points are located within a core point’s neighborhood but lack sufficient neighbors of their own [52]. The neighborhood is defined as:

$$N_{ϵ}\left(x\right)=\left\{yϵX\left|\right||x-y||\le ϵ\right\}$$

(3)

Mean Shift clustering is a centroid-based algorithm that detectes clusters by iteratively shifting data points towards regions of higher density. Each data point is moved into the density function’s mode by computing the gradient using a kernel density estimator [53]. The mean shift vector for a point x is calculated by:

$$m\left(x\right)={\displaystyle \frac{{\sum }_{x_iϵN\left(x\right)}K(x_i-x)x_i}{{\sum }_{x_iϵN\left(x\right)}K(x_i-x)}}-x,$$

(4)

where K is the kernel function (Gaussian).

Gaussian Mixture Model (GMM) is a method capable of modeling data representation clusters with elliptical shapes [54]. It assumes that data points are generated from a mixture of Gaussian distributions, characterized by its mean ($\mu$), covariance ($\sigma$), and weight ($\pi$). The probability density function for a data point x using GMM calculated as:

$$p\left(x\right)=\sum _{i=1}^k{\pi }_{i}N\left(x\right|{\mu }_i{\sigma }_i),$$

(5)

where $N\left(x\right|{\mu }_i{\sigma }_i)$ is the multivariate Gaussian distribution.

3.2.2. Classification Algorithm

Random Forest is an ensemble learning method that combines multiple decision trees to make robust and accurate predictions. To ensure diversity among the trees, it selects a random subset of features at each split and trains each tree on a bootstrap sample of the data [55]. The classifier uses majority voting for classification tasks to aggregate all trees’ predictions during inference. Mathematically, the prediction for a data point x is given by:

$$\widehat{y}=mode\left\{h_1\left(x\right),h_2\left(x\right),\dots ,h_T\left(x\right)\right\},$$

(6)

where $mode$ refers to the value that appears most frequently in a dataset, $h_t\left(x\right)$ is the prediction of the t -th tree, and T is the total number of trees.

3.3. Evaluation Metrics

In this subsection, the evaluation metrics used in this paper are presented. Three different metrics were selected, i.e., the Silhouette Score, Davies–Bouldin Index, and the Calinski–Harabasz Index, to exploit the different aspect and characteristics of each one in the evaluation process.

The Silhouette Score (SIL) measures the quality of clustering by evaluating how similar a data point is to points within its own cluster compared to points in the nearest neighboring cluster [45]. It ranges from $-1$ to 1, where values closer to 1 indicate well-separated clusters. The SIL for a single point i is calculated as:

$$s\left(i\right)={\displaystyle \frac{b\left(i\right)-a\left(i\right)}{max\left(a\right(i),b(i\left)\right)}},$$

(7)

where $a\left(i\right)$ is the average distance from point i to all other points in its cluster and $b\left(i\right)$ is the average distance from point i to all points in the nearest cluster.

Davies–Bouldin Index (DBI) uses the ratio of within-cluster scatter to between-cluster separation to assess the quality of clustering [56]. A lower DBI indicates better clustering, with more compact and well-separated clusters. It is calculated as:

$$DBI={\displaystyle \frac{1}{k}}\sum _{i=1}^{k}ma{x}_{j\ne i}\left({\displaystyle \frac{{\sigma }_i+{\sigma }_j}{d_{ij}}}\right),$$

(8)

where ${\sigma }_i$ is the average distance of points in cluster i to the centroid of cluster i, $d_{ij}$ is the distance between the centroids of clusters i and j, and k is the total number of clusters.

The Calinski–Harabasz Index (CHI), which is also known as the Variance Ratio Criterion, evaluates clustering quality by comparing the dispersion of points within clusters to the dispersion between clusters [57]. Higher CHI values indicate better-defined clusters. The CHI is calculated as:

$$CHI={\displaystyle \frac{Tr\left(B_k\right)/(k-1)}{Tr\left(W_k\right)/(n-k)}},$$

(9)

where $Tr\left(B_k\right)$ is the trace of the between-cluster dispersion matrix, $Tr\left(W_k\right)$ is the trace of the within-cluster dispersion matrix, k is the number of clusters, and n is the total number of data points.

4. Case Study

To validate the proposed MAS-DR framework, this section presents a real-world case study utilizing publicly available energy consumption data. The study aims to demonstrate the framework’s effectiveness in aggregating and segmenting residential users to assist DR strategies. The dataset originates from smart meters installed in London households, covering detailed energy consumption measurements across a significant time-frame. The case study is structured to evaluate the performance of MAS-DR services, including clustering, classification, and feature analysis.

4.1. Dataset

The case study that serves as an initial validation method of the proposed framework is based on open energy consumption data originated from residential houses in London [58]. The dataset contains measurements from 5567 households that span from November 2011 to February 2014 and captures energy readings at half-hour intervals and includes the following key attributes: the energy consumption (kWh) at half-hourly energy readings for each household, the date and time corresponding to each reading, and a unique ID for each household. It was sourced from publicly available repositories provided by the London Data Store, which aggregate energy use data from smart meters installed in residential buildings.

The households that belong to this dataset are separated into non-Time of Use customers, which were on a constant rate tariff, and customers that Dynamic Time of Use (dToU) were applied to. The 1100 dToU customers were given different price signals throughout the 2013 calendar period, information available on the original dataset. For the approach followed in the evaluation of the proposed framework, due to the huge amount of data, certain measures were followed to restrict their dimensionality. At first, only one year of data were used, more specifically, consumption measurements during 2013 were part of the experimentation. Additionally, the dataset was converted to hourly resolution, in order to both reduce the number of values, while also retaining sufficient granularity to capture meaningful temporal patterns.

As expected from real-life datasets, there were many missing and erroneous data. A rule-of-thumb approach was deployed to exclude these inconsequential data, based on which users with more than 20% missing values in their hourly readings were removed. Furthermore, the raw data measurements presented repeated identical readings over extended periods. In the evaluation of MAS-DR, residential houses, where more than 50% of their readings in a year were repeated, were excluded to ensure data integrity and meaningful representation of usage patterns. During the dataset initial pre-processing, the outlier values were not removed for each user, as they were considered useful in identifying anomalies and significant events in their consumption, which is the main purpose of Service 5.

After this initial pre-processing, the final dataset that was used included 5168 different users, spanning over a one-year period in hourly granularity. The MAS-DR framework’s services were applied in these all houses as well as subdividing them into smaller, user-defined blocks to examine specific consumption behaviors and clustering scenarios. An indicative sample of the dataset’s statistics is presented on

Table 3. Summary data for selected users and aggregate statistics.

	User_10	User_48	User_1683	User_3194	All Houses (Mean)	All Houses (Max)	All Houses (Min)
Avg Daily Loa	11.73	29.53	0.89	17.19	10.11	101.35	0.54
Peak Daily Load	23.13	105.83	2.64	80.31	25.08	321.70	1.48
Min Daily Load	3.10	4.57	0.22	5.55	3.21	61.95	0.00
St. Deviation	0.41	1.20	0.46	0.76	0.38	3.14	0.01

.

4.2. Experimental Setup

The experimental setup outlines the steps taken to validate the MAS-DR framework using real-world energy consumption data. The experimental process focuses on ensuring data quality, efficient processing, and robust segmentation performance to achieve meaningful insights for DR strategies. During the experimentation process, Python programming language was used for the implementation, Scikit-learn [59] was used for machine learning, Pandas [60] was used for data processing, and Plotly [61] was used for visualization. In addition, computational experiments were conducted on an Intel Core i5-10600K CPU with 16 GB RAM.

4.2.1. Pre-Process Finalization

After the acquisition, preliminary analysis and initial pre-process of the data, the next step on the design of the use case evaluation of the proposed framework is the experimental set-up. For each service, the experimental set-up varies, but there are steps that are common between all of them. These steps refer to the process of data, and more specifically the normalization and scaling of data.

4.2.2. Define the Optimal Number of Clusters

Regarding the segmentation of the users (Service 1 and Service 2), some of the algorithms used require to pre-define the number of clusters that data are going to be segmented to. For that purpose, the evaluation metrics were utilize to determine the optimal number of clusters for K-means, Agglomerative Clustering, Spectral Clustering, and BIRCH algorithms, in order to determine the optimal number of clusters. In

Table 4. Optimal number of clusters for Service 1: User segmentation after feature extraction.

	K-means	Hierarchical	Spectral	BIRCH
SIL	2	2	2	2
DBI	2	3	2	3
CHI	2	2	18	5

and

Table 5. Optimal number of clusters for Service 2: User segmentation based on HDL.

	K-means	Hierarchical	Spectral	BIRCH
SIL	2	2	2	2
DBI	2	2	2	6
CHI	2	2	4	5

, the optimal number of clusters for Service 1: User segmentation after feature extraction and Service 2: User segmentation based on HDL are presented, respectively. In order to have a deeper dive in the analysis, Figure 2 and Figure 3 present the Silhouette Score all clusters and algorithms for both services.

From the results, it can be observed that each metric has a slight difference regarding the number of clusters. In this paper, the most voted result was followed, which corresponded to the optimal number of clusters being two.

4.2.3. Check Feature Importance

The main idea when designing the framework, regarding the enhancement of Service 1, is to utilize all the available information derived form the consumption of residential users. Keeping that in mind, the efficiency of the service is of equal significance. Thus, it is valuable to further investigate the important of each feature. MAS-DR offers two methods to determine the feature importance (Service 1.1), Principal Component Analysis (PCA), to observe which features contribute most to the variance in the dataset, and iteratively clustering the dataset, excluding one feature and observing the impact on a cluster’s Silhouette Score.

More specifically, regarding the first method, the first step is to cluster the data in the optimal amount of clusters using one of the clustering algorithms. Then, feature importance for each cluster is computed using the PCA analysis. A dictionary of feature importance is created, with sorted and normalized results, where the higher values are better. The result of this analysis is presented in Figure 4.

The second method follows an iterate-based logic. More specifically, iteratively exclude each feature form the dataset and apply a clustering algorithm (in our case, K-means), to the reduced dataset without the feature. The next step is to compute the Silhouette Score, in each iteration, which measures how well-separated the clusters are. If the Silhouette Score is high, then the excluded feature is less important. The next step is the sorting of features based on the calculated Silhouette Scores. Finally, in order to have intuitive results, the scores are reverse and normalized, where a feature with a higher score is considered more important. The results for our dataset and for the features discussed in Section 3 are presented in Figure 5.

The final step of the feature importance is the combination of the two methods, based on which the feature vector that MAS-DR uses is decided.

5. Aggregation and Segmentation Insights

This section of the paper concerns the presentation of the insights derived from the MAS-DR framework. The results can be separated into three main categories regarding the objectives, namely cluster segmentation of the dataset, identification of each user’s DR insights, and the assignment of a new user to pre-defined clusters.

5.1. Segmenting Users into Clusters

This section presents the results of Service 1 and Service 2, which focus on clustering residential users based on their energy consumption patterns. Service 1 utilizes extended statistical, temporal, and behavioral features, while Service 2 applies the Household Daily Load (HDL) approach for seasonal and monthly analysis. The segmentation aims to identify distinct user groups, enabling tailored Demand Response (DR) strategies. Clustering performance is evaluated using multiple algorithms and metrics to ensure robust and meaningful results.

5.1.1. Service 1

This service focuses on clustering residential users based on a variety of statistical, temporal, and behavioral features derived from their energy consumption. Multiple clustering algorithms were evaluated using three performance metrics: Silhouette Score, Davies–Bouldin Index (DBI), and Calinski–Harabasz Index (CHI). The results for Service 1, as shown in

Table 6. Clustering performance metrics for Service 1: User segmentation after feature extraction.

Algorithm	Extended Features	Silhouette Score	Davies–Bouldin Index	Calinski–Harabasz Index
K-means	False	0.67	1.02	1327.38
	True	0.68	1.00	1348.05
Hierarchical	False	0.56	1.59	1185.57
	True	0.58	1.83	1225.88
Spectral	False	0.50	1.65	501.193
	True	0.83	0.47	1096.85
DBSCAN	False	0.73	0.84	1067.81
	True	0.75	1.49	1116.26
Mean Shift	False	0.44	1.58	175.68
	True	−0.16	1.59	60.47
GMM	False	0.33	2.03	98.34
	True	0.42	2.19	86.09

Note: Bold values indicate the best-performing algorithms based on each metric.

, indicate that the Spectral Clustering algorithm outperforms other methods, achieving the highest Silhouette Score (0.83) with extended features, the lowest Davies–Bouldin Index (0.47), suggesting well-separated clusters and a reasonable Calinski–Harabasz Index (1096.85). Furthermore, K-means performed well with extended features, achieving a CHI of 1348.05, but its Silhouette Score was slightly lower (0.68). Conversely, DBSCAN and Hierarchical Clustering showed competitive results but underperformed compared to Spectral Clustering. Additionally, it can be observed that in most of the approaches the extended features provide an increase in clustering efficiency.

The clustering analysis revealed two primary user clusters (Cluster 0 and Cluster 1). For Cluster 0 (Figure 6a), users exhibit lower overall power consumption, as indicated by their mean and median profiles.

Moreover, consumption remains stable with fewer extreme spikes throughout the year. For Cluster 1 (Figure 6b), users have higher average consumption with greater variability and their load profiles show occasional spikes, particularly in certain periods (e.g., winter).

The 2D visualization (Figure 7) highlights the distinct separation between the two clusters. While Cluster 0 users are densely grouped (lower consumption range), Cluster 1 users are more spread out, indicating higher variability and consumption.

Overall, spectral clustering delivers the best performance for user segmentation after feature extraction. The two clusters distinguish users based on consumption magnitude and variability. The extended feature set enhances clustering performance, as evident from improved metrics (Silhouette Score, DBI). This segmentation enables targeted DR strategies, allowing energy managers to optimize interventions for each user group effectively.

5.1.2. Service 2

This service analyzes the energy consumption of residential users by leveraging the Household Daily Load (HDL) concept, focusing on seasonal variations. The segmentation aims to uncover consumption trends that align with specific seasonal periods (winter, spring, summer, and autumn). Clustering performance was evaluated using multiple algorithms and three metrics: Silhouette Score, DBI, and CHI. As presented in

Table 7. Clustering performance metrics for Service 2: User segmentation based on Season-HDL.

Algorithm	Season	Silhouette Score	Davies–Bouldin Index	Calinski–Harabasz Index
K-means	winter	0.71	0.18	22.96
	spring	0.71	0.18	22.96
	summer	0.66	0.81	23.08
	autumn	0.71	0.18	22.96
Hierarchical	winter	0.33	1.50	15.42
	spring	0.71	0.18	22.96
	summer	0.66	0.81	23.08
	autumn	0.71	0.18	22.96
Spectral	winter	0.70	0.17	22.91
	spring	0.72	0.17	25.45
	summer	0.70	0.18	22.91
	autumn	0.73	0.17	28.96
DBSCAN	winter	0.42	1.57	13.43
	spring	0.46	1.38	16.65
	summer	0.52	1.32	16.52
	autumn	0.55	1.15	20.07
Mean Shift	winter	0.28	0.92	17.61
	spring	0.17	0.94	13.41
	summer	0.20	0.92	15.25
	autumn	0.28	1.07	15.83
GMM	winter	0.61	0.68	16.56
	spring	0.71	0.68	17.96
	summer	0.66	0.81	13.08
	autumn	0.71	0.68	17.96

Note: Bold values indicate the best-performing algorithms based on each metric for each season.

, spectral Clustering consistently outperforms other algorithms across all seasons—specifically, it has the highest Silhouette Score (e.g., 0.73 in autumn), the lowest DBI (e.g., 0.17 in autumn), indicating compact and well-separated clusters, and the highest CHI values (e.g., 28.96 in autumn), reflecting strong cluster structures. Furthermore, K-means also performs reliably with a Silhouette Score of 0.71 and competitive CHI values for multiple seasons, particularly in winter and autumn, while DBSCAN and Hierarchical Clustering show subpar performance in comparison, particularly due to less compact and less clearly separated clusters.

The summer season results, as visualized in Figure 8, highlight two distinct clusters. In Cluster 0 (Figure 8a), users with low and stable energy consumption are represented. The mean and median profiles remain consistently below 20 kW, with minimal fluctuations over time. In Cluster 1 (Figure 8b), users with higher and more variable energy consumption are represented. The mean and median power levels are significantly higher, with peaks occasionally exceeding 100 kW, indicating substantial energy usage variability.

Spectral Clustering provides the most effective segmentation for seasonal HDL, outperforming other methods across all metrics. Moreover, clear patterns emerge in user behavior, Cluster 0 users are characterized by low and stable energy consumption. Cluster 1 users exhibit higher and more dynamic consumption patterns. Seasonal segmentation (e.g., summer analysis) highlights the impact of seasonal variations, offering actionable insights for DR programs tailored to seasonal peaks. This segmentation allows energy managers to target high-impact users and tailor interventions, such as peak-load reduction strategies during summer months when consumption is elevated.

The 2D representation of clusters in Figure 9 highlights the separation between the two identified clusters for the summer season using K-means clustering. Cluster 0 (purple) are users with low and stable energy consumption are tightly grouped, indicating minimal variance in their energy usage. Cluster 1 (green) are users with higher and more varied consumption are more dispersed, reflecting significant differences in their energy behavior. The clear spatial separation between the clusters reinforces the effectiveness of the K-means algorithm in identifying distinct user groups based on the Season-HDL feature.

The clustering performance across months (January, April, July, and November) was analyzed using multiple algorithms and evaluation metrics, as presented in

Table 8. Clustering performance metrics for Service 2: User segmentation based on Month-HDL.

Algorithm	Season	Silhouette Score	Davies–Bouldin Index	Calinski–Harabasz Index
KMeans	January	0.61	0.97	23.44
	April	0.61	0.97	23.44
	July	0.66	0.81	23.08
	November	0.72	0.16	27.96
Hierarchical	January	0.38	1.46	16.46
	April	0.61	0.97	23.44
	July	0.66	0.81	23.08
	November	0.71	0.18	22.96
Spectral	January	0.61	0.97	23.44
	April	0.62	0.82	19.72
	July	0.66	0.81	23.08
	November	0.71	0.18	22.96
DBSCAN	January	0.29	1.57	12.78
	April	0.32	1.56	12.78
	July	0.66	0.81	23.08
	November	0.55	0.81	23.08
Mean Shift	January	0.21	0.94	14.44
	April	0.31	1.14	14.39
	July	0.34	0.94	19.50
	November	0.31	1.01	17.50
GMM	January	0.61	0.97	23.44
	April	0.61	0.97	23.44
	July	0.66	0.81	23.08
	November	0.71	0.18	22.96

Note: Bold values indicate the best-performing algorithms based on each metric for each month.

. The results indicate the following trends: K-means consistently delivers strong performance, particularly in November, with the highest Silhouette Score of 0.72, the lowest Davies–Bouldin Index of 0.16, and the highest Calinski–Harabasz Index of 27.96, reflecting compact and well-separated clusters. Spectral Clustering also performs well, with competitive Silhouette Scores and DBI values across all months. Hierarchical Clustering and DBSCAN show weaker performance, particularly in January and April, with lower cluster compactness and higher DBI scores.

The monthly analysis for January, visualized in Figure 10, reveals two primary clusters. Cluster 0 (Figure 10a) users in this cluster exhibit low and stable energy consumption, with both mean and median power values consistently below 20 kW. Consumption remains predictable, with minor fluctuations throughout the month. Cluster 1 (Figure 10b) represents higher energy consumers with occasional spikes, particularly during specific periods of the month. The mean consumption consistently exceeds 50 kW, indicating larger or more energy-intensive households. A 2D representation of clusters in Figure 11 highlights the separation between the two identified clusters.

Cluster 1 users can be prioritized for peak load reduction strategies to address occasional spikes. Cluster 0 users represent steady, manageable consumption patterns, contributing to grid stability. The insights derived from monthly segmentation enable more granular DR interventions, ensuring that seasonal or monthly energy patterns are addressed effectively.

5.2. Identification of Users’ DR Capability

This section evaluates the suitability of residential users for DR programs based on their energy consumption behaviors. Results from Service 3 (daily variation analysis), Service 4 (peak consumption identification), and Service 5 (high-impact user detection) are combined to provide a holistic view of user readiness. Users are categorized based on their behavior patterns, peak consumption periods, and potential grid impact, enabling energy managers to prioritize DR interventions effectively and ensure grid stability.

5.2.1. Service 3

Service 3 evaluates the daily energy consumption variation of individual users to assess their suitability for DR strategies. The analysis identifies users exhibiting consistent or predictable behavior based on two key thresholds are users with fewer than 3 clusters exhibit consistent behavior. Furthermore, if the top clusters cover more than 80% of daily consumption, the behavior is classified as predictable. The results in

Table 9. User daily variation analysis—indicative results.

User	Number of Clusters	Sum of Top Clusters	Behavior	Suitable for DR
user_3	3	100	Consistent	TRUE
user_12	2	100	Consistent	TRUE
user_68	3	100	Consistent	TRUE
user_209	2	100	Consistent	TRUE
user_330	8	63.29	Diverse	FALSE
user_450	11	47.12	Diverse	FALSE
user_1250	2	100	Consistent	TRUE
user_2655	4	82.47	Predictable	TRUE
user_3003	2	100	Consistent	TRUE
user_3651	2	100	Consistent	TRUE
user_4266	5	92.47	Predictable	TRUE

indicate that users such as user_3, user_12, and user_209 exhibit consistent behavior (fewer than 3 clusters with 100% of data in the top clusters), making them ideal candidates for DR programs. Users like user_2655 and user_4266 demonstrate predictable behavior, with the top clusters covering more than 80% of the data, also qualifying them for DR strategies. Users such as user_330 and user_450 display diverse behavior, where consumption is spread across many clusters (low top-cluster coverage), rendering them less suitable for DR.

Out of the analyzed users, most exhibit either consistent or predictable behavior, making them suitable for targeted DR interventions. Users with diverse behavior may require further analysis or alternative energy management strategies to optimize their participation in DR programs. These results help prioritize residential users for DR implementation, focusing on those with stable and predictable energy usage patterns.

5.2.2. Service 4

Service 4 identifies the peak energy consumption period for each user and assesses their suitability for DR strategies. The day is segmented into four periods:

• Midnight (00:00–05:00);
• Morning (05:00–12:00);
• Afternoon (12:00–18:00);
• Evening (18:00–24:00).

The objective is to pinpoint users whose peak consumption occurs during evening hours, as this aligns with typical residential demand spikes and offers the most potential for DR interventions. The results presented in

Table 10. User consumption peak identification—indicative results.

User	Afternoon	Evening	Midnight	Morning	Peak	DR Ready?
user_159	0.20	0.21	0.15	0.22	morning	FALSE
user_516	0.01	0.02	0.01	0.01	evening	TRUE
user_933	0.04	0.07	0.02	0.04	evening	TRUE
user_1162	0.12	0.10	0.27	0.07	midnight	FALSE
user_2383	0.07	0.04	0.01	0.05	afternoon	FALSE
user_3698	0.18	0.12	0.06	0.06	afternoon	FALSE
user_4268	0.17	0.21	0.03	0.11	evening	TRUE
user_5100	0.16	0.22	0.04	0.15	evening	TRUE

highlight that users such as user_516, user_933, user_4268, and user_5100 exhibit peak consumption during the evening, making them DR-ready. These users can benefit from load-shifting strategies to reduce stress on the grid. Users like user_159 (morning) and user_1162 (midnight) have peaks outside the evening window, rendering them less suitable for typical residential DR programs. Users with afternoon peaks, such as user_2383 and user_3698, may still require tailored interventions but are not prioritized for evening-focused DR strategies.

Overall, evening peak users are approximately half of the analyzed users, which demonstrate evening peaks, making them prime candidates for load reduction or price-based incentives during peak hours. Non-Evening Peaks are users with peaks in other time periods (i.e., morning, afternoon, and midnight) may require further analysis for customized DR solutions. The identification of peak periods allows energy managers to design targeted strategies that optimize grid stability and encourage user participation in Demand Response programs.

5.2.3. Service 5

Service 5 focuses on identifying high-impact users who may pose challenges to the energy grid or DR strategies due to irregular or extreme consumption behaviors. This service evaluates users based on the following criteria:

• High Peak User: Users with excessive peak consumption;
• Frequent Peak User: Users with recurring peaks;
• High Variability User: Users with significant fluctuations in energy usage;
• High Anomalies User: Users exhibiting frequent anomalies in their consumption patterns.

Users flagged in multiple categories are classified as High-Impact Users, and further analysis determines whether they qualify as Potential Issue Users.

The findings presented in

Table 11. High-impact user identification—indicative results.

User	High Peak User	Frequent Peak User	High Variability User	High Anomalies User	High-Impact User	Potential Issues User
user_52	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE
user_130	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE
user_245	FALSE	TRUE	FALSE	FALSE	FALSE	FALSE
user_398	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE
user_512	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE
user_673	TRUE	TRUE	TRUE	TRUE	TRUE	TRUE
user_918	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE
user_1327	FALSE	FALSE	FALSE	TRUE	FALSE	FALSE
user_1984	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE
user_2657	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE
user_3489	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE
user_4950	FALSE	FALSE	FALSE	FALSE	FALSE	FALSE

indicate that high Anomalies Users, as user_52, user_130, and user_1327 exhibit frequent anomalies, suggesting unpredictable or inconsistent energy consumption. High Peak Users like user_673 are characterized by excessive energy usage at certain times, which may stress the grid. Potential Issue Users like user_673 meet multiple criteria (High Peak, High Variability, and High Anomalies), requiring monitoring and tailored interventions.

Overall, most users flagged for high impact exhibit anomalous consumption patterns, emphasizing the need for anomaly detection tools in DR frameworks. User 673 is the most critical case, exhibiting peaks, variability, and anomalies, posing a significant risk to grid stability. Identifying such high-impact users allows energy managers to design targeted interventions to mitigate consumption irregularities and prioritize grid stability by monitoring and addressing potential issue users.

5.2.4. Overview of User Categorization

A summary of user analysis across Service 3, Service 4, and Service 5 is presented in

Table 12. Statistics for the identification of user consumption—indicative results.

	User Categorization	Count	Percentage (%)
	Total Users	5168	100.00
Service 3	Consistent Behavior Users	533	10.31
	Predictable Pattern Users	4872	94.27
	Diverse Behavior Users	296	5.73
Service 4	Evening Peak Users	2575	49.83
	Morning Peak Users	859	16.62
	Midnight Peak Users	81	1.57
	Afternoon Peak Users	1653	31.99
Service 5	High Peak Users	39	0.75
	Frequent Peak Users	345	6.68
	High Variability Users	34	0.66
	High Anomalies Users	1252	24.23
	High-Impact Users	5	0.10
	Potential Issues Users	4	0.08

, providing insights into user behavior, consumption peaks, and high-impact users. The majority of users (94.27%) display predictable patterns, making them ideal for tailored DR strategies. Nearly 50% of users have evening peaks, aligning with grid stress periods and presenting significant opportunities for load-shifting strategies. High-impact and potential issue users constitute a small but critical subset that requires continuous monitoring and customized solutions to maintain grid stability.

5.3. Assign New User to a Pre-Defined Group

In this section, the MAS-DR framework leverages a classification-based approach to assign new users to pre-existing groups or clusters. By utilizing historical data and machine learning models, such as the Random Forest classifier, the system effectively integrates new users into the established segmentation structure. This ensures that DR strategies can be promptly applied to new participants, enabling seamless scalability and enhanced grid optimization.

Service 6

For Service 6, a classification service was evaluated to integrate new users into pre-existing clusters, as part of the proposed MAS-DR framework. The performance of this service was analyzed using a confusion matrix, which provides a detailed breakdown of the classification results (Figure 12).

The results indicate that the classification service for Service 6 achieves exceptional performance. The overall accuracy of 98.55% demonstrates the model’s ability to effectively classify users into the correct clusters. The high precision (97.69%) for Class 1 shows that very few instances of Class 0 were misclassified as Class 1. Additionally, the recall (95.48%) highlights the model’s strong ability to detect most of the actual Class 1 instances. The F1-Score of 96.95% confirms a well-balanced trade-off between precision and recall, while the specificity of 99.38% emphasizes the model’s effectiveness in correctly identifying Class 0. The minimal false positives and false negatives further underscore the robustness of the classification model. Overall, Service 6 successfully integrates new users into the existing segmentation scheme, offering a reliable solution for expanding DR strategies with minimal performance loss.

The performance of the Random Forest classifier used in Service 6 is further quantified in

Table 13. Random Forest classification report.

	Precision	Recall	F1-Score	Support
Cluster 0	0.99	0.99	0.99	813
Cluster 1	0.98	0.95	0.97	221
Accuracy			0.99	1034
Macro Avg	0.98	0.97	0.98	1034
Weighted Avg	0.99	0.99	0.99	1034

, which summarizes the precision, recall, F1-score, and support for each cluster. The classification results show outstanding performance for both clusters, with overall accuracy reaching 99%. Specifically, the macro average precision, recall, and F1-score are 0.98, 0.97, and 0.98, respectively, indicating the model’s balanced performance across both clusters. The weighted average closely mirrors these values, further confirming that the model maintains high accuracy even with a class imbalance, where Cluster 0 significantly outweighs Cluster 1.

Figure 13 depicts the ROC curve for the classification model, with an Area Under the Curve (AUC) of 0.97. This high AUC value signifies the model’s excellent discriminative power in correctly identifying both clusters. The curve stays close to the top-left corner, indicating a strong true positive rate (TPR) with a minimal false positive rate (FPR). The True Positive Rate (Sensitivity) remains high across all thresholds. The model outperforms random guessing (represented by the diagonal line), highlighting its robustness and reliability.

Overall, these findings indicate that Service 6 is highly effective at classifying new users into pre-existing clusters, ensuring accurate segmentation and enabling tailored DR strategies for residential users.

6. Discussion

The MAS-DR framework represents a significant advancement in the field of residential energy management by integrating machine learning techniques for clustering, classification, and feature extraction. This work demonstrates the utility of unsupervised learning methods in analyzing consumption patterns, enabling energy managers to make informed decisions for Demand Response (DR) strategies. One of the key strengths of the MAS-DR framework is its ability to combine multiple clustering algorithms and evaluation metrics, providing robust segmentation insights. The integration of the Household Daily Load (HDL) concept adds an additional layer of temporal analysis, allowing stakeholders to design season-specific interventions. This contrasts with traditional optimization-based methods, as MAS-DR focuses on behavioral insights and adaptability to dynamic data.

The case study validates the framework’s effectiveness in segmenting users based on real-world energy data. However, there are limitations that warrant further exploration. For instance, while the framework effectively identifies user clusters and anomalies, the computational complexity of clustering algorithms like Spectral Clustering can pose challenges for large-scale datasets. Moreover, the framework relies on pre-defined thresholds in certain services, which may need dynamic tuning for diverse applications. Future improvements could address these limitations by exploring advanced clustering methods that scale better with data size and integrating adaptive mechanisms for threshold determination. Additionally, incorporating predictive capabilities, such as forecasting consumption patterns or detecting future anomalies, could further enhance the framework’s utility. Moreover, future work may consider methodologies to integrate uncertainty modeling in the analysis and segmentation processes, as highlighted in the work of Haopeng Zhang et al. [62]. Incorporating these concepts can enable the framework to provide more robust insights in the face of fluctuating and incomplete data. Additionally, the computational complexity of certain algorithms, such as Spectral Clustering, poses challenges for scalability with large datasets. Future enhancements will address these limitations by integrating real-time data processing and exploring more scalable methodologies.

Overall, The MAS-DR framework complements existing optimization-based models by offering a data-driven perspective on user segmentation and behavior analysis. Its deployment in real-world scenarios could revolutionize residential DR strategies, paving the way for smarter and more sustainable energy systems.

7. Conclusions

In this paper, the MAS-DR framework was introduced as a machine learning-based tool designed to provide aggregation and segmentation services for residential energy consumption users, assisting decision makers in implementing Demand Response (DR) strategies. The framework integrates clustering, classification, and feature extraction techniques to derive actionable insights into user consumption patterns.

The study demonstrated that the framework effectively segments users using a variety of clustering algorithms, such as K-Means, Hierarchical, and Spectral clustering, alongside evaluation metrics like the Silhouette Score, Davies–Bouldin Index, and Calinski–Harabasz Index. The Household Daily Load (HDL) concept further enables temporal analysis at monthly and seasonal levels, capturing fine variations in user behavior. Spectral Clustering exhibited the highest clustering quality, identifying two primary user clusters representing stable and variable consumption patterns. Regarding behavioral analysis, over 94% of users displayed consistent or predictable daily consumption behavior, making them suitable candidates for DR programs. Furthermore, approximately 50% of users demonstrated evening peak consumption, providing actionable insights for targeted load-shifting strategies to reduce peak demand. A small subset of users (0.1%) displayed extreme consumption patterns and anomalies, highlighting potential risks to grid stability. The Random Forest algorithm achieved high accuracy in assigning new users to pre-existing clusters, showcasing the framework’s robustness in handling new data.

Future work regarding the development of the framework will focus on integrating new technologies, such as deep learning for user segmentation, consumption forecasting and anomaly detection. In addition, the development of a user interface (UI) would benefit the framework’s users to visualize the results. Furthermore, in order to further analyze the framework’s usefulness, it could be validated on more datasets, either more diverse ones that include different climatic conditions or from different ministries. The framework can be easily adapted to include real-time measurements, but in its current format, most of the services that it provides focus on the historical consumption of the users in order to identify key insights. Real time could be incorporated into the framework for dynamic DR program adjustments, further enhancing grid resilience and consumer engagement. Finally, it is important to note that the overall purpose of the framework is the analysis of a user’s consumption, serving as a preliminary step in the design of DR strategies. The external factors (such as weather changes, tariff policy adjustments, socio-economic activities, etc.) and the residential users’ feedback, recommendations, and needs are really important in the design process, and could be added as a next step in the DR design process.