2.1. Materials
Two datasets were selected for evaluation. The first contained data on retail customers and was obtained from [
14]. It is a set that subjects itself well to clustering. In comparison, the second set containing wholesale customer data from [
15] is a set in which clustering makes no substantive sense.
2.1.1. Set Containing Data on Retail Customers
The dataset consists of 2240 customers. These customers are characterized by characteristics such as the customer’s ID, the customer’s birth year, the customer’s education level, the customer’s marital status, the customer’s annual household income, the number of children in the customer’s household, the number of teenagers in the customer’s household, the number of days since the last purchase made by the customer, the fact that the customer has complained, the amount of money the customer has spent on wine, fruit, meat, fish, candy, gold in the last 2 years, the number of purchases made at a discount, the use of six different campaign offers, the number of purchases made through the website, using a catalogue or directly in the store by the customer and the number of visits to the website by the customer in the last month. The majority of the dataset consists of married and highly educated customers.
2.1.2. Set Containing Data on Wholesale Customers
The set contains information on 701 business resellers, characterized by the reseller’s ID, business type, number of employees in the company, frequency of orders (defined by categories), the month of ordering, year of first ordering, year of most recent ordering, most frequently ordered product line, annual sales, type and amount of minimum payment, annual turnover, country name, city name, state or province name, the ratio of the sales value to the number of products ordered in the categories of bicycles, clothing, accessories, the ratio of the sales value to the number of products ordered in the various categories of promotions and without promotions, the average number of days to purchase a new product after it enters the market, the minimum number of days to purchase a new product after it enters the market, the maximum number of days to purchase a new product after it enters the market, the average delivery cost, the standard deviation of the delivery cost, the minimum delivery cost and the maximum delivery cost. Based on the City and StateProvinceName variables, information about the city’s population and the region’s GDP per capita was added to the set.
2.2. Methodology
The aim of this subsection is a detailed description of the methods we used in the following part. Specifically, the cluster analysis is outlined, and the classical k-means method, the DBSCAN method and three graph clustering algorithms (multilevel, greedy, label propagation) are described. Furthermore, the methods for determining the optimal number of clusters and tools for assessing the quality of the resulting clustering are presented.
2.2.1. Cluster Analysis
Cluster analysis aims to divide a dataset
composed of
n-dimensional feature vectors
into clusters (subsets, groups, classes) with respect to a certain criterion [
16]. The division must meet two assumptions:
We obtain a division of the set O consisting of N elements into k clusters and the matrix , whose successive elements are the belonging degrees of the element to the group . In the classical cluster analysis, a hard split is used in which each element belongs to only one group. Due to the clustering methods, cluster analysis algorithms are most commonly divided into hierarchical and nonhierarchical.
2.2.2. k-Means Clustering
The
k-means algorithm is one of the most widely used clustering algorithms [
16]. It clusters the data by attempting to separate the observations into
k groups to minimize the sum of squared distances between each point and its nearest cluster centre. Cluster centres are called centroids.
The k-means algorithm can be described in four steps:
- 1.
Choose k initial centroids: .
- 2.
For each , choose a cluster as a subset of all points that are closest to a given centroid .
- 3.
For each
, set a new centroid
for all points in cluster
as
- 4.
Repeat steps 2 and 3 until the set of does not change.
The method depends on the initiation of centroids. The most commonly used distance measures are:
Although the k-means algorithm is widely used, one must remember that it actually always partitions the data, regardless of its properties. One can use the k-means algorithm to partition uniform noise into k clusters in the sense of minimizing the Euclidean or other distances, without having a requirement for the clusters to be meaningful.
2.2.3. DBSCAN Clustering
The DBSCAN (density-based spatial clustering of applications with noise) algorithm [
17] was designed to extract clusters and noise in a spatial data set. This algorithm has two parameters:
and
, where
defines the distance from a given point
p such that each point
q, for which
, belongs to the neighbourhood of point
p, denoted by
;
determines the minimum number of points that must be in the cluster.
In a cluster, one can distinguish between two types of point: core points within the cluster and border points at the edge of the cluster. Two concepts are introduced regarding points in the cluster: directly density-reachable and density-reachable.
The point p is directly density-reachable from point q at a given and , if
- 1.
;
- 2.
(core points condition).
The direct density reachability is symmetrical for pairs among core points, but is not symmetrical for mixed pairs of core and border points. The point p is density-reachable from point q at a given and , if there is a chain of points , and , such that is directly density-reachable from point . This relationship is transitive but not symmetrical. The two border points of a cluster are not within their density reachability. Therefore, the concept of density-connected points is also introduced.
The point p is density-connected to the point q at a given and , if there is a point o such that both points p and q are density-reachable from point o at a given and .
Having introduced the above definitions, we can move on to a description of the DBSCAN algorithm itself. To find a cluster, DBSCAN starts from an arbitrary point p and selects all points within its density reachability at a given and . If p is a core point, this procedure yields a whole cluster. If p is a border point, then no point is density-reachable from p and the algorithm proceeds to the next point in the dataset.
Since global values for the and parameters are used, the algorithm can merge two clusters into one if they are “close enough” to each other. Let the distance between sets and be defined as . Then, two point sets having at least the rarest cluster density will only be separated if the distance between them is greater than . Consequently, it may be necessary to use the DBSCAN algorithm recursively for detected clusters with a higher value. However, this is not a disadvantage, as recursion leads to an elegant and efficient underlying algorithm. Moreover, recursive clustering is only necessary under a condition that is easy to verify.
The selection of and can be conducted by various methods, including checking the values of metrics available for unsupervised clustering, such as the elbow criterion or the silhouette criterion. The advantage of the DBSCAN algorithm is its applicability to spatial data and not having to determine the number of clusters in advance.
A more recent improvement to this algorithm is HDBSCAN [
18], which allows clusters of varying density. The algorithm works similarly to the original DBSCAN method but does not have an
parameter responsible for the cluster radius. Only the
parameter defining the minimum number of points in a cluster is responsible for the size of the clusters in this method. Both algorithms usually work well on data with noise, although the newer HDBSCAN algorithm is usually more efficient, and different cluster densities often produce a better solution. Since we obtained sufficient results with the DBSCAN method in the experimental part, we concluded that it was not necessary to compare it with the improved version of the algorithm.
2.2.4. Optimal Cluster Number Selection Methods
The clustering algorithms belong to the unsupervised learning group, which means that it is impossible to compare the results with the original classes to verify the clustering accuracy. Therefore, for this task, a number of coefficients exist to assess the clustering quality based on differences between observations in the same groups and between observations from different groups. At the outset, two commonly used methods are presented: the elbow method and the silhouette method.
The elbow method is based on measuring the within-cluster sum of squares errors (
) for a varying cluster number
k and then selecting a value
k, for which the change in
starts to decrease. The
is therefore defined as:
with
representing the
ith row of a data set having
n rows, and
representing the average value in the cluster.
The idea behind the elbow method is that the explained variability changes rapidly for a small number of clusters and then slows down, resulting in an elbow in the curve [
19]. The point at which the so-called elbow is formed indicates the number of clusters that should be optimal in a given clustering algorithm.
The silhouette coefficient is a slightly stricter method than the previously described one, but it is better at identifying relationships within and between groups. For consecutive observations
i belonging to the analysed data, the coefficient is defined as [
20]
whereby
is the smallest mean observation distance
i to all points in any other cluster, and
is the mean distance
i from all other points in the same group. For example, assuming we have three clusters
,
and
and observation
i belongs to cluster
, then
is calculated by measuring the mean distance
i from each point in cluster
, the mean distance
i from each point in cluster
and takes the smallest value. The silhouette coefficient for the dataset is the mean of the silhouette coefficient for individual points.
This measure indicates whether individual points are correctly assigned to their clusters. When using this coefficient, we can use the following interpretations:
close to 0 means that the point is between two clusters.
If is close to −1, then the studied point should be in a different cluster.
If is close to 1, then the point belongs to the “correct” cluster.
The described methods were supplemented by two less classic ones.
The Calinski–Harabasz index is based on the assumption that clusters are very compact and are sufficiently distant from each other to form a good distribution. The index is calculated by dividing the variance of the sums of squares of the distances of individual objects to their cluster centre by the sum of squares of the distance between the cluster centres. The higher that index value, the better the division. The formula for the Calinski–Harabasz index is defined as [
21]:
where
k is the number of clusters formed from a set of
n observations,
(
between cluster scatter matrix) determines the separation between groups and
(
Within cluster scatter matrix) determines the compactness within clusters. A higher value of the CH index means the clusters are dense and well-separated, although there is no “acceptable” cut-off value. We need to choose the solution which gives a peak or at least an abrupt elbow on the line plot of CH indices. On the other hand, if the line is smooth (horizontal, ascending or descending) then there is no such reason to prefer one solution over others.
Similarly to the silhouette method and the Calinski–Harabasz method, the Davies–Bouldin method tries to focus on both the separation and compactness of the clusters. This is due to the statement given in the maximum measure formula below repeatedly selecting values where the mean point is furthest from its centre and where the centres are closest to each other. However, in contrast to the silhouette and Calinski–Harabasz indexes, when the DB index decreases, it means an improvement in clustering. The Davies–Bouldin index is defined as [
22]:
where
k is the number of clusters, while
is the mean distance of all points in the
ith cluster from the
cluster centre, and
is the distance between the
ith and
jth cluster centroids.
The application of this method is similar to that of the CH index.
2.2.5. Graph Model
A graph is a mathematical structure consisting of a nonempty set V of objects called vertices and a family E of two-element sets of vertices called edges. These sets represent the presence of some relations between two vertices. For this purpose, we can assume that clients are vertices and two clients are joined by an edge if their features or shopping habits are similar. The graph of clients can be constructed in different ways depending on the structure of a data set. If only the information on products that were bought by specific clients is available, then we construct firstly an auxiliary graph of clients and products, where clients are joined with the product they bought. On this basis, we construct the main graph where two clients are joined with an edge if they bought the same product.
When looking for clusters, we are looking for similar clients. Firstly, we must define what it means for two clients to be similar. Similarity
is a function on the set of clients fulfilling the following conditions:
(or maximum similarity) if only
and
for all pairs
. There is no one universal similarity function (see e.g., [
23]). For the purpose of this research, the cosine similarity was applied [
24].
Let
and
be the vectors one wishes to compare. The similarity function expressed as a cosine measure is of the form:
where
is the scalar product of the vectors
x and
y.
The measure calculates the cosine of the angle between x and y. A cosine value of 0 means that the two vectors are at an angle of to each other (orthogonal) and are not similar. The closer the cosine value is to 1, the smaller the angle and the greater the similarity of the vectors.
Using the similarity function, a so-called similarity matrix can be constructed, where each entry is a value of the similarity function for an appropriate pair of customers. Based on that matrix, a customer graph can be constructed, in which an edge is placed between customers if their similarity function is sufficiently large. This cut-off threshold is a metaparameter of the construction. It may vary and depends on the data set. It is convenient that the output graph is connected, which means that there is a connection (not necessarily direct) between each pair of vertices.
2.2.6. Communities in Graphs
To search for a community in a graph is to divide the graph into disjoint subgraphs, which are characterized by a large number of edges inside the community and a small number outside. The structure of a community on the vertex set V is the division of the set V into a sequence of R subsets such that: and , with , . Subsets are such that are just communities in the graph.
There are many algorithms that create the aforementioned divisions. One of the parameters often used for this purpose is modularity [
25].
Modularity is a measure that takes into account the degree distribution of vertices in a network and is used to measure the strength of a network’s division into modules. Networks with a high modularity have dense connections between vertices in groups and sparse connections between vertices of different modules.
Modularity is the fraction of edges falling into given groups minus the expected fraction if the edges were randomly distributed. The value of this function is in the range . It is positive when the number of edges in the groups exceeds the number expected based on random selection. A modularity greater than 0.3 is assumed to mean that there are significant community divisions. The value 0.3 is not obligatory, but useful and easy to check.
There are a number of clustering algorithms. We provide a short description of those used in this research. We refer the reader to [
26] for more information.
2.2.7. Multilevel (Louvain) Algorithm [27]
- 1.
Initially, each node is assigned to a different cluster. The number of clusters is equal to the number of nodes.
For each neighbour j of node i, it is checked whether the total modularity increases by moving node i from its cluster to the cluster of node j. Node i is moved to the cluster of node j for which the modularity increment is the largest (positive). If none of the observed increments is positive, node i remains in its cluster.
Step two is repeated sequentially for all nodes. This is called a single iteration. The iteration is also repeated until modularity improvement cannot be achieved. Note that a node can be and usually is visited more than once.
- 2.
The second step is to rebuild the new network by grouping the nodes labelled in the same community (merging individual nodes) in step 1.
The edge weights between two new communities (containing more than 1 node) are determined by summing the edge weights from each node in one community to the other. From nodes in the same community, the edges form loops. The diagonal elements in the new neighbourhood matrix are nonzero.
One iteration of steps 1 and 2 is called a transition. After the first transition, the algorithm attempts a second transition and so on. The number of communities decreases after each transition. This continues until no changes in the communities are observed and maximum modularity is reached.
2.2.8. Greedy Algorithm [28]
The fast greedy algorithm involves optimizing the modularity factor. Initially, the vertices are not grouped and each vertex is a single community. Calculating the modularity improvement for the entire community and using the neighbourhood matrix makes it time-consuming to find the pair with the largest modularity increase (as in the multilevel algorithm). Instead, the expected modularity improvement is calculated for each pair of vertices. According to the modularity assumption, those pairs that achieve the greatest modularity improvement are combined into a new group. This procedure is repeated as long as the new mergers result in an increase in modularity.
2.2.9. Label Propagation Algorithm [29]
At the beginning of a given step, nodes have a label indicating the community to which they currently belong. Community membership changes based on the labels that neighbouring nodes have. This change is determined by the maximum number of labels within a node’s neighbours. Initially, each node is initialized with a unique label, and then the labels spread through the network. Consequently, densely connected groups quickly reach a common label. The main step is as follows
For each
from the sequence, let
where
are the neighbours of
x, of which
m obtained a label in the current step, and the others have labels from the previous step. The function
f returns the label occurring with the highest frequency among its neighbours. If there are multiple labels with the highest frequency,
is chosen randomly among them.
2.2.10. Community Comparison
In general, to perform proper clustering of a particular graph, partitioning is carried out using various algorithms. One of the parameters for evaluating partitioning is the aforementioned modularity.
To compare splits among themselves, the normalized mutual information parameter
(normalized mutual information) [
30] can be used.
Let ; denote two divisions of V into communities. A common part of A and B can be expressed as a contingency matrix C of size , where denotes the number of common vertices of and ().
The normalized mutual information parameter is defined as:
where
and
.
Parameter has its values in the interval . If communities A and B are identical, then , in the lack of similarity .
So, if the partition in clusters is repeated or similar in several algorithms, it can be considered appropriate.
The Fowlkes–Mallows Index (FMI) can also be used to compare the similarity of two clusterings [
31]. It is defined as the geometric mean of precision and recall:
where
is the number of pairs of points that belong to the same groups in both clusterings,
the number of pairs of points that belong to the same groups in the first clustering but not in the second,
the number of pairs of points that belong to the same groups in the second clustering but not in the first.
The index takes values between 0 and 1. The higher the value, the more similar the two clusterings.
2.2.11. Proposed Methodology
Clustering comparisons were made on two datasets: resellers and customers. First, embeddings for categorical variables were generated with the use of an autoencoder. An autoencoder is a neural network designed to encode input data in the form of the so-called hidden representation and then decode them in such a way that the reconstructed data are as similar to the original ones as possible. It is worth noting that most embedding generation methods require knowledge of the target variable. Using an autoencoder removes that need, since the loss function is calculated by comparing the reconstruction with the input data. A diagram of how an autoencoder combined with an embedding layer, known as AutoEmbedder, works is shown in
Figure 1. A similar usage of autoencoders can be seen in [
6,
32]. In the case of continuous variables, a transformation to a categorical variable was performed before their inclusion in the model by determining classes (ranges) of variable values. Embeddings were then generated for the categories using the neural network’s embedding layer. The embeddings were the input data for the autoencoder.
AutoEmbedder was trained on the training set. Once it had been trained, the embedding layer and encoder were used to generate the embeddings. A principal component analysis (PCA) [
33] was then performed on the coded data, assuming that 95% of the variance remained explained. After that procedure, each observation was in the form of a real number vector. For the observations coded as described above, with the selection of an appropriate similarity measure, we generated a similarity matrix. Further on, we proceeded to select clustering algorithms, including graph algorithms. The methodology is graphically illustrated by
Figure 2.
As a similarity metric, a cosine similarity scaled to a range from 0 to 1 was chosen.
The following algorithms were used to find groups of similar customers: k-means, DBSCAN, Louvain community, greedy modularity, label propagation. The last three are graph methods. For their implementation, a graph needed to be created to represent the relationship between customers in the set. When building a graph representing the relationship between the most similar customers, it was assumed that the graph was consistent. The graph was constructed based on the similarity matrix. Using the cut-off parameter, the creation of edges between similar clients was controlled. An edge was created between clients for which the similarity between them was greater than or equal to the set cut-off parameter. The cut-off parameters we tested were , and .
The resulting clusters were compared using metrics available for unsupervised clustering. Calinski–Harabasz and Davies–Bouldin indices were calculated for the clusterings, which helped to select the most optimal clustering. In order to compare how similar the two clustering data were, the NMI parameter (normalized mutual information) and the Fowlkes–Mallows index could be used. Moreover, for graph methods, one could compare their modularity. When determining the optimal cluster number in the k-means algorithm, we used the results of the elbow and silhouette methods and the Calinski–Harabasz and Davies–Bouldin index values. In DBSCAN clustering, the parameters were selected based on the silhouette criterion.
2.2.12. Computational Complexity
Firstly, AutoEmbedder was trained on the training dataset. Then, to encode categorical variables, only the embedding and encoder parts of AutoEmbedder were used.
If there were p categorical features given, and each of them had levels (), then the embedding layer generated values for each layer, resulting in a vector of size for each of the n samples in the dataset. The encoder consisted of dense layers with a ReLU activation function. If there were l layers with () nodes each, the complexity of calculating the prediction for one sample in each dense layer would be , where would be the size of the input—in the presented case, it was because of the embedding layer put before the encoder.
Standard PCA involves
operations, where
n is the number of samples and
p is the number of features for each sample [
34]. In our case, since the number of samples was much higher than the number of features, the time to complete the PCA algorithm was
. The complexity of computing the similarity matrix was
. The clustering algorithms used had the following complexities:
The Louvain algorithm is believed to run in
time, where
m is the number of edges in the graph [
35],
In the greedy modularity algorithm, in each step, the pair of nodes/communities that, joined, increase modularity the most, become part of the same community, and then the modularity is calculated for the full network. This step requires
calculations. After merging communities, the adjacency matrix of the network is updated in a worst-case scenario in
time. Each merging event is repeated
times. Hence, the overall complexity is
, or
in a sparse graph [
36],
The label propagation algorithm has
computational complexity [
37],
The DBSCAN algorithm scans the whole dataset only one time and needs to calculate the distance of any pair of objects in the dataset. Hence, the worst possible computational complexity of the whole algorithm is
[
38]. In our case, the distances did not need to be computed, because we took distances as one minus the value from the similarity matrix.
From the above analysis, it can be seen that the worst possible computational complexity occurs with the PCA algorithm, where it amounted to . Therefore, we could conclude that the worst possible complexity for the proposed methodology was .