Fairness First Clustering: A Multi-Stage Approach for Mitigating Bias

Abstract

Fair clustering aims to partition a dataset while mitigating bias in the original dataset. Developing fair clustering algorithms has gained increasing attention from the machine learning community. In this paper, we propose a fair k-means algorithm, fair first clustering (FFC), which consists of an initialization stage, a relaxation stage, and an improvement stage. In the initialization stage, k-means is employed to cluster each group. Then a combination step and a refinement step are applied to ensure clustering quality and guarantee almost fairness. In the relaxation stage, a commonly used fairness metric, balance, is utilized to assess fairness, and a threshold is set to allow for fairness relaxation while improving the clustering quality. In the improvement stage, a local search method is used to improve the clustering quality without changing the fairness. Comparisons of fairness and clustering quality are carried out between our method and other state-of-the-art fair clustering methods on 10 datasets, which include both synthetic and real-world datasets. The results show that compared to the method with the second highest balance value, FFC shares the same SSE value on one dataset and achieves lower SSE values on six datasets.

1. Introduction

Clustering is a fundamental technique in unsupervised machine learning and data mining that aims to divide a dataset into several clusters without any information. A lot of clustering algorithms have been proposed over the past few decades [1,2,3,4,5] and applied in many real-world applications, such as face detection [6], recommendation systems [7], information retrieval [8], and so on.

Cluster analysis is a widely used technique in the electronics domain. Li et al. [9] proposed a simplified electrochemical impedance spectroscopy model based on k-means to describe the battery characteristics of electric vehicles. Jinlei et al. [10] developed an active equalization strategy for series-connected battery packs based on clustering analysis with a genetic algorithm. Lei et al. [11] employed hierarchical clustering to cluster four kinds of VOCs detected by a fluorescent cross-responsive sensor array and obtained accurate results. Hwang and Kim [12] implemented the Gaussian mixture model to a variational autoencoder framework to cluster wafer maps in semiconductor manufacturing. Recently, researchers have paid more attention to fairness in clustering [13].

Traditional clustering algorithms often take the internal structure of the data into account, but ignore the bias in the data, which is associated with protected attributes (e.g., race and gender). In the process of clustering, the bias can be transmitted, or even amplified, and affect the clustering result. In bioelectronics, clustering methods can be used to analyze single-cell RNA-Seq data. Cell sequencing techniques can be considered the protected attribute since cells sequenced by different techniques would result in different expression levels [14]. Traditional clustering methods may perform well on cell sequences obtained by a specific sequence technique while struggling with the data obtained by others. Take feature extraction for face emotion recognition in electronic multimedia as another showcase; clustering methods can be used to group features that give the shape and location of the important biometric parts of the face, such as the eyes, nose, and mouth, based on similarity [15]. In this context, race can be considered the protected attribute, and there may be inherent physical differences in facial features between races. Traditional clustering-based feature extraction may perform well for one race but less well for other races. Therefore, the issue of fairness of the clustering should be considered in most domains where clustering is employed [16]. Such bias can be mitigated by adding fairness constraints that hide protected attributes from the clustering results. Generally, fairness in clustering requires that objects with different values of protected attributes have a uniform distribution in each cluster [17,18,19].

In order to achieve fair clustering, much effort has been made to incorporate fairness constraints into clustering [20,21,22,23,24,25]. They can be divided into: (1) pre-processing, (2) in-processing, and (3) post-processing according to when the fairness is handled. Chierichetti et al. [20], Backurs et al. [21], and Ahmadian et al. [22] partitioned the original dataset into small clusters where fairness is guaranteed and then applied clustering algorithms (e.g., k-center, k-median, and hierarchical clustering) on them to achieve the fair results. Hence, these works are all pre-processing-based methods. Both Kleindessner et al. [23] and Liu and Vicente [24] are in-processing-based methods. The former formulated fairness constraints and incorporated it into the spectral clustering objective function; the latter added the steps of adjusting the fairness of clusters to mini-batch k-means. Bera et al. [25] first employed a vanilla clustering algorithm and then changed the assignment of objectives to the clusters to improve fairness, so it is a post-processing-based method.

In general, enforcing fairness on clustering will lead to a loss of clustering quality. This is because the pursuit of fairness may result in the allocation of closely related objects to different clusters to mitigate bias. Such assignments can decrease the performance of the clustering. For example, post-processing-based methods often utilize the initial unfair clustering results obtained by vanilla clustering algorithms, which are expected to have high clustering quality. The fair clustering result generated by the fairness operation is usually of lower clustering quality than the original unfair one.

In this paper, we take a different path that modifies an initial fair clustering result rather than an unfair one. The method we propose consists of three stages: an initialization stage that obtains a fair clustering output, a relaxation stage that allows a certain degree of unfairness to achieve better clustering quality, and an improvement stage that improves clustering quality. Additionally, different from the post-processing-based approach that improves fairness while retaining clustering quality, our idea of an improvement stage is to improve clustering quality while maintaining fairness. The main contributions in this paper can be summarized as follows:

• An initialization method that can handle the case of multiple groups is presented to generate an initial fair clustering result, and, compared to random initialization, the result is of relatively good clustering quality.
• A local search method is employed to improve clustering quality. Specifically, the local search method can find a local optimum of the initial clustering result.
• Experimental results on both synthetic and real-world datasets demonstrate that our proposed method outperforms some of the other fair clustering methods in terms of fairness and clustering metrics.

2. Related Work

2.1. Fair Clustering

Chierichetti et al. proposed the first work on fair clustering [20]. They added fairness constraints for k-center and k-median clustering in the case of two protected groups. They presented the fairness notion of balance, which requires that two protected groups must have approximately equal representation in each cluster. To achieve fair clustering, they introduced the fairlet decomposition, which constructed small and balance subsets (i.e., fairlets). Classical clustering algorithms were applied to fairlets to obtain a fair clustering result. To scale to large datasets, Backurs et al. proposed a tree-metric-based nearly linear time algorithm for fairlet decomposition [21]. Rösner and Schmidt allowed multiple protected groups and obtained a constant approximation algorithm for the k-center clustering [26]. Bera et al. introduced the fairness notion of bounded representation, which adds upper and lower bounds to protected groups in clusters [25]. They first solved the vanilla clustering problem and then improved the fairness by fairly assigning objects to cluster centers. Ziko et al. proposed a general variational framework of fair clustering by integrating a fairness penalty term based on Kullback–Leibler (KL) divergence into the classic clustering methods [27]. Esmaeili et al. treated an upper bound on the clustering objective as a constraint and minimized a measure of unfairness [28].

Beyond center-based clustering, fairness has been achieved in other clustering algorithms. Kleindessner et al. incorporated fairness constraints into spectral clustering [23]. Ahmadian et al. extended fairness to hierarchical clustering under different hierarchical clustering objectives [22]. Li et al. explored fairness in deep clustering for large-scale and high-dimensional visual learning and developed a deep fair clustering model to learn the feature representations that are favorable for clustering and hiding protected attributes [29]. Chhabra et al. proposed a novel black-box attack against fair clustering algorithms where the attacker can perturb a small percentage of protected group memberships and a novel fair clustering approach that utilizes consensus clustering along with fairness constraints to output robust and fair clusters [30].

The works by [31,32] take the results of a vanilla clustering algorithm as the input and find a clustering close to the input while satisfying the fairness constraints. Both formulated the problem as an integer linear program and use ILP solver and LP solver to solve it, respectively.

2.2. K-Means Clustering

K-means clustering is a commonly used clustering method due to its simplicity. The goal of k-means clustering is to partition a dataset into k clusters, where each object is assigned to the cluster with the nearest cluster center. Formally, let $X=\{x_1,x_2,\dots ,x_n\}$ be a dataset with point $x_i={(x_{i1},\dots ,x_{ld})}^T\in {\mathcal{R}}^d$, k-means clustering aims to partition X into k clusters $C=\{C_1,C_2,\dots ,C_k\}$ to minimize the sum of the squared error (SSE) [33]. The SSE can be computed as the sum of the squared Euclidean distances of each point to its closest center:

$$SSE=\sum _{j=1}^k\sum _{x_i\in C_j}{\parallel x_i-c_j\parallel }^2,$$

(1)

where $c_j\in {\mathcal{R}}^d$ is the center of cluster $C_j$.

The most popular implementation of k-means clustering is Lloyd’s algorithm [34,35]. It starts by randomly choosing k points as centers. After that, it alternates between the two steps:

Assignment Step: Assign each point to the cluster with the nearest center:

$$C_j^{\left(t\right)}=\{x_i:\parallel x_i-c_j^{\left(t\right)}\parallel <\parallel x_i-c_p^{\left(t\right)}\parallel \forall p,1\le p\le k\}.$$

(2)

Update Step: Recalculate the center of each cluster:

$$c_j^{(t+1)}=\frac{1}{|C_j^{\left(t\right)}|}\sum _{x_i\in C_j^{\left(t\right)}}{x}_i.$$

(3)

Lloyd’s algorithm converges when no point changes its assignment or the maximum number of iterations is reached. It always converges to a local optimum but not necessarily to the global optimum.

Hartigan’s algorithm is another heuristic algorithm for k-mean clustering, but it is somewhat different from Lloyd’s algorithm [36]. It pays attention to a important fact: if you add a point to a cluster, the center will change. It starts from a random partition of X into k clusters and calculates the center of each cluster. For a point $x_i$, Hartigan’s algorithm calculates the change in Equation (1) if $x_i\in C_i$ is moved from $C_i$ to $C_j$:

$$\begin{array}{cc}\hfill & \Delta {SSE}_{x_i\to C_j}\hfill \\ \hfill =& \sum _{x\in C_i}{\parallel x-c_i\parallel }^2+\sum _{x\in C_j}{\parallel x-c_j\parallel }^2\hfill \\ \hfill & -\sum _{x\in C_i^{{}^{\prime }}}{\parallel x-c_i^{{}^{\prime }}\parallel }^2-\sum _{x\in C_j^{{}^{\prime }}}{\parallel x-c_j^{{}^{\prime }}\parallel }^2,\hfill \end{array}$$

(4)

$$c_i^{{}^{\prime }}=c_i+\frac{c_i-x_i}{|C_i|-1}\mathrm{and}{c}_j^{{}^{\prime }}=c_j+\frac{x_i-c_j}{|C_j|+1},$$

(5)

where $C_i^{{}^{\prime }}=C_i\setminus \left\{x_i\right\}$, $C_j^{{}^{\prime }}=C_j\cup \left\{x_i\right\}$ and $c_i^{{}^{\prime }}$ and $c_k^{{}^{\prime }}$ are the centers of $C_i^{{}^{\prime }}$ and $C_j^{{}^{\prime }}$, respectively. Then, $x_i$ is assigned to the cluster with the maximum decrease in the objective function value. Hartigan’s algorithm repeatedly picks a point and assigns it until Equation (4) is less than zero for all points.

3. Proposed Method

This section presents the fair k-means algorithm and describes the initialization method and the improvement for fairness.

3.1. Initialization

Fair clustering aims to ensure that the proportions of protected groups in each cluster are close to the proportions in the whole dataset [23,26,27,37]. Consequently, data points from different groups can be assigned the same clustering label. However, vanilla k-means clustering without consideration for protected attributes usually divides the dataset in such a way that similar points are grouped into the same cluster and conducts an unfair solution. In our study, we make the assumption that if data points belonging to the same group are similar, they should be placed in the same cluster in a fair clustering solution. Therefore, we propose an initialization strategy based on the k-means algorithm to ensure clustering quality and guarantee almost fairness.

Suppose that dataset X can be divided into m disjoint groups $X=\{G_1,G_2,\dots ,G_m\}$ and $|G_1|>|G_2|>|\dots |>|G_m|$. We first run k-means on every group to obtain m clustering results, $C^1,\dots ,C^i,\dots ,C^m$, where $C^i=\{C_1^i,C_2^i,\dots ,C_k^i\}$. $C^1$ is selected as the primary clustering result because $G_1$ has the most points compared with other groups. Then, we iteratively choose the clustering result corresponding to the largest group among the remaining groups as the secondary clustering result, refine the clusters obtained by combining the primary and secondary clustering results, and consider it the new primary clustering result. Figure 1 shows a complete initialization scheme on a dataset with 3 groups. The combination step and refinement step are described below.

3.1.1. Combination Step

Suppose we combine a primary clustering result $C^p=\{C_1^p,C_2^p,\dots ,C_k^p\}$ with a secondary clustering result $C^s=\{C_1^s,C_2^s,\dots ,C_k^s\}$, and $C^p$ contains more data points than $C^s$. In the context of clustering, cluster centers can be considered representatives of the clusters, as they are calculated as the mean of the data points within each cluster. To establish a one-to-one matching between the clusters in $C^p$ and $C^s$, we initially match the clusters based on their cluster centers. Specifically, let $M\left(C_i^p\right)\in C^s$ denote the cluster that has a match with cluster $C_i^p\in C^p$. We determine this matching by minimizing the following equation:

$$\mathsf{\Phi }\left(M\right)=\sum _{C_i^p\in C^p}d(C_i^p,M\left(C_i^p\right)),$$

(6)

where $d(·,·)$ represents the distance between two cluster centers. After the matching operation, k cluster pairs are obtained. Subsequently, each cluster pair is merged to generate a new primary clustering result $C^{p^{\prime }}=\{C_1^p\cup M\left(C_1^p\right),C_2^p\cup M\left(C_2^p\right),\dots ,C_k^p\cup M\left(C_k^p\right)\}$. The algorithm for combining the primary in Algorithm 1 and the secondary clustering results is summarized as follows:

Algorithm 1 Combine Algorithm

Input:  Primary clustering result $C^p=\{C_1^p,C_2^p,\dots ,C_k^p\}$, secondary clustering result $C^s=\{C_1^s,C_2^s,\dots ,C_k^s\}$. Output:  New primary clustering result.  1: Compute cluster centers of $C^p$ and $C^s$;  2: Compute all permutations of clusters in $C^p$;  3: For each permutation, calculate the value of Equation (6) when performing a one-to-one matching between the clusters in the permutation and the clusters in $C^p$;  4: Select the one-to-one matching with the minimum value of Equation (6);  5: Merge the matched cluster pairs of the one-to-one matching obtained in Step 4 to generate a new primary clustering result.

3.1.2. Refinement Step

Suppose that $C^s$ is produced by running k-means on $G_s$, and the data points in $C^p$ are considered to belong to the same group $G_p$. In the case of two groups, we measure the fairness of a cluster by calculating the ratio of the number of data points belonging to $G_s$ to the number of data points belonging to $G_p$ within the cluster. After the combination step, the fairness of the merged cluster $C_i^p\cup M\left(C_i^p\right)$ is formally defined as:

$$b(C_i^p\cup M\left(C_i^p\right))=\frac{\{C_i^p\cup M\left(C_i^p\right)\}\cap G_s}{\{C_i^p\cup M\left(C_i^p\right)\}\cap G_p}=\frac{|M(C_i^p)|}{|C_i^p|}.$$

(7)

The closer the value of Equation (7) is to $\frac{|G_s|}{|G_p|}$, the fairer the cluster is. However, it can be observed that almost all merged clusters are unfair. The reason is that we do not impose extra constraints, such as cluster size, during the process of k-means and the combination step to ensure that merged clusters maintain the distribution proportions of protected groups in the dataset. To cope with this challenge, a refinement step is designed.

The refinement step iteratively handles the cluster with the poorest fairness to make it nearly completely fair. Let $C_l^{p^{\prime }}=\{C_l^p\cup M\left(C_l^p\right)\}$ be the cluster with the lowest value for Equation (7). An intuitive approach to improve the fairness of $C_l^{p^{\prime }}$ is to move some data points belonging to $G_s$ to it. Specifically, we move these data points from a cluster $C_t^s\in C^s$ since the data points in $C_t^s$ all belong to $G_s$. We determine $C_t^s$ as the nearest unfair neighbor of $M\left(C_l^p\right)$ and move the $\frac{|G_s|\times |C_l^p|}{|G_p|}-|M\left(C_l^p\right)|$ data points that are closest to the center of $M\left(C_l^p\right)$. These two considerations will lead to a smaller increase in clustering cost, on the one hand, and make $C_l^{p^{\prime }}$ fair on the other hand. The algorithm of the refinement step in Algorithm 2 is summarized as follows:

Algorithm 2 Refine Algorithm

Input:  The primary clustering result $C^{p^{\prime }}$ obtained from the combination step. Output:  New primary clustering result.   1: Mark each cluster as unfair;   2: while the number of unfair clusters is greater than 1 do   3:     Select $C_l^{p^{\prime }}=\{C_l^p\cup M\left(C_l^p\right)\}$ as the cluster with the lowest value of Equation (7);   4:     Select $C_t^s$ as the nearest unfair neighbor of $M\left(C_l^p\right)$;   5:     Move the $\frac{|G_s|\times |C_l^p|}{|G_p|}-|M\left(C_l^p\right)|$ data points closest to the center of $M\left(C_l^p\right)$ from $C_t^s$ to $C_l^{p^{\prime }}$;   6:     Mark $C_l^{p^{\prime }}$ as fair;   7: end while

Consequently, the initialization stage is described in Algorithm 3.

Algorithm 3 Initialization

Input:  Dataset $X=\{G_1,G_2,\dots ,G_m\}$, cluster number k Output:  Clustering result.   1: Apply k-means to m groups to achieve m clustering results, $C^1,\dots ,C^i,\dots ,C^m$, where $C^i=\{C_1^i,C_2^i,\dots ,C_k^i\}$;   2: Primary clustering result $C^p=C^1$;   3: for $i=2,3,\cdots ,m$ do   4:     Secondary clustering result $C^s=C^i$;   5:     Apply Algorithm 1 to combine $C^p$ and $C^s$ and obtain a new primary clustering result $C^{p^{\prime }}$;   6:     Apply Algorithm 2 to refine $C^{p^{\prime }}$;   7:     Take the refined $C^{p^{\prime }}$ as the primary clustering result $C^p$;   8: end for   9: The final clustering result $C^p$;

3.2. Relaxation

The clustering result is almost exactly fair after the initialization stage. However, in certain cases, there is a desire to obtain a relatively relaxed fair clustering outcome. Specifically, when using a fairness measure to assess the fairness of clustering results, it is expected that the value of the fairness measure exceeds a predetermined threshold, rather than strictly achieving exact fairness.

Our proposed method relaxes the fairness constraint under a fairness measure called balance. Balance was first introduced by Chierichetii et al. [20] and generalized by Bera et al. [25]. It is measured by the ratio of different groups in each cluster. Suppose that $r(j,t)=\frac{|G_t\cap C_j|}{|C_j|}$ denotes the representation of group $G_t$ in cluster $C_j$, and $r\left(t\right)=\frac{|G_t|}{n}$ denotes the representation of group $G_t$ in the dataset. The balance of cluster $C_j$ is defined as:

$$Balance\left(C_j\right)=\underset{t\in m}{min}(\frac{r(j,t)}{r\left(t\right)},\frac{r\left(t\right)}{r(j,t)}),$$

(8)

The balance of a clustering result C is defined as:

$$Balance=\underset{j\in k}{min}Balance\left(C_j\right).$$

(9)

The value of the balance lies in the range from 0 to 1, with a higher value indicating a fairer clustering result. If the balance value of a clustering result is 1, it indicates that the clustering result is exactly fair.

To relax the fairness constraint, we introduce a parameter $\mu \in [0,1]$. The balance value of a clustering result is limited by $\mu$, that is, the balance value of each cluster should not less than $\mu$. We employ the greedy searching strategy based on moving an element from one cluster to another, similar to Hartigan’s algorithm. Given a point $x_i\in C_i$, which is moved to $C_j$, the movement will change the balance values of $C_i$ and $C_j$. If the balance values of $C_i$ and $C_j$ are still greater than or equal to $\mu$, this movement can be considered valid. We evaluate all valid movements and accept the movement with the maximum decrease the objective function value. We iteratively move points until there is no valid movement or until Equation (4) is less than zero for all valid movements.

By employing the greedy searching strategy, we can gradually optimize the objective function while maintaining a level of fairness. The algorithm of relaxation in Algorithm 4 is summarized as follows:

Algorithm 4 Relaxation

Input:  Dataset $X=\{x_1,x_2,\dots ,x_n\}$, clustering result $C=\{C_1,C_2,\dots ,C_k\}$, cluster number k. Output:  Clustering result,   1: repeat   2:     for each data point $x_i\in C_i$ do   3:         if $Balance(C_i\cap x_i)\ge \mu$ then   4:           for each clustert $C_j$ do   5:                if $Balance(C_j\cup x_i)\ge \mu$ then   6:                 Try to move $x_i$ to $C_j$ and calculate Equation (4);   7:                end if   8:           end for   9:           end if   10:          Move $x_i$ to the cluster with maximum decrease by Step 6;   11:          Update the centers of the clusters, where $x_i$ moves in and out by Equation (5);   12:      end for   13: until there is no valid movement or Equation (4) is less than zero for all valid movements

3.3. Improvement

In order to minimize the objective function value while maintaining fairness, a local search method based on interchanging two elements belonging to same group and different clusters is employed. Formally, give a set of k clusters belong to X

$$C=\{C_1,\dots ,C_a,\dots ,C_b,\dots ,C_k\},$$

(10)

the interchange of points $x_t\in C_a\cap G_e$ and $x_l\in C_b\cap G_e$ produces a new solution $C^{{}^{\prime }}=\{C_1,\dots ,C_a\setminus \left\{x_t\right\}\cup \left\{x_l\right\},\dots ,C_b\setminus \left\{x_l\right\}\cup \left\{x_t\right\},\dots ,C_k\}$. The change in the objective function value can be calculated by the following equation.

$$\begin{array}{cc}\hfill & {\Delta }_{C\to C^{{}^{\prime }}}\hfill \\ \hfill =& {\Delta }_{C_a\to C_a^{{}^{\prime }}}+{\Delta }_{C_b\to C_b^{{}^{\prime }}}\hfill \\ \hfill =& \sum _{x\in C_a}{\parallel x-c_a\parallel }^2-\sum _{x\in C_a^{{}^{\prime }}}{\parallel x-c_a^{{}^{\prime }}\parallel }^2\hfill \\ \hfill & +\sum _{x\in C_b}{\parallel x-c_b\parallel }^2-\sum _{x\in C_b^{{}^{\prime }}}{\parallel x-c_b^{{}^{\prime }}\parallel }^2,\hfill \end{array}$$

(11)

$$c_a^{{}^{\prime }}=c_a+\frac{x_l-x_t}{|C_a|}\mathrm{and}{c}_b^{{}^{\prime }}=c_b+\frac{x_t-x_l}{|C_b|},$$

(12)

where $C_a^{{}^{\prime }}=C_a\setminus \left\{x_t\right\}\cup \left\{x_l\right\}$, $C_b^{{}^{\prime }}=C_b\setminus \left\{x_l\right\}\cup \left\{x_t\right\}$ and $c_a^{{}^{\prime }}$ and $c_b^{{}^{\prime }}$ are the centers of $C_a^{{}^{\prime }}$ and $C_b^{{}^{\prime }}$, respectively.

Our local search strategy evaluates the interchange of every pair of data points that belong to same group and different clusters. It accepts the first interchange that results in an improvement of the objective function. The algorithm of improvement in Algorithm 5 is described as follows:

Algorithm 5 Improvement

Input:  Dataset $X=\{x_1,x_2,\dots ,x_n\}$, clustering result $C=\{C_1,C_2,\dots ,C_k\}$, cluster number k. Output:  Clustering result.   1: repeat   2:    $Found=False$;   3:    for $t=1,2,\dots ,n-1$ do   4:        for $l=t+1,\dots ,n$ do   5:            if $x_t$ and $x_l$ belong to different cluster and $x_t$ and $x_l$ belong to same groups, then   6:                Calculate Equation (11);   7:                if the value of Equation (11) is less than 0, then   8:                 Interchange points $x_t$ and $x_l$;   9:                 $Found=True$;   10:               end if   11:           end if   12:       end for   13:       if $Found=True,$ then   14:           break   15:        end if   16:     end for   17: until$Found=False$.

Finally, the overall algorithm of the proposed method in Algorithm 6 is summarized as follows:

Algorithm 6 FFC

Input:  Dataset $X=\{x_1,x_2,\dots ,x_n\}$, cluster number k. Output:  Clustering result.   1: Apply Algorithm 3 to generate an initial fair clustering result;   2: Apply Algorithm 4 to obtain a relatively relaxed fair clustering result;   3: Apply Algorithm 5 to improve the clustering quality while maintaining fairness, and the final clustering result is achieved.

3.4. Computational Complexity

The overall computational complexity of FFC can be evaluated as the sum of the computational complexity of each stage. In the initialization stage, the computational complexity of k-means is $O\left(n{I}_{1}kd\right)$, where $I_1$ is the number of iterations in the k-means. The computational complexities of the combination step and the refinement step are $O\left(\right(k+1)!d-kd)$ and $O(kd+nd)$, respectively. Then, the computational complexity of the initialization stage is $O\left(kd(n{I}_1+(k+1)!)\right)$. The computational complexity of the relaxation stage is $O\left(I_{2}nkd\right))$, where $I_2$ is the number of iterations in the relaxation stage. In the improvement stage, the computational complexity of the local search method is $O\left(I_{3}n(d+n)\right)$, where $I_3$ is the number of iterations in the improvement stage. To sum up, the computational complexity of our method is $O(kd(n{I}_1+n{I}_2+(k+1)!)+I_{3}n(d+n))$.

The computational complexity of vanilla k-means is $O\left(nIkd\right)$, where I is the number of iterations. It is lower than that of FFC since vanilla k-means do not account for fairness constraints. The computational complexity of fair spectral clustering [23] is $O\left({(n-m+1)}^3\right)$. The computational complexity of fair k-means proposed by Xu et al. [38] is $O(k(n^3+n^2)log\left(nh\right)+k^2{m}^2(m-1)(d+k))$, where h denotes the cost in the minimum cost flow they employed. Additionally, fair hierarchical agglomerative clustering, introduced by Chhabra et al. [39], has a computational complexity of $O\left(m{n}^3\right)$.

4. Experiments

In this section, we empirically evaluate the proposed algorithm on both synthetic and real-world datasets in terms of two clustering quality measurements and two fairness metrics.

4.1. Datasets and Experimental Setups

Datasets. The performance of our proposed method is evaluated on five synthetic and five real-world datasets.

Table 1. Dataset summaries.

Dataset	Instances	Features	Groups	Proportions
Elliptical	500	2	2	[0.5000, 0.5000]
DS-577	577	2	3	[0.3397, 0.3484, 0.3120]
2d-4c-no0	1572	2	4	[0.3359, 0.2214, 0.1730, 0.2697]
2d-4c-no1	1623	2	4	[0.3315, 0.1183, 0.3321, 0.2181]
2d-4c-no4	863	2	4	[0.4878, 0.0997, 0.3407, 0.0718]
Adult	3000	5	2	[0.6693, 0.3307]
Bank	4000	6	3	[0.1123, 0.2812, 0.6065]
Census1990	3000	25	2	[0.4847, 0.5153]
CreditCard	3000	14	2	[0.3963, 0.6037]
Diabetic	10,000	2	2	[0.4624, 0.5376]

provides a descriptive summary of the synthetic and real-world datasets. The synthetic datasets, including Elliptical, DS-577, 2d-4c-no0, 2d-4c-no1, and 2d-4c-no4, are commonly used in clustering studies. The Elliptical dataset is obtained from [40]. The DS-577 dataset is obtained from [41]. The datasets 2d-4c-no0, 2d-4c-no1, and 2d-4c-no4 are taken from Dr. Julia Handl’s work (https://personalpages.manchester.ac.uk/staff/Julia.Handl/data.tar.gz, accessed on 27 June 2023). Due to the absence of any explicitly protected attributes, we take the ground-truth label as the sensitive protected attribute for datasets DS-577, 2d-4c-no0, 2d-4c-no1, and 2d-4c-no4. For the Elliptical dataset, we divide it into two classes and take the ground-truth label of the two classes as the sensitive protected attribute. The synthetic datasets are shown in Figure 2. The shape of each object identifies the group it belongs to [14,29]. Regarding the real-world datasets, they are publicly available datasets from the UCI Machine Learning Repository (https://archive.ics.uci.edu/ml/datasets/, accessed on 27 June 2023) and have been used in previous fairness studies [25]. These datasets, such as Adult, Bank, Census1990, CreditCard, and Diabetic, cover diverse domains and exhibit various characteristics, including different feature dimensions and group structures.

Experimental Setups. Our proposed method is implemented in Python 3.9, and the source code is available on GitHub for public use (https://github.com/rebpan/FFC, accessed on 27 June 2023). Experiments are performed on a laptop with Windows 10 64-bit edition, which has a 2.90 GHz AMD Ryzen 7 4800H CPU and 32GB RAM. Each dataset is standardized by rescaling each attribute to have a mean of 0 and a standard deviation of unit variance. Then, L2 normalization is applied to each standardized dataset.

The baseline methods include vanilla k-means clustering (Lloyd), fair unnormalized spectral clustering (FSCUN) [23], fair normalized spectral clustering (FSCN) [23], fair algorithms for clustering (FAC) [25], and variational fair cluster (VFC) [27]. Publicly available implementations (FSCUNandFSCN:https://github.com/matthklein/fair_spectral_clustering; FAC: https://github.com/nicolasjulioflores/fair_algorithms_for_clustering; VFC: https://github.com/imtiazziko/Variational-Fair-Clustering, accessed on 27 June 2023) of these fair algorithms have been used.

FSCUN and FSCN consider fairness constraints to be linear constraints and incorporated them into the RatioCut objective. The fair version of spectral clustering is solved by finding the k smallest eigenvalues of a new Laplace matrix embedded in the fair subspace. Fair SC requires a weighted adjacency matrix W as an input, in our experiments, we construct W as:

$$w_{ij}=e^{-\frac{\parallel x_i-x_j{\parallel }_2^2}{\sigma }},\sigma =max\parallel x_i-x_j{\parallel }_2.$$

(13)

FAC employs a rounding approach based on linear programming to solve a fair-assignment problem, which improves the fairness after vanilla k-means clustering. FAC requires two parameters: the lower bound $\beta$ and the upper bound $\alpha$. The values of the two parameters are calculated as follows:

$${\beta }_l=\frac{|{\mathcal{G}}_l|(1-\delta )}{n},{\alpha }_l=\frac{|{\mathcal{G}}_l|}{n(1-\delta )},$$

(14)

where the value of $\delta$ is set as 0.2, as reported in [25].

VFC describes the fairness objective as the KL divergence between the demographic proportions in the dataset and the distributions of each cluster. The VFC objective is a trade-off between a clustering objective (e.g., k-medians, k-means, and NCut) and the fairness objective. The authors derived a convex–concave formulation for the overall objective and optimized it. We choose the best trade-off between the k-means clustering objective and fairness error.

4.2. Evaluation Metrics

In our experiments, both clustering quality and fairness are concerned. The metrics of clustering quality include:

• SSE: The SSE measures how much objects deviate from the centers of the clusters they are assigned to. The smaller the clustering cost, the better the clustering result.
• Dunn index (DI): The DI quantifies the ratio of the smallest distance between different clusters and the largest distance within a cluster [42]. It is defined as follows:

  $$DI=\frac{{min}_{1\le i\le k}{min}_{1\le j\le k,j\ne i}{min}_{a\in {\mathcal{C}}_i,b\in {\mathcal{C}}_j}\parallel a-b\parallel }{{max}_{1\le l\le k}ma{x}_{a\in {\mathcal{C}}_l,b\in {\mathcal{C}}_l}\parallel a-b\parallel }.$$

  (15)
  The DI has a value between zero and infinity and should be maximized.

The metrics of clustering fairness include:

• Balance: The balance measures the disparity in the distribution of a protected attribute across clusters and a dataset. The higher the value, the fairer the clustering result.
• Average Wasserstein Distance between Probability Vectors (AWD): The AWD measures the unfairness of the clustering result [16]. Let $P_i\in {\mathcal{R}}^{m\times 1}$ denote the distribution of groups in $C_i$, and $P_x\in {\mathcal{R}}^{m\times 1}$ denote the distribution of groups in X. The AWD of clustering is defined as:

  $$AWD=\frac{{\sum }_i^k|C_i|\times WD(P_i,P_x)}{n},$$

  (16)

  where $WD(.,.)$ is the Wasserstein distance. The smaller the value of AWD, the fairer the clustering result.

4.3. Fair Clustering Results Comparison

We relax the fairness constraint so that the balance value of our proposed method is slightly higher than or equal to the second highest. The clustering results of different algorithms in terms of clustering quality and fairness evaluation are shown in

Table 2. Clustering fairness.

Dataset	Balance ↑						AWD ↓
	Lloyd	FSCUN	FSCN	FAC	VFC	FFC	Lloyd	FSCUN	FSCN	FAC	VFC	FFC
Elliptical (k = 2)	0	0.8878	0.8835	0.8000	0.8835	0.8878	0.4940	0.0460	0.0480	0.1000	0.0480	0.0460
DS-577 (k = 3)	0	0	0	0.7988	0.5427	0.8005	0.4183	0.0995	0.1496	0.0462	0.1618	0.0381
2d-4c-no0 (k = 4)	0	0	0	0.7719	0.0201	0.7722	0.3127	0.0049	0.0649	0.0274	0.2226	0.0291
2d-4c-no1 (k = 4)	0	0	0	0.7780	0	0.7785	0.2705	0.0012	0.0701	0.0427	0.2341	0.0409
2d-4c-no4 (k = 4)	0	0	0	0.7369	0.3045	0.7369	0.1723	0.0759	0.0799	0.0459	0.1117	0.0296
Adult (k = 5)	0.6345	0.6755	0.6618	0.7978	0.9405	0.9405	0.0723	0.0402	0.0405	0.0429	0.0106	0.0143
Bank (k = 6)	0.2929	0.3368	0.5306	0.7958	0.8076	0.8079	0.0691	0.0357	0.0358	0.0334	0.0251	0.0360
Census1990 (k = 5)	0.0039	0.6964	0.7418	0.7984	0.9119	0.9127	0.2429	0.0511	0.0470	0.0554	0.0174	0.0218
CreditCard (k = 5)	0.8910	0	0.9259	0.9011	0.9037	0.9266	0.0193	0.0074	0.0128	0.0177	0.0173	0.0138
Diabetic (k = 10)	0.8376	0	0.8239	0.8239	0.8728	0.8731	0.0357	0.0235	0.0285	0.0289	0.0243	0.0330

and

Table 3. Clustering quality.

Dataset	DI ↑						SSE ↓
	Lloyd	FSCUN	FSCN	FAC	VFC	FFC	Lloyd	FSCUN	FSCN	FAC	VFC	FFC
Elliptical (k = 2)	0.0644	0.0118	0.0716	0.0010	0.0716	0.0118	206.2982	344.4216	343.9640	446.1201	343.9640	344.4216
DS-577 (k = 3)	0.0079	0.0005	0.0068	0.0005	0.0005	0.0005	71.0134	449.0292	361.4299	501.0075	377.3101	507.5545
2d-4c-no0 (k = 4)	0.0076	0.0032	0.0002	0.0002	0.0002	0	114.4834	1.5283 × 103	1.3558 × 103	1.4101 × 103	419.5988	1.3851 × 103
2d-4c-no1 (k = 4)	0.0017	0.0112	0.0001	0	0	0	82.3122	1.6150 × 103	1.2718 × 103	1.4734 × 103	287.2313	1.4632 × 103
2d-4c-no4 (k = 4)	0.0057	0.0002	0.0001	0.0001	0.0001	0.0001	104.0023	705.9991	666.3117	676.0752	420.1287	680.7263
Adult (k = 5)	0.0085	0.0023	0.0150	0.0148	0.0106	0.0091	1.3426 × 103	1.3529 × 103	1.3470 × 103	1.3443 × 103	1.3611 × 103	1.3672 × 103
Bank (k = 6)	0.0190	0	0.0018	0	0.0013	0	1.23134 × 103	1.7858 × 103	1.2543 × 103	1.2533 × 103	1.2774 × 103	1.2714 × 103
Census1990 (k = 5)	0.0663	0.0580	0.0258	0.0859	0.0527	0.0458	1.7604 × 103	1.8207 × 103	1.8219 × 103	1.8405 × 103	1.8807 × 103	1.8216 × 103
CreditCard (k = 5)	0.0288	0.0472	0.0184	0.0288	0.0288	0.0264	1.0805 × 103	1.8202 × 103	1.1119 × 103	1.0805 × 103	1.0805 × 103	1.0806 × 103
Diabetic (k = 10)	0.0406	0	0	0.0374	0	0	243.2634	3.2616 × 103	235.2583	231.9906	254.8696	241.3612

, where the results of our proposed method and the method with the second highest balance value are highlighted as the underlined items. From Table 2, one can find that the proposed FFC outperforms all other methods in terms of the balance metric. FFC achieves the lowest value for the AWD on four datasets. It is noticeable that FSCUN has a lower value for the AWD than FFC on datasets 2d-4c-no0 and 2d-4c-no1. On these two datasets, the balance value of FSCUN is 0 because it groups almost all data points into one cluster, as shown in Figure 3, which we did not expect.

Table 3 reports the comparative results for the clustering quality. The inclusion of fairness consideration in clustering algorithms can negatively impact their clustering quality. Generally, if one algorithm is more focused on fairness (i.e., has a higher value for the balance or a lower value for the AWD) compared to another, it is likely to result in lower clustering quality (i.e., has a lower value for the DI or a higher value for the SSE). Our proposed method performs better than other fair clustering algorithms in terms of fairness metrics, so it can be expected to perform poorly on clustering quality metrics.

It is worth noting the comparison between our proposed method and the methods with the second highest balance value in terms of the SSE. Among the 10 datasets, compared to the methods with the second highest balance value, FFC shares the same SSE values on one dataset and achieves lower SSE values on six datasets. On Elliptical, FFC and FSCUN share the highest balance, and they also have same values for the SSE. On DS-577, 2d-4c-no4, and Adult, FFC has higher SSE values than the methods with the second highest balance value. On 2d-4c-no0, 2d-4c-no1, Bank, Census1990, CreditCard, and Diabetic, FFC obtains lower SSE values than the methods with the second highest balance value. Therefore, FFC has a better ability to search for solutions that have a higher balance value and a low SSE value. Figure 3 depicts clustering results on the synthetic datasets. FFC assigns similar points belonging to the same group to the same cluster to make the clustering results meaningful.

4.4. Clustering Cost with Respect to k

We evaluate the clustering objectives and balance of Lloyd, FAC, and FFC with different numbers of clusters. Here, we relax the fairness constraint by ensuring that the balance value of FFC is slightly higher than or equal to the balance value of FAC for different numbers of clusters. Figure 4 shows the performance trends of these methods on Census with respect to the number of clusters. As the number of clusters increases, we have several observations. (1) The SSE values of these methods decrease. (2) The gap between the SSE values of Lloyd and FFC increases. (3) The balance value of Lloyd is reduced to zero. (4) Although the balance values for FFC and FAC are nearly equal, the SSE value for FFC is always lower than that of FAC, and the gap between the two increases. These observations are consistent with those in previous works [25,27,38].

5. Conclusions

In this paper, we incorporate fairness constraints into k-means clustering and propose FFC. The scheme of our method is to handle fairness before improving clustering quality, which is different from the existing post-processing-based work. We design an initialization stage that can handle the dataset with a multi-value protected attribute to produce an initial fair solution. Moreover, a relaxation stage and an improvement stage are presented to improve clustering quality while relaxing and maintaining fairness, respectively. Experimental results on synthetic and real-world datasets show that our proposed method outperforms some other fair clustering methods in terms of fairness metrics while the loss of clustering quality is acceptable or even better.

We achieved fairness in k-means clustering but sacrificed the quality of clustering. Even though the loss of clustering quality is acceptable, finding a way to minimize the cost of fairness is our future work. Since a local search method is employed, FFC does not provide a guarantee that any clustering output is the global optimum. In the future, we will find a strategy that can search for the near global optimum or even the global optimum.