Interpretable Market Segmentation on High Dimension Data ^†

Abstract

Obtaining relevant information from the vast amount of data generated by interactions in a market or, in general, from a dyadic dataset, is a broad problem of great interest both for industry and academia. Also, the interpretability of machine learning algorithms is becoming increasingly relevant and even becoming a legal requirement, all of which increases the demand for such algorithms. In this work we propose a quality measure that factors in the interpretability of results. Additionally, we present a grouping algorithm on dyadic data that returns results with a level of interpretability selected by the user and capable of handling large volumes of data. Experiments show the accuracy of the results, on par with traditional methods, as well as its scalability.

1. Introduction

Data obtained by monitoring a marketplace are mainly dyadic [1], that is, they represent the relation between two entities (for instance user vs products, buyers vs sellers or any other pairing of agents). This sort of data are also prevalent in common problems such as recommender systems [2], computational linguistics, information retrieval and preference learning [3], besides being used in more specific problems like automatic test grading [4].

A traditional problem to be solved with this kind of data consists on obtaining groups of entities that show a similar behavior. Market segmentation is the process of performing this analysis on market data [5]. The resulting grouping is coveted by companies since it offers valuable insight, but it is hard to obtain.

Also, having results that are easily interpretable by managers is essential. Interpretability is given by a collection of characteristics that promote ease of understanding of a model [6] and can be achieved by providing transparent models and algorithms or by offering additional explanations for the outputs of the model.

The algorithm introduced in this work aims to obtain informative and easy to interpret data for human supervisors. It is implemented in the Apache Spark [7] distributed framework, which enables the analysis of large amounts of data in a reasonable time.

2. Proposal

Given a dataset $\mathcal{X}$ containing data showing the interactions between two entities $\mathcal{U}$ and $\mathcal{V}$ in which each data point $x\in \mathcal{X}$ has the form $(u,i,f(u,i\left)\right)$ with f being a utility function$f:(\mathcal{U},\mathcal{I})\to \left\{-1,+1\right\}$, a grouping $Cl\left(\mathcal{U}\right)=\left\{Cl{u}_1,\cdots ,Cl{u}_m\right\}$ on one of the entities can be defined as a set of m groups containing all the elements in $\mathcal{U}$. The aptness of this grouping can be measured as the homogeneity of the value v across the elements in each $Cl{u}_k$ [8]. Using this measure, we can define the weighted entropy of a grouping as

$$WE\left(Cl\left(\mathcal{U}\right)\right)=\sum _{k,j}\frac{|Cl{u}_k|}{\left|\mathcal{U}\right|\left|\mathcal{I}\right|}H\left(\frac{|\left\{u\in Cl{u}_k:f(u,i_j)=+1\right\}|}{|Cl{u}_k|}\right).$$

(1)

where $H\left(x\right)$ represents the Shannon entropy of x.

Since each $u\in \mathcal{U}$ is a defined by a set of variables, each $Cl{u}_k$ is defined by giving a range for those variables. We can obtain a measure of the quality of $Cl\left(\mathcal{U}\right)$ by adding a factor that measures its interpretability. We do that by adding the number of such variables needed to define each group $Cl{u}_k$.

$$Q\left(Cl\left(\mathcal{U}\right)\right)=-WE\left(Cl\left(\mathcal{U}\right)\right)-\lambda \sum _{Cl{u}_k\in Cl\left(\mathcal{U}\right)}NV\left(Cl{u}_k\right).$$

(2)

where $NV\left(x\right)$ represents the number of variables needed to characterize x and $\lambda$ is a hyperparameter that enables the user to manage the balance between accuracy and ease of understanding.

The proposed algorithm takes a dataset $\mathcal{X}$ as input and returns a grouping $Cl\left(\mathcal{U}\right)$ that maximizes $Q\left(Cl\right(\mathcal{U}\left)\right)$. It does so by constructing a decision tree over the variables in $\mathcal{U}$ which defines the grouping $Cl\left(\mathcal{U}\right)$.

Algorithm 1: Grouping algorithm.

3. Results

Experiments performed on a large real world dataset containing information about readers and news items show that the proposed algorithm obtains a grouping consisting of 18 groups with a weighted entropy similar to that of the grouping with 100 elements obtained by Kmeans with $k=100$.

Additional experiments show that the Apache Spark implementation of the algorithm shows almost linear scalability when adding more computation nodes.