Multi-Label Classification from Multiple Noisy Sources Using Topic Models ^†

Abstract

Multi-label classification is a well-known supervised machine learning setting where each instance is associated with multiple classes. Examples include annotation of images with multiple labels, assigning multiple tags for a web page, etc. Since several labels can be assigned to a single instance, one of the key challenges in this problem is to learn the correlations between the classes. Our first contribution assumes labels from a perfect source. Towards this, we propose a novel topic model (ML-PA-LDA). The distinguishing feature in our model is that classes that are present as well as the classes that are absent generate the latent topics and hence the words. Extensive experimentation on real world datasets reveals the superior performance of the proposed model. A natural source for procuring the training dataset is through mining user-generated content or directly through users in a crowdsourcing platform. In this more practical scenario of crowdsourcing, an additional challenge arises as the labels of the training instances are provided by noisy, heterogeneous crowd-workers with unknown qualities. With this motivation, we further augment our topic model to the scenario where the labels are provided by multiple noisy sources and refer to this model as ML-PA-LDA-MNS. With experiments on simulated noisy annotators, the proposed model learns the qualities of the annotators well, even with minimal training data.

1. Introduction

With the advent of internet enabled hand-held mobile devices, there is a proliferation of user generated data. Often, there is a wealth of useful knowledge embedded within this data and machine learning techniques can be used to extract the information. However, as much of this data is user generated, it suffers from subjectivity. Any machine learning techniques used in this context should address the subjectivity in a principled way.

Multi-label classification is an important problem in machine learning where an instance d is associated with multiple classes or labels. The task is to identify the classes for every instance. In traditional classification tasks, each instance is associated with a single class; however, in multi-label classification, an instance can be explained with several classes. Multi-label classification finds applications in several areas, for example, text classification, image retrieval, social emotion classification [1], sentiment based personalized search [2], financial news sentiment [3,4], etc. Consider the task of classification of documents into several classes such as crime, politics, arts, sports, etc. The classes are not mutually exclusive since a document belonging to, say, the “politics” category may also belong to “crime”. In the case of classification of images, an image belonging to the “forest” category may also belong to the “Scenery” category, and so on.

A natural solution approach for multi-label classification is to generate a new label set that is a power set of the original label set, and then use traditional single label classification techniques. The immediate limitation here is an exponential blow-up of the label set ($2^C$ where C is the number of classes) and availability of only a small sized training dataset for each of the generated labels. Another approach is to build one-vs-all binary classifiers, where, for each label, a binary classifier is built. This method results in a lower number of classifiers than the power set based approach. However, it does not take into account the correlation between the labels.

Topic models [5,6,7] have been used extensively in natural language processing tasks to model the process behind generating text documents. The model assumes that the driving force for generating documents arise from “topics” that are latent. The alternate representation of a document in terms of these latent topics has been used in several diverse domains such as images [8], population genetics [9], collaborative filtering [10], disability data [11], sequential data and user profiles [12], etc. Though the motivation for topic models arose in an unsupervised setting, they were gradually found to be useful in the supervised learning setting [13] as well. The topic models available in literature for multi-label setting include Wang et al. [14], Rubin et al. [15]. However, these models either involve too many parameters [15] or learn the parameters by heavily depending on iterative optimization techniques [14], thereby making it hard to adapt to the scenario where labels are provided by multiple noisy sources such as crowd-workers. Moreover, in all of these models, The topics and hence words are assumed to be generated depending only on the classes that are present. They do not make use of the information provided by the absence of classes. The absence of a class often provides critical information about the words present. For example, a document labeled “sports” is less likely to have words related to “astronomy”. Similarly, in the images domain, an image categorised as “portrait” is less likely to have the characteristics of “Scenery”. Needless to say, such correlations are dataset dependent. However, a principled analysis must account for such correlations. Motivated by this subtle observation, we introduce a novel topic model for multi-label classification.

In the current era of big data where large amounts of unlabeled data are readily available, obtaining a noiseless source for labels is almost impossible. However, it is possible to get instances labeled by several noisy sources. An emerging example of this case occurs in crowdsourcing, which is the practice of obtaining work by employing a large number of people over the internet. The multi-label classification problem is more interesting in this scenario, where the labels are procured from multiple heterogeneous noisy crowd-workers with unknown qualities. We use the terms crowd-workers, workers, sources and annotators interchangeably in the paper. The problem becomes harder as now the true labels are unknown and the qualities of the annotators must be learnt to train a model. we non-trivially extend our topic model to this scenario.

Contributions

• We introduce a novel topic model for multi-label classification; our model has the distinctive feature of exploiting any additional information provided by the absence of classes. In addition, The use of topics enables our model to capture correlation between the classes. We refer to our topic model as ML-PA-LDA (Multi-Label Presence-Absence LDA).
• We enhance our model to account for the scenario where several heterogeneous annotators with unknown qualities provide the labels for the training set. We refer to this enhanced model as ML-PA-LDA-MNS (ML-PA-LDA with Multiple Noisy Sources). A feature of ML-PA-LDA-MNS is that it does not require an annotator to label all classes for a document. Even partial labeling by the annotators up to the granularity of labels within a document is adequate.
• We test the performance of ML-PA-LDA on several real world datasets and establish its superior performance over the state of the art.
• Furthermore, we study the performance of ML-PA-LDA-MNS, with simulated annotators providing the labels for these datasets. In spite of the noisy labels, ML-PA-LDA-MNS demonstrates excellent performance and the qualities of the annotators learnt approximate closely the true qualities of the annotators.

The rest of the paper is organized as follows. In Section 2, we describe relevant approaches in the literature. We propose our topic model for multi-label classification from a single source (ML-PA-LDA) in Section 3 and discuss the parameter estimation procedure for our model using variational expectation maximization (EM) in Section 4. In Section 5, we discuss inference on unseen instances using our model. We adapt our model to account for labels from multiple noisy sources in Section 6. Parameter estimation for this revised model, ML-PA-LDA-MNS, is described in Section 7. In Section 8, we discuss how our model can incorporate smoothing, so that, in the inference phase, new words which were never seen in the training phase can be handled. We discuss our experimental findings in Section 9 and conclude in Section 10.

2. Related Work

Several approaches have been devised for multi-label classification with labels provided by a single source. The most natural approach is the Label Powerset (LP) method [16] that generates a new class for every combination of labels and then solves the problem using multiclass classification approaches. The main drawback of this approach is the exponential growth in the number of classes, leading to several generated classes having very few labeled instances leading to overfitting. To overcome this drawback, a RAndom k-labELsets method (RAkEL) [17] was introduced, which constructs an ensemble of LP classifiers where each classifier is trained with a random subset of k labels. However, The large number of labels still poses challenges. The approach of pairwise comparisons (PW) improves upon the above methods, by constructing C(C-1)/2 classifiers for every pair of classes, where C is the number of classes. Finally, a ranking of the predictions from each classifier yields the labels for a test instance. Rank-SVM [18] uses the PW approach to construct SVM classifiers for every pair of classes and then performs a ranking. Further details on the above approaches can be obtained in the survey [19].

The previously described approaches are discriminative approaches. Generative models for the multi-label classification model the correlation between the classes by mixing weights for the classes [20]. Other probabilistic mixture models include Parametric Mixture Models PMM1 and PMM2 [21]. After the advent of the topic models like Latent Dirichlet Allocation (LDA) [5], extensions have been proposed for multi-label classification such as Wang et al. [14]. However, in [14], due to the non-conjugacy of the distributions involved, closed form updates cannot be obtained for several parameters and iterative optimization algorithms such as conjugate gradient and Newton Raphson are required to be used in the variational E step as well as M step, introducing additional implementation issues. Adapting this model to the case of multiple noisy sources would result in enormous complexity. The approach used in [22] makes use of Markov Chain Monte Carlo (MCMC) methods for parameter estimation, which is known to be expensive. The topic models proposed for multi-label classification in [15] involve far too many parameters that can be learnt effectively only in the presence of large amounts of labeled data. For small and medium sized datasets, The approach suffers from overfitting. Moreover, it is not clear how this model can be adapted when labels are procured from crowd-workers with unknown qualities. Supervised Latent Dirichlet Allocation (SLDA) [13] is a single label classification technique which works well on multi-label classification when used with the one-vs-all approach. SLDA inherently captures the correlation between classes through the latent topics.

With crowdsourcing gaining popularity due to the availability of large amounts of unlabeled data and difficulty in procuring noiseless labels for these datasets, aggregating labels from multiple noisy sources has become an important problem. Raykar et al. [23] look at training binary classification models with labels from a crowd with unknown annotator qualities. Being a model for multiclass classification, this model does not capture the correlation between classes and thereby cannot be used for multi-label classification from the crowd. Mausam et al. [24] look at multi-label classification for taxonomy creation from the crowd. They construct C classifiers by modeling the dependence between the classes explicitly. The graphical model representation involves too many edges especially when the number of classes is large and hence the model suffers from overfitting. Deng et al. [25] look at selecting the instance to be given to a set of crowd-workers. However, they do not look at aggregating these labels and developing a model for classification given these labels. In the report [26], Duan et al. look at methods to aggregate a multi-label set provided by crowd-workers. However, they do not look at building a model for classification for new test instances for which the labels are not provided by the crowd. Recently, the topic model, SLDA, has been adapted to learning from the labels provided by crowd annotators [27]. However, like its predecessor SLDA, it is only applicable to the single label setting and not to multi-label classification.

The existing topic models in the literature such as [28] assume that the presence of a class generates words pertaining to those classes and do not take into account the fact that the absence of a class may also play a role in generating words. In practice, The absence of a class may yield information about occurrence of words. We propose a model for multi-label classification based on latent topics where the presence as well as absence of a class could generate topics. The labels could be procured from multiple sources (e.g., crowd workers) whose qualities are unknown.

3. Proposed Approach for Multi-Label Classification from a Single Source: ML-PA-LDA

We now explain our model for multi-label classification assuming labels from a single source (The model from a single source has not explained in the conference version of this paper. Section 3, Section 4 and Section 5 entirely deal with the single source model.). For ease of exposition, we use notations from the text domain. However, the model itself is general and can be applied to several domains by suitable transformation of features into words. In our experiments, we have applied the model to domains other than text. We will explain the transformation of features to words when we describe our experiments.

Let D be the number of documents in the training set, also known as a corpus. Each document is a set of several words. Let C be the total number of classes in the universe. In multi-label classification, a document may belong to any ‘subset’ of the C classes as opposed to the standard classification setting where a document belongs to exactly one class. Let T be the number of latent topics responsible for generating words. The set of all possible words is referred to as a vocabulary. We denote by V the size of the vocabulary $\mathcal{V}=\{{\nu }_1,\dots ,{\nu }_V\}$, where ${\nu }_j$ refers to the $j^{th}$ word in $\mathcal{V}$. Consider a document d comprising N words $\mathbf{w}=\{w_1,w_2,\dots ,w_N\}$ from the vocabulary $\mathcal{V}$. Let $\lambda =\left[{\lambda }_1,\dots ,{\lambda }_C\right]\in {\{0,1\}}^C$ denote the true class membership of the document. In our notations, we denote by $w_{nj}$ the value $\mathrm{𝟙}[w_n={\nu }_j]$, which is the indicator that the word $w_n$ is the $j^{th}$ word of the vocabulary. Similarly, we denote by ${\lambda }_{ij}$, The indicator that ${\lambda }_i=j$, where $j=0$ or 1. Our objective is to predict the vector $\lambda$ for every test document.

Topic Model for the Documents

We introduce a model to capture the correlation between the various classes generating a given document. Our model is based on topic models [5] that were originally introduced for the unsupervised setting. The key idea in topic models is to get a representation for every document in terms of “topics”. Topics are hidden concepts that occur through the corpus of documents. Every document is said to be composed of some proportion of topics, where the proportion is specific to a document. Each topic is responsible for generating the words in the document and has its own distribution for generating the words. Neither the topic proportions of a document nor the topic-word distributions are known. The number of topics is assumed to be known. The topics can be thought of capturing the concepts present in a document. For further details on topic models, the reader may refer [5].

For the case of multi-label classification, we make the observation that the presence as well as absence of a class provides additional information about the topics present in a document. We now describe our generative process for each document, assuming labels are provided by a perfect source.

• Draw class membership ${\lambda }_i\sim \mathrm{Bern}\left({\xi }_i\right)$ for every class $i=1,\dots ,C$.
• Draw ${\theta }_{i,j,.}\sim \mathrm{Dir}\left({\alpha }_{i,j,.}\right)$ for $i=1,\dots ,C$, for $j\in \{0,1\}$, where ${\alpha }_{i,j,.}$ are the parameters of a Dirichlet distribution with T parameters. ${\theta }_{i,j,.}$ provides the parameters of a multinomial distribution for generating topics.
• For every word w in the document

  (a)

  Sample $u\sim \mathrm{Unif}\{1,\dots ,C\}$ from one of the C classes.

  (b)

  Generate a topic $z\sim \mathrm{Mult}\left({\theta }_{u,{\lambda }_u,.}\right)$, where ${\theta }_{i,j,.}$ are the parameters of a multinomial distribution in T dimensions.

  (c)

  Generate the word $w\sim \mathrm{Mult}\left({\beta }_{z.}\right)$ where ${\beta }_{z.}$ are the parameters of a multinomial distribution in V dimensions.

Intuitively, for every class i, its presence or absence (${\lambda }_i$) is first sampled from a Bernoulli distribution parameterized by ${\xi }_i$. The parameter ${\xi }_i$ is the prior for class i. We capture the correlations across classes through latent topics. The corpus wide distribution $\mathrm{Dir}\left({\alpha }_{i,j,.}\right)$ is the prior for the distribution ($\mathrm{Mult}\left({\theta }_{i,j,.}\right)$) of topics for class i taking the value j. Then, the latent class u is sampled, which, in turn, along with ${\lambda }_u$, generates latent topic z. The topic z is then responsible for a word. The same process repeats for the generation of every word in the document.

The generative process for the documents is depicted pictorially in Figure 1a. The parameters of our model consist of $\pi =\{\alpha$, $\xi$, $\beta \}$ and must be learnt. During the training phase, The observed variables for each document are $\mathbf{d}=\{\mathbf{w},{\lambda }_i\}$ for $i=1,\dots ,C$. The hidden random variables are $\mathsf{\Theta }=\{\theta ,u,z\}$. We refer to the above described topic model as ML-PA-LDA (Multi-Label Presence-Absence LDA).

4. Variational EM for Learning the Parameters of ML-PA-LDA

We now detail the steps for estimating the parameters of our proposed model ML-PA-LDA. Given the observed words w and the label vector $\lambda$ for a document d, The objective of the model described above is to first obtain $p\left(\mathsf{\Theta }\right|\mathbf{d})$, The posterior distribution over the hidden variables. Here, The challenge lies in the intractable computation of $p\left(\mathsf{\Theta }\right|\mathbf{d}),$ which arises due to the intractability in the computation of $p\left(\mathbf{d}\right|\pi )$. We use variational inference with mean field assumptions [29] to overcome this challenge.

The underlying idea in variational inference is the following. Suppose $q\left(\mathsf{\Theta }\right)$ is any distribution over $\mathsf{\Theta }$ for any arbitrary $\mathsf{\Theta }=\{\theta ,u,z\}$ that approximates $p\left(\mathsf{\Theta }\right|\mathbf{d})$. We refer to $q\left(\mathsf{\Theta }\right)$ as variational distribution.

Observe that

$$\begin{array}{cc}\hfill \ln & p\left(\mathbf{d}\right|\pi )=\ln \frac{p(\mathbf{d},\mathsf{\Theta }|\pi )}{p\left(\mathsf{\Theta }\right|\mathbf{d},\pi )}=\ln \frac{p(\mathbf{d},\mathsf{\Theta }|\pi \left)q\right(\mathsf{\Theta })}{q\left(\mathsf{\Theta }\right)p\left(\mathsf{\Theta }\right|\mathbf{d},\pi )}={\mathbb{E}}_{q\left(\mathsf{\Theta }\right)}\left[\ln \frac{p(\mathbf{d},\mathsf{\Theta }|\pi \left)q\right(\mathsf{\Theta })}{q\left(\mathsf{\Theta }\right)p\left(\mathsf{\Theta }\right|\mathbf{d},\pi )}\right]\\ & ={\mathbb{E}}_{q\left(\mathsf{\Theta }\right)}\left[\ln p(\mathbf{d},\mathsf{\Theta }|\pi )-\ln q(\mathsf{\Theta })\right]+{\mathbb{E}}_{q\left(\mathsf{\Theta }\right)}\left[\ln q\left(\mathsf{\Theta }\right)-\ln p\left(\mathsf{\Theta }\right|\mathbf{d},\pi )\right]\hfill \\ & =\mathcal{L}\left(\mathsf{\Theta }\right)+\mathbf{KL}\left(q\right(\mathsf{\Theta }\left)\right|\left|p\right(\mathsf{\Theta }|\mathbf{d},\pi )).\hfill \end{array}$$

(1)

Variational inference involves maximizing $\mathcal{L}\left(\mathsf{\Theta }\right)$ over the variational parameters $\{\gamma ,\varphi ,\delta \}$ so that $\mathbf{KL}\left(q\right(\mathsf{\Theta }\left)\right|\left|p\right(\mathsf{\Theta }|\mathbf{d},\pi ))$ also gets minimized. Our underlying variational model is provided in Figure 1b. In our notations, $u_{ni}=\mathrm{𝟙}\left[u_n=i\right]$ for $i\in \{1,\dots ,C\}$, $z_{nt}=\mathrm{𝟙}\left[z_n=t\right]$ for $t\in \{1,\dots ,T\}$ and $\mathsf{\Gamma }$ denotes the gamma function. From our model (Figure 1a),

$$\begin{array}{cc}\hfill \ln p(\mathbf{d},\mathsf{\Theta }|\pi )& =\ln p(\mathbf{w},\theta ,\lambda ,u,z|\pi )\hfill \\ & =\ln p\left(\lambda \right|\xi )+\ln p(\theta \left|\alpha \right)+\ln p\left(u\right)+\ln p\left(z\right|\lambda ,u,\theta )+\ln p(\mathbf{w}|z,\beta ),\hfill \end{array}$$

(2)

and, in turn, from the assumptions on the distributions of the variables involved,

$$\ln p\left(\lambda \right|\xi )=\sum _{i=1}^C{\lambda }_i\ln {\xi }_i+(1-{\lambda }_i)\ln (1-{\xi }_i),$$

(3)

$$\ln p\left({\theta }_{ij}\right|\alpha )=\ln \mathsf{\Gamma }\left(\sum _{t=1}^T{\alpha }_{ijt}\right)-\sum _{t=1}^T\ln \mathsf{\Gamma }{\alpha }_{ijt}+\sum _{t=1}^T({\alpha }_{ijt}-1)\log {\theta }_{ijt},$$

(4)

$$\log p\left(u\right)=\sum _{n=1}^N\sum _{i=1}^C{u}_{ni}\log 1/C,$$

(5)

$$\log p\left(z\right|u,\lambda ,\theta )=\sum _{n=1}^N\sum _{t=1}^T\sum _{i=1}^C\sum _{j=0}^1{u}_{ni}{\lambda }_{ij}{z}_{nt}\log {\theta }_{ijt},$$

(6)

$$\log p\left(w\right|z,\beta )=\sum _{n=1}^N\sum _{t=1}^T\sum _{j=1}^V{w}_{nj}{z}_{nt}\log {\beta }_{tj}.$$

(7)

Assume the following variational distributions (as per Figure 1b) over $\mathsf{\Theta }$ for a document d. These assumptions on the independence between the latent variables are known as mean field assumptions [29]:

$$\begin{array}{c}{u}^d\sim \mathrm{Mult}\left({\delta }^d\right),z^d\sim \mathrm{Mult}\left({\varphi }^d\right),{\theta }_{ij}^d\sim \mathrm{Dir}({\gamma }_{ij}^d)\mathrm{for}i=1,\dots ,C\mathrm{and}j=0,1.\hfill \end{array}$$

Therefore, for a document d,

$$\begin{array}{c}\hfill q\left({\mathsf{\Theta }}^d\right)=\prod _{i=1}^C\prod _{j=0}^{1}q({\theta }_{ij}^d)\prod _{n=1}^N\prod _{i=1}^{C}q\left(u_{ni}^d\right)q\left(z_{ni}^d\right).\end{array}$$

4.1. E-Step Updates for ML-PA-LDA

The E-step involves computing the document-specific variational parameters ${\mathsf{\Theta }}^d=\{{\delta }^d,{\gamma }^d,{\varphi }^d\}$, for every document d, assuming a fixed value for the parameters $\pi =\{\alpha ,\xi ,\beta \}$. As a consequence of the mean field assumptions on the variational distributions [29], we get the following update rules for the distributions by maximising $\mathcal{L}\left(\mathsf{\Theta }\right)$. From now on, when clear from context, we omit the superscript d:

$$\begin{array}{cc}\hfill \log q\left(z\right)& ={\mathbb{E}}_{\mathsf{\Theta }\backslash z}\left[p(\mathbf{d},\mathsf{\Theta })\right]\propto {\mathbb{E}}_{u,\theta }\left[\log p\left(z\right|u,\lambda ,\theta )\right]+\log p\left(w\right|z,\beta )\hfill \\ & \propto \sum _{n=1}^N\sum _{t=1}^T{z}_{nt}\left[\sum _{i=1}^C\sum _{j=0}^1\mathbb{E}\left[u_{ni}\right]{\lambda }_{ij}\mathbb{E}\left[\log {\theta }_{ijt}\right]\right]+\sum _{n=1}^N\sum _{t=1}^T{z}_{nt}\left[\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}\right].\hfill \end{array}$$

(8)

In the computation of the expectation of ${\mathbb{E}}_{\mathsf{\Theta }\backslash z}\left[p(\mathbf{d},\mathsf{\Theta })\right]$ above, only the terms in $p(\mathbf{d},\mathsf{\Theta })$ that are a function of z need to be considered as the rest of the terms contribute to the normalizing constant for the density function $q\left(z\right)$. Hence, all terms in the right hand side of Equation (2) need not be considered and expectations of only $\log p\left(z\right|u,\lambda ,\theta )$ (Equation (6)) and $\log p\left(w\right|z,\beta )$ (Equation (7)) must be taken with respect to $u,\lambda ,\theta$. Observe that Equation (8) follows the structure of the multinomial distribution as assumed, and, therefore,

$$\begin{array}{cc}\hfill \log {\varphi }_{nt}& \propto \sum _{i=1}^C\sum _{j=0}^1\mathbb{E}\left[u_{ni}\right]{\lambda }_{ij}\mathbb{E}\left[\log {\theta }_{ijt}\right]+\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1{\delta }_{ni}{\lambda }_{ij}\mathbb{E}\left[\log {\theta }_{ijt}\right]+\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}.\hfill \end{array}$$

(9)

Similarly, The updates for the other variational parameters are as follows:

$$\begin{array}{cc}\hfill \log q\left(u\right)& ={\mathbb{E}}_{\mathsf{\Theta }\backslash u}\left[\log p\left(u\right)+p\left(z\right|u,\lambda ,\theta )\right]\hfill \\ & \propto \sum _{n=1}^N\sum _{i=1}^C{u}_{ni}\log 1/C+\sum _{n=1}^N\sum _{t=1}^T\sum _{i=1}^C\sum _{j=0}^1{u}_{ni}{\lambda }_{ij}\mathbb{E}\left[z_{nt}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right].\hfill \end{array}$$

(10)

Again, Equation (10) follows the structure of the multinomial distribution, and so

$$\begin{array}{c}\hfill \log {\delta }_{ni}\propto \log \frac{1}{C}+\sum _{t=1}^T{\varphi }_{nt}{\lambda }_i\mathbb{E}\left[\log {\theta }_{i1t}\right]+{\varphi }_{nt}(1-{\lambda }_i)\mathbb{E}\left[\log {\theta }_{i0t}\right].\end{array}$$

(11)

We have used that $\mathbb{E}\left[z_{nt}\right]={\varphi }_{nt}$. Additionally, since $\mathbb{E}\left[u_{ni}\right]={\delta }_{ni}$,

$$\begin{array}{cc}\hfill \log & q\left(\theta \right)={\mathbb{E}}_{\mathsf{\Theta }\backslash \theta }\left[p(\mathbf{d},\mathsf{\Theta })\right]\propto \mathbb{E}\left[p\left(\theta \right|\alpha )+p(z|u,\lambda ,\theta )\right]\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T({\alpha }_{ijt}-1)\log {\theta }_{ijt}+\sum _{n=1}^N\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T\mathbb{E}\left[u_{ni}\right]E\left[z_{nt}\right]{\lambda }_{ij}\log {\theta }_{ijt}\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T({\alpha }_{ijt}-1)\log {\theta }_{ijt}+\sum _{n=1}^N\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T{\delta }_{ni}{\varphi }_{nt}{\lambda }_{ij}\log {\theta }_{ijt}\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T({\gamma }_{ijt}-1)\log {\theta }_{ijt},\hfill \end{array}$$

where

$$\begin{array}{c}\hfill {\displaystyle {\gamma }_{ijt}={\alpha }_{ijt}+{\lambda }_{ij}\sum _{n=1}^N{\delta }_{ni}{\varphi }_{nt}.}\end{array}$$

(12)

In all of the above update rules, $\mathbb{E}[\log {\theta }_{ijt}^{\mathbf{d}}]=\psi ({\gamma }_{ijt})-\psi ({\sum }_{t^{\prime }=1}^T{\gamma }_{ij{t}^{\prime }}^d)$, where $\psi (.)$ is the digamma function.

4.2. M-Step Updates for ML-PA-LDA

In the M-step, The parameters $\xi$, $\beta$ and $\alpha$ are estimated using the values of ${\varphi }^d,{\delta }^d,{\gamma }^d$ estimated from the E-step. The function $\mathcal{L}\left(\mathsf{\Theta }\right)$ in Equation (1) is maximized with respect to the parameters $\pi$ yielding the following update equations.

Updates for $\xi$: for $i=1,\dots ,C$:

$$\begin{array}{c}\hfill {\xi }_i=\frac{{\sum }_{d=1}^D{\lambda }_i^d}{D}.\end{array}$$

(13)

Intuitively, Equation (13) makes sense as ${\xi }_i$ is the probability that any document in the corpus belongs to class i. Therefore, ${\xi }_i$ is the mean of ${\lambda }_i^d$ over all documents.

Updates for $\beta$: for $t=1,\dots ,T$; for $j=1,\dots ,V$:

$$\begin{array}{c}\hfill {\beta }_{tj}=\frac{{\sum }_{d=1}^D{\sum }_{n=1}^{N_d}{w}_{nj}^d{\varphi }_{nt}^d}{{\sum }_{d=1}^D{N}_d}.\end{array}$$

(14)

Intuitively, The variational parameter ${\varphi }_{nt}^d$ is the probability that the word $w_n^d$ is associated with topic t. Having updated this parameter in the E-step, ${\beta }_{tj}$ computes the fraction of times the word j is associated with topic t by giving a weight ${\varphi }_{nt}^d$ to its occurrence in document d.

Updates for $\alpha$:

There do not exist closed form updates for $\alpha$ parameters. Hence, we use the Newton Raphson (NR) method to iteratively obtain the solution as follows:

$$\begin{array}{c}\hfill {\alpha }_{ijr}^{t+1}={\alpha }_{ijr}^t-\frac{g_r-c}{h_r},\end{array}$$

(15)

where

$$\begin{array}{c}c=\frac{{\sum }_{\tau =1}^T{g}_{\tau }/h_{\tau }}{z^{-1}+{\sum }_{\tau =1}^{T}1/h_{\tau }},z=D{\psi }^{\prime }\left(\sum _{t^{\prime }=1}^T{\alpha }_{ij{t}^{\prime }}^t\right),h_{\tau }=-D{\psi }^{\prime }({\alpha }_{ijr}^t),\hfill \end{array}$$

$$g_r=D\left[\psi \left(\sum _{\tau =1}^T{\alpha }_{ij\tau }^t\right)-\psi \left({\alpha }_{ijr}^t\right)\right]+\sum _{d=1}^D\left[\psi \left({\gamma }_{ijr}^d\right)-\psi \left(\sum _{\tau =1}^T{\gamma }_{ij\tau }^d\right)\right].$$

5. Inference in ML-PA-LDA

In the previous sections, we introduced our model ML-PA-LDA and described how the parameters of the model $\pi =\{\alpha ,\beta ,\xi \}$ are learnt from a training dataset $D=\left\{(\mathbf{d},{\lambda }_1^d,\dots ,{\lambda }_C^d)\right\}$ comprising documents d and their corresponding labels. More specifically, we used variational inference for learning $\pi$. We now describe how inference can be performed on a test document d (We provide complete details of the inference phase in this section. This is a new section and was not present in our conference paper.). Here, unlike the previous scenario, The labels ${\lambda }_1^d,\dots ,{\lambda }_C^d$ are unknown and the task is to predict these labels.

The graphical model depicting this scenario is provided in Figure 2a. The model is similar to the case of training except that the variable $\lambda$ is no longer observed and must be estimated. Therefore, the set of hidden variables is now $\mathsf{\Theta }=\{\lambda ,z,u,\theta \}$. We use ideas from variational inference to estimate $\lambda$. Now, the approximating variational model is given by Figure 2b. Note that $\lambda$ is not observed in the original model (Figure 2a); therefore, an approximating distribution for $\lambda$ is required in the variational model (Figure 2b).

In this testing phase, The parameters $\pi =\{\alpha ,\xi ,\beta \}$ are known (from the training phase) and do not need to be estimated. Therefore, only the E-step of variational inference is required to be derived and executed, the end of which estimates for the variational parameters $\{\Delta ,\gamma ,\varphi ,\delta \}$ are obtained.

We begin by deriving the updates for the parameters of the posterior distribution of the latent variable z for a new document d.

From our model (Figure 2a),

$$\begin{array}{cc}\hfill \ln p(\mathbf{d},\mathsf{\Theta }|\pi )& =\ln p(\mathbf{w},y,\theta ,\lambda ,u,z|\pi )\hfill \\ & =\ln p\left(\lambda \right|\xi )+\ln p(\theta \left|\alpha \right)+\ln p\left(u\right)+\ln p\left(z\right|\lambda ,u,\theta )+\ln p(\mathbf{w}|z,\beta )+\ln p\left(y\right|\lambda ,\rho ).\hfill \end{array}$$

Assume the following independent variational distributions (as per Figure 2b) over each of the variables in $\mathsf{\Theta }$ for a document d:

$$\begin{array}{c}{u}^d\sim \mathrm{Mult}\left({\delta }^d\right),{\lambda }_i^d\sim \mathrm{Bern}\left({\Delta }_i^d\right)\mathrm{for}i=1,\dots ,C,\hfill \\ z^d\sim \mathrm{Mult}\left({\varphi }^d\right),{\theta }_{ij}^d\sim \mathrm{Dir}({\gamma }_{ij}^d)\mathrm{for}i=1,\dots ,C\mathrm{and}j=0,1.\hfill \end{array}$$

Therefore, $q\left({\mathsf{\Theta }}^d\right)={\prod }_{i=1}^{C}q({\lambda }_i^d){\prod }_{i=1}^C{\prod }_{j=0}^{1}q({\theta }_{ij}^d){\prod }_{n=1}^N{\prod }_{i=1}^{C}q\left(u_{ni}^d\right)q\left(z_{ni}^d\right)$. Note that, in the training phase, $\lambda$ was observed or known and there was no variational distribution over $\lambda$. However, in the inference phase, $\lambda$ is not observed and therefore a variational distribution over $\lambda$ is also required. we are now set to derive the updates for the various distributions $q(.)$. The key rule [29] that arises as a consequence of variational inference with mean field assumptions is that

$$\begin{array}{c}\hfill q\left(h\right)\propto {\mathbb{E}}_{\mathsf{\Theta }\backslash h}\left[p(\mathbf{d},\mathsf{\Theta })\right]\forall h\in \mathsf{\Theta }.\end{array}$$

(16)

We now apply Equation (16) to obtain the updates for the parameters of $q(.)$:

$$\begin{array}{cc}\hfill \log q\left(z\right)& ={\mathbb{E}}_{\mathsf{\Theta }\backslash z}\left[p(\mathbf{d},\mathsf{\Theta })\right]\propto {\mathbb{E}}_{u,\lambda ,\theta }\left[\log p\left(z\right|u,\lambda ,\theta )\right]+\log p\left(w\right|z,\beta )\hfill \\ & \propto \sum _{n=1}^N\sum _{t=1}^T{z}_{nt}\left[\sum _{i=1}^C\sum _{j=0}^1\mathbb{E}\left[u_{ni}\right]\mathbb{E}\left[{\lambda }_{ij}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right]\right]+\sum _{n=1}^N\sum _{t=1}^T{z}_{nt}\left[\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}\right].\hfill \end{array}$$

(17)

Similar to the derivations for the training phase, in the computation of the expectation of ${\mathbb{E}}_{\mathsf{\Theta }\backslash z}\left[p(\mathbf{d},\mathsf{\Theta })\right]$ in Equation (17), only the terms in $p(\mathbf{d},\mathsf{\Theta })$ that are a function of z need to be considered as the rest of the terms contribute to the normalizing constant for the density function $q\left(z\right)$. Hence, only expectations of $\log p\left(z\right|u,\lambda ,\theta )$ (Equation (6)) and $\log p\left(w\right|z,\beta )$ (Equation (7)) need to be taken with respect to $u,\lambda ,\theta$. Therefore,

$$\begin{array}{cc}\hfill \log {\varphi }_{nt}& \propto \sum _{i=1}^C\sum _{j=0}^1\mathbb{E}\left[u_{ni}\right]\mathbb{E}\left[{\lambda }_{ij}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right]+\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1{\delta }_{ni}{\Delta }_i^j{(1-{\Delta }_i)}^{1-j}\mathbb{E}\left[\log {\theta }_{ijt}\right]+\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}.\hfill \end{array}$$

(18)

Note that Equation (18) is different from Equation (9), as, in Equation (18), we also have an expectation over $\lambda$ terms in the inference phase. Similarly, The updates for the other variational parameters are as follows:

$$\begin{array}{cc}\hfill \log q\left(u\right)& \propto {\mathbb{E}}_{\mathsf{\Theta }\backslash u}\left[\log p\left(u\right)+p\left(z\right|u,\lambda ,\theta )\right]\hfill \\ & \propto \sum _{n=1}^N\sum _{i=1}^C{u}_{ni}\log 1/C+\sum _{n=1}^N\sum _{t=1}^T\sum _{i=1}^C\sum _{j=0}^1{u}_{ni}\mathbb{E}\left[{\lambda }_{ij}\right]\mathbb{E}\left[z_{nt}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right].\hfill \end{array}$$

(19)

Therefore,

$$\begin{array}{c}\hfill \log {\delta }_{ni}\propto \log \frac{1}{C}+\sum _{t=1}^T{\varphi }_{nt}{\Delta }_i\mathbb{E}\left[\log {\theta }_{i1t}\right]+{\varphi }_{nt}(1-{\Delta }_i)\mathbb{E}\left[\log {\theta }_{i0t}\right].\end{array}$$

(20)

Following similar steps, we get

$$\begin{array}{c}\hfill {\displaystyle {\gamma }_{ijt}={\alpha }_{ijt}+{\left({\Delta }_i\right)}^j{(1-{\Delta }_i)}^{1-j}\sum _{n=1}^N{\delta }_{ni}{\varphi }_{nt},}\end{array}$$

(21)

$$\begin{array}{cc}\hfill \log q\left(\lambda \right)& \propto \mathbb{E}\left[\log p\left(\lambda \right|\xi )+\log p(z|u,\lambda ,\theta )\right]\hfill \\ & =\sum _{i=1}^C{\lambda }_i\log {\xi }_i+(1-{\lambda }_i)\log (1-{\xi }_i)+\sum _{n=1}^N\sum _{t=1}^T\sum _{i=1}^C\sum _{j=0}^1{\lambda }_i^j{(1-{\lambda }_i)}^{1-j}\mathbb{E}\left[u_{ni}\right]\mathbb{E}\left[z_{nt}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right].\hfill \end{array}$$

Hence,

$$\begin{array}{c}\hfill \log {\Delta }_i\propto \log {\xi }_i+\sum _{n=1}^{N_d}\sum _{t=1}^T{\delta }_{ni}{\varphi }_{nt}\mathbb{E}\left[\log {\theta }_{i1t}\right],\end{array}$$

(22)

$$\begin{array}{c}\hfill \log (1-{\Delta }_i)\propto \log 1-{\xi }_i+\sum _{n=1}^{N_d}\sum _{t=1}^T{\delta }_{ni}{\varphi }_{nt}\mathbb{E}\left[\log {\theta }_{i0t}\right].\end{array}$$

(23)

In all of the above update rules, $\mathbb{E}[\log {\theta }_{ijt}^d]=\psi \left({\gamma }_{ijt}\right)-\psi ({\sum }_{t^{\prime }=1}^T{\gamma }_{ij{t}^{\prime }}^d)$, where $\psi (.)$ is the digamma function.

Aggregation rule for predicting document labels: Algorithm 1 provides the algorithm for the inference phase of ML-PA-LDA. After execution of Algorithm 1, ${\Delta }_i$ gives a probabilistic estimate corresponding to class i. In order to predict the labels of any document, a suitable threshold (say 0.5) can be applied on the value of ${\Delta }_i$ so that, if ${\Delta }_i>\mathrm{threshold}$, The estimate for ${\lambda }_i$, that is, ${\widehat{\lambda }}_i=1$.

Algorithm 1 Algorithm for inferring class labels for a new document d during test phase of Multi-Label Presence-Absence Latent Dirichlet Allocation (ML-PA-LDA).

Require: Document d, Model parameters $\pi =\{\xi ,\alpha ,\beta \}$

Initialize ${\mathsf{\Theta }}^d$

repeat

Update $\varphi$ using Equation (18)

Update $\delta$ using Equation (20)

Update $\gamma$ using Equation (21)

Update $\Delta$ using Equations (22) and (23)

until convergence

6. Proposed Approach for Multi-Label Classification from Multiple Sources: ML-PA-LDA-MNS

So far, we have assumed the scenario where the labels for training instances are provided by a single source. Now, we move to the more realistic scenario where the labels are provided by multiple sources with varying qualities that are unknown to the learner. These sources could even be human workers with unknown and varying noise levels. We adopt the single coin model for the sources, which we will now explain.

Single Coin Model for the Annotators

When the true labels of the documents are not observed, $\lambda$ is unknown. Instead, noisy versions $y_1,\dots ,y_K$ of $\lambda$ provided by a set of K independent annotators with heterogeneous unknown qualities $\{{\rho }_1,\dots ,{\rho }_K\}$ are observed. $y_j$ is the label vector given by annotator j. $y_{ji}$ can be either $0,1$ or $-1$. $y_{ji}=1$ indicates that, according to annotator j, The class i is present, while $y_{ji}=0$ indicates that the class i is absent as per annotator j. $y_{ji}=-1$ indicates that the annotator j has not made a judgement on the presence of class i in the document. This allows for partial labeling up to the granularity of labels even within a document. This flexibility in the modeling is essential, especially when the number of classes is large. ${\rho }_j$ is the probability with which an annotator reports the ground truth corresponding to each of the classes. ${\rho }_j$ is not known to the learning algorithm. For simplicity, we have assumed the single coin model for annotators and also that the qualities of the annotators are independent of the class under consideration. That is, $P(y_{j1}=1|{\lambda }_1=1)=P(y_{j1}=0|{\lambda }_1=0)=\dots$ $=P(y_{jC}=1|{\lambda }_C=1)=P(y_{jC}=0|{\lambda }_C=0)={\rho }_j$. This is a common assumption in literature [23].

The generative process for the documents is depicted pictorially in Figure 3a. The parameters of our model consist of $\pi =\{\alpha$, $\xi$, $\rho$, $\beta \}$. The observed variables for each document are $\mathbf{d}=\{\mathbf{w},y_{ji}\}$ for $i=1,\dots ,C$, $j=1,\dots ,K$. The hidden random variables are $\mathsf{\Theta }=\{\theta ,\lambda ,u,z\}$. We refer to our topic model trained with labels from multiple noisy sources as ML-PA-LDA-MNS (Multi-Label Presence-Absence LDA with Multiple Noisy Sources).

7. Variational EM for ML-PA-LDA-MNS

We now detail the steps for estimating the parameters of our proposed model ML-PA-LDA-MNS. Given the observed words w and the labels $y_1,\dots ,y_K$ for a document d, where $y_j$ is the label vector provided by annotator j. The objective of the model described above is to obtain $p\left(\mathsf{\Theta }\right|\mathbf{d})$. Here, The challenge lies in the intractable computation of $p\left(\mathsf{\Theta }\right|\mathbf{d}),$ which arises due to the intractability in the computation of $p\left(\mathbf{d}\right|\pi )$, where $\pi =\{\alpha ,\beta ,\rho ,\xi \}$. As in ML-PA-LDA, we use variational inference with mean field assumptions [29] to overcome this challenge.

Suppose $q\left(\mathsf{\Theta }\right)$ is the variational distribution over $\mathsf{\Theta }$ for any arbitrary $\mathsf{\Theta }=\{\theta ,\lambda ,u,z\}$ that approximates $p\left(\mathsf{\Theta }\right|\mathbf{d})$. The underlying variational model is provided in Figure 3b. As stated earlier,

$$\begin{array}{cc}\hfill \ln p\left(\mathbf{d}\right|\pi )& ={\mathbb{E}}_{q\left(\mathsf{\Theta }\right)}\left[\ln p(\mathbf{d},\mathsf{\Theta }|\pi )-\ln q(\mathsf{\Theta })\right]+{\mathbb{E}}_{q\left(\mathsf{\Theta }\right)}\left[\ln q\left(\mathsf{\Theta }\right)-\ln p\left(\mathsf{\Theta }\right|\mathbf{d},\pi )\right]\hfill \\ & =\mathcal{L}\left(\mathsf{\Theta }\right)+\mathbf{KL}\left(q\right(\mathsf{\Theta }\left)\right|\left|p\right(\mathsf{\Theta }|\mathbf{d},\pi )).\hfill \end{array}$$

(24)

Variational inference involves maximizing $\mathcal{L}\left(\mathsf{\Theta }\right)$ over the variational parameters $\{\Delta ,\gamma ,\varphi ,\delta \}$ so that $\mathbf{KL}\left(q\right(\mathsf{\Theta }\left)\right|\left|p\right(\mathsf{\Theta }|\mathbf{d},\pi ))$ also gets minimized. As earlier, $u_{ni}=\mathrm{𝟙}\left[u_n=i\right]$ for $i\in \{1,\dots ,C\}$, $z_{nt}=\mathrm{𝟙}\left[z_n=t\right]$ for $t\in \{1,\dots ,T\}$ and $\mathsf{\Gamma }$ denotes the gamma function. From our model (Figure 3a),

$$\begin{array}{cc}\hfill \ln p(\mathbf{d},\mathsf{\Theta }|\pi )& =\ln p(\mathbf{w},y,\theta ,\lambda ,u,z|\pi )\hfill \\ & =\ln p\left(\lambda \right|\xi )+\ln p(\theta \left|\alpha \right)+\ln p\left(u\right)+\ln p\left(z\right|\lambda ,u,\theta )+\ln p(\mathbf{w}|z,\beta )+\ln p\left(y\right|\lambda ,\rho ),\hfill \end{array}$$

and, in turn, similar to ML-PA-LDA,

$$\ln p\left(\lambda \right|\xi )=\sum _{i=1}^C{\lambda }_i\ln {\xi }_i+(1-{\lambda }_i)\ln (1-{\xi }_i),$$

(25)

$$\ln p\left({\theta }_{ij}\right|\alpha )=\ln \mathsf{\Gamma }\left(\sum _{t=1}^T{\alpha }_{ijt}\right)-\sum _{t=1}^T\ln \mathsf{\Gamma }{\alpha }_{ijt}+\sum _{t=1}^T({\alpha }_{ijt}-1)\log {\theta }_{ijt},$$

(26)

$$\log p\left(u\right)=\sum _{n=1}^N\sum _{i=1}^C{u}_{ni}\log 1/C,$$

(27)

$$\log p\left(z\right|u,\lambda ,\theta )=\sum _{n=1}^N\sum _{t=1}^T\sum _{i=1}^C\sum _{j=0}^1{u}_{ni}{\lambda }_{ij}{z}_{nt}\log {\theta }_{ijt},$$

(28)

$$\log p\left(w\right|z,\beta )=\sum _{n=1}^N\sum _{t=1}^T\sum _{j=1}^V{w}_{nj}{z}_{nt}\log {\beta }_{tj}.$$

(29)

In addition, we have the following distribution over the annotators’ labels:

$$\begin{array}{cc}\hfill \log p\left(y\right|\lambda ,\rho )=& \sum _{j=1}^K\sum _{i=1}^C\left[{\lambda }_i{y}_{ji}+(1-{\lambda }_i)(1-y_{ji})\right]\log {\rho }_j+\left[(1-{\lambda }_i)y_{ji}+{\lambda }_i(1-y_{ji})\right]\log 1-{\rho }_j.\hfill \end{array}$$

(30)

Assume the following mean field variational distributions (as per Figure 3b) over $\mathsf{\Theta }$ for a document d:

$$\begin{array}{c}{u}^d\sim \mathrm{Mult}\left({\delta }^d\right),{\lambda }_i^d\sim \mathrm{Bern}\left({\Delta }_i^d\right)\mathrm{for}i=1,\dots ,C,\hfill \\ z^d\sim \mathrm{Mult}\left({\varphi }^d\right),{\theta }_{ij}^d\sim \mathrm{Dir}({\gamma }_{ij}^d)\mathrm{for}i=1,\dots ,C\mathrm{and}j=0,1.\hfill \end{array}$$

Therefore, for a document d,

$$\begin{array}{c}\hfill q\left({\mathsf{\Theta }}^d\right)=\prod _{i=1}^{C}q\left({\lambda }^d\right)\prod _{i=1}^C\prod _{j=0}^{1}q({\theta }_{ij}^d)\prod _{n=1}^N\prod _{i=1}^{C}q\left(u_{ni}^d\right)q\left(z_{ni}^d\right).\end{array}$$

7.1. E-Step Updates for ML-PA-LDA-MNS

The E-step involves computing the document-specific variational parameters ${\mathsf{\Theta }}^d=\{{\delta }^d,{\Delta }^d,{\gamma }^d,{\varphi }^d\}$, for every document d, assuming a fixed value for the parameters $\pi =\{\alpha ,\xi ,\rho ,\beta \}$. As a consequence of the mean field assumptions on the variational distributions, we get the following update rules for the distributions by maximising $\mathcal{L}\left(\mathsf{\Theta }\right)$. When it is clear from context, we omit the superscript d:

$$\begin{array}{cc}\hfill \log q\left(z\right)& ={\mathbb{E}}_{\mathsf{\Theta }\backslash z}\left[p(\mathbf{d},\mathsf{\Theta })\right]\propto {\mathbb{E}}_{u,\lambda ,\theta }\left[\log p\left(z\right|u,\lambda ,\theta )\right]+\log p\left(w\right|z,\beta )\hfill \\ & \propto \sum _{n=1}^N\sum _{t=1}^T{z}_{nt}\left[\sum _{i=1}^C\sum _{j=0}^1\mathbb{E}\left[u_{ni}\right]\mathbb{E}\left[{\lambda }_{ij}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right]\right]+\sum _{n=1}^N\sum _{t=1}^T{z}_{nt}\left[\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}\right].\hfill \end{array}$$

(31)

In the computation of the expectation of ${\mathbb{E}}_{\mathsf{\Theta }\backslash z}\left[p(\mathbf{d},\mathsf{\Theta })\right]$ in Equation (31), The terms in $p(\mathbf{d},\mathsf{\Theta })$ that are a function of z need to be considered as the rest of the terms contribute to the normalizing constant for the density function $q\left(z\right)$. Hence, expectations of $\log p\left(z\right|u,\lambda ,\theta )$ (Equation (28)) and $\log p\left(w\right|z,\beta )$ (Equation (29)) must be taken with respect to $u,\lambda ,\theta$. Therefore,

$$\begin{array}{cc}\hfill \log {\varphi }_{nt}& \propto \sum _{i=1}^C\sum _{j=0}^1\mathbb{E}\left[u_{ni}\right]\mathbb{E}\left[{\lambda }_{ij}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right]+\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1{\delta }_{ni}{\Delta }_i^j{(1-{\Delta }_i)}^{1-j}\mathbb{E}\left[\log {\theta }_{ijt}\right]+\sum _{j=1}^V{w}_{nj}\log {\beta }_{tj}.\hfill \end{array}$$

(32)

Similarly, The updates for the other variational parameters are as follows:

$$\begin{array}{cc}\hfill \log q\left(u\right)& ={\mathbb{E}}_{\mathsf{\Theta }\backslash u}\left[\log p\left(u\right)+p\left(z\right|u,\lambda ,\theta )\right]\hfill \\ & \propto \sum _{n=1}^N\sum _{i=1}^C{u}_{ni}\log 1/C+\sum _{n=1}^N\sum _{t=1}^T\sum _{i=1}^C\sum _{j=0}^1{u}_{ni}\mathbb{E}\left[{\lambda }_{ij}\right]\mathbb{E}\left[z_{nt}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right].\hfill \end{array}$$

(33)

Therefore,

$$\begin{array}{c}\hfill \log {\delta }_{ni}\propto \log \frac{1}{C}+\sum _{t=1}^T{\varphi }_{nt}{\Delta }_i\mathbb{E}\left[\log {\theta }_{i1t}\right]+{\varphi }_{nt}(1-{\Delta }_i)\mathbb{E}\left[\log {\theta }_{i0t}\right],\end{array}$$

(34)

$$\begin{array}{cc}\hfill \log & q\left(\theta \right)={\mathbb{E}}_{\mathsf{\Theta }\backslash \theta }\left[p(\mathbf{d},\mathsf{\Theta })\right]\propto \mathbb{E}\left[p\left(\theta \right|\alpha )+p(z|u,\lambda ,\theta )\right]\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T({\alpha }_{ijt}-1)\log {\theta }_{ijt}+\sum _{n=1}^N\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T\mathbb{E}\left[u_{ni}\right]E\left[z_{nt}\right]E\left[{\lambda }_{ij}\right]\log {\theta }_{ijt}\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T({\alpha }_{ijt}-1)\log {\theta }_{ijt}+\sum _{n=1}^N\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T{\delta }_{ni}{\varphi }_{nt}{\Delta }_i^j{(1-{\Delta }_i)}^{1-j}\log {\theta }_{ijt}\hfill \\ & =\sum _{i=1}^C\sum _{j=0}^1\sum _{t=1}^T({\gamma }_{ijt}-1)\log {\theta }_{ijt},\hfill \end{array}$$

where

$$\begin{array}{c}\hfill {\displaystyle {\gamma }_{ijt}={\alpha }_{ijt}+{\left({\Delta }_i\right)}^j{(1-{\Delta }_i)}^{1-j}\sum _{n=1}^N{\delta }_{ni}{\varphi }_{nt},}\end{array}$$

(35)

$$\begin{array}{c}\log q\left(\lambda \right)\propto \mathbb{E}\left[\log p\left(\lambda \right|\xi )+\log p(z|u,\lambda ,\theta )+\log p\left(y\right|\lambda ,\rho )\right]\hfill \\ =\sum _{i=1}^C{\lambda }_i\log {\xi }_i+(1-{\lambda }_i)\log (1-{\xi }_i)+\sum _{n=1}^N\sum _{t=1}^T\sum _{i=1}^C\sum _{j=0}^1{\lambda }_i^j{(1-{\lambda }_i)}^{1-j}\mathbb{E}\left[u_{ni}\right]\mathbb{E}\left[z_{nt}\right]\mathbb{E}\left[\log {\theta }_{ijt}\right]\hfill \\ +\sum _{j=1}^K\sum _{i=1}^C\left[{\lambda }_i{y}_{ji}+(1-{\lambda }_i)(1-y_{ji})\right]\log {\rho }_j+\left[(1-{\lambda }_i)y_{ji}+{\lambda }_i(1-y_{ji})\right]\log 1-{\rho }_j.\hfill \end{array}$$

Hence,

$$\begin{array}{c}\hfill \log {\Delta }_i\propto \log {\xi }_i+\sum _{j=1}^K{y}_{ji}\log {\rho }_j+(1-y_{ji})\log 1-{\rho }_j+\sum _{n=1}^{N_d}\sum _{t=1}^T{\delta }_{ni}{\varphi }_{nt}\mathbb{E}\left[\log {\theta }_{i1t}\right],\end{array}$$

(36)

$$\begin{array}{c}\hfill \log (1-{\Delta }_i)\propto \log 1-{\xi }_i+\sum _{j=1}^K(1-y_{ji})\log {\rho }_j+y_{ji}\log (1-{\rho }_j)+\sum _{n=1}^{N_d}\sum _{t=1}^T{\delta }_{ni}{\varphi }_{nt}\mathbb{E}\left[\log {\theta }_{i0t}\right].\end{array}$$

(37)

In all of the above update rules, $\mathbb{E}[\log {\theta }_{ijt}^d]=\psi \left({\gamma }_{ijt}\right)-\psi ({\sum }_{t^{\prime }=1}^T{\gamma }_{ij{t}^{\prime }}^d)$, where $\psi (.)$ is the digamma function. In addition, terms involving $y_{ji}$ are considered only when $y_{ji}\ne -1$. Observe that, for the single source model ML-PA-LDA, The variational parameter ${\Delta }_i$ is absent as ${\lambda }_i$ is observed.

7.2. M-Step Updates for ML-PA-LDA-MNS

In the M-step, The parameters $\xi$, $\rho$, $\beta$ and $\alpha$ are estimated using the values of ${\Delta }^d,{\varphi }^d,{\delta }^d,{\gamma }^d$ estimated from the E-step. The function $\mathcal{L}\left(\mathsf{\Theta }\right)$ in Equation (24) is maximized with respect to the parameters $\pi$ yielding the following update equations.

Updates for $\xi$: for $i=1,\dots ,C$:

$$\begin{array}{c}\hfill {\xi }_i=\frac{{\sum }_{d=1}^D{\Delta }_i^d}{D}.\end{array}$$

(38)

Intuitively, Equation (38) makes sense as ${\xi }_i$ is the probability that any document in the corpus belongs to class i. ${\Delta }_i^d$ is the probability that document d belongs to class i and is computed in the E-step. Therefore, ${\xi }_i$ is an average of ${\Delta }_i^d$ over all documents. In ML-PA-LDA, as ${\lambda }_i^d$ was observed, The average was taken over ${\lambda }_i^d$ instead of ${\Delta }_i^d$.

Updates for $\rho$: for $j=1,\dots ,K$:

$$\begin{array}{c}\hfill {\rho }_j=\frac{{\displaystyle \sum _{d=1}^D\sum _{i=1}^C}\mathrm{𝟙}\left[y_{ji}^d\ne -1\right]\left[y_{ji}^d{\Delta }_i^d+(1-y_{ji}^d)(1-{\Delta }_i^d)\right]}{{\displaystyle \sum _{d=1}^D\sum _{i=1}^C}\mathrm{𝟙}\left[y_{ji}^d\ne -1\right]\left[y_{ji}^d{\Delta }_i^d+(1-y_{ji}^d)(1-{\Delta }_i^d)+y_{ji}^d(1-{\Delta }_i^d)+(1-y_{ji}^d){\Delta }_i^d\right].}\end{array}$$

(39)

From Equation (39), we observe that ${\rho }_j$ is the fraction of times that crowd-worker j has provided a label that is consistent with the probability estimate ${\Delta }_i^d$ over all classes i. The implicit assumption is that every crowd-worker has provided at least one label; otherwise, such a crowd-worker need not be considered in the model.

Updates for $\beta$: for $t=1,\dots ,T$; for $j=1,\dots ,V$:

$$\begin{array}{c}\hfill {\beta }_{tj}=\frac{{\sum }_{d=1}^D{\sum }_{n=1}^{N_d}{w}_{nj}^d{\varphi }_{nt}^d}{{\sum }_{d=1}^D{N}_d}.\end{array}$$

(40)

Intuitively, The variational parameter ${\varphi }_{nt}^d$ is the probability that the word $w_n^d$ is associated with topic t. Having updated this parameter in the E-step, ${\beta }_{tj}$ computes the fraction of times the word j is associated with topic t by giving a weight ${\varphi }_{nt}^d$ to its occurrence in document d.

Updates for $\alpha$:

There do not exist closed form updates for $\alpha$ parameters. Hence, we use the Newton Raphson (NR) method to iteratively obtain the solution as follows:

$$\begin{array}{c}\hfill {\alpha }_{ijr}^{t+1}={\alpha }_{ijr}^t-\frac{g_r-c}{h_r},\end{array}$$

(41)

where

$$\begin{array}{c}c=\frac{{\sum }_{\tau =1}^T{g}_{\tau }/h_{\tau }}{z^{-1}+{\sum }_{\tau =1}^{T}1/h_{\tau }},z=D{\psi }^{\prime }\left(\sum _{t^{\prime }=1}^T{\alpha }_{ij{t}^{\prime }}^t\right),h_{\tau }=-D{\psi }^{\prime }({\alpha }_{ijr}^t),\hfill \end{array}$$

$$g_r=D\left[\psi \left(\sum _{\tau =1}^T{\alpha }_{ij\tau }^t\right)-\psi \left({\alpha }_{ijr}^t\right)\right]+\sum _{d=1}^D\left[\psi \left({\gamma }_{ijr}^d\right)-\psi \left(\sum _{\tau =1}^T{\gamma }_{ij\tau }^d\right)\right].$$

The M-step updates for $\beta$ and $\alpha$ involved in ML-PA-LDA (single source version) are the same as the updates in the ML-PA-LDA-MNS. The parameter $\rho$ is absent in ML-PA-LDA. The overall algorithm for learning the parameters is provided in Algorithm 2.

Algorithm 2 Algorithm for learning the parameters $\pi$ during the training phase of ML-PA-LDA-MNS.
repeat
for d = 1 to D do
Initialize ${\mathsf{\Theta }}^d$	▹ E-step
repeat
Update ${\mathsf{\Theta }}^d$ sequentially using Equations (32), (34)–(37).
until convergence
end for	▹ M-step
Update $\xi$ using Equation (38)
Update $\rho$ using Equation (39)
Update $\beta$ using Equation (40)
Perform NR updates for $\alpha$ using Equation (41), till convergence.
until convergence

Inference

The inference in ML-PA-LDA-MNS is identical to the inference in ML-PA-LDA, as, for both of the models, in the test phase, the labels of a new document are unknown. More specifically, in the inference stage for ML-PA-LDA-MNS, the sources also do not provide any information.

8. Smoothing

In the model for the documents described in Section 3, we modeled $\beta$ to be a parameter that governs the multinomial distributions for generating the words from each topic. In general, a new document can include words that have not been encountered in any of the training documents. The unsmoothed model described earlier does not handle this issue. In order to handle this, we must “smoothen” the multinomial parameters involved [5]. One way to perform smoothing is to model $\beta$ as a multinomial random variable over the vocabulary $\mathcal{V}$, with parameters $\eta$. Again, due to the intractable nature of the computations, we model the variational distribution for $\beta$ as $\beta \sim \mathrm{Dir}\left(\chi \right)$. We estimate the variational parameter $\chi$ in the E-step of variational EM using Equation (42), assuming $\eta$ is known:

$$\begin{array}{c}\hfill {\chi }_{tj}={\eta }_{tj}+\sum _{d=1}^D\sum _{n=1}^{N_d}{\varphi }_{nt}^d{w}_{nj}^d.\end{array}$$

(42)

The model parameter $\eta$ is estimated in the M-step using Newton Raphson method as follows:

$$\begin{array}{c}\hfill {\eta }_{ir}^{t+1}={\eta }_{ir}^t-\frac{g_r-c}{h_r},\end{array}$$

(43)

where

$$\begin{array}{c}c=\frac{{\sum }_{\tau =1}^V{g}_{\tau }/h_{\tau }}{z^{-1}+{\sum }_{\tau =1}^{T}1/h_{\tau }},z={\psi }^{\prime }\left(\sum _{j^{\prime }=1}^V{\eta }_{i{j}^{\prime }}^t\right),h_r=-{\psi }^{\prime }\left({\eta }_{ir}^t\right),\hfill \end{array}$$

$$g_r=\left[\psi \left(\sum _{j^{\prime }=1}^V{\eta }_{i{j}^{\prime }}^t\right)-\psi \left({\eta }_{ir}^t\right)\right]+\left[\psi \left({\chi }_{ir}\right)-\psi \left(\sum _{j^{\prime }=1}^V{\chi }_{i{j}^{\prime }}\right)\right].$$

The steps for the derivation are similar to the steps for non-smooth version.

9. Experiments

In order to test the efficacy of the proposed techniques, we evaluate our model on datasets from several domains.

9.1. Dataset Descriptions

We have carried out our experiments on several datasets from the text domain as well as the non-text domain. Our code is available on bitbucket [30]. We now describe the datasets and the pre-processing steps below.

9.1.1. Text Datasets

In the text domain, we have performed studies on the Reuters-21578, Bibtex and Enron datasets.

Reuters-21578: The Reuters-21578 dataset [31] is a collection of documents with news articles. The original corpus had 10,369 documents and a vocabulary of 29,930 words. We performed stemming using the Porter Stemmer algorithm [32] and also removed the stop words. From this set, the words that occurred more than 50 times across the corpus were retained and only documents which contained more than 20 words were retained. Finally, the most commonly occurring top 10 labels were retained, namely, acq, crude, earn, fx, grain , interest, money, ship, trade, and wheat. This led to a total of 6547 documents and a vocabulary of size 1996. Of these, a random 80% was used as a training set and the remaining 20% as test.

Bibtex: The Bibtex dataset [33] was released as part of the ECML-PKDD 2008 Discovery Challenge. The task is to assign tags such as physics, graph, electrochemistry, etc. to Bibtex entries. There are a total of 4880 and 2515 entries in the training set and test, respectively. The size of the vocabulary is 1836 and the number of tags is 159.

Enron: The Enron dataset [34] is a collection of emails for which a set of pre-defined categories are to be assigned. There are a total of 1123 and 573 training and test instances, respectively, with a vocabulary of 1001 words. The total number of email tags are 53.

Delicious: The Delicious dataset [35] is a collection of web pages tagged by users. Since the tags or classes are assigned by users, each web page has multiple tags. The corpus from Mulan [34] had a vocabulary of 500 words and 983 classes. The training set had 14,406 instances and the test set had 4671 instances. From the training set, only documents that contained more than 50 words were retained, and the most commonly occurring top 20 classes were retained. The final training set used had 430 training instances and 108 test instances.

9.1.2. Non-Text Datasets

We also evaluate our model on datasets from domains other than text, where the notion of words is not explicit.

Converting real valued features to words: Since we assume a bag-of-words model, we must replace every real-valued feature with a ‘word’ from a ‘vocabulary’. We begin by choosing an appropriate size for the vocabulary. Thereafter, we collect every real number that occurs across features and instances in the corpus into a set. We then cluster this set into V clusters, using the k-means algorithm, where V is the size of the vocabulary previously chosen. Therefore, each real valued feature has a new representative word given by the nearest cluster center to the feature under consideration. The corpus is then generated with this new feature representation scheme.

Yeast: The Yeast dataset [18] contains a set of genes that may be associated with several functional classes. There are 1500 training examples and 917 examples in the test set with a total of 14 classes and 103 real valued features.

Scene: The Scene dataset [36] is a dataset of images. The task is to classify images into the following six categories: beach, sunset, fall, field, mountain, and urban. The dataset contains 1211 instances in the training set and 1196 instances in the test set with a total of 294 real valued features.

In our experiments, we use the measures, accuracy across classes, micro-f1 score and average class log likelihood on the test sets to evaluate our model. Let $TP$, $TN$, $FP$ and $FN$ denote the number of true positives, true negatives, false positives and false negatives, respectively, with respect to all classes. Then, the overall accuracy is computed as:

$$\mathrm{accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}.$$

(44)

The micro-f1 is the harmonic of micro-precision and micro-recall where

$$\begin{array}{cc}\hfill \mathrm{micro}-\mathrm{precision}& =TP/(TP+FP)\mathrm{and}\hfill \\ \hfill \mathrm{micro}-\mathrm{recall}& =TP/(TP+FN).\hfill \end{array}$$

(45)

The average class log-likelihood on the test instances is computed as follows:

$$\begin{array}{c}\hfill \log -\mathrm{l}=\frac{{\sum }_{d=1}^{D_{test}}{\sum }_{i=1}^C{\lambda }_i^d\log {\Delta }_i^d+(1-{\lambda }_i^d)\log (1-{\Delta }_i^d)}{D_{test}\times C},\end{array}$$

where $D_{test}$ is the number of instances in the test set. Further details on these measures can be found in the survey [37].

9.2. Results: ML-PA-LDA (with a Single Source)

We run our model first assuming labels from a perfect source.

In

Table 1. Comparison of average accuracy of various multi-label classification techniques. The following abbreviations have been used: RAKel: Random k-label sets, MCC: Monte Carlo Classifier Chains, BRq: Binary Relevance Method - Random Subspace, BCC: Bayesian Chain Classifiers, SLDA: Supervised Latent Dirichlet Allocation

Dataset	RAKel (J48)	MCC	BRq	BCC	SLDA	ML-PA-LDA	ML-PA-LDA-MNS
Reuters	0.881	0.876	0.863	0.867	0.897	0.969	0.942
Bibtex	0.293	0.290	0.309	0.299	0.984	0.984	0.981
Enron	0.402	0.389	0.430	0.411	0.937	0.939	0.938
Delicious	0.316	0.307	0.338	0.345	0.799	0.804	0.803
Scene	0.577	0.580	0.550	0.594	0.823	0.823	0.818
Yeast	0.415	0.432	0.462	0.413	0.767	0.767	0.767

, we compare the performance of our non-annotator model vs. other methods such as RAKel, Monte Carlo Classifier Chains (MCC) [38], Binary Relevance Method - Random Subspace (BRq) [39], Bayesian Chain Classifiers (BCC) [40] and SLDA. BCC [40] is a probabilistic method that constructs a chain of classifiers by modeling the dependencies between the classes using a Bayesian network. MCC instead uses a Monte Carlo strategy to learn the dependencies. BRq improves upon binary relevance methods of combining classifiers by constructing an ensemble. As mentioned earlier, RAKel draws subsets of the classes, each of size k and constructs ensemble classifiers. The implementations of RAKel, MCC, BRq and BCC provided by Meka [41] were used. For SLDA, the code provided by the authors of [13] was used. Since SLDA does not explicitly handle multi-label datasets, we built C SLDA classifiers (where C is the number of classes) and used them in a one-vs-rest fashion.

On the Reuters, Bibtex and Enron datasets, ML-PA-LDA (without the annotators) performs significantly better than SLDA. On Scene and Yeast datasets, ML-PA-LDA and SLDA give the same performance. It is to be noted that these datasets, known to be hard, are from the images and biology domains, respectively. As can be seen from Table 1, our model ML-PA-LDA gives a better overall performance than SLDA and also does not require training C binary classifiers. This advantage is a significant one, especially in datasets such as Bibtex where the number of classes is 159.

We compared the performance of our algorithm with the size of the datasets used for training as well as the number of topics used. The results of our model are shown in Figure 4c,f,i. An increase in the size of the dataset improves the performance of our model with respect to all of the measures in use. Similarly, an increase in the number of topics generally improves the measures under consideration. Note that an increase in the number of topics corresponds to an increased model complexity (and also increased running time). A striking observation is the low accuracy, log likelihood and micro-f1 scores associated with the model when the number of topics = 80 (eight times the number of classes) and the size of the dataset is low (S = 25%). This is expected as the number of parameters to be estimated is too large (as the model complexity is high) to be learned using very few training examples. However, as more training data is available, the model achieves enhanced performance. This observation is consistent with Occam’s razor [42].

Statistical Significance Tests:

From Table 1, since the performance of SLDA and ML-PA-LDA are similar in some of the datasets such as Enron and Bibtex, we performed tests to check for statistical significance.

Binomial Tests: We first performed binomial tests [43] for statistical significance. The null hypothesis was fixed as “Average accuracy of SLDA is the same as that of ML-PA-LDA” and the alternate hypothesis was fixed as “Average accuracy of ML-PA-LDA is better than that of SLDA”. We executed our method as well as SLDA with 10 random initializations each on the data sets Reuters, Enron, Bibtex and Delicious. In order to reject the null hypothesis in favour of the alternate hypothesis, at a level $\alpha =0.05$, SLDA should have been better than ML-PA-LDA in either $0,1$ or 2 runs. However, during this experiment, it was observed that the accuracy of ML-PA-LDA was better than SLDA in every run for all the datasets. Therefore, for each of these datasets, the null hypothesis was rejected with a p-value of 0.0009.

t-Tests: We also performed the unpaired two-sample t-test [44] with the null hypothesis “Mean accuracy of SLDA is the same as the mean accuracy of ML-PA-LDA (${\mu }_{\mathrm{SLDA}}={\mu }_{\mathrm{ML}-\mathrm{PA}-\mathrm{LDA}}$)” and alternate hypothesis “Mean accuracy of ML-PA-LDA is higher than SLDA (${\mu }_{\mathrm{SLDA}}<{\mu }_{\mathrm{ML}-\mathrm{PA}-\mathrm{LDA}}$)”. The null hypothesis was rejected at the level $\alpha =0.05$ for all of the datasets—Reuters, Delicious, Bibtex and Enron. In Reuters, for instance, the null hypothesis was to be rejected if ${\widehat{\mu }}_{\mathrm{SLDA}}-{\widehat{\mu }}_{\mathrm{ML}-\mathrm{PA}-\mathrm{LDA}}<-0.0073$, where ${\widehat{\mu }}_M$ denotes the empirically obtained mean accuracy for method M. The actual difference obtained was ${\widehat{\mu }}_{\mathrm{SLDA}}-{\widehat{\mu }}_{\mathrm{ML}-\mathrm{PA}-\mathrm{LDA}}=-0.048<-0.0073$, and the null hypothesis was rejected with a p-value of $8.47\times {10}^{-8}$ for the Reuters dataset. For the Delicious dataset, the rule for rejecting the null hypothesis was ${\widehat{\mu }}_{\mathrm{SLDA}}-{\widehat{\mu }}_{\mathrm{ML}-\mathrm{PA}-\mathrm{LDA}}<-0.001,$ while the actual difference obtained was ${\widehat{\mu }}_{\mathrm{SLDA}}-{\widehat{\mu }}_{\mathrm{ML}-\mathrm{PA}-\mathrm{LDA}}=-0.003<-0.001$.

The above tests show that our results are statistically significant.

9.3. Results: ML-PA-LDA-MNS (with Multiple Noisy Sources)

To verify the performance of the annotator model ML-PA-LDA-MNS where the labels are provided by multiple noisy annotators, we simulated 50 annotators with varying qualities. The $\rho$-values of the annotators were sampled from a uniform distribution. For 10 of these annotators, $\rho$ was sampled from $U[0.51,0.65]$. For another 20 of them, $\rho$ was sampled from $U[0.66,0.85],$ and, for the remaining 20 of them, $\rho$ was sampled from $U[0.86,0.9999]$. This captures the heterogeneity in the annotator qualities. For each document in the training set, a random $10\%(=5)$ of annotators were picked for generating the noisy labels.

In Table 1, we report the performance of the ML-PA-LDA-MNS. We find that the performance of ML-PA-LDA-MNS is close to that of ML-PA-LDA and often better than or at par with SLDA (from Table 1), in spite of having access to only noisy labels. On Scene and Yeast datasets, ML-PA-LDA-MNS, ML-PA-LDA and SLDA give the same performance. In

Table 2. Performances of ML-PA-LDA and ML-PA-LDA-MNS for different sizes of training sets, for a fixed number of topics (=20). Results are shown for the Reuters dataset. A similar trend is demonstrated by other datasets (omitted for space).

% of Training	ML-PA-LDA	ML-PA-LDA	ML-PA-LDA-MNS	ML-PA-LDA-MNS	Ann RMSE
Set Used	Avg Accuracy	Avg Microf1	Avg Accuracy	Avg Microf1
10	0.949	0.762	0.927	0.616	0.023
30	0.953	0.784	0.930	0.619	0.014
50	0.955	0.787	0.936	0.629	0.011
70	0.961	0.828	0.937	0.650	0.010
100	0.969	0.829	0.942	0.669	0.009

, we compare the performance of ML-PA-LDA-MNS and ML-PA-LDA on the Reuters dataset under varying amounts of training data. With more training data, both models perform better. We also report the annotator root mean square error or “Ann RMSE”, which is the L2 norm of the difference in predicted qualities of the annotators vs. the true qualities. $\mathrm{Ann}\mathrm{RMSE}=\sqrt{{\sum }_{j=1}^K{|{\widehat{\rho }}_j-{\rho }_j|}^2/K},$ where ${\widehat{\rho }}_j$ is the quality of annotator j as predicted by our variational EM algorithm and ${\rho }_j$ is the true annotator quality, which is unknown during training. We find that “Ann RMSE” decreases as more training data is available, showing the efficacy of our model for learning the qualities of the annotators.

Similar to the experiment carried out on ML-PA-LDA, we vary the number of topics as well as dataset sizes and compute all of the measures used. The plots are shown in Figure 4 (first two columns) and help in understanding how T, the number of topics, must be tuned depending on the size of the available training set. As in ML-PA-LDA, an increase in the topics as well as dataset size improves the performance of ML-PA-LDA-MNS in general. Therefore, as more training data become available, having a larger number of topics helps.

Adversarial Annotators

We also tested the robustness of our model against labels from adversarial or malicious annotators. Such malicious annotators occur in many scenarios such as crowdsourcing. An adversarial annotator is characterized by a quality parameter $\rho <0.5$. As in the previous case, we simulated 50 annotators. The $\rho$-values of 10 of them were sampled from $U[0.0001,0.1]$. For another 15 annotators, $\rho$ was sampled from $U[0.51,0.65]$. For another 20 of them, $\rho$ was sampled from $U[0.66,0.85],$ and, for the remaining five of them, $\rho$ was sampled from $U[0.86,0.9999]$. The choice of the proportion of malicious annotators is as per literature [23]. From the Reuters dataset, we obtained an average accuracy of 0.955, average class log likelihood of −0.193, average micro-f1 of 0.793 and an average Ann-RMSE of 0.002 over five runs with 40 topics. This shows that, even in the presence of malicious annotators, our model remains unaffected and performs well.

10. Conclusions

We have introduced a new approach for multi-label classification using a novel topic model, which uses information about the presence as well as absence of classes. In the scenario when the true labels are not available and instead a noisy version of the labels is provided by the annotators, we have adapted our topic model to learn the parameters including the qualities of the annotators. Our experiments indeed validate the superior performance of our approach.