A Parametric Bayesian Approach in Density Ratio Estimation
Abstract
:1. Introduction
2. Density Ratio Estimation for the Exponential Distribution Family
3. Bayesian DRE
3.1. log–Huber’s Robust Loss
3.1.1. Log– Loss Function
3.1.2. Log- Loss Function
3.2. Examples of the Bayesian DRE and Some Applications
4. Other Applications
- 1
- Estimating the -divergence function between two probability densities:A discrepancy measure between densities and applicable to the class of Ali-Silvey [23] distance, also known as -divergence (Csiszàr, [24]), is given byNote that some of the notable divergence functions, such as Kullback–Leibler, reverse Kullback–Leibler (RKL), and Hellinger divergence functions correspond to , and 0, respectively, and belong to this class. So, if is estimated by under log– (or (t) under log– losses), then applying the Monte Carlo approximations method, the -divergence is also estimated by
- 2
- Plug-in-type estimation of the density ratio under KL loss function:Consider the plug-in density estimator, say , for estimating in the exponential family (1), based on the KL loss. We haveNext, finding the plug-in density estimator that minimizes KL loss is equivalent to finding the point estimator which minimizes the posterior expectation associated with the loss function in (24). Therefore, implies , and hence the Bayes estimator of is given bySimilar arguments can be applied to for estimating in the exponential family (2), and we have:By setting the case when both densities follow the identical distribution from (1) (e.g., the ratio of two normal or two Poisson, etc.), substituting the Bayes estimators obtained in (25) and (26) into the plug-in estimator of givesNote that can be obtained similarly by replacing posterior medians instead of posterior expectations for above.
5. Numerical Illustrations
6. Concluding Remarks
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Sugiyama, M.; Suzuki, T.; Kanamori, T. Density Ratio Estimation in Machine Learning; Cambridge University Press: Cambridge, UK, 2012. [Google Scholar]
- Kanamori, T.; Hido, S.; Sugiyama, M. A Least-squares Approach to Direct Importance Estimation. J. Mach. Learn. Res. 2009, 10, 1391–1445. [Google Scholar]
- Sugiyama, M.; Kanamori, T.; Suzuki, T.; Hido, S.; Sese, J.; Takeuchi, I.; Wang, L. A density-ratio framework for statistical data processing. IPSJ Trans. Comput. Vis. Appl. 2009, 1, 183–208. [Google Scholar] [CrossRef]
- Sugiyama, M.; Yamada, M.; Bunau, P.V.; Suzuki, T.; Kanamori, T.; Kawanabe, M. Direct density-ratio estimation with dimensionality reduction via least-squares hetero-distributional subspace search. Neural Netw. 2011, 24, 183–198. [Google Scholar] [CrossRef] [PubMed]
- Sugiyama, M.; Müller, K.R. Input-dependent estimation of generalization error under covariate shift. Stat. Decis. 2005, 23, 249–279. [Google Scholar] [CrossRef]
- Quiñonero-Candela, J.; Sugiyama, M.; Schwaighofer, A.; Lawrence, N. Dataset Shift in Machine Learning; MIT Press: Cambridge, MA, USA, 2009. [Google Scholar]
- Suzuki, T.; Sugiyama, M.; Kanamori, T.; Sese, J. Mutual information estimation reveals global associations between stimuli and biological processes. BMC Bioinform. 2009, 10, S52. [Google Scholar] [CrossRef] [PubMed]
- Suzuki, T.; Sugiyama, M. Sufficient dimension reduction via squared-loss mutual information estimation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia Laguna, Italy, 13–15 May 2010; pp. 804–811. [Google Scholar]
- Sugiyama, M.; Hara, S.; von Bünau, P.; Suzuki, T.; Kanamori, T.; Kawanabe, M. Direct density ratio estimation with dimensionality reduction. In Proceedings of the SIAM International Conference on Data Mining, Columbus, OH, USA, 29 April–1 May 2010. [Google Scholar]
- Hido, S.; Tsuboi, Y.; Kashima, H.; Sugiyama, M.; Kanamori, T. Inlier–Based Outlier Detection via Direct Density Ratio Estimation. In Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy, 15–19 December 2008; pp. 223–232. [Google Scholar]
- Thomas, O.; Dutta, R.; Corander, J.; Kaski, S.; Gutmann, M.U. Likelihood-free inference by ratio estimation. arXiv 2016, arXiv:1611.10242. [Google Scholar]
- Gretton, A.; Smola, A.; Huang, J.; Schmittfull, M.; Borgwardt, K.; Schölkopf, B. Covariate shift by kernel mean matching. Dataset Shift Mach. Learn. 2009, 3, 5. [Google Scholar]
- Sugiyama, M.; Suzuki, T.; Nakajima, S.; Kashima, H.; von Bunau, P.; Kawanabe, M. Direct importance estimation for covariate shift adaptation. Ann. Inst. Stat. Math. 2008, 60, 699–746. [Google Scholar] [CrossRef]
- Nguyen, X.; Wainwright, M.J.; Jordan, M.I. Estimating divergence functional and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory 2010, 56, 5847–5861. [Google Scholar] [CrossRef]
- Deledalle, C.A. Estimation of Kullback-Leibler losses for noisy recovery problems within the exponential family. Electron. J. Stat. 2017, 11, 3141–3164. [Google Scholar] [CrossRef]
- Sugiyama, M.; Suzuki, T.; Kanamori, T. Density-ratio matching under the Bregman divergence: A unified framework of density-ratio estimation. Ann. Inst. Stat. Math. 2012, 64, 1009–1044. [Google Scholar] [CrossRef]
- Lehman, E.L.; Casella, G. Theory of Point Estimation, 2nd ed.; Springer Texts in Statistics; Springer: New York, NY, USA, 1998. [Google Scholar]
- Huber, P. Robust estimation of a location parameter. Ann. Math. Stat. 1964, 53, 73–101. [Google Scholar] [CrossRef]
- Huber, P.J.; Ronchetti, E.M. Robust Statistics; John Wiley & Sons: Hoboken, NJ, USA, 2009; Volume 2. [Google Scholar]
- Brown, L.D. Fundamentals of Exponential Families; IMS: Hayward, CA, USA, 1986. [Google Scholar]
- Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat. 1951, 22, 79–86. [Google Scholar] [CrossRef]
- Nielsen, F.; Nock, R. Entropies and cross-entropies of exponential families. In Proceedings of the 2010 IEEE International Conference on Image Processing, Hong Kong, China, 26–29 September 2010; pp. 3621–3624. [Google Scholar]
- Ali, S.M.; Silvey, S.D. A general class of coefficients of divergence of one distribution from another. J. R. Stat. Soc. Ser. B 1966, 28, 131–142. [Google Scholar] [CrossRef]
- Csiszàr, I. Information-type measures of difference of probability distributions and indirect observation. Stud. Sci. Math. Hung. 1967, 2, 299–318. [Google Scholar]
- Póczos, B.; Schneider, J. On the estimation of alpha-divergences. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Ft. Lauderdale, FL, USA, 11–13 April 2011; pp. 609–617. [Google Scholar]
- Krnjajic, M.; Kottas, A.; Draper, D. Parametric and nonparametric Bayesian model specification: A case study involving models for count data. Comput. Stat. Data Anal. 2008, 52, 2110–2128. [Google Scholar] [CrossRef]
Density | Conjugate Prior | Correction Factor | Condition for |
---|---|---|---|
, | |||
, where | |||
, | |||
1 | , and | ||
Density | KL |
---|---|
Bet | |
Gam | |
log– | |
Ray | |
Pa | |
U | Provided |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Sadeghkhani, A.; Peng, Y.; Lin, C.D. A Parametric Bayesian Approach in Density Ratio Estimation. Stats 2019, 2, 189-201. https://doi.org/10.3390/stats2020014
Sadeghkhani A, Peng Y, Lin CD. A Parametric Bayesian Approach in Density Ratio Estimation. Stats. 2019; 2(2):189-201. https://doi.org/10.3390/stats2020014
Chicago/Turabian StyleSadeghkhani, Abdolnasser, Yingwei Peng, and Chunfang Devon Lin. 2019. "A Parametric Bayesian Approach in Density Ratio Estimation" Stats 2, no. 2: 189-201. https://doi.org/10.3390/stats2020014