Surveying Racial Bias in Facial Recognition: Balancing Datasets and Algorithmic Enhancements
Abstract
:1. Introduction
2. Methods
2.1. Datasets Methods
2.2. Dataset Balancing through Generation Methods
2.3. Network Improvements across Human Race Methods
2.4. Terminology
3. Facial Recognition Dataset Comparison
3.1. Unbalanced Datasets
3.2. Balanced Datasets
3.3. Evaluating Networks Trained on Unbalanced Datasets
3.4. Intersection of Racially Balanced Datasets and Gender Bias
3.5. Discussion on Racially Balanced Datasets
4. Balancing Datasets through Data Generation
4.1. Understanding the Impact of Data Generation
4.2. Cross-Race Transformation
4.3. Intersection of Racial Dataset Balancing and Gender
4.4. Discussion on Balancing Datasets through Data Generation
5. Network Improvements across Human Race
5.1. Network Improvements on the RFW Dataset
5.2. Racial Classification
5.2.1. Two-Class Race Classification
5.2.2. Multiple-Class Race Classification
5.3. Related Tasks
5.4. Intersection of Network Improvements across Race and Gender
5.5. Discussion on Network Improvements
6. Future Work
7. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Coe, J.; Atay, M. Evaluating impact of race in facial recognition across machine learning and deep learning algorithms. Computers 2021, 10, 113. [Google Scholar] [CrossRef]
- Cavazos, J.G.; Phillips, P.J.; Castillo, C.D.; O’Toole, A.J. Accuracy comparison across face recognition algorithms: Where are we on measuring race bias? IEEE Trans. Biom. Behav. Identity Sci. 2020, 3, 101–111. [Google Scholar] [CrossRef] [PubMed]
- Krishnapriya, K.; Albiero, V.; Vangara, K.; King, M.C.; Bowyer, K.W. Issues related to face recognition accuracy varying based on race and skin tone. IEEE Trans. Technol. Soc. 2020, 1, 8–20. [Google Scholar] [CrossRef]
- Pagano, T.P.; Loureiro, R.B.; Araujo, M.M.; Lisboa, F.V.N.; Peixoto, R.M.; Guimaraes, G.A.d.S.; Santos, L.L.d.; Cruz, G.O.R.; de Oliveira, E.L.S.; Cruz, M.; et al. Bias and unfairness in machine learning models: A systematic literature review. arXiv 2022, arXiv:2202.08176. [Google Scholar]
- Parikh, R.B.; Teeple, S.; Navathe, A.S. Addressing bias in artificial intelligence in health care. JAMA 2019, 322, 2377–2378. [Google Scholar] [CrossRef] [PubMed]
- Ntoutsi, E.; Fafalios, P.; Gadiraju, U.; Iosifidis, V.; Nejdl, W.; Vidal, M.E.; Ruggieri, S.; Turini, F.; Papadopoulos, S.; Krasanakis, E.; et al. Bias in data-driven artificial intelligence systems—An introductory survey. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2020, 10, e1356. [Google Scholar] [CrossRef]
- Fountain, J.E. The moon, the ghetto and artificial intelligence: Reducing systemic racism in computational algorithms. Gov. Inf. Q. 2022, 39, 101645. [Google Scholar] [CrossRef]
- Abdurrahim, S.H.; Samad, S.A.; Huddin, A.B. Review on the effects of age, gender, and race demographics on automatic face recognition. Vis. Comput. 2018, 34, 1617–1630. [Google Scholar] [CrossRef]
- Suresh, H.; Guttag, J.V. A framework for understanding unintended consequences of machine learning. arXiv 2019, arXiv:1901.10002. [Google Scholar]
- Wang, M.; Deng, W. Mitigating bias in face recognition using skewness-aware reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual, 14–19 June 2020; pp. 9322–9331. [Google Scholar]
- Morales, A.; Fierrez, J.; Vera-Rodriguez, R.; Tolosana, R. Sensitivenets: Learning agnostic representations with application to face images. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 43, 2158–2164. [Google Scholar] [CrossRef]
- Darabant, A.S.; Borza, D.; Danescu, R. Recognizing Human Races through Machine Learning—A Multi-Network, Multi-Features Study. Mathematics 2021, 9, 195. [Google Scholar] [CrossRef]
- Karkkainen, K.; Joo, J. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Virtual, 5–9 January 2021; pp. 1548–1558. [Google Scholar]
- Hupont, I.; Fernández, C. Demogpairs: Quantifying the impact of demographic imbalance in deep face recognition. In Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France, 14–18 May 2019; pp. 1–7. [Google Scholar]
- Wang, M.; Deng, W.; Hu, J.; Tao, X.; Huang, Y. Racial faces in the wild: Reducing racial bias by information maximization adaptation network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 692–702. [Google Scholar]
- Li, A.; Tan, Z.; Li, X.; Wan, J.; Escalera, S.; Guo, G.; Li, S.Z. CASIA-SURF CeFA: A Benchmark for Multi-modal Cross-ethnicity Face Anti-spoofing. arXiv 2020, arXiv:2003.05136. [Google Scholar]
- Robinson, J.P.; Qin, C.; Henon, Y.; Timoner, S.; Fu, Y. Balancing biases and preserving privacy on balanced faces in the wild. IEEE Trans. Image Process. 2023, 32, 4365–4377. [Google Scholar] [CrossRef] [PubMed]
- Ning, X.; Nan, F.; Xu, S.; Yu, L.; Zhang, L. Multi-view frontal face image generation: A survey. Concurr. Comput. Pract. Exp. 2023, 35, e6147. [Google Scholar] [CrossRef]
- Page, M.J.; Moher, D.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. bmj 2021, 372, n160. [Google Scholar] [CrossRef] [PubMed]
- Serna, I.; Morales, A.; Fierrez, J.; Obradovich, N. Sensitive loss: Improving accuracy and fairness of face representations with discrimination-aware deep learning. Artif. Intell. 2022, 305, 103682. [Google Scholar] [CrossRef]
- Samaria, F.S.; Harter, A.C. Parameterisation of a stochastic model for human face identification. In Proceedings of the 1994 IEEE Workshop on Applications of Computer Vision, Sarasota, FL, USA, 5–7 December 1994; pp. 138–142. [Google Scholar]
- Yucer, S.; Akçay, S.; Al-Moubayed, N.; Breckon, T.P. Exploring racial bias within face recognition via per-subject adversarially-enabled data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Virtual, 14–19 June 2020; pp. 18–19. [Google Scholar]
- Merriam-Webster. Race. Available online: https://www.merriam-webster.com/dictionary/race (accessed on 24 May 2024).
- Merriam-Webster. Ethnicity. Available online: https://www.merriam-webster.com/dictionary/ethnicity (accessed on 24 May 2024).
- Merriam-Webster. Ethnic. Available online: https://www.merriam-webster.com/dictionary/ethnic (accessed on 24 May 2024).
- Office of Management and Budget. Revisions to OMB’s Statistical Policy Directive No. 15: Standards for Maintaining, Collecting, and Presenting Federal Data on Race and Ethnicity; Office of Information and Regulatory Affairs, Office of Management and Budget, Executive Office of the President: Washington, DC, USA, 2024.
- Howell, A.J.; Buxton, H. Towards unconstrained face recognition from image sequences. In Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, Killington, VT, USA, 14–16 October 1996; pp. 224–229. [Google Scholar]
- Huang, G.B.; Mattar, M.; Berg, T.; Learned-Miller, E. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Proceedings of the Workshop on faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, Marseille, France, 17–20 October 2008. [Google Scholar]
- Bae, G.; de La Gorce, M.; Baltrušaitis, T.; Hewitt, C.; Chen, D.; Valentin, J.; Cipolla, R.; Shen, J. Digiface-1m: 1 million digital face images for face recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 3–7 January 2023; pp. 3526–3535. [Google Scholar]
- Shorten, C.; Khoshgoftaar, T.M. A survey on image data augmentation for deep learning. J. Big Data 2019, 6, 60. [Google Scholar] [CrossRef]
- Mehrabi, N.; Morstatter, F.; Saxena, N.; Lerman, K.; Galstyan, A. A survey on bias and fairness in machine learning. ACM Comput. Surv. (CSUR) 2021, 54, 1–35. [Google Scholar] [CrossRef]
- Fisher, R.A. Statistical methods for research workers. In Breakthroughs in Statistics: Methodology and Distribution; Springer: Berlin/Heidelberg, Germany, 1970; pp. 66–70. [Google Scholar]
- Cowles, M.; Davis, C. On the origins of the. 05 level of statistical significance. Am. Psychol. 1982, 37, 553. [Google Scholar] [CrossRef]
- Ortega-Garcia, J.; Fierrez, J.; Alonso-Fernandez, F.; Galbally, J.; Freire, M.R.; Gonzalez-Rodriguez, J.; Garcia-Mateo, C.; Alba-Castro, J.L.; Gonzalez-Agulla, E.; Otero-Muras, E.; et al. The multiscenario multienvironment biosecure multimodal database (bmdb). IEEE Trans. Pattern Anal. Mach. Intell. 2009, 32, 1097–1111. [Google Scholar] [CrossRef]
- Kumar, N.; Berg, A.; Belhumeur, P.N.; Nayar, S. Describable visual attributes for face verification and image search. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 1962–1977. [Google Scholar] [CrossRef] [PubMed]
- Wolf, L.; Hassner, T.; Maoz, I. Face recognition in unconstrained videos with matched background similarity. In Proceedings of the CVPR 2011, Colorado Springs, CO, USA, 21–25 June 2011; pp. 529–534. [Google Scholar] [CrossRef]
- Yi, D.; Lei, Z.; Liao, S.; Li, S.Z. Learning face representation from scratch. arXiv 2014, arXiv:1411.7923. [Google Scholar]
- Yang, S.; Luo, P.; Loy, C.C.; Tang, X. From facial parts responses to face detection: A deep learning approach. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 13–16 December 2015; pp. 3676–3684. [Google Scholar]
- Parkhi, O.M.; Vedaldi, A.; Zisserman, A. Deep Face Recognition. In Proceedings of the BMVC 2015—British Machine Vision Conference 2015, British Machine Vision Association, Swansea, UK, 7–10 September 2015. [Google Scholar]
- Kemelmacher-Shlizerman, I.; Seitz, S.M.; Miller, D.; Brossard, E. The megaface benchmark: 1 million faces for recognition at scale. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 4873–4882. [Google Scholar]
- Guo, Y.; Zhang, L.; Hu, Y.; He, X.; Gao, J. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. In Proceedings of the European Conference on Computer Vision, 2016, Amsterdam, The Netherlands, 11–14 October 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 87–102. [Google Scholar]
- Zhang, Z.; Song, Y.; Qi, H. Age progression/regression by conditional adversarial autoencoder. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 5810–5818. [Google Scholar]
- Cao, Q.; Shen, L.; Xie, W.; Parkhi, O.M.; Zisserman, A. Vggface2: A dataset for recognising faces across pose and age. In Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China, 15–19 May 2018; pp. 67–74. [Google Scholar]
- Maze, B.; Adams, J.; Duncan, J.A.; Kalka, N.; Miller, T.; Otto, C.; Jain, A.K.; Niggel, W.T.; Anderson, J.; Cheney, J.; et al. Iarpa janus benchmark-c: Face dataset and protocol. In Proceedings of the 2018 International Conference on Biometrics (ICB), Gold Coast, QLD, Australia, 20–23 February 2018; pp. 158–165. [Google Scholar]
- Grother, P.; Ngan, M.; Hanaoka, K. Face Recognition Vendor Test (FVRT): Part 3, Demographic Effects; National Institute of Standards and Technology: Gaithersburg, MD, USA, 2019.
- Golzar, J.; Tajik, O.; Noor, S. Convenience Sampling. IJELS 2022, 1, 72–77. [Google Scholar] [CrossRef]
- Thomee, B.; Shamma, D.A.; Friedland, G.; Elizalde, B.; Ni, K.; Poland, D.; Borth, D.; Li, L.J. YFCC100M: The new data in multimedia research. Commun. ACM 2016, 59, 64–73. [Google Scholar] [CrossRef]
- Merler, M.; Ratha, N.; Feris, R.S.; Smith, J.R. Diversity in faces. arXiv 2019, arXiv:1901.10436. [Google Scholar]
- Birmajer, D.; Blount, B.; Boyd, S.; Einsohn, M.; Helmreich, J.; Kenyon, L.; Lee, S.; Taub, J. Introductory Statistics. 2021. Available online: https://touroscholar.touro.edu/oto/26/ (accessed on 5 April 2024).
- s-celeb-1m Challenge 3: Face Feature Test/Trillion Pairs. Originally. Available online: http://trillionpairs.deepglint.com/overview (accessed on 14 June 2020).
- Google. Freebase Data Dumps. Originally. Available online: https://developers.google.com/freebase/data (accessed on 24 May 2024).
- Face++ Research Toolkit. Available online: https://www.faceplusplus.com/ (accessed on 24 May 2024).
- Merriam-Webster. Nationality. Available online: https://www.merriam-webster.com/dictionary/nationality (accessed on 24 May 2024).
- Fu, S.; He, H.; Hou, Z.G. Learning race from face: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 2483–2509. [Google Scholar] [CrossRef] [PubMed]
- Microsoft Azure. Available online: https://www.azure.cn. (accessed on 24 May 2024).
- Zhou, E.; Cao, Z.; Yin, Q. Naive-deep face recognition: Touching the limit of LFW benchmark or not? arXiv 2015, arXiv:1501.04690. [Google Scholar]
- Baidu Cloud Vision Api. Available online: http://ai.baidu.com/ (accessed on 24 May 2024).
- Amazon’s Rekognition Tool. Available online: https://aws.amazon.com/rekognition/ (accessed on 24 May 2024).
- Wen, Y.; Zhang, K.; Li, Z.; Qiao, Y. A discriminative feature learning approach for deep face recognition. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 499–515. [Google Scholar]
- Liu, W.; Wen, Y.; Yu, Z.; Li, M.; Raj, B.; Song, L. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 212–220. [Google Scholar]
- Deng, J.; Guo, J.; Xue, N.; Zafeiriou, S. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 July 2019; pp. 4690–4699. [Google Scholar]
- Acien, A.; Morales, A.; Vera-Rodriguez, R.; Bartolome, I.; Fierrez, J. Measuring the gender and ethnicity bias in deep models for face recognition. In Proceedings of the Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications: 23rd Iberoamerican Congress, CIARP 2018, Madrid, Spain, 19–22 November 2018; Proceedings 23. Springer: Berlin/Heidelberg, Germany, 2019; pp. 584–593. [Google Scholar]
- Sarridis, I.; Koutlis, C.; Papadopoulos, S.; Diou, C. Towards Fair Face Verification: An In-depth Analysis of Demographic Biases. arXiv 2023, arXiv:2307.10011. [Google Scholar]
- Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems 27 (NIPS 2014), Montreal, QC, Canada, 8–13 December 2014. [Google Scholar]
- Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Machine Learning, PMLR, Lille, France, 7–9 July 2015; pp. 2256–2265. [Google Scholar]
- Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–24 June 2022; pp. 10684–10695. [Google Scholar]
- Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. [Google Scholar]
- Karras, T.; Laine, S.; Aila, T. A Style-Based Generator Architecture for Generative Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019. [Google Scholar]
- Jin, Y.; Zhang, J.; Li, M.; Tian, Y.; Zhu, H.; Fang, Z. Towards the automatic anime characters creation with generative adversarial networks. arXiv 2017, arXiv:1708.05509. [Google Scholar]
- Croitoru, F.A.; Hondru, V.; Ionescu, R.T.; Shah, M. Diffusion models in vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 10850–10869. [Google Scholar] [CrossRef]
- Peebles, W.; Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–6 October 2023; pp. 4195–4205. [Google Scholar]
- Radford, A.; Metz, L.; Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv 2015, arXiv:1511.06434. [Google Scholar]
- Jain, N.; Manikonda, L.; Hernandez, A.O.; Sengupta, S.; Kambhampati, S. Imagining an engineer: On GAN-based data augmentation perpetuating biases. arXiv 2018, arXiv:1811.03751. [Google Scholar]
- Ting, D.S.W.; Cheung, C.Y.L.; Lim, G.; Tan, G.S.W.; Quang, N.D.; Gan, A.; Hamzah, H.; Garcia-Franco, R.; San Yeo, I.Y.; Lee, S.Y.; et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 2017, 318, 2211–2223. [Google Scholar] [CrossRef] [PubMed]
- Silberzahn, R.; Uhlmann, E.L.; Martin, D.P.; Anselmi, P.; Aust, F.; Awtrey, E.; Bahník, Š.; Bai, F.; Bannard, C.; Bonnier, E.; et al. Many analysts, one data set: Making transparent how variations in analytic choices affect results. Adv. Methods Pract. Psychol. Sci. 2018, 1, 337–356. [Google Scholar] [CrossRef]
- Wheatley, G. Quick Draw; Mathematics Learning: Bethany Beach, DE, USA, 2007. [Google Scholar]
- Miyato, T.; Koyama, M. cGANs with projection discriminator. arXiv 2018, arXiv:1802.05637. [Google Scholar]
- Miyato, T.; Kataoka, T.; Koyama, M.; Yoshida, Y. Spectral normalization for generative adversarial networks. arXiv 2018, arXiv:1802.05957. [Google Scholar]
- Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; Courville, A.C. Improved training of wasserstein gans. In Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Dumoulin, V.; Shlens, J.; Kudlur, M. A learned representation for artistic style. arXiv 2016, arXiv:1610.07629. [Google Scholar]
- Shmelkin, R.; Wolf, L.; Friedlander, T. Generating master faces for dictionary attacks with a network-assisted latent space evolution. In Proceedings of the 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), Jodhpur, India, 15–18 December 2021; pp. 1–8. [Google Scholar]
- King, D.E. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 2009, 10, 1755–1758. [Google Scholar]
- Schroff, F.; Kalenichenko, D.; Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 815–823. [Google Scholar]
- Mandis, I.S. Reducing Racial and Gender Bias in Machine Learning and Natural Language Processing Tasks Using a GAN Approach. Int. J. High Sch. Res. 2021, 3, 17–24. [Google Scholar] [CrossRef]
- Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. [Google Scholar]
- He, L.; Wang, Z.; Li, Y.; Wang, S. Softmax dissection: Towards understanding intra-and inter-class objective for embedding learning. In Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA, 7–12 February 2020; Volume 34, pp. 10957–10964. [Google Scholar]
- Wang, H.; Wang, Y.; Zhou, Z.; Ji, X.; Gong, D.; Zhou, J.; Li, Z.; Liu, W. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 5265–5274. [Google Scholar]
- Ge, J.; Deng, W.; Wang, M.; Hu, J. FGAN: Fan-Shaped GAN for Racial Transformation. In Proceedings of the 2020 IEEE International Joint Conference on Biometrics (IJCB), Houston, TX, USA, 28 September–1 October 2020; pp. 1–7. [Google Scholar] [CrossRef]
- Choi, Y.; Choi, M.; Kim, M.; Ha, J.W.; Kim, S.; Choo, J. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 8789–8797. [Google Scholar]
- Kim, Y.H.; Nam, S.H.; Hong, S.B.; Park, K.R. GRA-GAN: Generative adversarial network for image style transfer of Gender, Race, and age. Expert Syst. Appl. 2022, 198, 116792. [Google Scholar] [CrossRef]
- Kim, Y.H.; Lee, M.B.; Nam, S.H.; Park, K.R. Enhancing the accuracies of age estimation with heterogeneous databases using modified CycleGAN. IEEE Access 2019, 7, 163461–163477. [Google Scholar] [CrossRef]
- Kim, Y.H.; Nam, S.H.; Park, K.R. Enhanced cycle generative adversarial network for generating face images of untrained races and ages for age estimation. IEEE Access 2020, 9, 6087–6112. [Google Scholar] [CrossRef]
- Jain, A.; Memon, N.; Togelius, J. Zero-shot racially balanced dataset generation using an existing biased stylegan2. In Proceedings of the 2023 IEEE International Joint Conference on Biometrics (IJCB), Ljubljana, Slovenia, 25–28 September 2023; pp. 1–18. [Google Scholar]
- Sumsion, A.; Torrie, S.; Sun, Z.; Lee, D.J. Overcoming deep learning subclass imbalances: Comparing the transfer of identity across a racial transformation. Electron. Imaging 2023, 35, 1–6. [Google Scholar] [CrossRef]
- Yang, C.; Lv, Z. Gender based face aging with cycle-consistent adversarial networks. Image Vis. Comput. 2020, 100, 103945. [Google Scholar] [CrossRef]
- Lui, N.; Chia, B.; Berrios, W.; Ross, C.; Kiela, D. Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision. In Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, 20–27 February 2024; Volume 38, pp. 14220–14228. [Google Scholar]
- Kang, D.; Dhar, D.; Chan, A. Incorporating side information by adaptive convolution. In Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Shi, Y.; Jain, A.K. Probabilistic face embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 6902–6911. [Google Scholar]
- Gong, S.; Liu, X.; Jain, A.K. Jointly de-biasing face recognition and demographic attribute estimation. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020; Proceedings, Part XXIX 16. Springer: Berlin/Heidelberg, Germany, 2020; pp. 330–347. [Google Scholar]
- Gong, S.; Liu, X.; Jain, A.K. Mitigating face recognition bias via group adaptive classifier. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual, 19–25 June 2021; pp. 3414–3424. [Google Scholar]
- Wang, M.; Zhang, Y.; Deng, W. Meta balanced network for fair face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 44, 8433–8448. [Google Scholar] [CrossRef] [PubMed]
- Gwilliam, M.; Hegde, S.; Tinubu, L.; Hanson, A. Rethinking common assumptions to mitigate racial bias in face recognition datasets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Virtual, 11–17 October 2021; pp. 4123–4132. [Google Scholar]
- Li, Y.; Sun, Y.; Cui, Z.; Shan, S.; Yang, J. Learning fair face representation with progressive cross transformer. arXiv 2021, arXiv:2108.04983. [Google Scholar]
- Huang, L.; Wang, M.; Liang, J.; Deng, W.; Shi, H.; Wen, D.; Zhang, Y.; Zhao, J. Gradient attention balance network: Mitigating face recognition racial bias via gradient attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 18–22 June 2023; pp. 38–47. [Google Scholar]
- Li, Z.; Liu, F.; Yang, W.; Peng, S.; Zhou, J. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Trans. Neural Netw. Learn. Syst. 2021, 33, 6999–7019. [Google Scholar] [CrossRef] [PubMed]
- Chan, E.R.; Lin, C.Z.; Chan, M.A.; Nagano, K.; Pan, B.; De Mello, S.; Gallo, O.; Guibas, L.J.; Tremblay, J.; Khamis, S.; et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–24 June 2022; pp. 16123–16133. [Google Scholar]
- Chen, S.; Wang, C.; Chen, Z.; Wu, Y.; Liu, S.; Chen, Z.; Li, J.; Kanda, N.; Yoshioka, T.; Xiao, X.; et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE J. Sel. Top. Signal Process. 2022, 16, 1505–1518. [Google Scholar] [CrossRef]
- Wang, M.; Deng, W. Deep face recognition: A survey. Neurocomputing 2021, 429, 215–244. [Google Scholar] [CrossRef]
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. Nature 2015, 518, 529–533. [Google Scholar] [CrossRef]
- Watkins, C.J.; Dayan, P. Q-learning. Mach. Learn. 1992, 8, 279–292. [Google Scholar] [CrossRef]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems 25 (NIPS 2012), Lake Tahoe, NV, USA, 3–6 December 2012. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- Hoffer, E.; Ailon, N. Deep metric learning using triplet network. In Proceedings of the Similarity-Based Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, 12–14 October 2015; Proceedings 3. Springer: Berlin/Heidelberg, Germany, 2015; pp. 84–92. [Google Scholar]
- Wang, M.; Deng, W. Adaptive Face Recognition Using Adversarial Information Network. IEEE Trans. Image Process. 2022, 31, 4909–4921. [Google Scholar] [CrossRef]
- Manesh, F.S.; Ghahramani, M.; Tan, Y.P. Facial part displacement effect on template-based gender and ethnicity classification. In Proceedings of the 2010 11th International Conference on Control Automation Robotics & Vision, Singapore, 7–10 December 2010; pp. 1644–1649. [Google Scholar]
- Toderici, G.; O’malley, S.M.; Passalis, G.; Theoharis, T.; Kakadiaris, I.A. Ethnicity-and gender-based subject retrieval using 3-D face-recognition techniques. Int. J. Comput. Vis. 2010, 89, 382–391. [Google Scholar] [CrossRef]
- Wang, W.; He, F.; Zhao, Q. Facial ethnicity classification with deep convolutional neural networks. In Proceedings of the Chinese Conference on Biometric Recognition, Chengdu, China, 14–16 October 2016; pp. 176–185. [Google Scholar]
- Krizhevsky, A.; Hinton, G. Convolutional deep belief networks on cifar-10. Unpubl. Manuscr. 2010, 40, 1–9. [Google Scholar]
- Ricanek, K.; Tesafaye, T. Morph: A longitudinal image database of normal adult age-progression. In Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR06), Southamtpon, UK, 10–12 April 2006; pp. 341–345. [Google Scholar]
- Gao, W.; Cao, B.; Shan, S.; Chen, X.; Zhou, D.; Zhang, X.; Zhao, D. The CAS-PEAL large-scale Chinese face database and baseline evaluations. IEEE Trans. Syst. Man Cybern.-Part A Syst. Hum. 2007, 38, 149–161. [Google Scholar]
- Phillips, P.J.; Moon, H.; Rizvi, S.A.; Rauss, P.J. The FERET evaluation methodology for face-recognition algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1090–1104. [Google Scholar] [CrossRef]
- Katti, H.; Arun, S. Can you tell where in India I am from? Comparing humans and computers on fine-grained race face classification. arXiv 2017, arXiv:1703.07595. [Google Scholar]
- Vo, T.; Nguyen, T.; Le, C. Race recognition using deep convolutional neural networks. Symmetry 2018, 10, 564. [Google Scholar] [CrossRef]
- Lyons, M.; Akamatsu, S.; Kamachi, M.; Gyoba, J. Coding facial expressions with gabor wavelets. In Proceedings of the Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, 14–16 April 1998; pp. 200–205. [Google Scholar]
- Szegedy, C.; Ioffe, S.; Vanhoucke, V.; Alemi, A.A. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017. [Google Scholar]
- Das, A.; Dantcheva, A.; Bremond, F. Mitigating bias in gender, age and ethnicity classification: A multi-task convolution neural network approach. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Wang, Y.; Feng, Y.; Liao, H.; Luo, J.; Xu, X. Do they all look the same? Deciphering chinese, japanese and koreans by fine-grained deep learning. In Proceedings of the 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), Miami, FL, USA, 10–12 April 2018; pp. 39–44. [Google Scholar]
- Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 7132–7141. [Google Scholar]
- Howard, A.; Sandler, M.; Chu, G.; Chen, L.C.; Chen, B.; Tan, M.; Wang, W.; Zhu, Y.; Pang, R.; Vasudevan, V.; et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 1314–1324. [Google Scholar]
- King, D.E. Max-Margin Object Detection. arXiv 2015, arXiv:1502.00046. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Liu, Z.; Luo, P.; Wang, X.; Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 13–16 December 2015; pp. 3730–3738. [Google Scholar]
- Vrandečić, D.; Krötzsch, M. Wikidata: A free collaborative knowledgebase. Commun. ACM 2014, 57, 78–85. [Google Scholar] [CrossRef]
- Rothe, R.; Timofte, R.; Van Gool, L. Deep expectation of real and apparent age from a single image without facial landmarks. Int. J. Comput. Vis. 2018, 126, 144–157. [Google Scholar] [CrossRef]
- Somandepalli, K. Prediction Race from Face for Movie Data; University of Southern California: Los Angeles, CA, USA, 2017. [Google Scholar]
- Ryu, H.J.; Adam, H.; Mitchell, M. Inclusivefacenet: Improving face attribute detection with race and gender diversity. arXiv 2017, arXiv:1712.00193. [Google Scholar]
- Pastaltzidis, I.; Dimitriou, N.; Quezada-Tavarez, K.; Aidinlis, S.; Marquenie, T.; Gurzawska, A.; Tzovaras, D. Data augmentation for fairness-aware machine learning: Preventing algorithmic bias in law enforcement systems. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, 21–24 June 2022; pp. 2302–2314. [Google Scholar]
- Cheng, M.; Cai, K.; Li, M. RWF-2000: An open large scale video database for violence detection. In Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 10–15 January 2021; pp. 4183–4190. [Google Scholar]
- Conti, J.R.; Noiry, N.; Clemencon, S.; Despiegel, V.; Gentric, S. Mitigating gender bias in face recognition using the von mises-fisher mixture model. In Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA, 17–23 July 2022; pp. 4344–4369. [Google Scholar]
- Gwyn, T.; Roy, K. Examining Gender Bias of Convolutional Neural Networks via Facial Recognition. Future Internet 2022, 14, 375. [Google Scholar] [CrossRef]
- Tian, F.; Liu, W.; Zhao, S.; Liu, J. Face Recognition Fairness Assessment based on Data Augmentation: An Empirical Study. In Proceedings of the 2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C), Guangzhou, China, 5–9 December 2022; pp. 315–318. [Google Scholar]
- Avrahami, O.; Fried, O.; Lischinski, D. Blended latent diffusion. ACM Trans. Graph. (TOG) 2023, 42, 1–11. [Google Scholar] [CrossRef]
- Palmer, M.A.; Brewer, N.; Horry, R. Understanding gender bias in face recognition: Effects of divided attention at encoding. Acta Psychol. 2013, 142, 362–369. [Google Scholar] [CrossRef] [PubMed]
- Ng, C.B.; Tay, Y.H.; Goi, B.M. A review of facial gender recognition. Pattern Anal. Appl. 2015, 18, 739–755. [Google Scholar] [CrossRef]
- Terhörst, P.; Kolf, J.N.; Huber, M.; Kirchbuchner, F.; Damer, N.; Moreno, A.M.; Fierrez, J.; Kuijper, A. A comprehensive study on face recognition biases beyond demographics. IEEE Trans. Technol. Soc. 2021, 3, 16–30. [Google Scholar] [CrossRef]
Dataset | Year | Number of Images (In Thousands) | Number of Identities (In Thousands) | Average Images Per Identity | Caucasian | African | Indian | Asian | Racially Balanced/Unbalanced |
---|---|---|---|---|---|---|---|---|---|
LFW [28] | 2008 | 13 | 5.7 | 2 | 77.6% | 12.9% | ** Combined In African | 9.4% | Unbalanced |
BioSecure [34] | 2009 | 2.7 | 0.667 | 4 | 86.1% | 5.2% | ** Combined In African | 8.8% | Unbalanced |
PubFig [35] | 2011 | 58 | 0.2 | 294 | 85.0% | 12.0% | ** Combined In African | 3.0% | Unbalanced |
YouTube Faces [36] | 2011 | 621 | 1.6 | 390 | 77.2% | 11.7% | ** Combined In African | 10.9% | Unbalanced |
CASIA [37] | 2014 | 500 | 10.5 | 48 | 82.0% | 12.9% | ** Combined In African | 5.2% | Unbalanced |
CelebA [38] | 2015 | 203 | 10.2 | 20 | 75.4% | 14.6% | ** Combined In African | 9.9% | Unbalanced |
VGGFace [39] | 2015 | 2600 | 2.6 | 1000 | 82.3% | 12.7% | ** Combined In African | 5.0% | Unbalanced |
MegaFace [40] | 2016 | 4700 | 660 | 7 | 70.3% | 10.9% | ** Combined In African | 18.7% | Unbalanced |
MSCeleb1M [41] | 2016 | 10,000 | 100 | 100 | 71.6% | 16% | ** Combined In African | 12.2% | Unbalanced |
UTKFace [42] | 2017 | 24 | - | - | 46.2% | 37.8% | ** Combined In African | 16.0% | Unbalanced |
IJB-C [44] | 2018 | 21 | 3.5 | 6 | 70.5% | 17.8% | ** Combined In African | 11.6% | Unbalanced |
VGGFace2 [43] | 2018 | 3300 | 9 | 370 | 76.1% | 16.8% | ** Combined In African | 7% | Unbalanced |
DemogPairs [14] | 2019 | 10.8 | 0.6 | 18 | 33.3% | 33.3% | ** Combined In African | 33.3% | Balanced |
FRVT2018 [45] | 2019 | 2700 | 1200 | 2 | 64.9% | 27.3% | ** Combined In African | 1.6% | Unbalanced |
RFW [15] | 2019 | 40 | 12 | 3.3 | 25% | 25% | 25% | 25% | Balanced |
BUPT-Balancedface [10] | 2020 | 1300 | 28 | 46.4 | 25% | 25% | 25% | 25% | Balanced |
BUPT-Globalface [10] | 2020 | 2000 | 38 | 52.6 | 38% | 13% | 18% | 31% | Unbalanced |
CeFA [16] | 2020 | 23.5 | 1.6 | 14.7 | N/A | 33.3% | 33.3% | 33.3% | Balanced |
DiveFace [11] | 2020 | 125 | 24 | 5.2 | 33.3% | 33.3% | ** Combined In African | 33.3% | Balanced |
FaceARG [12] | 2021 | 175 | - | - | 24.42% | 24.02% | 25.94% | 25.60% | Balanced |
FairFace [13] | 2021 | 108 | - | - | * 29.7% | * 29.5% | * 26.6% | * 14.2% | * Unbalanced |
BFW [17] | 2023 | 20 | 0.8 | 25 | 25% | 25% | 25% | 25% | Balanced |
Dataset | Overlaps with Other Datasets | Percentage Overlap |
---|---|---|
DemogPairs [14] | VGGFace2, VGGFace, and CWF | 0.13%, 0.02% and 1.27% |
RFW [15] | MS-Celeb-1M | 0.4% |
BUPT-Balancedface [10] | MS-Celeb-1M | Not Available |
BUPT-Globalface [10] | MS-Celeb-1M | Not Available |
DiveFace [11] | Megaface dataset MF2 | 2.7% |
Model | LFW Verification | RFW | |||||
---|---|---|---|---|---|---|---|
Accuracy | Caucasian | Indian | Asian | African | SER (↓) | ||
Commercial API | Microsoft [55] | - | 87.60% | 82.83% | 79.67% | 75.83% | 1.95 |
Face++ [52,56] | 99.5% | 93.90% | 88.55% | 92.47% | 87.50% | 2.05 | |
Baidu [57] | - | 89.13% | 86.53% | 90.27% | 77.97% | 2.26 | |
Amazon [58] | - | 90.45% | 87.20% | 84.87% | 86.27% | 1.58 | |
mean | - | 90.27% | 86.28% | 86.82% | 81.89% | 1.96 | |
Algorithms trained on racially unbalanced datasets | Center-loss [59] | 99.0% | 87.18% | 81.92% | 79.32% | 78.00% | 1.72 |
Sphereface [60] | 99.42% | 90.80 | 87.02 | 82.95 | 82.28 | 1.93 | |
Arcface [61] | 99.40% | 92.15% | 88.00% | 83.98% | 84.93 | 2.04 | |
VGGface2 [43] | 99.0% | 89.90% | 86.13% | 84.93% | 83.38 | 1.64 | |
mean | 99.21% | 90.01% | 85.77% | 82.80% | 82.15% | 1.83 |
Dataset | Racially | Caucasian | African | Indian | Asian | Overall | Δ | |||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Balanced | Female | Male | Female | Male | Female | Male | Female | Male | Female | Male | ||
LFW [28] | ✗ | 18.7% | 58.9% | 3.3% | 9.6% | – | – | 2.2% | 7.2% | 24.2% | 75.7% | 51.5% |
RFW [15] | ✓ | 7.9% | 17.2% | 1.3% | 24.4% | 9.1% | 14.7% | 7.7% | 17.6% | 26.1% | 73.9% | 47.8% |
YouTube Faces [36] | ✗ | 20.3% | 56.9% | 4.0% | 7.7% | – | – | 3.0% | 7.9% | 27.3% | 72.5% | 45.2% |
FRVT2018 [45] | ✗ | 16.5% | 48.4% | 7.4% | 19.9% | – | – | 0.4% | 1.2% | 24.3% | 69.5% | 45.2% |
MSCeleb1M [41] | ✗ | 19.2% | 52.4% | 3.9% | 12.1% | – | – | 4.5% | 7.7% | 27.6% | 72.2% | 44.6% |
VGGFace2 [43] | ✗ | 30.2% | 45.9% | 6.3% | 10.5% | – | – | 3.6% | 3.4% | 40.1% | 59.8% | 19.7% |
CASIA [37] | ✗ | 33.2% | 48.8% | 5.7% | 7.2% | – | – | 2.6% | 2.6% | 41.5% | 58.6% | 17.1% |
PubFig [35] | ✗ | 35.5% | 49.5% | 5.5% | 6.5% | – | – | 1.0% | 2.0% | 42.0% | 58.0% | 16.0% |
IJB-C [44] | ✗ | 30.2% | 40.3% | 6.0% | 11.8% | – | – | 6.2% | 5.4% | 42.4% | 57.5% | 15.1% |
BioSecure [34] | ✗ | 36.0% | 50.1% | 2.1% | 3.1% | – | – | 4.5% | 4.3% | 42.6% | 57.5% | 14.9% |
MegaFace [40] | ✗ | 30.3% | 40.0% | 4.7% | 6.2% | – | – | 8.1% | 10.6% | 43.1% | 56.8% | 13.7% |
FaceARG [12] | ✓ | 14.5% | 9.8% | 10.3% | 13.8% | 15.7% | 10.3% | 16.2% | 9.5% | 56.7% | 43.3% | 13.4% |
CeFA [16] | ✓ | – | – | – | – | – | – | – | – | 43.9% | 56.1% | 12.2% |
CelebA [38] | ✗ | 41.5% | 33.9% | 8.2% | 6.4% | – | – | 5.5% | 4.4% | 55.2% | 44.7% | 10.5% |
UTKFace [42] | ✗ | 20.0% | 26.2% | 16.3% | 21.5% | – | – | 8.9% | 7.1% | 45.2% | 54.8% | 9.6% |
FairFace [13] | ✗ | 12.3% | 17.4% | 14.8% | 14.7% | 6.8% | 7.3% | 13.1% | 13.6% | 47.0% | 53.0% | 6% |
VGGFace [39] | ✗ | 38.6% | 43.7% | 6.9% | 5.8% | – | – | 2.9% | 2.1% | 48.4% | 51.6% | 3.2% |
DiveFace [11] | ✓ | 19.7% | 20.2% | 14.0% | 15.0% | – | – | 16.3% | 14.8% | 50.5% | 49.5% | 1.0% |
DemogPairs [14] | ✓ | 16.7% | 16.7% | 16.7% | 16.7%% | – | – | 16.7% | 16.7% | 50% | 50% | 0% |
BFW [17] | ✓ | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 12.5% | 50.0% | 50.0% | 0% |
Method | Loss- ArcFace | White | Black | East Asian | South Asian | Average (↑) | STD (↓) | SER (↓) |
---|---|---|---|---|---|---|---|---|
IMAN [15] | ✓ | 93.92 | 92.98 | 90.60 | 90.98 | 92.12 | 1.38 | 1.55 |
CosFace [87] | X | 95.12 | 93.93 | 92.98 | 92.93 | 93.74 | 0.89 | 1.45 |
ArcFace [61] | ✓ | 96.18 | 94.67 | 93.72 | 93.98 | 94.67 | 0.96 | 1.64 |
ACNN [97] | ✓ | 96.12 | 94.00 | 93.67 | 94.55 | 94.59 | 0.94 | 1.63 |
PFE [98] | ✓ | 96.38 | 95.17 | 94.27 | 94.60 | 95.10 | 0.80 | 1.58 |
DebFace [99] | ✓ | 95.95 | 93.67 | 94.33 | 94.78 | 94.68 | 0.83 | 1.56 |
GAC [100] | ✓ | 96.20 | 94.77 | 94.87 | 94.98 | 95.21 | 0.58 | 1.37 |
RL-RBN [10] | ✓ | 96.27 | 94.68 | 94.82 | 95.00 | 95.19 | 0.63 | 1.43 |
MBN [101] | ✓ | 96.25 | 95.38 | 95.32 | 94.85 | 95.45 | 0.51 | 1.37 |
Rethinking [102] | ✓ | 89.1 | 85.5 | 71.8 | 75.8 | 80.55 | 7.01 | 2.59 |
PCT [103] | ✓ | 97.00 | [96.22] | 95.73 | 96.38 | 96.33 | 0.52 | 1.42 |
Sensitive Loss [20] | X | [97.23] | 95.82 | [96.50] | [96.95] | [96.63] | 0.53 | 1.51 |
GABN [104] | ✓ | 95.78 | 94.71 | 94.51 | 95.21 | 95.05 | [0.49] | [1.30] |
Race | Metric | VGG-19 | Inception ResNet-v2 | SeNet | MobileNetV3 |
---|---|---|---|---|---|
Afro-American | Pr | 96.68 | 96.88 | 95.27 | [96.97] |
Afro-American | Re | 97.96 | 98.20 | 98.32 | [98.44] |
Afro-American | F1 | 97.32 | 97.54 | 96.77 | [97.70] |
Asian | Pr | 98.27 | 98.43 | 97.94 | [98.52] |
Asian | Re | 97.76 | 97.92 | 96.96 | [98.20] |
Asian | F1 | 97.32 | 98.18 | 97.45 | [98.36] |
Caucasian | Pr | 94.91 | 95.61 | 95.43 | [96.20] |
Caucasian | Re | 94.00 | [95.00] | 94.28 | 94.12 |
Caucasian | F1 | 94.45 | [95.30] | 94.85 | 95.15 |
Indian | Pr | 94.37 | 94.51 | [94.95] | 94.89 |
Indian | Re | 94.52 | 94.32 | 94.00 | [95.80] |
Indian | F1 | 94.44 | 94.41 | 94.47 | [95.34] |
Overall | Pr | 94.06 | 96.36 | 95.90 | [96.64] |
Overall | Re | 96.06 | 96.36 | 95.89 | [96.64] |
Overall | F1 | 96.06 | 96.36 | 95.89 | [96.64] |
(a) | |||||||||||
Tested on (White Accuracy—Not White Accuracy): | |||||||||||
FairFace (Balanced) | UTKFace (Unbalanced) | LFWA+ (Unbalanced) | |||||||||
Trained | FairFace (Balanced) | [93.7%]–[75.4%] | 93.6–80.1% | [97.0%]–[96.0%] | |||||||
on: | UTKFace (Unbalanced) | 80.0–69.3% | 91.8%–[83.9%] | 92.5–88.7% | |||||||
LFWA+ (Unbalanced) | 87.9–54.1% | [94.7%]–38.0% | 96.1–86.6% | ||||||||
(b) | |||||||||||
Tested on (White Accuracy—Not White Accuracy): | |||||||||||
FairFace (Balanced) | UTKFace (Unbalanced) | LFWA+ (Unbalanced) | CelebA (Unbalanced) | ||||||||
Trained | FairFace (Balanced) | [94.2%]–[94.4%] | [94.0%]–[93.9%] | 92.0%–[93.0%] | [98.1%]–[98.1%] | ||||||
on: | UTKFace (Unbalanced) | 86.0–82.3% | 93.5–92.5% | 91.6–90.8% | 96.2–96.2% | ||||||
LFWA+ (Unbalanced) | 76.1–73.8% | 84.2–83.3% | [93.0%]–89.4% | 94.0%–94.0% | |||||||
CelebA (Unbalanced) | 81.2–78.1% | 88.0–88.6% | 90.5–90.1% | 97.1–92.1% | |||||||
(c) | |||||||||||
Tested on (White Accuracy—Not White Accuracy): | |||||||||||
FairFace (Balanced) | UTKFace (Unbalanced) | ||||||||||
Trained | FairFace (Balanced) | [59.7%]–[60.7%] | 56.5–61.6% | ||||||||
on: | UTKFace (Unbalanced) | 41.3–41.8% | [57.6%]–[61.7%] |
Method | Year | Loss/Training | Architecture | Dataset Modification | Additional Data | Benefits from Race Classification |
---|---|---|---|---|---|---|
ACNN [97] | 2017 | ✓ | ||||
InclusiveFaceNet [137] | 2017 | ✓ | ✓ | ✓ | ||
CosFace [87] | 2018 | ✓ | ||||
ArcFace [61] | 2019 | ✓ | ||||
IMAN [15] | 2019 | ✓ | ✓ | ✓ | ||
PFE [98] | 2019 | ✓ | ||||
DebFace [99] | 2020 | ✓ | ✓ | ✓ | ||
RL-RBN [10] | 2020 | ✓ | ✓ | |||
GAC [100] | 2021 | ✓ | ✓ | |||
MBN [101] | 2021 | ✓ | ✓ | ✓ | ||
PCT [103] | 2021 | ✓ | ✓ | |||
Rethinking [102] | 2021 | ✓ | ✓ | ✓ | ||
Using VGG16 [1] | 2021 | ✓ | ||||
AIN [115] | 2022 | ✓ | ✓ | |||
Fairness aware augmentation [138] | 2022 | ✓ | ✓ | |||
Sensitive Loss [20] | 2022 | ✓ | ✓ | |||
GABN [104] | 2023 | ✓ | ✓ | ✓ |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Sumsion, A.; Torrie, S.; Lee, D.-J.; Sun, Z. Surveying Racial Bias in Facial Recognition: Balancing Datasets and Algorithmic Enhancements. Electronics 2024, 13, 2317. https://doi.org/10.3390/electronics13122317
Sumsion A, Torrie S, Lee D-J, Sun Z. Surveying Racial Bias in Facial Recognition: Balancing Datasets and Algorithmic Enhancements. Electronics. 2024; 13(12):2317. https://doi.org/10.3390/electronics13122317
Chicago/Turabian StyleSumsion, Andrew, Shad Torrie, Dah-Jye Lee, and Zheng Sun. 2024. "Surveying Racial Bias in Facial Recognition: Balancing Datasets and Algorithmic Enhancements" Electronics 13, no. 12: 2317. https://doi.org/10.3390/electronics13122317