Feature Interaction-Based Face De-Morphing Factor Prediction for Restoring Accomplice’s Facial Image
Abstract
:1. Introduction
- A novel face de-morphing method is proposed, which significantly diverges from direct identity disentanglement approaches. This method leverages a pre-trained StyleGAN inversion model to embed facial images into a latent space and employs the predicted de-morphing factors to perform inverse linear interpolation within the latent space, thereby obtaining the identity features of accomplices to generate high-quality facial images.
- A feature interaction-based de-morphing factor prediction network is proposed, employing channel-wise attention mechanisms to effectively integrate features from the morphed face and the real-time captured face. This approach enables the prediction of the de-morphing factor by exploring feature correlations.
- The experimental results demonstrate that the proposed method can effectively restore the accomplice’s face images. Its image quality and restoration accuracy outperform those of existing face de-morphing methods.
2. Related Work
2.1. Generation of Morphed Face Images
2.1.1. Landmark-Based Generation
2.1.2. Deep Learning-Based Generation
2.2. Face Morphing Attack Detection
2.2.1. Single Image-Based Morphing Attack Detection
2.2.2. Differential Image-Based Morphing Attack Detection
3. Proposed Method
3.1. Motivation
3.2. Overall Framework
3.3. Feature Interaction-Based De-Morphing Factor Prediction Network
4. Experiments
4.1. Datasets
4.2. Implementation Details and Evaluation Metrics
4.3. Ablation Experiments
4.3.1. Effectiveness of the Feedforward Network
4.3.2. The Universality of the Feature Interaction-Based De-Morphing Factor Prediction Network
4.4. Performance Comparison
4.4.1. Performance Comparison on HNU-FM Dataset
4.4.2. Performance Comparison on MIPGAN Dataset
4.4.3. Analysis of Restoration Effects
4.4.4. Evaluation of Model Performance
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Abbreviations
FRSs | Face recognition systems |
MAD | Morphing attack detection |
S-MAD | Single image-based morphing attack detection |
D-MAD | Differential image-based morphing attack detection |
BSIF | Binarized statistical image feature |
SPN | Sensor pattern noise |
GAN | Generative adversarial network |
References
- Ferrara, M.; Franco, A.; Maltoni, D. The magic passport. In Proceedings of the IEEE International Joint Conference on Biometrics, Clearwater, FL, USA, 29 September–2 October 2014; pp. 1–7. [Google Scholar]
- Makrushin, A.; Neubert, T.; Dittmann, J. Automatic generation and detection of visually faultless facial morphs. In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Porto, Portugal, 27 February–1 March 2017; pp. 39–50. [Google Scholar]
- Qin, L.; Peng, F.; Venkatesh, S.; Ramachandra, R.; Long, M.; Busch, C. Low visual distortion and robust morphing attacks based on partial face image manipulation. IEEE Trans. Biom. Behav. Identity Sci. 2021, 3, 72–88. [Google Scholar] [CrossRef]
- Damer, N.; Saladié, A.M.; Braun, A.; Kuijper, A. MorGAN: Recognition vulnerability and attack detectability of face morphing attacks created by generative adversarial network. In Proceedings of the 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), Redondo Beach, CA, USA, 22–25 October 2018; pp. 1–10. [Google Scholar]
- Venkatesh, S.; Zhang, H.; Ramachandra, R.; Raja, K.; Damer, N.; Busch, C. Can GAN generated morphs threaten face recognition systems equally as landmark based morphs?—Vulnerability and detection. In Proceedings of the International Workshop on Biometrics and Forensics (IWBF 2020), Porto, Portugal, 29–30 April 2020; pp. 1–6. [Google Scholar]
- Zhang, H.; Venkatesh, S.; Ramachandra, R.; Raja, K.; Damer, N.; Busch, C. MIPGAN—Generating strong and high quality morphing attacks using identity prior driven GAN. IEEE Trans. Biom. Behav. Identity Sci. 2021, 3, 365–383. [Google Scholar] [CrossRef]
- Damer, N.; Fang, M.; Siebke, P.; Kolf, J.N.; Huber, M.; Boutros, F. MorDIFF: Recognition vulnerability and attack detectability of face morphing attacks created by diffusion autoencoders. In Proceedings of the IWBF 2023: 11th International Workshop on Biometrics and Forensics, Barcelona, Spain, 19–20 April 2023; pp. 1–6. [Google Scholar]
- Blasingame, Z.W.; Liu, C. Leveraging Diffusion for Strong and High Quality Face Morphing Attacks. IEEE Trans. Biom. Behav., Identity Sci. 2024, 6, 118–131. [Google Scholar] [CrossRef]
- Kramer, R.S.S.; Mireku, M.O.; Flack, T.R.; Ritchie, K.L. Face morphing attacks: Investigating detection with humans and computers. Cogn. Res. Princ. Implic. 2019, 4, 28. [Google Scholar] [CrossRef] [PubMed]
- Scherhag, U.; Rathgeb, C.; Merkle, J.; Breithaupt, R.; Busch, C. Face recognition systems under morphing attacks: A survey. IEEE Access 2019, 7, 23012–23026. [Google Scholar] [CrossRef]
- Venkatesh, S.; Ramachandra, R.; Raja, K.; Busch, C. Face morphing attack generation and detection: A comprehensive survey. IEEE Trans. Technol. Soc. 2021, 2, 128–145. [Google Scholar] [CrossRef]
- Raghavendra, R.; Raja, K.B.; Busch, C. Detecting morphed face images. In Proceedings of the 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS), Niagara Falls, NY, USA, 6–9 September 2016; pp. 1–7. [Google Scholar]
- Zhang, L.-B.; Peng, F.; Long, M. Face morphing detection using Fourier spectrum of sensor pattern noise. In Proceedings of the 2018 IEEE International Conference on Multimedia and Expo (ICME), San Diego, CA, USA, 23–27 July 2018; pp. 1–6. [Google Scholar]
- Raghavendra, R.; Raja, K.B.; Venkatesh, S.; Busch, C. Transferable deep-CNN features for detecting digital and print-scanned morphed face images. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017; pp. 1822–1830. [Google Scholar]
- Zhang, L.-B.; Cai, J.; Peng, F.; Long, M. MSA-CNN: Face morphing detection via a multiple scales attention convolutional neural network. In Proceedings of the 20th International Workshop, IWDW 2021, Beijing, China, 20–22 November 2021; pp. 17–31. [Google Scholar]
- Soleymani, S.; Dabouei, A.; Taherkhani, F.; Dawson, J.; Nasrabadi, N.M. Mutual information maximization on disentangled representations for differential morph detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Virtual Conference, 5–9 January 2021; pp. 1731–1741. [Google Scholar]
- Damer, N.; Spiller, N.; Fang, M.; Boutros, F.; Kirchbuchner, F.; Kuijper, A. PW-MAD: Pixel-wise supervision for generalized face morphing attack detection. In Proceedings of the 16th International Symposium, ISVC 2021, Virtual Event, 4–6 October 2021; pp. 291–304. [Google Scholar]
- Qin, L.; Peng, F.; Long, M. Face morphing attack detection and localization based on feature-wise supervision. IEEE Trans. Inf. Forensics Secur. 2022, 17, 3649–3662. [Google Scholar] [CrossRef]
- Kashiani, H.; Talemi, N.A.; Saadabadi, M.S.E.; Nasrabadi, N.M. Towards Generalizable Morph Attack Detection with Consistency Regularization. In Proceedings of the IEEE International Joint Conference on Biometrics, IJCB 2023, Ljubljana, Slovenia, 25–28 September 2023; pp. 1–10. [Google Scholar]
- Damer, N.; Boller, V.; Wainakh, Y.; Boutros, F.; Terhörst, P.; Braun, A.; Kuijper, A. Detecting face morphing attacks by analyzing the directed distances of facial landmarks shifts. In Pattern Recognition: Proceedings of the 40th German Conference, GCPR 2018, Stuttgart, Germany, 9–12 October 2018; Springer: Cham, Switzerland, 2018; pp. 518–534. [Google Scholar]
- Scherhag, U.; Rathgeb, C.; Busch, C. Towards detection of morphedface images in electronic travel documents. In Proceedings of the 2018 13th IAPR International Workshop on Document Analysis Systems (DAS), Vienna, Austria, 24–27 April 2018; pp. 187–192. [Google Scholar]
- Scherhag, U.; Rathgeb, C.; Merkle, J.; Busch, C. Deep face representations for differential morphing attack detection. IEEE Trans. Inf. Forensics Secur. 2020, 15, 3625–3639. [Google Scholar] [CrossRef]
- Chaudhary, B.; Aghdaie, P.; Soleymani, S.; Dawson, J.; Nasrabadi, N.M. Differential morph face detection using discriminative wavelet subbands. In Proceedings of the IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 1425–1434. [Google Scholar]
- Ortega-Delcampo, D.; Conde, C.; Palacios-Alonso, D.; Cabello, E. Border control morphing attack detection with a convolutional neural network de-morphing approach. IEEE Access 2020, 8, 92301–92313. [Google Scholar] [CrossRef]
- Ferrara, M.; Franco, A.; Maltoni, D. Face demorphing. IEEE Trans. Inf. Forensics Secur. 2018, 13, 1008–1017. [Google Scholar] [CrossRef]
- Peng, F.; Zhang, L.-B.; Long, M. FD-GAN: Face de-morphing generative adversarial network for restoring accomplice’s facial image. IEEE Access 2019, 7, 75122–75131. [Google Scholar] [CrossRef]
- Banerjee, S.; Jaiswal, P.; Ross, A. Facial de-morphing: Extracting component faces from a single morph. In Proceedings of the 2022 IEEE International Joint Conference on Biometrics (IJCB), Abu Dhabi, United Arab Emirates, 10–13 October 2022; pp. 1–10. [Google Scholar]
- Long, M.; Yao, Q.; Zhang, L.-B.; Peng, F. Face De-Morphing Based on Diffusion Autoencoders. IEEE Trans. Inf. Forensics Secur. 2024, 19, 3051–3063. [Google Scholar] [CrossRef]
- OpenCV. Available online: https://learnopencv.com/face-morph-using-opencv-cpp-python/ (accessed on 12 March 2023).
- Facemorpher. Available online: https://github.com/yaopang/FaceMorpher (accessed on 15 April 2024).
- Dlib C++ Library. Available online: http://dlib.net/ (accessed on 12 March 2023).
- Milborrow, S.; Nicolls, F. Active shape models with SIFT descriptors and Mars. In Proceedings of the 2014 International Conference on Computer Vision Theory and Applications (VISAPP), Lisbon, Portugal, 5–8 January 2014; Volume 2, pp. 380–387. [Google Scholar]
- Abrosoft FantaMorph. Available online: http://www.fantamorph.com/ (accessed on 15 April 2024).
- Karras, T.; Laine, S.; Aila, T. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 4396–4405. [Google Scholar]
- Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; Aila, T. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 14–19 June 2020; pp. 8107–8116. [Google Scholar]
- Banerjee, S.; Ross, A. Conditional identity disentanglement for differential face morph detection. In Proceedings of the IEEE International Joint Conference on Biometrics (IJCB), Shenzhen, China, 4–7 August 2021; pp. 1–8. [Google Scholar]
- Long, M.; Zhou, J.; Zhang, L.-B.; Peng, F.; Zhang, D. ADFF: Adaptive de-morphing factor framework for restoring accomplice’s facial image. IET Image Process. 2024, 18, 470–480. [Google Scholar] [CrossRef]
- Richardson, E.; Alaluf, Y.; Patashnik, O.; Nitzan, Y.; Azar, Y.; Shapiro, S.; Cohen-Or, D. Encoding in style: A StyleGAN encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20–25 June 2021; pp. 2287–2296. [Google Scholar]
- Wei, T.; Chen, D.; Zhou, W.; Liao, J.; Zhang, W.; Yuan, L.; Hua, G.; Yu, N. E2Style: Improve the efficiency and effectiveness of StyleGAN inversion. IEEE Trans. Image Process. 2022, 31, 3267–3280. [Google Scholar] [CrossRef] [PubMed]
- Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.-H. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR), New Orleans, LA, USA, 18–24 June 2022; pp. 5718–5729. [Google Scholar]
- Zhang, L.-B.; Cai, J.; Peng, F.; Long, M. A benchmark database for the comparison of face morphing detection methods. In Proceedings of the International Conference on Electronic Information Technology and Smart Agriculture (ICEITSA), Huaihua, China, 10–12 December 2021; pp. 393–401. [Google Scholar]
- Huber, M.; Boutros, F.; Luu, A.T.; Raja, K.; Ramachandra, R.; Damer, N.; Neto, P.C.; Gonçalves, T.; Sequeira, A.F.; Cardoso, J.S.; et al. SYN-MAD 2022: Competition on face morphing attack detection based on privacy-aware synthetic training data. In Proceedings of the IEEE International Joint Conference on Biometrics (IJCB), Abu Dhabi, United Arab Emirates, 10–13 October 2022; pp. 1–10. [Google Scholar]
- Face++ Compare API. Available online: https://www.faceplusplus.com/face-comparing/ (accessed on 6 March 2024).
- DeBruine, L.; Jones, B. Face Research Lab London Set. 2017. Available online: https://figshare.com/articles/dataset/Face_Research_Lab_London_Set/5047666/5 (accessed on 11 June 2023).
Method | Protocol I | Protocal III | ||
---|---|---|---|---|
Scenario 1 | Scenario 2 | Scenario 1 | Scenario 2 | |
96.12% | 90.03% | 78.42% | 69.48% | |
99.16% | 98.86% | 77.55% | 74.39% | |
98.31% | 92.23% | 80.71% | 79.48% |
Method | Face Demorphing [25] | FD-GAN [26] * | DAD [27] | CNN [24] | cGAN [36] | |
---|---|---|---|---|---|---|
Scenario 1 | 49.82% | 66.49% | 44.21% | 65.79% | 64.91% | 80.71% |
Scenario 2 | 49.12% | 53.73% | 44.21 | 54.39% | 53.33% | 79.84% |
Method | Face Demorphing [25] | FD-GAN [26] * | DAD [27] | CNN [24] | cGAN [36] | |
---|---|---|---|---|---|---|
Scenario 1 | 56.08% | 77.03% | 36.49% | 77.03% | 75.34% | 98.31% |
Scenario 2 | 52.70% | 70.61% | 36.49 | 67.91% | 63.51% | 92.23% |
Method | Face Demorphing [25] | FD-GAN [26] * | DAD [27] | CNN [24] | cGAN [36] | |
---|---|---|---|---|---|---|
MIPGAN-I | 56.21% | 75.74% | 24.85% | 69.82% | 50.89% | 85.80% |
MIPGAN-II | 66.01% | 71.43% | 25.12% | 60.01% | 32.02% | 91.38% |
Method | Model Capacity | Inference Time |
---|---|---|
(Million) | (img/sec NVIDIA 3090 24 GB) | |
Face Demorphing [25] | / | 0.55 |
FD-GAN [26] * | 44.00 | 10.50 |
DNN [27] | 43.80 | 6.50 |
CNN [24] | 4.16 | 5.21 |
cGAN [36] | 54.53 | 13.04 |
DMFP | 9.12 | / |
252.53 | 15.58 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Cai, J.; Duan, Q.; Long, M.; Zhang, L.-B.; Ding, X. Feature Interaction-Based Face De-Morphing Factor Prediction for Restoring Accomplice’s Facial Image. Sensors 2024, 24, 5504. https://doi.org/10.3390/s24175504
Cai J, Duan Q, Long M, Zhang L-B, Ding X. Feature Interaction-Based Face De-Morphing Factor Prediction for Restoring Accomplice’s Facial Image. Sensors. 2024; 24(17):5504. https://doi.org/10.3390/s24175504
Chicago/Turabian StyleCai, Juan, Qiangqiang Duan, Min Long, Le-Bing Zhang, and Xiangling Ding. 2024. "Feature Interaction-Based Face De-Morphing Factor Prediction for Restoring Accomplice’s Facial Image" Sensors 24, no. 17: 5504. https://doi.org/10.3390/s24175504