HoloForkNet: Digital Hologram Reconstruction via Multibranch Neural Network
Abstract
:1. Introduction
2. The Proposed Method
3. Numerical Experiments
3.1. Numerical Experiment Conditions
3.2. Assessing Image Quality
3.3. Reconstructing Images of Objects 1
3.4. Reconstructing Object Images 2
4. Optical Experiments
4.1. Conditions for Optical Experiments
4.2. Results of Optical Experiments
5. Analysis of the Obtained Results
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Schnars, U.; Falldorf, C.; Watson, J.; Jüptner, W. Digital Holography and Wavefront Sensing; Springer: Berlin/Heidelberg, Germany, 2015; ISBN 9783662446928. [Google Scholar]
- Javidi, B.; Carnicer, A.; Anand, A.; Barbastathis, G.; Chen, W.; Ferraro, P.; Goodman, J.; Horisaki, R.; Khare, K.; Kujawinska, M.; et al. Roadmap on digital holography [Invited]. Opt. Express 2021, 29, 35078. [Google Scholar] [CrossRef]
- Lam, V.K.; Nguyen, T.C.; Chung, B.M.; Nehmetallah, G.; Raub, C.B. Quantitative assessment of cancer cell morphology and motility using telecentric digital holographic microscopy and machine learning. Cytom. Part A 2017, 93, 334–345. [Google Scholar] [CrossRef]
- Wu, Y.; Rivenson, Y.; Zhang, Y.; Wei, Z.; Günaydin, H.; Lin, X.; Ozcan, A. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery. Optica 2018, 5, 704–710. [Google Scholar] [CrossRef]
- Li, H.; He, G.; Song, Q.; Xia, H.; Liu, Z.; Liang, J.; Li, T. The Study of Tooth Erosion Tested by the Color Digital Holography (CDH) Detection System. Appl. Sci. 2022, 12, 8613. [Google Scholar] [CrossRef]
- Balasubramani, V.; Kujawińska, M.; Allier, C.; Anand, V.; Cheng, C.-J.; Depeursinge, C.; Hai, N.; Juodkazis, S.; Kalkman, J.; Kuś, A.; et al. Roadmap on Digital Holography-Based Quantitative Phase Imaging. J. Imaging 2021, 7, 252. [Google Scholar] [CrossRef]
- Petrov, V.; Pogoda, A.; Sementin, V.; Sevryugin, A.; Shalymov, E.; Venediktov, D.; Venediktov, V. Advances in Digital Holographic Interferometry. J. Imaging 2022, 8, 196. [Google Scholar] [CrossRef]
- Dyomin, V.; Semiletov, I.; Chernykh, D.; Chertoprud, E.; Davydova, A.; Kirillov, N.; Konovalova, O.; Olshukov, A.; Osadchiev, A.; Polovtsev, I. Study of Marine Particles Using Submersible Digital Holographic Camera during the Arctic Expedition. Appl. Sci. 2022, 12, 11266. [Google Scholar] [CrossRef]
- Galande, A.S.; Gurram, H.P.R.; Kamireddy, A.P.; Venkatapuram, V.S.; Hasan, Q.; John, R. Quantitative phase imaging of biological cells using lensless inline holographic microscopy through sparsity-assisted iterative phase retrieval algorithm. J. Appl. Phys. 2022, 132, 243102. [Google Scholar] [CrossRef]
- Zhu, Y.; Yeung, C.H.; Lam, E.Y. Microplastic pollution monitoring with holographic classification and deep learning. J. Phys. Photonics 2021, 3, 024013. [Google Scholar] [CrossRef]
- Kim, J.; Go, T.; Lee, S.J. Accurate real-time monitoring of high particulate matter concentration based on holographic speckles and deep learning. J. Hazard. Mater. 2020, 409, 124637. [Google Scholar] [CrossRef]
- Kemper, B.; von Bally, G. Digital holographic microscopy for live cell applications and technical inspection. Appl. Opt. 2007, 47, A52–A61. [Google Scholar] [CrossRef]
- Muslimov, E.R.; Sakhabutdinov, A.Z.; Morozov, O.G.; Pavlycheva, N.K.; Akhmetov, D.M.; Kharitonov, D.Y. Digital Holographic Positioning Sensor for a Small Deployable Space Telescope. Appl. Sci. 2022, 12, 4427. [Google Scholar] [CrossRef]
- Schnars, U.; Jüptner, W.P.O. Digital recording and numerical reconstruction of holograms. Meas. Sci. Technol. 2002, 13, R85–R101. [Google Scholar] [CrossRef]
- Verrier, N.; Atlan, M. Off-axis digital hologram reconstruction: Some practical considerations. Appl. Opt. 2011, 50, H136–H146. [Google Scholar] [CrossRef]
- Memmolo, P.; Bianco, V.; Paturzo, M.; Ferraro, P. Numerical Manipulation of Digital Holograms for 3-D Imaging and Display: An Overview. Proc. IEEE 2016, 105, 892–905. [Google Scholar] [CrossRef]
- Cheremkhin, P.A.; Kurbatova, E.A.; Evtikhiev, N.N.; Krasnov, V.V.; Rodin, V.G.; Starikov, R.S. Adaptive Digital Hologram Binarization Method Based on Local Thresholding, Block Division and Error Diffusion. J. Imaging 2022, 8, 15. [Google Scholar] [CrossRef]
- Cheremkhin, P.; Evtikhiev, N.; Krasnov, V.; Rodin, V.; Rymov, D.; Starikov, R. Machine learning methods for digital holography and diffractive optics. Procedia Comput. Sci. 2020, 169, 440–444. [Google Scholar] [CrossRef]
- Zeng, T.; Zhu, Y.; Lam, E.Y. Deep learning for digital holography: A review. Opt. Express 2021, 29, 40572. [Google Scholar] [CrossRef]
- Situ, G. Deep holography. Light. Adv. Manuf. 2022, 3, 1–23. [Google Scholar] [CrossRef]
- Montresor, S.; Tahon, M.; Picart, P. Deep learning speckle de-noising algorithms for coherent metrology: A review and a phase-shifted iterative scheme [Invited]. J. Opt. Soc. Am. A 2022, 39, A62. [Google Scholar] [CrossRef]
- Cuche, E.; Marquet, P.; Depeursinge, C. Spatial filtering for zero-order and twin-image elimination in digital off-axis holography. Appl. Opt. 2000, 39, 4070–4075. [Google Scholar] [CrossRef]
- Stoykova, E.S.E.; Kang, H.K.H.; Park, J.P.J. Twin-image problem in digital holography—A survey (Invited Paper). Chin. Opt. Lett. 2014, 12, 060013–60024. [Google Scholar] [CrossRef]
- Li, Y.-L.; Li, N.-N.; Di Wang, D.; Chu, F.; Lee, S.-D.; Zheng, Y.-W.; Wang, Q.-H. Tunable liquid crystal grating based holographic 3D display system with wide viewing angle and large size. Light Sci. Appl. 2022, 11, 188. [Google Scholar] [CrossRef]
- Cheremkhin, P.A.; Evtikhiev, N.; Kozlov, A.V.; Krasnov, V.; Rodin, V.; Starikov, R. An optical-digital method of noise suppression in digital holography. J. Opt. 2022, 24, 115702. [Google Scholar] [CrossRef]
- Wang, D.; Li, N.-N.; Li, Y.-L.; Zheng, Y.-W.; Wang, Q.-H. Curved hologram generation method for speckle noise suppression based on the stochastic gradient descent algorithm. Opt. Express 2021, 29, 42650. [Google Scholar] [CrossRef]
- Wang, D.; Li, Z.-S.; Zheng, Y.-W.; Li, N.-N.; Li, Y.-L.; Wang, Q.-H. High-Quality Holographic 3D Display System Based on Virtual Splicing of Spatial Light Modulator. ACS Photonics 2022. [Google Scholar] [CrossRef]
- Yamaguchi, I.; Zhang, T. Phase-shifting digital holography. Opt. Lett. 1997, 22, 1268–1270. [Google Scholar] [CrossRef]
- Leith, E.N.; Upatnieks, J. Wavefront Reconstruction with Diffused Illumination and Three-Dimensional Objects. J. Opt. Soc. Am. 1964, 54, 1295–1301. [Google Scholar] [CrossRef]
- Denis, L.; Lorenz, D.; Thiébaut, E.; Fournier, C.; Trede, D. Inline hologram reconstruction with sparsity constraints. Opt. Lett. 2009, 34, 3475–3477. [Google Scholar] [CrossRef]
- Momey, F.; Denis, L.; Olivier, T.; Fournier, C. From Fienup’s phase retrieval techniques to regularized inversion for in-line holography: Tutorial. J. Opt. Soc. Am. A 2019, 36, D62–D80. [Google Scholar] [CrossRef]
- Berdeu, A.; Flasseur, O.; Méès, L.; Denis, L.; Momey, F.; Olivier, T.; Grosjean, N.; Fournier, C. Reconstruction of in-line holograms: Combining model-based and regularized inversion. Opt. Express 2019, 27, 14951–14968. [Google Scholar] [CrossRef]
- Zhang, W.; Cao, L.; Brady, D.J.; Zhang, H.; Cang, J.; Zhang, H.; Jin, G. Twin-Image-Free Holography: A Compressive Sensing Approach. Phys. Rev. Lett. 2018, 121, 093902. [Google Scholar] [CrossRef]
- Zhang, G.; Guan, T.; Shen, Z.; Wang, X.; Hu, T.; Wang, D.; He, Y.; Xie, N. Fast phase retrieval in off-axis digital holographic microscopy through deep learning. Opt. Express 2018, 26, 19388–19405. [Google Scholar] [CrossRef]
- Tahon, M.; Montrésor, S.; Picart, P. Deep Learning Network for Speckle De-Noising in Severe Conditions. J. Imaging 2022, 8, 165. [Google Scholar] [CrossRef]
- Galande, A.S.; Thapa, V.; Gurram, H.P.R.; John, R. Untrained deep network powered with explicit denoiser for phase recovery in inline holography. Appl. Phys. Lett. 2023, 122, 133701. [Google Scholar] [CrossRef]
- Zhang, K.; Zuo, W.; Gu, S.; Zhang, L. Learning Deep CNN Denoiser Prior for Image Restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 2808–2817. [Google Scholar] [CrossRef]
- Nguyen, T.; Bui, V.; Lam, V.; Raub, C.B.; Chang, L.-C.; Nehmetallah, G. Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection. Opt. Express 2017, 25, 15043–15057. [Google Scholar] [CrossRef]
- Pan, F.; Dong, B.; Xiao, W.; Ferraro, P. Stitching sub-aperture in digital holography based on machine learning. Opt. Express 2020, 28, 6537–6551. [Google Scholar] [CrossRef]
- Wang, H.; Li, K.; Jiang, X.; Wang, J.; Zhang, X.; Liu, X. Zero-order term suppression in off-axis holography based on deep learning method. Opt. Commun. 2023, 537, 129264. [Google Scholar] [CrossRef]
- Rymov, D.A.; Starikov, R.S.; Cheremkhin, P.A. Neural-network-enabled holographic image reconstruction via amplitude and phase extraction. J. Opt. Technol. 2022, 89, 511–516. [Google Scholar] [CrossRef]
- Jiao, S.; Jin, Z.; Chang, C.; Zhou, C.; Zou, W.; Li, X. Compression of phase-only holograms with JPEG standard and deep learning. Appl. Sci. 2018, 8, 1258. [Google Scholar] [CrossRef]
- Zhang, X.; Wang, H.; Wang, W.; Yang, S.; Wang, J.; Lei, J.; Zhang, Z.; Dong, Z. Particle field positioning with a commercial microscope based on a developed CNN and the depth-from-defocus method. Opt. Lasers Eng. 2022, 153, 106989. [Google Scholar] [CrossRef]
- Shao, S.; Mallery, K.; Kumar, S.S.; Hong, J. Machine learning holography for 3D particle field imaging. Opt. Express 2020, 28, 2987–2999. [Google Scholar] [CrossRef]
- Terbe, D.; Orzó, L.; Zarándy, A. Classification of Holograms with 3D-CNN. Sensors 2022, 22, 8366. [Google Scholar] [CrossRef]
- Pitkäaho, T.; Manninen, A.; Naughton, T.J. Focus prediction in digital holographic microscopy using deep convolutional neural networks. Appl. Opt. 2019, 58, A202–A208. [Google Scholar] [CrossRef]
- Cuenat, S.; Andréoli, L.; André, A.N.; Sandoz, P.; Laurent, G.J.; Couturier, R.; Jacquot, M. Fast autofocusing using tiny transformer networks for digital holographic microscopy. Opt. Express 2022, 30, 24730. [Google Scholar] [CrossRef]
- Jaferzadeh, K.; Fevens, T. HoloPhaseNet: Fully automated deep-learning-based hologram reconstruction using a conditional generative adversarial model. Biomed. Opt. Express 2022, 13, 4032. [Google Scholar] [CrossRef]
- Ju, Y.-G.; Choo, H.-G.; Park, J.-H. Learning-based complex field recovery from digital hologram with various depth objects. Opt. Express 2022, 30, 26149. [Google Scholar] [CrossRef]
- Pirone, D.; Sirico, D.G.; Miccio, L.; Bianco, V.; Mugnano, M.; Ferraro, P.; Memmolo, P. Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning. Lab Chip 2022, 22, 793–804. [Google Scholar] [CrossRef]
- Wang, K.; Dou, J.; Kemao, Q.; Di, J.; Zhao, J. Y-Net: A one-to-two deep learning framework for digital holographic reconstruction. Opt. Lett. 2019, 44, 4765–4768. [Google Scholar] [CrossRef]
- Rivenson, Y.; Zhang, Y.; Günaydın, H.; Teng, D.; Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl. 2018, 7, 17141. [Google Scholar] [CrossRef]
- Chen, H.; Huang, L.; Liu, T.; Ozcan, A. Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization. Light Sci. Appl. 2022, 11, 254. [Google Scholar] [CrossRef]
- Li, H.; Chen, X.; Chi, Z.; Mann, C.; Razi, A. Deep DIH: Single-Shot Digital In-Line Holography Reconstruction by Deep Learning. IEEE Access 2020, 8, 202648–202659. [Google Scholar] [CrossRef]
- Wang, K.; Kemao, Q.; Di, J.; Zhao, J. Y4-Net: A deep learning solution to one-shot dual-wavelength digital holographic reconstruction. Opt. Lett. 2020, 45, 4220–4223. [Google Scholar] [CrossRef]
- Niknam, F.; Qazvini, H.; Latifi, H. Holographic optical field recovery using a regularized untrained deep decoder network. Sci. Rep. 2021, 11, 10903. [Google Scholar] [CrossRef]
- Ren, Z.; Xu, Z.; Lam, E.Y. End-to-end deep learning framework for digital holographic reconstruction. Adv. Photonics 2019, 1, 016004. [Google Scholar] [CrossRef]
- Wu, Y.; Wu, J.; Jin, S.; Cao, L.; Jin, G. Dense-U-net: Dense encoder–decoder network for holographic imaging of 3D particle fields. Opt. Commun. 2021, 493, 126970. [Google Scholar] [CrossRef]
- Cai, S.; Wu, Y.; Chen, G. A Novel Elastomeric UNet for Medical Image Segmentation. Front. Aging Neurosci. 2022, 14, 31. [Google Scholar] [CrossRef]
- Neven, R.; Goedemé, T. A Multi-Branch U-Net for Steel Surface Defect Type and Severity Segmentation. Metals 2021, 11, 870. [Google Scholar] [CrossRef]
- Dhillon, A.; Verma, G.K. Convolutional neural network: A review of models, methodologies and applications to object detection. Prog. Artif. Intell. 2019, 9, 85–112. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Proceedings of the 18th International Conference, Munich, Germany, 5–9 October 2015; Springer International Publishing: Berlin/Heidelberg, Germany, 2015; Volume 9351, pp. 234–241. [Google Scholar]
- Falk, T.; Mai, D.; Bensch, R.; Çiçek, Ö.; Abdulkadir, A.; Marrakchi, Y.; Böhm, A.; Deubner, J.; Jäckel, Z.; Seiwald, K.; et al. U-Net: Deep learning for cell counting, detection, and morphometry. Nat. Methods 2018, 16, 67–70. [Google Scholar] [CrossRef]
- Li, L.; Li, X.; Jiang, L.; Su, X.; Chen, F. A review on deep learning techniques for cloud detection methodologies and challenges. Signal Image Video Process. 2021, 15, 1527–1535. [Google Scholar] [CrossRef]
- Xiao, H.; Li, L.; Liu, Q.; Zhu, X.; Zhang, Q. Transformers in medical image segmentation: A review. Biomed. Signal Process. Control 2023, 84, 104791. [Google Scholar] [CrossRef]
- Hao, X.; Yin, L.; Li, X.; Zhang, L.; Yang, R. A Multi-Objective Semantic Segmentation Algorithm Based on Improved U-Net Networks. Remote Sens. 2023, 15, 1838. [Google Scholar] [CrossRef]
- Wang, A.; Togo, R.; Ogawa, T.; Haseyama, M. Defect Detection of Subway Tunnels Using Advanced U-Net Network. Sensors 2022, 22, 2330. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar] [CrossRef]
- Gu, J.; Wang, Z.; Kuen, J.; Ma, L.; Shahroudy, A.; Shuai, B.; Liu, T.; Wang, X.; Wang, G.; Cai, J.; et al. Recent advances in convolutional neural networks. Pattern Recognit. 2018, 77, 354–377. [Google Scholar] [CrossRef]
- Ding, B.; Qian, H.; Zhou, J. Activation functions and their characteristics in deep neural networks. In Proceedings of the 2018 Chinese Control And Decision Conference (CCDC), Shenyang, China, 9–11 June 2018; pp. 1836–1841. [Google Scholar] [CrossRef]
- Shi, W.; Caballero, J.; Huszár, F.; Totz, J.; Aitken, A.P.; Bishop, R.; Rueckert, D.; Wang, Z. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. arXiv 2016, arXiv:1609.05158. [Google Scholar] [CrossRef]
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Available online: http://yann.lecun.com/exdb/mnist/ (accessed on 1 March 2021).
- Gonzalez, R.C.; Woods, R.E. Digital Image Processing; Pearson: New York, NY, USA, 2018; ISBN 9781292223049. [Google Scholar]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef]
Metric | 1st Plane | 2nd Plane |
---|---|---|
SSIM | 0.86 ± 0.04 | 0.85 ± 0.04 |
CC | 0.97 ± 0.02 | 0.95 ± 0.02 |
Metric | HoloForkNet | Dense-U-net |
---|---|---|
SSIM | 0.972 ± 0.004 | 0.93 ± 0.09 |
CC | 0.98 ± 0.04 | 0.7 ± 0.2 |
Metrics | Objects 1 | Objects 2 | ||
---|---|---|---|---|
1st Plane | 2nd Plane | 1st Plane | 2nd Plane | |
Small distance between planes | ||||
SSIM | 0.84 ± 0.04 | 0.85 ± 0.03 | 0.84 ± 0.04 | 0.930 ± 0.009 |
CC | 0.994 ± 0.003 | 0.994 ± 0.003 | 0.994 ± 0.003 | 0.99 ± 0.01 |
Large distance between planes | ||||
SSIM | 0.85 ± 0.03 | 0.80 ± 0.03 | 0.85 ± 0.03 | 0.923 ± 0.007 |
CC | 0.993 ± 0.003 | 0.989 ± 0.006 | 0.993 ± 0.003 | 0.992 ± 0.002 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Svistunov, A.S.; Rymov, D.A.; Starikov, R.S.; Cheremkhin, P.A. HoloForkNet: Digital Hologram Reconstruction via Multibranch Neural Network. Appl. Sci. 2023, 13, 6125. https://doi.org/10.3390/app13106125
Svistunov AS, Rymov DA, Starikov RS, Cheremkhin PA. HoloForkNet: Digital Hologram Reconstruction via Multibranch Neural Network. Applied Sciences. 2023; 13(10):6125. https://doi.org/10.3390/app13106125
Chicago/Turabian StyleSvistunov, Andrey S., Dmitry A. Rymov, Rostislav S. Starikov, and Pavel A. Cheremkhin. 2023. "HoloForkNet: Digital Hologram Reconstruction via Multibranch Neural Network" Applied Sciences 13, no. 10: 6125. https://doi.org/10.3390/app13106125