Regionally Adaptive Active Learning Framework for Nuclear Segmentation in Microscopy Image
Abstract
:1. Introduction
- Poor image quality. Image quality is limited by the acquisition microscope hardware and imaging time. As shown in Figure 1, intensity inhomogeneity (displayed in the red box) is a common issue in microscopy. It is difficult to find an optimal threshold to segment nuclei that works well for the entire microscopy image. Other difficulties include image noise and low image contrast.
- Densely packed nuclei. When nuclei are in close proximity, the boundaries between nuclei are difficult to delineate. As shown in Figure 1, the cells in the blue box are densely packed, making accurate segmentation more difficult. The situation is very common, such as in the dentate gyrus of brain hippocampus. Since the topology of densely packed nuclei is very complex in terms of the number of nuclei in close proximity, current state-of-the-art methods, which work well to segment a single nucleus, usually fail to separate the boundaries between multiple nuclei touching one another.
- Large image dimensionality. In general, the computational challenges are based on the data size and complexity. A typical mouse brain (volume of ∼1000 mm3) imaged at high-resolution (e.g., 0.4853 μm × 0.4853 μm × 1 μm) results in approximately 8 TB of data (16-bit) for each acquired channel. Thus, it is computationally prohibitive to use sophisticated image processing methods that require matrix inversion or fit the contour of nuclei in an iterative manner.
2. Related Work on Nuclei Segmentation
- Classic image processing techniques. The majority of current nucleus/cell segmentation methods are based on several classic image processing algorithms such as intensity thresholding [10,11,12,13,14], morphology operations [15,16,17], region accumulation [18,19,20,21], and deformable models [22,23]. Intensity thresholding might be the simplest and computationally cheapest method for cell segmentation. However, intensity thresholding, even the local thresholding technique such as local Otsu [24], does not lead to accurate segmentations within regions of densely packed nuclei since the boundaries are usually blurry. Similarly, a multi-level segmentation method using Darwinian particle swarm optimization (DPSO) was proposed in [25], which works in the frequency domain and assumes the nuclei voxels usually have a high response to certain specifically designed band-pass filters. The region-growing technique starts from a set of selected seed points in the image and iteratively adds connected points to form the labeled regions [5]. The significant limitation of the region-growing and watershed transform is its propensity to over-segment, due to image noise. To address this limitation, marker-based watershed methods have been proposed [18,26] to improve the segmentation accuracy by using the detected nuclei as the seeds to guide the segmentation of nuclei. However, it is very challenging to determine the localization of markers in large-scale microscopy images [27]. Recently, a cell detection method called CUBIC was proposed in [3] to construct a single-cell resolution whole brain atlas. CUBIC is a hybrid 2D/3D method where a 2D mean filter is first used to detect low maximum intensity values at the center of the nuclei and then a 3D filter is applied to unify the multiply-detected cells. However, CUBIC is unable to provide morphological information such as size and shape of nuclei, which might limit its application.
- Nuclei segmentation using unsupervised learning. K-means clustering has been applied to obtain an initial segmentation in [28,29], followed by a set of post-processing steps to refine the segmentation results. Probability inference under the EM (expectation maximization) framework is also used in automated nuclei segmentation for various biomedical data [30,31,32]. In [33,34], spectral clustering is used to identify the boundary of a cell, where the spatial and color information are used to measure similarity between the neighboring points. Furthermore, Song et al. proposed an improved nuclei segmentation method using the adaptive hierarchical clustering algorithm [35]. To improve the robustness of nuclei segmentation results, Su et al. [36] proposed a semi-automatic segmentation method that uses super-pixels to determine the segmentation label for each super-pixel, instead of each point. These unsupervised approaches, which are globally optimized for all the points in the entire image, are often not robust against the presence of substantial intensity inhomogeneity.
- Nuclei segmentation using supervised learning techniques. A coarse-to-fine nucleus segmentation framework was proposed in [37], where a multiscale convolutional network (MSCN) was integrated with the graph partition-based method for cytoplasm and nucleus segmentation in cervical images. Ronneberger et al. proposed the U-net architecture and applied the U-net to a cell segmentation task in light microscopic images [38]. Attention U-net introduces the attention and residual modules in the U-Net for the segmentation of live cells in microscopy images [39]. Cellpose proposed a generalist for a wide range of images of cell-based U-net [40]. Mask-RCNN used object detection technologies to achieve instance segmentation of nuclei [41]. ClearMap 1.0 [2] is a widely used computational software in the neuroscience community that provides functions such as cell detection, localization, and quantification of nuclei. The ilastik [42] is the component in ClearMap for nuclear segmentation, which uses random forest techniques [43].
3. Methods
3.1. Cascading Architecture with Cross-Training
3.2. Regionally Adaptive Fine-Tuning Strategy
- Ovality is measured by the ratio of the major axis and minor axis of fitting an ellipse (red ellipse in Figure 3a). For most individual nuclei, the contour is close to a circle. We can identify the region of suspected nuclei clumps if the ratio is greater than a threshold (here we use 3 for our image data).
- Concavity is measured by the difference in size between convex hulls (red curve in Figure 3b) and connected regions (green curve in Figure 3b). If the concavity degree is greater than a certain threshold (here we use 30 for our image data), it is highly possible that the underlying region is a clump of nuclei.
- The area of connected region. As shown in Figure 3c, it is straightforward to detect a region containing densely packed nuclei based on the area of the connected region being larger than a common nucleus area (here we used 500 pixels for our image data as threshold).
3.3. Complete Workflow of Regionally Adaptive Network via Cascading Architecture (RANCA)
- 1.
- Apply the cascaded models at each point of a microscopy image patch, where the output of a layer is the average of output from the concurrent training models. The input is the image intensity image patch (Figure 6a). The image intensities are normalized by the mean of image pixel values. The output is the probability of being background, edge, and nuclei for the center point. After that, we can obtain the probability map for background, nuclei, and nuclei edge, as shown in Figure 6b;
- 2.
- Apply the nuclei clump detection approach to partition whole images into regions with multiple touching nuclei and unconnected nuclei, respectively.
- 3.
- 4.
- Combine (by adding) the segmentation results to obtain the final segmentation, as shown in Figure 6e.
4. Experiments and Results Analysis
4.1. Experimental Setup
4.2. Evaluation of Segmentation Performance
4.2.1. Comparion with Other Methods
4.2.2. Ablation Study
4.3. Nuclear Counting Application
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Fürth, D.; Vaissière, T.; Tzortzi, O.; Xuan, Y.; Märtin, A.; Lazaridis, I.; Spigolon, G.; Fisone, G.; Tomer, R.; Deisseroth, K.; et al. An interactive framework for whole-brain maps at cellular resolution. Nat. Neurosci. 2018, 21, 139–149. [Google Scholar] [CrossRef] [PubMed]
- Renier, N.; Adams, E.L.; Kirst, C.; Wu, Z.; Azevedo, R.; Kohl, J.; Autry, A.E.; Kadiri, L.; Venkataraju, K.U.; Zhou, Y.; et al. Mapping of brain activity by automated volume analysis of immediate early genes. Cell 2016, 165, 1789–1802. [Google Scholar] [CrossRef] [PubMed]
- Murakami, T.C.; Mano, T.; Saikawa, S.; Horiguchi, S.A.; Shigeta, D.; Baba, K.; Sekiya, H.; Shimizu, Y.; Tanaka, K.F.; Kiyonari, H.; et al. A three-dimensional single-cell-resolution whole-brain atlas using CUBIC-X expansion microscopy and tissue clearing. Nat. Neurosci. 2018, 21, 625–637. [Google Scholar] [CrossRef] [PubMed]
- Iritani, S. Neuropathology of schizophrenia: A mini review. Neuropathology 2007, 27, 604–608. [Google Scholar] [CrossRef]
- Meijering, E. Cell segmentation: 50 years down the road [life sciences]. IEEE Signal Process. Mag. 2012, 29, 140–145. [Google Scholar] [CrossRef]
- Bai, M.; Urtasun, R. Deep watershed transform for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 5221–5229. [Google Scholar]
- Lin, T.Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2980–2988. [Google Scholar]
- Januszewski, M.; Kornfeld, J.; Li, P.H.; Pope, A.; Blakely, T.; Lindsey, L.; Maitin-Shepard, J.; Tyka, M.; Denk, W.; Jain, V. High-precision automated reconstruction of neurons with flood-filling networks. Nat. Methods 2018, 15, 605–610. [Google Scholar] [CrossRef] [PubMed]
- Hollandi, R.; Moshkov, N.; Paavolainen, L.; Tasnadi, E.; Piccinini, F.; Horvath, P. Nucleus segmentation: Towards automated solutions. Trends Cell Biol. 2022, 32, 295–310. [Google Scholar] [CrossRef]
- Callau, C.; Lejeune, M.; Korzynska, A.; García, M.; Bueno, G.; Bosch, R.; Jaén, J.; Orero, G.; Salvadó, T.; López, C. Evaluation of cytokeratin-19 in breast cancer tissue samples: A comparison of automatic and manual evaluations of scanned tissue microarray cylinders. BioMedical Eng. OnLine 2015, 14, 1–11. [Google Scholar] [CrossRef]
- Kong, J.; Wang, F.; Teodoro, G.; Liang, Y.; Zhu, Y.; Tucker-Burden, C.; Brat, D.J. Automated cell segmentation with 3D fluorescence microscopy images. In Proceedings of the 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), Brooklyn, NY, USA, 16–19 April 2015; pp. 1212–1215. [Google Scholar]
- Forsberg, D.; Monsef, N. Evaluating cell nuclei segmentation for use on whole-slide images in lung cytology. In Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden, 24–28 August 2014; pp. 3380–3385. [Google Scholar]
- Wienert, S.; Heim, D.; Saeger, K.; Stenzinger, A.; Beil, M.; Hufnagl, P.; Dietel, M.; Denkert, C.; Klauschen, F. Detection and segmentation of cell nuclei in virtual microscopy images: A minimum-model approach. Sci. Rep. 2012, 2, 503. [Google Scholar] [CrossRef]
- Su, H.; Xing, F.; Lee, J.D.; Peterson, C.A.; Yang, L. Automatic myonuclear detection in isolated single muscle fibers using robust ellipse fitting and sparse representation. IEEE/ACM Trans. Comput. Biol. Bioinform. 2013, 11, 714–726. [Google Scholar] [CrossRef]
- Schmitt, O.; Hasse, M. Morphological multiscale decomposition of connected regions with emphasis on cell clusters. Comput. Vis. Image Underst. 2009, 113, 188–201. [Google Scholar] [CrossRef]
- Wang, Q.; Niemi, J.; Tan, C.M.; You, L.; West, M. Image segmentation and dynamic lineage analysis in single-cell fluorescence microscopy. Cytom. Part A J. Int. Soc. Adv. Cytom. 2010, 77, 101–110. [Google Scholar] [CrossRef] [PubMed]
- Dorini, L.B.; Minetto, R.; Leite, N.J. Semiautomatic white blood cell segmentation based on multiscale analysis. IEEE J. Biomed. Health Inform. 2012, 17, 250–256. [Google Scholar] [CrossRef]
- Koyuncu, C.F.; Akhan, E.; Ersahin, T.; Cetin-Atalay, R.; Gunduz-Demir, C. Iterative h-minima-based marker-controlled watershed for cell nucleus segmentation. Cytom. Part A 2016, 89, 338–349. [Google Scholar] [CrossRef]
- Qu, L.; Long, F.; Liu, X.; Kim, S.; Myers, E.; Peng, H. Simultaneous recognition and segmentation of cells: Application in C. elegans. Bioinformatics 2011, 27, 2895–2902. [Google Scholar] [CrossRef] [PubMed]
- Liu, F.; Xing, F.; Zhang, Z.; Mcgough, M.; Yang, L. Robust muscle cell quantification using structured edge detection and hierarchical segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015; Proceedings, Part III 18. Springer: Cham, Switzerland, 2015; pp. 324–331. [Google Scholar]
- Santamaria-Pang, A.; Huang, Y.; Rittscher, J. Cell segmentation and classification via unsupervised shape ranking. In Proceedings of the 2013 IEEE 10th International Symposium on Biomedical Imaging, San Francisco, CA, USA, 7–11 April 2013; pp. 406–409. [Google Scholar]
- Delgado-Gonzalo, R.; Uhlmann, V.; Schmitter, D.; Unser, M. Snakes on a plane: A perfect snap for bioimage analysis. IEEE Signal Process. Mag. 2014, 32, 41–48. [Google Scholar] [CrossRef]
- Dufour, A.; Thibeaux, R.; Labruyere, E.; Guillen, N.; Olivo-Marin, J.C. 3-D active meshes: Fast discrete deformable models for cell tracking in 3-D time-lapse microscopy. IEEE Trans. Image Process. 2010, 20, 1925–1937. [Google Scholar] [CrossRef]
- Baxi, A. A review on Otsu image segmentation algorithm. Int. J. Adv. Res. Comput. Eng. Technol. (IJARCET) 2013, 2, 387–389. [Google Scholar]
- Ghamisi, P.; Couceiro, M.S.; Martins, F.M.; Benediktsson, J.A. Multilevel image segmentation based on fractional-order Darwinian particle swarm optimization. IEEE Trans. Geosci. Remote Sens. 2013, 52, 2382–2394. [Google Scholar] [CrossRef]
- Yang, L.; Qiu, Z.; Greenaway, A.H.; Lu, W. A new framework for particle detection in low-SNR fluorescence live-cell images and its application for improved particle tracking. IEEE Trans. Biomed. Eng. 2012, 59, 2040–2050. [Google Scholar] [CrossRef]
- Xing, F.; Yang, L. Robust nucleus/cell detection and segmentation in digital pathology and microscopy images: A comprehensive review. IEEE Rev. Biomed. Eng. 2016, 9, 234–263. [Google Scholar] [CrossRef] [PubMed]
- Chen, X.; Zhu, Y.; Li, F.; Zheng, Z.Y.; Chang, E.C.; Ma, J.; Wong, S.T. Accurate segmentation of touching cells in multi-channel microscopy images with geodesic distance based clustering. Neurocomputing 2015, 149, 39–47. [Google Scholar] [CrossRef]
- Kothari, S.; Chaudry, Q.; Wang, M.D. Automated cell counting and cluster segmentation using concavity detection and ellipse fitting techniques. In Proceedings of the 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Boston, MA, USA, 28 June–1 July 2009; pp. 795–798. [Google Scholar]
- Fatakdawala, H.; Xu, J.; Basavanhally, A.; Bhanot, G.; Ganesan, S.; Feldman, M.; Tomaszewski, J.E.; Madabhushi, A. Expectation–maximization-driven geodesic active contour with overlap resolution (emagacor): Application to lymphocyte segmentation on breast cancer histopathology. IEEE Trans. Biomed. Eng. 2010, 57, 1676–1689. [Google Scholar] [CrossRef]
- Irshad, H.; Veillard, A.; Roux, L.; Racoceanu, D. Methods for nuclei detection, segmentation, and classification in digital histopathology: A review—Current status and future potential. IEEE Rev. Biomed. Eng. 2013, 7, 97–114. [Google Scholar] [CrossRef]
- Jung, C.; Kim, C.; Chae, S.W.; Oh, S. Unsupervised segmentation of overlapped nuclei using Bayesian classification. IEEE Trans. Biomed. Eng. 2010, 57, 2825–2832. [Google Scholar] [CrossRef] [PubMed]
- Kovacheva, V.N.; Khan, A.M.; Khan, M.; Epstein, D.B.; Rajpoot, N.M. DiSWOP: A novel measure for cell-level protein network analysis in localized proteomics image data. Bioinformatics 2014, 30, 420–427. [Google Scholar] [CrossRef]
- Ge, J.; Gong, Z.; Chen, J.; Liu, J.; Nguyen, J.; Yang, Z.; Wang, C.; Sun, Y. A system for counting fetal and maternal red blood cells. IEEE Trans. Biomed. Eng. 2014, 61, 2823–2829. [Google Scholar] [CrossRef]
- Song, Y.; Tan, E.L.; Jiang, X.; Cheng, J.Z.; Ni, D.; Chen, S.; Lei, B.; Wang, T. Accurate cervical cell segmentation from overlapping clumps in pap smear images. IEEE Trans. Med. Imaging 2016, 36, 288–300. [Google Scholar] [CrossRef]
- Su, H.; Yin, Z.; Huh, S.; Kanade, T.; Zhu, J. Interactive cell segmentation based on active and semi-supervised learning. IEEE Trans. Med. Imaging 2015, 35, 762–777. [Google Scholar] [CrossRef]
- Song, Y.; Zhang, L.; Chen, S.; Ni, D.; Lei, B.; Wang, T. Accurate segmentation of cervical cytoplasm and nuclei based on multiscale convolutional network and graph partitioning. IEEE Trans. Biomed. Eng. 2015, 62, 2421–2433. [Google Scholar] [CrossRef]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 5–9 October 2015; Proceedings, Part III 18. Springer: Cham, Switzerland, 2015; pp. 234–241. [Google Scholar]
- Ghaznavi, A.; Rychtáriková, R.; Saberioon, M.; Štys, D. Cell segmentation from telecentric bright-field transmitted light microscopy images using a Residual Attention U-Net: A case study on HeLa line. Comput. Biol. Med. 2022, 147, 105805. [Google Scholar] [CrossRef] [PubMed]
- Stringer, C.; Wang, T.; Michaelos, M.; Pachitariu, M. Cellpose: A generalist algorithm for cellular segmentation. Nat. Methods 2021, 18, 100–106. [Google Scholar] [CrossRef]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Berg, S.; Kutra, D.; Kroeger, T.; Straehle, C.N.; Kausler, B.X.; Haubold, C.; Schiegg, M.; Ales, J.; Beier, T.; Rudy, M.; et al. Ilastik: Interactive machine learning for (bio) image analysis. Nat. Methods 2019, 16, 1226–1232. [Google Scholar] [CrossRef] [PubMed]
- Ho, T.K. The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 1998, 20, 832–844. [Google Scholar]
- Kim, M.; Wu, G.; Guo, Y.; Shen, D. Joint labeling of multiple regions of interest (ROIS) by enhanced auto context models. In Proceedings of the 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), Brooklyn, NY, USA, 16–19 April 2015; pp. 1560–1563. [Google Scholar]
- Tu, Z.; Bai, X. Auto-context and its application to high-level vision tasks and 3D brain image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2009, 32, 1744–1757. [Google Scholar]
- Van Valen, D.A.; Kudo, T.; Lane, K.M.; Macklin, D.N.; Quach, N.T.; DeFelice, M.M.; Maayan, I.; Tanouchi, Y.; Ashley, E.A.; Covert, M.W. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS Comput. Biol. 2016, 12, e1005177. [Google Scholar] [CrossRef]
- Caicedo, J.C.; Goodman, A.; Karhohs, K.W.; Cimini, B.A.; Ackerman, J.; Haghighi, M.; Heng, C.; Becker, T.; Doan, M.; McQuin, C.; et al. Nucleus segmentation across imaging experiments: The 2018 Data Science Bowl. Nat. Methods 2019, 16, 1247–1253. [Google Scholar] [CrossRef]
Touching | Morphology | Adaption | w/o Post-Processing | |
---|---|---|---|---|
ilastik | × | ✓ | ✓ | ✓ |
U-net | × | ✓ | ✓ | ✓ |
Attention U-net | × | ✓ | ✓ | ✓ |
Mask-RCNN | ✓ | ✓ | ✓ | ✓ |
Cellpose | ✓ | ✓ | ✓ | × |
RANCA | ✓ | ✓ | ✓ | ✓ |
Method | Dice (%) | Dice FP (%) | Dice FN (%) | Jaccard (%) |
---|---|---|---|---|
ilastik | 90.24 | 8.72 | 10.80 | 82.22 |
U-net | 82.50 | 35.00 | 0.02 | 70.21 |
Attention U-net | 83.14 | 33.70 | 0.02 | 77.12 |
Mask-RCNN | 90.22 | 1.69 | 17.86 | 82.19 |
Cellpose | 90.69 | 2.64 | 16.14 | 82.31 |
RANCA | 92.16 | 9.47 | 6.21 | 85.46 |
Method | Comonents | Size = 30 | Size = 40 | Size = 50 | Size = 60 | Size = 70 | Size = 80 | |
---|---|---|---|---|---|---|---|---|
CA | RA | |||||||
CAC | ✓ | 81.65 ± 1.76 | 83.35 ± 3.26 | 85.79 ± 4.86 | 85.98 ± 2.10 | 88.04 ± 2.44 | 86.56 ± 3.01 | |
RANCA | ✓ | ✓ | 82.85 ± 2.44 | 85.73 ± 1.73 | 86.14 ± 1.70 | 86.08 ± 1.77 | 89.24 ± 1.85 | 87.92 ± 1.09 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, Q.; Wei, J.; Quan, B. Regionally Adaptive Active Learning Framework for Nuclear Segmentation in Microscopy Image. Electronics 2024, 13, 3430. https://doi.org/10.3390/electronics13173430
Wang Q, Wei J, Quan B. Regionally Adaptive Active Learning Framework for Nuclear Segmentation in Microscopy Image. Electronics. 2024; 13(17):3430. https://doi.org/10.3390/electronics13173430
Chicago/Turabian StyleWang, Qian, Jing Wei, and Bo Quan. 2024. "Regionally Adaptive Active Learning Framework for Nuclear Segmentation in Microscopy Image" Electronics 13, no. 17: 3430. https://doi.org/10.3390/electronics13173430