*Proceedings* **Deep Image Segmentation for Breast Keypoint Detection †**

#### **Tiago Gonçalves 1,\* , Wilson Silva <sup>1</sup> , Maria J. Cardoso <sup>2</sup> and Jaime S. Cardoso <sup>1</sup>**


Published: 21 August 2020

**Abstract:** The main aim of breast cancer conservative treatment is the optimisation of the aesthetic outcome and, implicitly, women's quality of life, without jeopardising local cancer control and overall survival. Moreover, there has been an effort to try to define an optimal tool capable of performing the aesthetic evaluation of breast cancer conservative treatment outcomes. Recently, a deep learning algorithm that integrates the learning of keypoints' probability maps in the loss function as a regularisation term for the robust learning of the keypoint localisation has been proposed. However, it achieves the best results when used in cooperation with a shortest-path algorithm that models images as graphs. In this work, we analysed a novel algorithm based on the interaction of deep image segmentation and deep keypoint detection models capable of improving both state-of-the-art performance and execution-time on the breast keypoint detection task.

**Keywords:** aesthetic assessment of breast cancer surgery outcomes; artificial intelligence; breast cancer; breast cancer conservative treatment; computer vision; deep learning; image segmentation; machine learning; keypoint detection

#### **1. Introduction**

Breast cancer is a highly mutable and rapidly evolving disease, however, thanks to the generalised use of breast cancer screening and better treatments, approximately 90% of the cases can be cured [1,2]. Therefore, it is now possible to employ breast cancer conservative treatment (BCCT) approaches, which only require the removal of cancerous tissue with a rim of healthy tissue, instead of radical mastectomy-based approaches, which require the removal of the entire breast and posterior breast reconstruction [3]. In both cases, it is possible to obtain good cosmetic results, and consequently, improve patients' quality of life. The objective assessment of the cosmetic results, which acts as a reliable proxy of the quality of the treatment and valuable input to the improvement of current techniques, is performed through the analysis of digital photographs, from which, several features are extracted and used to train a classification algorithm [4]. To facilitate the extraction of such features, the annotation of several breast keypoints is required. Several semi-automatic methods to perform this keypoint annotation are already available, however, they still need the input from the user and are time-consuming and computationally demanding. Recently, Silva et al. showed that deep learning algorithms may be part of the answer. They introduced a deep algorithm based deep neural networks (DNN) which receives an image as input and returns the coordinates of the breast keypoints as output [5], which are then given to a shortest-path algorithm that models images as graphs to refine breast keypoint localisation. Although this increased the performance of the task of breast

keypoint annotation, this is still computationally complex. To overcome this issue, we proposed a novel deep keypoint detection algorithm, which combines the approach by Silva et al. together with deep image segmentation model that refines breast keypoint localisation in lesser time, and with improved precision [6].

#### **2. Materials and Methods**

Our approach combines the DNN proposed by Silva et al. and a deep image segmentation model, U-Net++ [7], in a pipeline (see Figure 1). The intuition behind this approach is that it is easier to detect breast contours if one is capable to detect breasts first. We started by the training of both the U-Net++ and the DNN by Silva et al. in the breast detection (i.e., generate breast masks) and in the keypoint detection tasks, respectively. The complete pipeline works as follows: the image is given as input to U-Net++, which returns a breast segmentation mask; from this mask, contours are extracted using the marching squares algorithm [8]; the image is given as input to the DNN by Silva et al., which returns a set of breast keypoints; the refined localisation of breast keypoints is obtained with the projection of this set onto the breast segmentation mask contours through the minimisation of the Euclidean Distance of a given breast keypoint and a given contour (processing step). We also trained both the DNN and the complete keypoint method proposed by Silva et al. for comparison purposes. All the experiments were performed taking into account 5-fold cross-validation. In addition to the study of algorithms' performance, a study on the algorithms' execution time, in seconds, was also done, regarding the interest in the deployment of these algorithms into a web-application for both the research and medical communities.

**Figure 1.** Proposed deep image segmentation keypoint detection method [6].

#### **3. Results**

Table 1 presents the average error distance (measured in pixels) and the average execution time (measured in seconds) of each model inference on the test set. Figure 2 shows a visual example of the obtained results.

**Figure 2.** Example of the results obtained with the proposed deep image segmentation method. The first image is the input photograph and the second image is the U-Net++ predicted mask with the detected breast keypoints (after the processing step).



#### **4. Discussion**

Our keypoint detection method surpassed both the DNN-based keypoint detection and the keypoint detection method from Silva et al. in the endpoints and breast contours detection tasks, which were, to our knowledge, the state-of-the-art breast keypoint detection algorithms. Besides, this novel algorithm achieves lower values of standard deviation and maximum error, which suggests more consistency when compared with the other two. Regarding the study of performance, our keypoint detection method presents the best balance between time-efficiency and accuracy, being the most accurate model, with a time efficiency comparable to the most time-efficient method.

#### **5. Conclusions**

In this work, we proposed keypoint detection method that combines a deep keypoint detection and deep image segmentation models, and capable of achieving a good balance between both performance and execution-time. Further studies should be focused on the development of a fully end-to-end deep keypoint detection model, trained with a multi-term loss function, and on the deployment of these breast keypoint detection algorithms into a fully-functional web-application for both the research and medical communities.

**Author Contributions:** Conceptualization, T.G., W.S. and J.S.C.; methodology, T.G., W.S. and J.S.C.; software, T.G. and W.S.; validation, J.S.C.; formal analysis, J.S.C.; investigation, T.G., W.S., M.J.C. and J.S.C.; resources, M.J.C. and J.S.C.; data curation, M.J.C. and J.S.C.; writing—original draft preparation, T.G.; writing—review and editing, W.S. and J.S.C.; visualization, M.J.C. and J.S.C.; supervision, J.S.C.; project administration, J.S.C.; funding acquisition, J.S.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was partially financed by the ERDF - European Regional Development Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme and by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia within project "POCI-01-0145-FEDER-028857" and PhD grant number SFRH/BD/139468/2018.

**Acknowledgments:** The authors thank Florian Fitzal for sharing the VIENNA dataset.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
