Deep Image Segmentation for Breast Keypoint Detection ^†

Abstract

The main aim of breast cancer conservative treatment is the optimisation of the aesthetic outcome and, implicitly, women’s quality of life, without jeopardising local cancer control and overall survival. Moreover, there has been an effort to try to define an optimal tool capable of performing the aesthetic evaluation of breast cancer conservative treatment outcomes. Recently, a deep learning algorithm that integrates the learning of keypoints’ probability maps in the loss function as a regularisation term for the robust learning of the keypoint localisation has been proposed. However, it achieves the best results when used in cooperation with a shortest-path algorithm that models images as graphs. In this work, we analysed a novel algorithm based on the interaction of deep image segmentation and deep keypoint detection models capable of improving both state-of-the-art performance and execution-time on the breast keypoint detection task.

1. Introduction

Breast cancer is a highly mutable and rapidly evolving disease, however, thanks to the generalised use of breast cancer screening and better treatments, approximately 90% of the cases can be cured [1,2]. Therefore, it is now possible to employ breast cancer conservative treatment (BCCT) approaches, which only require the removal of cancerous tissue with a rim of healthy tissue, instead of radical mastectomy-based approaches, which require the removal of the entire breast and posterior breast reconstruction [3]. In both cases, it is possible to obtain good cosmetic results, and consequently, improve patients’ quality of life. The objective assessment of the cosmetic results, which acts as a reliable proxy of the quality of the treatment and valuable input to the improvement of current techniques, is performed through the analysis of digital photographs, from which, several features are extracted and used to train a classification algorithm [4]. To facilitate the extraction of such features, the annotation of several breast keypoints is required. Several semi-automatic methods to perform this keypoint annotation are already available, however, they still need the input from the user and are time-consuming and computationally demanding. Recently, Silva et al. showed that deep learning algorithms may be part of the answer. They introduced a deep algorithm based deep neural networks (DNN) which receives an image as input and returns the coordinates of the breast keypoints as output [5], which are then given to a shortest-path algorithm that models images as graphs to refine breast keypoint localisation. Although this increased the performance of the task of breast keypoint annotation, this is still computationally complex. To overcome this issue, we proposed a novel deep keypoint detection algorithm, which combines the approach by Silva et al. together with deep image segmentation model that refines breast keypoint localisation in lesser time, and with improved precision [6].

2. Materials and Methods

Our approach combines the DNN proposed by Silva et al. and a deep image segmentation model, U-Net++ [7], in a pipeline (see Figure 1). The intuition behind this approach is that it is easier to detect breast contours if one is capable to detect breasts first. We started by the training of both the U-Net++ and the DNN by Silva et al. in the breast detection (i.e., generate breast masks) and in the keypoint detection tasks, respectively. The complete pipeline works as follows: the image is given as input to U-Net++, which returns a breast segmentation mask; from this mask, contours are extracted using the marching squares algorithm [8]; the image is given as input to the DNN by Silva et al., which returns a set of breast keypoints; the refined localisation of breast keypoints is obtained with the projection of this set onto the breast segmentation mask contours through the minimisation of the Euclidean Distance of a given breast keypoint and a given contour (processing step). We also trained both the DNN and the complete keypoint method proposed by Silva et al. for comparison purposes. All the experiments were performed taking into account 5-fold cross-validation. In addition to the study of algorithms’ performance, a study on the algorithms’ execution time, in seconds, was also done, regarding the interest in the deployment of these algorithms into a web-application for both the research and medical communities.

Figure 1. Proposed deep image segmentation keypoint detection method [6].

3. Results

Table 1 presents the average error distance (measured in pixels) and the average execution time (measured in seconds) of each model inference on the test set. Figure 2 shows a visual example of the obtained results.

Table 1. Average error distance for endpoints, breast contours and nipples, measured in pixels and average execution time of the models’ inferences. Best results are highlighted in bold. Note: STD stands for standard deviation and Max stands for maximum error.

Model	Endpoints			Breast Contours			Nipples			Execution Time (s)
	Mean	STD	Max	Mean	STD	Max	Mean	STD	Max
Silva et al. Keypoint Detection DNN	40	33	182	21	8	72	70	39	218	150
Silva et al. Keypoint Detection Method	40	33	182	13	14	104	70	39	218	1704
Proposed Keypoint Detection Method	38	34	195	11	5	34	70	39	218	280

Figure 2. Example of the results obtained with the proposed deep image segmentation method. The first image is the input photograph and the second image is the U-Net++ predicted mask with the detected breast keypoints (after the processing step).

4. Discussion

Our keypoint detection method surpassed both the DNN-based keypoint detection and the keypoint detection method from Silva et al. in the endpoints and breast contours detection tasks, which were, to our knowledge, the state-of-the-art breast keypoint detection algorithms. Besides, this novel algorithm achieves lower values of standard deviation and maximum error, which suggests more consistency when compared with the other two. Regarding the study of performance, our keypoint detection method presents the best balance between time-efficiency and accuracy, being the most accurate model, with a time efficiency comparable to the most time-efficient method.

5. Conclusions

In this work, we proposed keypoint detection method that combines a deep keypoint detection and deep image segmentation models, and capable of achieving a good balance between both performance and execution-time. Further studies should be focused on the development of a fully end-to-end deep keypoint detection model, trained with a multi-term loss function, and on the deployment of these breast keypoint detection algorithms into a fully-functional web-application for both the research and medical communities.