*Article* **Heuristic Attention Representation Learning for Self-Supervised Pretraining**

**Van Nhiem Tran 1,2, Shen-Hsuan Liu 1,2, Yung-Hui Li 2,\* and Jia-Ching Wang 1**

> 1 Department of Computer Science and Information Engineering, National Central University, Taoyuan 3200, Taiwan; tvnhiemhmus@g.ncu.edu.tw (V.N.T.); 109522071@cc.ncu.edu.tw (S.-H.L.); jcw@csie.ncu.edu.tw (J.-C.W.)

2 AI Research Center, Hon Hai Research Institute, Taipei 114699, Taiwan

**\*** Correspondence: yunghui.li@foxconn.com; Tel.: +886-2-2268-3466

**Abstract:** Recently, self-supervised learning methods have been shown to be very powerful and efficient for yielding robust representation learning by maximizing the similarity across different augmented views in embedding vector space. However, the main challenge is generating different views with random cropping; the semantic feature might exist differently across different views leading to inappropriately maximizing similarity objective. We tackle this problem by introducing **H**euristic **A**ttention **R**epresentation **L**earning (HARL). This self-supervised framework relies on the joint embedding architecture in which the two neural networks are trained to produce similar embedding for different augmented views of the same image. HARL framework adopts prior visual object-level attention by generating a heuristic mask proposal for each training image and maximizes the abstract object-level embedding on vector space instead of whole image representation from previous works. As a result, HARL extracts the quality semantic representation from each training sample and outperforms **existing** self-supervised baselines on several downstream tasks. In addition, we provide efficient techniques based on conventional computer vision and deep learning methods for generating heuristic mask proposals on natural image datasets. Our HARL achieves +1.3% advancement in the ImageNet semi-supervised learning benchmark and +0.9% improvement in AP50 of the COCO object detection task over the previous state-of-the-art method BYOL. Our code implementation is available for both TensorFlow and PyTorch frameworks.

**Keywords:** heuristic attention; perceptual grouping; self-supervised learning; visual representation learning; deep learning; computer vision
