Cascaded Searching Reinforcement Learning Agent for Proposal-Free Weakly-Supervised Phrase Comprehension

Abstract

Phrase comprehension (PC) aims to locate a specific object in an image according to a given linguistic query. The existing PC methods work in either a fully supervised or proposal-based weakly supervised manner, which rely explicitly or implicitly on expensive region annotations. In order to completely remove the dependence on the supervised region information, this paper proposes to address PC in a proposal-free weakly supervised training paradigm. To this end, we developed a novel cascaded searching reinforcement learning agent (CSRLA). Concretely, we first leveraged a visual language pre-trained model to generate a visual–textual cross-modal attention heatmap. Accordingly, a coarse salient initial region of the referential target was located. Then, we formulated the visual object grounding as a Markov decision process (MDP) in a reinforcement learning framework, where an agent was trained to iteratively search for the target’s complete region from the salient local region. Additionally, we developed a novel confidence discrimination reward function (ConDis_R) to constrain the model to search for a complete and exclusive object region. The experimental results on three benchmark datasets of Refcoco, Refcoco+, and Refcocog demonstrated the effectiveness of our proposed method.

1. Introduction

Phrase comprehension (PC) is a fundamental task in the multi-modal learning community and serves as the basis for many downstream tasks, including image captioning [1,2], visual question answering [3,4], etc. The purpose of PC is to locate a specific entity in an image according to a given linguistic query. It requires a comprehensive understanding of both vision and language, and there has been a surge of research interest in computer vision (CV) and natural language processing (NLP). Most traditional PC methods follow a fully supervised training approach, where the mappings between the entity regions and the query texts during the training stage are available. However, the annotated data are noticeably expensive due to the labor-intensive labeling process in practical applications and noise may be introduced when labeling the training data.

In recent years, some efforts have tackled PC in a proposal-based weakly supervised training paradigm, to alleviate the strong dependence on annotations, where region–text pairs are no longer required, but the entity regions still need to be provided in advance. The predominant methods formulate weakly supervised PC as an entity region retrieval problem, which is based on a modality reconstruction mechanism [5,6,7], where the highest-score bounding box among all the candidate regions is selected as the referential target. However, weakly supervised training methods do not completely remove the dependence on expensive grounding annotations, due to the given entity regions. Therefore, it is of great significance to explore specific solutions for PC in a proposal-free weakly supervised training paradigm. To this end, this paper focuses on exploring the proposal-free weakly supervised phrase comprehension task to remove the constraint of expensive grounding annotations completely, where any explicit or implicit supervised region information does not need to be available during the training stage. To distinguish the traditional fully supervised, proposal-based weakly supervised, and the proposal-free weakly supervised PC methods, we denote them as FPC, BWPC, and FWPC, respectively. Figure 1 depicts the differences among them.

Take FPC for instance (Figure 1a), the instance-level annotations of the queries “blurry man”, “clear man”, and the corresponding bounding-boxes of these two instances in the image are provided when training an FPC model. For BWPC in Figure 1b, although the mappings between the specific entity regions and the corresponding query texts are unavailable in the training stage, those entity regions need to be given in advance. In Figure 1c for the proposal-free weakly supervised training paradigm, explicit or implicit supervised region information is unavailable during the training phase.

On the other hand, reinforcement learning (RL) has seen a rapid development in recent years, including practical applications in the fields of visual grounding (VG) [9] and object detection (OD) [10]. As a pioneering RL-based work in VG, Sun et al. [9] first proposed an iterative shrinking mechanism to localize the target, where the shrinking direction is decided by an RL agent. In the OD field, Liu et al. [10] proposed a pay attention to them (PAT) general object detection framework, which first applies a CNN regression detector to the entire input image, and then conducts refinement via RL in a cascaded way. Essentially, differently from supervised learning, RL relies on an unsupervised learning approach, where an agent gains and maximizes the rewards in a constant trial-and-error interaction with the environment. Consequently, RL continuously constrains the optimization direction of the model. This inherent characteristic and the successful application of RL inspired us to approach FWPC in a sequential-decision-making form. The core insight of our idea is illustrated in Figure 2. Thus, we conceive a strategy that starts with a part of an object that has been located in a certain way and then iteratively searches for the complete object.

Based on the above observations and conception, in this paper we propose a cascaded searching reinforcement learning agent, namely CSRLA, to address FWPC. We first leverage the visual language pre-trained model ALBEF [11] and calculate the cross-attention weights to locate the most salient region according to the alignment restricted by a threshold of the image and the query. Then, we formulate PC as a Markov decision process (MDP), where the target grounding is decomposed into sequential cascaded searching actions started from the salient region within an RL framework. An actor-net and a critic-net are developed for the agent to conduct an action and assess its value, respectively. Additionally, we design a novel confidence discrimination reward function (ConDis_R) to constrain the agent to search for a complete and exclusive target entity.

To summarize, the main contributions are as follows:

• This paper makes the first attempt to address PC in an proposal-free weakly supervised paradigm.
• We propose a cascaded searching reinforcement learning agent (CSRLA), which formulates PC as a Markov decision process (MDP) within an RL framework, where the target grounding is decomposed into RL-based sequential cascaded searching actions, to perform a cascaded search for the complete referent target from the initial salient region.
• We design a novel confidence discrimination reward function (ConDis_R) to constrain the agent to search for a complete and exclusive target.
• Extensive experiments on three benchmark datasets demonstrated the effectiveness of our proposed method.

2. Related Work

PC techniques have made great progress in recent years. The existing methods follow two training paradigms: FPC and BWPC. On the other hand, RL has been widely employed to address CV tasks, and remarkable achievements have been made. This section introduces some representative efforts in these directions.

2.1. Fully-Supervised Phrase Comprehension

The patch–query pairs are available during the training stage for FPC. Zhao et al. [12] propose a encoder–decoder transformer-based architecture Word2Pix, which employs a word-to-pixel attention mechanism to learn textual to visual features for visual grounding. Deng et al. [13] presented a transformer-based framework TransVG, which establishes cross-modal correspondence using simply stacked transformer layers. Hong et al. [14] proposed an RVG-TREE model, which could automatically generate a binary tree to parse the language and implement grounding with the tree in a bottom-up manner. Hu et al. [15] proposed a CMNs model that analyzes referring texts into components, and recognizes entities and relationships mentioned in the query for object grounding in an end-to-end manner. Liao et al. [16] proposed an RCCF method that reformulates the phrase comprehension as a correlation filtering process to achieve a faster inference speed. Liu et al. [17] developed an NMTREE network that is capable of efficiently parsing the query sentence in a tree structure, where each node calculates the visual attention according to its linguistic features. Deng et al. [18] explored neat yet effective purely transformer-based frameworks for visual grounding. Su et al. [19] proposed an active perception visual grounding framework based on language adaptive weights (VG-LAW) to effectively alleviate the mismatches of redundant and missing features.Yang et al. [20] developed a margin-based loss for tuning joint vision language models, the gradient-based explanations are consistent with human region-level grounding annotations. Li et al. [21] presented a end-to-end transformer-based grounding approach that focuses on capturing the cross-modality correlations between text and visual patches. Unfortunately, FPC relies heavily on expensive annotations of patch–query pairs.

It is worth noting that certain single-stage, proposal-free PC methods have been introduced [9,22,23,24], aiming to enhance the efficiency of target grounding by bypassing the need for proposal generation. However, a fundamental distinction arises when comparing these approaches with our proposed method: while these methods adhere to fully supervised training paradigms, our approach embraces an proposal-free weakly supervised training paradigm.

2.2. Weakly Supervised Phrase Comprehension

In this setting, the entity regions are provided in advance, but the specific patch–query mapping is unavailable. As a pioneering work, Rohrbach et al. [5] proposed a method that learns to locate objects by reconstructing a given text via an attention mechanism. KAC-Net [6] explores the consistency contained in both visual and textual modalities by reconstructing input phrase and proposal information. Liu et al. [7] introduced an end-to-end adaptive reconstruction network (ARN) model that captures the relationship between the image region proposals and the texts in an adaptive grounding and reconstruction manner. Niu et al. [25] proposed a variational Bayesian method (variational context) to deal with complex context modeling in PC. Zhang et al. [26] proposed a novel counterfactual contrastive learning (CCL) model to implement sufficient contrastive learning between the positive and negative results for BWPC. Sun et al. [27] proposed a cycle-free pipeline to mitigate the target deviation problem, where a region describer network is developed to predict the textual description for each entity region. Liu et al. [28] introduced an entity-enhanced adaptive reconstruction network (EARN) to address BWPC via entity enhancement, adaptive grounding, and collaborative reconstruction. Despite the patches and queries being disconnected in this paradigm, the reliance on expensive grounding annotations is not completely eliminated in BWPC.

2.3. Vision-and-Language Pre-Training

Recently, vision-and-language pre-training (VLP) has shown great ability in multi-modal learning by pre-training the model on large-scale vision-language datasets and fine-tuning on specific multi-modal learning tasks, including visual question answering, visual commonsense reasoning, and cross-modal retrieval, etc. ALIGN [29] utilized a noisy dataset of more than 1B image–text pairs, obtained without expensive filtering or post-processing steps from the Conceptual Captions dataset [30], to pre-train a visual-language model. The recent CLIP model [31], pre-trained on 400 million image–test pairs collected from the web, achieved impressive performance by performing zero-shot transfer on downstream tasks. ALBEF [11] proposed a momentum distillation (MoD), to deal with noisy image–text pairs drawn from the web, and introduced image–text contrastive (ITC) loss for the cross-modal alignment.

2.4. Reinforcement Learning in Computer Vision

RL is a machine learning paradigm that learns a sequence of actions in a continuous trial-and-error interaction with the environment in order to obtain and maximize rewards. Since AlphaGo [32] came to prominence, RL has attracted much attention.

The past decade has witnessed a surge of the practical applications of RL in CV, due to its powerful decision-making ability. Generally, RL implements an MDP to model the CV tasks in an appropriate manner; the representative applications include object tracking [33,34], object detection [35,36], image segmentation [37,38], and video analysis [39,40], etc.

In addition, since this paper focuses on PC in the multi-modal learning community, we mention some representative works of RL in this domain. In the multi-modal learning community, Ren et al. [41] introduced a decision-making framework for the image captioning task, where a policy and a value network are employed to collaboratively generate image captions. SCST [42] also dealt with the same task by proposing a self-critical sequence training algorithm that utilizes the output of its own test-time inference rather than estimating a baseline to normalize the rewards to reduce training variances. N2NMNs [43] addressed the visual question answering task by leveraging RL to jointly optimize the neural modules and the layout policy. Lu et al. [44] introduced a novel visual question answering task called textual distractors generation, which was also modeled by RL. In cross-modal retrieval tasks, Cai et al. [45] proposed a novel retrieval framework that conducts the interactive process in an ask-and-confirm manner, where an RL policy is utilized to search for discriminative objects. Yan et al. [46] proposed an RL-based attention mechanism to make the attention module optimize directly towards an expected goal.

It is worth mentioning that Sun et al. [9] made the first attempt at RL in FPC, they modeled visual grounding in an RL framework through an iterative image-cropping manner. However, different from the task setting of [9], this paper deals with PC in a proposal-free weakly supervised training paradigm via an RL-based target searching policy.

3. Methodology

3.1. Overview

We propose a cascaded searching reinforcement learning agent (CSRLA) to address FWPC. We first leverage the visual language pre-trained ALBEF [11] to locate the entity salient region, then we formulate FWPC as a Markov decision process (MDP) in an RL framework to cascade search for the complete referent target from the initial local region, as illustrated in Figure 3.

3.2. Visual Encoder

We employ an off-the-shelf visual-transformer, i.e., swin-transformer [47], to encode each input image patch, since its characteristic of multi-scale receptive field fusion is suitable for our variable-sized target localization task. The official released version “Swin-S” pre-trained on ImageNet-1K [48] and fine-tuned on MS-COCO [49] is adopted. We employ the features of the feature pyramid network (FPN) of the neck block, since they integrate all levels of visual relationships.

3.3. Textual Encoder

For each query text, we leverage the vanilla language processing model BERT [50] to extract the word-level representations. BERT is a transformer-based pre-trained linguistic model, in which a bi-directional self-attention mechanism is capable of capturing the contextual semantics of sentences. The linguistic query is encoded into a D-dimensional vector, which is denoted as $T=\left\{t_1,···,t_n\right\}\in {\mathbb{R}}^{n\times D}$.

3.4. Initially Salient Region Localization

We leverage the pre-trained ALBEF to locate the coarse salient regions of the referential instances in the form of initial boxes. ALBEF adopts a cross-modal transformer to mine the inter-modal correlations among vision–language elements via its cross-attention mechanism.

Grad-CAM [51] provides an operational neural network visualization technique that can highlight specific-instance regions through class activation mapping. We utilize this technique to activate and highlight a coarse region of images where the visual and the linguistic modalities are strongly correlated to each other. Concretely, we localize the cross-modal interrelated salient region via the cross-attention map between the visual and the linguistic modalities. To do this, the Grad-CAM [51] of CSRLA is calculated as follows:

$$H_{CAAM}=max\left(0,\sum _k{\alpha }_k{A}^k\right),$$

(1)

where $H_{CAAM}\in {\mathbb{R}}^{w\times h}$ refers to cross-attention activation mapping (CAAM) in the 8th layer of the cross-modal transformer, ∑ is the global average pooling operation, and ${\alpha }_k$ stands for the cross-attention importance weights. To obtain ${\alpha }_k$, the gradient of the attention map A, i.e., $\frac{\partial s}{\partial A_{ji}^k}$ is first calculated:

$${\alpha }_k=\frac{1}{Z}\sum _i\sum _j\frac{\partial s}{\partial A_{ji}^k},$$

(2)

where s is the similarity between the cross modalities, and $\frac{1}{Z}\sum \sum$ denotes the global average pooling operation.

The $H_{CAAM}$ reflects how closely the different modalities are related. In order to generate the initial region of a referential instance, we set a threshold for CAAM weights, and the region whose weight is greater than this threshold is located in the form of a box:

$$B_p\left((x_0,y_0),(x_1,y_1)\right)=F\left(H_{CAAM},\delta \right),$$

(3)

where $B_p$ stands for the generated initial region boxes, ($x_0$, $y_0$) and ($x_1$, $y_1$) are the top-left and bottom-right coordinates of the boxes, respectively, $\delta$ denotes the weight threshold, and F is a function that checks the salient heatmap contours and generates boxes according to the threshold.

3.5. Formulation of Reinforcement Learning

Reinforcement learning is a machine learning paradigm that learns a sequence of decisions in MDPs by gaining and maximizing the rewards in the constant trial-and-error interaction with the environment. This section presents our proposed RL-based searching policy for searching a complete target region from the aforementioned initial salient patch. We describe the modeling process in detail, in terms of state, action, reward, and loss function.

3.5.1. State

State refers to the environment with which an agent interacts, it should cover all the scenarios that might be observed during the agent learning process. Specifically, in our CSRLA, for an iteration i, the state $s_i$ consists of four components: sentence-level linguistic representation is generated by the textual encoder, i.e., BERT [50]. We take a global average pooling for each token representation of the output, and finally obtain a 768-dimensional semantic feature ${\mathit{s}}_{t_i}$ of the query text. Similarly, patch-level visual representation is generated by the visual encoder, i.e., Swin-transformer [47]. We employ the features of the feature pyramid network (FPN), since they integrate all levels of visual relationships, then we flatten the [256 × 6 × 6] embedding of the neck block into a 9216-dimensional visual representation ${\mathit{s}}_{v_i}$. Furthermore, we record the agent’s history actions vector to guide future decisions, denoted as ${\mathit{s}}_{a_i}$. For the spatial-vector ${\mathit{s}}_{s_i}$ of a current patch i, we argue that this is the key information for target localization, since it is important for an agent to know its spatial location in the image environment in an instance iterative searching process. Following [9], we encode using a 5-dimensional vector, i.e., [$\frac{x_0}{W}$, $\frac{y_0}{H}$, $\frac{x_1}{W}$, $\frac{y_1}{H}$, $\frac{w·h}{W·H}$ ], where [$x_0$, $y_0$] and [$x_1$, $y_1$] represent the top-left and bottom-right coordinates of the i-th patch in the iterations, w and h stand for the width and height of the patch, and W and H denote the width and height of the image.

Finally, these four components are stacked together to obtain a cross-modal joint state (CJS), as follows:

$$S_i={\mathit{s}}_{t_i}\oplus {\mathit{s}}_{v_i}\oplus {\mathit{s}}_{a_i}\oplus {\mathit{s}}_{s_i},$$

(4)

where ⊕ denotes the concatenation operation. CJS allows the agent to continuously observe the visual environment, especially the semantic-rich linguistic environment and cross-modal alignment environment, which provides the basis for the MDPs.

3.5.2. Action

In RL, an agent learns to perform one action at a time after observing the current environment. Specifically, in CSRLA, the agent conducts a cascade searching strategy of patch expansion from a local region to a complete entity. Therefore, we formulate the action space of CSRLA as a set of five actions, denoted as [$a_1$, $a_2$, $a_3$, $a_4$, $a_5$], where $a_1$, $a_2$, $a_3$, $a_4$ indicate that the i-th patch expands in the directions left, right, up, and down, respectively. Note that $a_5$ is a terminator, which represents that the current patch is not expanded in this action. The agent performs the actions to complete the search of the reference object. Figure 4 shows the presented action space.

At each state transition step t of an episode, the agent first performs an action, then the searched area expands in the corresponding direction, which indicates that the state has been iteratively transmitted to the next MDP time-step. The new region coordinates (($x_0$, $y_0$), ($x_1$, $y_1$)) that have been moved to are as follows:

$$\left\{\begin{array}{cc}{x}_0=x_0-\lambda (x_1-x_0),\hfill & action=a_1\hfill \\ x_1=x_1+\lambda (x_1-x_0),\hfill & action=a_2\hfill \\ y_0=y_0-\lambda (y_1-y_0),\hfill & action=a_3\hfill \\ y_1=y_1+\lambda (y_1-y_0),\hfill & action=a_4\hfill \\ \varphi ,\hfill & action=a_5,\hfill \end{array}\right.$$

(5)

where the hyperparameter $\lambda$ denotes the region expansion ratio. In our setting, this is assigned in a discrete decreasing manner:

$$\lambda =\left\{\begin{array}{cc}0.2,\hfill & steps\le 5\hfill \\ 0.1,\hfill & steps\ge 8\hfill \\ 0.15,\hfill & otherwise,\hfill \end{array}\right.$$

(6)

where $steps$ refers to the number of actions that have been performed in the current episode.

3.5.3. Reward

When an agent observes an environment, performs an action, and obtains a reward from the environment, CSRLA completes a state transition process (i.e., a RL step). In essence, rewards determine the direction of an agent’s learning, i.e., the optimization direction of the network.

Specifically, CSRLA aims to search a more complete region of the object from the salient patch. We consider that a well-trained classification network should be sensitive to the local-region and complete-region of visual objects. Generally, the category confidence of the complete-region is higher than that of the local-region. In our task scenario, for example, when an agent observes an image patch of black animal hair, it cannot clearly judge whether this instance category is “cat” or “dog” or other, since each category confidence level is not high due to the average distribution. However, as the patch expands from partial to complete, the agent becomes more confident of which animal category is observed.

To this end, we designed a reward function based on a neat yet effective multi-label classification network (MLC), which consists of a Resnet backbone and a multi-label classification head. The novel confidence discrimination reward function (ConDis_R) is developed as

$$R(S_{t-1},S_t)={\eta }_t-{\eta }_{t-1}-\Delta \left(t\right),$$

(7)

where $\Delta \left(t\right)$ is a penalty term to constrain the search efficiency of the agent, $\eta$ is the category confidence of each time-step, which is formulated as follows:

$${\eta }_t=SoftExp\left(argmax\left(SoftMax\left(p_t\right)\right)\right),$$

(8)

where $SoftExp$ and $SoftMax$ are two activation functions, $p_t$ is the category confidence score of the current state $S_t$. $SoftExp$ is a confidence sensitivity perception function, which maps the category confidence score via an exponential function, making image patches have different confidence sensitivities at different expansion stages. The $SoftMax$ is a probabilistic constraint that penalizes the search for instances other than the target. The calculation formula of the $SoftExp$ activation function is as follows:

$$f\left(x\right)=\left\{\begin{array}{cc}-\frac{1}{\alpha }\times ln\left(1-\alpha \left(x+\alpha \right)\right),\hfill & \alpha <0\hfill \\ x,\hfill & \alpha =0\hfill \\ \frac{1}{\alpha }\times e^{\alpha x-1},\hfill & \alpha >0.\hfill \end{array}\right.$$

(9)

Transitions: After an action is performed, the current state is transformed into the next one. In our approach, the MDP transitions are deterministic; that is, a state is transferred to a specific state after an action has been performed.The state transfer formula is as follows:

$$\begin{array}{ccc}\hfill \mathcal{T}(s,a)& =& \mathcal{T}(({\mathit{s}}_{t_i},{\mathit{s}}_{v_i},{\mathit{s}}_{a_i},{\mathit{s}}_{s_i}),a)\hfill \\ & =& \mathcal{T}({\mathit{s}}_{t_{i+1}},{\mathit{s}}_{v_{i+1}},{\mathit{s}}_{a_{i+1}},{\mathit{s}}_{s_{i+1}})\hfill \\ & =& s^{\prime },\hfill \end{array}$$

(10)

where $\mathcal{T}$ is the RL transition, $\mathcal{T}(s,a)$ represents that the agent’s state transits to $s^{\prime }$ by executing action a while in s.

3.5.4. Optimization

Our CSRLA was optimized in the advantage-actor-critic framework [52], which is one of the popular algorithms in the policy-gradient family. It consists of an actor-net and a critic-net; the former is used to select and execute an action from the action space in the current patch environment $S_i$, while the latter is applied to evaluate the executed action value, which serves as the basis for the subsequent decisions.

The objective function $J_a\left(\theta \right)$ of actor-net is defined as:

$${\nabla }_{\theta }{J}_a\left(\theta \right)=A\left(s_t,a_t\right){\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right),$$

(11)

where $a_t$ and $s_t$ are the action and the state at time step t, ${\pi }_{\theta }$ denotes the policy according to the current action, and $A\left(s_t,a_t\right)$ is the advantage function that takes the mathematical expectation of the action value to reduce its variance. This is defined as

$$A\left(s_t,a_t\right)={\mathbb{E}}_{{\pi }_{\theta }}^n\left[r_t^n+V^{\pi }\left(s_{t+1}^n\right)-V^{\pi }\left(s_t^n\right)\right],$$

(12)

where $r_t^n$ refers to the immediate reward for performing the current action, and $V^{\pi }(·)$ represents the state value generated by the critic-net.

Accordingly, the objective function $J_c\left(\omega \right)$ of the critic-net is defined as

$${\nabla }_{\omega }{J}_c\left(\omega \right)={\nabla }_{\omega }{\left(A_{\omega }\left(s_t,a_t\right)\right)}^2.$$

(13)

The pseudocode for the proposed model is shown in Algorithm 1.

Algorithm 1 Advantage-Actor-Critic for CSRLA

Input: Image I and referring text T 1: Generate Grad-CAM by ALBEF 2: Generate salient local region 3: Initialize action-value parameters $\theta$ 4: Initialize state-value parameters $\omega$ 5: Initialize S (first state of an episode) 6: for t = 1, 2, ⋯ do 7:       Take $a_t$ according to policy ${\pi }_{\theta }\left(a_t|s_t\right)$ 8:       Execute $a_t$ (patch expansion), observe reward $r_t$, next state $s_{t+1}$ 9:       $r_t={\eta }_t-{\eta }_{t-1}-\Delta \left(t\right)$ 10:     R←$r_t$ + $\gamma R$ 11:     $A\left(s_t,a_t\right)\leftarrow {\mathbb{E}}_{{\pi }_{\theta }}^n\left[r_t^n+V^{\pi }\left(s_{t+1}^n\right)-V^{\pi }\left(s_t^n\right)\right]$ 12:     Update $\theta \leftarrow \theta +A\left(s_t,a_t\right){\nabla }_{\theta }log{\pi }_{\theta }\left(a_t|s_t\right)$ 13:     Update $\omega \leftarrow \omega +{\nabla }_{\omega }{\left(A_{\omega }\left(s_t,a_t\right)\right)}^2$ 14:     t←$t+1$ 15: end for

3.5.5. Discussion

In this paper, we propose the CSRLA model to address phrase comprehension in a proposal-free weakly supervised training paradigm. Our approach does not rely on region supervised information for the RefCOCO, RefCOCO+, and RefCOCOg datasets. Despite the encouraging performance, our proposed CSRLA still suffers from one limitation. Although CSRLA no longer relies on regional supervision information, strictly speaking, the referential annotated texts provide supervision information to a certain extent. Given this consideration, we define the proposed supervisory paradigm as a proposal-free weakly supervised paradigm rather than an unsupervised paradigm. Therefore, exploring a thoroughly unsupervised training method that does not rely on manual annotation for both region and text information is of great significance.

4. Experiments

4.1. Experimental Setup

Datasets: We evaluated our approach on three benchmark datasets, i.e., RefCOCO [8], RefCOCO+ [8], and RefCOCOg [53]. RefCOCO includes 50,000 target instances and 142,209 referring texts. It is split into training, validation, test A, and test B sets, which have 120,624, 10,834, 5657, and 5095 query–patch pairs, respectively. RefCOCO+ contains 49,856 referents and 141,546 queries. In the same way, RefCOCO+ is split into training, validation, test A, and test B sets, which have 120,191, 10,758, 5726, and 4889 query–patch pairs, respectively. RefCOCOg consists of 95,010 longer query sentences containing appearances and locations for 49,822 entities and is an auxiliary dataset for PC. It is split into training and validation with 21,149 and 4650 images, respectively. It does not have an open test split.

Evaluation Metrics: The metric used to evaluate CSRLA was the popular intersection-over-union (IoU) between the searched bounding box and the ground-truth, where the correct determination threshold was 0.5.

Implementation Details: For the image regions, the local fragment features were encoded by a Swin-transformer [47], whose dimensionality was 9216. As for the query sentence, word-level tokens were encoded by the vanilla BERT [50] into 768-dim vectors. The activation threshold of the cross-modal attention salient region was 0.15. In the RL searching stage, the list length of the history RL action ${\mathit{s}}_{a_i}$ was set to 5. The maximum number of action iterations was 9. The initial expansion ratio was 0.2, and the searching stride decreased by 0.05 at specific time steps. The learning rates of the actor and the critic networks were both set to $1\times {10}^{-6}$. The reward discount factor $\gamma$ was 0.99. We trained the MLC network for 15 epochs on MS-COCO [49] and trained the actor and the critic networks for 500 K episodes. All experiments were implemented on the Pytorch platform with 3 GeForce GTX 1080 GPUs.

4.2. Experimental Results

Since this paper was the first to address PC in an proposal-free weakly supervised paradigm, there are no previously published FWPC results.Thus, we had no choice but to choose the state-of-the-art BWPC methods, including VC [25], ARN [7], IGN [26], DTMR [54], C-FREE [27] and EARN [28] for comparison. It should be noted that, although VC [25] reports proposal-free weakly supervised experiment results on RefCOCO [8], RefCOCO+ [8], and RefCOCOg [53], its experimental setting still belonged to the BWPC category in the task definition of this paper, since entity regions were required.

Table 1. Grounding results (IoU > 0.5, %) on the benchmark dataset RefCOCO+ compared with the state-of-the-art BWPC methods.

Methods	Published on	Supervised Manners	Settings	RefCOCO+			AVG
				Val	Test A	Test B
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$w/oreg$ − $det$	-	19.74	24.05	21.90
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$det$	-	25.79	25.54	25.67
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$w/o\alpha$ − $det$	-	34.68	28.10	31.39
VC [25]	TPAMI-19	proposal-based weakly-supervised	$w/oreg$	-	18.79	24.14	21.47
VC [25]	TPAMI-19	proposal-based weakly-supervised	-	-	23.24	24.91	24.08
VC [25]	TPAMI-19	proposal-based weakly-supervised	$w/o\alpha$	-	34.60	31.58	33.09
ARN [7]	ICCV-19	proposal-based weakly-supervised	$L_{adp}+L_{att}$	33.53	36.40	29.23	33.05
ARN (det) [7]	ICCV-19	proposal-based weakly-supervised	$L_{lan}+L_{adp}$ − $det$	31.73	34.23	29.35	31.77
ARN (det) [7]	ICCV-19	proposal-based weakly-supervised	$det$	32.78	34.35	32.13	33.09
ARN [7]	ICCV-19	proposal-based weakly-supervised	$L_{lan}+L_{adp}+L_{att}$	34.53	36.01	33.75	34.76
IGN [26]	NIPS-20	proposal-based weakly-supervised	$Base$	31.13	34.44	29.59	31.72
IGN [26]	NIPS-20	proposal-based weakly-supervised	$CCL$	34.29	36.91	33.56	34.92
C-FREE [27]	TMM-21	proposal-based weakly-supervised	-	39.20	39.63	37.59	38.81
EARN [28]	TPAMI-23	proposal-based weakly-supervised	$L_{adp}$	35.50	37.39	33.65	35.51
EARN [28]	TPAMI-23	proposal-based weakly-supervised	$w/o{L}_{adp}$	35.31	33.46	37.27	35.35
EARN [28]	TPAMI-23	proposal-based weakly-supervised	$L_{lan}+L_{adp}+L_{att}$	37.54	37.58	37.92	37.68
CSRLA (Ours)	-	proposal-free weakly-supervised	-	32.37	32.56	31.26	32.06

,

Table 2. Grounding results (IoU > 0.5, %) on the benchmark dataset RefCOCO compared with the state-of-the-art BWPC methods.

Methods	Published on	Supervised Manners	Settings	RefCOCO			AVG
				Val	Test A	Test B
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$w/oreg$ − $det$	-	17.14	22.30	19.72
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$det$	-	20.91	21.77	21.34
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$w/o\alpha$ − $det$	-	32.68	27.22	29.95
VC [25]	TPAMI-19	proposal-based weakly-supervised	$w/oreg$	-	13.59	21.65	17.62
VC [25]	TPAMI-19	proposal-based weakly-supervised	-	-	17.34	20.98	19.16
VC [25]	TPAMI-19	proposal-based weakly-supervised	$w/o\alpha$	-	33.29	30.13	31.71
ARN [7]	ICCV-19	proposal-based weakly-supervised	$L_{adp}+L_{att}$	33.07	36.43	29.09	32.86
ARN (det) [7]	ICCV-19	proposal-based weakly-supervised	$L_{lan}+L_{adp}$ − $det$	31.58	35.50	28.32	31.80
ARN (det) [7]	ICCV-19	proposal-based weakly-supervised	$det$	32.17	35.35	30.28	32.60
ARN [7]	ICCV-19	proposal-based weakly-supervised	$L_{lan}+L_{adp}+L_{att}$	34.26	36.01	33.07	34.45
IGN [26]	NIPS-20	proposal-based weakly-supervised	$Base$	31.05	34.39	28.16	31.20
IGN [26]	NIPS-20	proposal-based weakly-supervised	$CCL$	34.78	37.64	32.59	35.00
DTMR [54]	TPAMI-21	proposal-based weakly-supervised	-	39.21	41.14	37.72	39.36
EARN [28]	TPAMI-23	proposal-based weakly-supervised	$L_{adp}$	35.31	37.07	32.66	35.01
EARN [28]	TPAMI-23	proposal-based weakly-supervised	$w/o{L}_{adp}$	34.93	33.76	36.98	35.22
CSRLA (Ours)	-	proposal-free weakly-supervised	-	31.86	32.06	32.19	32.04

and

Table 3. Grounding results (IoU > 0.5, %) on the benchmark dataset RefCOCOg compared with the state-of-the-art BWPC methods.

Methods	Published on	Supervised Manners	Settings	Val	mA
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$w/oreg$ − $det$	28.14	22.27
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$det$	33.66	25.53
VC (det) [25]	TPAMI-19	proposal-based weakly-supervised	$w/o\alpha$ − $det$	29.65	30.47
VC [25]	TPAMI-19	proposal-based weakly-supervised	$w/oreg$	25.14	20.66
VC [25]	TPAMI-19	proposal-based weakly-supervised	-	33.79	24.05
ARN [7]	ICCV-19	proposal-based weakly-supervised	$L_{adp}+L_{att}$	33.19	32.99
ARN (det) [7]	ICCV-19	proposal-based weakly-supervised	$L_{lan}+L_{adp}$ − $det$	32.60	31.90
ARN (det) [7]	ICCV-19	proposal-based weakly-supervised	$det$	33.09	32.88
ARN [7]	ICCV-19	proposal-based weakly-supervised	$L_{lan}+L_{adp}+L_{att}$	34.66	34.61
IGN [26]	NIPS-20	proposal-based weakly-supervised	$Base$	32.17	31.56
IGN [26]	NIPS-20	proposal-based weakly-supervised	$CCL$	34.92	34.96
EARN [28]	TPAMI-23	proposal-based weakly-supervised	$L_{adp}$	38.99	35.80
EARN [28]	TPAMI-23	proposal-based weakly-supervised	$w/o{L}_{adp}$	38.37	35.73
CSRLA (Ours)	-	proposal-free weakly-supervised	-	31.89	32.03

present the quantitative grounding results of CSRLA and the competitors on the three benchmark datasets. Surprisingly, we observed that the grounding accuracy of our proposal-free weakly supervised method was quite close to those of the state-of-the-art weakly supervised methods.

Specifically, on the RefCOCO+ dataset, our method achieved accuracies of 32.37%, 32.56%, and 31.26% on split validation, test A, and test B, respectively, as well as an average accuracy of 32.06% on the three test sets. Encouragingly, CSRLA outperformed most VC [25] settings on testA and testB, as well as for most ARN settings [7] and IGN (base) [26], and outperformed most VC [25] settings on testA. From the average accuracy of the overall test set, the proposed CSRLA outperformed most experimental settings of VC [25], ARN [7], and IGN [26].

On the RefCOCO dataset, CSRLA also achieved a competitive performance, surpassing all experimental setup results of VC [25] on testB and most experimental setup results of VC [25] on testA, as well as most of the experimental results of ARN [7] and IGN [26]. From the average accuracy of the three test sets, the proposed CSRLA achieved a detection accuracy of 32.04%, surpassing VC [25] and performing similarly to ARN [7] and IGN [26].

The experimental results on the RefCOCOg dataset show that the proposed method achieved an accuracy of 31.89%, which was also better than the results of several experimental settings on VC [25]; the average accuracy (mA) of the proposed CSRLA on the three datasets was 32.03%, which exceeded VC [25] and was basically on par with the performance of ARN [7] and IGN [26].

We emphasize that, differently from our proposal-free weakly supervised training approach, all the above comparative methods used weakly supervised settings, all of which relied on the supervised region information. CSRLA approximated the state-of-the-art BWPC methods in performance, which demonstrates its superiority and effectiveness, since it does not take advantage of any explicit or implicit supervised region information.

4.3. Ablation Studies

Impact of various components of the model. We performed a set of experiments on the RefCOCO dataset to study the effectiveness of the different components of CSRLA.

Table 4. Impact of various components of the model (grounding accuracy, IoU > 0.5, %). The best results are marked in bold.

Settings	Val	Test A	Test B	Avg
w/o RL (ALBEF for FWPC)	19.17	15.82	22.65	19.21
MCC	25.20	23.54	27.14	25.34
fixed stride	29.05	28.32	30.17	29.18
CSRLA	32.37	32.56	31.26	32.06

presents the experimental results.

First, we conducted an experiment that only carried out the first stage, to generate the salient-region of the referential target by ALBEF [11] and to verify the impact of RL without the target searching process (w/o RL, ALBEF for FWPC). We observed that the salient initial regions generated by ALBEF had a poor grounding accuracy on 3 splits in the FWPC setting without the RL-based cascading searching process, which were 19.17%, 15.82%, and 22.65% on val, testA, and testB, respectively, and the average accuracy was 19.21%, which was 12.85% lower than that of CSRLA. This illustrates the necessity of an RL-based cascading searching process for FWPC.

Then, we investigated the impact of replacing the multi-label classification network (MLC) of the reward function with a multi-class classification network (MCC). MCC, by contrast, produced a significant performance degradation, whose average accuracy on the three splits was 6.72% lower than that of CSRLA. This performance degradation may have been due to the training dataset’s deviation and the category inconsistency of the MS-COCO dataset [49] and the Imagenet dataset [48].

Last, we explored the performance difference between a decreasing expansion ratio and a fixed one (fixed stride). The decreasing expansion-stride strategy makes sense, as our search mechanism scales from a region to all sides. The actual expanded area increases with the number of iterations. Actually, the strategy of regional fine-tuning is preferable when approaching the target, as we can observe from the experimental results; the average accuracy of fixed stride scheme was 2.86% lower than that of CSRLA, which suggests the effectiveness of the decreasing expansion-stride strategy.

Impact of Cross-Attention Activation Mapping thresholds. One of the key hyperparameters of CSRLA is the cross-attention activation mapping threshold CAAM threshold. A small value may result in a large salient region being generated in the first stage, and vice versa. Both cases would affect the grounding performance.

Table 5. Impact of different cross-attention activation mapping thresholds (grounding accuracy, IoU > 0.5, %). The best results are marked in bold.

CAAM Thresholds	Val	Test A	Test B	Avg
0.11	26.52	28.76	27.71	27.66
0.13	31.64	32.48	30.69	31.60
0.15	32.37	32.56	31.26	32.06
0.17	30.22	29.83	28.20	29.42
0.20	24.14	27.54	25.35	25.68

shows the impact of different CAAM thresholds; we observed a performance increase from the threshold of 0.09 to 0.15, and then a decrease to 0.20. The peak 32.06% was reached at 0.15, thus we chose the threshold value of 0.15.

4.4. Visualization and Analysis

We performed a set of visual experiments to provide a more comprehensive understanding of the grounding process of our method.

4.4.1. Visualization of Success Cases

Figure 5 shows 10 sets of successful examples. Specifically, each set of examples consists of the query expression, the input image, and the output image with the visualized cross-attention activate mapping and the grounding result.

The goal of CSRLA is to locate the complete object via a cascaded searching strategy from a partial salient region. For instance, the query sentence of example (e) is “boy in gray”, and a prominent gray-clothes-region bounding box is generated according to this in the first stage; then the CSRLA network starts from this box and gradually searches for the object of the boy wearing gray clothes. In another example (g), the salient region focuses on the leg area based on a person’s kneeling motion, and then searches and finds the subject performing the action, i.e., the kneeling person. These examples illustrate that our CSRLA could search the complete object effectively from the local part of the referential target in terms of its ability to use a cascaded searching policy.

4.4.2. Visualization of Failure Cases

Additionally, Figure 6 shows 10 sets of failure examples. Taking (b) and (g) as examples, if the CAAM thresholds selected are too small, objects other than the reference object will be selected incorrectly by large red grounding bounding boxes. On the contrary, examples (d) and (i) illustrate that a large threshold results in a small salient region box, making it difficult to search for the complete referential object.

In (c), when the objects in the image overlap heavily, this also increases the difficulty of referential target grounding. When facing scenarios where the difference between the referential target and other objects is slight, or where the reference is ambiguous, e.g., examples (e) and (j), our model would become confused, resulting in a decrease in grounding accuracies. Moreover, in the face of complex query sentences, grounding is often defective, such as (f), where the query is “boy with white collar looking up behind brown hair head with no face showing”, and CSRLA does not have a good understanding of what this refers to.

In addition, we also noticed that some annotation biases existed in the datasets. For example in (h), the query is “colorful keyboard”, according to which the target grounded by our model is indeed the “colorful keyboard”, whereas the annotated ground-truth target is the puzzling “laptop”.

5. Conclusions

Differently from all existing efforts, we first proposed to address PC tasks in an proposal-free weakly supervised setting (i.e., FWPC) with the intention of completely eliminating the dependence on the expensive supervised region information. To address this new challenge, we proposed a novel model, named CSRLA. We first leveraged a visual language pre-trained model to generate a visual salient region that the query attended closely to. Then, we developed a RL-based regional searching network for searching referential objects from the partial salient region to the complete entity. We verified the performance of our approach on three PC benchmark datasets, i.e., RefCOCO, RefCOCO+, and RefCOCOg. CSRLA’s performance was close to those of the state-of-the-art BWPC methods, but it does not need any explicit or implicit supervised region information, which demonstrates its superiority and effectiveness.