Multimodal Attention-Based Instruction-Following Part-Level Affordance Grounding

Abstract

The integration of language and vision for object affordance understanding is pivotal for the advancement of embodied agents. Current approaches are often limited by reliance on segregated pre-processing stages for language interpretation and object localization, leading to inefficiencies and error propagation in affordance segmentation. To overcome these limitations, this study introduces a unique task, part-level affordance grounding, in direct response to natural language instructions. We present the Instruction-based Affordance Grounding Network (IAG-Net), a novel architecture that unifies language–vision interactions through a varied-scale multimodal attention mechanism. Unlike existing models, IAG-Net employs two textual–visual feature fusion strategies, capturing both sentence-level and task-specific textual features alongside multiscale visual features for precise and efficient affordance prediction. Our evaluation on two newly constructed vision–language affordance datasets, ITT-AFF VL and UMD VL, demonstrates a significant leap in performance, with an improvement of 11.78% and 0.42% in mean Intersection over Union (mIoU) over cascaded models, bolstering both accuracy and processing speed. We contribute to the research community by releasing our source code and datasets, fostering further innovation and replication of our findings.

1. Introduction

The ability to comprehend and execute human instructions in natural language is important for the functionality of embodied agents. Once agents gain this ability, they can engage in natural and intuitive communication with humans. This ability necessitates an in-depth understanding of both natural language processing and visual content analysis. To address this challenge, a significant body of research has emerged, encompassing areas such as vision and language navigation [1,2,3], instruction-following manipulation [4], and visual grounding [5].

Despite considerable progress in various related research areas, formidable challenges, particularly in complex, real-world scenarios, remain in the domain of instruction-following manipulation. Most existing methods predominantly focus on object-level vision–language comprehension, which limits their ability to achieve fine-grained, part-level comprehension. Object-level grounding enables agents to identify the target object, such as “the glass on the right”. However, agents fail to consider the specific functional parts of the object that are necessary to complete the task. For example, the instruction “Hand over the glass on the right”, as depicted in Figure 1, requires the agent to understand not only the object itself (the glass on the right) but also the specific interaction ("hand over" involves the body of the glass in the green area of Figure 1). Conversely, for the instruction “Please pour some water into the right glass”, the agent needs to associate the mouth area of the right glass (the pink area in Figure 1) with the task of “pouring water into”. Object-level comprehension methods [6,7,8,9,10,11] excel at locating objects but struggle to discern part-level differences for the same object.

While object-level grounding is sufficient for basic manipulation tasks such as “pick up”, “hand over”, and “move”, it is inadequate for more complex actions like ‘use the knife to cut’. These actions require semantic grasping [12]—a nuanced comprehension of the object’s different parts and their associated functionalities or “affordance”, a term coined by psychologist James J. Gibson [13]. In robotic manipulation, understanding affordances at a granular level is crucial for determining the exact location for task-specific actions within a real-world context. Interpreting directives such as "Hand over the glass" or "Pour water into the glass" requires an advancement beyond traditional object-level grounding techniques. Existing research typically concentrates on either tracking bounding boxes with language cues [6,7,8,9,10,11] or segmenting object areas without considering the semantic context of tasks [14,15,16], as depicted at the bottom of Figure 1. Such methods, while effective for object detection and instance segmentation, fall short in their granularity, failing to meet the demands of manipulation-oriented robotic interactions. Current affordance detection methods can identify parts of objects, but they are confined to simplistic, pre-defined affordance categories and do not factor in the task semantics. This necessitates additional pre-processing steps for language interpretation and object localization prior to affordance segmentation. Our work seeks to fill this gap, providing a direct and nuanced understanding of instructions for fine-grained affordance segmentation that aligns with the complexities of real-world applications.

To overcome this challenge, we explore the problem of part-level affordance grounding based on natural language instructions. Specifically, we address the problem of segmenting the relevant functional region of a referenced object when given an image and an expression describing a manipulation instruction. In this paper, we introduce the Instruction-following Affordance Grounding Network (IAG-Net), targeting the nuanced task of part-level affordance grounding based on natural language instructions, as illustrated in the upper right of Figure 1. Unlike previous methods that rely on coarse object-level prediction and fixed task representations, our approach represents a significant advancement, achieving an end-to-end understanding of manipulation instructions for precise, fine-grained affordance grounding within images. We introduce a varied-scale multimodal attention mechanism for efficient feature fusion and extract task-specific contexts for precise affordance prediction. Our approach bridges the gap between language instructions and visual cues, facilitating a more nuanced interaction between humans and robots—beyond the capabilities of current object recognition systems. In addition, different from several works using simulated or fixed environment settings [8,11], we establish an instruction image dataset with realistic images and semantic grasping tasks.

Affordance grounding from natural language expressions has broad applications, including human–robot communication and natural language-facilitated human–robot interaction [17]. For instance, when faced with different instructions for the same object, such as “Help me move the table” and “Help me clean the table”, a robot should distinguish the relevant object regions and act appropriately. The part-level affordance grounding can enhance human–robot interaction capabilities and bolster the autonomy of robots in decision-making processes. In the context of dialog response generation, natural language-based affordance grounding can provide specific cues for predicting future human actions and understanding the semantics of manipulation tasks. Compared to one-shot manipulation task representation, natural language offers flexibility for various tasks, such as “Pour the milk” and “Hand over the milk box”. Therefore, this research has the potential to advance the field of robotics by empowering robots with enhanced interaction and decision-making abilities, thus allowing them to operate more autonomously and effectively [18,19] in dynamic environments or under remote control. Moreover, by integrating robust security measures and defenses against potential cyberattacks [20,21], this work contributes to enhancing reliability and trustworthiness in practical applications. In a brief overview, our contributions consist of the following:

(1)

We introduce an end-to-end framework for instruction-following part-level affordance grounding that streamlines the process and enables the development of more efficient robotics applications.

(2)

A realistic instruction dataset for instruction-following affordance grounding is used. Our multimodal dataset, featuring realistic images paired with manipulation instruction expressions, offers a more comprehensive and practical resource.

(3)

We introduce two language–vision feature fusion strategies. By leveraging both sentence-level and task-specific textual features, our approach incorporates a soft event detection strategy in the task-information extraction module, enhancing semantic information guidance for part-level affordance grounding.

The remainder of this paper is structured as follows. Section 2 presents existing works related to our work. In Section 3, we delineate the architecture and three components of our IAG-Net. Section 4 discusses the experimental results and analysis conducted on two datasets. We conclude the paper and discuss potential applications in Section 5.

2. Related Work

In this section, we survey three key areas related to our work: instruction-following robot manipulation, visual grounding, and affordance detection. We outline the interconnections between these areas and highlight the gaps targeted by this work.

Table 1. Comparison affordance grounding with existing related tasks.

Tasks	Language- Following	Object Location	Object Segmentation	Affordance Segmentation
Instruction-following Manipulation	✓	✓
REC	✓	✓
RIS	✓		✓
Affordance Detection				✓
Affordance Grounding	✓			✓

shows a comparison of existing tasks in related work.

2.1. Instruction-Following Robot Manipulation

Understanding robot manipulation in response to instructions is foundational to our research. In robotics, grounding language includes grounding the navigation instructions and the manipulation instructions. Both types of grounding ground the spatial relations of verbs [22,23,24] and the visual attributes of objects, such as colors [25,26] and shapes [27]. However, former work has focused on grounding navigation verbs, such as "Go near the blue cone" and "Walk to the white barrel”. Latter work has focused on grounding manipulation verbs, such as "Pick up the yellow box on the upper side" and "Fetch the red box on the left”.

Our work is most related to grounding manipulation instructions. Early works used semantic structures such as spatial description clauses [28] and the verb-environment-instruction library structure [4] to transform natural language into structured sequences. Recently, many studies have applied deep neural networks to this task by encoding and comparing image features and language features [29,30]. To explore the ambiguity of spatial and perspective problems in "pick up and place" tasks, many studies have introduced probability models [30], commonsense reasoning [31], feedback and conversation [8,26], user gestures [32], spatial reasoning [23], and perspective analysis [33]. In addition to the output of the referred object location, some works have directly mapped an image and language to motor control [27], grasp strategy [34], and structured action sequences. Recently, Mees et al. [11] attempted to perform language-conditioned manipulation with visual affordance following a U-Net [35] architecture. However, their work focused on completing long-horizon manipulation tasks under a simulated environment. Our method is most similar to the instruction grounding methods based on deep learning [7,9]. Li et al. [9] used the “Grasp A to B” as the robot control language format, in which A and B correspond to the target object and the delivery place, respectively. However, existing methods use a two-stage neural network, which first generates candidate objects and then infers the most likely objects. In contrast, we propose a one-stage framework to learn part-level segmentation based on its suitability for a given task specified using natural language. In addition, existing methods focus on grounding "pick up and place” tasks, while our work addresses semantic grasp tasks such as "cut and contain", which is the foundation for learning complex manipulation. Existing language-following datasets [6,7,8,9,10] usually provide object-level annotations rather than part-level annotations and grounding.

2.2. Visual Grounding

As a fundamental multimodal task, visual grounding mainly involves vision and language comprehension and aims to localize the referent object in an image according to the given natural language. This task has gained much attention in both the natural language and computer vision community. Typical visual grounding tasks include referring expression comprehension (REC) and referring image segmentation (RIS).

REC focuses on locating the most related objects in an image based on a given referring expression [36]. In this task, it is assumed that the referring expression unambiguously describes an object. The typical two-stage methods [37,38,39] first generate several candidate regions via a pretraining proposal framework, e.g., Faster-RCNN [40] or Mask-RCNN [41]. Afterward, the candidate regions and expression map are embedded in a common space, and joint embedding is used to maximize the match score of region–expression pairs. However, whole-sentence embedding may fail to consider fine-grained details that are crucial for accurate localization. Compositional modular networks [37] first identify the entities and their relationships in the input expression and then implement end-to-end visual reasoning. MAttNet [38] uses Bi-LSTM to parse an expression into subject, location, and relationship representations and computes individual match scores. However, these two-stage methods tend to be restricted by the generated proposal quality in the first stage. More recently, one-stage methods [42,43] eliminate the offline object detector and directly output the target bounding boxes of the referring expression. Following one-stage methods, several researchers [44,45,46] have explored end-to-end frameworks based on transformers.

Similar to the REC task, the RIS task aims to segment the referring object in an image that matches the corresponding referring expression [47]. In early works [48,49], textual features and image features were usually fused together and then the label for each pixel was predicted following classical image segmentation methods. To improve the performance of feature fusion, many methods have utilized attention mechanisms in image segmentation. Shi et al. [50] proposed a key-word-aware network to evaluate the importance of words according to their contribution to identifying an object. Ye et al. [51] considered the long-range correlations between text and vision modalities. Since most approaches usually perform multimodal fusion between language and vision at the decoding side, the EFN [52] employs a coattention mechanism to achieve simultaneous updates of multimodal features. Due to the excellent performance of transformers [53] based on self-attention mechanisms, many researchers [54,55,56] have extracted and fused vision and language features with self-attention mechanisms.

Although the above methods also input a picture and an expression and output the target object guided by the expression, they are different from our method in two aspects. First, the referring expression and user instructions are different in content. While referring expressions describe the referent object within an image (e.g., “the man on the left of the tree”), manipulation instructions consist of objects and tasks (e.g., “Use the knife to cut”). On the other hand, the above methods achieve object-level location or segmentation. These methods do not adequately address the part-level granularity required for complex manipulation tasks.

2.3. Object Affordance Detection

The concept of affordance encompasses both object affordance and environment affordance [57]. We mainly focus on object affordance in this paper. Vision-based object affordance understanding aims to learn object affordance from vision, which falls within the domain of affordance learning. Visual object affordance is normally associated with three main tasks: object affordance categorization, object affordance detection, and affordance reasoning.

Similar to vision tasks such as object detection and semantic segmentation, affordance detection aims to detect the object and assign pixel-level labels for the object according to the affordance categories [58,59,60,61,62,63]. In early work, manually designed geometric features were extracted from RGB-D images to detect the affordance of objects [58]. As deep learning has progressed, an increasing number of deep-learning-based methods have been proposed to avoid hand-designed features. Nguyen et al. [59] trained an encoder–decoder convolutional network to obtain pixel label predictions. Subsequently, Ref. [60] employed a sequence of deconvolutional layers with a resizing strategy and introduced a multitask loss function to effectively handle the multiclass problem in the affordance mask. AffordanceNet [62] uses two branches to localize the object and assign affordance labels separately. Without object candidates, Zhao et al. [63] designed a relationship-aware network to produce pixel-wise affordance segmentation in an end-to-end way. Considering the object shape, ADOSMNet [16] uses a novel shape mask-guided encoder that aggregates global features and locally relevant object features. In addition, several researchers have introduced weakly supervised and few-shot affordance detection methods [64].

In these studies, the affordance detection and segmentation problem is considered a classification or pixel-level prediction problem. Due to the predefined and fixed nature of the affordance category, it is challenging for robots to directly understand language instructions. Consequently, existing approaches often fail to consider the crucial relation between visual features and task-specific semantic contexts. Our approach innovatively integrates natural language instruction with affordance detection and bridges the gap between language instructions and visual affordance, facilitating a more convenient interaction between humans and robots.

3. Method

In this section, we initially present the problem formulation and describe the overall architecture of our method. Then, we explain the three main modules: the sentence-level multiscale vision language encoder, task-specific context extraction module, and part-level affordance-grounding module.

3.1. Problem Formulation

The primary objective of our method is to generate a part-level mask, M, for the image, I, based on a corresponding natural language instruction, T, that identifies the functional part of the target object relevant to the given manipulation task. With $\theta$ denoted as the model parameters, the task can be formulated as follows:

$$M=Af{f}_{\theta }(I,T).$$

(1)

In contrast to existing object-level segmentation tasks, M is a part-level mask corresponding to the affordance described in the language instruction. This task requires not only grounding between the image and the natural language but also grounding between the manipulation task and the functional part of the object. The former is object-level grounding and the latter is part-level grounding. Inspired by the characteristics of this task, we developed a method that employs two language–vision feature fusion strategies, one for sentence-level semantic features and the other for task-specific semantic features.

3.2. Overview

The architecture of our approach, as shown in Figure 2, comprises two foundational modules for extracting visual and linguistic features, followed by a multiscale vision language encoder to integrate sentence-level linguistic features with visual features from multiple scales. To effectively learn the fused feature from two different modalities, the encoder fuses sentence-level language features with multiscale visual features. The multiscale vision language encoder focuses on grounding the image with the sentence-level language feature. In addition to sentence-level semantic features, different words in the instructions contribute differently to the manipulation task. Recognizing the varying importance of different words in the instructions for the manipulation task, we introduce a task-specific context extraction and fusion module. This module is designed to identify and emphasize words that are crucial for understanding the manipulation task. Finally, the part-level affordance-grounding module combines the outputs of the above two modules and predicts a mask, M, for the functional part of the target object.

3.3. Vision and Language Encoders

For multimodal tasks involving both vision and language, the two-tower model architecture has emerged as a widely adopted approach for effectively extracting features from each modality. Therefore, in our approach we employ two distinct backbones, a visual encoder and a language encode, to separately extract visual and textual features. The network inputs consist of an image, $I\in {\mathbb{R}}^{FH\times FW\times C}(C=3)$, and a language instruction, $T=\{w_1,w_2,\cdots ,w_t\}$, where $FH$, $FW$, and C represent the height, width, and color channel number of the image, respectively, and t denotes the length of the instruction. The visual encoder processes the image input, while the language encoder handles the textual input.

Visual Features: We adopt ResNet 50 [65] as our visual backbone, following the approach of many prior works in image segmentation and detection [55,66]. ResNet 50 is known for its ability to effectively capture detailed multiscale visual features. The backbone produces multilevel visual features, $Re{s}_i\in {\mathbb{R}}^{F{H}_i\times F{W}_i\times C_i}$, where $Re{s}_i$ corresponds to a ResNet 50 feature and $i\in 1,2,3,4$ denotes the block index. Here, $F{H}_i$ and $F{W}_i$ represent the height and width of the feature map, respectively, and $C_i$ indicates the dimension of the feature channels.

Text Features: Language feature are extracted using BERT [67], a language encoder that levarages a bidirectional self-attention mechanism within transformer layers to generate rich linguistic representations. Given an instruction sentence, $T=\{w_1,w_2,\cdots ,w_t\}$, where $w_i$ represents the i-th word with $i=\{1,\cdots ,t\}$, BERT outputs a final hidden state, $F_w=\{f_1,f_2,\cdots ,f_t\}\in {\mathbb{R}}^{t\times d}$, where d is the dimension of the word embedding. Each embedding vector, $f_i$, corresponds to the contextual representation of the i-th word, $w_i$, in the instruction.

3.4. Multiscale Vision Language Encoder

A multimodal learning study [68] demonstrated the significance of high-level visual features, especially when language terms are directly related to objects such as “Hand over the cup on the right of the desk”. This insight serves as a guiding principle for our approach, where we harness high-level visual features and sentence-level textual features to capture object-level vision and language attributes effectively. The multiscale visual language encoder is designed to extract object-level multimodal features for affordance grounding.

We obtain the global sentence representation by concatenating the representations of all words in the sentence, as expressed in the following equation:

$$F_{gt}=[f_1,\cdots ,f_t],$$

(2)

where $[,]$ represents the concatenation operation. Initially, we transform both the textual and visual features into a uniform feature dimension using two separate linear layers. Subsequently, these features are fused, as shown in Equation (3).

$$F_m=[g\left(F_v{W}_v\right),g\left(F_{gt}{W}_t\right)],$$

(3)

where $W_v$ and $W_t$ are two learnable parameter matrices and g denotes the activation function Relu. The concatenation operation, $[,]$, is employed to combine features from two different modalities. The visual feature, $F_v$, is extracted from the visual backbone, specifically using the visual feature $Re{s}_4$. The textual feature, $F_{gt}$, is computed according to Equation (2). The multimodal feature, $F_m$, results from fusing the sentence-level semantic and high-level visual feature, providing a global multimodal representation.

In order to enhance the fusion of visual–linguistic features, we propose a varied-scale multimodal attention (VSMA) mechanism designed to effectively capture cross-modal features. The incorporation of multiscale visual features has demonstrated efficacy in image segmentation, particularly in handling objects of varied scales. Additionally, cross-modal attention mechanisms guide the network to focus on image regions relevant to the language context, thereby enhancing performance in vision–language tasks. Motivated by these insights, we introduce the VSMA mechanism, which integrate a multiscale structure with cross-modal attention to refine the features and concentrate on the essential multimodal features of the target object. The structure of the VSMA mechanism is depicted in Figure 3. Specifically, VSMA comprises channel attention, spatial attention at various scales, and multimodal attention.

For channel attention, we compute the attention weights from the average pooling and max pooling of the multimodal feature, $F_m$, following channel attention in the CBAM [69]. Since various scales have been efficient for various image processing tasks in previous works such as ASPP [70], we introduce varied-scale spatial attention to improve the robustness of the affordance parts in the images. For the feature of modality, m, we first compute the max pooling and average pooling features as $F_{max}^m$ and $F_{avg}^m$, respectively. Next, atrous convolutional layers [70] with different sampling rates, $Aco{n}_s$, are employed to capture varied-scale spatial information, the results of which are concatenated. Then, a convolution operation is applied to obtain the varied-scale spatial attention weights.

$$VSA\left(F_m\right)=Conv\left(Aco{n}_s\left([F_{max}^m,F_{avg}^m]\right)\right)F_m,$$

(4)

where $Aco{n}_s$ is the atrous convolutional layer for scale s, as in ASPP, and $Conv$ is the convolution layer.

In addition to spatial attention, multimodal attention is useful for determining the role of each modality in the affordance-grounding tasks. Therefore, multimodal attention is utilized to leverage features from different modalities before fusion, as shown in Figure 3. The whole VSMA can be represented as follows:

$$\begin{array}{cc}\hfill F_{ma}^h& =VSMA\left(F_m\right)\hfill \\ \hfill & =MA\left(VSA\left(CA\left(F_m\right)\right)\right)\hfill \end{array},$$

(5)

where $CA$ represents the channel attention, $VSA$ is the varied-scale spatial attention, and $MA$ denotes the modality attention. $F_{ma}$ is the refined multimodal feature after the h-th VSMA and $h=\{1,\cdots ,H\}$ represents the index of heads in the varied-scale multimodal attention (VSMA). The number of VSMA mechanisms, indexed by h, collectively captures varied-scale cross-modal attention. The details of the head number, H, settings are described in the experiments (Section 4.3).

Finally, the outputs of different heads are fused by concatenation and convolution operations to obtain the multiscale attention features, $F_s$.

$$F_s=con{v}_{1\times 1}[F_{ma}^1,\cdots ,F_{ma}^H]$$

(6)

Through channel attention, varied-scale spatial attention, and multimodal attention, the features from two different modalities are fused deeply and densely. To further obtain part-level results, the output of this module is passed into the part-level affordance to refine the segmentation.

3.5. Task-Specific Context Extraction

Task information, exemplified by actions and objects such as “cup” and “contain”, plays a crucial role in grounding object parts for specific tasks. For instance, given two similar language instructions, “Grasp the cup” and “Grasp the knife”, distinguishing the target object in the image is essential. Similarly, for instructions like “Use the cup to contain” and “Grasp the cup”, the different actions lead to distinct affordance areas. Both action and object information are necessary for part-level affordance grounding. Therefore, this module is designed to extract a task-related textual context, which we denote as the task-specific context for part-level affordance grounding.

The target object and responding action/verb are two important elements for the task, such as “Use the knife to cut” and “Hand over the cup”. We conceptualize the task context as a pair of key elements: <object, action>. Drawing inspiration from event extraction (EE) [71] in natural language processing (NLP), we treat the task-specific context extraction problem akin to the event extraction problem, where the action serves as the event trigger and the object serves as the argument. Since there is no annotation for the trigger or the argument in the datasets, we use a soft classifier to predict the probability that each word belongs to one of three types (i.e., trigger, argument, or unnecessary words). Then, the task-specific context of the entire expression can be calculated from the probability of each word.

Word Feature Embedding: By capturing the syntactic and semantic essence of each word, we can map words in the instruction to individual vectors using pre-trained BERT, as outlined in Section 3.2. After this step, the instruction is represented as a sequence of vectors, $\{f_1,f_2,\cdots ,f_t\}$. These word representations are then augmented with position embedding, $p_i$, resulting in a composite feature for each word, $w_i$, denoted by $[f_i,p_i]$.

Graph-Based Feature Encoding: Following word embedding, we transform the dependency tree of the instructions into an undirected syntactic graph. This graph effectively links words to their contextual counterparts, enhancing relational understanding. For instance, in the instruction "Use the knife on the right to cut", the action word "cut" and its corresponding object "the knife" are closely linked in the graph, despite being distant in the sequential representation. In the sequential representation, the two words are 5 steps apart, whereas in the dependency tree they are 2 steps apart (as shown in Figure 4).

In the generated graph $G=\{V,E\}$, each node, $v\in V$, represents a word in the given instruction, while each directed edge, $e\in E$, corresponds to a syntactic arc in the dependency tree (see Figure 4). To consider and learn the syntactic context for each word, we employ a graph neural network (GNN) [72] to aggregate information from the connected neighbors. After passing through L layers, the GNN outputs a representation for each word, $f_i^L$.

Task-Specific Context Computing: Each word’s feature from the graph, denoted as $f_i^L$, is passed through a multi-layer perceptron (MLP), resulting a soft probability vector, $s_i\in {\mathbb{R}}^{1\times 3}$. Each vector, $s_i$, indicates the likelihood of each word, $w_i$, belonging to one of three categories: action, object, or unnecessary word. The task-specific context of the instruction, T, is then computed using a weighted combination of all the words, as shown in Equation (7).

$$f_c=\sum _{i=1}^t(s_i^1\times h_i^L+s_i^2\times h_i^L),$$

(7)

where $s_i^1$ and $s_i^2$ represent the probabilities of the action and the object for each word, $w_i$, respectively. $h_i^L$ is the word embedding for the word, $w_i$, which is encoded by GNN. $f_c$ represents the task-specific context for the entire sentence. The task-specific context, $F_t$, is then fused with visual features, $Re{s}_1$, through a $1\times 1$ convolution operation, resulting $F_t$ as follows:

$$F_{tc}=con{v}_{1\times 1}[Re{s}_1,f_c],$$

(8)

where the combined feature, $F_{tc}$, is subsequently utilized in the part-level affordance-grounding module. The $F_{tc}$ fuses the task-specific context with the visual features, which provide a rich semantic of action and target object.

In summary, the task-specific context extraction module extracts a task-related textual context, represents it as a pair of key elements, <object, action>, and integrates this context with visual object features for further use in the part-level affordance-grounding module.

3.6. Part-Level Affordance Grounding

In this module, we leverage the extracted task-specific information to precisely guide part-level affordance grounding. The multiscale feature, $F_s$, provide cross-modal features with varied scales, which provides global multimodal information. The task-specific features, $F_{tc}$, further refine the semantic information and pay more attention to local information. To closely align object parts with their respective functional tasks, we employ an “upsample after fusion” strategy, wherein we first resize the multiscale feature, $F_s$, and task-specific features, $F_{tc}$, to a uniform size. These features are then concatenated and subjected to convolution operations for effective integration. This fusion process is crucial for aligning visual and task-specific cues at a fine-grained level. We repeatedly apply deconvolution layers to the fused features until the resulting feature map is upscaled to match the size of the original image. This process ensures that the part-level affordance grounding is detailed and aligns well with the original image scale, which is denoted as:

$$M_a=decon{v}_{repeated}\left(con{v}_{3\times 3}\left([reshape\left(F_s\right),F_{tc}]\right)\right).$$

(9)

3.7. Loss Function

To optimize the part-level affordance grounding, we define our loss function using binary cross-entropy [73]. This loss function is particularly suited for binary classification tasks, where each pixel in the image is classified as relevant or irrelevant for the task. The function is formulated as follows:

$$L_{aff}=-\sum _{k=1}^{H\times W}(y_{k}log\left(p_k\right)+(1-y_k)log(1-p_k)),$$

(10)

where $y_k$ represents the pixel element of the ground-truth mask and $p_k$ represents the corresponding pixel element in the predicted mask. The pseudocode for the IAG-Net training process is outlined in Algorithm 1.   

Algorithm 1: Training process for IAG-Net

4. Experiments and Evaluation

To thoroughly assess our method, we conducted extensive experiments on two public datasets, focusing on investigating the following research problems:

RQ1: How does IAG-Net perform compared with existing related methods?

RQ2: How does each design choice made in IAG-Net affect its performance?

RQ3: How sensitive is IAG-Net with different hyperparameters and optimizations?

4.1. Dataset Construction

This task necessitates paired data, $<T,I,M>$, comprising natural language instructions, T, corresponding RGB images, I, and pixel-level part affordance masks, M. As no existing datasets fulfill these requirements, we constructed a new affordance-grounding dataset (https://github.com/WenQu-NEU/Affordance-Grounding-Dataset accessed on 30 March 2024) with the following methodology:

• Image Selection: The first step involves selecting an image, I, with existing affordance annotations from two widely recognized computer vision datasets: the IIT-AFF dataset [59] and the UMD dataset [58]. We detail the specific characteristics and selection criteria for each dataset below.
• Language Instruction Labeling: After image selection, we manually added instructions, T, for each image, I.
• Affordance Mask Generation: We generated the affordance mask, M, based on the image annotations corresponding to the given instructions, T.

Image Selection: In computer vision, there are two widely used datasets with pixel ground-truth affordance labels for objects, i.e., the IIT-AFF dataset [59] and the UMD dataset [58]). The IIT-AFF dataset (https://sites.google.com/site/iitaffdataset/) has 8835 real-world images with cluttered scenes. This dataset contains 10 object categories: bowl, tv, pan, hammer, knife, cup, drill, racket, spatula, and bottle, and 9 affordance classes: contain, cut, display, engine, grasp, hit, pound, support, and w-grasp. In contrast, the images in the UMD dataset (https://users.umiacs.umd.edu/~fer/affordance/part-affordance-dataset/index.html) are objects on a rotating table with a clutter-free setup. These images were captured by a Kinect camera. We utilized the IIT-AFF and UMD datasets for their extensive annotations and the affordance categories they offer, which align with our objective to examine object affordances in diverse scenarios. To ensure the UMD dataset’s variety and to avoid redundancy, we implemented a filtering process to remove highly similar images. By computing pairwise image similarity, any pair with a similarity score exceeding the threshold of $0.9$ had one image randomly removed. This curation resulted in a refined UMD dataset consisting of 7070 unique images for our experiments. Consequently, our enhanced dataset now comprised 14,642 object bounding boxes and $24,677$ pixel-level affordance regions, making it a robust and comprehensive resource for affordance analysis.

Language Instruction Labeling: To create a semantic link between functional areas and language instructions, we manually crafted language instructions for each image, adhering to standardized guidelines to ensure consistency, relevance, and diversity across annotations. Two volunteers were tasked with generating instructions based on the content of each image. Each instruction was meticulously designed to encompass three critical elements: the object, the action to be performed, and the affordance related to the action, such as “Use the cup to contain”. To ensure accuracy and consistency, each volunteer worked independently, followed by a review process where any discrepancies between their instructions were discussed and resolved. This process resulted in 8835 and 7070 expressions for the IIT-AFF and UMD datasets, respectively. To enrich the diversity of our language instructions, we specifically focused on varying the expressions used for objects with multiple affordances. For example, a knife, which possesses affordances such as “grasp” and “cut” was labeled with distinct instructions such as “Grasp the knife” or “Use the knife to cut”. Additionally, we incorporated positional descriptors such “the right” and “the left” descriptors to provide more precise and contextually relevant descriptions of the objects. Figure 5 shows examples of our dataset. The first and third rows display the original images along with their affordance annotations from the IIT-AFF and UMD datasets. The second and fourth rows further demonstrate how these images are paired with language instructions and their corresponding ground-truth masks.

Affordance Mask Generation: Generating affordance masks posed challenges in accurately aligning textual instructions with visual annotations, which we addressed through iterative refinement and validation processes. The IIT-AFF and UMD datasets provide affordance annotations for each image. For each image annotation, the value of each pixel ranges from 0 to 9 and 0 to 7 for the IIT-AFF and UMD datasets, respectively. In the third row of Figure 5, each color corresponds to a different affordance category. We automatically generated affordance masks corresponding to the text instructions. Therefore, a binary mask generated corresponds to each instruction focusing on one affordance area at a time, which is more aligned with our research questions.

Table 2. Statistics of instruction-following affordance-grounding dataset.

	Dataset	ITT-AFF VL		UMD VL		All
Number		Image	Instructions	Image	Instructions	Pairs
contain		1143	1143	949	949	2092
cut		101	101	790	790	891
display		1594	1594	-	-	1594
engine		71	71	-	-	71
grasp		1617	1617	2611	2611	4228
hit		414	414	-	-	414
pound		180	180	419	419	599
support		440	440	1060	1060	1500
w-grasp		926	926	671	671	1597
scoop		-	-	570	570	570
Average		721	721	1010	1010	1731
All		8835	8835	7070	7070	13,556

shows the statistics of the constructed affordance-grounding dataset, which are denoted as the IIT-AFF VL dataset and UMD VL dataset. Since IIT-AFF VL and UMD VL have different affordance categories, we used "-" to represent absent affordance categories in the table. For a comprehensive and fair evaluation, we randomly divided the dataset into distinct sets: training ($80\%$), testing ($10\%$), and validation (the remaining $10\%$). This division ensures a balanced representation of the data across all the experimental phases. Considering the diversity of expressions, we used different affordance categories to construct instructions for each object category. For instance, the object “knife” can provide two affordance categories, i.e., “cut” and “grasp”. We used different instructions to cover them, such as “Grasp the knife” and "Use the knife to cut".

4.2. Experimental Settings

4.2.1. Implementation Details

We used the Python and PyTorch frameworks via NVIDIA RTX3060 on Windows to implement our method. In the experiment, the input images were resized to $512\times 512$ pixels following previous works [70,74]. For model optimization, we utilized stochastic gradient descent (SGD) [75] with a weight decay of $1\times {10}^{-4}$ and momentum of $0.9$. The initial learning rate was set to $0.001$ and gradually decreasing with the polynomial decay strategy.

4.2.2. Evaluation Metrics

We employed the mean intersection-over-union (mIoU) and $F_{\beta }^{\omega }$ metrics [76] because they are widely recognized for assessing segmentation quality in contexts similar to ours. The mIoU is calculated as the average ratio between the intersection area and the union areas across all pixels using both the ground-truth mask, M, and the predicted probability result, P. For affordance segmentation, most studies employed the $F_{\beta }^{\omega }$ metric based on the parameters defined in [76]. Therefore, we also reported the $F_{\beta }^{\omega }$ for the experiments. Since our datasets include diverse affordance categories, we conducted evaluations per the affordance category, allowing for a more granular understanding of the model performance across different types of object interactions.

4.3. Comparison Results (RQ1)

To demonstrate the effect of our model, we conducted comparative analyses with three distinct methodological paradigms: image-only segmentation methods, cascaded methods, and multimodal-based RIS methods. Image-only methods were selected to determine the influence of visual data in isolation, providing a baseline to analyze the impact of incorporating linguistic instructions. Cascaded methods and multimodal-based methods were chosen to represent common practical applications and to assess how the fusion of text and image data informs the performance within these contexts. Specifically, we selected image-only segmentation methods (DeepLabV3+ [74], OCRNet [77], AffordanceNet [62], RelaNet [63], BPN [14], GSE [15], and ADOSMNet [16]) that are considered benchmarks within the field of affordance segmentation. It is important to note the fundamental difference between image segmentation and affordance segmentation tasks. While image segmentation predicts masks for an entire image indiscriminately, affordance segmentation is more intricate, often requiring a pre-processing step for object detection or an integrated approach that accomplishes object detection and segmentation simultaneously. In the realm of multimodal approaches, we incorporated cascaded methods (combining Faster-RCNN [40] and DeepLabV3+ [74]) owing to their prevalent adoption in practical scenarios, thus enabling an evaluation that reflects real-world application performance. Furthermore, we evaluated our model against a state-of-the-art RIS method (BKINet [56]), which represents the latest innovations in multimodal understanding. These baselines were chosen due to their proven robustness and widespread recognition in the literature, providing a well-understood standard from language-independent to language-integrated approaches, against which to measure our advancements. The details of these methods are as follows:

• DeepLabV3+ [74]: The semantic segmentation method DeepLabV3 was employed for its semantic segmentation capabilities. By fine-tuning this method in our dataset, we aimed to assess the additional impact, if any, that language processing contributes to overall affordance-grounding performance.
• OCRNet [77]: OCRNet is notable for its object-contextual representations for semantic segmentation, which characterize each pixel by leveraging the features of its corresponding object class. Since affordance segmentation is closely related to object class, OCRNet serves as an appropriate image-only baseline for our study.
• AffordanceNet [62]: AffordanceNet, a classical affordance detection method, segments images based on affordance categories without the integration of language instructions, providing a baseline for comparing language-agnostic approaches.
• RelaNet [63]: RelaNet is an end-to-end affordance segmentation method without separate and intermediate object detection steps. It considers relationships between the affordance and corresponding objects. Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original papers (the backbone with ResNet50).
• BPN [14]: BPN considers the potential relationship between object categories and object affordances and proposes a boundary-preserving network. It is an image-only method for affordance detection. Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original papers.
• GSE [15]: GSE utilizes a repeated multi-scale feature-map-fusion network to produce category-relevant feature maps. Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original papers (the backbone with ResNet50).
• ADOSMNet [16]: This state-of-the-art affordance detection method operates without language instructions. We considered it an upper bound in our comparisons since it eliminates the need to interpret language and achieve state-of-the-art results on the two datasets (IIT-AFF and UMD). Because this dataset is a closed-source dataset, we employed the results of the IIT-AFF dataset reported in the original papers (the backbone with ResNet50). Unfortunately, ADOSMNet’s performance on the UMD VL dataset could not be compared because the dataset selected is specific to our study (details in Section 4.1).
• Cascaded Method: Our custom-designed cascaded method combines object detection and affordance segmentation. It begins by localizing the object mentioned in the instructions using Faster-RCNN [40]. The target object is the region with the highest prediction probability, and affordance segmentation is performed within the target region with DeepLabV3+ [74]). This approach provides insights into the impact of object grounding on the whole process.
• BKINet [56]: BKINet is a state-of-the-art method for referring image segmentation that involves uniquely segmenting a reference object directly based on the input text. We trained this model on our dataset to evaluate its performance under the same conditions as though in our proposed method.
• TransVG [44]: TransVG is a classical visual grounding method that predict object localization with language. We trained this model on our dataset to provide the object-level grounding results, which is easier than our part-level task.

In

Table 3. Performance on the UMD vision language dataset. Bold text is utilized to highlight the best values achieved for mIoU and F-score among multimodal-based methods.

	Method	Image Only					Multimodal-Based
		DeepLabV3+		OCRNet		AffordanceNet	Cascaded		BKINet		IAG-Net (Ours)
Affordance		mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$
contain		85.80	81.27	85.40	81.72	59.75	86.11	83.38	79.03	47.81	85.24	82.02
cut		82.3	78.6	82.76	79.42	64.26	82.07	80.62	42.32	43.65	78.34	79.89
grasp		69.5	80.03	71.02	80.54	65.97	71.39	78.70	75.98	51.80	76.24	81.08
pound		85.42	81.29	85.61	81.70	76.73	85.20	82.08	61.45	67.17	85.13	83.14
scoop		85.57	79.74	85.92	80.65	77.0	85.79	81.65	70.84	86.83	86.34	81.22
support		83.77	81.11	85.03	82.04	79.45	85.86	81.34	89.37	90.47	85.20	81.41
w-grasp		80.80	76.04	81.01	77.06	78.65	78.97	73.00	86.46	88.08	81.81	75.45
Average		81.88	79.73	82.39	80.45	71.6	82.19	80.11	72.20	67.97	82.61	80.60

and

Table 4. Performance on the IIT vision language dataset. Bold text is utilized to highlight the best values achieved for mIoU and F-score among multimodal-based methods.

	Method	Image Only									Multimodal-Based
		DeepLabV3+		OCRNet		AffordanceNet	RelaNet	GSE	BPN	ADOSMNet	Cascaded		BKINet		IAG-Net (Ours)
Affordance		mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$
contain		85.68	63.98	88.35	70.85	79.61	80.20	87.92	80.62	87.77	73.18	66.11	81.79	65.90	87.72	70.25
cut		69.29	61.58	79.20	61.94	75.68	78.04	65.34	79.23	84.68	67.45	58.68	63.37	43.01	73.29	65.11
display		87.03	39.30	89.46	41.42	77.81	79.14	91.90	80.55	84.26	74.37	39.18	96.68	77.17	91.55	42.84
engine		78.60	73.99	79.82	73.96	77.50	81.22	81.91	81.49	84.37	58.94	51.53	78.52	75.87	85.39	79.58
grasp		75.95	29.28	78.59	31.61	68.48	71.59	79.76	72.96	83.07	70.60	29.42	69.12	59.31	83.49	33.68
hit		85.08	40.07	94.97	39.12	70.75	88.52	90.51	88.84	84.64	89.96	35.93	64.25	75.00	92.55	43.19
pound		79.47	54.82	79.21	53.16	69.57	76.91	75.95	77.59	82.68	72.66	46.09	35.56	51.28	75.43	55.21
support		79.60	67.88	81.36	61.09	69.81	80.12	78.41	80.96	83.74	76.98	56.59	71.44	68.63	77.99	51.31
w-grasp		88.58	70.08	82.13	73.20	70.98	74.56	89.43	74.56	83.35	67.56	64.06	52.43	76.06	90.29	70.31
Average		81.03	55.66	83.68	56.26	73.35	78.92	82.33	79.64	84.28	72.41	49.73	68.13	65.83	84.19	56.83

, we present the accuracy values of the evaluated methods across the two datasets. The clear background and minimal environmental interference of the UMD dataset likely contribute to its generally higher performance than that of IIT-AFF. This observation has shown the importance of dataset characteristics in influencing the model accuracy. Among the three categories of methods, image-only methods, by concentrating solely on visual cues, avoid the complexities and potential inaccuracies introduced by language data integration, which explains their superior performance. However, when object grounding is integrated with the image-only method in the cascaded approach, there is a notable performance decrease. For instance, compared with that of DeepLabV3+, the cascaded method shows a decrease in performance by $-1.41\%$, $-1.69\%$, $-8.59\%$, and $-43.07\%$ on the two datasets. The cascaded method’s performance drop, as indicated by the comparison with DeepLabV3+, suggests that the sequential process of object grounding followed by affordance segmentation may introduce compounded errors, leading to lower overall accuracy. Considering the image characteristics of UMD and IIT-AFF, we find that the more complex the image content is, the larger the performance decreases for the cascaded method. We also evaluated the accuracy of object localization with the visual grounding method TransVG [44], which has average accuracy values of approximately $87.28$ on the IIT-AFF dataset and $99.78$ on the UMD dataset. This result highlights a challenge for both cascaded and multimodal methods, which process both image and language instructions, as they tend to underperform compared to image-centric approaches. Our IAG-Net notably outperforms the cascaded and multimodal RIS methods across most categories, demonstrating its robustness and effectiveness. BKINet’s underperformance in categories such as “cut”, “hit”, “pound”, and “support” could be attributed to the limited sample size in these affordance categories within the IIT-AFF VL dataset. Our method, IAG-Net, demonstrates competitive results comparable to those of image-only methods and appears to be less influenced by dataset variability. In addition to the above methods, we also tested the large multimodal model NExT-Chat [78] for this task using the provided demo (https://next-chatv.github.io/). NExT-Chat is a large multimodal model for detection and segmentation that can handle multiple tasks, such as visual grounding, region captioning, and grounded reasoning. While NExt-Chat is capable of bounding boxes or segmentation for entire objects, it is unable to achieve part-level visual understanding. Enhancing large multimodal models such as NExT-Chat for more granular tasks points to an area for future exploration and could be a valuable direction for subsequent research.

Visualization: To further illustrate the subjective performance of IAG-Net, we present a series of visualizations that compare our model’s affordance-grounding results with the ground truth. Each set in our visual representation, as displayed in Figure 6, comprises four parts. The first column displays the original image paired with its corresponding language instruction. The second column shows the overlap of the predicted affordance mask on the original image. The third and fourth columns provide a side-by-side comparison of the affordance prediction maps of IAG-Net against the ground-truth annotations. These visual examples demonstrate that IAG-Net is capable of effectively interpreting language instructions and aligning closely with ground-truth affordance maps.

Statistical Analysis: To evaluate the statistical significance of the performance disparities among multimodal-based approaches, we undertook a statistical analysis focusing on the mean IoU and F-score values across two datasets. The results revealed a statistically significant difference in segmentation performance among the three methods. In Figure 7, we present a visualization depicting the mean IoU and F-score values alongside their respective confidence intervals for two datasets. These confidence intervals were calculated using the t-distribution, assuming a $95\%$ confidence level and considering the sample size. The inclusion of confidence intervals in our visualization effectively displays the uncertainty associated with these performance estimates. Notably, our method is distinguished by its tighter confidence intervals for the mean IoU across both datasets, indicating a more uniform performance across various affordance categories and underscoring the robustness of our approach. Nevertheless it is important to note that there remains room for improvement in terms of the F-score performance of our method. While our method shows promising results in terms of mean IoU, it requires further refinement to improve F-score outcome.

Model Efficiency Analysis: In terms of model efficiency, IAG-Net has a more streamlined architecture with fewer parameters than the cascaded method, as illustrated in

Table 5. Model efficiency analysis of image-only and multimodal based methods. The ↑ signifies that larger values are preferable.

Method	Type	Params	FPS ↑
DeepLabV3+	Image only	$39.76$	$4.86$
Cascaded	Mutimodal-based	$81.86$	$2.61$
BKINet	Multimodal-based	$138.55$	$10.85$
IAGNet (Ours)	Multimodal-based	$42.35$	$4.24$

. This efficiency stems from IAG-Net’s integrated approach, in contrast to the cascaded method’s separate training for visual grounding and affordance segmentation, which results in a larger parameter set and lower FPS. Compared to image-only methods, IAG-Net achieves a similar balance of FPS and parameters, indicating its efficiency in handling both visual and language data without significant resource overhead. BKINet achieves the highest FPS, partly because of the parallel computing of the architecture.

Failure Cases: Our analysis of failure cases (as shown in Figure 8), particularly in complex scenes from the IIT-AFF dataset, reveals that IAG-Net sometimes struggles with images containing multiple objects matching the instructions, leading to partial target recognition. Occlusions present another challenge, affecting the model’s performance. These errors predominantly arise from the model’s current limitations in simple decoder architecture. Future work will focus on enhancing this component to better interpret instructions and handle complex visual scenes.

4.4. Ablation Study (RQ2)

To verify the contribution of each component, we conducted extensive ablation experiments on the two datasets and analyzed the results of different operations. As shown in

Table 6. Results of the ablation study for each component. The symbol ✓ indicates the addition of the corresponding component. Bold text is utilized to emphasize the best values achieved for each metric.

Method	TaskInf	MH-VSMA	Overall Acc	Mean Acc	Freq Acc	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$
Base			$94.8$	$70.5$	$90.5$	$74.9$	$43.0$
Base+TaskInf	✓		$\mathbf{95.4}$	$71.7$	$\mathbf{91.4}$	$75.8$	$43.9$
IAGNet	✓	✓	$94.0$	$\mathbf{85.0}$	$89.0$	$\mathbf{79.3}$	$\mathbf{46.0}$

, the components of the network include the following:

• Baseline: In the baseline model, the encoder extracts visual and textual features, and the fused multimodal features are fed into the decoder for affordance segmentation. This model serves as a foundational comparison point without the task-specific context and VSMA.
• Baseline+TaskContext: This model incorporates task-specific context into the baseline, enhancing the multimodal fusion process.
• IAG-Net: The full model integrates both the multihead VSMA (MH-VSMA) and the task-specific context, offering advanced affordance-grounding capabilities.

For the baseline, we fused the visual and textual features in the encoder, and the decoder outputs the affordance segmentation mask. For Baseline+TaskContext, we added the task-specific context and the multimodal fusion to the encoder. In this case, for all five measurements (overall accuracy, mean accuracy, frequency accuracy, mIoU, and $F_{\beta }^{\omega }$), improvements are observed to varying degrees. After adding MH-VSAM, the full model significantly improves in both the mIoU ($+3.5\%$) and $F_{\beta }^{\omega }$ ($+2.1\%$). The results are presented in Table 6. The introduction of a task-specific context significantly improves the overall and frequency accuracies, demonstrating its effectiveness in capturing task-relevant information. The inclusion of MH-VSAM further improves the mean accuracy, mIoU, and $F_{\beta }^{\omega }$, underscoring its role in refining affordance segmentation. The performance of overall accuracy and frequency accuracy have declined slightly. While mIoU and F-score focus on segmentation quality and boundary delineation, overall accuracy and frequency accuracy consider pixel-level classification correctness. The increase in mIoU and F-score suggests an improvement in segmentation quality, which may come at the cost of overall accuracy as the model might prioritize correctly segmenting complex regions over accurately classifying all pixels. The sample imbalance issues could also cause results where certain classes are underrepresented in the dataset. Since the frequency accuracy is computed by averaging the frequency accuracies for all classes in the dataset, the sample imbalance will cause a decrease if less frequent classes are misclassified more often. The ablation study indicates that both the task-specific context and the MH-VSAM benefit the affordance-grounding task. These enhancements validate our two-stage fusion strategy, which combines sentence-level fusion and task context fusion for more accurate affordance grounding.

4.5. Influence of Hyperparameters and Optimization (RQ3)

When we used the multi-head VSMA in the encoder, there was an improvement in the two measurements. For the multi-head VSMA module, we evaluated the method with different numbers of heads. As shown in

Table 7. Hyperparameters for MH-VSMA, with different head numbers of VSMA. The bold text is used to highlight the best value for each metric.

Head Number	Mean Acc	mIoU	${\mathit{F}}_{\mathit{\beta }}^{\mathit{\omega }}$	FPS
1-VSMA	$74.63$	$74.08$	$41.18$	$\mathbf{4.10}$
2-VSMA	$82.85$	$76.39$	$44.10$	$4.02$
3-VSMA	$81.99$	$76.31$	$44.30$	$3.82$
4-VSMA	$83.94$	$76.68$	$45.86$	$3.61$
5-VSMA	$\mathbf{84.99}$	$\mathbf{79.32}$	$\mathbf{45.99}$	$3.56$
6-VSMA	$81.41$	$75.48$	$41.77$	$3.55$

, our evaluation of the multi-head VSMA module with varying head numbers reveals a trade-off between accuracy and FPS. While increasing the number of heads enhances the mean accuracy, mIoU, and $F_{\beta }^{\omega }$, it concurrently reduces the FPS. The use of 5 heads strikes a balance, achieving improved accuracy while maintaining reasonable computational efficiency. The results presented in Table 7 are derived from averaging multiple runs, with data randomly sampled from the two datasets for each iteration. This process ensures that our findings are robust and representative of various data scenarios.

We also conducted compaarative experiments with two different optimizers, SGD and Adam. Although Adam demonstrated faster convergence during the initial epochs and overall convergence speed, SGD achieved higher metrics in both mIoU and mean accuracy. This outcome underscores the suitability of SGD for the affordance segmentation tasks. Consequently, in our experimental setup we opted for the SGD optimizer, configured with a weight decay of $1\times {10}^{-4}$ and a momentum of $0.9$, as detailed in the experiment setting subsection.

5. Conclusions

In this work, we address the problem of affordance grounding by introducing the instruction-following affordance-grounding network (IAG-Net), representing a significant advancement in affordance segmentation. This is particularly notable due to its ability to directly interpret natural language instructions and bypass traditional object detection stages. Our key findings emphasize the importance of both sentence-level and task-specific contexts in visual language tasks, especially in achieving precise part-level affordance grounding.

We contribute to the field by creating two multimodal datasets, namely, IIT-AFF VL and UMD VL, which are built upon the existing image-only datasets IIT-AFF and UMD. By augmenting images with affordance annotation with text instructions, we generate instruction-following affordance masks. The creation of the IIT-AFF VL and UMD VL datasets represents a notable contribution to the field, providing comprehensive resources for future research in fine-grained vision–language affordance understanding. These multimodal datasets not only extend existing image-only resources but also offer rich opportunities for exploring advanced multimodal interactions. While our comprehensive evaluations demonstrate the effectiveness and generalizability of IAG-Net, it is essential to acknowledge the limitations and challenges encountered. Since existing affordance datasets lack variety, performance in highly cluttered scenes or with objects exhibiting complex or subtle affordances may be affected.

In summary, our work represents a significant step forward in vision–language affordance grounding. There remain opportunities for further exploration and refinement. Further enhancements to IAG-Net will involve capitalizing on large-scale multimodal models and advancing robustness in challenging contexts. Another avenue for improvement involves integrating additional knowledge sources, such as knowledge graphs, to enhance the understanding of affordance grounding. Furthermore, we aim to integrate richer knowledge sources and explore expansive applications of our methodology, particularly in interactive robotics and complex task execution.