Enhancing Large Language Models with RAG for Visual Language Navigation in Continuous Environments

Abstract

The task of Visual Language Navigation in Continuous Environments (VLN-CE) aims to enable agents to comprehend and execute human instructions in real-world environments. However, current methods frequently face challenges such as insufficient knowledge and difficulties in obstacle avoidance when performing VLN-CE tasks. To address these challenges, this paper proposes a navigation approach guided by a Retrieval Augmented Generation (RAG) large language model (RAGNav). By constructing a navigation knowledge base, we leverage RAG to enhance the input of LLM, enabling the generation of more precise navigation plans. Furthermore, we introduce a Prompt Enhanced Obstacle Avoidance strategy (PEOA) to improve the flexibility and robustness of agents in complex environments. Experimental results indicate that our method not only increases the navigation accuracy of agents but also enhances their obstacle avoidance capabilities, achieving a 2% and 2.32% increase in success rates on the public datasets R2R-CE and RxR-CE, respectively.

1. Introduction

In recent years, Visual Language Navigation (VLN) [1] has emerged as a cutting-edge research area in artificial intelligence, garnering significant attention from the research community. VLN aims to integrate natural language processing (NLP) and computer vision (CV) technologies, enabling agents to recognize environmental information and understand natural language instructions for autonomous navigation. In traditional VLN settings, the environment is often abstracted as a graph structure, where the navigation task involves selecting one node from a set of neighboring nodes at each time step. However, this approach assumes ideal conditions for navigation between locations and nodes, which does not accurately represent real-world scenarios. To address these limitations and create more realistic navigation tasks, ref. [2] introduced the concept of Visual Language Navigation in Continuous Environments (VLN-CE). In this setting, agents operate in a 3D space, navigating to any accessible point by executing low-level navigation actions (e.g., moving forward 0.25 m or rotating 15 degrees). This shift allows for a more accurate representation of real-world navigation challenges.

Recent research on VLN-CE has increasingly focused on leveraging large language models (LLMs), which enhance the agents’ ability to understand complex instructions and effectively guide navigation. Ref. [3] proposed a zero-shot agent based on the Generative Pre-Trained Transformer(GPT), utilizing the capabilities of GPT-4 [4] for navigation decision-making in the R2R dataset. Another study [5] introduced the Navigational Chain of Thought (NavCoT) strategy, which employs parameter efficient, domain specific training to enable LLMs to make self-guided navigation decisions, thereby mitigating the domain gap. NavGPT further ehances navigation by taking visual observations, navigation history, and textual descriptions of future navigable directions as inputs to infer the agent’s current state and make informed decisions to approach the target. Despite these advancements, the reliance on LLMs’ internal knowledge alone has resulted in insufficient grounding for developing navigation plans, and there has been limited focus on improving the agents’ obstacle avoidance capabilities.

Previous research has shown that agents often become trapped by obstacles or move too close to walls in complex environments, which not only reduces task completion efficiency but also limits their potential for real-world applications. To enhance agents’ obstacle avoidance abilities, SafeVLN [6] was introduced, employing simulated 2D LiDAR occupancy masks to avoid predicting waypoints in obstacle areas. This method utilizes a “reselect after collision” strategy to prevent agents from becoming trapped in a loop of continuous collisions. Another study [7] presented a new controller that assists agents in escaping deadlocks through a trial-and-error heuristic, effectively minimizing performance losses due to sliding prohibitions. These approaches primarily focus on decision-making after encountering obstacles, with less emphasis on preemptively preventing collisions.

To overcome these limitations, we propose a novel visual language navigation approach for continuous environments that enhance large language models through Retrieval-Augmented Generation (RAG). Our approach aims to improve the accuracy and robustness of agents in continuous environments by constructing a navigation knowledge base, utilizing RAG for knowledge retrieval, which is then combined with the language understanding capabilities of LLMs. Additionally, we introduce a Prompt-Enhanced Obstacle Avoidance Strategy (PEOA), which integrates LLMs with structured prompts to facilitate better obstacle avoidance and generate fine-grained navigation plans.

In summary, our work makes the following key contributions:

1.

We introduce the use of RAG to retrieve knowledge from a navigation knowledge base, enhancing LLMs to generate more accurate navigation plans.

2.

We introduce a Prompt-Enhanced Obstacle Avoidance Strategy that leverages LLMs with structured prompts to improve obstacle avoidance and fine-grained navigation.

3.

Extensive experiments on publicly available datasets (R2R-CE and RxR-CE) demonstrate the superiority of our approach, achieving success rate improvements of at least 2% and 2.32% over baseline models, respectively.

2. Related Work

2.1. Vision and Language Navigation

Research on Visual Language Navigation (VLN) [1,8,9,10] has garnered increasing attention from scholars. Datasets such as R2R [1] and RxR [11] provide researchers with rich resources to explore the interaction between natural language and visual environments, thereby advancing technological progress in AI regarding the understanding of complex instructions, precise localization, and autonomous navigation. Early studies primarily focused on discrete environments, such as those simulated using the Matterport3D simulator [12]. In these settings, agents observe panoramic RGB images and can teleport step-by-step to target nodes based on natural language instructions. Basic agents typically incorporate cross-modal attention modules for cross-modal alignment, along with LSTM and transformer-based networks to model contextual history and decode sequences of navigation actions. They are trained using a hybrid approach that combines reinforcement learning and imitation learning [13,14,15].

To achieve more precise navigation planning in continuous environments, researchers have proposed a series of methods that improve navigation from various perspectives [5,6,7,16,17,18]. Among these, integrating contemporary large language model (LLM) technology with VLN has emerged as a particularly popular approach. Ref. [5] presents a strategy known as the Navigational Chain-of-Thought (NavCoT) to address the domain gap issue encountered when utilizing LLMs for VLN tasks. The core idea of NavCoT is to enable LLMs to perform self-guided navigation reasoning through parameter-efficient, domain-specific training, thereby simplifying the decision-making process and enhancing interpretability. Additionally, ref. [18] introduces a novel planning framework called AO-Planner, which aims to tackle some of the challenges and gaps faced by LLMs in VLN tasks.

Multimodal transformers, such as VL-BERT [19] and Flamingo [20], have also shown remarkable success in VLN tasks by jointly modeling visual and textual information. VL-BERT integrates visual and linguistic features through a unified transformer architecture, enabling the model to perform cross-modal reasoning effectively. Flamingo leverages large-scale pretraining on multimodal datasets to achieve state-of-the-art performance in tasks requiring visual and textual understanding. However, these methods often rely heavily on large-scale pretraining, and may struggle with generalization to unseen environments or instructions. In contrast, retrieval-augmented generation (RAG)-based methods, such as ours, focus on dynamically retrieving and utilizing task-specific knowledge, providing greater flexibility and adaptability in complex navigation scenarios. For example, our method outperforms VL-BERT and Flamingo in handling ambiguous instructions by leveraging external knowledge to disambiguate user intent.

2.2. Large Language Models

Since the introduction of BERT by Google in 2018 [21], Large Language Models (LLMs) such as GPT-2 [22] and GPT-3 [23] have demonstrated remarkable language understanding and generation capabilities through unsupervised learning on vast amounts of text data, achieving outstanding results on various NLP benchmark tests. Simultaneously, the application of LLMs in VLN is flourishing, with researchers exploring the integration of language understanding with visual perception, enabling agents to perform navigation tasks in complex environments based on natural language instructions. Related studies have investigated the use of LLMs to generate executable action plans, and the ${\mathit{LLM}}_{pla}$ method has been proposed for few-shot planning in embodied agents [23]. Furthermore, the Navigational Chain-of-Thought (NavCoT) strategy aids LLMs in self-guiding navigation decisions through efficient in-domain training, effectively mitigating domain gap issues [5].

Recent advancements have explored the application of LLMs in robotic navigation. For instance, study [24] demonstrates the use of pre-trained models for navigation tasks. Similarly, ref. [25] investigates the feasibility of using LLMs for path planning. However, these studies primarily focus on the general application of LLMs in navigation tasks and do not delve deeply into the specific challenges of obstacle avoidance in continuous environments.

Currently, the application of RAG [26] in LLMs has become a prominent trend. RAG models process queries and utilize techniques such as semantic search to retrieve relevant materials from associated text databases. They then leverage generative models to integrate this information with the input, providing richer contextual support for navigation tasks. Consequently, RAG-enhanced LLMs exhibit tremendous potential in visual language navigation within continuous environments, further improving the accuracy and effectiveness of navigation strategies.

2.3. Collision in Autonomous Navigation

In the field of robotic navigation, collision avoidance research has garnered significant attention [27,28,29]. For instance, imitation learning, a popular approach in this domain, has been employed in visual navigation [30] to guide agents with expert experience in effectively avoiding collisions. However, in the VLN-CE context, the challenges posed by longer navigation paths, complex environments, and additional requirements for instruction alignment hinder the acquisition of expert actions necessary for collision handling, thus limiting the applicability of imitation learning methods. ETPNav represents a recent study that addresses collisions within VLN-CE [31]. It designs a trial-and-error heuristic controller that avoids agents becoming trapped in collision zones by randomly selecting actions from a predefined action set. Ref. [6] introduces a classification of collisions in VLN-CE for the first time, defining three types: waypoint collisions, navigation collisions, and dynamic collisions. It also proposes the Safe-VLN algorithm, which includes two key components a waypoint predictor and a re-selection navigator to enhance the collision avoidance capabilities of the agent.

There are various approaches to enable effective obstacle avoidance for agents. Inspired by the application of prompts in large language models (LLMs), this paper primarily focuses on the role of prompts in enhancing obstacle avoidance capabilities and introduces the PEOA module to further improve this ability.

3. Proposed Approach

In this paper, we propose a navigation method guided by the RAG large language model, referred to as RAG-Nav. The entire process is illustrated in Figure 1. First, we construct a navigation knowledge base containing general conceptual knowledge related to areas and objects. Next, we utilize LLMext to analyze the input instructions I, extracting two sets of parameterized data, ${\mathit{O}}_n$ and ${\mathit{R}}_n$. Here, ${\mathit{O}}_n$ represents object-related parameters, while ${\mathit{R}}_n$ represents room-related parameters. These are combined with the most relevant knowledge K retrieved from the knowledge base through the RAG mechanism, which can be formulated as:

$$K=\mathrm{Retrieve}(I,\mathrm{Knowledge}\mathrm{Base}),$$

where Retrieve is the retrieval function that returns the most relevant knowledge K based on the input instruction I and the knowledge base. The extracted parameters ${\mathit{O}}_n$ and ${\mathit{R}}_n$ are obtained as:

$${\mathit{O}}_n,{\mathit{R}}_n=\mathrm{LLMext}\left(I\right).$$

The retrieved knowledge K, along with ${\mathit{O}}_n$, ${\mathit{R}}_n$, and the scene graph description ${\mathit{S}}_t$ generated by CLIP, are input to LLMpla to enhance the accuracy of the large language model when planning coarse-grained navigation ${\mathit{G}}_t$. This process is expressed as:

$${\mathit{G}}_t=\mathrm{LLMpla}(I,K,{\mathit{O}}_n,{\mathit{R}}_n,{\mathit{S}}_t).$$

Figure 1. The core of this framework consists of three large language models with different roles: ${\mathcal{LLM}}_{\mathrm{ext}}$, ${\mathcal{LLM}}_{\mathrm{pla}}$, and ${\mathcal{LLM}}_{\mathrm{oa}}$. The main architecture is divided into two primary modules. The first half analyzes the current RGB image using instructions and CLIP, and then combines this analysis with RAG to generate a plan for the next action. The second half comprises the PEOA module, which generates fine-grained actions for the agent at each step.

Figure 1. The core of this framework consists of three large language models with different roles: ${\mathcal{LLM}}_{\mathrm{ext}}$, ${\mathcal{LLM}}_{\mathrm{pla}}$, and ${\mathcal{LLM}}_{\mathrm{oa}}$. The main architecture is divided into two primary modules. The first half analyzes the current RGB image using instructions and CLIP, and then combines this analysis with RAG to generate a plan for the next action. The second half comprises the PEOA module, which generates fine-grained actions for the agent at each step.

At the fine-grained operation execution level, we use PEOA. The module uses coarse-grained target ${\mathit{G}}_t$ and scene description ${\mathit{S}}_t$ as input, and also the depth map data as auxiliary input. It is most critical is to judge the obstacles and distances that may be encountered according to the coarse-grained action and depth map data. Input to the ${\mathrm{LLM}}_{\mathrm{oa}}$ generates fine-grained decisions that do not encounter obstacles. Generates A set of parameterized discrete actions ${\mathit{A}}_t$. The fine-grained action generation formula is

$${\mathit{A}}_t={\mathrm{LLM}}_{\mathrm{oa}}({\mathit{G}}_t,{\mathit{S}}_t,\mathrm{Depth}\mathrm{Map}),$$

where ${\mathrm{LLM}}_{\mathrm{oa}}$ is the LLM responsible for generating discrete actions with obstacle avoidance.A concrete LLM example is shown in Figure 2.There is also a pseudo-code Table 1 that supports my approach.

3.1. Task Setup

We focus on a practical setting, VLN-CE [6]. Specifically, our task is to enable an agent to effectively navigate from a starting point to a target location within a 3D grid environment constructed from the Matterport3D dataset, guided by natural language instructions W = ${\left\{w_i\right\}}_{i=1}^L$. At each location, the agent receives panoramic RGB observations $\mathbf{O}=\{I^{RGB},I^d\}$, which include 12 RGB images and 12 depth images, captured at 12 evenly spaced horizontal orientations (0°, 30°, …, 330°). The action space comprises parameterized discrete actions, such as moving forward (0.25 m), rotating left or right (15°), and stopping. The agent greedily selects the navigation action with the highest predicted probability. Ongoing navigation is terminated if the agent triggers the STOP action or exceeds the maximum number of predictions for the target.

Figure 2. A simple example for LLM.

Figure 2. A simple example for LLM.

Table 1. Pseudocode for the RAG-Nav Method.

Step	Description	Functions/Inputs
1	Construct Navigation Knowledge Base	KnowledgeBase = BuildKnowledgeBase()
2	Analyze Input Instructions	O_n, R_n = LLMext(Instructions)
3	Retrieve Relevant Knowledge	RelevantKnowledge = RAG(KnowledgeBase, O_n, R_n)
4	Generate Coarse-Grained Navigation Goal (G_t)	G_t = LLMpla(O_n, R_n, RelevantKnowledge, S_t)
5	Generate Scene Graph Description	S_t = CLIP(InputImage)
6	Define Obstacle Avoidance Strategy	PEOA = DefineObstacleAvoidanceStrategy()
7	Input for Fine-Grained Action Execution	A_t = PEOA(G_t, S_t, DepthMap)
8	Execute Actions	ExecuteActions(A_t)
9	Evaluate Success Rate	SuccessRate = EvaluatePerformance()

Table 1. Pseudocode for the RAG-Nav Method.

Step	Description	Functions/Inputs
1	Construct Navigation Knowledge Base	KnowledgeBase = BuildKnowledgeBase()
2	Analyze Input Instructions	O_n, R_n = LLMext(Instructions)
3	Retrieve Relevant Knowledge	RelevantKnowledge = RAG(KnowledgeBase, O_n, R_n)
4	Generate Coarse-Grained Navigation Goal (G_t)	G_t = LLMpla(O_n, R_n, RelevantKnowledge, S_t)
5	Generate Scene Graph Description	S_t = CLIP(InputImage)
6	Define Obstacle Avoidance Strategy	PEOA = DefineObstacleAvoidanceStrategy()
7	Input for Fine-Grained Action Execution	A_t = PEOA(G_t, S_t, DepthMap)
8	Execute Actions	ExecuteActions(A_t)
9	Evaluate Success Rate	SuccessRate = EvaluatePerformance()

3.2. Building a Navigation Knowledge Base

The knowledge base is fundamentally designed to integrate and structure information for efficient retrieval and localization. Specifically, the navigation knowledge base [32] is tailored for VLN, aiming to enhance the agent’s comprehension and navigation accuracy.

This paper leverages scene understanding auxiliary tasks to explore adjacency relationships between regions and objects in discrete navigation environments. We first segment the scene into independent regions to ensure accurate connectivity statistics, where the connectivity between regions is based on the adjacency of nodes. For example, if region A contains node C, and node D in region B is adjacent to node C, then regions A and B are considered connected. Object proximity is determined by assuming adjacency when objects appear within the same view, and their proximity count is incremented accordingly.

After statistical analysis, we obtain region and object proximity count matrices. Outliers in these matrices are addressed, and they are then converted into proximity probability matrices using normalization to reduce the impact of extreme values. We employ the following normalization formula:

$$P_{i,j}=0.95\times {\displaystyle \frac{N_{i,j}-N_{i,\min }}{N_{i,\max }-N_{i,\min }}},$$

where $P_{i,j}$ represents the proximity probability of region or object, $N_{i,\min }$ and $N_{i,\max }$ represent the minimum and maximum values of the i-th row of the matrix N, respectively. This process not only enhances the agent’s cognitive capabilities regarding the layout of the environment but also lays the groundwork for the subsequent retrieval processes of the RAG, thereby improving the agent’s ability to navigate accurately and efficiently in complex environments.

3.3. Coarse-Grained Planning with RAG

In VLN systems, to enhance the global planning capabilities of agents during navigation tasks, we employ RAG techniques to extract relevant knowledge from our constructed navigation knowledge base, thereby supporting the global planning and decision-making processes of LLMs.

3.3.1. Knowledge Base Retrieval

For each navigation task, the agent uses a retrieval model (e.g., DPR or BM25) to extract relevant knowledge from the knowledge base based on the given instructions. For example, with the instruction “Go to the restroom and get a basin”, the agent extracts keywords like “restroom” and “basin” using an extraction model $L_{\mathrm{ext}}$ and retrieves related entries, such as $\{\mathrm{restroom},\mathrm{living}\mathrm{room},t_1\}$ and $\{\mathrm{basin},\sin \mathrm{k},t_2\}$, where $t_1$ and $t_2$ represent association probabilities. The top-k most relevant pieces of knowledge are then selected.

3.3.2. Enhanced Generation

Subsequently, the retrieved relevant knowledge is combined with the navigation instructions and the scene description of the current view $S_t$, and input into the planning model $L_{\mathrm{pla}}$. This allows us to leverage its powerful textual parsing capability to interpret the information and convert it into sub-goal descriptions $G_t$. Furthermore, to mitigate the inaccuracies introduced by the static planner $L_{\mathrm{pla}}$, we incorporate a Room and Object Awareness Scene Perceptor (ROASP) proposed in [33].

At each time step t, the agent perceives the environment and obtains a panoramic visual observation $O=\{I^{RGB},I^d\}$. The image features are extracted using the CLIP Image Encoder as follows:

$$f_r=E_c^v\left(o_i\right),$$

where $E_c^v(·)$ represents the CLIP image encoder. For each room category $C_r$ and each object category $C_o$, the text features are extracted using the pre-trained CLIP text encoder as follows:

$$f_r=E_c^t\left(T_r\right),$$

$$f_o=E_c^t\left(T_o\right),$$

where $E_c^t(·)$ represents the CLIP text encoder. Finally, the similarity score $S_r$ between the image feature $f_r$ and the text feature $f_r^T$ as well as the similarity score $S_o$ between the image feature $f_r$ and the text feature $f_o^T$ are respectively computed as:

$$S_r=S(f_r,f_r^T),$$

$$S_o=S(f_o,f_r^T).$$

We compute the average of the scores obtained from $S_r$ and select the highest-scoring prediction as $C_r$. The score $S_o$ increases with the proportion of the object in the view. We then select the top k matching objects with the highest scores as auxiliary environmental feedback. This approach allows ROASP not only to describe and predict the current environment but also to provide the large language model with crucial contextual information, thereby enabling the generation of more accurate ground truth $G_t$.

3.4. Prompt-Enhanced Obstacle Avoidance Strategy

In exploring the obstacle avoidance strategies for agents in 3D continuous environments, we propose the PEOA strategy, which involves designing formatted prompts to guide the agents’ obstacle avoidance behavior and fine-grained planning. The obstacle avoidance performance is enhanced by optimizing the prompt structure, ultimately resulting in the planning of a parameterized discrete action $A_t$.

In this context, we have designed various formats for prompts aimed at providing agents with richer and more specific environmental information and task instructions. For instance, a typical prompt may contain the following content:

“Nearby obstacles: {(wall, 0.60 m), (table, 0.36 m), (chair, 0.30 m, 31°–32°), (door, 0.2 m)}; sub-goal: {living room}.”

This format not only clarifies the target task but also details the potential obstacles that may be encountered along the planned route, as well as the distances $D_t$ to these obstacles obtained from depth images. By providing this information to the obstacle avoidance model $L_{\mathrm{o}}$, we can generate parameterized discrete actions that ensure each movement progresses toward the goal while minimizing deviations and errors.

4. Experimental Findings and Interpretation

4.1. Evaluation Metrics

For evaluation, we adopted several navigation metrics based on prior work. These include:

• Trajectory Length (TL): measuring average path length;
• Navigation Error (NE): the average distance between the final and target positions;
• Success Rate (SR): the proportion of paths with NE less than 3 m;
• Oracle Success Rate (OSR): SR with the Oracle stopping strategy;
• Success weighted by Path Length (SPL): to assess efficiency;
• Normalized Dynamic Time Warping (NDTW): to evaluate path fidelity;
• SDTW: penalizing NDTW by SR.

In the R2R-CE dataset, SR and SPL are the primary metrics, while RxR-CE emphasizes path fidelity, focusing on NDTW and SDTW.

4.2. Datasets

This study evaluates performance on the R2R-CE and RxR-CE datasets. R2R-CE contains 5611 shortest path trajectories, with each trajectory paired with approximately three English instructions, averaging 9.89 m in length and 32 words per instruction. Val Seen includes novel paths and instructions within known scenes, while Val Unseen introduces novel paths, instructions, and scenes.

In contrast, RxR-CE is larger and more challenging, with instructions averaging 120 words and paths averaging 15.23 m. Additionally, agents in RxR-CE face stricter constraints, including a larger chassis radius (0.18 m) and a prohibition on sliding along obstacles, increasing the risk of collisions.

4.3. Implementation Details

Our model was trained on a Linux server equipped with two NVIDIA RTX 3090 GPUs (NVIDIA Corporation, Santa Clara, CA, USA). During the training process, we set the batch size to 8 and the learning rate to $2\times {10}^{-5}$, training for a total of 20,000 iterations. The optimal model was selected based on performance on the validation set with unseen data.

For the LLM, we utilized the Llama-3-8B-Instruct [34] model for navigation planning. For the scene perception module, we retained the top three object prediction results for each location. We built our agent using the Habitat [35] platform.

4.4. Comparison with SOTA Methods

4.4.1. R2R-CE

Table 2 shows that our proposed method demonstrates superior performance compared to state-of-the-art models, particularly in two critical metrics: SR and SPL. Our method achieved an SPL of 60 and an SR of 68 on the Val Seen set, outperforming all other models, including Reborn and ETPNav. Additionally, our method achieved the lowest NE of 3.66 and the highest OSR of 74 on the Val Seen set, demonstrating exceptional navigation precision and potential for success under optimal conditions. On the Val Unseen set, we obtained SPL and SR scores of 51 and 58, respectively, along with a low NE of 4.34 and a high OSR of 66, indicating strong generalization capabilities. Furthermore, on the Test Unseen set, we maintained SPL and SR scores of 50 and 57, respectively, with a competitive NE of 4.85 and an OSR of 65, highlighting robust performance in challenging unseen environments. While our TL is slightly longer compared to some models, it remains competitive with state-of-the-art methods, balancing path efficiency with task success. These results affirm the effectiveness and robustness of our method in navigation tasks, establishing it as a leading approach in the field.

4.4.2. RxR-CE

Table 3 presents a performance comparison of our model with several other methods under different evaluation conditions. The Seq2Seq model employs a sequence-to-sequence architecture, typically used for prediction and generation tasks involving sequential data. CWP-CMA and CWP-RecBERT utilize conditional weighting models and contextual information to optimize navigation decisions and improve success rates, while ETPNav significantly enhances navigation capabilities through an augmented task planning mechanism. On the unseen validation set, ETPNav achieved a NE of 5.64 and a SR of 54.79, whereas our model attained values of 4.58 and 56.85 for the corresponding metrics, demonstrating a superior success rate and indicating its better performance in unknown environments.

Table 2. Comparison with state-of-the-art methods on R2R-CE dataset.

Model	Val Seen					Val Unseen					Test Unseen
	TL	NE	OSR	SR	SPL	TL	NE	OSR	SR	SPL	TL	NE	OSR	SR	SPL
Seq2Seq [1]	9.26	7.12	46	37	35	8.64	7.37	40	32	30	8.85	7.91	36	28	25
SASRA [36]	8.89	7.71	-	36	34	7.89	8.32	-	24	22	-	-	-	-	-
HPN [37]	8.54	5.48	53	46	43	7.62	6.31	40	36	34	8.02	6.65	37	32	30
CM2 [38]	12.05	6.10	51	43	35	11.54	7.02	42	34	28	13.90	7.70	39	31	24
WS-MGMAP [39]	10.12	5.65	52	47	43	10.00	6.28	48	39	34	12.30	7.11	45	35	28
CWP-RecBERT [40]	12.50	5.02	59	50	44	12.23	5.74	53	44	39	13.51	5.89	51	42	36
Sim2Sim [41]	11.18	4.67	61	52	44	10.69	6.07	52	43	36	11.43	6.17	52	44	37
Reborn [42]	10.29	4.34	67	59	56	10.06	5.40	57	50	46	11.47	5.55	57	49	45
ETPNav [7]	11.78	3.95	72	66	59	11.99	4.71	65	57	49	12.87	5.12	63	55	48
Ours	12.11	3.66	74	68	60	11.69	4.34	66	58	51	12.66	4.85	65	57	50

Table 2. Comparison with state-of-the-art methods on R2R-CE dataset.

Model	Val Seen					Val Unseen					Test Unseen
	TL	NE	OSR	SR	SPL	TL	NE	OSR	SR	SPL	TL	NE	OSR	SR	SPL
Seq2Seq [1]	9.26	7.12	46	37	35	8.64	7.37	40	32	30	8.85	7.91	36	28	25
SASRA [36]	8.89	7.71	-	36	34	7.89	8.32	-	24	22	-	-	-	-	-
HPN [37]	8.54	5.48	53	46	43	7.62	6.31	40	36	34	8.02	6.65	37	32	30
CM2 [38]	12.05	6.10	51	43	35	11.54	7.02	42	34	28	13.90	7.70	39	31	24
WS-MGMAP [39]	10.12	5.65	52	47	43	10.00	6.28	48	39	34	12.30	7.11	45	35	28
CWP-RecBERT [40]	12.50	5.02	59	50	44	12.23	5.74	53	44	39	13.51	5.89	51	42	36
Sim2Sim [41]	11.18	4.67	61	52	44	10.69	6.07	52	43	36	11.43	6.17	52	44	37
Reborn [42]	10.29	4.34	67	59	56	10.06	5.40	57	50	46	11.47	5.55	57	49	45
ETPNav [7]	11.78	3.95	72	66	59	11.99	4.71	65	57	49	12.87	5.12	63	55	48
Ours	12.11	3.66	74	68	60	11.69	4.34	66	58	51	12.66	4.85	65	57	50

Although our model scored 52.76 on the SPL, slightly higher than ETPNav’s score of 50.83, it achieved a NE of 5.72 and an SR of 53.53 on the unseen test set. In comparison, ETPNav recorded scores of 6.99 and 51.21, respectively. Overall, our model exhibited outstanding performance across various evaluation conditions, particularly in terms of success rate and navigation efficiency. This demonstrates its effectiveness and robustness in complex environments, further validating its advantages in task execution.

Table 3. Comparison with state-of-the-art methods on RxR-CE dataset. The uparrow (↑) indicates that higher values are better, while the downarrow (↓) indicates that lower values are better.

Methods	Val Seen					Val Unseen					Test Unseen
	NE↓	SR↑	SPL↑	NDTW↑	SDTW↑	NE↓	SR↑	SPL↑	NDTW↑	SDTW↑	NE↓	SR↑	SPL↑	NDTW↑	SDTW ↑
Seq2Seq [1]	-	-	-	-	-	-	-	-	-	-	12.10	13.93	11.96	30.86	11.01
CWP-CMA [40]	-	-	-	-	-	8.76	26.59	22.16	47.05	-	10.40	24.08	19.07	37.39	18.65
CWP-RecBERT
citepp36	-	-	-	-	-	8.98	27.08	22.65	46.71	-	10.40	24.85	19.61	37.30	19.05
ETPNav [7]	5.03	61.46	50.83	66.41	51.28	5.64	54.79	44.89	61.90	45.33	6.99	51.21	39.86	54.11	43.30
Ours	4.14	63.27	52.76	68.43	53.87	4.58	56.85	46.34	63.08	47.64	5.72	53.53	41.10	55.73	44.92

Table 3. Comparison with state-of-the-art methods on RxR-CE dataset. The uparrow (↑) indicates that higher values are better, while the downarrow (↓) indicates that lower values are better.

Methods	Val Seen					Val Unseen					Test Unseen
	NE↓	SR↑	SPL↑	NDTW↑	SDTW↑	NE↓	SR↑	SPL↑	NDTW↑	SDTW↑	NE↓	SR↑	SPL↑	NDTW↑	SDTW ↑
Seq2Seq [1]	-	-	-	-	-	-	-	-	-	-	12.10	13.93	11.96	30.86	11.01
CWP-CMA [40]	-	-	-	-	-	8.76	26.59	22.16	47.05	-	10.40	24.08	19.07	37.39	18.65
CWP-RecBERT
citepp36	-	-	-	-	-	8.98	27.08	22.65	46.71	-	10.40	24.85	19.61	37.30	19.05
ETPNav [7]	5.03	61.46	50.83	66.41	51.28	5.64	54.79	44.89	61.90	45.33	6.99	51.21	39.86	54.11	43.30
Ours	4.14	63.27	52.76	68.43	53.87	4.58	56.85	46.34	63.08	47.64	5.72	53.53	41.10	55.73	44.92

4.5. Ablation Study

We conducted a series of ablation experiments on the R2R-CE dataset. All results presented in this section are reported on the validation unseen split.

Table 4 presents the contributions of the main modules of the model. The first row represents the Transformer-based framework without any of the proposed modules. The second row utilizes RAG to extract knowledge from the knowledge base, resulting in a 3.6% increase in success rate, indicating that the knowledge effectively guides navigation decisions. The third row adds the ROASP module, yielding results superior to those of the previous two rows, demonstrating that ROASP enables the agent to better perceive the environment. Finally, the last row incorporates an intelligent obstacle avoidance strategy, significantly enhancing model performance, particularly in terms of SPL, which improved by 2.5%. This indicates that the strategy facilitates efficient fine-grained navigation planning.

Table 4. Performance Metrics with Different Configurations.

RAG	ROASP	PEOA	TL	NE	OSR	SR	SPL
×	×	×	10.31	5.59	56.63	49.86	45.32
✔	×	×	11.64	5.03	59.76	53.42	47.16
✔	✔	×	11.79	4.80	62.59	55.83	48.67
✔	✔	✔	11.69	4.34	65.83	58.32	51.09

Table 4. Performance Metrics with Different Configurations.

RAG	ROASP	PEOA	TL	NE	OSR	SR	SPL
×	×	×	10.31	5.59	56.63	49.86	45.32
✔	×	×	11.64	5.03	59.76	53.42	47.16
✔	✔	×	11.79	4.80	62.59	55.83	48.67
✔	✔	✔	11.69	4.34	65.83	58.32	51.09

Table 5 examines the impact of the number of relevant pieces of information retrieved from RAG on model performance. It is evident that the success rate tends to decrease as the number of retrieved items increases. However, the SPL performance is optimal when Top-K is set to 3. A lower number of retrieved items may result in the omission of the most useful relevant information, while an excessive number can lead to a decline in overall performance.

Table 5. Top-K Ablations.

Top-K	TL	NE	OSR	SR	SPL
1	11.92	4.38	65.97	58.44	50.80
3	11.69	4.34	65.83	58.32	51.09
5	11.63	4.45	65.02	57.10	50.43
10	12.43	4.60	65.81	57.68	49.18

Table 5. Top-K Ablations.

Top-K	TL	NE	OSR	SR	SPL
1	11.92	4.38	65.97	58.44	50.80
3	11.69	4.34	65.83	58.32	51.09
5	11.63	4.45	65.02	57.10	50.43
10	12.43	4.60	65.81	57.68	49.18

Table 6 presents the results of our method implemented on different LLMs. We selected three of the most popular models currently available. The data indicates that, overall, Llama3 consistently achieves the best performance, although there is a slight decrease in the success rate (SR) compared to GPT-3.0. This suggests that Llama3 exhibits superior reasoning capabilities in this domain.

Table 6. Different LLM Ablations.

LLM	TL	NE	OSR	SR	SPL
GPT3.0	11.77	4.43	64.77	58.34	50.96
qwen2	11.91	4.52	65.21	57.66	49.74
llama3	11.69	4.34	65.83	58.32	51.09

Table 6. Different LLM Ablations.

LLM	TL	NE	OSR	SR	SPL
GPT3.0	11.77	4.43	64.77	58.34	50.96
qwen2	11.91	4.52	65.21	57.66	49.74
llama3	11.69	4.34	65.83	58.32	51.09

Table 7 compares different prompt structures for PEOA. Structure 1 is a simple query: There are obstacles, and the goal is to reach the living room. What should you do? Structure 2 provides more details: Environmental information: Obstacles {wall, table, chair, door}. Target location: living room. Note: Distance to obstacles is 0.2 m–0.6 m. Suggested actions: {move forward, turn left, turn right}. Structure 3 is the most detailed: Current environment: Nearby obstacles {(wall, 0.60 m), (table, 0.36 m), (chair, 0.30 m, 31°–32°), (door, 0.2 m)}. Target: Sub-goal: living room. The results show that Structure 3 achieves the highest SPL and performs well across other metrics. Despite a slightly lower SR than Structure 2, Structure 3 is preferred for its concise yet informative content, enhancing the agent’s environmental understanding.

Table 7. Performance Metrics for Different Prompt Structures.

Prompt	TL	NE	OSR	SR	SPL
Structure 1	11.76	4.50	65.72	57.92	50.62
Structure 2	11.79	4.41	65.92	58.02	50.77
Structure 3	11.69	4.34	65.83	58.32	51.09

Table 7. Performance Metrics for Different Prompt Structures.

Prompt	TL	NE	OSR	SR	SPL
Structure 1	11.76	4.50	65.72	57.92	50.62
Structure 2	11.79	4.41	65.92	58.02	50.77
Structure 3	11.69	4.34	65.83	58.32	51.09

4.6. Qualitative Examples

As depicted in Figure 3, the agent successfully navigated the environment utilizing RAG-enhanced LLM decision-making and PEOA obstacle avoidance strategies. Beginning at the initial point, the agent proceeded to point A, where it paused adjacent to a potted plant. Confronted with two potential paths, the agent inferred a stronger association between the bathroom and bedroom compared to the staircase, prompting a right turn towards the bedroom.

Employing the PEOA strategy, the agent effectively processed wall proximity and angular data (wall, 0.3 m, 30°–68°), thereby avoiding collisions and entering the bedroom, where it halted at point B after detecting a television. Upon failing to locate the bathroom en route to point C, the agent retraced its trajectory to point B, subsequently navigating to point D, and ultimately entering the bathroom.

Figure 3. Example of navigation using RAGNav.

Figure 3. Example of navigation using RAGNav.

5. Conclusions

This paper introduces RAG-Nav, a novel navigation method that integrates Retrieval-Augmented Generation (RAG) with large language models to enhance navigation accuracy and robustness in complex, continuous 3D environments. By retrieving relevant information from a constructed knowledge base, RAG-Nav augments the generative capabilities of the language model for precise coarse-grained planning. Additionally, it incorporates an intelligent obstacle avoidance strategy using structured prompts for fine-grained planning, effectively mitigating collision risks.

Experimental results on the public datasets R2R-CE and RxR-CE demonstrate that RAG-Nav achieves a 2% and 2.32% increase in navigation success rates, respectively, confirming its effectiveness in improving navigation accuracy avoidance capabilities.