Case-Based Deduction for Entailment Tree Generation

Abstract

Maintaining logical consistency in structured explanations is critical for understanding and troubleshooting the reasoning behind a system’s decisions. However, existing methods for entailment tree generation often struggle with logical consistency, resulting in erroneous intermediate conclusions and reducing the overall accuracy of the explanations. To address this issue, we propose case-based deduction (CBD), a novel approach that retrieves cases with similar logical structures from a case base and uses them as demonstrations for logical deduction. This method guides the model toward logically sound conclusions without the need for manually constructing logical rule bases. By leveraging a prototypical network for case retrieval and reranking them using information entropy, CBD introduces diversity to improve in-context learning. Our experimental results on the EntailmentBank dataset show that CBD significantly improves entailment tree generation, achieving performance improvements of 1.7% in Task 1, 0.6% in Task 2, and 0.8% in Task 3 under the strictest Overall AllCorrect metric. These findings confirm that CBD enhances the logical consistency and overall accuracy of AI systems in structured explanation tasks.

1. Introduction

Neural networks have demonstrated exceptional performance across various tasks [1]. However, these models often remain opaque to human understanding, prompting researchers to explore methods for providing more interpretable explanations. Three main approaches have emerged: highlighting relevant features in the input data to explain decisions [2], generating natural language explanations alongside predicted answers [3], and employing structured reasoning processes [4].

Our work focuses on leveraging structured reasoning processes to explain decisions in a clear and understandable manner. Traditionally, structured explanations have been constructed using reasoning chains, as demonstrated in earlier studies [2,5]. More recent research has explored entailment trees as an alternative method for generating explanations. Entailment trees offer a more structured and manageable way of conducting reasoning, making them a promising tool for explanation generation [6,7,8]. Entailment trees, as proposed by [6], illustrate the logical connections between multiple premises, systematically organizing the reasoning chains that underpin question-answering tasks. This approach provides a transparent means of understanding how hypotheses are derived from a series of entailment steps based on multiple premises.

Previous methods for constructing entailment trees typically employed a single, sequential process that generated the entire tree at once [9]. For instance, the sequential form of the entailment tree in Figure 1b can be represented as “$sen{t}_1\&sen{t}_6\to in{t}_1:$ Chocolate is a kind of solid substance $;in{t}_1\&sen{t}_2\to in{t}_2:$ Chocolate in solid state has definite shape $;\dots ;in{t}_2\&in{t}_4\to \mathrm{Hypothesis}$”. However, this approach often struggled to generate complete and accurate entailment trees, leading to unreliable or hallucinatory results. To mitigate these issues, recent studies [7,8,10,11] have adopted an iterative approach, where one-step entailments are generated first and then iteratively expanded to form the entire tree. Despite these improvements, a common challenge persists: ensuring logical consistency within individual steps. As shown in Figure 1c, given the sentences “Chocolate is usually a solid” and “Chocolate is a kind of substance”, the correct entailment is “Chocolate is a kind of solid substance”. However, a pretrained language model might incorrectly predict that “Substance is usually a solid”, highlighting the issue of logical inconsistency.

To address these challenges, researchers have explored symbolic methods [12,13,14] known for their logical reasoning capabilities. However, these methods face difficulties in enumerating all first-order logical rules due to the diverse nature of natural language expressions, and constructing rule bases through human engineering is time-consuming. A promising alternative is case-based learning [15,16,17,18,19], where systems operate on the principle that similar problems have similar solutions. Case-based reasoning is a method in artificial intelligence where solutions to new problems are derived by referring to similar, previously solved cases. By leveraging past cases to solve new problems, case-based reasoning allows for guiding logical generation using patterns from past gold cases without the need for explicit rule induction.

The case-based reasoning framework typically follows a retrieve-reuse-refine paradigm [20]. A key challenge in this approach is retrieving similar cases, as logical patterns can manifest in various natural language forms. To address this, we propose using prototypical networks [21] to represent logical patterns as prototype embeddings. Prototypical networks are a type of neural network that learns a metric space in which classification is performed by computing the distance to prototype representations of each class. This method captures the essence of logical patterns, enhancing the retrieval process within the case-based reasoning framework. Our research focuses on three prevalent logical patterns: conjunction, if–then, and substitution. The prototypical network generates three prototype embeddings, one for each pattern, by averaging the embeddings of cases in each category. During reasoning, these prototype embeddings help identify the logical pattern of the current step, narrowing the retrieval scope within our proposed framework, CBD.

Once similar cases are retrieved, they are used as demonstrations for in-context learning. In-context learning benefits from diverse demonstrations [22,23,24] and is sensitive to their order [25,26]. To enhance diversity, we incorporate information entropy [27], a measure from information theory used to quantify the diversity of information in a set, which is applied here to rerank the retrieved cases. We then select the top-n cases as demonstrations. Inspired by curriculum learning [28], we organize the demonstrations by difficulty, arranging them from easy (low information entropy) to difficult (high information entropy). The deduction process is carried out iteratively until the target hypothesis is proven or the maximum deduction depth is reached. To facilitate iterative generation, we utilize the controller from MetGen [8].

Our contributions are as follows:

• Case-based learning for step generation: We implement case-based learning to eliminate the need for manual construction of logical rules. The retrieved cases are utilized as demonstrations for in-context learning, effectively guiding large language models in logical text generation.
• Prototypical networks for inducing abstract logical structures: To improve the efficiency of identifying cases with similar logical structures, we employ a prototypical network to capture implicit logical patterns. This enables more targeted case retrieval and provides insights into logical pattern attribution.
• Integration of information entropy to enhance diversity: We incorporate information entropy to increase the diversity of retrieved cases, enhancing the model’s ability to learn underlying logical patterns and improving in-context learning.

The rest of this paper is organized as follows: Section 2 defines the problem and provides background knowledge on case-based reasoning and prototypical networks. Section 3 describes the architecture and implementation of CBD, detailing the case-based deduction process for entailment tree generation. Section 4 outlines our experimental setup, including the benchmark dataset, baseline models, evaluation metrics, and implementation details. Section 5 presents our results, discusses their implications, and highlights the advantages of CBD over existing methods. Section 6 reviews related work in entailment tree generation and explanation for question-answering tasks, situating our contributions within the broader research context. Finally, Section 8 concludes the paper by summarizing our contributions and discussing potential directions for future work.

2. Background

2.1. Task Definition

Entailment tree generation, as illustrated in Figure 1, involves creating an entailment tree with a hypothesis ($hyp$), which is a declarative form of a question and answer, and a set of natural language sentences $\mathcal{K}=\{s_1,s_2,\dots ,s_n\}$, referred to as facts. Here, n represents the number of candidate facts. Entailment tree generation is a process that organizes reasoning into steps to demonstrate how a conclusion is derived from a set of premises. The objective is to generate a valid entailment tree $T_{pred}$, where the hypothesis ($hyp$) serves as the root node, the leaves are facts selected from $\mathcal{K}$, and the intermediate nodes consist of novel intermediate facts ($in{t}_m$) representing conclusions generated by the model. The tree $T_{pred}$ is considered valid if each nonleaf node is a valid entailment of its immediate children. The annotated gold tree is denoted as $T_{gold}$, with its leaf facts forming $S_{gold}$. Following [6], we consider three increasingly challenging tasks based on the scope of candidate facts $\mathcal{K}$:

• Task 1 (no-distractor): $\mathcal{K}$ contains only the gold standard facts (${\mathcal{K}}_{gold}$).
• Task 2 (distractor): $\mathcal{K}$ includes ${\mathcal{K}}_{gold}$ along with 15–20 distractor facts, challenging the model’s ability to distinguish relevant information.
• Task 3 (full-corpus): This scenario uses the entire corpus as $\mathcal{K}$, presenting the most comprehensive and challenging setting for entailment tree generation. In our experiments, the entire corpus refers to the WorldTree corpus.

2.2. Case-Based Reasoning

Case-based reasoning (CBR) is a well-established problem-solving approach where new problems are addressed by adapting solutions from past cases that share similar characteristics [16,17,29]. It follows the retrieve-reuse-refine paradigm [30], which has been widely applied in fields such as legal question answering [31] and numerical reasoning [32], where reasoning from past experiences proves valuable for addressing new challenges [33]. CBR mimics human learning through analogies, functioning similarly to analogical reasoning [34]. Numerous studies [35,36,37,38] have demonstrated the effectiveness of CBR in various domains. In knowledge base question answering (KBQA), CBR has been applied to predict answers in a Knowledge Base (KB) given a query node and a relation [35,36,37]. Additionally, CBR has shown promise in semantic parsing tasks, where logical forms of similar cases are retrieved and passed through seq2seq models along with a target query to generate semantic outputs [39,40,41].

Building on prior CBR research [20,29,42], our work CBD extends the principles of CBR to the task of entailment tree generation. Rather than merely solving new problems, CBD retrieves cases with similar logical structures from a preannotated case base and adapts them as demonstrations for generating logical entailment steps. This approach ensures that the generated entailment trees maintain both structural and logical consistency, making it particularly suitable for complex, multistep reasoning tasks. CBD introduces unique contributions such as logical pattern retrieval and the integration of innovative techniques like prototypical networks and information entropy-based reranking, which we detail in the Methodology section. To our knowledge, this is the first application of case-based reasoning in entailment tree generation.

2.3. Prototypical Networks

Prototypical networks (ProtoNet) [21] pioneered the introduction of prototypes into deep learning. This approach suggests that each category can be succinctly represented by a single prototype within the feature space. ProtoNet computes prototype vectors by averaging instance vectors within each category, leveraging metric-based comparisons between prototypes and query instances to make predictions [43,44,45]. For instance, Yue et al. [46] explored cross-domain instance-to-prototype matching for Unsupervised Domain Adaptation (UDA), while Pan et al. [47] used prototype learning to bridge domain gaps and construct classifiers in the target domain for UDA. The effectiveness of prototype-based models highlights their ability to capture class-level semantic features through representative prototype embeddings of instances from the same classes.

Given the intrinsic similarity between prototypes and the underlying logical patterns of entailment steps, it is both natural and effective to introduce prototypes into case-based learning for modeling logical patterns. Researchers [43,48,49,50] have further explored the interpretability of prototypical networks. By applying this concept to text classification, they demonstrated how prototypical texts can aid in interpreting predictions. Building on this foundation, we employ a prototypical network to capture abstract logical patterns, which not only facilitates a more focused search for relevant cases but also provides valuable insights into the attribution of logical patterns, thereby contributing to clearer decision-making processes.

3. Case-Based Deduction

The objective of CBD is to construct an entailment tree based on a set of knowledge $\mathcal{K}=\{s_1,s_2,\dots ,s_n\}$ and a declarative hypothesis $hyp$, which consists of a question and its correct answer. For example, given the hypothesis “the shape of the chocolate changes when the chocolate melts”, our task is to build an explanation that justifies $hyp$ by selecting and combining pairs of sentences to generate new intermediate conclusions, thereby forming an entailment tree. This is achieved through a case-based reasoning approach, which involves three main phases:

• Retrieval Phase: Relevant cases are retrieved to serve as demonstrations, ensuring the logical consistency of the generated results. A prototypical network predicts the logical pattern of promising steps, categorizing them into one of three types: if–then, substitution, or conjunction relations. After determining the logical pattern, cosine similarity is used to identify the most relevant cases for the current fact pairs.
• Reuse Phase: To mitigate bias toward entities that might result from the cosine similarity method and to encourage diversity among the selected cases, we integrate information entropy into the ranking procedure. Retrieved cases are then ranked based on their information entropy scores.
• Refine Phase: The ordered cases are concatenated and used as demonstrations to guide a pretrained language model for logical text generation.

Through iterative application of this process, the entire entailment tree is constructed. Figure 2 provides a comprehensive overview of the CBD framework. Further details are discussed in the following sections.

3.1. Retrieve Phase

Using the controller proposed by Hong et al. [8], we identify promising steps in the reasoning process. For instance, the sentences “Chocolate is usually a solid” and “Chocolate is a kind of substance” exemplify such a promising step. The case-based deduction module then generates intermediate conclusions for each identified step. This process begins by retrieving analogous cases that follow the same logical pattern, leveraging a prototypical network to capture these implicit patterns. The prototypical network helps recognize abstract logical patterns in the data, which ensures that retrieved cases align with the input logical structures. By doing so, the network facilitates the retrieval of cases that are more semantically relevant and helps improve the logical consistency of intermediate conclusions. We categorize the types of inferences into three prevalent categories, as shown in

Table 1. The distribution of logical categories in individual entailment tree steps based on a sample of 100 random steps from the training corpus. Here, $s_1$ and $s_2$ represent input sentences, while $in{t}_1$ represents the deduction outcome (intermediate nodes in the entailment tree).

Categories	Logical Pattern	Proportion	Example
Substitution	$s_1: \forall x\in XP\left(x\right)$	46%	$s_1:$ coal is a kind of fossil fuel.
	$s_2: a\in X$		$s_2:$ fossil fuels are acquired by mining the lithosphere.
	$in{t}_1: P\left(a\right)$		$in{t}_1:$ coal is acquired by mining the lithosphere.
If-then	$s_1: P\left(x\right)\to Q\left(x\right)$	33%	$s_1:$ if a thing converts something into something else then that thing uses that something.
	$s_2: P\left(x\right)$		$s_2:$ bees convert nectar into honey for food.
	$in{t}_1: Q\left(x\right)$		$in{t}_1:$ bees use nectar for food.
Conjunction	$s_1: P\left(x\right)$	21%	$s_1:$ sunlight is a kind of electromagnetic radiation.
	$s_2: Q\left(x\right)$		$s_2:$ sunlight can shine through a window.
	$in{t}_1: P\wedge Q\left(x\right)$		$in{t}_1:$ sunlight is a kind of electromagnetic radiation that can shine through a window.

. The case base, denoted as $\mathcal{C}$, consists of three logical categories: ${\mathcal{C}}_{conj}$, ${\mathcal{C}}_{if}$, and ${\mathcal{C}}_{sub}$.

To encode the semantics of promising steps into embeddings, we use the DeBERTa model [51], chosen for its effectiveness in capturing sentence semantics. Although other neural architectures, such as recurrent neural networks (RNNs), could also be utilized for this purpose, DeBERTa is our preferred choice. After computing the sentence pair embeddings within the case base $\mathcal{C}$, we use a prototypical network to generate a prototype for each logical pattern based on these embeddings. By measuring the distance between the embedding of a promising step and each prototype, we can accurately determine the logical pattern corresponding to that step.

Given a promising step $S_{pair}=(s_1,s_2)$, where $s_1,s_2\in \mathcal{K}$ and are represented as $s_1=\{w_1,\dots ,w_k\}$ and $s_2=\{w_1^{\prime },\dots ,w_k^{\prime }\}$, we apply the DeBERTa model to encode the sentence pair into a continuous low-dimensional embedding ${\mathbf{H}}_{pair}$, capturing the semantics of the sentence pair. For all input tokens of the sentence pairs, we concatenate the two sentences using the specific token [SEP], forming $S_{pair}=\{\left[\mathrm{CLS}\right],w_1,\dots ,w_k,\left[\mathrm{SEP}\right],w_1^{\prime },\dots ,w_k^{\prime },\left[\mathrm{SEP}\right]\}$. The embedding is computed as follows:

$${\mathbf{H}}_{pair}=\mathrm{DeBERTa}\left({\varphi }^{emb}\left(S_{pair}\right)\right),$$

(1)

where ${\varphi }^{emb}$ refers to the embedding layer of DeBERTa, which maps discrete words into continuous embeddings. The output layer is represented as $\{{\mathbf{h}}_{pair},{\mathbf{h}}_1,\dots ,{\mathbf{h}}_k,{\mathbf{h}}_{\left[\mathrm{SEP}\right]},{\mathbf{h}}_1^{\prime },\dots ,{\mathbf{h}}_k^{\prime }\}$. The vector representation of the sentence pair, corresponding to the token [CLS], is denoted as ${\mathbf{h}}_{pair}$.

In our approach, the prototypical network represents each logical pattern with a single vector, known as a prototype. The logical patterns $\mathcal{L}=\{conj,if,sub\}$ correspond to conjunction, if–then, and substitution. We calculate prototypes by averaging all sentence pair embeddings within the annotated entailment step set for each category:

$$\begin{array}{cc}\hfill {\overline{\mathbf{h}}}_{conj}& ={\displaystyle \frac{1}{N_{conj}}}\sum _{\ell =1}^{N_{conj}}{\mathbf{h}}_{conj}^{\ell },\hfill \\ \hfill {\overline{\mathbf{h}}}_{if}& ={\displaystyle \frac{1}{N_{if}}}\sum _{\ell =1}^{N_{if}}{\mathbf{h}}_{if}^{\ell },\hfill \\ \hfill {\overline{\mathbf{h}}}_{sub}& ={\displaystyle \frac{1}{N_{sub}}}\sum _{\ell =1}^{N_{sub}}{\mathbf{h}}_{sub}^{\ell },\hfill \end{array}$$

(2)

where ${\overline{\mathbf{h}}}_{conj}$ is the prototype embedding for the conjunction pattern, and ${\mathbf{h}}_{conj}^{\ell }$ represents the ℓ-th sentence pair embedding within the conjunction category. The same process is applied to calculate the prototypes for the other two logical categories: ${\overline{\mathbf{h}}}_{if}$ for the if–then pattern and ${\overline{\mathbf{h}}}_{sub}$ for the substitution pattern.

Next, we calculate the probabilities $\widehat{p}$ for predicting the logical pattern to the promising step $x=(s_o,s_t)$ using the following equations:

$$\begin{array}{cc}\hfill {\mathbf{h}}_x& =\mathrm{DeBERTa}{\left({\varphi }^{emb}\left(x\right)\right)}_{\left[CLS\right]},\hfill \\ \hfill D_{euc}^i& =\sqrt{\sum _{m=1}^d{\left({\left[{\mathbf{h}}_x\right]}_m-{\left[{\overline{\mathbf{h}}}_i\right]}_m\right)}^2},\hfill \\ \hfill {\widehat{p}}_i& ={\displaystyle \frac{exp\left(-D_{euc}^i\right)}{{\sum }_{j=1}^{\left|\mathcal{L}\right|}exp\left(-D_{euc}^j\right)}},\hfill \end{array}$$

(3)

where ${\mathbf{h}}_x$ is the vector representation of the promising step, ${\overline{\mathbf{h}}}_i$ is the prototype embedding for the i-th logical category, and $D_{euc}^i$ is the Euclidean distance between two vectors. ${\widehat{p}}_i$ represents the probability of the i-th logical category, with $\widehat{p}=max\left({\widehat{p}}_i\right)$ as the predicted category. Euclidean distance is used here, following findings from [21], where it was shown to outperform other distance metrics.

The loss function is then defined as

$$\mathcal{L}=-\sum _{i=1}^N\left(t_{i}log\left({\widehat{p}}_i\right)\right),$$

(4)

where $t_i$ is 1 if the i-th case is predicted correctly; otherwise, $t_i$ is 0, and N is the total number of samples. We minimize the cross-entropy loss between the prediction and the ground truth. Through the prototypical network, we narrow the retrieval scope to focus on cases with analogical logical patterns. For instance, as shown in Figure 2, the promising step ($s_1,s_6$) exemplifies a conjunction pattern. Once the logical pattern is determined, we proceed to gather similar cases for demonstration.

We hypothesize that employing analogous single-step deduction can effectively assist a large language model in performing logical deductions. Through these illustrative examples, the model can better comprehend underlying logical patterns, enabling it to extrapolate from these patterns and generate logical outcomes for promising steps. Specifically, we retrieve cases from the corresponding case base ${\mathcal{C}}_{conj}$ based on the logical pattern identified in the previous phase. The top-$\alpha$ cases are selected based on their cosine similarity:

$$sim({\mathbf{h}}_x,{\mathbf{h}}_{c_i})={\displaystyle \frac{\left({\mathbf{h}}_x·{\mathbf{h}}_{c_i}\right)}{∥{\mathbf{h}}_x∥\parallel {\mathbf{h}}_{c_i}\parallel }},$$

(5)

where ${\mathbf{h}}_x$ represents the vector representation of the promising step, and ${\mathbf{h}}_{c_i}$ represents the vector presentations of the logical pattern category.

3.2. Reuse Phase

Cosine similarity tends to emphasize entities within the query, often resulting in retrieval outcomes dominated by cases with similar entities. This homogeneity can limit the model’s performance in in-context generation. Studies [25,26] have shown that model performance can vary significantly based on the selection of in-context examples, with diverse demonstrations leading to better outcomes. To address this limitation, we apply an information entropy-based reranking mechanism to prioritize and rank the retrieved cases, introducing greater diversity into the results. The information entropy-based reranking ensures that the model encounters a broader range of distinct cases within the same logical pattern. By diversifying the retrieved cases, this approach enhances in-context learning, enabling the model to better learn from these varied demonstrations. This ultimately improves the model’s performance in generating logically consistent entailment trees.

The information entropy of a sentence $s=\{w_1,w_2,\dots ,w_k\}$ is calculated as follows:

$$H\left(s\right)=-\sum _{i=1}^{k}p\left(w_i\right){log}_{2}p\left(w_i\right),$$

(6)

where $p\left(w_i\right)$ denotes the probability of each word $w_i$, calculated based on its frequency in the case base. Higher entropy indicates a sentence with more information content. Previous studies [52,53] have demonstrated that diverse demonstrations significantly improve the performance of large language models.

After retrieving a set of cases $\{c_1,c_2,\dots ,c_{\alpha }\}$ using cosine similarity, we rerank them based on information entropy to enhance diversity. The information entropy of each case $c_i$ is calculated as

$$I{E}_{c_i}=H\left(s_1\right)+H\left(s_2\right)+H\left(in{t}_{\ast }\right).$$

(7)

We then select the top-$\beta$ cases with the highest information entropy scores. The information entropy score of a case provides insights into its complexity, indicating the amount of information it contains. Recognizing that in-context learning is sensitive to the order of demonstrations, we organize the filtered cases by difficulty, arranging them from easy (low entropy) to difficult (high entropy), following a curriculum learning approach [28].

3.3. Refine Phase

In the refine phase, the ordered cases are concatenated and used as demonstrations to guide a pretrained language model for logical text generation. We use the Llama 3 [54] model as the backbone. Each selected case (i.e., $s_1+s_2\to in{t}_{\ast }$) is translated into a natural language format as follows: $s_1$ [AND] $s_2$ [ENTAIL] $in{t}_{\ast }$. Through in-context learning, the LLM generates logical text based on these demonstrations. Below is an example prompt for case-based deduction using five cases (see Figure 3):

Following this case-based deduction process, the controller facilitates reasoning. Since entailment trees are built iteratively, with each step expanding the reasoning space, we employ beam search for effective reasoning. The process continues until the target hypothesis is validated or the maximum reasoning depth is reached.

4. Experiment

4.1. Benchmark Dataset

We conducted experiments using the EntailmentBank dataset [6], which contains 1840 expert-annotated entailment trees. Each tree corresponds to a hypothesis derived from a question and its correct answer in the ARC dataset [55]. The leaf facts of these trees are selected from the WorldTreeV2 corpus [56]. The statistics for the EntailmentBank dataset are detailed in

Table 2. Summary Statistics for the EntailmentBank dataset splits.

	Train	Dev	Test	All
Questions	1313	187	340	1840
Entailment Steps	4175	597	1109	5881

.

We chose EntailmentBank because it offers a structured entailment reasoning task, where multistep reasoning is crucial to constructing logically consistent entailment trees. This dataset provides a well-established benchmark for evaluating models that generate logical entailments and intermediate reasoning steps, aligning with the capabilities of our proposed method. EntailmentBank’s tasks involve reasoning from multiple premises to conclusions, which allows us to test CBD’s ability to retrieve, reuse, and refine logical patterns, making it an ideal dataset for assessing multistep reasoning models.

Following [6], three explanation tasks are defined based on a hypothesis (question + answer), each with increasing difficulty: (a) generating a valid entailment tree given all relevant sentences (the leaves of the gold entailment tree), (b) generating the tree given all relevant sentences along with some irrelevant ones, or (c) generating the tree given a full corpus. The objective is to produce explanations in the form of entailment trees that represent a sequence of multipremise entailment steps from known facts through intermediate conclusions to the final hypothesis (the question + answer).

The case base used in our study comprises the entailment steps from the training dataset. We utilized the manually annotated dataset from [8], which includes 400 annotated reasoning steps in the training split (Train-manual) and 275 steps in the development split (Dev-manual).

Table 3. Statistics of logical pattern annotations for steps.

Split	Substitution	Conjunction	If-then	All
Train	211	105	84	400
Dev	153	71	51	275

presents the statistics of the logical pattern annotations. These entailment steps are categorized into three types: conjunction, if–then, and substitution patterns. We used the training dataset to train the prototypical network and the development dataset to evaluate and optimize its performance.

4.2. Baseline Models

To evaluate the effectiveness of our proposed method, we conducted a comparative analysis between CBD and several existing methods:

• EntailmentWriter [6]: This method generates the entire entailment tree “all at once”, directly producing linearized trees. Hong et al. [8] extended this approach by developing EntailmentWriter-Iter, which iteratively generates individual steps of the tree and then concatenates them to form the final tree.
• IRGR [10]: This method designs an iterative retrieval-generation framework that enhances retrieval outcomes. It iteratively retrieves premises and constructs step-by-step explanations from textual evidence.
• MetGen [8]: This method employs a reasoning controller and a T5 model to independently select premises and generate conclusions. For a fair comparison, we also experimented with replacing the T5 model with Llama 3 [54].

In our comparison, we integrated CBD with the controller from MetGen to effectively select steps, which then facilitated the generation of logical intermediate conclusions.

4.3. Evaluation Metrics

We evaluate entailment trees in a two-step process. First, nodes in the predicted tree $T_{pred}$ are aligned with nodes in the gold tree $T_{gold}$ using the $sen{t}_{\ast }$ labels and Jaccard similarity for intermediate nodes, aiming to capture semantic equivalences rather than exact matches. Next, we assess the generated tree by comparing it to the gold tree across three dimensions, following established methodologies from previous studies [6,7,8]:

• Leaves: This dimension evaluates the use of correct leaf facts. We calculate the $F_1$ score by comparing the predicted leaf facts with those in the gold tree. Additionally, we report the AllCorrect score, which indicates exact matches. The AllCorrect score is 1 when the $F_1$ score is 1, and 0 otherwise.
• Steps: This dimension assesses the structural correctness of entailment steps. We compare the steps in both trees using the $F_1$ and AllCorrect scores. A predicted step is considered correct if its children’s identifiers perfectly match those in the gold tree.
• Intermediates: This dimension measures the accuracy of intermediate conclusions, also using the $F_1$ and AllCorrect scores. A predicted intermediate conclusion is considered correct if the BLEURT-Large-512 [57] score between the aligned predicted intermediate and the corresponding gold intermediate is greater than 0.28. Using BLEURT allows us to account for semantic equivalence between the generated and gold conclusions beyond surface-level matching, which is critical for assessing complex multistep reasoning tasks.

Finally, we employ a strict metric called Overall Allcorrect. This score is 1 only if all leaves, steps, and intermediates in the generated tree are correct, meaning the tree exactly matches $T_{gold}$. Any error in the tree results in a score of 0 under this metric.

We chose these evaluation metrics because they directly measure both the structural and semantic accuracy of the generated entailment trees. By using a combination of F1 and AllCorrect scores, we ensure that the model is evaluated on both partial correctness and exact matches, providing a nuanced view of its performance. Additionally, the inclusion of BLEURT-Large-512 for intermediate conclusions allows us to capture semantic similarity, which is essential for assessing the quality of multistep reasoning. These metrics have been widely used in prior work, ensuring that our evaluation is comparable with other models in the field and that our method’s improvements are consistently measured across all reasoning dimensions.

4.4. Implementation Details

All models and baselines were implemented using the PyTorch framework [58] and Huggingface transformers [59]. During the logical generation in the refine phase, we used Llama 3 [54] as the pretrained language model. For Task 1, we iterated until all facts in $\mathcal{K}$ were used. For Task 2 and 3, we applied the same settings as MetGen to ensure a fair comparison, using a fact score threshold of 0.001 to filter out distractors and a maximum reasoning depth of 5. We selected the top 10% of steps for each state and set the beam size to 10. The prototypical network architecture is based on the DeBERTa model [51], with the final layer producing a 768-dimensional embedding for each case. The network was trained using the Adam optimizer with an initial learning rate of 2 × 10⁻⁵ and a batch size of 8, and over 30 epochs. Additionally, we optimized the loss function using the AdamW optimizer [60].

5. Results and Analysis

In this section, we evaluate the performance of CBD and provide a comprehensive analysis of the results, including the impact of varying the number of selected cases.

5.1. Main Results

The evaluation results, summarized in

Table 4. Automatic evaluation results on the EntailmentBank test split. ◊ indicates results reported in published papers. † denotes results obtained using the official public code with the Llama 3 language model. The best results are highlighted in bold, and the second-best results are underlined.

Method	Leaves		Steps		Intermediates		Overall
	F1 (%)	AllCorrect (%)	F1 (%)	AllCorrect (%)	F1 (%)	AllCorrect (%)	AllCorrect (%)
Task 1 (no-distractor)
EntailmentWriter [6] ◊	98.4	84.1	50.0	38.5	67.0	35.9	34.4
EntailmentWriter-Iter [6] ◊	99.8	97.6	51.6	38.5	68.3	36.5	35.0
IRGR [10] ◊	97.6	89.4	50.2	36.8	62.1	31.8	32.4
MetGen (T5) [8] ◊	100.0	100.0	57.7	41.9	70.8	39.2	36.5
MetGen (Llama 3) [8] †	99.8	99.4	58.6	42.4	71.6	40.3	36.5
CBD (Ours)	99.8	99.4	58.5	42.4	72.2	42.1	38.2
Task 2 (distractor)
EntailmentWriter [6] ◊	83.2	35.0	39.5	24.7	62.2	28.2	23.2
EntailmentWriter-Iter [6] ◊	85.2	40.9	38.9	26.8	63.5	29.1	25.0
IRGR [10] ◊	69.9	23.8	30.5	22.4	47.7	26.5	21.8
MetGen (T5) [8] ◊	82.7	46.1	41.3	29.6	61.4	32.4	27.7
MetGen (Llama 3) [8] †	80.9	43.2	41.2	29.4	60.3	32.9	27.9
CBD (Ours)	81.3	44.1	42.5	30.3	60.9	33.5	28.5
Task 3 (full-corpus)
EntailmentWriter [6] ◊	30.9	1.2	4.4	1.2	28.8	5.6	1.2
EntailmentWriter-Iter [6] ◊	32.4	1.8	4.4	1.5	29.7	6.5	1.5
IRGR [10] ◊	46.6	10.0	11.3	8.2	38.7	20.9	8.2
MetGen (T5) [8] ◊	34.8	8.7	9.8	8.6	36.6	20.4	8.6
MetGen (Llama 3) [8] †	36.2	9.1	10.6	8.2	36.1	18.2	8.2
CBD (Ours)	35.7	9.7	11.5	9.4	36.7	20.9	9.4

, show that multistep methods (IRGR, EntailmentWriter-Iter, and MetGen) outperform the single-step method (EntailmentWriter), highlighting the effectiveness of multistep approaches in selecting premises and generating intermediate conclusions that closely align with the ground truth. When comparing CBD with MetGen for intermediate conclusion generation, we observe a notable performance improvement, particularly in Task 1. Specifically, under the strict Overall AllCorrect metric, CBD shows performance gains of 1.7% in Task 1, 0.6% in Task 2 and 0.8% in Task 3. A key advantage of CBD is its use of a prototypical network to recognize abstract logical patterns in the promising step. Unlike traditional methods that often focus on surface-level similarities, the prototypical network allows CBD to capture the underlying logical patterns, which is particularly valuable in retrieving similar cases. By leveraging these similar cases, CBD ensures that the intermediate conclusions maintain logical consistency, contributing to higher performance in the Intermediates AllCorrect metric, where it achieves a top score of 42.1% on Task 1. In contrast, methods like EntailmentWriter and IRGR, which lack the capability to abstract these logical patterns, produce less accurate intermediate conclusions.

EntailmentWriter and EntailmentWriter-Iter, particularly the single-step EntailmentWriter, struggle to generate consistent intermediate conclusions, as they produce the entire entailment tree “all at once”. This makes it difficult to maintain logical coherence across multiple reasoning steps. In contrast, CBD’s multistep reasoning, supported by case-based learning, leads to a more accurate generation of intermediate conclusions. While MetGen benefits from its reasoning controller and performs well across most tasks, CBD outperforms it in both the Intermediates and Overall AllCorrect metrics. The integration of the prototypical network and information entropy-based reranking in case-based reasoning enables CBD to demonstrate strong performance, effectively handling a wide range of reasoning complexities and excelling across diverse tasks.

5.2. Ablation Study

In the ablation study, we evaluate the effectiveness of three key components of CBD: the prototypical network (ProtoNet), ranking by information entropy (IE), and curriculum reasoning (Curriculum). The results, based on experiments conducted on Task 1, are presented in

Table 5. Ablation study on the EntailmentBank dataset for Task 1. “w/o” indicates the removal of the corresponding module from the model. $\Delta$ represents the percentage difference compared with CBD.

Method	Intermediates		Overall	$\Delta$
	F1 (%)	AllCorrect (%)	AllCorrect (%)
CBD (Ours)	72.2	42.1	38.2	-
CBD (w/o ProtoNet)	67.8	37.1	34.7	−3.5
CBD (w/o IE)	71.5	40.9	36.8	−1.4
CBD (w/o Curriculum)	71.8	40.6	37.1	−1.1

Note: Bold values indicate the best performance.

. First, we assess the model without ProtoNet, which directly searches the case base without distinguishing among different logical patterns. This version shows a significant performance drop, underscoring the crucial role of ProtoNet in selecting relevant cases. Next, we examine the model without reranking by information entropy scores. While this model maintains the same architecture as the full CBD, it ranks analogical cases solely based on cosine similarity during the reuse phase. The observed performance degradation highlights the importance of information entropy in introducing diversity into the demonstrations. Despite this, the model still outperforms MetGen (36.8 vs. 36.5), indicating that case-based deduction contributes to generating better conclusions. Additionally, we observe that the order of selected cases affects overall performance, as captured by the curriculum reasoning component. Importantly, CBD (w/o ProtoNet) closely mirrors traditional case-based reasoning methods, which rely purely on similarity-based retrieval without the advanced features we propose. The performance drop shows that our enhancements in case retrieval and diversification significantly improve model performance over standard case-based reasoning techniques.

5.3. Prototypical Network Results

To improve the efficiency of identifying cases with similar logical patterns, we use a prototypical network to determine the logical pattern of each promising step. The DeBERTa model [51] serves as the encoder for the prototypical network. The statistics for logical pattern annotations are shown in Table 3. Our model achieves an accuracy of 81% on the development set. We also experimented with a direct classifier approach, which produced results comparable to the prototypical network. However, the prototypical network is preferred due to its more interpretable decision-making process.

While evaluating the performance of our method, it’s important to consider the distinction between our approach and clustering algorithms such as ECA* [61] and iECA* [62]. These clustering algorithms are designed to identify clusters without predefined centers, making them ideal for applications where categories are unknown in advance. In contrast, our approach leverages a prototypical network, where each data point’s category is known beforehand, enabling targeted logical pattern recognition.

5.4. Comparative Analysis of Retrieval and Reranking Method Combinations

This section presents a comparative analysis of different retrieval and reranking method combinations within the CBD framework. Our experiments focused on comparing the proposed method, which employs a prototypical network in the retrieval phase and information entropy for reranking in the reuse phase (ProtoNet + IE), against commonly used approaches, such as Dense Retrieval combined with a RoBERTa-based reranker (Dense + Reranker), on the EntailmentBank dataset for Task 1. Dense Retrieval operates by encoding both the query and candidate cases into dense vector representations and then calculating their cosine similarity within the same vector space. The RoBERTa-based Reranker refines the ranking of candidate cases by jointly encoding the query and a candidate case, scoring their relevance based on the [CLS] token’s embedding.

As summarized in

Table 6. Experimental results of different retrieval and reranking method combinations for Task 1.

Method	Intermediates		Overall
	F1 (%)	AllCorrect (%)	AllCorrect (%)
Dense + Reranker	68.0	36.2	33.5
Dense + IE	67.8	37.1	34.7
ProtoNet + Reranker	70.6	39.4	35.9
ProtoNet + IE (Ours)	72.2	42.1	38.2

Note: Bold values indicate the best performance.

, our ProtoNet + IE approach outperforms all other methods across all metrics, achieving the highest overall accuracy of 38.2%. These results underscore the effectiveness of our method in both the retrieval and reuse stages, delivering more accurate results than alternative approaches. Specifically, in the retrieval phase, the prototypical network effectively narrows the retrieval scope, reducing noise from irrelevant cases. Additionally, the information entropy-based reranker enhances diversity, which is crucial for in-context learning. This comparative analysis highlights the significant advantages of integrating ProtoNet with information entropy in improving the performance of entailment tree generation within our framework.

5.5. Effect of Number of Selected Cases

We next investigate the influence of the number of selected cases in the reused phase on Task 1. This experiment was conducted using the top 10 results based on cosine similarity. We focused on the top 10 because it likely includes a more diverse set of entities. To explore this, we tested CBD with varying numbers of selected cases. The results, presented in Figure 4 show a clear improvement in the Overall-Allcorrect accuracy of our model when three cases are selected. We hypothesize that selecting three cases strikes a balance between the diversification provided by information entropy ranking and the curriculum learning process from easy to hard. The improvement with one or two cases is less significant, possibly due to insufficient diversity, which may cause the model to confuse in-context cases with knowledge. It is important to note that accuracy declines when more than three cases are selected, suggesting that noise may be introduced into the demonstrations, leading the model to focus more on entities rather than the logical pattern. Nevertheless, even with just one selected case, the model’s performance is comparable to MetGen, indicating that CBD is beneficial for the entailment task.

5.6. Visualization of Logical Patterns

We visualized the prototype embeddings of different logical patterns (conjunction, if-then, substitution) using t-SNE [63], as shown in Figure 5. Each color represents a prototype logical pattern induced by the prototypical network, with the prototypical features clearly organized into distinct clusters. This clustering suggests that the model can effectively differentiate among the different logical patterns, enhancing its ability to accurately classify promising steps.

6. Related Work

6.1. Explanation for Question Answering

Extensive research has explored interpretability in question-answering (QA) systems, with a focus on understanding how models make predictions and provide explanations in various forms [64,65]. These efforts include techniques such as using attention maps over passages [66], extracting snippets of textual evidence [2], and selecting answer-supporting sentences from input paragraphs [56,67]. With the advent of large language models (LLMs) [52,68,69], recent research has shifted toward improving interpretability. One prominent approach is generating reasoning steps before producing answers, often referred to as chain-of-thought (CoT) reasoning [3,70,71,72,73]. However, several studies [74,75] have pointed out the limitations of this strategy. For example, Turpin et al. [76] highlight that CoT explanations can misrepresent the true reasoning behind a model’s prediction. Additionally, evaluations of CoT explanations [77] reveal several issues, such as contradictions and mathematical errors [78]. These findings underscore the risk that CoT-generated answers may not align with the reasoning process, which can erode user trust and complicate the identification of errors. Unlike free-form text explanations, which are prone to hallucinations [70,79,80,81] or rationales that only provide supporting evidence [2,82], structured explanations like entailment trees [6,83] are gaining popularity as a more reliable approach. These methods use a tree structure to outline how information is combined to derive the answer systematically. In our work, we adopt case-based deduction to iteratively construct entailment trees, ensuring that each step logically follows from the previous one, thus enhancing interpretability and minimizing errors.

6.2. Entailment Tree Generation

Entailment trees [6], which illustrate the logical relationships between multiple premises, have proven effective for structuring text-only question-answering tasks by systematically representing logical reasoning chains. They provide a transparent way to understand how hypotheses logically follow from a series of entailment steps based on multiple premises [84]. Recent advancements in the field have focused on the reconstruction task [6,10,85] and on leveraging entailment trees for neuro-symbolic reasoning [11,84]. For example, Weir et al. [84] introduced a QA system that employs backward chaining to search for entailment trees, allowing users to inspect and debug the system’s reasoning process. This approach holds promise for developing interactive and educational QA systems [86]. In our work, we focus on the reconstruction task, specifically on entailment tree generation.

EntailmentBank [6] proposes representing QA system reasoning steps as multistep textual entailments, offering detailed and informative explanations. Various methods have been developed to reconstruct tree-structured explanations for correct answers, such as those introduced by Hong et al. [8], Yang et al. [7], and Ribeiro et al. [10]. Recent studies [7,8,10,11] have adopted an iterative strategy for performing single-step neural inferences and combining these results to construct multistep reasoning chains. This iterative technique has been successful, leading to improved performance. Hong et al. [8] use concise prompts, such as “deductive conjunction”, tailored to reasoning patterns. Their method employs supervised learning to train one-step generation. In contrast, our method adopts an unsupervised in-context learning approach without parameter fine-tuning. By providing retrieved cases with promising steps during each inference, our method effectively guides the model toward the desired outputs. On the other hand, Ribeiro et al. [10] directly input fact pairs, prompting the language model to deduce based on these pairs. However, this approach lacks considering an underlying logical pattern, making it susceptible to inaccuracies or hallucinations.

7. Limitations

While the case-based deduction method demonstrates improvements in entailment tree generation, there are some considerations regarding its current design. One aspect to acknowledge is the method’s reliance on prototypical networks, which could potentially introduce biases based on the distribution of training data. If the training data emphasize certain logical patterns more than others, the model may show a preference for these patterns. However, with a sufficiently diverse and well-balanced dataset, this issue can be mitigated, allowing the model to generalize more effectively. Additionally, the method is particularly well suited for binary tree structures, which makes it robust in handling such reasoning tasks. While this focus contributes to the method’s strong performance in these contexts, it may be less adaptable when dealing with more complex or nonbinary tree structure reasoning. Nonetheless, for tasks that adhere to binary structures, the approach provides significant advantages in terms of logical consistency and accuracy.

8. Conclusions and Future Research

Existing methods for entailment tree generation often struggle to maintain logical consistency within individual steps. To address this challenge, we introduce a novel approach called case-based deduction (CBD), which retrieves cases with similar logical patterns to the input pairs, ensuring consistent logical generation. Within the retrieve-reuse-refine paradigm, we use a prototypical network to identify logical patterns, enabling the accurate selection of analogous cases. By ranking these cases based on information entropy scores, we diversify the demonstrations and effectively teach a large language model the foundational logical patterns for reasoning. Our experimental results show that this method significantly improves the quality of entailment tree generation compared with state-of-the-art approaches. Additionally, using a prototypical network in the retrieval phase provides interpretable logical pattern induction. While our current approach has demonstrated effectiveness in entailment tree generation tasks, its potential extends beyond this specific application. The principles of CBD, retrieving and reusing logical patterns from similar cases, can be applied to a broader range of reasoning domains. For example, CBD could be leveraged in commonsense reasoning to infer conclusions based on everyday scenarios by retrieving cases with similar logical structures. In legal reasoning, the method could assist in drawing parallels between legal precedents and current cases. Additionally, mathematical proof generation could benefit from CBD’s ability to retrieve similar logical steps from previously solved proofs, aiding in new problem-solving tasks.

In future work, we plan to address the limitations identified in our current approach and focus on more specific improvements to enhance the performance and flexibility of case-based deduction. One promising direction is the exploration of alternative retrieval mechanisms. While our current method leverages a prototypical network to retrieve cases based on logical patterns, we aim to investigate more adaptive retrieval strategies that adapt to diverse reasoning contexts. For instance, context-aware retrieval mechanisms or reinforcement learning techniques could allow the model to better select cases tailored to specific problem domains. Another area for improvement is integrating more complex and diverse logical patterns into the prototypical network. This enhancement would allow the model to capture a broader range of reasoning strategies, particularly for more challenging entailment tasks. By expanding the variety of logical patterns the network can recognize and retrieve, we expect the model to handle diverse and nonbinary reasoning structures more effectively. We also see value in incorporating human evaluation to complement the current automated metrics. Human evaluators can provide additional insights into aspects like interpretability, logical flow, and the practical utility of generated entailment trees, which may not be fully captured by automated evaluation methods.