A Purely Entity-Based Semantic Search Approach for Document Retrieval

Abstract

Over the past decade, knowledge bases (KB) have been increasingly utilized to complete and enrich the representation of queries and documents in order to improve the document retrieval task. Although many approaches have used KB for such purposes, the problem of how to effectively leverage entity-based representation still needs to be resolved. This paper proposes a Purely Entity-based Semantic Search Approach for Information Retrieval (PESS4IR) as a novel solution. The approach includes (i) its own entity linking method and (ii) an inverted indexing method, and for document retrieval and ranking, (iii) an appropriate ranking method is designed to take advantage of all the strengths of the approach. We report the findings on the performance of our approach, which is tested by queries annotated by two known entity linking tools, REL and DBpedia-Spotlight. The experiments are performed on the standard TREC 2004 Robust and MSMARCO collections. By using the REL method on the Robust collection, for the queries whose terms are all annotated and whose average annotation scores are greater than or equal to 0.75, our approach achieves the maximum nDCG@5 score (1.00). Also, it is shown that using PESS4IR alongside another document retrieval method would improve performance, unless that method alone achieves the maximum nDCG@5 score for those highly annotated queries.

1. Introduction

Because of their semi-structured, rich, and strong semantics, knowledge bases are exponentially used in the different information retrieval tasks. Furthermore, the quality and quantity of the knowledge bases such as DBpedia [1] are continuously increasing, which gives us an idea of the knowledge bases’ usefulness now and in the future [2]. Moreover, knowledge bases such as DBpedia and Freebase [3] are among the most widely used.

Based on knowledge bases, a given text could be represented by a suitable set of entities. This representation of the text by these entities would be called entity-based representation. Moreover, there are several ways to utilize knowledge bases to improve the representation of queries and documents for better ad hoc document retrieval task. In the case when entity-based representation is used alongside term-based representation for query representation, knowledge bases are used for query expansion [4,5]. Furthermore, they are used for both query and document representation to complete and enrich the term representation for better document retrieval [6,7,8].

Although many entity-based representation approaches have been proposed for semantic document retrieval, the problem of how to effectively leverage entity-based representation still needs to be resolved [9,10] (see Section 2 for more details). To answer the question “How to know when an approach (a retrieval method) does its best, and for what type of queries?”, it is essential to explore the strengths and weaknesses of the approach and, on the other hand, to analyze its performance with regard to different queries. On the other hand, because of the complexity and dynamic nature of the queries, no document retrieval approach achieves the same performance for every query. Although a better approach has a higher performance with regard to the entire query set, a weak approach might have a better performance for some of those queries. Once these questions are effectively addressed, a document retrieval system, such as a search engine, could leverage many approaches alongside each other with regard to the different kinds of queries for better information retrieval.

We propose a novel semantic search approach, named Purely Entity-based Semantic Search for Information Retrieval (PESS4IR (https://github.com/El-Emin/PESS4IR-source-code)), which is based purely on entity representation, and we explore its strengths and weaknesses with regard to better document retrieval. In other words, in our approach, the “Purely entity-based representation” concept means that documents and queries are only represented by entities. The approach is mainly composed of three components. The first one is its own entity linking method, which is more appropriate for document text and is named Entity Linking for Document Text (EL4DT). We note that, for the entity linking task, there are many tools available online, such as DBpedia Spotlight [11], TagMe [12], and REL [13]. However, the main reason for designing EL4DT is that these tools do not provide certain information and statistics necessary for our document retrieval and ranking method. The second component is an inverted indexing method to achieve the indexing task. Finally, the third component is an appropriate document retrieval and ranking method, which is designed to leverage the strengths of the approach.

Our approach introduces the concept of “strong entity” to describe entities annotated by EL4DT with high scores (an annotation score higher than or equal to 0.85). The concept plays an important role in our retrieval and ranking method. Moreover, our approach introduces the “related entities number” concept to describe the number of entities semantically related to an entity, for each entity in the paragraph. Furthermore, alongside the “strong entity” and “related entities number” concepts, our retrieval and ranking method leverages many other aspects of our approach, such as document title weighting and the usage of all information and statistics stored in the index, such as the number of semantically related entities in the same paragraph (see Section 3.2).

Before discussing the evaluation of our approach, it is vital to understand the nature of the purely entity-based representation of queries and documents. In fact, queries are ambiguous [6] due to the nature of their text, which is usually short and suffers from a lack of context, in contrast to document text, which could be well represented by entities because of its textual richness. Annotating the query set of the TREC 2004 Robust collection (250 queries) with the DBpedia Spotlight and REL entity linking tools highlights the “completely annotated query” concept; we consider a query as completely annotated when all its terms are annotated and stopwords are not obligatory. DBpedia Spotlight and REL annotate 72% and 6.8% of queries, respectively, as completely annotated queries (see Appendix A.1 and Appendix A.2). Thus, in our experiments, only completely annotated queries are considered. As our approach is designed to handle only completely annotated queries, the evaluation will be based on the corresponding results. In other words, to conduct a comparison with another method, one required the results for corresponding queries, which could be extracted from their respective result or run files. Moreover, we use the well-known baseline Galago tool [14], a search engine extended by the Lemur and Indri projects for research purposes [15], to obtain the corresponding results performed by Galago (Dirichlet method). Also, the result files of the LongP (Longformer) [16] model are used in our experiments to demonstrate the added value achieved by our approach. For our experiments, we use the TREC 2004 Robust and MSMARCO collections. We use the REL and DBpedia Spotlight entity linking tools as arbitrary entity linking methods for the query annotation process instead of our entity linking method. In other words, our approach is tested with the queries annotated by other methods. Also, in the evaluation, we use standard evaluation metrics such as normalized Discounted Cumulative Gain (nDCG@k), Mean Average Precision (MAP), and Precision (P@k).

In this work, we explore the strengths and weaknesses of our approach by taking advantage of its strengths and avoiding its weaknesses. The exploration study allowed us to effectively address the main question: “for which query a purely entity-based approach would be recommended?”. Furthermore, for queries with an average annotation score (average annotation score of query entities) higher than or equal to 0.75, as annotated by the REL method, our approach achieves the maximum nDCG@5 score (1.000), which would be an added value for any ad hoc document retrieval method which does not reach the maximum nDCG@5 score for those highly annotated queries.

The remainder of the paper is organized as follows: Section 2 discusses the related work and background. Section 3 introduces the proposed approach, PESS4IR, by explaining all its components in detail. Section 4 provides the results of the performed experiments. Then, Section 5 provides the discussion of the results. Finally, the conclusion is given in Section 6.

2. Related Works

A semantic search significantly improves information retrieval (IR) tasks, including the ad hoc document retrieval task [6,7,8,10,17,18,19,20,21,22], which is our concern in this paper. On the other hand, knowledge bases have recently been increasingly used to improve the semantic search [6,7,8,19,22]. Knowledge bases allow an entity-based representation of text instead of the lexical representation used in traditional models such as the BM25 model [23]. Below, we provide the non-entity-based document retrieval and entity-based document retrieval, our primary concern in this paper, and the entity linking task as a secondary concern.

2.1. Non-Entity-Based Document Retrieval

The approaches based on pre-trained Transformer language models such as BERT [24] are the current state of the art (SOTA) for text re-ranking [25]. In this section, we present some state-of-the-art document retrieval models which are not based on knowledge bases. Li et al. (2020) [26] proposed an approach named PARADE, which is a re-ranking model. They claimed that an improvement was achieved by the model on TREC Robust04 and GOV2 collections; the model achieves its most effective performance when it is adopted as a re-ranker, namely PARADE-Transformer. The Longformer model [27] is also a Transformer-based model. Its performance on the Robust04 and MSMARCO collections is presented in [16]. We use the LongP Longformer model [16] in our comparisons with the state-of-the-art methods. Moreover, Gao and Callan (2022) [25] proposed the MORES+ model, which is a re-ranking method, and tested it on two classical IR collections: Robust04 and ClueWeb09. Wang et al. (2023) [28] proposed a ranking method named ColBERT-PRF. They evaluated it on the MSMARCO and TREC Robust collections for document ranking tasks. The approach could be exploited for both end-to-end ranking and re-ranking scenarios.

2.2. Entity-Based Document Retrieval

In the literature, over the last decade, many document retrieval approaches have explored different types of uses of entity-based representation to improve the representation of documents and queries. Xiong et al. (2017) [6] proposed a neural information retrieval approach using both term-based and entity-based representations for queries and documents. The approach performs four-way interactions, allowing four matching possibilities between the query and the document. The four possible representations of the document and query are: query terms (words) to document terms, query entities to document entities, query entities to document terms, and query terms to document entities. Thus, they achieve their document retrieval and ranking by integrating those combinations into neural models. Liu et al. (2018) [7] also introduced a neural ranking model that combines entity-based and term-based representations of documents and queries. They used a translation layer in their neural architecture, matching queries and documents without using handcrafted features. Bagheri et al. (2018) [19] proposed a document retrieval approach that uses neural embeddings that consider both word embeddings and entity embeddings. Also, they compared word and entity embedding performances. Lashkari et al. (2019) [8] proposed a neural embeddings-based representation for documents by considering the term, entity, and semantic type within the same embedding space. Gerritse et al. (2022) [22] proposed the EM-BERT model, which incorporates entity embeddings into a point-wise document ranking approach. The model combines words and entities into an embedding representation to represent both the query and the document using the BERT [24] model. Although many entity-based document retrieval approaches have been proposed, the problem of how to effectively leverage entity-based representation still needs to be resolved. Guo et al. [10] presented a survey on existing neural ranking models, highlighting the models that learn with external knowledge, such as knowledge bases. They indicated that more research is needed to improve the effectiveness of neural ranking models with external knowledge and to understand external knowledge’s role in ranking tasks. Moreover, Reinanda et al. [9] explain that the problem of how to effectively leverage entity-based representation in conjunction with term-based representation still needs to be solved. On the other hand, these approaches and models use knowledge graphs as embedding-based representations, where entity embeddings are learned from knowledge graphs in many ways in the literature [29,30,31].

To understand how to effectively leverage the knowledge graph alongside any other model, we introduced PESS4IR, a novel solution, to empirically study the impact of purely entity-based representation for document retrieval. With PESS4IR, queries and documents are represented only by an entity-based representation, without the use of a neural network and embedding-based representation. Below, we discuss the background related to the entity-based document retrieval task.

2.3. Entity Linking

Entity linking is the task concerned with linking terms of a given text to appropriate entities extracted from knowledge bases; in other words, it gives an entity-based representation which suits the given text. There are many entity linking tools, among which DBpedia Spotlight [11], TagMe [12], REL [13], WAT [32], and FEL [33] are the most widely used. Most of these entity-linking tools are designed for general text annotation purposes. Moreover, some of them would perform better for short text than others, such as TagMe, which is known for its performance with short text [34]. However, to obtain a more appropriate entity linker for document text, we developed a novel entity linking method, which provides the specific information and statistics our approach needs. In the literature, an effective entity linking method is generally designed based on the general pipeline, which is composed of three steps: mention detection, candidate selection, and disambiguation. Moreover, disambiguation is the most challenging step [35]. According to Balog [35], modern disambiguation should consider three important types of evidence: prior importance, contextual similarity, and coherence. Many researchers deal with disambiguation via a graph-based approach. Kwon et al. [36] recently dealt with the disambiguation issue by proposing a graph-based solution. Our entity linking method (EL4DT) uses a graph-based method for the disambiguation task, and the three types of evidence described in [35] are considered.

3. Materials and Methods

This section presents our approach (PESS4IR), which mainly includes three methods: the entity linking method (EL4DT), the indexing method, and the retrieval and ranking method. In the following subsections, we explain each method in detail.

3.1. Entity Linking Method

We designed and developed an entity linking method for document annotation, knowing that many available entity linking methods exist. The main reason is to provide our approach with some required information and statistics, which are not provided by the available entity linking tools. We store the information and statistics in the inverted index to make them available for our retrieval and ranking method. In the following subsections, we provide the details of our entity linking method.

3.1.1. Overview

Our entity linking method is designed precisely for the document text and is named Entity Linking for Document Text (EL4DT). It is based on two knowledge bases, DBpedia [1] and Facc1 [37], from which the surface forms are constructed. In addition, EL4DT respects the general pipeline of entity linking methods, which supposes that an entity linking method considers three steps: mention detection, candidate selection, and disambiguation. They are explained in more detail in the following three subsections, respectively.

Before going into the details, we briefly describe our entity linking method by explaining its three main steps. In the first step, which contains mention detection and candidate selection tasks, the process starts with a given preprocessed text and, by the n-gram method, mentions are generated. For each generated mention, the candidate entities are extracted from the surface form. During the first step, a pre-disambiguation process is performed by selecting, in the case of many entities proposed by one class of the surface form classes, the more probable candidate entity according to the context similarity score. Moreover, for each candidate entity, the score is computed. In the second step, we have the disambiguation method, which performs the disambiguation task by using graphs. Figure 1 represents an initialized graph for the entities of a given paragraph, where each entity (represented by a node in the graph) could be connected to another if there are some relationships between them. Their relationships (defined by an edge in the graph) are scored according to the nature of those relationships. Additionally, the entities which are semantically related will be grouped as a cluster in the graph. Moreover, the relationships between the entities are found in the article categories (DBpedia), the SKOS relationships, such as broader and related (DBpedia), and document entity coherence relationships (introduced in Equation (1)). In the last step, we select the graph’s highest score cluster. The selected graph cluster, a set of entities among the paragraph’s entities, includes the disambiguated entities. It is important to note that sure entities (entities with a score greater than or equal to 0.5) do not need a disambiguation step. Furthermore, only weak entities (entities with a score of less than 0.5) which are not related to the selected cluster are ignored. The EL4DT algorithm (Algorithm 1) introduces the three steps of mention detection, candidate selection, and disambiguation. Moreover, the frequently used symbols are listed in

Table 1. Frequent symbols.

Symbol	Description
E	All entities.
Eq	Entities in query q.
Ep	Entities in paragraph p.
Eqp	Common entities between query q and paragraph p.
Edt	Entities in document title dt.
SEp	Strong entities in paragraph p.
SEd	Strong entities in document d.
SEqp	Entities in query q, which are found in paragraph p as strong entities.
Td	Document text
Tp	Paragraph text

.

3.1.2. Mention Detection

A mention refers to a contiguous sequence of terms in the text to be annotated, which refers to one or more particular entities in the surface form [35]. The surface forms are the structures that include all possible mentions extracted from knowledge bases. As mentioned earlier, our surface forms are constructed from the DBpedia and Facc1 (http://lemurproject.org/clueweb12/FACC1/) knowledge bases. The surface forms of our EL4DT are constructed from the components listed in

Table 2. Surface forms of EL4DT.

Component (Class)	Description	Knowledge Base
E_db	Entities extracted from Article categories (without stopwords)	DBpedia
cE_db	Entities extracted from Article categories (with stopwords)	DBpedia
ED_db	Entities extracted from Disambiguation (without stopwords)	DBpedia
cED_db	Entities extracted from Disambiguation (with stopwords)	DBpedia
RE_db	Entities extracted from Redirects (without stopwords)	DBpedia
cRE_db	Entities extracted from Redirects (with stopwords)	DBpedia
E_dbFacc	Common entities extracted from Facc1 and DBpedia’s Article categories (without stopwords)	Facc1 and DBpedia
cE_dbFacc	Common entities extracted from Facc1 and DBpedia’s Article categories (with stopwords)	Facc1 and DBpedia
E_Similar	Upper- and lower-case modified entities from DBpedia’s Article categories	DBpedia

, where each component/class in the table corresponds to a surface form.

For a given text, which is assumed to be a paragraph, an n-gram method is used to find all the possible candidate entities corresponding to each mention. The candidate entities are extracted from the surface forms. Therefore, from a given paragraph, all possible mentions, which exist on the surface, would be detected.

3.1.3. Candidate Selection

The main role of the candidate selection method is to select the most probable candidate entity for each mention. There is at least one candidate entity for each mention. Moreover, some candidate entities could be included in others. Also, a mention could be included in another one; in such a case, the included one would be ignored. For example, if we consider the following sequence of words “The Empire State Building …” with the mention detection step, the three following mentions could be detected from the surface forms: “Empire”, “Empire State”, and “Empire State Building”. One can observe that the first two mentions are included in the third one; if there are one or more candidate entities, in the surface forms for the third mention, then the first and second mentions will be ignored. In the same way, the candidate entities detected for the first and second mentions will be ignored.

The selection score of a candidate entity is computed by considering different factors, such as:

• The component weight: This is a defined weight according to each component of the surface forms (components) (Table 2).
• The contextual similarity score: This is defined as a score of similarity between the entity terms and the given paragraph.
• The number of terms in the entity. These score computations are used in the candidate selection algorithm to select the most appropriate entity for each mention.

3.1.4. Disambiguation

The disambiguation task is achieved using a graph-based algorithm, which is the central part of our EL4DT method. The constructed graph is a weighted graph G = (V, E), where the node set V contains all the selected candidate entities from a given paragraph, and each edge represents the semantic relationship between two entities. The main goal of the disambiguation algorithm is to select among the ambiguated entities (entities with weak scores) only those related to sure entities (entities with scores higher than or equal to 0.5). Thus, other weak entities are ignored. Furthermore, EL4DT identifies the best cluster, the group of related entities with the higher score among other clusters in the graph. In other words, the best cluster in the graph is supposed to be the paragraph’s main idea. In a graph, we note that a cluster is a set of entities connected by edges whose weights are greater than zero.

The set of entities in a paragraph is expressed by $E_p=\{e_1,e_2,....,e_n\}$, where n represents the number of entities in the paragraph.

The $E_{coh}$ symbol stands for coherence entities (document entity coherence), which exist in the document title and the document’s strong entities. The strong entity concept refers to entities annotated by EL4DT with an annotation score higher than or equal to 0.85. Thus, $E_{coh}$ represents the intersection between the document’s strong entities ${SE}_d$ and the document’s title entities $E_{dt}$, as Equation (1) shows.

$$E_{coh}=E_{dt}\bigcap {SE}_d$$

(1)

After the graph initialization step, the graph scoring expression is based on Equation (2), which shows how the graph edges are scored:

$$GS\left(e_i,e_j\right)=\frac{rScore(e_i,e_j)\times \left(LScore\right(e_i)+LScore(e_j\left)\right)}{\left|E_p\right|}$$

(2)

where $\mathrm{r}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}(e_i,e_j)$ gives the number of relationships between two entities, as computed by Equation (3), where the R symbol represents the different types of relationships between the two entities. Moreover, the relationship between the two entities is either direct or indirect. Direct relationships exist when entities share the common article categories of DBpedia. Moreover, from the SKOS of DBpedia, there are some relationships between entities such as the <skos:broader> and <skos:related> predicates. However, an indirect relationship means that $e_i$ and $e_j$ have no direct relationships but are related by $E_{coh}$ as an indirect relationship. The following Equations (3)–(6) explain Equation (2), respectively:

$$rScore(e_i,e_j)=n(e_i,e_j,R)$$

(3)

$$\begin{array}{cc}\hfill n\left(e_i,e_j,R\right)=& {\displaystyle \sum _{c\in ACatg\left(e_i\right)}}\left\{exist\left(c,ACatg\left(e_j\right)\right)\right\}+exist\left(e_i,SkosR\left(e_j\right)\right)+exist\left(e_j,SkosR\left(e_i\right)\right)\hfill \\ & +exist\left(e_i,e_j,E_{coh}\right)\end{array}$$

(4)

$$exist(c,ACatg(e_j\left)\right)=\left\{\begin{array}{c}1,c\in ACatg\left(e_j\right)\\ 0,otherwise.\hfill \end{array}\right.$$

(5)

$$exist\left(e_i,e_j,E_{coh})\right)=\left\{\begin{array}{l}1,{(e}_i{e}_j)\in E_{coh}\\ 0,otherwise.\end{array}\right.$$

(6)

Moreover, the $\mathrm{L}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\left(e_i\right)$ gives the entity score computed by EL4DT in the candidate selection process, which mainly calculates the three factors (assignment of components weights, score of contextual similarity, and number of terms in the entity) when considering the different surface forms and the three types of entity, such as the non-sure, sure, and strong entities. $ACatg\left(e_i\right)$ provides the entity category of the entity $e_i$, which includes all entities of the same DBpedia category. $SkosR\left(e_i\right)$ provides the Skos relations (<skos:broader> and <skos:related> predicates) of the entity $e_i$, which are extracted from DBpedia.

3.1.5. Algorithm

The following algorithm (Algorithm 1) represents an overview of our entity linking method. It gives the main processes of entity linking for a given document, starting with the document text as input and the corresponding annotations as output.

Algorithm 1: EL4DT algorithm (Mention Detection, Candidate Selection, Disambiguation)

1: Input: $T_d$ ← document_text 2: Output: $document\_annotations$ 3: for $T_p$ ∈ $T_d$ do 4:                  ms ← find_allPossible_candidate_entities ($T_p$) 5:                  $E_p$ ← select_candidate_entity (ms, $T_p$) 6: end for 7: $E_{coh}\leftarrow E_{dt}\bigcap {SE}_d$ 8: for $E_p$ ∈ $E_d$ do 9:                   G (V, E) ← graph_initialization ($E_p$) 10:                  for v, e ∈ G do 11:$\mathrm{e}\leftarrow \frac{\mathrm{r}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}({\mathrm{e}}_{\mathrm{i}},{\mathrm{e}}_{\mathrm{j}})\times \left(\mathrm{L}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\right({\mathrm{e}}_{\mathrm{i}})+\mathrm{L}\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}({\mathrm{e}}_{\mathrm{j}}\left)\right)}{\left|E_p\right|}$ 12:                  end for 13:                  $E_p$ ← select_disambiguited_entity_set (G) 14:                  $documen{t}_{annotations}=adding\left(E_p\right)$ 15: end for

3.2. Indexing

Our approach needs an appropriate indexing method that considers all the required information and statistics for our entity linking method (EL4DT). In fact, there are many indexing techniques; the inverted index technique is among the most popular ones and is known for its efficiency and simplicity [38]. Our inverted index method performs the indexing task. It considers all the needed information for our retrieval and ranking method. Figure 2 illustrates the index structure performed by our inverted index method.

In the figure, each line corresponds to an entity ${\mathrm{e}}_{\mathrm{i}}$ in all the documents in which it occurs and to all the other important details. docNo represents the document’s identifier; EOccNbD represents the entity occurrence number in the document; pargNo is the paragraph’s identifier; NbEp represents the number of entities in the paragraph; and isStrong takes the values (0 or 1), which stand for a non-strong or strong entity, respectively. NbSEp represents the number of strong entities in the paragraph, and NbRE is the number of semantically related entities identified in our entity linking method.

3.3. Retrieval and Ranking Method

This section introduces the retrieval and ranking method we designed for our approach as one of its key elements. The retrieval process of the method is an end-to-end process that retrieves all relevant documents for a given query. In this section, we first provide the ranking function and its key elements; then, we provide the algorithm of the retrieval and ranking method (Algorithm 2).

3.3.1. Document Scoring

According to our approach, the computing document scores for a given query (completely annotated query) need an appropriate and relevant solution. To achieve this goal, we designed and developed the following ranking method, which is mainly based on the following Equation (7). The equation allows the computation of the document relevance score by summing the relevance score of each paragraph in the document against the query entities.

$$S\left(q,d\right)=\sum _{p\in d}\frac{{\sum }_{e\in E_q}[nb\_rE(e)\times nbT(e\left)\right]\times {\left|E_{qp}\right|}^2\times e^{\left[\left|{SE}_{qp}\right|\right]}}{\left|E_p\right|+e^{\left[\left|S{E}_p\right|-\left|S{E}_{qp}\right|\right]}}$$

(7)

where $nb\_rE\left(e\right)$ gives the number of related entities (from the index), and $nbT\left(e\right)$ gives the number of terms in entity $e$. Moreover, $\left|E_{qp}\right|$ represents the number of query entities found in the paragraph $p$. Furthermore, the exponential function in $e^{\left[\left|{SE}_{qp}\right|\right]}$ is used to weight the number of query entities located in paragraph $p$ as strong entities according to the index information.

3.3.2. Title Weighting

The document title plays an important role in the document retrieval task. In our approach, we also consider document titles and compute their weights in our retrieval and ranking method. The document title weight is computed according to Equation (8):

$$TW\left(q,E_t\right)=\left|E_{qt}\right|\times S\left(q,d\right)\times w,$$

(8)

where $\left|E_{qt}\right|$ is the number of query entities present in the document title, and $w$ is a parameter used to balance the influence of the title weight in the document scoring process. Its value is an arbitrary value established after many tests ($w=0.01$). Finally, the document title weight is added to the document score.

3.3.3. Algorithm

In this section, we provide the algorithm of our document retrieval and ranking method (Algorithm 2) and highlight all the main details. The algorithm shows how to compute the ranking score for retrieved documents corresponding to a given query, where only completely annotated queries are considered.

The inputs are $E_p$, which represents the entities of the given query; subIndexAsRaws represents the loaded lines from the index corresponding to each entity of the query. In line three (3) of the algorithm, the getStatisctics() function extracts all the statistics and information from the raw lines of subIndexAsRaws. In line (4), the getAllFoundDocsIDs() function retrieves all documents that contain at least one entity of the given query entities, which are the documents of concern. The rest of the algorithm shows how the ranking score is computed for each pair of document queries. Finally, the algorithm returns $ScoreQ\_docs$, which represents the ranked document list with a ranking score for each retrieved document.

Algorithm 2: Retrieval and Ranking Method

1: Input: $q,E_p$, subIndexAsRaws 2: Output: $ScoreQ\_docs$ 3: entity_index_info ← getStatisctics(subIndexAsRaws) 4: retrievedDocs ← getAllFoundDocsIDs(entity_index_info) 5: for $d$ ∈ retrievedDocs do 6:       $S\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{Q}\_\mathrm{d}\mathrm{o}\mathrm{c}\mathrm{s}\left(q,d\right)\leftarrow {\sum }_{p\in d}\frac{{\sum }_{e\in E_q}[nb\_rE(e)\times nbT(e\left)\right]\times {\left|E_{qp}\right|}^2\times e^{\left[\left|{SE}_{qp}\right|\right]}}{\left|E_p\right|+e^{\left[\left|S{E}_p\right|-\left|S{E}_{qp}\right|\right]}}$ 7:       $TW\left(q,E_t\right)\leftarrow \left|E_{qt}\right|\times S\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{Q}\_\mathrm{d}\mathrm{o}\mathrm{c}\mathrm{s}\left(q,d\right)\times w$ 8:       $S\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{Q}\_\mathrm{d}\mathrm{o}\mathrm{c}\mathrm{s}\left(q,d\right)\leftarrow S\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{Q}\_\mathrm{d}\mathrm{o}\mathrm{c}\mathrm{s}\left(q,d\right)+TW\left(q,E_t\right)$ 9: end for

4. Results

In the results section, we provide the data collection used in the experiments, the evaluation metrics, and the implementation details used for conducting the experiments.

4.1. Data

In our experiments, we use the standard TREC 2004 Robust collection, which was used in TREC 2004 Robust Track. Also, we use the MS MACRO collection [39], which is a large-scale dataset focused on machine reading comprehension, question answering, and passage/document ranking.

Table 3. Usage of MSMARCO and Robust04 collections.

Collection	Queries (Title Only)	#Docs	Qrels
TREC Disks 4 & 5 minus CR	TREC 2004 Robust Track, topics 301–450 & 601–700	528k	Complete qrels 1
MSMARCO v1	TREC-DL-2019 and TREC-DL-2020 (200, 200)	3.2M	ir_datasets (Python API) 2

¹ https://trec.nist.gov/data/robust/qrels.robust2004.txt; ² https://ir-datasets.com/msmarco-document.html. Accessed on 2 August 2023.

shows information about our use of these collections.

For the TREC 2004 Robust collection, the query set includes 250 queries, and only the query titles are annotated without their descriptions. Moreover, for the MSMARCO collection, version 1 is used. Two query sets are used: TREC-DL-2019 and TREC-DL-2020; these two sets are provided with qrels suitable for evaluation with the nDCG metric.

4.2. Evaluation Metrics

Three standard evaluation metrics are used to evaluate the results. The normalized Discounted Cumulative Gain (nDCG) is the metric for non-binary relevance judgments. NDCG@20 is the official TREC Web Track ad hoc task evaluation metric for the top 20 ranked documents. The second metric is the mean average precision (MAP) of the top 1000 ranked documents. The third metric is P@20, which provides the precision of the top 20 retrieved documents. Moreover, with regard to the importance of the top five ranked documents, NDCG@5 is also used as an evaluation metric for an ad hoc document retrieval task. It is important to note that the nDCG@5 evaluation metric is used to evaluate the performances of many ranking models [40,41,42].

4.3. Results of Experiments on Robust04

4.3.1. Query Annotation

Our approach is purely based on entity representation for documents and queries. Assuming that we have the best-designed purely entity-based retrieval system, including the best representation of the document and the best ranking method, if the given query is not completely annotated, the system will not work well because of the ignored term(s) from that query (not-annotated term(s)). Therefore, query annotation is the critical factor in a purely entity-based retrieval system, which is why we consider only completely annotated queries in our experiments. We tested our retrieval approach by using two arbitrary entity linking methods for query annotation, including DBpedia Spotlight [11] and REL [13]. Moreover, the two Python APIs provided in

Table 4. Completely annotated queries by DBpedia Spotlight and REL for Robust collection queries.

Entity Linking Method	#Completely Annotated Queries	% of Completely Annotated Queries	Usage
DBpedia Spotlight	180	72%	Spotlight Python Library (v0.7) 1
REL	17	6.8%	Python APl 2

¹ https://pypi.org/project/spotlight/; ² https://github.com/informagi/REL. Accessed on 2 August 2023.

are the used implementations corresponding to each of these two entity linking methods. Then, to check whether a query was completely annotated, we compared the found mentions with the original query text. Also, the stopwords were not considered if they were not represented in the detected mentions. Furthermore, we considered the average score of the query entities’ annotation scores to be the indicator of query annotation quality. Table 4 presents the number of completely annotated queries by each entity linking method.

Table 4 contains only the queries completely annotated by both the DBpedia Spotlight and the REL entity linkers. Hence, the process was applied to all Robust04 queries (250 queries). Also, we note that no changes were made to REL’s results or DBpedia Spotlight’s query annotations. Later in our tests, we classified the queries according to the average scores of their annotation scores to show the corresponding performances and to understand how to effectively leverage a purely entity-based approach.

Before explaining the results of the experiment achieved with the Robust collection, it is crucial to clarify some information about the annotation scores achieved by both the DBpedia Spotlight and the REL entity linkers, where both methods offered scores between 0 and 1. However, the scoring systems and the meanings of the scores are different. As with the probability logic, the general interpretation is the same for both methods, which states that the closer the annotation score is to 1, the more accurate the annotation is, and vice versa when the annotation score is closer to zero.

We classified the completely annotated queries into four classes according to their annotation average scores with regard to each entity linking method. The reason for this classification is to observe the performance of the PESS4IR method while the annotation score increases. Moreover, four classes (four is an arbitrary number) are suitable for the results’ readability. So, the four average score classes are (min = 0.65, 0.85, 0.95, and 1.00) and (min = 0.50, 0.65, 0.70, and 0.75) for DBpedia Spotlight and REL, respectively. Moreover, for each of these average scores, the corresponding query numbers are (154, 132, 109, and 3) and (12, 9, 4, and 2), respectively, for each method. We note that DBpedia Spotlight tends to assign higher scores than the REL tool; thus, we assigned the average score class values of DBpedia Spotlight a higher place than those of the REL entity linker. Finally, for both entity linkers, the completely annotated queries were separately classified into these different classes. Figure 3 shows the performances of PESS4IR and the Galago (Dirichlet model) according to each class of queries, where the PESS4IR queries are annotated by the DBpedia Spotlight and REL entity linkers (respectively in (a) and (b)). In the figure, the performance is presented by the NDCG@20 scores.

In Figure 3a, Galago (Dirichlet model) outperforms our approach for the first three classes of queries, where the queries are annotated by the DBpedia Spotlight entity linker. However, for the last query class, PESS4IR outperforms Galago, where the average annotation score of that class is equal to (1.0). The corresponding nDCG@20 scores are provided in

Table 5. nDCG@20 scores for queries’ classes given by DBpedia spotlight.

Method	nDCG@20
	AVGs ≥ Min (154 Queries)	AVGs ≥ 0.85 (132 Queries)	AVGs ≥ 0.95 (109 Queries)	AVGs = 1.0 (3 Queries)
Galago (Dirichlet)	0.3498	0.3643	0.3839	0.5702
PESS4IR	0.2160	0.2257	0.2355	0.6207

, with more details.

In Figure 3b, for the last queries class, which corresponds to average annotation scores larger than or equal to (0.75), PESS4IR outperforms the Galago method. Moreover, the corresponding nDCG@20 scores are listed in

Table 6. nDCG@20 scores for queries’ classes given by REL.

Method	nDCG@20
	AVGs ≥ Min (12 Queries)	AVGs ≥ 0.65 (9 Queries)	AVGs ≥ 0.7 (4 Queries)	AVGs ≥ 0.75 (2 Queries)
Galago (Dirichlet)	0.4216	0.4500	0.4360	0.6759
PESS4IR	0.3036	0.3670	0.4038	0.7306

.

From the perspective of a retrieval system based on a multi-method approach, which leverages different approaches (such as the retrieval method) for better document information retrieval, PESS4IR could be leveraged for the well-represented queries (queries with high annotation scores, such as the class of the highest average annotation scores, in (Table 5 and Table 6)), and other methods for the rest of queries. In fact, due to its autonomy, the PESS4IR approach could be used alongside any other document retrieval method.

Table 7. PESS4IR’s added value with Galago.

Method	nDCG@5	MAP	P@5
Galago (Dirichlet)	0.3729	0.1534	0.3855
Galago + PESS4IR	0.3758	0.1540	0.3855

illustrates the added value of PESS4IR when it is used alongside Galago. Moreover, for Galago, the added value is expressed by all the used metrics. Furthermore, PESS4IR provides an added value for any state-of-the-art method used on the TREC 2004 Robust collection, such as when one uses PESS4IR for the highly annotated queries (AVG_score ≥ 0.75, annotated by REL) and any other state-of-the-art method for the rest of the queries. In this case, an added value is offered by PESS4IR unless the SOTA method alone can reach the maximum nDCG@5 score for those highly annotated queries. In the discussion session (see Section 5.1), we give more details about the added value offered by PESS4IR.

4.3.2. PESS4IR with LongP (Longformer) Model

We compared PESS4IR with the LongP (Longformer) model and combined them to have a better performance for the ad hoc document retrieval task. Moreover, in this experiment, PESS4IR was used for the set of queries with the highest representation (AVG_score ≥ 0.75, annotated by REL), and LongP (Longformer) was used for the rest of the queries. The added value achieved by PESS4IR appeared clearly when it was used alongside the LongP model. The results are illustrated in

Table 8. PESS4IR and LongP (Longformer) on Robust Collection.

Method	nDCG@5	MAP	P@5
LongP (Longformer)	0.6542	0.3505	0.6723
LongP (Longformer) + PESS4IR	0.6551	0.3492	0.6731

.

In this table, the evaluation is given by the nDCG@5 metric, where it shows a better ranking performance; this is due to the outperformance of PESS4IR upon the LongP (Longformer) model for the queries with the highest annotation. In Section 5.1, we provide the details of the added value achieved by PESS4IR.

4.4. Results of the Experiment on MSMARCO

We would have tested PESS4IR by queries annotated by both the REL and the DBpedia Spotlight entity linking methods; however, we tested it only with the DBpedia Spotlight tool (see Appendix B.3 and Appendix B.4). The reason for not testing PESS4IR with queries annotated by the REL tool is that among the completely annotated queries, of the query sets of both TREC-DL-2019 and TREC-DL-2020, there is no query whose annotation avg score is greater than 0.75 (for the TREC-DL-2020 query set, see Appendix B.1); and for the query set of TREC-DL-2019, there is only one query, but it has a scoring issue (see Appendix B.2). In the discussion section (see Section 5.3), we provide the details related to that scoring issue.

In the following table (

Table 9. nDCG@20 scores for PESS4IR and LongP (Longformer) on MSMARCO.

Method	nDCG@20
	TREC DL 2019			TREC DL 2020
	Avg ≥ Min (24 Queries)	Avg ≥ 0.95 (16 Queries)	Avg = 1.0 (1 Query)	Avg ≥ Min (24 Queries)	Avg ≥ 0.95 (10 Queries)	Avg = 1.0 (1 Query)
LongP (Longformer)	0.7179	0.7464	None	0.6850	0.6605	None
PESS4IR	0.3734	0.3970	None	0.2970	0.3055	None

), we provide the performance and results of our approach (PESS4IR) and of the LongP (Longformer) model on the MSMARCO collection. Moreover, the nDCG@20 metric is used. It is important to note that the Python API of the MSMARCO collection does not provide judgment values (qrels) for some queries of the TREC DL 2019 and TREC DL 2020 query subsets. Among them, the queries with the highest annotation exist (annotated by the DBpedia Spotlight tool, whose annotation avg score is equal to 1), and with the queries with the highest annotation, PESS4IR is supposed to achieve its best performance. However, the corresponding judgment values are provided by the MSMARCO collection.

The LongP (Longformer) model outperforms PESS4IR for the first two classes for each set (TREC DL 2019 and TREC DL 2020). And for the last class, where PESS4IR is supposed to achieve its best performance with the queries with the highest annotation, there are no corresponding qrels; this why we put “None”. Consequently, the absence of highly annotated queries (with an annotation avg ≥ 0.75) annotated by the REL tool and the absence of corresponding qrels for the highly annotated queries annotated by the DBpedia Spotlight tool (with an annotation avg = 1.0) are the two reasons why we did not provide the combination of both PESS4IR and Long (Longformer) on the MSMARCO collection.

5. Discussion

In the discussion section, we explain and discuss the added value achieved by PESS4IR when it is tested with queries annotated by the REL and DBpedia Spotlight entity linking methods. Moreover, we discuss our experiments presenting the strengths and limitations of our approach (PESS4IR).

5.1. PESS4IR Tested by REL

As a purely entity-based method is appropriate only for completely annotated queries, the results are partial when only completely annotated queries are considered. Among them, the ones with higher scores perform better than the rest. Thus, the purely entity-based approach is recommended for highly represented queries (whose entities have high annotation scores). Furthermore, this session shows how our approach achieves the maximum nDCG@5 score. The following experiment shows how our approach offers added value. In the experiment, we use the REL entity linking method for query annotation (see Appendix A.2).

Table 10. REL annotations whose average scores are higher than or equal to 0.75.

qID	Query Text	REL Annotations (“Mention” → Entity → Score)	Annotation Avg Scores
365	El Nino	“El Nino” → El_Niño → 0.76	0.76
423	Milosevic, Mirjana Markovic	“Milosevic” → Slobodan_Milošević → 0.99 “Mirjana Markovic” → Mirjana_Marković → 0.98	0.98

contains highly represented queries whose entities have an average annotation score greater than or equal to 0.75. The table also contains the query text which is the original text (query title). For each query, in REL annotation column, it shows the detected mentions with the corresponding entities and their annotation scores.

In addition, to analyze the performance achieved by our purely entity-based approach, we present the results of the PESS4IR together with the results of the LongP (Longformer) and the Galago (Dirichlet) models. Moreover, Figure 4 shows the results of our approach in comparison with those of the two models. In the experiment, as illustrated in

Table 11. nDCG@5 scores for queries’ classes given by REL on Robust collection.

Method	nDCG@5
	AVGs ≥ Min (12 Queries)	AVGs ≥ 0.65 (9 Queries)	AVGs ≥ 0.7 (4 Queries)	AVGs = 0.75 (2 Queries)
Galago (Dirichlet)	0.4976	0.5097	0.5463	0.9152
LongP (Longformer)	0.7122	0.7085	0.7769	0.8962
PESS4IR	0.3336	0.4071	0.4781	1.0000

and Figure 4, the nDCG@k scores are for the top five ranked documents, and the results of the LongP (Longformer) model are provided.

With these results, our approach achieves the maximum nDCG@5 score of 1.000 for the queries with the highest representation (annotated by the REL entity linking method). This score is an added value for any document retrieval method which does not reach that score. Moreover, in this experiment, the achieved score is the maximum nDCG@5 score, corresponding to only two queries. This low number of queries represents a limitation of our approach. But this limitation is caused by the comparison requirements, where we obtain that maximum nDCG@5 score after selecting the queries with the highest representation (with the highest annotation average score). Thus, this is how we compared our approach to any other document retrieval approach.

Table 11 shows how the LongP (Longformer) model outperforms the PESS4IR and Galago methods, with a big difference for the first three classes of queries. However, with the queries of the last class (queries whose entities have an average annotation score greater than or equal to 0.75), the LongP (Longformer) model and Galago (Dirichlet) are outperformed by our PESS4IR approach. It is important to note that for the queries of the last class, Galago (Dirichlet) outperforms the LongP (Longformer) model. While a strong approach has a good performance for the entire query set, another approach which is not as strong might offer an even better performance for some of those queries. This is one of the motivations of our approach.

5.2. PESS4IR Tested by DBpedia Spotlight

As we mentioned before, the DBpedia Spotlight entity linker tends to assign higher scores for annotations than the REL entity linking method, which is the reason why the group of queries with the highest representation has a high value.

Table 12. DBpedia Spotlight annotations whose average scores are equal to 1.0.

qID	Query Text	REL Annotations (“Mention” → Entity → Score)	Annotation Avg Scores
396	sick building syndrome	“sick building syndrome” → Sick_building_syndrome → 1.0	1.0
400	Amazon rain forest	“Amazon rain forest” → Amazon_rainforest → 1.0	1.0
429	Legionnaires’ disease	“Legionnaires’ disease” → Legionnaires’_disease → 1.0	1.0

contains highly represented queries whose entities have an average annotation score equal to 1.0. These queries are the queries with the highest representation according to the DBpedia Spotlight entity linker. For each query whose annotation average score is equal to 1.0, Table 12 shows the detected mentions with the corresponding entities and their annotation scores.

Moreover, the performance of the test performed with the DBpedia Spotlight entity linking method corresponding to the nDCG@5 scores is presented in

Table 13. nDCG@5 scores for queries’ classes given by DBpedia Spotlight on Robust collection.

Method	nDCG@5
	AVGs ≥ Min (154 Queries)	AVGs ≥ 0.85 (132 Queries)	AVGs ≥ 0.95 (109 Queries)	AVGs ≥ 1.0 (3 Queries)
Galago (Dirichlet)	0.4001	0.4138	0.4392	0.6667
LongP (Longformer)	0.6737	0.6896	0.6946	0.8284
PESS4IR	0.2599	0.2739	0.2864	0.7380

.

As one can see in Table 12, PESS4IR with the queries annotated by DBpedia Spotlight achieves an added value for Galago with the higher represented queries. However, it is outperformed by LongP (Longformer) model for all the query classes. Thus, when PESS4IR is tested on the Robust04 collection by the queries annotated by the DBpedia Spotlight tool, it does not outperform the LongP (Longformer) model.

5.3. Query Annotation Weaknesses

The weakness of the annotation of a given query could be represented by using the DBpedia spotlight, REL, and TagMe tools for the query sets such as TREC 2019 and TREC 2020 of the MSMACO collection. In the following table (

Table 14. Example of query annotation weakness.

qID	Query Text	REL Annotations (Entity → Score)	DBpedia Spotlight (Entity → Score)	TagMe Annotations (Entity → Score)
1037798	“who is robert gray”	Robert_Gray_(poet) → 0.94	2015_Mississippi_gubernatorial_election → 0.69	Robert_Gray_(sea_captain) → 0.3

), we provide a weakness point of a purely entity-based approach, which is caused by the query annotation process.

We note that REL gives its annotations of this query as the highest-scoring annotated query, with an annotation score of 0.94. Such an annotation would surely negatively affect any purely entity-based approach. Generally, entity linking methods use a disambiguation process to select an entity among many candidate entities, as we explain in Section 3.1.4. In this case, for the given query “who is robert gray”, among many entities (known people, with the same name) the REL method selects “Robert_Gray_(poet)” to be the entity. However, the issue is the computed annotation score, which means that the selected entity is sure. In other words, the 0.94 score means that there is no way it could be another “robert gray”. However, with TagMe tool the selected entity was “Robert_Gray_(sea_captain)”, with 0.3 as the annotation score. Thus, such a case could negatively affect our approach and lead to a weak performance when it is supposed to have a better performance.

6. Conclusions

We introduce a purely entity-based semantic search approach for ad hoc information retrieval (PESS4IR) as a novel solution. The main goal of this paper is to analyze the impact of the purely entity-based semantic search on the effectiveness of ad hoc document retrieval by giving clear answers about when such an approach achieves its best and when it does not, showing its strengths and weaknesses. Our proposed approach represents queries and documents only by entity-based representation. It mainly includes its own entity linking method, which is appropriate for document text (EL4DT); it is an inverted indexing method and a method for document retrieval and ranking which is designed to leverage all the strengths of the approach. To evaluate the approach, we used the TREC 2004 Robust and MSMARCO collections, and we linked queries with two different entity linking methods, DBpedia Spotlight and REL. As our approach uses a purely entity-based representation for queries and documents, only completely annotated queries are considered. Galago (Dirichlet) and LongP (Longformer) models are used to compare the performance on the corresponding group of queries of the two collections and to show how PESS4IR could be compared to any other retrieval method.

In the experiments, we used the average annotation score of each query’s entities as only completely annotated queries are considered. The results indicate that as the average annotation score increases, the ranking score gets higher as well. Indeed, with the highest-scoring queries annotated by DBpedia Spotlight and REL, our approach outperforms the Galago method based on the nDCG@20 evaluation metric. Thus, our approach offers an added value when used with the Galago (Dirichlet method) or LongP (Longformer) model. For the queries with the highest annotation average score (avg_score ≥ 0.75) among the queries annotated by the REL entity linking method, our approach achieved the maximum nDCG@5 score (1.00), which would be an added value for any ad hoc document retrieval method that does not reach the same score for the same queries. The LongP (Longformer) model is the confirmation of this added value reached by PESS4IR, where it is among the current state-of-the-art models.

For further research, how to increase the quality and number of completely annotated queries can be investigated. It would also be interesting to investigate how to conduct automatic query reformulation and query recommendation tasks based on knowledge bases, with purely entity-based representation as the output of these techniques.