Individual- vs. Multiple-Objective Strategies for Targeted Sentiment Analysis in Finances Using the Spanish MTSA 2023 Corpus

Abstract

Multitarget sentiment analysis extracts the subjective polarity of text from multiple targets simultaneously in a given context. This approach is useful in finance, where opinions about different entities affect the target differently. Examples of possible targets are other companies and society. However, typical multitarget solutions are resource-intensive due to the need to deploy multiple classification models for each target. An alternative to this is the use of multiobjective training approaches, where a single model is capable of handling multiple targets. In this work, we propose the Spanish MTSACorpus 2023, a novel corpus for multitarget sentiment analysis in finance, and we evaluate its reliability with several large language models for multiobjective training. To this end, we compare three design approaches: (i) a Main Economic Target (MET) detection model based on token classification plus a multiclass classification model for sentiment analysis for each target; (ii) a MET detection model based on token classification but replacing the sentiment analysis models with a multilabel classification model; and (iii) using seq2seq-type models, such as mBART and mT5, to return a response sequence containing the MET and the sentiments of different targets. Based on the computational resources required and the performance obtained, we consider the fine-tuned mBART to be the best approach, with a mean F1 of 80.300%.

1. Introduction

Analyzing sentiment in the context of business and finance presents a unique set of challenges. Financial documents can contain complex language, jargon, or numerical data that can confound sentiment analysis (SA) models. The sensitivity of markets to rapidly changing events and their susceptibility to small linguistic changes make it difficult to accurately predict market reactions. Another challenge for SA in finance is that there are targets other than the primary target that may perceive sentiment differently than the primary target. Examples of these targets include entire industries, consumers, or the market as a whole. Moreover, a change in sentiment towards one target can have a domino effect on other related targets due to the interconnectedness of different targets. For example, a positive earnings report from a technology company can not only affect sentiment toward that company’s stock but also have a broader impact on the technology sector and even the entire stock market. Similarly, changes in economic policy or geopolitical events can simultaneously affect sentiment toward currencies, commodities, and international markets.

Multitarget sentiment analysis (MTSA) enhances the analysis of text data by providing a more granular understanding of sentiment. Unlike plain SA, MTSA examines sentiment toward specific targets, such as investors, competitors, or society, within the same document. This granularity allows for a more nuanced and meaningful analysis. These insights are not only more informative but also more actionable. For example, decision makers can identify which specific targets are driving sentiment, enabling more precise strategies and responses. Other applications include product reviews, social media monitoring, or customer support.

One way to use MTSA is to train individual classification models for each target. Because each model is trained to analyze sentiment specifically for its designated target, there are several advantages: (i) specificity, as individual models allow for fine-grained SA and potentially obtaining more accurate results; (ii) scalability, as it is easier to scale model analysis across many targets when models are trained individually; and (iii) the isolation of training data, as models can be trained on target-specific datasets, ensuring that the training data are relevant to the sentiment of that particular target. However, the resulting models have two major drawbacks: (i) isolation, as models are not able to learn jointly from the domain, and (ii) time, memory, and inference resources grow in proportion to the number of targets, making these models infeasible in certain real-world situations.

In this work, we evaluate different MTSA strategies for the financial domain. Specifically, we collect and annotate a novel dataset of Spanish tweets about the economy, expressing the Main Economic Target (MET) and sentiments about the MET, the rest of the companies, and society. Next, we evaluate two strategies to reduce the requirements for training reliable MTSA models. The first strategy is multilabel training to reduce the need for separate sentiment classifiers for each target. The second strategy is a novel approach based on multitask learning, where all sentiments and the MET are extracted by using generative LLMs, such as BLOOM and BART.

The remainder of the manuscript is organized as follows. Section 2 reviews recent research on MTSA approaches and a general SA analysis on the financial domain. Section 3 describes the Spanish MTSACorpus 2023 dataset and the pipeline of our benchmark to compare the proposed strategies with the baselines. In Section 4, the results of the baselines, the multilabel strategy, and the generative approach are presented, and they are discussed in Section 5. Finally, the conclusions and suggestions for further work are given in Section 6.

2. Background Information

In this section, we examine novel approaches and strategies for conducting MTSA (see Section 2.1) and background information on SA applied to finances and economics in Spanish and other languages (see Section 2.2).

2.1. Multitarget Sentiment Analysis

SA focuses on extracting the subjective opinions expressed in a document, usually expressed as a binary classification (‘positive’ or ‘negative’), although other classifications add more degrees of subjectivity.

According to the specificity of SA, three levels can be distinguished depending on whether the document is analyzed as a whole (document-based), sentence-by-sentence (sentence-based), or aspect-based (ABSA). Of these approaches, ABSA is the only one that considers that a piece of text can express more than one sentiment. Specifically, a sentiment is identified for each subtopic. A detailed survey on ABSA can be found in [1], where the authors discover that deep learning architectures based on transformers are the most popular way to build ABSA systems. It is worth noting that depending on the domain, it is relevant to develop custom datasets for specific domains.

A novel strategy for performing SA that is attracting some academic attention is known as MTSA. Unlike traditional SA, which classifies the overall sentiment, or ABSA, which focuses on specific aspects within a single target, MTSA expands its scope by considering that there is more than one target that could be interested in the sentiment. Therefore, MTSA allows for the simultaneous analysis of sentiment toward multiple targets, such as the industry, company, or consumer, all within the same text. Similar to other automatic document classification tasks, recent advances in natural language processing (NLP) have led to the development of deep learning models, including transformer-based architectures, designed to capture complex relationships between different targets and their associated sentiments.

In Spanish, the work described in [2] describes an MTSA strategy composed of several individual models that are able to detect the MET and obtain its polarity and the sentiments toward other companies and citizens. This work also proposes a novel corpus composed of two data sources: tweets about finance and headlines from digital newspapers, both in Spanish. This dataset was evaluated by using automatic document classification models based on different LLMs, where the best results were obtained with LLMs trained on Spanish datasets (MarIA and BETO). In this work, a multilabel classification approach was also evaluated, and similar results were obtained. However, this approach does not take into account the MET extraction. It is worth mentioning that the [2] dataset has been extended and used as a common task in IberLEF 2023 [3].

2.2. Sentiment Analysis toward Finances and Economics

In the bibliography, it is possible to find several papers on SA applied in the financial domain. In [4], the authors used a dataset of about 1 million messages from StockTwits to test the reliability of different features and machine learning algorithms used in financial SA. The authors found that statistical features such as bigrams or emoticons significantly improved the performance of the models. However, the authors also found that other traditional machine learning algorithms did not improve performance. The authors also provide empirical evidence that there is a correlation between the preprocessing method and the size of the dataset with two variables: investor sentiment and stock returns. In [5], the authors developed a platform to evaluate several SA methods based on the integration of text features with deep and machine learning classifiers on several financial and economic datasets. Their results showed that transformers achieved state-of-the-art SA results. It is worth noting that lightweight distilled versions of transformers achieved similar results to their larger counterparts. In [6], a method called FiGAS (Fine-Grained Aspect-based Sentiment Analysis) was presented, which identifies the sentiment associated with specific topics in each sentence and assigns these topics polarity scores with real values between −1 and +1. It is an unsupervised approach for economic and financial domains, using a specialized lexicon provided by the authors. Some financial datasets that have been used in various studies such as [7,8] for the evaluation and development of sentiment analysis systems are:

• TRC2-Financial. This is a subset of the TRC24 (https://trec.nist.gov/data/reuters/reuters.html accessed on 7 February 2024) dataset from Reuters, consisting of 1.8 million articles between 2008 and 2010. In this case, the dataset used in [7] contains 46,143 documents with more than 29 million words and almost 400,000 sentences, which have been filtered for some key financial terms to make the corpus more relevant.
• Financial PhraseBank. This is a dataset consisting of 4845 English sentences randomly selected from financial news available in the LexisNexis database [9]. These sentences were then annotated by 16 people with experience in finance and economics. The annotators were asked to assign labels based on how they thought the information in the sentence might affect the stock price of the company mentioned. The dataset also provides information on the level of agreement between the annotators [7].
• FiQA. This is a dataset consisting of pairs of questions and answers about financial reports written by financial experts with annotated gold reasoning programs to ensure a complete explanation [9].

In Spanish, the work is scarce. For example, in [10], the authors explored different strategies for combining feature sets to improve the performance of applying SA to financial texts. In this paper, the authors published a dataset of nearly 16k tweets annotated with three levels of sentiment. The combination of linguistic features and embedding-based features achieved the best result in their experiments (a weighted F1 score of 73.159%).

3. Materials and Methods

In this section, we present a summary of the material, methods, and proposed approach. Figure 1 shows the pipeline of this work. In summary, the process is as follows. First, we collected tweets from Spanish economists and digital newspapers by using the UMUCorpusClassifier tool [11]. Second, the corpus is preprocessed to normalize the text format. Third, the corpus is divided into training, validation, and test subsets in a 60-20-20 ratio. Fourth, to analyze sentiment for different targets in financial tweets, we experimented with three approaches: (i) fine-tuning pretrained models based on transformers for SA and MET entity detection, i.e., a sentiment classification model for each target and a MET detection model; (ii) a MET detection model and a multilabel classification model for different targets; and (iii) fine-tuning an encoder–decoder model for target extraction and SA for three targets. For each approach, we evaluated different pretrained models, both monolingual and multilingual. Finally, we performed an evaluation to select the best approach.

3.1. Spanish MTSACorpus 2023

The Spanish MTSACorpus 2023 is compiled from X (formerly Twitter) by using the UMUCorpusClassifier tool [11]. Twitter is a valuable data source for SA because of several advantages. First, this social network provides a large and continuously updated stream of text data, offering a wide range of topics and real-time insights. Second, the brevity of microblogging encourages concise and focused language, making it suitable for text classification tasks. In addition, tweets often reflect the latest trends and public opinion, making them relevant for monitoring and analyzing current events and sentiment related to finance. Finally, hashtags and keywords facilitate the compilation process.

The Spanish MTSACorpus 2023 is compiled from the timelines of Spanish economists and digital newspapers focused on finance, following an approach similar to [10]. Some of the users are (i) el Economista (https://www.eleconomista.es/ accessed on 7 February 2024), (ii) Expansión (https://www.expansion.com accessed on 7 February 2024), and (iii) Cinco Días (https://cincodias.elpais.com/ accessed on 7 February 2024), to name a few. Figure 2 contains an example of a tweet from the dataset, stating that employment in the euro area and the EU has reached a new record, despite the economic slowdown.

The annotation process is performed by a group of 4 annotators who annotate the sentiment toward the three targets and extract the MET. The annotation rules of the MET were that the MET should appear literally in the text, the MET should be as short as possible while preserving the semantics, and starting pronouns should be absent.

In terms of sentiment, these were defined on a three-point scale: positive, neutral, and negative, with positive reflecting favorable opinions, negative indicating unfavorable sentiments, and neutral indicating the absence of a strong opinion. In terms of targets, the annotators consider society as the general public and companies as other companies or businesses apart from the MET, but not limited to the same sector.

Disagreements among the annotators were addressed in daily meetings, where annotations that deviated from the annotation guidelines were corrected. Cases where consensus could not be reached were removed from the final dataset.

The resulting dataset contains 7020 instances, which are divided into training, validation, and test sets in a ratio of 60-20-20.

Table 1. Spanish MTSACorpus 2023 statistics per target.

	Label	Train	Val	Test	Total
MET	Positive	1848	658	629	3135
	Neutral	692	214	230	1136
	Negative	1672	532	545	2749
Companies	Positive	954	329	342	1625
	Neutral	2090	702	679	3471
	Negative	1168	373	383	1924
Society	Positive	948	334	338	1620
	Neutral	1977	653	651	3281
	Negative	1287	417	415	2119
	Total	4212	1404	1404	7020

shows the statistics of the Spanish MTSACorpus 2023 for the three evaluated targets. Note that the training and validation sets are stratified to avoid introducing bias by splitting the dataset.

To analyze the dataset, we observe the information gain for each linguistic feature obtained with the UMUTextStats tool [12]. Figure 3a–c show the top ten normalized linguistic features for the MET, companies, and society, respectively. It can be observed that, regardless of the target, the presence of negative words is correlated with negative emotions, especially lexical items related to sadness and, to a lesser extent, to anger. However, in the case of documents labeled as positive, only positive words are strongly correlated with MET, but not with companies or society. In fact, we observed that the feature most correlated with positive documents is the percentage symbol for companies and the country lexicon for society. There are other features that are relevant for two of the targets, as it is the case of determiners, since we observed a correlation when the targets are other companies or society. Finally, we also observe that suffixes with an appropriate connotation are relevant for the MET.

The dataset is available at the following link: https://github.com/NLP-UMUTeam/Spanish-MTSA-2023 (accessed on 7 February 2024). The dataset includes the MET, the sentiment of the three targets, and the split, but in order to respect Twitter’s policy and users’ right to privacy, we only share the Twitter IDs, not the text.

3.2. Approaches

In this section, we explain the different methods and approaches used to create a MTSA system. The MTSA models will output the MET, which is the main target of the text, and the sentiments toward three targets: the MET, other companies, and society. Thus, we have one sequence classification problem, i.e., the extraction of the MET, and three sentiment classification problems, one for each target. In the following sections, we evaluate different strategies for solving all these subtasks.

3.2.1. Strategy A: MET Detection Using Token Classification and Sentiment Classification Using Multiclass Classification

In this approach, we considered the problem of MET detection as a token classification task and SA as a multiclass classification task. In Figure 4, we can see the architecture of the system, and we proceeded to fine-tune different pretrained models by using the examples available in the dataset we collected.

Fine-tuning refers to the process of taking a pretrained model and adjusting it to perform a specific task by using the knowledge acquired in the pretraining phase of the models. Moreover, we can observe that before the fine-tuning process, the dataset was tokenized, which involves converting text strings into integer token IDs that can be read by transformer-based pretrained models through the associated tokenizer.

We considered MET detection as a token classification task, where labels are assigned to each token in a text sequence, similar to Named Entity Recognition (NER) models. However, instead of including multiple entities like other existing NER models, we included only the “target” entity. Therefore, using the IOB2 (Inside, Outside, and Beginning) format to label the tokens, there are a total of three entity classes for each word token: (i) B-TARGET, indicating the beginning of the target entity; (ii) I-TARGET, indicating that the token belongs to the target entity; and (iii) O, indicating that the token does not belong to any target entity. To allow the fine-tuned models to assign labels to each token in an input sequence, we added a token classification layer to the model.

When annotating the datasets, we considered that the sentiment of the entities is represented by different emotional states, such as positive, neutral, and negative. Therefore, the goal of the SA model is to assign one of the three categories to a given sequence, indicating the sentiment it represents. In Figure 4, we can see that the architecture of the SA model is similar to that of MET detection, but in this case, a sequence classification is added, which is responsible for assigning labels to text sequences based on the available categories.

Recent developments in LLM have demonstrated their success in many NLP tasks in different languages. For Spanish, there are several monolingual and multilingual models based on BERT [13], RoBERTa [14], ALBERT [15], DistilBERT [16], and XLM-RoBERTa [17], pretrained on different text sources. Therefore, we evaluated the performance of different pretrained LLMs for the tasks of MET detection and SA.

The following models were evaluated: (i) BETO [18], an LLM model based on BERT and trained with the Spanish Unannotated Corpora; (ii) MarIA [19], based on RoBERTa and pretrained with web crawlings from the National Library of Spain; (iii) BERTIN [20], which is another model based on RoBERTa but trained with the Spanish part of the mC4 dataset; (iv) ALBETO [21], a version of ALBERT, which is a lightweight version of BERT, pretrained only with documents written in Spanish; (v) DistilBETO (DBETO) [21], a version of DistilBERT (another lightweight version of BERT), trained by using distillation techniques to transfer the weights of BETO to a new model with fewer layers and less complexity; (vi) RoBERTuito [22], a pretrained language model for user-generated content in Spanish, trained following RoBERTa guidelines on 500 million tweets; and (vii) XLM-RoBERTa [17], a multilingual version of RoBERTa, trained with data filtered from CommonCrawl from 100 different languages.

The hyperparameters used to fine-tune the pretrained models for MET detection are a 16-train batch size, 10 epochs, 0.01 weight decay, and a learning rate of 2 × 10⁻⁵. For sentiment detection, we perform hyperparameter tuning of the LLMs, evaluating the learning rate, number of training epochs, batch size, warmup steps, and weight decay. This experiment is performed by using the RayTune tool [23]. The number of models evaluated is 10 for each LLM and target. The parameters are selected by using Distributed Asynchronous Hyperparameter Optimization (HyperOptSearch) with the Tree of Parzen Estimators (TPE) [24] and the ASHA scheduler.

Table 2. Results of the hyperparameter tuning for the multiclass classification model of sentiments. The parameters are the learning rate (LR, units: ×10⁻⁵), the number of training epochs (E, units: integer), the batch size (B, units: integer), the warm-up steps (WS, units of 1k), and the weight decay (WD, units: float).

	Target					Companies					Society
LLM	LR	E	B	WS	WD	LR	E	B	WS	WD	LR	E	B	WS	WD
ALBETO	4	4	16	0.5	0.19	4.2	5	8	0.25	0.20	3.1	5	8	1	0.016
BERTIN	3.6	3	16	0	0.27	3	4	16	0.5	0.04	2.9	5	8	1	0.028
BETO	1.7	5	8	0	0.03	2.7	3	8	0.25	0.11	2.9	3	8	0.25	0.15
DBETO	4.1	3	8	0.5	0.26	4.9	3	8	0	0.00	3	3	8	0	0.035
MarIA	2.6	3	8	1	0.05	1.7	3	8	0.25	0.19	2.9	2	8	0.25	0.02
RoBERTuito	2	6	16	0.5	0.01	2	6	16	0.5	0.01	2	6	16	0.5	0.01
XLM	2	4	8	0	0.09	4.2	3	16	0.25	0.12	3.2	5	8	0	0.13

shows the best configuration for each LLM and target. Note that the best results are different for each goal. For example, the number of training epochs is larger when the target is society and the warm-up steps are smaller when the target is other companies.

3.2.2. Strategy B: MET Detection Using Token Classification and Sentiment Classification Using Multilabel Classification

In the first approach (see Section 3.2.1), we created an SA model for each target. This results in a higher computational cost and longer inference time. To address this issue, in this approach, we kept MET detection as a token classification model and approached the multitarget SA problem from a multilabel perspective. In a multilabel classification problem, each text or tweet can be associated with multiple labels, which means that an instance can belong to more than one category at the same time as they are not mutually exclusive.

To achieve multilabel classification, we transformed the labels of the dataset through the “one-hot” representation, considering the sentiment of a tweet as a set of emotional states associated with each entity. Thus, the sentiment set is “target_pos”, “target_neu”, “target_neg”, “other_pos”, “other_neu”, “other_neg”, “society_pos”, “society_neu”, and “society_neg”, where each label represents the emotional state of a specific target. In this case, labels with the prefix ’other’ indicate the sentiments of companies, labels with the prefix “society” refer to the consumer entity, and labels with the prefix “target” refer to the sentiments of the MET.

Table 3. Illustration of the multilabel sentiment classification model with three simplification examples.

		Labels
	Text	Target	Companies	Society
1	Turistas chinos en España en Julio 74,367, crecimiento interanual 51%, YTD 309,742, interanual 60% cc @josecdiez @migsebastiang @plalanda_II. (Chinese tourists in Spain in July 74,367, year-on-year growth 51%, YTD 309,742, year-on-year 60% cc @josecdiez @migsebastiang @plalanda_II.)	POS	POS	POS
2	PepsiCo invertirá 31 millones de euros para construir una nueva fábrica de gazpacho Alvalle (PepsiCo will invest 31 million euros to build a new Alvalle gazpacho factory.)	POS	NEU	NEU
3	Mercedes prepara el ataque a Tesla: invertirá 1.000 millones para asaltar ‘su’ mercado eléctrico… (Mercedes prepares attack on Tesla: will invest 1 billion to assault ‘its’ electric market…)	NEG	NEU	NEU

shows the output of the multilabel model with a set of three simple examples. In this approach, the same pretrained models were evaluated as in the previous approach (see Section 3.2.1). For multilabel classification, we performed an epoch-based hyperparameter search. That is, we keep variables such as weight decay and learning rate fixed while varying the value of epochs. This allows the model to perform evaluation and adjustment after each epoch, thus selecting the best model. The hyperparameters used to fine-tune pretrained models for multilabel classification are a training batch size of 8, 6 epochs, a weight decay of 0.01, and a learning rate of 2 × 10⁻⁵.

3.2.3. Strategy C: Multitarget Sentiment Classification Model

An inconvenience of the previous approaches (see Section 3.2.1 and Section 3.2.2) is that at least two models are required, one for MET detection and another for SA, which still implies a high computational burden. To address this issue, we propose an MTSA approach that involves building a sequence-to-sequence (Seq2Seq) model capable of simultaneously providing MET and sentiment for different targets given a single input.

Seq2Seq models, also known as encoder–decoder models, use both parts of the transformers architecture, i.e., they use both the encoder and the decoder of a transformers network. In this case, the attention layers of the encoder can access all words in the initial sentence, while the attention layers of the decoder can only access words that are positioned before a specific word in the input. These models are commonly used for natural language processing tasks that involve understanding input sequences and generating output sequences, often of different lengths and structures. Common use cases for encoder–decoder models include text translation and summarization. In addition, these models have shown flexibility in their responses because they do not rely on a sequence classification layer, as encoder models do for classification, but instead rely on the decoding mechanism to allow for specific output definition.

We based our approach on the REBEL framework [25], which uses mBART to extract entities and relationships from text. In this case, we used different seq2seq models such as mBART [26] and mT5 [27] as a base model and fine-tuned them to make the decoder generate the appropriate response, which in this case is the MET of the text and the sentiments of the respective targets.

On the one hand, mBART is an encoder–decoder model pretrained on a large monolingual corpus in multiple languages by using the BART goal. mBART is one of the first methods to pretrain a full sequence-to-sequence model by denoising complete texts in multiple languages, while other models focus only on the encoder, decoder, or reconstruction of parts of the text. The mT5 model, on the other hand, is based on the T5 (Text-to-Text Transfer Transformer) and has been pretrained on a new CommonCrawl-based dataset covering 101 languages.

In order to extract the MET and the sentiments for each target within a single sequence, we expressed the output of the model as a sequence of tokens so that we could easily retrieve each feature and minimize the number of tokens to generate, making the decoding process more efficient. To accomplish this, we introduced a number of new tokens into the seq2seq model that act as markers. The token <absa> marks the beginning of MET. The token <target> marks the end of MET and the start of the sentiment for the MET entity. The token <companies> marks the end of the sentiment for the MET entity and the beginning of the sentiment for the companies’ entity. On the other hand, the token <consumers> marks the end of the sentiment for the companies entity and the beginning of the sentiment for the society entity.

The hyperparameters used to fine-tune the seq2seq models of both mBART and mT5 are as follows: a train batch size of 4, a weight decay of 0.01, a learning rate of 2 × 10⁻⁵, 6 epochs for mBART, and 24 epochs for mT5. An epoch-based evaluation approach was used, where the model is evaluated and adjusted after each epoch to select the best model from a given epoch.

Figure 5 shows three examples of MTSA with the mBART model. We can observe that the model is able to identify METs and assign corresponding sentiments to each target through an input text.

4. Results

In this section, we explore the capabilities and limitations of the different deep learning approaches that we evaluated for MTSA. To perform this, we used the standard metrics in NLP tasks, such as precision (P), recall (R), and the macroaverage F1 score. Since MTSA involves two goals, target detection and sentiment identification for different entities, we averaged the F1 scores (Mean F1) of the MET detection and sentiment for each target, giving them equal weight. This allows us to compare the overall performance of the different approaches.

4.1. Strategy A: MET Detection Using Token Classification and Sentiment Classification Using Multiclass Classification

The results of the evaluation of different pretrained models for MET detection and the multiclass sentiment classification of different entities are shown in

Table 4. Benchmark of the different LLMs and multiclass sentiment classification with test splits evaluated for MTSA. Mean F1 is the average of F1 _MET and F1 _Sentiment. F1 _Sentiment is the average of the sentiment of the MET, companies, and society.

LLM	Mean F1	F1 MET	F1 Sentiment
ALBETO	77.002	80.719	73.284 (67.936, 76.963, 74.954)
BERTIN	75.330	76.693	73.967 (69.932, 75.602, 76.368)
BETO	79.990	83.584	76.395 (72.048, 78.885, 78.253)
DBETO	76.635	80.176	73.094 (68.075, 76.482, 74.724)
MarIA	77.940	78.161	77.720 (73.837, 80.357, 78.965)
RoBERTuito	59.886	46.386	73.751 (70.376, 75.857, 75.019)
XLM	76.243	78.978	73.507 (69.924, 75.577, 75.020)

. Regarding the experiments conducted for MET detection, BERT-based models such as BETO, ALBETO, and DBETO achieved better results than models based on RoBERTa. In terms of the macro F1 score, BETO achieved the best result with 83.584%, and lightweight models such as ALBETO and DBETO obtained competent results with 80.719% and 80.176%, respectively, surpassing other more complex models trained on a larger corpus such as MarIA and BERTIN. Furthermore, we can observe that monolingual models for Spanish generally perform better than multilingual models such as XLM-RoBERTa, with the exception of RoBERTuito, which was pretrained on a less extensive corpus.

In this approach, we considered the SA problem for different entities as a multiclass problem, so we evaluated the fine-tuning of different pretrained models for each of the targets. In terms of performance, the MarIA model achieved the best results in sentiment classification for the three entities, with a macro F1 score of 73.837%, 80.357%, and 78.965%.

If we compare the overall performance, i.e., the Mean F1 score metric, we can see that BETO has the best result with 79.990%, although the SA models based on BETO have worse results than MarIA thanks to its performance in MET detection. MarIA has the second-best result with 77.940%, surpassing the multilingual model (XLM-RoBERTa). We can also observe that the lightweight models (ALBETO and DBETO) achieve competent results with 77.002% and 76.635%, respectively, outperforming more complex models such as BERTIN and RoBERTuito.

Table 5. Precision (P), recall (R), and F1 score (F1) for the MET using model with the best Mean F1 score (BETO).

Label	P	R	F1
Target (MET)	82.925	84.254	83.584

and

Table 6. Sentiment analysis classification report for each target in Strategy A. The selected model is BETO, which achieved the best average F1 score. For each target, the precision (P), recall (R), and F1 score (F1) are reported.

	Target			Companies			Society
Label	P	R	F1	P	R	F1	P	R	F1
Negative	83.932	81.468	82.682	77.624	73.368	75.436	77.674	80.482	79.053
Neutral	50.230	47.391	48.770	81.241	84.831	82.997	81.558	78.802	80.156
Positive	82.827	86.645	84.693	79.279	77.193	78.222	74.783	76.331	75.549

show a classification report for MET detection and SA using the model that achieved the best Mean F1 (BETO). In Table 5, we can see that BETO achieved an accuracy of 82.92% and a recall of 84.25% for MET detection. On the other hand, MarIA obtained the best F1 for calculating the sentiment of each target. It can be seen that all models, including the lightweight models ALBETO and DBETO, are very competitive. The only exception is RoBERTuito, which obtained limited results. The main reason for this can be related to the fact that this LLM is trained on general Spanish tweets. However, the knowledge of a complex domain such as finance may make it difficult to rely on short tweets, which is why the model underperformed in the MET detection task with an F1 score of 59.89%.

Table 6 shows the classification report for the BETO model to extract the sentiment polarity of each target. It can be seen that the precision, recall, and F1 scores are similar. The most limited result is obtained with the neutral class for the MET with precision, recall, and F1 score results of 50.230%, 47.391%, and 48.770%, respectively. Looking at the confusion matrix of this model, we notice that the false predictions of the neutral class were made with the same proportion of positive and negative examples, highlighting the difficulty of determining which texts have a neutral polarity toward the MET.

4.2. Strategy B: MET Detection Using Token Classification and Sentiment Classification Using Multilabel Classification

This approach for the MTSA system considers the identification of METs as a token classification problem, just like the previous approach (see Section 3.2.1), and the SA of different targets as a multilabel problem. Therefore, there is no need for a separate analysis model for each entity. A multilabel model that includes sentiments for all entities is sufficient since the classes are not mutually exclusive. In

Table 7. Benchmark of the different LLMs and multilabel sentiment classification with test splits evaluated for MTSA.

LLM	Mean F1	MET			Multilabel Sentiment
		P	R	F1	P	R	F1
ALBETO	78.336	80.082	81.367	80.719	74.675	72.078	73.194
BETO	80.022	82.925	84.254	83.584	77.685	75.398	76.461
BERTIN	76.957	77.502	75.901	76.693	76.683	74.700	75.523
DBETO	76.108	79.729	80.628	80.176	74.384	71.395	72.737
MARIA	76.457	78.854	77.481	78.161	79.668	77.510	78.510
RoBERTuito	60.955	65.310	35.965	46.386	76.683	74.700	75.523
XLM	72.221	78.789	78.978	70.351	75.542	72.994	74.090

, we can see that the scores obtained from different pretrained models for MET detection in the first approach (see Section 3.2.1) were maintained, and the same models were evaluated for multilabel sentiment classification.

In terms of the macro F1 score, MarIA achieved the best results in multilabel classification with 78.510%. Lighter models such as ALBETO and DBETO achieved lower results with 73.194% and 72.737%, respectively. In this case, the XLM-RoBERTa model performed better than the light models (74.090% in macro F1 score) but lower than the RoBERTuito model, which is a Spanish model trained on a much smaller corpus.

In terms of overall performance, BETO achieved the best result with 80.023%, although it is not the best model for SA due to its performance in MET detection. The classification report for MET detection and sentiment analysis is shown in Table 5 and

Table 8. Sentiment analysis classification report for each target in Strategy B. The selected model is BETO, which achieved the best average F1 score. For each target, the precision (P), recall (R), and F1 score (F1) are reported.

	Target			Companies			Society
Label	P	R	F1	P	R	F1	P	R	F1
Negative	82.109	80.000	81.041	79.143	72.324	75.580	80.977	75.904	78.358
Neutral	49.315	46.957	48.107	80.441	86.009	83.132	79.592	83.871	81.675
Positive	85.096	84.420	84.757	81.505	76.023	78.669	80.984	73.077	76.827

. In Table 5, we can see that it achieved a precision of 82.925% and a recall of 84.254% for MET detection. Table 8 shows the precision, recall, and F1 score obtained for each sentiment polarity. We can see that the model obtained an F1 score greater than 72% in all the polarities of different targets, except for the neutral polarity of MET, where it obtained an F1 score of 48.107%.

4.3. Strategy C: Multitarget Sentiment Classification Model

Unlike other approaches that require at least two models for MTSA, this approach is based on using a seq2seq model as a base and adapting it to generate MET and the sentiments of different targets as a response sequence.

In this paper, we evaluated mBART and mT5.

Table 9. Benchmark of the different seq2seq models with test splits evaluated for MTSA. Task 2 is the average of the sentiment of the MET, companies, and society.

LLM	Mean F1	F1 MET	F1 Sentiment
mBART	80.300	85.053	75.548 (71.936, 78.368, 76.340)
mT5	78.289	82.246	74.333 (68.977, 77.146, 76.875)

shows the overall performance and the results obtained in each task of the two models. Regarding MET detection, mBART achieved the best result with a macro F1 score of 85.053%. It also achieved the best results in target and company SA with 71.936% and 78.368% macro F1 scores, respectively. The mT5 model achieved the best result in consumer SA with a macro F1 score of 76.875%. According to the Mean F1 score metric, the mBART model achieved the best overall performance with 80.3%.

Table 10. Precision (P), recall (R), and F1 score (F1) for the MET using seq2seq model with the best Mean F1 score (mBART).

Label	P	R	F1
Target (MET)	85.455	84.654	85.053

shows the MET detection classification report for the mBART model. It can be observed that it achieved a precision of 85.455% and a recall of 84.654%, surpassing the best MET detection model from the two previous approaches, such as BETO (see Table 5). Regarding the SA with the mBART model, in

Table 11. Sentiment analysis classification report for each target in Strategy C. The selected model is mBART, which achieved the best Mean F1 score (mBART). For each target, the precision (P), recall (R), and F1 score (F1) are reported.

	Target			Companies			Society
Label	P	R	F1	P	R	F1	P	R	F1
Negative	81.768	81.468	81.618	78.177	73.890	75.973	78.316	73.976	76.084
Neutral	52.315	49.130	50.673	81.571	84.094	82.814	79.173	82.335	80.723
Positive	82.481	84.579	83.517	76.316	76.316	76.316	72.537	71.894	72.214

, we can see the precision, recall, and F1 score obtained in different sentiment polarities. It is noticeable that the model obtained an F1 score greater than 72% in all polarities of different targets, except for the neutral polarity of MET, where it obtained an F1 score of 50.673%.

5. Discussion

This section compares the best model of each approach and explains some limitations and which of them is more suitable for a multitarget analysis of targeted sentiment analysis.

Table 12. Comparison table of the best model among the three approaches. Mean F1 refers to the overall performance, F1-MET indicates the macro F1 score of the MET detection model, and F1-Sentiment indicates the average F1 score of the SA models for the entities. No. Model indicates the number of models required to address the MTSA problem and Size indicates the size required for the models.

Approach	Model	Mean F1	F1-MET	F1-Sentiment	No. Model	Size (GB)
MET+Multiclass	BETO	79.990	83.584	76.395	4	1.76
MET+Multilabel	BETO	80.023	83.584	76.461	2	0.8
Multitarget	mBART	80.300	85.053	75.548	1	2.3

presents a comparison between the best model of each approach. This table provides a summary of the Mean F1, which refers to the overall performance; the F1-MET, which indicates the macro F1 score of the MET detection model; and the F1-Sentiment, which indicates the average F1 score of the SA models for the targets. It can be observed that the approach using a seq2seq model for MTSA gave the best result with an average F1 score of 80.30%, followed by the approach using a MET detection model and a multilabel sentiment classification model for entities, which achieved an average F1 score of 80.022%.

In terms of the MET identification performance, mBART achieved the best result with a macro F1 score of 85.053%, approximately 1.5% better than the BETO-based model. In terms of SA, the multitarget approach produced the worst result with a macro F1 score of 75.548%, while the multilabel classification approach based on BETO produced the best result with a macro F1 score of 76.461%.

Although mBART is the heaviest model, it requires less inference time because it does not have to load multiple models. However, because it is a seq2seq model, it is sometimes unable to extract the sentiment of certain targets, leaving the sentiment null. For example, in the test set, the model was unable to identify sentiments for 12 instances of certain targets out of a total of 1404 instances, resulting in an error rate of 0.85%. To address this issue, we replaced null sentiments with the most common sentiment from the training set. It is possible that this caused the model to perform about 1% worse than the BETO-based fine-tuned SA model. Therefore, we performed an error analysis to identify the weaknesses of the system that we used to identify the sentiment of different entities by using the mBART model. For each entity, we examined instances in the test set that were misclassified. The confusion matrices for this system for each entity are shown in Figure 6a–c.

Using the confusion matrix, we can see that the model is not biased toward making serious mistakes, such as identifying a positive sentiment as a negative sentiment and vice versa. We can see that the model mislabels 9.36% of the negative and 8.27% of the positive texts for the entity target (see Figure 6a) and confuses neutral texts with positive ones at 27.39%. Regarding the prediction of sentiments for the companies entity, the model predicts 18.28% of the negative texts and 17.25% of the neutral texts. Regarding the prediction of the sentiments for the society target, we can see that it behaves similarly to the prediction of the companies sentiments, mistaking 17.59% of the negative texts and 20.12% of the positive texts as neutral.

6. Conclusions

MTSA is an approach that enriches the analysis of textual data by providing a more granular understanding of sentiment as it extracts the subjective polarity of text from different multiple targets simultaneously in a given context. This approach is useful in the financial field because it allows for extracting opinions about different entities that affect the actor differently, including other companies and society. To this end, we evaluated three system design approaches: (i) a MET detection model based on token classification plus multiclass classification models for the SA of different entities; (ii) a MET detection model based on token classification plus a multilabel classification model for SA; and (iii) using seq2seq type models (mBART and mT5) as a base model and fine-tuning it to return a response sequence that includes the MET and the sentiments of different targets.

The multiclass and multilabel classification approaches are based on the tuning and evaluation of different pretrained models, both monolingual and multilingual and basic and lightweight. The BETO model is the one that obtained the best overall performance in the first two proposed architectures, with a Mean F1 of 79.90% and 80.02%, respectively. The third proposed architecture, based on fine-tuning an mBART model, obtained the best performance with an 80.301% Mean F1 and 85.053% MET detection, outperforming the other approaches. Based on the computational resources required by each approach and the performance obtained, we considered that fine-tuned mBART is the best approach to address the MTSA problem since it has only one model and has better scalability to more targets.

As a promising future line of research, we propose to focus on the interpretability of the results. In this sense, a limitation of our pipeline is that we only focus on the interpretability of the linguistic features applied to the targets, but this approach is model agnostic, and it does not consider the performance of the evaluated models. In this sense, we propose to evaluate other techniques such as LIME or SHAP to gain insight into the predictions obtained. Furthermore, we want to evaluate the effects of figurative language in economics and how it affects the polarity of the sentiment. To perform this, we will use some available datasets [28] to find clues in the texts that could determine the presence of satire, irony, or parody. In addition, we will investigate the effects of offensive content in finance and economics [29], as hostile environments are expected to be negatively correlated with SA in finance. Another improvement is the inclusion of cross-validation in our pipeline to avoid bias in the selection of the best model with a fixed validation split. The sentiment analysis model, trained on data from various official accounts, has no limitations in its application to text or news from different platforms. The uniformity of text features, syntax, and writing style across platforms supports its general use. However, it is important to note that the model was primarily trained on short texts from microblogging platforms such as X (formerly Twitter) and news headlines. Therefore, an interesting possibility for further evaluation is to test the model on text extracted from long articles and financial reports. This suggestion is identified as a potential area for future work.

The source code of the project is available in https://github.com/NLP-UMUTeam/Spanish-MTSA-2023.