Mental-Health: An NLP-Based System for Detecting Depression Levels through User Comments on Twitter (X)

Abstract

The early detection of depression in a person is of great help to medical specialists since it allows for better treatment of the condition. Social networks are a promising data source for identifying individuals who are at risk for this mental disease, facilitating timely intervention and thereby improving public health. In this frame of reference, we propose an NLP-based system called Mental-Health for detecting users’ depression levels through comments on X. Mental-Health is supported by a model comprising four stages: data extraction, preprocessing, emotion detection, and depression diagnosis. Using a natural language processing tool, the system correlates emotions detected in users’ posts on X with the symptoms of depression and provides specialists with the depression levels of the patients. By using Mental-Health, we described a case study involving real patients, and the evaluation process was carried out by comparing the results obtained using Mental-Health with those obtained through the application of the PHQ-9 questionnaire. The system identifies moderately severe and moderate depression levels with good precision and recall, allowing us to infer the model’s good performance and confirm that it is a promising option for mental health support.

1. Introduction

According to the World Health Organization (WHO), mental health is a state of well-being in which a person is aware of their capabilities, works productively, faces everyday stress, and can contribute to their community. Unfortunately, there are currently several factors, such as economic stress, urbanization, chronic stress, adverse experiences, and the excessive use of social media, that have contributed to the rise in mental health issues, affecting around 450 million people worldwide [1], with depression being one of the leading causes of disability in the world [2].

Emotions are an essential part of people. Emotions express the most intimate aspect of a person and reflect their behavior. Some works have attempted to determine the relationship between emotions and symptoms of depression [3,4,5]; for example, in [6], the authors assessed the emotional states that occur across the clinical disorders of depression, anxiety, and mixed anxiety and depression. The emotion states were evaluated using the Basic Emotions Scale, which includes a set of simple and complex emotions rationally derived from the basic emotions of sadness, anger, fear, disgust, and happiness. According to Stets [7], emotions and sentiments are defined by their interactions.

Natural language processing techniques can detect emotions from the texts that users post on social networks. Social networks provide a space where individuals can share their experiences, thoughts, moods, ideas, and feelings and discuss their daily struggles with mental health. Some studies that combine social networks and mental illnesses have already been carried out [8,9,10]. As a result, social networks hold great potential as a data source for identifying individuals at high risk for developing mental illness and can help in providing timely intervention and improving public health [11]. Integrating computational approaches to understand mental health status could have a significant impact, allowing new data to support clinical care, identify risky behaviors, provide timely interventions, assess developing conditions, or reach populations that are difficult to access through traditional clinical methods [12].

Based on this understanding, this work establishes the relationship between six of Plutchik’s eight emotions and the 20 symptoms related to depression proposed by Ahmed et al. [13] and Chan [14]. Regarding depression diagnosis, most works carry out a binary classification, that is, they only provide a diagnosis regarding whether a patient has depression. In this case, a multiclass classification is proposed that allows for the five levels of depression to be diagnosed (through the PHQ-9 questionnaire). It is essential to mention that this work represents an attempt to determine the level of depression from the comments made by patients through their social networks (in this case, through X). This process is carried out by combining NLP techniques, specifically emotion detection from text and the correlation between emotions and depression symptoms. We developed an NLP-based system called Mental-Health as proof of concept for this model. The model has four stages: the post stage, preprocessing stage, emotion detection stage, and forecasting stage. Mental-Health identifies moderately severe and moderate depression levels with good precision and recall, allowing us to infer the model’s good performance and confirm that it is a promising option for mental health support. It is expected that Mental-Health could be used by medical professionals focused on mental illness as a support tool for detecting their patients’ depression levels and aiming to improve their mental situation.

The rest of this paper is defined as follows: Section 2 discusses related work, and Section 3 presents the materials and methods. The results, including a case study and evaluation, are described in Section 4. The discussion is presented in Section 5. Finally, Section 6 describes the conclusions and future work.

2. Related Work

Several studies that have employed social media and artificial intelligence to detect and assess depression have recently automatically received increased attention [15]. This section describes relevant works grouped into two main areas according to their focus: models and methods and applications and frameworks.

2.1. Models and Methods for Depression Detection

Some authors have proposed models for detecting depression. For example, Li et al. [16] designed a computational model for detecting depression expressions in Chinese social media posts (Sina Weibo). Specifically, 15,879 posts were obtained and analyzed. Skaik and Inkpen [17] designed a model for detecting depression in X users that is representative of the people in Canada. Using classical machine learning techniques, they achieved a 0.961 F1 score using 10-fold cross-validation. Shah et al. [18] proposed a deep learning-based model to detect depression by analyzing users’ textual posts. This model was evaluated using a Reddit dataset. Nusrat et al. [19] proposed a model for predicting depression using tweets from the Twitter database based on lexicon labeling. Explainable Artificial Intelligence was used to provide reasoning by highlighting the parts of tweets that represent type of depression.

Biradar and Totad [20] proposed a model to classify comments on Twitter as having depression or without depression. Rao et al. [21] proposed a model named Multi-Gated LeakyReLU for identifying depression in social media. The model worked on a post level and a user level to obtain users’ emotional moods. Malviya et al. [22] implemented baseline models, such as the Transformers model, Linear Classifiers Support Vector Machines, and the Reddit dataset, to achieve a better method of detecting users with depression. Yang et al. [23] designed a model to identify the degree of depression. They determined the levels of depression in users, and the way the scores are defined and can be related to the data. Wang et al. [24] proposed a node-and-linkage-feature-based model to detect a user with depression. Several psychologists supported them. The model used psychological criteria to detect depression. Stephen and Prabu [25] proposed a method to determine Twitter users’ levels of depression by combining emotions and sentiment scores. Angskun et al. [26] introduced a model to obtain users’ depressive moods from Twitter comments by applying demographic characteristics and a sentiment analysis. Leis et al. [27] conducted a study to identify the behavioral patterns of users and the linguistic features of tweets in Spanish to determine who generates them, which could indicate signs of depression. Nadeem [28] employed a crowdsourced method to compile a list of Twitter users to being diagnosed with depression. Alsagri and Ykhlef [29] presented a study that detected a depressed user by applying machine learning techniques based on tweets and network behavior.

2.2. Applications and Frameworks for Depression Detection

Some authors have developed systems to identify depression, such as Arora and Arora [30], who created a system that analyzes tweets for anxiety and depression using Support Vector Regression (SVR) and the Multinomial Naive Bayes algorithm as a classifier. Narynov et al. [31] created a system named VKontakte that collects data from a social network, applying keywords to detect depressive moods. Lin et al. [32] designed a system dubbed SenseMood, a deep approach to detect the psychological states of users on social media. The system classifies users with or without depression through a neural network. Ricard et al. [33] created a web-based application for detecting depression based on Instagram profiles. Linguistic features were extracted from Instagram comments to build a predictive model.

Regarding frameworks for depression detection, Hussain et al. [8] proposed a framework to identify depression. They developed an application to detect depression-related markers in Facebook users using machine learning classification techniques with a data-driven approach. They identified the dominant features to identify individuals with or without depression. Martínez-Castaño et al. [34] proposed a modular platform to obtain social media data with crawlers to extract relevant content. They built a classifier that follows the thread of posts sent by users and predicts the occurrence of depressive signs. Safa et al. [35] created an approach to obtain and analyze user comments on X sustained by mentions and used a framework to predict depression symptoms.

Chiong et al. [36] created an approach using social media texts for depression detection. The experimental results indicate that the proposed approach can detect depression even when the texts do not contain specific keywords. Tlatelpa et al. [37] designed an approach that analyzes sentiments defined in the comments through a new text representation that obtains their polarity, specializing in the classification framework for profiles of users with depression. Yang et al. [38] proposed a machine learning-based framework to detect depression in users of social networks. The framework is based on syntactic and semantic features and pragmatic features. The framework outperforms similar methods.

Various classifications of works enable the identification of depression through the texts that users post on social networks. When comparing these works, we can note that the primary distinctions between them and our work are as follows: Firstly, we identified the symptoms (20) experienced by individuals suffering from depression, which has not been addressed in any of the works we have analyzed. This identification is based on research about the symptoms of people suffering from depression. Once these symptoms were identified, specialists in psychology provided support to determine the relative impact of each symptom on individuals’ behavior, assigning them a scale from highest to lowest depending on their influence. Secondly, we related the symptoms and the weights obtained with the values of the emotions (8) that were extracted from the comments written on the social network X through the MeaningCloud tool. (It should be noted that several of the works analyzed used the eight emotions and polarity that were extracted with NLP tools, and from them, we determined depressive states. In our case, the research goes further since we not only used emotions and polarity to assess depression, but the extracted emotions were correlated with the weights of the symptoms already determined.) Once the correlation of symptoms with emotions was made and had the corresponding value, we evaluated it based on another study carried out in the way that psychologists typically evaluate patients with the PHQ-9 questionnaire to determine degrees of depression. This questionnaire has five levels, so we carried out an analysis supported by medical specialists so that these levels were equal to those used in our study (so we can identify our model as multiclass). Finally, the levels were determined, and the values obtained in the application of our model are presented as a depression prognosis, which is what a specialist uses to assess a patient’s condition.

The model proposed in this work differs from the existing ones because it is not based on machine learning algorithms, but on a correlation between depression symptoms and emotions detected on social network posts through NLP techniques. Mental health specialists supported this correlation. This model is described in the section below.

3. Materials and Methods

3.1. Model Description

To carry out detection based on the existing literature related to mental disorders, as an alternative solution to the process for identifying levels of depression, Mental-Health is based on a four-stage model that would be able to identify a set of emotions related to symptoms and, with the application of NLP, obtain levels of depression detected in an individual based on what was written in their comments on social networks. The model begins with the study of the psychological emotions that occur in human beings, specifically based on the survey carried out by Plutchik [39]. This study has been widely used in the identification of emotions based on the particularity with which these emotions are defined and which has served to develop research that deals with the way a person feels [40,41,42,43]. In his work, Plutchik [44] defined eight primary emotions, joy, trust, fear, surprise, sadness, aversion, anger, and anticipation, and possible combinations.

Likewise, our work is mainly based on the studies carried out by Ahmed et al. [13] and Chan [14]. They defined 20 symptoms related to depression: sadness, pessimism, past failure, loss of pleasure, guilty feeling, punishment feeling, self-dislike, self-criticalness, suicidal thoughts, crying, agitation, loss of interest, indecisiveness, worthlessness, loss of energy, changes in sleeping pattern, irritability, changes in appetite, concentration difficulty, and tiredness. The relationship between the symptoms and the primary emotions already mentioned and obtained with a natural language processing tool are those that allow us to establish a parameter for identifying the levels of depression in the texts that users write on social networks and that are schematically represented in the model defined for such a task. To achieve this, interviews were conducted with a group of three experienced specialists in the psychological care of patients to obtain timely information on what symptoms a person with depression suffers from. Additionally, this group of specialists weighted each of the symptoms using the AHP method [45] to identify the most relevant symptoms of depression. This method makes decisions between alternatives based on the variables to be managed that have a hierarchical value, allowing us to transform qualitative criteria into quantitative ones. Due to its value in decision making, it has been used in various fields of research since it is the best alternative [46,47,48,49].

3.2. Correlation between Emotions and Symptoms

Within the research activities, a significant step was to establish the correlation of emotions with symptoms of depression. Next, the process of determining the degree of correlation between emotions and symptoms is described below.

• A group of three experts determined the weight that each of the symptoms has with respect to the others using the Analytical Hierarch Process (AHP), which is a method for making decisions that allows for priority scales to be generated based on expert judgments expressed through pairwise comparisons via a preference scale [50]. The result of this stage can be seen in Appendix A.
• Once the weighted matrix of depression symptoms was obtained, the next step was to obtain the normalized pairwise comparison matrix. This technique has been widely used to tackle the subjective and objective judgments about qualitative and/or quantitative criteria in multi-criteria decision making [51], where each data point obtained in every column in the first matrix is divided by the sum of the score of each symptom represented in that same matrix. To indicate how the weights assigned to each symptom were obtained concerning the other, they received a weight determined by psychologists. This weight could be 1, 3, 5, 7, and 9 per degree, with the highest value compared to the others, or there could be intermediate values such as 2, 4, or 6. In the first row, we can see the values of 1.00, 0.50, 2.00, etc., and in the first column, we can see the values of 1.00, 2.00, 0.50, etc. The values obtained at the intersection are the weights each symptom has divided according to the previous values (1, 2, 3, 4, 5, 7, and 9) concerning the others. The sum of the values obtained in each comparison (last row) is the value that will be used to obtain the normalized pairwise comparison matrix (see Appendix B).
• The weights of each of the symptoms of depression in relation to the others were obtained, and the resulting percentages were converted to integer values (see Appendix C).
• Three steps are utilized in multi-criteria decision making to obtain alternative relations: (1) Determine the relevant criteria and alternatives. (2) Attach numerical measures to the relative importance of the criteria and the impacts of the alternatives on these criteria. (3) Process the numerical values to determine the ranking score of each alternative [52]. In our study, to determine the correlation between emotions and symptoms of depression, each expert defined the corresponding symptom–emotion relationships. The responses that specialists agreed on were accepted, and the rest were discarded (see Appendix D). It should be clarified that joy and trust were not related to the symptoms of depression in any of the cases, so only six emotions at most were related to each of the symptoms.
• According to Lawshe [53], if the subject matter experts are generally perceived as true experts, then it is unlikely that there is a higher authority on the content validity of one test. As mentioned, three psychologists validated our model. This group of experts distributed 10 points between the emotions related to a symptom according to the importance they felt the emotion has concerning the symptom (see Appendix E).
• The values given by each expert were added to obtain a single value for each symptom–emotion relationship (see Appendix F).
• The final weights of the emotions regarding symptoms of depression are obtained in this part. It should be noted that the weight of the emotions regarding symptoms is used in decimal value so that when multiplied by the weight of the symptom, the sum of the final weights accumulates a total of 100 (see Appendix G).
• The weight of the emotions regarding symptoms of depression (values obtained in step 7) is multiplied by the weight of symptoms obtained in the first phase of this process (step 3) and then multiplied by the emotion value obtained by the MeaningCloud tool to obtain the final score that indicates the total value obtained from the model. Figure 1 shows the correlation between symptoms and emotions resulting from this process. As can be seen, there is a strong correlation between the emotion of aversion and the symptom of self-criticalness, between the emotion of sadness and the symptom of crying, as well as between the emotion of surprise and the symptom of agitation.

3.3. Model for Determining Levels of Depression

The model for determining levels of depression is outlined as follows: It starts with collecting comments from social media, and then establishing the correlation between emotions and identified symptoms, and finally calculating the overall level of depression.

The model and its stages are shown in Figure 2.

Next, each model stage is described in detail.

Data extraction: This stage of the model is where user comments are obtained through the social network. In the case of comments on X, it is important to indicate that the characteristics of the network must be followed in terms of the maximum number of characters, which is 280. It is also important to obtain the date and time of the comment to obtain the time lapse.

Preprocessing: Data preprocessing is a significant and essential stage whose primary goal is to obtain final datasets that can be considered correct and valid for further data mining algorithms [54]. This module performs NLP preprocessing tasks to make collected data suitable for analysis and modeling and to achieve good performance on emotion detection tasks, specifically hashtag extraction, hashtag segmentation, URL removal, mention removal, tokenization, lower-case conversion, stopword removal, and stemming.

Emotion detection: In this stage, MeaningCloud API uses NLP techniques to identify entities and concepts from the text [55]. The Emotion Recognition pack is based on Robert Plutchik’s Wheel of Emotions because of its clarity and potential [56]. Likewise, the emotions listed are obtained: sadness, happiness, anger, aversion, trust, fear, surprise, and anticipation.

Depression diagnosis: This stage is further divided into two parts: the correlation between emotions and symptoms of depression and the diagnosis of the level of depression. These two parts are described below.

Correlation between emotions and symptoms: This process is responsible for establishing the relationship between emotions and symptoms of depression (Section 3.2).

Diagnosing the level of depression: The total depression value (final score) shows the predefined value to indicate the value of the prognosis and is presented to the medical specialist. Finally, the values of symptoms and emotions collected by the NLP-based system are evaluated.

An NLP-based system called Mental-Health was developed as proof of concept of the proposed model. Next, we describe the Mental-Health architecture.

3.4. Mental-Health Architecture

According to Perry and Wolf [57] software architecture is a set of architectural elements with a particular form. They distinguish three different classes of architectural elements: processing elements, data elements, and connecting elements. In such a way, we designed software as proof of concept called Mental-Health. The software architecture created is shown below to indicate the elements that integrate it, which will allow us to provide an idea of how information is interconnected and processed to clearly illustrate the structure of our model applied to the software. This architecture shows how each layers interact, as shown in Figure 3.

Presentation layer. The architecture workflow begins with the login and user access to the Mental-Health platform. This layer corresponds to a set of graphic interfaces developed where the main Mental-Health functionalities are displayed, such as the medical specialist dashboard, patient profile, scheduled appointments, patient list, list of comments extracted from the patient, diagnosis, emotions and symptoms obtained, prognosis, and reporting.

Service layer. The service layer is divided into four modules. The first is the social media data extraction/collection module, which is responsible for extracting and gathering comments from registered users on social media. The NLP-based preprocessing module is responsible for carrying out NLP-based preprocessing tasks to effectively understand and analyze the data collected. The emotion detection module is responsible for detecting emotions from the text; this module is where the emotion–symptom correlation that was previously explained is applied. The diagnosis module is responsible for applying the depression detection model and determining the value that allows for the depression level to be determined.

Persistence layer. This layer is responsible for storing patient information, such as ID, name, the clinic where they are treated, email, mobile phone number, registration date, status, family background, and the information corresponding to the comments written in X that are used to make the diagnoses of depression organized in the corresponding dataset. It stores each of the social network analysis reports or depression prognoses and ensures they are always available so that they can be viewed by a medical specialist at any time.

The hardware and software tools used for the implementation of Mental-Health are shown in Appendix I.

4. Results

4.1. Case Study: Detecting Users’ Depression Levels

This case study aims to determine how efficient Mental-Health is in detecting users’ depression levels through social network posts. For this purpose, a group of 20 patients were involved; precisely, 13 were previously diagnosed as depressed, and 7 were non-depressed. The sample included people between the ages of 15 and 56 of both the male and female sexes. These patients have an X account and computing devices through which they post whenever they want. They were asked to publish one daily tweet describing their feelings over a period of two weeks. This time was selected since it is the period needed to apply a PHQ-9 questionnaire to detect depression. It is essential to mention that the people were asked to publish at least one tweet per day because, according to the specialists, it is very complex for patients with depression to write extensively about their condition.

It should be mentioned that the consent of all patients was requested through a signature authorizing the researchers to take their information to extract their tweets to identify their levels of depression through the Mental-Health platform and with the advice of the medical specialists who have been treating them. The following sections describe the most relevant aspects of this case study.

Patient List. This section shows the doctor’s patient list (see Figure 4). The information presented include the patients’ names, clinics, emails, mobile device numbers, registration dates, statuses, and actions to be carried out.

Patients’ posts. When the doctor selects a patient, Mental-Health shows the patient’s posts obtained from a social network, in this case, from X (see Figure 5).

Detection of patients’ depression levels. Appendix H presents the algorithm detailing the internal functioning of the depression detection model within the Mental-Health system to provide a more precise representation of its operation.

To start the depression screening,

Table 1. The values of the emotions obtained with MeaningCloud.

Emotion	API Value
Sadness	0.22
Aversion	0.15
Anger	0.26
Fear	1.0
Joy	0.26
Trust	0.11
Anticipation	0.41
Surprise	0.07

shows the emotion scores obtained from the patients’ posts using MeaningCloud. This tool assigns a value to each emotion detected on a scale from 0 to 1, where a value close to 0 denotes a weak emotion and a value close to 1 denotes a stronger emotion.

Next, to determine the depression scores of the patients, the model proposed in this work (

Table 2. Application of depression level detection model.

Symptom	Weight	Related Emotion	Weight of Emotion Regarding Symptom	Final Weight	Normalized Value
Crying	15	Sadness	0.7	10.5	5.770
		Anger	0.066	0.99
		Fear	0.2	3
		Anticipation	0.033	0.495
Self-criticalness	11	Sadness	0.266	2.926	3.203
		Aversion	0.333	3.663
		Anger	0.266	2.926
		Fear	0.1	1.1
		Anticipation	0.033	0.363
Punishment feeling	9	Sadness	0.333	2.997	2.928
		Anger	0.533	4.797
		Fear	0.1	0.9
		Anticipation	0.033	0.297
Loss of pleasure	7	Sadness	0.3	2.1	2.556
		Aversion	0.1	0.7
		Anger	0.266	1.862
		Fear	0.133	0.931
		Anticipation	0.2	1.4
Pessimism	6	Sadness	0.133	0.798	2.031
		Aversion	0.133	0.798
		Anger	0.466	2.796
		Fear	0.1	0.6
		Anticipation	0.166	0.996
Loss of interest	6	Sadness	0.4	2.4	2.382
		Anger	0.1	0.6
		Fear	0.133	0.798
		Anticipation	0.366	2.196
Agitation	5	Anger	0.166	0.83	1.254
		Fear	0.1	0.5
		Anticipation	0.166	0.83
		Surprise	0.566	2.83
Indecisiveness	5	Sadness	0.133	0.665	3.494
		Fear	0.533	2.665
		Anticipation	0.333	1.665
Sadness	4	Sadness	0.633	2.532	1.525
		Aversion	0.1	0.4
		Fear	0.2	0.8
		Anticipation	0.066	0.264
Suicidal thoughts	4	Sadness	0.5	2	1.560
		Aversion	0.066	0.264
		Anger	0.033	0.132
		Fear	0.166	0.664
		Anticipation	0.233	0.932
Irritability	4	Sadness	0.1	0.4	1.110
		Aversion	0.266	1.064
		Anger	0.4	1.6
		Fear	0.066	0.264
		Anticipation	0.1	0.4
		Surprise	0.066	0.264
Changes in appetite	4	Sadness	0.3	1.2	1.538
		Aversion	0.366	1.464
		Anger	0.066	0.264
		Fear	0.233	0.932
		Anticipation	0.033	0.132
Concentration difficulty	4	Sadness	0.3	1.2	1.658
		Aversion	0.133	0.532
		Anger	0.066	0.264
		Fear	0.2	0.8
		Anticipation	0.266	1.064
		Surprise	0.033	0.132
Tiredness	4	Sadness	0.433	0.9	1.567
		Aversion	0.166	0.9
		Anger	0.066	0.5
		Fear	0.2	0.5
		Anticipation	0.133	0.2
Past failure	3	Sadness	0.5	1.5	1.344
		Aversion	0.033	0.099
		Anger	0.1	0.3
		Fear	0.266	0.798
		Anticipation	0.1	0.3
Changes in sleep pattern	3	Sadness	0.3	0.9	1.267
		Aversion	0.233	0.699
		Anger	0.1	0.3
		Fear	0.266	0.798
		Anticipation	0.066	0.198
		Surprise	0.033	0.099
Self-dislike	2	Sadness	0.3	0.6	0.477
		Aversion	0.266	0.532
		Anger	0.3	0.6
		Anticipation	0.133	0.266
Worthlessness	2	Sadness	0.3	0.6	0.784
		Aversion	0.2	0.4
		Anger	0.1	0.2
		Fear	0.2	0.4
		Anticipation	0.3	0.6
Guilty feeling	1	Sadness	0.4	0.9	0.440
		Aversion	0.2	0.9
		Fear	0.3	0.5
		Anticipation	0.1	0.2
Loss of energy	1	Sadness	0.533	0.533	0.415
		Aversion	0.1	0.1
		Anger	0.033	0.033
		Fear	0.233	0.233
		Anticipation	0.1	0.1
Depression score					37.30

) and the emotion values obtained through the MeaningCloud tool (Table 1) were used, resulting in the scores presented in Table 2. Table 2 contains a new column named “Normalized value” that represents the sum of the multiplications of the final weight value by the emotion value. Furthermore, at the end of Table 2, the depression score is presented, which was obtained by adding all normalized values. Specifically, the formula used to obtain the depression score is shown below.

Depression score =
Normalized Value1(FinalWeight1*ApiValue1 + FinalWeight2*ApiValue2 + … + FinalWeightN*ApiValueN)
+
Normalized Value2(FinalWeight1*ApiValue1 + FinalWeight2*ApiValue2 + … + FinalWeightN*ApiValueN)
+
…
N

Once the depression score is obtained, Mental-Health determines a patient’s depression level according to the ranges presented in

Table 3. Depression levels.

Depression Level	Range
Severe	80–100
Moderately severe	60–79.99
Moderate	40–59.99
Mild	20–39.99
Minimal	0–19.99

. The three medical specialists established these ranges.

Also, Mental-Health provides a description of the symptoms detected, which contains the scores obtained using the model (refer to Figure 6).

Finally, Mental-Health also provides users with the emotions detected from the posts (see Figure 7).

4.2. Evaluation

To determine whether Mental-Health allows for patients’ depression levels to be correctly detected from social network posts, the results were compared with those obtained through the PHQ-9 questionnaire [58,59], which allowed us to identify the depression level that a person can have in a period of two weeks. This questionnaire consists of nine questions with four possible responses (0 to 3), resulting in a score between 0 and 27. Based on the obtained score, a patient is diagnosed with a depression level according to the following ranges:

• A score of 0–4: This indicates an absence or minimal presence of depressive symptoms. This suggests that a person is unlikely to have a significant depressive disorder.
• A score of 5–9: This indicates a mild presence of depressive symptoms. The person may experience some depressive symptoms, but this is not considered a significant level of depression.
• A score of 10–14: This indicates a moderate presence of depressive symptoms. The symptoms can affect the person’s daily life, and they may require interventions or treatment.
• A score of 15–19: This indicates a moderately severe presence of depressive symptoms. The person is likely experiencing significant difficulties due to depressive symptoms, and it may be appropriate to consider therapeutic intervention.
• A score of 20 or more: This indicates a severe presence of depressive symptoms. The person may be experiencing significant depression that can affect their daily functioning. Urgent therapeutic intervention is recommended.

As can be seen, both the PHQ-9 questionnaire and Mental-Health define five depression levels (see Figure 8).

First, all 20 patients were asked to answer the PHQ-9 questionnaire, and then the obtained results were compared with the depression levels provided by Mental-Health. The dataset obtained from the comments of the 20 patients is available at the following link: https://mental-health.com.mx/datasets/Dataset-patients.xlsx (accessed on 1 April 2024).

In any study utilizing information processing, it is crucial to specify how the data were extracted and which data points are significant. Presented below is the information that we consider important for dataset processing. The description and details of the dataset shown below can be found in Appendix J, Appendix K, Appendix L, Appendix M, Appendix N and Appendix O.

• Many comments range from 20 to 30 characters (Appendix J).
• Most comments are 8 to 10 words long (Appendix K).
• The most repeated words are “I feel” (En) = “siento” (Sp) and “Restless” (En) = “inquieto” (Sp), to mention a few (Appendix L).
• The most common bigrams are “I feel” (En) = “me siento” (Sp) and “I can’t” (En) = “no puedo” (Sp) (Appendix M).
• The most common trigrams are “I do not feel “(En) = “no me siento” (Sp) and “I feel very” (En) = “me siento muy” (Sp) (Appendix N).
• A word cloud of the dataset was created (Appendix O).

Concerning the presented dataset, it is necessary to indicate that the three medical specialists who helped us define the depression detection model through texts provided us with information about their patients from the corresponding ethics collection. In addition, all of the patients who provided information consented to our use of the data, signed a data use approval document, and wrote their comments to process them and obtain valuable data regarding the determination of depression levels. These specialists provided us with information on 100 patients. However, only 20 of these patients were included in the dataset because only those 20 met the requirement of writing at least one comment per day during the 14-day evaluation period. The rest of the patients wrote sporadically, and to validate their comments with the PHQ-9 questionnaire, they needed to write for 14 consecutive days. Once the users wrote the texts on social networks with the values obtained using Mental-Health, we create the confusion matrix to obtain the efficiency of the application of our system. It should be noted that most of the patients had a history of depression, which is why they did not write much. Furthermore, comments were only collected at the express request of the medical specialists.

Many clinical tests are used to indicate a disease. The tests correctly identify patients with or without a disease [60]. In this work, the confusion matrix [61] was used to determine the classification performance of the proposed model. The diagnosis provided by Mental-Health was compared with the diagnosis provided by the specialists.

Table 4. Confusion matrix and metrics.

	PHQ-9 Questionnaire Depression Levels					Metrics
Mental-Health Depression Levels	Severe	Moderately Severe	Moderate	Mild	Minimal	Precision	Recall
Severe	0	0	0	0	0	0	0
Moderately severe	0	3	0	0	0	1.0	1.0
Moderate	0	0	8	0	0	1.0	1.0
Mild	0	0	0	1	1	1.0	0.5
Minimal	0	0	2	1	4	0.8	0.57

presents the confusion matrix obtained using Mental-Health; furthermore, the statistical values are presented.

We can infer several findings about the classification performance of the model for different depression levels as assessed using the PHQ-9 questionnaire:

• Severe Depression: The model did not identify cases of severe depression because there were no people who had comments that indicated a high level of depression.
• Moderate Depression: The model shows excellent precision and recall for moderate depression. This indicates that the model correctly identifies all cases of moderate depression, and all predictions made by the model for moderate depression are accurate.
• Mild Depression: The model has a precision of 1.0. Since only two cases were presented, its recall is 0.5; at this point, the score of the incorrect case, “minimal”, was very close to “mild” (the correct one), which was due to the similarity in the comments.
• Minimal Depression: The model’s precision for minimal depression is 0.8, meaning that 80% of the cases predicted as minimal depression are correct. Its recall is 0.57, indicating that it correctly identifies 57% of the cases of minimal depression. In addition to this point, we can say that in some cases, very negative or very positive comments on the emotions with the highest values can affect the obtained results.

Overall, the model performs well in identifying moderately severe and moderate depression levels with good precision and recall.

Given the above data, Mental-Health is a good tool for determining depression levels since the patients were diagnosed with depression by the specialists. At the same time, some of the cases did not have a previous diagnosis of depression. Therefore, the first parameter that indicates whether they have any depressive traits is a good indicator. Secondly, the average falls within the appropriate range as it matches the values defined in Mental-Health concerning the values determined using the PHQ-9 questionnaire.

5. Discussion

Mental illnesses have become a global health problem because the people who suffer from them present characteristics such as a loss of interest in work, fun, and family relationships. Among these mental illnesses, the one that most affects the population is depression, because in extreme cases of this illness, people die by suicide. The coronavirus 2019 (COVID-19) pandemic contributed to increased suicidal behaviors [62]. This fact marked a milestone in the magnification of mental illnesses such as depression due to its explosive growth. Strategies have been defined to identify depression in its earliest stage to provide medical treatment that can prevent people’s symptoms from increasing and help improve their moods with the aim of avoiding suicide.

The detection of depression was first carried out through questionnaires that specialists in mental illnesses applied to patients, such as the PHQ-9 questionnaire or the Hamilton questionnaire [63]. However, with the advent of new information and communication technologies, the detection of depression can be carried out through the texts that users write on their social networks. In this work, an NLP-based system for identifying levels of depression through social network posts was presented. The development of Mental-Health involved three specialists in psychology and psychiatry who helped establish the appropriate values for the symptoms and emotions found. In future work, it is planned to adapt Mental-Health so that it helps to identify diseases related to depression, such as anxiety.

6. Limitations

In this work, the proposed model as well as Mental-Health have the following limitations:

• Due to the characteristics of the information obtained by medical specialists which the evaluated patients had to provide, the dataset presented 20 patients with their corresponding comments made each day.
• The model shows some limitations in identifying cases of mild and minimal depression. Further improvements might be necessary to enhance its performance for these categories.
• To determine effectiveness in detecting depression, a comparison must be made with the ranges used by the PHQ-9 questionnaire.
• Mental-Health does not identify the intentions of certain hidden phrases in texts, such as irony, sarcasm, or figurative language.
• Mental-Health does not identify new forms of textual expression, such as emoticons.
• Mental-Health is only applicable to the Spanish and English languages.

7. Conclusions

Great efforts have been made to detect the initial symptoms of depression and to be able to influence the earliest stages of this disease to help reduce the number of people who suffer from it. Even though different types of works have been conducted on matters that are related to the topic in this study, like models, methods, approaches, systems, studies, web applications, frameworks, classifiers, analyses, platforms, and algorithms, we have not found any study that focuses on the detection of the symptoms that a person can present and on the relationship that exists with the emotions detected through the analysis of feelings using comments that users write on social networks. Overall, the proposed model offers a systematic and data-driven approach to detecting levels of depression based on user-generated content on social networks. By leveraging psychological research, NLP techniques, and expert judgment, it provides a valuable tool for identifying individuals needing mental health support. This allows us to feel motivated by being able to contribute to the health sector to improve people’s states and, based on the work carried out, focus on mental illnesses related to depression, such as anxiety, which allows us to extend our model to become the basis for the detection of other mental illnesses.