*4.2. Experimental Design*

The first evaluation consisted of verifying the ability of the context-aware sociability patterns to model and explain social behaviors, i.e., we verified whether sociability patterns could explain and predict stable social routine (i.e., individuals repeating their social behaviors over the days) and less able to explain unstable social routine. For this, we used Pearson correlation coefficient [41] to assess the association between the ability of sociability patterns to explain social routine and stability of the individual's social routine. This coefficient measures linear correlation between two variables, which assumes values between −1 (perfect negative correlation) and 1 (perfect positive correlation). Therefore, higher levels of positive associations indicate that the proposed solution recognizes consistent sociability patterns and capable of modeling monitored people's social behaviors.

In the end, we evaluated the proposed solution to detect social behavior changes. For this, we joined data of two users who had different social routine, hence enabling to identify the moment when the change occurs. We verified whether the proposed solution can accurately detect this change and its ability to adapt to the new sociability pattern.

### *4.3. Ability to Model Social Routines*

This experiment aimed to assess the ability of the sociability patterns to model social routine. Specifically, a sociability pattern should explain the social behavior of the monitored individual, which should be correlated with his/her social habit, because the more stable the individual's social behavior, the greater the pattern's ability to predict it. We analyzed association between the identified sociability patterns' prediction level and the social routine's stability. For this reason, we used Pearson correlation coefficient for

quantifying this association. By using this correlation coefficient, patterns' ability to explain and predict social routine of the monitored individuals were identified.

We defined that a social observation consisted of a data window of one week (i.e., seven days). For example, for the *MONDAY* context, an observation consisted of data from 1 day, since a week has only 1 day with that context. Therefore, the first step of this experiment was to define the number of observations necessary to extract patterns consistent with the monitored individual's social behavior. For this, we identified sociability patterns considering each day of the week as a CA (i.e., *MONDAY*, *TUESDAY*, *WEDNES-DAY*, *THURSDAY*, *FRIDAY*, *SATURDAY*, and *SUNDAY*), and we checked the predictive performance of sociability patterns when using different numbers of social observations to design them. We defined prediction performance as the ability of sociability patterns to explain and predict individuals' social routines.

Figure 9 presents an example of scenario for assessing the prediction performance of sociability patterns using two observations. We performed this evaluation using one, two, three, and four observations to project sociability patterns, so allowing us to compare the predictive performance of each configuration. Each execution consisted of the following steps: (i) recognizing the sociability pattern with the number of specified observations (i.e., one, two, three, or four); (ii) measuring the Jaccard similarity index between the pattern extracted with the next social observations; and (iii) identifying a new social pattern from the evaluated observations to represent a new reference pattern.

**Figure 9.** Evaluation of the prediction performance of sociability patterns using two observations.

We performed the experiment for all users considering CAs specified previously (e.g., *MONDAY*, *TUESDAY*, and *SUNDAY*), so making it possible to recognize the predictive performance (i.e., the similarity between sociability patterns and observations) of sociability patterns for these CAs. We calculated the average prediction performance to identify the number of observations required to extract sociability patterns to explain social routine.

Figure 10 shows the average predictive performance of the sociability patterns identified using one, two, three, and four observations. From results, we identified that extracting the sociability pattern using only one observation resulted in poor predictive performance compared to other configurations. Extraction of social patterns with two, three, or four observations showed similar predictions. Therefore, we concluded that the most appropriate approach was to use two observations to extract sociability patterns since we could identify them in less time with predictive performance similar to other configurations.

**Figure 10.** Prediction performance of sociability patterns.

After quantifying prediction levels of the extracted sociability patterns, we measured stability of the social routine of the individuals who participated in this study. We calculated average similarity between one day and the subsequent day (Figure 11), i.e., similarity of the individual's social behavior between consecutive days. CAs considered each day of the week, similar to the previous experiment. In the end, we calculated average stability of the individuals' social routine, so allowing us to correlate this variable with prediction levels of sociability patterns.

**Figure 11.** Configuration used to evaluate stability of individuals' social routines.

We performed a stability analysis of the individuals' social routines to recognize essential information to understand their social behaviors. Figure 12 shows the stability of the individuals' social routines (i.e., the similarity of social behaviors between days), so making it possible to identify that most users had social habits with stability below 40%. However, some users had more stable routines, such as *u04* and *u27*. From this analysis, we expect that sociability patterns could explain and predict more consistently social behaviors of the more stable users and that present lower levels of predictions when applied to individuals with more unstable social routines.

So far, we quantified the prediction performance of the extracted sociability patterns and the stability of the individuals' social routine. Therefore, we can perform association analysis between these two variables using Pearson correlation coefficient. Figure 13 shows this association, in which y-axis represents the average of prediction performance of the social patterns, and x-axis represents the average of the individuals' social routine stability. When analyzing Figure 13, we can identify a clear correlation between these two variables, so representing a linear relationship. Pearson correlation coefficient resulted in +0.86, which represents a strong positive association between these variables.

**Figure 12.** Stability of the individuals' social routines.

**Figure 13.** Correlation between prediction performance of sociability pattern and social routine stability.

Figure 14 shows the relationship between prediction performance of the sociability patterns and stability of the individuals' social routines for each CA, so making possible to identify a linear relationship between these variables. Figure 15 shows the result of applying the Pearson correlation coefficient between these variables for each specified CA, so indicating that association levels were higher than 0.7, which represents strong positive correlations. When analyzing these results, we can recognize that prediction performance of the extracted sociability patterns remains related to stability of the social routine in all evaluated CAs.

**Figure 14.** Correlation between prediction performance of sociability pattern and social routine for each Context Attribute (CA).

**Figure 15.** Average correlation level for each CA.

From this experiment, we can recognize that sociability patterns of the proposed solution satisfactorily model social routines of individuals. Therefore, sociability patterns can be used to reliably understand and predict social behavior since they have strong correlations with users' social habits.

### *4.4. Evaluation of Social Behavior Change Detection*

This experiment aimed to verify the performance of the social behavior change detection solution. In particular, we evaluate its ability to detect social behavior changes and adaption to the new social behavior of the monitored individual. Therefore, we expect that the solution to accurately identify observations that represent abnormal social behaviors and social routine changes.

Firstly, we defined a similarity threshold that represents a change in social behavior. For this reason, we calculated mean and standard deviation of the stability of social routines for all users, so defining this threshold as [*μ* + *<sup>σ</sup>*]. Similar to the configuration of the previous experiment (Figure 11), we calculated mean and standard deviation of the similarity of social behaviors of individuals between consecutive days considering each day of the week as CAs. In the end, we identified that users had an average of 35.4% of stability in their social routines and standard deviation of 10.7%. Therefore, we specified the threshold for social behavior changes at 46.1%.

We combined data from two users with significantly different social routines to simulate a change in social behavior. For this, we selected users who had more stable social habits. We identified that users u27 and u04 presented social routines with satisfactory stability for this experiment by considering results from the previous experiment.

Figure 16 shows social routines of the individuals u27 and u04, in which each cell contains the number of social events in a given time slot of 30 min (t = 24 0.5 ). From this visualization, we can recognize an evident change in social behavior at the limit that separates the two users' data. In this scenario, we expect that the proposed solution would detect abnormal behaviors when starting to process data of the user u04 and recognize the social routine change. Additionally, the solution adapt to the new pattern, then providing a new sociability pattern capable of explaining and predicting the new social routine.

**Figure 16.** Merge of the social routines of the users u27 and u04.

Social event streams for the proposed solution were created from data of the selected users. After processing data stream, our solution detected social behavior changes presented in Table 2. These results demonstrate that it identified abnormal behaviors and social routine changes precisely. Specifically, our solution identified a high similarity between social patterns and observations (i.e., similarity > 46.1%) while processing user data u27,

so not representing behavioral changes. From the first observation of the user u04, our solution started to detect abnormal social behaviors, so recognizing the social routine change by extracting the first pattern using data from this user (i.e., 5th pattern). This new reference pattern remained consistent with the next processed social observations, i.e., the solution can efficiently adapt to data stream changes.

**Table 2.** The flow of social behavior change detection. Red lines represent the identification of abnormal behaviors and social routine change.


### *4.5. Discussion and Limitations*

This section presented an experimental evaluation of the proposed solution that significantly extended experimental evaluations reported in [20]. We performed an indepth evaluation of the components designed to detect context-aware sociability patterns and behavior changes. From these experiments, we found that patterns recognized by the solution can model and predict social routines of individuals considering context information. Moreover, it can detect significant changes in social habits consistently.

In [20], we compared the similarity between social intervals detected by the proposed solution with those recognized by a batch processing algorithm. From the evaluation in [20], we recognized that the compared solutions identified sociability patterns with 86.33% similarity. We also analyzed sociability patterns based on CAs to investigate their contribution to understand social habits. We identified that context-based recognition provides insights into sociability patterns hidden in the context-free analysis. Therefore, the detection of sociability patterns based on CAs improves the understanding of social habits because it enables to distinguish abnormal behaviors from expected changes due to the context.

The aims of our experiments differ from the evaluations performed by the related works, hence it is a challenge to compare their results using specific metrics. For example, studies that design machine learning models to classify and predict mental states use metrics such as accuracy, precision, recall, whereas in our unsupervised sociability pattern learning approach, we use methods to assess the ability to model social routine and detect behavior changes. The works [22–24] also assess sociability patterns, but they differ from our experiments. Harari et al. [23] computed test-retest correlations between the observed behavior durations for adjacent weeks. Barnett et al. [22] analyzed the rate of anomalies in

behavioral patterns based on a statistical test inspired by Filzmoser's approach to predict schizophrenic relapse. Bonilla et al. [24] analyzed the applying of the Poisson mixture model to obtain the intensity functions of all calls in which patients were involved.

Our experiments evaluated the ability of the proposed solution to identify contextaware sociability patterns capable of explaining social routine and detecting social behavior changes. From this analysis, we identified that the prediction performance of social patterns has a strong positive correlation with the stability of the social routine (i.e., Pearson correlation coefficient greater than +0.7) in all CAs considered, so enabling to recognize that the proposed solution detects patterns consistent with the social behaviors of the monitored individuals. The evaluation of the social behavior change detection solution analyzed results of the detection processes performed by the solution when processing data containing changes in social routines. This experiment recognizes that our proposed solution can accurately detect and report abnormal social behaviors and social routine changes. Therefore, this is a promising tool for monitoring mental health, since reports of behavioral changes may indicate the onset, presence, or development of mental disorders.

This study has some limitations. First, the algorithm requires to manually enter two parameters: *ϕ* and *θ*. Therefore, the recognized sociability pattern depends on the predefined values chosen empirically rather than automatically setting the best values based on processed data. Second, the experimental evaluation was based on only one type of social activity (i.e., conversations). Other sources of social interactions should be considered, such as interactions on mobile social networks and telephone call communications. Third, social routine change does not necessarily imply change in sociability, as individuals can change their routine and maintain the same sociability level. However, the solution can identify social routine changes (i.e., whether there is a change in sociability), so allowing mental health professionals to interpret and investigate changes in the user's social aspect. Finally, another aspect is the homogeneous essence of the study participants (i.e., university students). Our solution should be validated with a more heterogeneous population, especially mental health professionals and their patients.

### **5. Conclusions and Future Work**

This work presented an approach for monitoring mental health through awareness of the social situation. Specifically, we introduced a solution based on FPM and CEP concepts to recognize intervals of the day when a monitored individual habitually socializes for each contextual condition. We also presented the solution developed to recognize abnormal behaviors and routine changes. Additionally, we introduced specialized knowledge modeling through fuzzy logic to allow the solution to send notifications of behavioral changes considering the imprecision of this task. From the evaluation conducted, we demonstrate that the predictive performance of context-aware sociability patterns has a strong correlation with social routine stability and that the solution can detect social behavior changes. We conclude that our proposed solution can be integrated into mental health monitoring tools to objectively collect patterns and changes in social behaviors, then providing support to mental health professionals and contributing to the effectiveness of the treatment proposed for patients.

As future work, we plan to address some open issues. The first one is to detect sociability patterns previously specified by mental health professionals, so enabling to identify situations of interest. We would also like to create dashboards for sociability patterns in an appropriate way for professionals, so facilitating analysis of social behavior. Another task is to update the solution to add information about the sociability level (e.g., social interaction intensity) to extracted patterns and identified behavioral changes. We also intend to develop an approach capable of automatically defining the best values for the parameters of the algorithm. Moreover, we plan to extend our solution to detect patterns related to other behaviors, such as physical activity and mobility. Plans also include validating our solution with professionals and their patients.

**Author Contributions:** I.R.d.M. implemented the solution and wrote this manuscript. A.S.T. and I.R.d.M. conducted experimental evaluations. F.J.d.S.eS., L.R.C. and M.E. supervised the research. F.J.d.S.eS., L.R.C., A.S.T. and M.E. participated in the revision of the manuscript, so supporting the writing process. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Coordination for the Improvement of Higher Education Personnel—CAPES (grant 88887.200532/2018-00); INCT of the Future Internet for Smart Cities (CNPq 465446/ 2014-0, CAPES 88887.136422/2017-00, and FAPESP 14/50937-1 and 15/24485-9); National Council for Scientific and Technological Development—CNPq (311608/2017-5, 420907/2016- 5, 312324/2015-4); and the State of Maranhão Research Funding Agency (FAPEMA).

**Conflicts of Interest:** The authors declare no conflict of interest.
