This section presents the experimental results of the proposed multimodal SBB system. The section is divided into five subsections where the first two describe the dataset used in the experiments and the experimental setup. Next, person identification results are analyzed for each SBB trait. Finally, the results of the proposed multimodal SBB system are discussed, followed by a comparison with the state-of-the-art.
4.3. Results of Individual SBB Traits
In this section, an analysis of the individual Social Behavioral Biometrics (SBB) traits’ performance is conducted. According to the methods described in [
9,
10,
28], the original retweet network, reply network, URL network, hashtag network, temporal profile, and linguistic profile for all available users are constructed. The results of person identification by standalone performance of the SBB characteristics are shown in
Table 1. In the context of existing SBB methodologies, this analysis provides insightful information on the efficacy and potential of human micro-expression SBB trait to enhance multimodal SBB systems’ overall person identification performance if integrated.
The performance of the temporal profile as a social behavioral biometric is found to be the weakest, achieving a mere 9.34% accuracy in identifying individuals. On the other hand, when considering network-based SBB characteristics, the reply network outperforms others with an accuracy of 60.74%. It is crucial to highlight that for all network-based SBB traits, except for the retweet network, the F1-scores are disappointingly low compared to human micro-expression. This indicates that the human micro-expression SBB significantly outperforms the network-based and temporal SBB traits across all assessment measures.
When assessing the novel SBB feature, human micro-expression, it consistently scores higher in precision, recall, and F1-score than its traditional SBB counterparts, except for the linguistic profile. The precision score of linguistic profile is as high as 64.36%, compared to 62.14% for human micro-expression. Additionally, the linguistic profile’s recall score is around 6% higher than that of human micro-expression. However, the accuracy of human micro-expression is more than 5% lower than that of the linguistic profile. A possible explanation for the high performance of linguistic features in person identification is the considerably larger feature space generated by the linguistic profile, presented in
Table 2, which consists of approximately 57,967 dimensions, due to the richness of vocabulary usage.
In contrast, the feature vector size of the human micro-expression SBB trait is only 15,000, nearly one-fourth the size of the linguistic profile’s feature space. Notably, even with a significantly smaller feature space, the accuracy, precision, recall, and F1-score of the proposed method are only slightly behind the linguistic profile. This not only emphasizes the preeminence of human micro-expression but also highlights the inherent potential of improving overall identification if integrated into a multimodal system.
Another important aspect is that due to the smaller dataset size, with only 200 tweets available for every user, the method of human micro-expression can only generate a limited number of emotion signals. This, in turn, limits the number of emotion signals to generate the samples for training, which significantly influences the learning process of the model. If the proposed method is trained on a larger dataset, it is expected to achieve better performance since the learning of the model would be much more efficient with a higher number of emotion signals being available. This suggests that the human micro-expression SBB trait would be even more efficient for person identification if trained on a larger dataset while still having a significantly smaller feature space than linguistic profile.
In addition to time efficiency, a compelling rank-wise accuracy comparison of the recently introduced human micro-expression and the linguistic profile, as depicted in
Table 3, further demonstrates the superior performance of the human micro-expression SBB trait, while the linguistic profile holds a slight edge in rank-1 accuracy with 67.62% versus the human micro-expressions’s 61.73%, the human micro-expression SBB trait begins to outperform the linguistic profile from rank-2 onwards. It achieves a higher rank-2 accuracy of 77.45%, compared to the linguistic profile’s 76.23%.
This performance advantage continues to escalate as it moves further down the ranks, with the human micro-expression consistently outpacing the linguistic profile in accuracy. By rank-5, it already demonstrates a 3.19% point lead with an accuracy of 93.24%, compared to 90.05% for the linguistic profile. At the higher ranks (6 to 10), the human micro-expression SBB trait not only maintains but also extends its lead, culminating in a perfect accuracy score of 100% at ranks 9 and 10. In contrast, the linguistic profile only reaches this perfect accuracy at rank-10. Despite a slightly slower start at rank-1, the human micro-expression SBB trait swiftly surpasses the linguistic profile in accuracy from rank-2 onwards, maintaining this superiority consistently across all subsequent ranks.
As depicted in
Figure 9, the human micro-expression shows an identification accuracy that is over 12% greater than that of the other network-based SBB traits, except Retweet Network. Temporal profile and hashtag network exhibit poor performance, with identification accuracies below 30%. URL network and reply network fall within a similar range, achieving accuracies between 30% and 50%. Retweet network, human micro-expression, and linguistic profile SBB traits demonstrate closer proximity in terms of rank-1 accuracy, ranging from 60% to 70%.
The in-depth analysis presented above reveals that the human micro-expression SBB trait excels in both accuracy and efficiency. Its performance surpasses the linguistic profile when it comes to rank-2 accuracy and higher, and it does so while utilizing a much smaller feature space. These findings imply that the human micro-expression SBB trait holds considerable potential for incorporation into a multimodal SBB system, improving the overall identification accuracy.
4.4. Performance Analysis of Proposed Multimodal SBB System
This section presents a detailed analysis of the results of the proposed multimodal SBB system. The performance of the proposed multimodal system with different sets of features is examined to determine whether the novel SBB trait, human micro-expression, can be integrated with other SBB traits to contribute to improved person identification accuracy. Subsequently, a comparative analysis of the proposed multimodal system with state-of-the-art multimodal systems is provided.
To determine the potential of human micro-expression to improve person identification accuracy when integrated with other SBB traits, experiments were designed and carried out in four steps. First, a baseline feature set consisting of network-based SBB traits, namely URL network, retweet network, hashtag network, and reply network, is considered to observe the performance of person identification. Next, the human micro-expression SBB trait is integrated with the baseline SBB traits, and the impact on person identification accuracy is examined. The integration of the human micro-expression biometric is carried out with different emotion signal lengths. Since the language-based SBB trait, linguistic profile, performs better as a standalone trait, it is also added to the baseline features to understand the difference in performance improvement. Finally, the human micro-expression SBB trait is integrated with all the state-of-the-art SBB traits except the temporal profile (as it has the lowest performance that does not affect the overall system) to prove the hypothesis that human micro-expression SBB characteristic can improve person identification accuracy in a multimodal SBB system architecture. The experimental results on the proposed multimodal SBB system are provided in
Table 4. The features of the URL network, retweet network, hashtag network, and reply network are termed baseline features. The integration of human micro-expression with baseline features is annotated as BH. The baseline features and the linguistic profile are called BL. Finally, the incorporation of human micro-expression with BL features, where the emotion signal length is 50 [
11], is termed BLH50.
From
Table 4, the person identification accuracy reaches 63.47% when only the baseline SBB traits (URL network, retweet network, hashtag network, and reply network) are considered. Compared to the highest standalone performance of baseline SBB traits, which is approximately 60.74% achieved by the retweet network, this represents a 2.73% improvement when these features are combined.
Upon integrating the human micro-expression with the baseline SBB traits, the overall person identification accuracy sees a noticeable improvement. With a signal length of 50, the improvement peaks at around 4%. With a signal length of 75, the improvement is at its lowest, less than one percent. However, the performance improves at a signal length of 25, and the maximum improvement is observed at a signal length of 50. An approximate 3% improvement in person identification accuracy is observed when the signal length is 25. The most significant improvement is found, however, in the precision scores. When transitioning from the baseline features to BH (integration of human micro-expression with baseline features), there is a 7% precision score improvement from 61% to 68%, given that the signal length is 50. Additionally, there is a 4% improvement in the F1-score.
Excluding the human micro-expression and integrating the linguistic profile in the feature set shows performance improvement since the standalone performance of linguistic profile for person identification is also high. In fact, higher than the human micro-expression. Integrating the linguistic profile with baseline SBB traits results in about 6% improvement. However, the performance improvement in precision score, 7%, is as same as BH. Both in recall and F1 score, the BL feature set improves the performance by 1%.
Finally, the human micro-expression SBB trait is integrated with both the linguistic features and the baseline features. Once the human micro-expression is added to the BL (baseline features and linguistic profile), the overall accuracy of person identification in the proposed multimodal SBB system is significantly enhanced to 73.87%. This represents a 10.4% improvement from the baseline SBB traits, up to a 6.53% improvement from BH, and finally up to a 3.97% improvement from BL with a signal length of 50. In BLH (integration of human micro-expression to BL), when the signal length is 25 and 75, a performance difference (1.23%) is observed. Interestingly, this difference is smaller than that of the BH with signal lengths of 25 and 75. One possible reason for this could be the presence of linguistic features, which are already high-performing standalone features, whereas in BH there are no linguistic features involved. The model’s performance may be influenced by the inherent ambiguity and multifaceted nature of human language, potentially leading to uncertainties when deciphering intricate textual nuances or contexts. The precision score no longer improves beyond 70% when signal lengths are 25 and 50, respectively. However, the recall score reaches its highest at 74%. The overall balance score of precision and recall is 72% as measured by the F1-score. Overall, with signal lengths of 25, 50, and 75, the BLH feature set identifies individuals with 71.56%, 73.87%, and 70.33% accuracy, respectively. Therefore, the BLH50 feature set, which incorporates the baseline features, linguistic profile, and the human micro-expression with a signal length of 50, is found to be the most optimum choice for maximizing person identification performance in the proposed multimodal SBB system.
A comprehensive understanding of the performance in rank-level person identification with these optimum feature-set settings is further analyzed through the cumulative matching characteristic (CMC) curve depicted in
Figure 10. It is observed that although the person identification accuracy starts at 63.47% and 67.34% with baseline and BH feature sets, respectively, the identification rate becomes significantly higher over different ranks when the human micro-expression trait is incorporated with the baseline SBB traits. With baseline features, the person identification rate increases approximately by 6.28% at rank 2 from 63.47% to 69.77%, whereas with integrating human micro-expression, the identification rate at rank 2 has about 13% increase from rank 1. Interestingly, when the human micro-expression SBB trait is integrated, the identification accuracy becomes saturated at rank 9. On the other hand, the person identification accuracy does not become saturated within rank 10; the maximum identification accuracy at rank 10 with the baseline features is approximately 84%.
When linguistic profile is integrated with baseline features, it does not outperform the human micro-expression integrated with baseline features at rank 2. The identification improvement with BL features at rank 2 is about 12%, whereas it is about 13% with BH features. However, the identification rate with the BL feature in the subsequent ranks becomes better than with BH, as the BL reaches 100% accuracy at rank 7. Now, integrating the human micro-expression on top of BL features produces significantly better results at different ranks.
Within the first four ranks, the BLH50 feature set can accurately identify individuals 94.70% of the time, while it is 91.44% with the BL feature set. Within ranks 5 through 7, the proposed BLH50 feature set produces an accuracy score of 97.26% to 100%. As demonstrated in
Figure 10, the BLH50 feature set improves the person identification rate at every rank from rank 1 to rank 10. For most feature sets, the biggest improvement in person identification takes place from rank 1 to rank 3. With BL and BH, the identification rate demonstrates a flat difference from ranks 3 through 7. On the other hand, the identification rate increases higher and then decreases lower with BLH50 from rank 3 to rank 6. Overall, the CMC curve demonstrates consecutively higher performance improvement at different ranks when the human micro-expression is incorporated. This indicates that the inclusion of human micro-expression enhances person identification across multiple ranks.
Through the extensive analysis conducted above, it is evident that incorporating human micro-expression with other SBB traits substantially enhances person identification accuracy. In each rank, the human micro-expression SBB trait demonstrates a marked improvement in person identification when compared to the majority of other SBB traits when integrated. This indicates that the integration of human micro-expression with other SBB traits, including the linguistic profile, can yield significant improvements in person identification accuracy, underlining the effectiveness and potential of human micro-expression as a valuable addition to the existing SBB traits in creating a more robust and accurate person identification system.
4.5. Performance Comparison with the State of the Art
This section offers a comparative performance analysis of the proposed multimodal SBB system with state-of-the-art multimodal SBB systems. To ensure a fair comparison, all state-of-the-art multimodal SBB approaches are re-implemented following the same experimental setup used for the proposed system. The multimodal approach in [
9] combines the baseline features with the temporal profile. The multimodal SBB systems presented in [
44] incorporate linguistic profile and stylometry features using weighted sum rule techniques, in addition to the baseline features, to enhance overall performance. Since the state-of-the-art multi-modal systems utilize score-level fusion, the proposed method has also experimented with score-level fusion alongside rank-level fusion. The comparative performance differentials yielded by these two fusion techniques are illustrated in
Table 5.
This comparison reveals a negligible variance in overall performance regardless of the fusion technique implemented. The accuracy of the proposed system is 73.84% and 73.87% when score-level and rank-level fusion are utilized, respectively. Therefore, the data suggest that the enhanced performance of the method is not derived primarily from the fusion technique chosen. Instead, it is largely attributed to the prominent role of human micro-expression SBB trait, which illustrates its considerable feature importance within the models.
Now,
Table 6 demonstrates the comparative results of the existing SBB systems and the proposed multimodal SBB system, showcasing the effectiveness of the proposed approach in relation to other methods.
It is noticeable that the integration of the temporal profile with the baseline features yields the same experimental results as the baseline features presented in
Table 4. Given the low standalone person identification accuracy (below 10%) of the temporal profile based on the experimental setup used in this study, the weight of the temporal profile is minimal, which does not contribute to the overall performance improvement. The multimodal SBB system in [
44,
50] achieves an accuracy of 70.59%, while the precision score is relatively low at 66%, the recall score is 60% with an F1 score of 68%. This represents a 7% improvement over the results from [
9]. The updated SBB system incorporating stylometry features in written samples [
44] further improves identification accuracy by around 1%. Interestingly, precision and F1 scores increase by 4% and 2% respectively.
With the proposed multimodal system that integrates the human micro-expression SBB trait alongside the baseline and linguistic features, there is an approximate increase of 3% in person identification accuracy. The F1 score also improves to 72%, representing a 4% increase. The lowest performing multimodal SBB approach is the incorporation of baseline and temporal profile, with an accuracy score of 63.47%. The second-highest performing multimodal SBB system is [
44,
50], achieving an accuracy score of 71.38%. The proposed multimodal SBB system, which integrates the human micro-expression SBB trait with baseline and linguistic features and utilizes the rank-level weighted Borda count technique, outperforms the state-of-the-art multimodal SBB systems with an impressive accuracy score of 73.87%. Although the obtained result is not perfect, it is crucial to note that behavioral-based biometrics tend to be less precise than physiological biometrics [
4,
51]. Given that this approach is centered on human communication, the results show promise. Importantly, the state-of-the-art SBB traits and mulitmodal systems originally required at least 200 tweets to identify a person. On the other hand, the proposed multimodal system integrating human micro-expression requires only 100 tweets to identify a person, which is a significantly lower data requirement than the state-of-the-art. This highlights the significant potential and effectiveness of the novel human micro-expression SBB trait in enhancing person identification performance when integrated with other SBB traits.