Contextual Enrichment of Crowds from Mobile Phone Data through Multimodal Geo-Social Media Analysis

Abstract

The widespread use of mobile phones and social media platforms provides valuable information about users’ behavior and activities. Mobile phone data are rich on positional information, but lack semantic context. Conversely, geo-social media data reveal users’ opinions and activities, but are rather sparse in space and time. In the context of emergency management, both data types have been considered separately. To exploit their complementary nature and potential for emergency management, this paper introduces a novel methodology for improving situational awareness with the focus on urban events. For crowd detection, a spatial hot spot analysis of mobile phone data is used. The analysis of geo-social media data involves building spatio-temporal topic-sentiment clusters of posts. The results of the spatio-temporal contextual enrichment include unusual crowds associated with topics and sentiments derived from the analyzed geo-social media data. This methodology is demonstrated using the case study of the Vienna Pride. The results show how crowds change over time in terms of their location, size, topics discussed, and sentiments.

1. Introduction

In the field of emergency management, accurate information and near real-time situational awareness are crucial for crisis prevention and mitigation [1,2]. Past disasters, such as the 2010 Love Parade disaster in Duisburg [3], or attempted attacks, like the one during the 2023 Vienna Pride Parade [4], that have occurred during large-scale events underline the importance of accurate information about crowd dynamics and sentiments on specific issues, as well as the intentions of individuals.

In this context, the high coverage of mobile phones, especially in urban contexts, makes them a valuable source for information on human mobility [2,5]. At the same time, the widespread use of social media platforms, such as Facebook and Twitter, provides information about individuals’ opinions on a wide array of topics and reveals their activities and social connections. As a result, social media data have been used in several different applications such as event detection and diagnostics [6] and the crisis management of disasters [1].

Numerous studies have analyzed mobile phone data in the form of Call Detail Records (CDRs), mainly focusing on disasters triggered by natural hazards [2]. Disasters caused by anthropogenic hazards, such as riots or terrorist attacks, are strongly influenced by human actions, intentions, and decisions, resulting in unpredictable and rapidly changing situations. Gathering accurate information about such disasters is therefore a major challenge. Yet, their prevention and management is crucial for the safety of individuals worldwide. Social media data have the potential to address this challenge but are typically a source that is rather sparse in space and time, and therefore lack information on physical movements. Mobile phone data, on the other hand, provide valuable positional information of users, but lack contextual information that would explain the findings in the data. Consequently, the combination of these two data types may lead to more efficient approaches. However, there remains a lack of research in this area [2,7].

To address the shortcomings of both data types and exploit the potential of their complementary nature in the context of emergency management, we present a novel methodology to improve situational awareness with a focus on organized, large-scale urban events. This methodology could be applied in two different ways. On the one hand, it is suitable for a post-event analysis of major urban events, the results of which could be used to derive findings for possible prevention measures. On the other hand, if adequate data are available, near-real-time use during events is conceivable, where the results could provide a detailed insight into what is happening on site. The application is conceivable for urban events of different scales, e.g., music festivals, concerts, city festivals, or regular events such as demonstrations or football matches. In this paper, we address the following research questions:

RQ1: 

How can crowds that form during urban events be detected in aggregated mobile phone data?

RQ2: 

How can crowd information be contextually enriched with semantic and sentiment information from geo-social media data?

To answer these research questions, we propose a methodology consisting of three parts. The first one focuses on the detection of unusual crowds, i.e., crowds that form during urban events, through a spatial hot spot analysis of mobile phone data, previously normalized with respect to a baseline. It is assumed that the mobile phone data are aggregated uniformly in space and time, and that the number of visitors, defined in this paper as an estimate of the number of people with an active mobile device in a defined grid cell, is available for each grid cell and time step. The second part concerns the analysis of geo-social media posts, which is based on spatio-temporal topic-sentiment clustering [8]. The third part deals with the contextual enrichment of crowds with the results of the analyzed posts.

2. Related Works

When it comes to mobile phone data analysis, the main focus of past research was on mobile phone data in the form of CDRs, which contain information about calls and SMS messages. The features of the calls usually include the IDs of the participants and their location together with the time and duration of the call [2,5]. Dong et al. [9] present a methodology for the detection of crowd events from CDRs and the subsequent identification of unusual crowd events, which is carried out by comparing individuals’ trajectories and their historical mobility profiles. In Sun et al. [10], the space–time structure of urban dynamics is uncovered by applying Principal Component Analysis (PCA) on data collected by cellular networks. The used data are aggregated into grid cells and thus describe the population distribution of the area of interest (AOI). The results show that urban dynamics have a low intrinsic dimensionality, a dominance of periodic trends, and temporal stability. The researchers also identify three categories of common patterns, namely, regular patterns, unusual patterns, and insignificant patterns. By isolating the principal components that describe regular patterns, significant deviations from general trends can be identified. Gundogdu et al. [11] use spatially and temporally aggregated CDRs to identify and analyze emergency and non-emergency events such as protests, violent incidents, holidays, and major sport events. Based on the assumption that people make more calls when an extraordinary event takes place, anomalies are detected via a Markov-modulated Poisson process. Their results show that those events can be captured by anomalous calling patterns and that considering the records as a time series outperforms tracing movements of masses. Van Dijcke and Wright [12] use aggregated mobile phone data to identify anomalous device surges, i.e., rapidly developing crowds, basing their analysis on the George Floyd protests.

The contextual enrichment of locations or events through social media data involves extracting meaningful insights and information from geotagged posts, check-ins, and other location-based content shared on social media platforms. Among others, it is widely used in the field of road data enrichment where Rettore et al. [13] propose a spatio-temporal model that provides a description of traffic conditions based on geotagged Tweets. After segmenting the AOI, Tweets are spatially and temporally matched with individual segments, and routes are enriched based on the Tweet analysis of the segments they pass. In the context of emergency management of mass gatherings, Haghighi et al. [14] introduce an approach on providing real-time situational information about crowd behavior and mood. Their classification model identifies the main crowd type categories, which are then associated with a sentiment and a level of emergency and medical workload. Ngo et al. [15] also present a framework that includes a model representing different crowd types, but their approach is to identify them based on the emotion analysis of social media posts associated with the event. Duan et al. [16] use Sina Weibo check-in data to detect large crowds in Shanghai and perform a topic modeling to extract posts about the 2014 stampede. They also find an increase in negative posts right after the event. Marwen et al. [17] use geo-social media data to enrich remote sensing imagery on large-scale events, mainly focusing on natural and anthropogenic hazards. There is also a lot of research on the use of social media in the context of such events, involving a large number of studies from a communication research perspective (e.g., Soler I Martí et al. [18], Sinpeng [19]).

Even though mobile phone data and social media data have mainly been studied separately, researchers have also leveraged the content of both combined. For example, Cecaj and Mamei [20] present a method for discovering urban events in spatio-temporal datasets based on statistical anomaly detection. The experiments are run on CDRs data and Twitter posts. After spatial and temporal aggregation, events occurring in spatial segments of the AOI are identified by detecting outliers in the time series of calls made or posts shared. For the detection of events from several data sources, the union of outliers from the datasets is considered. To describe individual events, georeferenced Tweets from the area of the event and its neighborhood posted at the time of the event are used to create word clouds. In Wu et al. [21], the aim is to find the semantics of the location history of a mobile user, such as events attended or landmark information. These semantics are extracted from geo-social media data as a list of words that best describe the purpose of a user’s visit at a specific location and time. To achieve this goal, three relevance measures of words are tested: frequency-based, Gaussian mixture model, and Kernel density estimation (KDE). Their results show that KDE best captures the locality and relevance of words. For uncovering urban functions, Tu et al. [7] present an approach on inferring human activities based on positioning data from mobile phones and social media check-in data. After estimating the home and work places from the mobile phone positioning data, the remaining activities such as shopping or transportation are labeled with a hidden Markov model using the social media check-in data. Urban function and their dynamics are then inferred by aggregating the activities found. The results from Sobolev et al. [22] reinforce the joint use of mobile and social media data, since they find a high correlation between person count estimations from Twitter and mobile phone visitor counts for the 2017 Women’s March.

Especially with the backdrop of the General Data Protection Regulation (GDPR) in the European Union, approaches that require data linked to individuals can no longer be used due to privacy issues. Methods using aggregated data reduce these privacy concerns. So far, however, unusual events or anomalies have only been identified by looking at the temporal aspect within individual spatial segments of aggregated datasets. These approaches do not take into account the spatial interactions between segments. The contextual enrichment of locations has been carried out either by first filtering potentially relevant posts or by weighting them with a relevance measure based on their distance from the location, and then analyzing them. These approaches may not capture events happening nearby that may influence what is happening at the location and contribute to a better overview of what is happening in the area. We address this research gap by providing a method for the spatial analysis of aggregated mobile phone data, taking into account the spatial interaction inherent in the data, and its subsequent semantic enrichment using the analysis results of geo-social media data, all in the context of large urban events.

3. Methodology

This section introduces a methodology which provides information about large crowds at urban events by fusing mobile phone data and geo-social media data. A detailed workflow is visualized in Figure 1, where input datasets are depicted in green, individual steps in blue, and intermediate results in red. This approach detects unusual crowds, taking into account the spatial interactions inherent in the data, and enriches them with the results of previously analyzed geo-social media posts. This ensures that the contextual enrichment is carried out by considering the whole AOI rather than isolated crowds. We used Python 3.9.16 and ArcGIS Pro 3.1.0 for our analysis.

3.1. Detection of Unusual Crowds

To detect unusual crowds, we used mobile phone data aggregated spatially into grid cells and temporally into regular time steps. For each time step of each grid cell, the data provide the number of visitors, which is computed from the raw mobile phone records. For the spatial neighborhood definition described in Section 3.1.2, information about the age distribution of mobile phone users from whom the data were collected is additionally required.

3.1.1. Baseline Calculation and Data Normalization

In urban areas, the usual number of people at a specific location depends on several factors such as the location itself (e.g., tourist areas tend to be more crowded compared to residential areas) but also the time. If the analysis of mobile phone data is performed on the raw number of visitors, it will mainly identify generally more or generally less crowded areas. Therefore, to detect unusual crowds, the data should be normalized with respect to a baseline. We assumed that the usual number of people depends on the location, weekday, and time. Therefore, the baseline was calculated as the mean number of visitors per grid cell, weekday, and time step during a time period of four weeks before the event. A spatial hot spot analysis was then performed on the relative number of visitors with respect to this baseline. The explicit formula for the calculation of the normalized number of visitors $v\_nor{m}_{i,j}$ for grid cell i and time step j is as follows:

$$v\_nor{m}_{i,j}={\displaystyle \frac{v_{i,j}-b_{i,j}}{b_{i,j}}}$$

(1)

where $v_{i,j}$ is the number of visitors and $b_{i,j}$ the baseline value.

3.1.2. Spatial Hot Spot Analysis

Due to the spatial aggregation of the mobile phone data and human mobility behavior, we assumed that neighboring grid cells in each time step strongly influence each other. This is particularly relevant in the context of large crowds, which usually extend over several grid cells of a mobile dataset. To capture this influence, a spatial hot spot analysis was carried out. To this end, a spatial neighborhood had to be defined based on the area that can be reached by a pedestrian within a time step. For this purpose, we proposed to include information on the age distribution of visitors for each grid cell and time step. The average walking speed for older pedestrians (age $\ge 65$) was set as $1.14$ m/s and for younger pedestrians (age $<65$) as $1.36$ m/s [23]. The average walking speed $aws$ for the neighborhood definition of each grid cell was then calculated by considering the portion of older $\left(ov\right)$ and younger $\left(yv\right)$ visitors in the dataset with the following formula (in m/s):

$$aws=1.14\ast ov+1.36\ast yv$$

(2)

The estimate of the distance (d) a pedestrian can cover within the time period of one time step of the dataset (t in seconds) was then calculated using the average walking speed $aws$ and the Euclidean distance measure. Since information on age distribution is not always contained in mobile phone data, the mean value of 1.25 m/s could also be used.

The spatial neighborhood of a grid cell was then defined as all grid cells within the distance d and thus represented the area around the grid cell that can be reached by a pedestrian in the given time unit. In urban areas, direct routes without turns are often unavailable, particularly when the starting point is further away from the destination. Additionally, we assumed that the likelihood of a pedestrian reaching a specific location diminishes with increasing distance. To represent this, we employed an inverse distance squared weighting of our spatial neighborhood matrix. This ensured that more distant grid cells were weighted less than closer ones. For the individual time steps, we then performed a spatial hot spot analysis based on the Getis–Ord Gi* statistic introduced by Getis and Ord [24], which identifies spatial clusters of either high or low values in the data. The z-scores and p-values returned for each grid cell indicate the statistical significance of the clustering of either high values as hot spots or low values as cold spots. As each grid cell is being considered in the context of its spatial neighborhood, it can only be statistically significant if it has a high/low value and its neighboring grid cells also have high/low values. The Getis–Ord local statistic was calculated separately for each grid cell using the following formula:

$$G_i^{*}={\displaystyle \frac{{\sum }_{j=1}^n{\omega }_{i,j}{x}_j-\overline{X}{\sum }_{j=1}^n{\omega }_{i,j}}{S\sqrt{\frac{n{\sum }_{j=1}^n{\omega }_{i,j}^2-{\left({\sum }_{j=1}^n{\omega }_{i,j}\right)}^2}{n-1}}}}$$

(3)

where $x_j$ is the field value for the grid cell j, ${\omega }_{i,j}$ is the spatial weight between the data points i and j, n is the total number of grid cells, and:

$$\overline{X}={\displaystyle \frac{{\sum }_{j=1}^n{x}_j}{n}}$$

(4)

$$S=\sqrt{{\displaystyle \frac{{\sum }_{j=1}^n{x}_j^2}{n}}-{\overline{X}}^2}$$

(5)

3.1.3. Crowd Detection, Labeling, and Size Calculation

After performing the spatial hot spot analysis, a grid cell was defined as a “crowd cell” if it was classified as a hot spot, i.e., the positive z-score of the grid cell was in the 90% confidence interval. Due to the relatively small spatial resolution of the mobile phone data used in the case study (cf. Section 4), we defined a “crowd” as hot spot cells sharing vertices. All contiguous crowd cells were then unified into a polygon and associated with a unique numeric ID. To calculate the crowd size, the number of visitors across all grid cells within the crowd during the corresponding time step was summed up. Consequently, the resulting dataframe for each time step contained all the crowds with the following features: crowd ID, the geometry of the crowd, and the total number of visitors per crowd.

3.2. Geo-Social Media Data Analysis

It was assumed that the social media input data consisted of posts with a timestamp and a point georeference, which ensures the spatio-temporal relation to an event or at least its direct vicinity. To get more interpretable topics, all posts were translated into English using the Google Translate API. Then, the multimodal JSTTS-GeoGSOM model introduced by Hanny and Resch [8] was used to build spatio-temporal topic–sentiment clusters. It uses a Geographic Growing Self-Organizing Map to cluster a multi-dimensional feature vector, consisting of semantic embeddings, sentiment classifications, and spatio-temporal information, coherently in space. The output are the top k keywords, the prevalent sentiment, a convex hull in geographic space, and a mean time for each cluster. The input data for the model included the translated post texts with georeferences in the form of longitude/latitude coordinates and timestamps transformed to Unix time. An equidistant map projection had to be specified according to the AOI. Experiments by Hanny and Resch [25] showed that a sentiment weight set to $sw={\displaystyle \frac{1}{3}}$ gave high sentiment uniformity and topic quality and was therefore used for the analysis. The spatial and temporal granularity of the output clusters was controlled by the coordinate weight $cw$ and the time weight $tw$. These parameters were relative to the time period and area covered by the dataset, and their value was determined according to the use case. The default parameters for the clustering process were set to achieve a good balance between speed and quality, and thus remained unchanged for the analysis, except for the spread factor $SF$. This parameter was used to control the number of output clusters. The higher the spread factor is set, the more clusters will be formed.

The output of the multimodal JSTTS-GeoGSOM model consisted of two dataframes. The post dataframe contained posts with the input features along with a sentiment prediction, probabilities for each sentiment, and the ID of the cluster it belonged to. For each cluster, the cluster dataframe contained the size, a topic described by keywords ordered by their importance to the cluster, a sentiment, and the duration, which was determined by the time difference between the earliest and latest timestamp of the posts within that cluster.

To make the topics of the clusters more interpretable, the Llama2-70b-chat generative language model introduced by Touvron et al. [26] was used. Separately for each cluster, the model was asked to create a short label describing its content based on the posts in that cluster and keywords. The resulting labels were added to the cluster dataframe as a new feature.

3.3. Contextual Enrichment of Crowds

The contextual enrichment of crowds was performed for each time step separately by adding features to the crowd dataframe, which is described in Section 3.1.3. The added features for each crowd contained a list of IDs of posts that were associated with the respective crowd, the overall sentiment of the crowd, and spatio-temporal topic–sentiment clusters, which were linked to each crowd, with their size and weight.

First, the post dataframe was temporally filtered. In order to mitigate the effects of low post numbers due to the short time periods, posts shared within an hour prior to the end time of the respective time step were used for this. Considering that people close to the crowd may also post relevant information, filtered posts associated with individual crowds were those with a geolocation either within the polygon of the crowd or within the distance of one side length of the mobile phone data grid cells.

Intuitively, the sentiment of the crowd would be determined as the most common sentiment of the crowd’s posts. Due to the rather low number of posts for each crowd, this may often not lead to a clear result. Therefore, to determine the sentiment of each crowd, the average of the sentiment probabilities of the crowd’s posts was calculated for each sentiment. The sentiment with the highest average probability was then declared the overall sentiment of the crowd.

The spatio-temporal topic–sentiment clusters associated with the crowd were defined as all the clusters to which the crowd’s posts belonged. The size of the cluster corresponded to the number of posts it contained, which also provided an idea of how important the cluster was for the event. The importance weight was the proportion of the crowd’s posts covered by the cluster and gave information of how important the cluster was to the crowd compared to other clusters.

The resulting crowd dataframe, in combination with the post and cluster dataframes, provided an overview of the number and distribution of visitors in the AOI, an estimate of their mood, and some of the topics discussed in relation to a mood.

4. Case Study: Vienna Pride Parade 2023

The annual Vienna Pride Parade is Austria’s largest demonstration. It strongly demands equal rights for all people, regardless of their sexual orientation, gender identity, and gender characteristics. In 2023, with more than 300,000 participants, it represented a challenging event in terms of crisis management and was therefore used to demonstrate the above presented methodology [27]. On Saturday, 17 June 2023, the event began at 11 AM near the Rathausplatz, a large central square. At 1 PM, the participants started walking along the Ringstraße, the inner city circular road, against the flow of traffic, i.e., in an anti-clockwise direction, and were expected to return at around 5 PM [28].

4.1. Data

This section describes the datasets used for the case study. In addition, the preprocessing steps are explained to illustrate how the datasets were prepared to fulfill the requirements for use in the methodology presented above.

4.1.1. Mobile Phone Data

The mobile phone data consisted of two datasets that were prepared by one of the major mobile network providers in Austria. The first dataset was collected for the Vienna Pride Parade 2023 spanning the period from Friday, 16 June until Sunday, 18 June 2023. Spatially, it covered the city center of Vienna by dividing it into square grid cells with a resolution of 50 m. For each grid cell, it provided the estimated number of visitors in time steps of five minutes. Due to GDPR requirements, the minimum number of visitors for an entry in the aggregated dataset cannot be less than five. In such a case, there was no entry for that grid cell and time step.

For a sensible baseline, a dataset covering a longer time period was needed. For this purpose, mobile phone data were acquired from the previous year (6 June to 7 July 2022). It should be noted that the Donauinselfest (DIF) 2022, a large open air festival, took place in Vienna from 24 June until 26 June 2022. The dataset covered the whole city of Vienna, albeit with a coarser spatial resolution of 250 m and time steps of one hour. Additionally, it provided information about the age distribution of the users, aggregated into age groups.

4.1.2. Geo-Social Media Data

The geo-social media dataset consisted of posts collected from two social media platforms, namely, Twitter and Facebook. These posts were dated from 16 June until 18 June 2023, corresponding to the time period of the mobile phone data for the Pride Parade. Their georeference was either Vienna as a “place”, i.e., a bounding box of the entire city, or a point geolocation within Vienna. Additionally, we extracted Facebook posts from CrowdTangle. Due to data acquisition requirements, each Facebook post was required to contain the keywords “pride”, “parade”, and/or “LGBT”, which slightly biased a subset of the dataset towards the Pride theme. The Tweets were solely extracted based on their geolocation and time stamp. With 3356 Twitter posts and 188 Facebook posts, our final dataset contained a total of 3543 posts.

Since only 158 posts in our dataset had a precise point location, we randomly assigned point georeferences to all posts with a polygon geometry within the area covered by the Pride mobile phone dataset. After translating the posts into English using the Google Translate API, pride-related posts dated on the day of the parade were identified by semantic clustering using the JTS-GSOM model [25]. The resulting clusters were described with keywords sorted by their importance in the cluster. All posts that belonged to clusters including the word “pride” in the first five keywords were considered as pride-related. To ensure that pride-related posts were mainly placed around the Ringstraße, where the actual event took place, we assigned georeferences randomly in two different areas. Pride-related posts were distributed in the area covering the Ringstraße with a 100-m buffer on either side, while all other posts were distributed in the entire area covered by the Pride mobile phone dataset. These two areas and the resulting georeferences are shown in Figure 2.

4.2. Methodology Application

This section explains how the proposed methodology was applied to the datasets described above and shows the results.

4.2.1. Mobile Phone Data Analysis

To calculate the baseline, the 2022 mobile phone dataset was used. Since the DIF is a highly attended event that has a significant impact on the number and distribution of people in Vienna, the week in which it took place was excluded from the dataset. We calculated the baseline as the average number of visitors by grouping the dataset by grid cell, weekday, and hour. Missing entries were assumed to represent zero visitors.

Since the spatial and temporal resolution of the two mobile phone datasets was different, the number of visitors could not be normalized as described in Section 3.1.1. To make the two datasets comparable, the number of visitors in the Pride dataset and the baseline number of visitors were scaled to values between 0 and 1. To add the scaled baseline to the 2023 Pride dataset, the entries from the datasets were joined spatially (i.e., every 50 m cell received the baseline value of the respective 250m cell from the previous year) and matched by weekday and hour. The input feature for the hot spot analysis was then calculated by subtracting the scaled baseline from the scaled number of visitors.

To calculate a more detailed spatial weights matrix for our hot spot analysis, we used additional information on the age distribution of visitors. The portion of older visitors, i.e., visitors aged 65+, in the 2022  dataset was $0.07$, and assuming that the age distribution of mobile phone users had not changed significantly in one year, the average walking speed was calculated using the age distribution from the 2022 dataset, resulting in $1.34$ m/s. Thus, the estimated distance d a pedestrian could cover in five minutes was 402 m.

With the normalized number of visitors and the values for the spatial neighborhood definition, the hot spot analysis was performed and unusual crowds were detected as described in Section 3.1.

4.2.2. Geo-Social Media Data Analysis

For the spatio-temporal topic–sentiment clustering of the social media posts, the JSTTS-GeoGSOM model introduced by Hanny and Resch [8] was used with the following parameters. The coordinates were reprojected into an equidistant conic projection for Europe (ESRI:102031). The coordinate weight was set to $cw=1$, the time weight to $tw=1.5$, and the spread factor to $SF=0.75$. This resulted in 41 clusters, which were well localized in space (cf. Figure 3) and most of them ($80.5\%$) had a duration of less than 2 hours, providing a good basis for gaining insights into what happened during the event at specific locations at a specific time. Based on the keywords and labels, 26 of these clusters were somewhat related to the Vienna Pride. The results used for the contextual enrichment were the post dataframe (cf.

Table 1. Sample of the post dataframe. User names and URLs were anonymized. Geometries were omitted for formatting reasons.

Input_Text	Sentiment	P_Neg	P_Neut	P_Pos	Cluster_Id	Post_Id	Date_Time
Happy #ViennaPride @url …	positive	0.0202	0.2704	0.7093	0	478	17 June 2023 12:12:45
@user Absolutely. I’m satisfied.	positive	0.0260	0.0776	0.8963	25	482	17 June 2023 12:15:12
@user GameDay! Early bird beer …	positive	0.0145	0.2683	0.7171	0	497	17 June 2023 12:32:14

) and the cluster dataframe (cf.

Table 2. Sample of pride-related clusters from the cluster dataframe.

Id	Size	Keywords	Sentiment	Duration	Label
0	44	pride, happy, vienna, lgbtiq, …	positive	01:27:00	Vienna Pride Parade 2023
3	33	pride, beautiful, vienna, today, many, …	positive	01:04:00	Vienna Pride Parade 2023
15	42	vienna, viennapride, live, pride, rainbow, …	neutral	01:23:00	Vienna Pride Parade 2023
20	47	vienna, viennapride, time, pride, rainbow, …	neutral	01:29:00	Vienna Pride Parade and Events
24	25	spiritually, gain, ground, things, …	neutral	01:26:00	Spirituality, queerness, and social issues
25	28	spiritually, gain, pride, ground, …	neutral	01:37:00	Pride Parade Vienna

), which was enhanced with labels generated using Llama 2. The quality of the labels was evaluated by manually checking a random sample of Tweets from each topic.

4.2.3. Contextual Enrichment of Crowds and Results

The contextual enrichment of crowds was performed for each time step separately by adding three features to the crowd dataframe. The first feature contained the IDs of posts which were associated with the crowds. For each crowd, these were posts which had been posted within an hour before the time step and a distance of 50 m from the crowd. The second feature provided an estimate of the crowd’s sentiment, which was determined by calculating the average probabilities of each sentiment of the crowd’s posts and then choosing the sentiment with the highest average probability. The third feature contained clusters associated with the crowds together with the cluster size, i.e., the number of posts it contained, and the portion of the crowd’s posts contained in that cluster. The entries of this feature were in the form $\{c_1:(s_1,p_1),...\}$, where $c_1$ is the cluster ID, $s_1$ its size, and $p_1$ the portion of posts. An example of the enriched crowd dataframe is shown in

Table 3. Enriched crowds for 1 PM.

Crowd	Geometry	Visitors	Post_Ids	Sentiment	Cluster_Size_Weight
6	POLYGON ((16.35871 48.20962,…	19,764	475	negative	{24: (25, 1.0)}
7	POLYGON ((16.36614 48.20964,…	1226	493	neutral	{15: (42, 1.0)}
10	POLYGON ((16.36542 48.21465,…	3764	1140	neutral	{3: (33, 1.0)}
11	POLYGON ((16.37133 48.20799,…	2419	516	neutral	{20: (47, 1.0)}
12	POLYGON ((16.36833 48.21628,…	1331	1095	positive	{0: (44, 1.0)}
13	POLYGON ((16.37045 48.21706,…	654	478, 482, 497	positive	{0: (44, 0.667), 25: (28, 0.333)}

, which includes all enriched crowds for 1PM, and the information on the respective clusters is provided in Table 2.

These tables show that the semantic content of most of the clusters for the enriched crowds was about the Vienna Pride Parade, but there were differences in the sentiment. Crowds $7,10$, and 11 had a neutral position towards the parade and Crowds 12 and 13 a positive one. Crowd 6 exhibited a rather negative attitude towards spirituality, queerness, and related social issues.

The information gained from the mobile phone and social media data provided a good basis for visualizations that could help to improve situational awareness during the event. Figure 4 and Figure 5 show the crowds detected in the area around Rathausplatz on the day of the parade. It also shows the clusters which were associated with them and their sentiment. As the topics of the clusters were all related to the Pride topic, this figure gives an idea of the sentiment towards the event in different crowds at different times. It is also noticeable that the crowd at the Rathausplatz (cf. Crowd 6 in Figure 4) was quite large at the time of the gathering, which was at 1PM.

The results and their visualizations have shown that the contextual enrichment method introduced in this paper can provide valuable insights into the event. By analyzing each time step, it detected unusual crowds, estimated their mood, and associated them with topics. By looking at multiple time steps, it is possible to observe how crowds, their mood, and topics changed over time at different locations. This can provide an overview of what is happening during an event, potentially leading to improved situational awareness.

5. Discussion

In the following, we discuss some limitations of the methodology. In particular, we address shortcomings arising from the different data inputs and design decisions of the methodology. Due to the somewhat synthetic nature of the use case, we will not interpret and contextualize its results, but mainly highlight findings that arose from conducting the case study.

5.1. Methodology

To find unusual crowds, we normalized the mobile phone data with respect to a baseline to ensure that the spatial hot spot analysis identified areas with unusually high/low number of visitors rather than areas with generally high/low number of visitors. The disadvantage of this approach is that if an event takes place at a location that is regularly very crowded, the crowds in that location may not be identified. As a consequence, these locations may need to be analyzed separately.

As a part of our crowd detection, we had to define a spatial neighborhood for the spatial hot spot analysis, which only considered pedestrians. However, mobile phone data is also being collected from people traveling by car or other means of transport. In order to include these people in the analysis, additional information on the means of transport used by visitors would be required. In addition, taking into account the street structure of the area in the spatial neighborhood definition could provide a more refined analysis. However, the associated increase in computational complexity would likely have a negative impact on the performance of the analysis.

To conduct a meaningful analysis of geo-social media data for the contextual enrichment of mobile phoned data, rather large datasets of posts with exact georeferences are required. However, access to such datasets may be limited. Georeferences in social media posts are often provided in the form of large-scale polygons, i.e., bounding boxes of districts, cities, or even larger regions, that are not suitable for the application of the presented methodology. Therefore, in our use case, we converted the polygon geometries of the Tweets to points, which led to a potentially strong distortion of the geometries. In real-world applications, the coarse geometry of much geo-social media data is certainly a major issue. Methods for improving these geometries, e.g., the extraction of specific locations (in this context, e.g., squares and public transport stations) from the texts [29] or attached imagery [30], should therefore be tested.

Apart from spatial characteristics, the pure number of posts available for the analysis also has a strong impact on the contextual enrichment. If a crowd is associated with only a few posts, the results of the methodology may not provide a good representation of the topics discussed or an accurate estimate of the mood of the crowd. Depending on the use case (e.g., the size of the area of interest and timeframe), this could represent a key limitation, particularly in the context of a potential near real-time application of the methodology. Specific crawling strategies (e.g., the extraction of posts based on specific locations), as well as data augmentation methods (such as the previously described location extraction), could at least mitigate these shortcomings. In order to increase the pure quantity of data, the inclusion of other social media platforms (e.g., TikTok, Telegram, and Mastodon) could also be tested, although we expect similar limitations with these data sources with regard to their spatial accuracy.

5.2. Case Study and Results

Our methodology was demonstrated on the basis of an analysis of the 2023 Vienna Pride, which uncovered some potential shortcomings that might also arise for other use cases.

Our methodology assumes the availability of mobile phone data from an earlier point in time in order to calculate a baseline for the identification of unusual crowds. However, as in our use case, these data may not be available in exactly the same format in terms of spatial and temporal resolution. We therefore had to apply a modified version of the first part of the methodology for our use case. The baseline was calculated using the 2022 mobile phone dataset, which had a coarser spatial and temporal resolution. This difference in resolution was also visible in the results of the spatial hot spot analysis (cf. Figure 6), where large continuous cold spots still reflected the underlying structure of the 2022 data. Furthermore, the results showed the gathering that took place on the Rathausplatz at 1 PM (Crowd 6 in Figure 4), but the crowd moving along the Ringstraße afterwards was not clearly visible.

The detailed resolution of the Pride dataset and GDPR restrictions, which set the minimum number of visitors for an entry to five, resulted in many missing entries in our dataset. Therefore, an approach to smoothing the data was tested. Since the dataset covered the city center of Vienna, it was reasonable to assume that some people were present. Therefore, all missing entries were imputed with having two visitors, the mean value between zero and four. As the results were not significantly different from the results of the original data, this smoothing approach was not applied in the case study. We assumed that a more sophisticated method, such as imputing the missing number based on its direct neighborhood, would not significantly change this either.

Unfortunately, there was a lack of geo-social media data with exact georeferences for the Vienna Pride Parade 2023. For most of the posts in our dataset, the georeference was based on the “place” Vienna, i.e., a bounding box of the entire city. For the application of the methodology, these posts were randomly assigned exact georeferences within the area of the mobile phone data. As the clustering process with the JSTTS-GeoGSOM model also takes the georeference into account, the content of the geo-social media data analysis was not exact. This was then also reflected in the contextual enrichment, which, in this case, could not accurately describe the topics discussed and the mood. In order to obtain more semantically precise clusters, a cascading topic modeling approach could be tested, in which pride-related clusters are subjected to further topic modeling. This could be used, for example, to identify posts that relate to a specific performance in the context of pride.

6. Conclusions

This paper introduces a novel methodology for improving situational awareness during organized urban events through the contextual enrichment of crowds detected from mobile phone data with geo-social media data. The development was motivated by the complementary properties of these data types and the lack of research on their combination in the field of emergency management.

We introduced an approach for analyzing mobile phone data to detect crowds during urban events. This approach involved a spatial hot spot analysis of the data normalized with respect to a baseline (RQ1). The hot spot analysis considered the data in the context of their spatial neighborhood, which was intended to capture the spatial interaction inherent in the data. Since we assumed that the mobile phone data provided an estimate of the number of people present for each grid cell and time step, the size of each crowd was calculated as the sum of the estimates for the respective grid cells and time steps. The results of the first part of the methodology included crowds that do not form regularly, together with their size.

Additionally, we introduced an approach to contextually enrich these detected crowds with semantic and sentiment information from geo-social media data. For this purpose, we considered the whole event rather than isolated crowds. To pursue this goal, the second part of the methodology focused on the spatio-temporal topic–sentiment clustering of geo-social media posts. The third and final part of our approach presented the actual contextual enrichment. After associating the crowd with posts and topic clusters based on spatio-temporal criteria, semantic information was extracted and summarized from clusters that had a topic associated with the respective event. Furthermore, the overall sentiment of the crowd was determined from the sentiments of the analyzed posts. To demonstrate the proposed methodology and the results, a case study of the Vienna Pride Parade was conducted.

A central limitation of our methodology is the dependence on a sufficient quantity of geo-referenced social media posts in order to be able to make meaningful statements. Therefore, for a real-world application of the methodology presented here, we estimate that at least several hundred georeferenced social media posts within a short timeframe (e.g., a day) would be necessary, depending on the size of the study area. As our case study showed, the often coarse spatial resolution of Tweets is a major restriction, especially when analyzing small-scale events. Future research could therefore investigate the inclusion of other data sources in the methodology, as well as the application of location extraction methods to geocode spatially implicit posts.

Another important point that should be considered more closely in future research are privacy-related factors that affect both mobile phone and geo-social media data. Particularly against the background of geo-privacy [31], further measures need to be investigated to prevent the potential identification of individuals in the results of our methodology. In the context of social media data, this could be done, e.g., by the automated summarization of the post texts in pre-processing. For the aggregated mobile phone data used in this study, legal measures have already been implemented due to the minimum number of persons prescribed in the GDPR. Such restrictions, however, are not available in other regions of the world and could possibly still lead to a certain identifiability of groups through complementary data [32]. Dedicated research is needed to mitigate these issues.