LPaMI: A Graph-Based Lifestyle Pattern Mining Application Using Personal Image Collections in Smartphones

Abstract

Normally, individuals use smartphones for a variety of purposes like photography, schedule planning, playing games, and so on, apart from benefiting from the core tasks of call-making and short messaging. These services are sources of personal data generation. Therefore, any application that utilises personal data of a user from his/her smartphone is truly a great witness of his/her interests and this information can be used for various personalised services. In this paper, we present Lifestyle Pattern MIning (LPaMI), which is a personalised application for mining the lifestyle patterns of a smartphone user. LPaMI uses the personal photograph collections of a user, which reflect the day-to-day photos taken by a smartphone, to recognise scenes (called objects of interest in our work). These are then mined to discover lifestyle patterns. The uniqueness of LPaMI lies in our graph-based approach to mining the patterns of interest. Modelling of data in the form of graphs is effective in preserving the lifestyle behaviour maintained over the passage of time. Graph-modelled lifestyle data enables us to apply variety of graph mining techniques for pattern discovery. To demonstrate the effectiveness of our proposal, we have developed a prototype system for LPaMI to implement its end-to-end pipeline. We have also conducted an extensive evaluation for various phases of LPaMI using different real-world datasets. We understand that the output of LPaMI can be utilised for variety of pattern discovery application areas like trip and food recommendations, shopping, and so on.

1. Introduction

Smartphone usage is common for a variety of auxiliary services like photography, game playing, schedule planning, and so on. Utilisation of these services generates large amounts of data with respect to personal use for individuals. This data contains the use patterns of various applications [1,2,3,4,5] and also helps to understand personality traits [6]. For instance, Vijay et al. [5] mined frequent co-occurrence patterns in smartphone application usage. Similarly, Frey et al. [2] proposed detection of life events based on a smartphone user’s app installation behaviour. Do et al. [3] also proposed analysis of the usage of smartphone applications. Therefore, we find that utilizing the data from smartphone usage can help facilitate learning of the lifestyle patterns/behaviour of a user.

Among various types of accumulated data in a smartphone, photographs taken by smartphone cameras are also of vital importance. Photograph collections in smartphones truly demonstrate the interests and likes of a user. A person normally takes photographs of all those things and places which are of interest to him/her. For instance, a tourist may have a large collection of photos of various places, or a person interested in shopping may take photographs of various shopping malls and shopping items, and so on. Therefore, the photo collection in the smartphone can help give information on the lifestyle patterns of a user. The utilisation of a smartphone camera to monitor daily eating patterns of a user [7], and use of a wearable camera to automatically record images for healthcare [8], are some recent studies in this line of work. It has also been observed that images from online resources, like Flicker, have been used for various purposes like predicting home and vacation locations and for trip recommendations [9,10,11]. Interested readers can refer to [12] to obtain a detailed survey on the utilisation of image data from Flicker in various research applications. In contrast, our aim is to focus on personal photograph collections for behaviour analysis rather than relying on external resources.

In this paper, we present Lifestyle Pattern MIning (LPaMI), which was introduced and demonstrated at the World IT Show organised in COEX Mall, Seoul, South Korea from 24 to 27 May 2017 as part of G-ITRC project from Kyung Hee University, South Korea. It is a lifestyle pattern mining application to mine intrinsic patterns existing in user smartphone photo collections. We define the lifestyle pattern of a user as a frequent group of actions performed in different intervals of time. For example, a group of frequent actions for a person of a tourist nature can be visiting a park, followed by a trip to lake, then to mountains etc., over the weekends in a month. We understand that such lifestyle pattern information with respect to the user can be utilised in various types of personalised recommendation services. Hence, LPaMI can be used to extract the naturally occurring lifestyle patterns followed by a user. The novelty of our proposal lies in the proposal of an end-to-end solution for lifestyle behaviour learning that involves extracting objects of interest (OOIs) from photos, modelling OOIs in a graph format, and graph-based pattern mining. OOIs refer to scenes in photographs related to given application area. For example, areas of application can include tourism, shopping, and eating, while their relevant objects can be outing spots, gift items, and food items, respectively. To model the data in the form of graphs, each extracted OOI is declared as a node and an edge indicates a connection between them, for example, two places visited in a sequence. We then apply various graph mining techniques to discover numerous patterns of a user for lifestyle analysis. The key instinct of LPaMI is that its output i.e., a list of lifestyle patterns, can enhance the functionality of any state-of-the-art recommendation application. For instance, various trip advisor applications contain sufficient information for different type of venues for any given city of the world. Some of them are customizable to provide only relevant types of recommendations like suggesting specific types of restaurants in a city to person who loves eating different types of food. However, the customisation capability of these applications is limited i.e., the user must specify his/her likes and dislikes, personal information, and interests. In this way, they are restricted to making recommendations based on a user’s supplied information and do not use his/her patterns followed over the passage of time. We understand that consideration of the lifestyle patterns provides a better overview of the personality traits of the user, which can be used for better recommendations. We show the step-by-step functionalities of LPaMI in Figure 1 as a bird’s-eye view to clearly show our concept. We summarise the contributions of our paper as below:

• We propose mining the lifestyle patterns of a user through his/her personal photo collection in a smartphone. In this way, our goal is to avoid dependency on external resources like GPS and online connectivity. Rather, we want to make maximum use of the local and free resources available. Nonetheless, the support of any external resource can be embedded to further enhance the lifestyle pattern mining capabilities.
• To mine the lifestyle patterns through the photo collections found on user smartphones, we present a prototype application, LPaMI, that recognises the OOIs via state-of-the-art scene recognition techniques. We then model the OOIs in the form of a graph(s) and apply various graph pattern mining techniques to learn the patterns.
• We have designed and implemented a prototype application for LPaMI to demonstrate its functionality and effectiveness. For each phase, like scene/OOI recognition, data migration, graph modelling, and graph pattern mining, we have implemented various state-of-the-art algorithms to present a robust and effective solution. Our implementation of data migration is in Spark environment to make the solution scalable in the case of big data with respect to personal images and to make it useable if the data source is changed from the mobile image collection to any other data repository.
• We also present an extensive evaluation of various phases using real-world datasets to prove the effectiveness of our proposal.

2. Overview of LPaMI

In this section, we present a quick overview of our application, LPaMI, along with its pipeline algorithm. We also present some formal modelling practices like a sequence diagram, use case diagram, and data flow diagram along with a demo/illustration of LPaMI to interact with them. The aim is to explain the internals of LPaMI to create ease of effective utilisation.

2.1. Features of LPaMI

LPaMI allows a user to either consider selected photographs from his/her collection or download in bulk from cloud storage. The retrieved photographs to be processed further can be seen in the “Show Images” screen, as shown in Figure 2a. As soon as the photographs are selected, the back-end process of OOI recognition starts. The retrieved data on OOIs is written to the disk, and the data migration routines start processing to clean, filter, and then load the data in the persistent storage. We then perform graph modelling based on different configuration settings like granularity level, the set of photographs to consider for mining in certain period of time, modelling as either a single large graph or as a set of graphs, visualisation layout, and basic quantitative information of the graphs. Finally, the data of OOIs takes up the shape of a graph and is visualised for analysis, as shown in Figure 2b.

As soon as the OOIs are modelled as a graph, pattern mining can be applied. We show the screen for this purpose in Figure 2c. It allows a user to select the type of patterns to be mined. The user is provided a control using a slider to set a minimum support threshold for pattern mining. Resultantly, all the patterns satisfying the input threshold criteria are displayed. LPaMI also provides a “Search” screen to the user as shown in Figure 2d. Here, a user can select some nodes of interest using a dialogue box and can create a query graph. The query graph is then used to mine patterns containing or similar to the input pattern with the aim of analysing the behaviour in a customised manner. In this way, LPaMI provides an interactive procedure to understand the details of the data.

LPaMI also describes the data using a storyboard, shown at the bottom of the Figure 2b,c. The storyboard explains the data in a textual format and presents the details of data and of mined patterns too. Our idea is to make LPaMI a user-friendly product, so this feature is a great facilitation option. To explain OOIs in terms of a graph, the storyboard reads data from persistent storage along with its metadata to explain the situation. On the other hand, it explains each pattern obtained as a result of applying pattern mining. To build a story, we have a database of different statements, and each is randomly picked and concatenated with the name of every OOI along with the metadata information of the concerned graph. Finally, the statements built using metadata and other statements, describing OOIs, are combined to shape a story. We understand that such procedure of generating the storyboard may be a naïve approach, however, we plan to make it richer in semantics in next release of LPaMI.

2.2. Formal Modelling Practices for LPaMI

We present some formal modelling practices, like sequence diagrams, use-case diagrams, and data flow diagrams to explain interactions with LPaMI. A sequence diagram represents the scenario or flow of events in one single use case, so first we need to define a use case. Let a user aim to visualise search output. The list of steps is as follows. Firstly, the user gets access to the system. Secondly, the user selects the photographs. Thirdly, the data is modelled as a graph. Finally, some query nodes are selected and edges are created between them. We illustrate the sequence diagram for the aforementioned objective in Figure 3a. Figure 3b shows the use case diagram which demonstrates the step-by-step process for the said task. Finally, Figure 3c displays the data flow diagram (DFD) which also demonstrates the above-listed tasks.

Finally, we present a high-level pseudocode (Algorithm 1) that states the step-by-step outline of LPaMI, consisting of four main components.

2.3. Example Illustration

Consider Figure 4 that shows different activities performed by a person for each weekend of last three months i.e., a set of photos provided as input. We find that the person is busy with the family-related activities in the first and second weekends of every month, whereas he/she goes for swimming every third weekend, followed by an outdoor sport at the next weekend. This is a natural lifestyle pattern of interest for the given user, which can be utilised in various personalised recommendation applications. For example, a personalised recommendation service can use this pattern to suggest her playing either tennis or baseball in the last weekend of month 3. This example clearly illustrates the objective of LPaMI, in that how it finds the lifestyle patterns from the photo collection.

Algorithm 1 Pipleline Algorithm for LPaMI

Require: Image collection C of given application area obtained from the smartphone of a user U

Ensure: Patterns of interest mined for U

/* Object Recognition Phase */

1: Extract Objects of Interest from C into text files on disk

/* Data Migration Phase */

2: Load the data into staging area in the database

3: Filter and clean the noise created during processing of the images

4: Migrate the filtered data into the normalised data model

/* Data Transformation into Graph Format Phase */

5: Generate sequence graphs based on given time window

6: Model a graph database or one large graph from the sequence graphs

/* Pattern Mining Phase */

7: Mine user-defined type of patterns of actions performed by U

3. Component-Wise Details of LPaMI

We present details of each component of LPaMI in this section. For each of the phases, we utilise state-of-the-art algorithms and procedures to design an effective product. We accept the fact that our intention is not to propose any new solution/algorithm for any specific phase of LPaMI. However, we aim to design a robust and scalable architecture for LPaMI using already proven solutions so that the output of LPaMI is really effective and meaningful.

3.1. Recognizing OOIs from Photographs

Scene understanding and recognition has attracted significant attention from researchers in the computer vision and artificial intelligence community. Its major objective is to understand or recognise the world captured in a photograph, at a single glance. However, in this work our task is to recognise the OOIs (e.g., beaches, mountains, and so on) from photographs such as scenes. Hence, the OOIs represent the high-level view of a photograph. However, there are various challenges to recognizing OOIs. Typically, OOIs have to be recognised against global properties, noisy surroundings, and under different illumination environments. To deal with these challenges, one of the most popular approaches relies on local features. In particular, a set of local features (like scale-invariant feature transform, SIFT [13]) is extracted from an image and then converted into the global image representation using encoding techniques (e.g., BoV [14], and the Fisher Vector [15]). Lastly, this image representation is fed into the classifier (e.g., SVM, random forest) for recognizing the scene category. Some recent works [16,17,18] have investigated deep convolutional neural networks (CNNs) to compute image representation. These approaches employ fully connected networks to encode an input of raw image as a high-dimensional vector, and hence are computationally expensive too.

We now explain our proposed approach to detect OOIs from the given photo collection from a smartphone user. Our idea is to extract low-level features, like shape features, to recognise the scenes. The block diagram in Figure 5a shows the proposed framework for the scene recognition from photographs.

3.1.1. Pre-Processing

We first pre-process the photographs before the extraction of OOIs. This phase includes processing, for example image conversion and image resizing. Image conversion is employed to change an RGB image to a gray scale image. Image resizing is then used to resize images to a fixed size.

3.1.2. Feature Extraction

We select a state-of-the-art approach, gradient local auto-correlations (GLACs) [19], for the feature extraction process in LPaMI. The GLAC descriptor is an efficient method for obtaining shift-invariant features. This technique is considered as an extended form of the histogram of oriented gradients (HOG) [20] and scale-invariant feature transform (SIFT) [13]. It can also be employed as a local feature descriptor. GLACs extract more features since they are second-order statistics of gradients. They are built on the spatial and orientational auto-correlations of image gradients. Below we formally explain our feature extraction process using GLACs.

Let R be an image region and p = (x, y)t is a position vector. For each pixel, the image gradient ${\left(\frac{\delta R}{\delta x},\frac{\delta R}{\delta y}\right)}^t$ can be represented as the magnitude $m=\sqrt[]{\frac{\delta R^2}{\delta x}+\frac{\delta R^2}{\delta y}}$ and the orientation angle $\theta =arctan\left(\frac{\delta R}{\delta x},\frac{\delta R}{\delta y}\right)$ . Voting weights to the nearest bins are applied to code the orientation angle $\theta$ into D orientation bins. With the gradient orientation vector f along with the gradient magnitude m, the Nth order auto-correlation function of gradients in local neighbours can be described as

$$G\left(d_0,\dots d_N,a_1,\dots ,a_N\right)=\int w\left[m\left(p\right),\dots ,m\left(p+a_N\right)\right]f_{d_0}\left(p\right)f_{d_1}\left(p+a_1\right)\dots f_{d_N}\left(p+a_N\right)dp$$

(1)

where $a_i$ represents displacement vectors from the reference point p, w is a weighting function, and $f_d$ is the dth element of f. Figure 5b shows spatial auto-correlation patterns. In this study, we consider N ∈ {0, 1}, $a_{1_{x,y}}$ ∈ {± Δr, 0}, and w(·) = min(·) according to [19]. For N ∈ {0, 1}, GLACs can be expressed as

$$0^{th}order,G_{N=0}\left(d_0\right)=\sum _{p\in R}^{}m\left(p\right)f_{d_0}\left(p\right)$$

(2)

$$1^{st}order,G_{N=1}\left(d_0,d_1,a_1\right)=\sum _{p\in R}^{}min\left[m\left(p\right),m\left(p+a_1\right)\right]f_{d_0}\left(p\right)f_{d_1}\left(p+a_1\right)$$

(3)

3.1.3. Classification

In order to recognise the OOIs effectively, we have adopted the extremely randomised trees classifier. Extremely randomised trees (extra-trees) are an ensemble of absolutely randomised decision trees, where the complete training sample is applied to build every tree in the tree ensemble [21]. Extra-trees have three important parameters that include the number of attributes K, the number of trees M, and the minimum sample size $nmin$ to divide a node. The extra-trees classifier applies the following process to split the nodes. Firstly, it randomly picks K attributes from all candidate attributes, and selects a random splitting value for every attribute. A score is then assigned to each attribute, and the attribute with maximum variance reduction abilities is picked to split the node. Experimental results in  [21] show that this technique is most of the time superior to random forests in terms of accuracy and time complexity since it eliminates the necessity for the optimisation of the discretisation thresholds.

We encapsulate the execution of this phase in Algorithm 2 that contains details of scene recognition for the OOIs from given set of photographs.

Algorithm 2 Recognition of Objects of Interest from Photographs

Require: Image collection C of a given application area obtained from the smartphone of a user U, Number of Trees $numTrees$, Number of Attributes $numAttr$, Splitting Node Size $nodeSize$

Ensure: Classify the OOI and Extract the Date and Time from C onto the disk

1: Read images from C into I 2: Find Date and Time as image infomation from I 3: Perform RGB to Gray scaling of I and save into $I{m}_{cov}$ 4: Resize the $I{m}_{cov}$ into $I{m}_{resize}$ 5: Perform feature extraction using GLACs as $\left(I{m}_{resize}\right)$ –> $Featur{e}_{vec}$ 6: Compute scores as $[ensemble,scores]=ExtraTree(numTrees,numAttr,nodeSize,trainSet,inputType)$ 7: Determine the scene type as $Category$ via $predictWit{h}_{ExtraTree}(ensemble,Featur{e}_{vec})$ 8: Write the output in the file $(Category,Date,Time)$

3.2. Data Filtering and Migration Phase

As soon as we extract OOIs from the image collection, we place the resultant raw data in the HDFS. In this way, we aim to perform a faster processing of this data using Spark. We use Spark as we understand that photo collections can be very large in various application areas. For example, a user may have taken photographs of the ocean on a certain day, so OOIs can be water, beaches, ships, fish, and so on. Resultantly, big data of time-stamped photos is accumulated for a user over the passage of time. We present a step-by-step procedure of raw data processing in Figure 6a and explain the details of every step and operation in the following sub-sections.

3.2.1. Data Migration from HDFS to Relational Database

We now present details of processing of the raw text data (stored in the HDFS) containing OOIs. The aim of this phase is to clean, filter, format, and normalise the data. A sample of raw text data is shown in Figure 6b that hints at the travel record of a person in terms of visiting different places. Each row contains the name of an OOI along with its timestamp. We also observe that there are few statements with information errors resulting in the non-recognition of entities. This occurs due to classification inaccuracy.

3.2.2. Data Reading from HDFS

We begin with reading the raw text data from HDFS using Apache Spark. Apache Spark is a fast, in-memory data processing engine containing useful APIs that allow developers to execute variety of data-intensive workloads by an easy integration with the HDFS. Initially, we load the batch of text data from HDFS storage into a Spark resilient distributed dataset (RDD) for parallel in-memory distributed processing. RDD is a fundamental data structure of Spark and is an immutable, partitioned collection of elements to be operated on in parallel. On reading the raw data, we load it into an RDD called rawDataInputRDD, as shown in the Figure 6a.

3.2.3. Data Processing in RDDs

Having loaded the data in rawDataInputRDD, we are now able to partition it across the cluster for parallel processing. Since each RDD is immutable, we create rawDataMapRDD that obtains data from rawDataInputRDD and splits it into place and time attributes. This timestamp creation is necessary as it helps in graph modelling, which is the next phase in LPaMI. Finally, the resultant data is stored in rawDataDateFMapRDD.

3.2.4. Transformations, Data Processing, and Normalisation of DataFrames

Data storage in the tabular format is flexible enough to support a wide variety of data transformation operations. For this purpose, Spark provides DataFrame, which is a distributed collection of data organised into named columns. It is conceptually equivalent to a table in a relational database. One can incorporate SQL while working with DataFrames using Spark SQL. Spark SQL is a component on the top of Spark core that introduces a form of data abstraction called SchemaRDD. SchemaRDD enables us to perform in-memory SQL operations on structured and semi-structured data.

To perform in-memory SQL operations on the data, we move the data from rawDataDateFMapRDD into SparkSQL tabularDataFrame. The tabular data may consist of noise or errors e.g., the object not found error. Thus the tabularDataFrame is transformed to remNoiseDataFrame to remove noise or filter unwanted data from the tabularDataFrame and then the refined remNoiseDataFrame is saved on the HDFS in Parquet file format which is a column-based storage format. Finally, the refined remNoiseDataFrame is normalised i.e. assigned to two DataFrames named pictureDataFrame and pictureObjectDataFrame, and then permanently loaded into a persistent relational database.

3.3. Modelling Data as a Graph

With the successful migration of filtered and cleaned data into persistent storage, we obtain the big data of OOIs. Each OOI is a notable entity extracted from every photo. Fortunately, the time stamp factor attached to each photo enables us to align the OOIs into a sequence of actions. We then model list of sequences into graph(s), where a node denotes an OOI and the edges between them demonstrate the order for performing a particular action in certain period of time, like visiting one place after ther other. The main advantage is that this graph modelling enables us to apply various graph pattern mining algorithms for lifestyle behaviour learning. To model graph(s) from sequences, we use the state-of-the-art MUSCLE [22] algorithm. MUSCLE is an iteration-based multiple sequence alignment method, which has been found to be the fastest for large numbers of sequences. Using MUSCLE, we can generate undirected, directed, or weighted graph(s).

Graph Construction

Since we have a time-stamped collection of OOIs, the domain of time is a linearly ordered set T. A time-dependent network is a directed graph $G=(V,E)$, where every edge e has an associated link-traversal function $f_e:T\to T$. We consider two same OOIs as one instance in LPaMI if the time gap between them $▵t=(t_{i+1}-t_i)$ is below a given threshold TimeThreshold. We consider such photographs to be taken in same place and represent them as a single node. Furthermore, if $v_i=v_{i+1}$ and $▵t<TimeThreshold$, they can be converged into one record such as $v=v_i=v_{i+1};t=(t_i+t_{i+1})/2$. If $v_i\ne v_{i+1}$ or $▵t>TimeThreshold$, we create an edge $e=(v_i,v_{i+1})$. We illustrate the sample travel data of a person and display it in a graph format in the Figure 7.

To support applying versatile graph mining operations in LPaMI, we propose generating graphs of two types: (1) a set of sequence graphs; and (2) a cumulated graph. We define both of them below.

Sequence Graph Model: Let f be the function to create the set of sequence graphs $S_t(V_t,E_t)\in S$ from given graph G and time set T. If f satisfies $f\left(t\right)\le f(t+1)$ then the total numbers of vertices and edges are monotonically increasing over time.

Suppose $S_t(V_t,E_t)$ is a sequence graph, where $V_t\subseteq V$ is the set of vertices $v_1,v_2,\dots ,v_k\in V_t$, and $E_t\subseteq E$ is the set edges for the time-instance t. Each edge connects the consecutive two vertices $e(v_i,v_{i+1})\in E_t,i=1,\dots ,k-1$. The vertex $v_1$ is called the source, $t_1$ the starting time, $v_k$ the destination, and $t_k$ the arrival time in time instance t. The set of sequence graphs is homogeneous in nature, where all the nodes are of same type.

Definition 1.

(Set of Sequence Graphs). We define S, $S=S_{t_1}\to S_{t_2}\to \dots \to S_{t_k}$, as a sequence graph where $t_i<t_{i+1}$.

For instance, a user visits every place in a sequence in a real-world scenario. Using the given data, we create a set of sequence graphs and align them to generate cumulated graphs based on the defined granularity level of a user (granularity level can be either year, quarter, month, weekly, or day) using the MUSCLE algorithm. A set of sequence graphs is illustrated in Figure 1 in the graph modelling phase, where the size of each sequence is of one month.

Cumulated Graph Model: Our model takes as input a sequence of edge sets $\left\{E_1,E_2,\dots E_n\right\}$ arriving at times $\left\{T_1,T_2,\dots T_n\right\}$ and a function $fW\left(G\right)$, which outputs the value of a specified statistic on graph G. At each time $T_i$, g is applied to the current aggregated graph $G_i$ to obtain value $w{\left(e\right)}_i$ for edge weight. Each vertex $v_i$ in $G_i$ is merged by node type i.e, if they are same OOIs. Given a large graph and the granularity level, the proposed model creates a set of heterogeneous graphs $G_t(V_t,E_t,T_V,g)$ that consists of various types of nodes. Member graphs at each time interval are combined to create an aggregated graph. Here, $T_V$ is a finite set of node types and the function g is applied to the current aggregated graph $G_t$ to obtain edge weight value $w{\left(e\right)}_t$ for each edge. In each graph $G_t$, the number of vertices is much smaller because the vertex $v_i$ in $G_t$ is merged by node type.

Definition 2.

(Cumulated Graph): The cumulated graph $G(V_t,E_t,T_V,g)$ is a single large graph, where $V_t\in V$ for the set of edges $E_t\in E$. $T_V$ is a finite set of node types, and a function g assigns edge weight in $G_t$.

We illustrate a cumulated graph from a set of sequence graphs in Figure 1 in the graph modelling phase and present the pseudocode of this process in Algorithm 3.

Algorithm 3 Data Transformation into Graph Format Phase

Require: Image data I is produced by Object Recognition and Data Migration Phases for Data Transformation, only such data tuples i have the recognised object o and time information t, given time window $\tau$

Ensure: Graph data $G(V,E)$

/* Sequence Graphs Creation */

Generate sequence graphs S based on given time window τ

1: Assign each tuple $i(o,t)\in I$ into appropriate set $t_i\in T$ where $startf\left(t_i\right)<f\left(t\right)<endf\left(t_i\right)$ and $startf\left(t_i\right)\dots endf\left(t_i\right)=\tau$.

2: Align the created daily sequences $S_d$ using the MUSCLE algorithm, $S_d\to S$

3: Create a set of sequence graphs $S=S_{(}{t}_1)\to S_{(}{t}_2)\to \dots \to S_{(}{t}_k)$.

/* Graph Formating */

Model a graph database $G_t$ or one large graph G from the sequence graphs

4: Identify vertices and edges $O\to V,v_i\in V$ and $e(v_i,v_{i+1})\in E$.

5: Calculate graph statistics and assign attributes by function $fw$.

6: Create the sequence graph $G_t=(V_t,E_t,fW)$; a large graph $G_t\to G$

3.4. Graph Pattern Mining

As soon as graph modelling is performed, we apply state-of-the-art frequent subgraph mining algorithms to extract lifestyle patterns. A variety of algorithms have been utilised to discover various interesting lifestyle patterns, both from our cumulated graph model (Definition 2) and from the graph database (Definition 1).

Table 1. List of algorithms along with their features.

Input	Algorithm	Output
Set of sequence graphs	gSpan	Frequent patterns
Set of sequence graphs	Gaston	Maximal frequent patterns
Set of sequence graphs	kFISM	Top-k frequent induced patterns
Cumulated graph	GRAMI	Frequent patterns

shows the selected algorithms and their characteristics. We use gSpan [23], Gaston [24], and kFISM [25] to mine graph database. The gSpan finds patterns whose occurrence count satisfies the minimum support threshold. The maximal frequent subgraph, found using Gaston, is defined if a frequent graph pattern g is maximal in a database when none of its supergraphs is frequent. Similarly, an induced subgraph of a graph, obtained using kFISM, is a subgraph that is formed from a subset of the vertices of the graph and contains edges connecting pairs of vertices in that subset. On the other hand, GRAMI [26] is used to mine an cumulated graph. It mines the frequent patterns, satisfying some minimum support threshold criteria from a single large graph.

4. Experimental Evaluation

4.1. Dataset Description and System Configuration

We used various datasets for the evaluation of LPaMI as displayed in the

Table 2. List of datasets used.

Name	Instances	Class Labels	Type of Evaluation
8-Scenes (8S) [1]	2688	8	accuracy of processing OOIs
INRIA Holidays [2]	900	8	accuracy of processing OOIs, pattern mining
Memetracker (MT)	13 M		execution time of data migration phase
Brightkite (BK)	4.4 M		execution time of data migration phase
Twitter7 (T7)	15 M		execution time of data migration phase

. The 8-Scene and INRIA Holidays datasets are used to evaluate the performance of our utilised classifiers to recognise OOIs from the photograph collection. The 8-Scene dataset has various images containing eight class labels i.e., coasts, inside city, forests, highways, open country, mountains, streets, and tall buildings. Similarly, the INRIA holidays dataset also has eight categories, e.g., mountains, waterfalls, sunsets, beaches, historical buildings, restaurants, marine habitats, and rivers. For the INRIA dataset we have used 700 images for training and 200 images for testing. For the 8-Scene dataset we have used 1880 images for training and 808 images for testing. Since both of these datasets are much smaller in size, they cannot be used to evaluate the scalability of LPaMI in terms of its performance in data migration from the HDFS to a structured format in a persistent storage space. Hence, we simulate the MT, BK, and T7 datasets for the aforementioned purposes. For the data migration phase, we set up the distributed Hortonworks Data Platform (HDP 2.6.1.0) cluster consisting of one server (Intel machine Core i7-6950X CPU @ 3.00 GHz) and four agents while using the Ubuntu 14.04 LTS operating system. Each machine consists of 4 processors, 8 GB of RAM and 100 GB of hard disk.

All the aforementioned datasets are used for the performance evaluation of its OOI extraction plus data migration phases. For the lifestyle pattern mining phase, we opt to perform effectiveness analysis of LPaMI using the INRIA dataset only since it is a real-world dataset of photographs. The INRIA Holidays dataset contains outdoor scene images. This dataset is helpful for us to simulate the photograph collection created in a smartphone of a user because the outdoor scene images can be referred to as photographs taken during different trips. The results discovered in the INRIA datasets are of travelling patterns and we present the visualisations obtained using LPaMI.

4.2. Performance Analysis

All of the classification results are achieved by utilizing extremely randomised trees (extra-trees) along with the feature extraction by GLACs. In order to evaluate our proposed method, we have employed two benchmark datasets: 8-Scenes and INRIA Holidays. For the 8-Scene dataset, our method shows 86.32% accuracy as shown in Figure 8b. Figure 8a illustrates the performance of our proposed approach after employing GLACs and extra-trees on the 8-Scene dataset. Here the categories like “open country’ and “streets” show lower accuracy since the “open country” and “mountain” categories contain almost the similar scenes. Figure 8b,c illustrate the comparison between GLACs and other feature extraction techniques that include HOG, GIST and SIFT and comparison with existing works for the 8-Scenes dataset. On the other hand, for the INRIA Holidays dataset our approach shows an average accuracy of 88.0%. Finally, Figure 8d, shows the performance of our approach after employing GLACs and extra-trees. These experiments demonstrate a reasonable accuracy of our proposed approach and also that of the selected classifiers for LPaMI.

It should be recalled that in the pre-processing stage, we have resized the images to a fixed size of $384\times 256$. The reason for setting such a low resolution is that the size of the images in the INRIA dataset is 3268 × 2448 and that of the images in the 8-Scene dataset is 256 × 256. The depth of the input images in both the INRIA dataset and 8-Scene dataset is 24 bits and after converting image from RGB image to gray image the depth becomes 8 bits. In case of INRIA dataset, if we use the original size of the image then the total time required to extract the features using GLACs is approximately 3372.5817 s (average accuracy of 90.1%), and on the other hand if we resize the images to 384 × 256, then the total time required to extract the features using GLACs is approximately 302.0094 seconds (average accuracy of 88.0%). Therefore, in order to reduce the feature extraction time, we have reduced the size of the images to a low resolution.

For the data migration phase, our aim is to check the scalability of data transfer from HDFS to persistent storage under different constraints. These constraints are the varying types of operations for data, as shown in

Table 3. Operations performed during data migration phase from the disk to persistent storage.

Data Migration Mode	Set of Operations
Type-1	1. Data loading 2. Mapping data attributes 3. In-memory SQL operations 4. Data cleaning 5. Save to the HDFS in Parquet file format 6. Data normalisation
Type-2	1. Data loading 2. Mapping data attributes 3. In-memory SQL operations 4. Data cleaning 5. Save to the HDFS in Parquet file format 6. Data normalisation
Type-3	1. Data loading 2. Mapping data attributes 3. In-memory SQL operations 4. Data cleaning 5. Save to the HDFS in Parquet file format 6. Data normalisation

. From the results in Figure 9, it is obvious that execution time of data migration improves with an increase in the number of processing nodes for Spark. Hence, we observe a sharp decrease in the execution time in all the datasets except the 8-Scenes dataset. This happens because the size of the 8-Scenes dataset is much smaller than the rest, and hence, data is processed efficiently without much impact on the processing nodes. Next, we observe that the execution time of data migration mode of type-1 is more computationally expensive than the remaining two. We observe the key difference happens due to data normalisation. We perform data normalisation so that there is no redundancy in the data in persistent storage media and to facilitate the graph modelling phase without scanning the entire data.

4.3. Effectiveness Analysis of LPaMI Using Graph Pattern Mining

We use the INRIA Holidays dataset for effectiveness analysis of LPaMI. This dataset contains various images and we consider them as photographs taken by a user during his/her holiday period. To mine the patterns, we utilise gSpan, Gaston, kFISM, and GRAMI. Recall that gSpan, Gaston, and kFISM operate on the graph database, whereas GRAMI mines the patterns from the cumulated graph. Furthermore, kFISM also requires an input about the size of the pattern (number of nodes in a subgraph) to return. So, we can observe a slight variation in the results by the said algorithm.

On obtaining the results from gSpan and Gaston, we observe that gSpan outputs large numbers of patterns whereas Gaston only outputs maximal frequent patterns which are fewer in number. Hence, the output of Gaston is more meaningful for learning the lifestyle patterns and behaviour of the user. For a lower minimum support threshold like 10%, we observe that there are some common patterns found by both of the algorithms (as shown in Figure 10 and Figure 11). However, the output of Gaston contains only maximal frequent patterns which are not found by gSpan. Furthermore, the number of nodes in such patterns is normally greater than for gSpan, hence providing us with more useful information about the travelling patterns. Figure 12, Figure 13, Figure 14 and Figure 15 further show the results for minimum support thresholds of 20% and 30%, respectively, for both of the algorithms.

In contrast to gSpan and Gaston, kFISM facilitates learning lifestyle patterns in the form of subgraphs of certain sizes by imposing restrictions on the size of the patterns. In this way, it can serve to analyse the patterns containing certain specific nodes. The patterns of size 2, for instance, show many results with the node “historical building” as displayed in Figure 16. We find that there are many visits to historical places after a variety of activities like visits to the beach and amusement parks, playing with pets, and enjoying night scenes. By further detailed analysis of the results of increasing sizes, we find that this image collection is heavily dominated by trips to historical places, as demonstrated in Figure 17, Figure 18 and Figure 19. This helps us to learn the travelling behaviour of the user; the user pays a visit to a historical place after a visit to almost every other type of place i.e., every alternative trip of the user is towards a historical place. Therefore, we find the effectiveness of kFISM in real life to be really useful. gSpan, on the other hand, outputs all the frequent subgraph patterns, which may be larger in number. Hence, it may be hard to interpret the results of a real-life scenario like that of LPaMI. Gaston produces only the maximal frequent patterns; hence their numbers are fewer than those of gSpan. However, it outputs patterns similar to those of gSpan at a higher threshold. On the other hand, by setting patterns of certain sizes, kFISM gives us more flexibility to learn lifestyle patterns with a detailed focus.

We use the GRAMI algorithm to mine frequent subgraph patterns from the cumulated graph. We adjust the frequency to be 2, 3, 4 and observe the results in Figure 20, Figure 21 and Figure 22. Frequency 2 means that the subgraph occurred at least twice in the cumulated graph. Such output by GRAMI demonstrates the frequent behaviour of the user towards obeying certain lifestyle patterns.

Discussion on the Results: We now discuss how the obtained result/output of LPaMI reveals the lifestyle patterns encapsulated in the photograph collections of a user. We understand that the existence/analysis of lifestyle is learned through obtained patterns which contain certain types of nodes and their frequent occurrences with the other nodes. It should be noted that we aggregate and represent similar types of places like historical venues with a unique same label. Hence, we find only one type of node, which basically encapsulates multiple related places.

Based on the obtained results, we observe that almost all of the results contain the node referring to a historical place, which is connected to various other types of nodes like visits to the park, playing with pets, enjoying night scenes, and so on. Such connectivity of nodes demonstrates the travel pattern of the user and her interest towards the subject of history. The user pays many visits to such places and takes the photographs. By further focusing the patterns obtained from various algorithms, we find that most of the edges of the node showing historical places are with node types like night scenes, sunsets, playing with pets, and amusement parks. These patterns provide a hint with respect to the nature of travel/lifestyle of the user in that after some trips to historical places, the user spends some time with pets, and similarly visits some places to enjoy night scenes and sunsets. We find instances of the historical venue multiple times along with aforementioned types of nodes, therefore, they appear at different support thresholds.

Since our definition of lifestyle patterns (stated in the Introduction Section) corresponds to a frequent group of actions (travel to different places in the datasets used), this very aspect is also the focus of the utilised graph-mining algorithms like gSpan, Gaston, kFISM, and GRAMI. Hence, they rightly match with the anticipated outcomes of LPaMI. As a result of frequent visits to certain places, the utilised graph mining algorithms in LPaMI also reveal the corresponding patterns.

5. Related Works

In this section, we present a literature review of existing works related to various aspects of LPaMI.

5.1. Learning User’s Lifestyle Patterns via Smartphone Data

The MIT Technology Review reports 99.5% of newly created digital data remains unanalysed, as highlighted by Liew et al. in their study [27]. Motivated by this fact, there is a great need to make use of the digital data from various possible computational perspectives. Personal data collection in the smartphone of a user is one of the potential areas to focus on. This data can be mined to discover hidden knowledge patterns like frequent activities, etc. Therefore, LPaMI aims to mine the personal photograph collection available in the smartphone of a user to analyse his/her interests and likes. In this way, the deriving force behind the proposal of LPaMI is bi-folded: maximum utilisation of local resources (photograph collection in this case) to analyse personal lifestyle traits while preserving privacy and security. We now review some of the existing studies which use smartphone data for lifestyle analysis and show the uniqueness of our proposal.

Lane et al. [28] present a smartphone app, BeWell, that tracks the activities of a user which impact physical, social, and mental wellbeing, namely sleep, physical activity, and social interactions. They concurrently monitor multiple dimensions (e.g., sleep patterns, social interaction, and physical activity), representing a more complete picture of the user’s overall wellbeing. For this purpose, they use multiple sensors available in smart-like accelerometers, digital compasses, gyroscopes, GPSs, microphones, and cameras. Similarly, Fahim et al. [29] utilise a smartphone to track sedentary lifestyle in daily routine. The authors propose helping ordinary persons to monitor their lifestyles in a proactive manner. For this purpose, they rely on the accelerometer sensor available in a smartphone to sense the motion patterns of a user. In another study [30], they also focus on learning the sedentary behaviour of a person through the smartphone. In this study, they utilise the audio sensor of smartphone in addition to the accelerometer. Similarly, we find Raghu et al. [31] develop a method to assess daily living patterns using a smartphone equipped with microphones and inertial sensors. They fuse the information retrieved from various sensors of a smartphone and present a computationally light-weight algorithm to identify various high level activities or lifestyle patterns of a user. Wang et al. [32] propose a smartphone app, called StudentLife, that evaluates the impact of the workload on stress, sleep, activity, mood, sociability, and mental well-being of students on a daily and weekly basis. StudentLife makes use of MobileEMA, which is an ecological momentary assessment (EMA) component available for android and iPhone, to examine the behaviour of the student’s mental state across the defined study period. In a similar manner, Xiang Sheng et al. [33] propose a smartphone sensing app (called LIPS), to learn the lifestyles of a user. Their idea is to leverage multiple sensors on a smartphone to obtain a comprehensive view of the context (such as location, local weather, activities etc.) of a mobile user over a long period, and to find out what a mobile user likes to do (characterisation) and what she/he will do next (prediction) based on the collected sensor data about the places of interest (PoIs). We find that the idea of LIPS is similar to our idea of LPaMI, however, we go few steps ahead and mine the patterns using graph-based approach to learn the interactions among different actions performed or places visited in a sequence. We understand that all the aforementioned studies are quite useful. However, they rely on multiple sensory data sources. On the other hand, we only focus on the photo collection to model the lifestyle data. Therefore, by imposing a lesser dependency on the available resources, we aim to retrieve maximum information. We accept the fact that only relying on the photographs collection for lifestyle analysis may not be accurate and detailed enough for different application areas, however, we aim to design a light-weight application that is purely a reflection of the interests of a smartphone user.

The above-mentioned studies consider various sensors. However, smartphone cameras and wearable cameras have also been utilised for lifestyle data analysis. Gupta et al. [7] use the smartphone camera to observe the daily eating patterns of various individuals. The authors argue that food diary and food frequency questionnaires are typically used to learn food eating patterns. However, they cannot focus on learning daily eating pattern data and its frequent variations. Therefore, many researchers are developing methods to collect evidence-based and time-stamped ingestion data and to analyse it for daily patterns of eating and fasting. Hence, camera phones are increasingly being used in nutritional studies. In another study, Doherty et al. [8] focus on the use of a wearable camera in the healthcare domain. The authors state that these devices can be worn all day and automatically record images from a first-person point of view, requiring no intervention or attention from the subject or the researcher. In this case, one of the most mature visual lifelogging devices is Microsoft’s SenseCam, a wearable camera. All the above mentioned studies are very useful for lifestyle analysis, however, the uniqueness of our proposal lies in its graph-based modelling. Modelling data as a graph preserves an order and shows relationships between different actions performed, hence, the output is more meaningful.

We find that Apple [34] provides a machine learning framework, Core ML, to develop apps for iPhone and iPads. Core ML helps to develop various apps for vision aspects (like face tracking, face detection, landmarks, text detection, and so on.) and natural language processing applications (like language identification, tokenisation, lemmatisation, part of speech, and named entity recognition.) In particular, Core ML provides various ready-to-use models like SqueezeNet, ResNet50, Places205-GoogLeNet, and so on. The models like SqueezeNet and ResNet50 detect the dominant objects present in an image from a set of 1000 categories such as trees, animals, food, vehicles, people, and more. On the other hand, Places205-GoogLeNet is very similar to what we propose in LPaMI in that it detects the scene of an image from 205 categories such as airport terminals, bedrooms, forests, coasts, and more. Hence, we find that our idea of LPaMI is a practical step towards learning the lifestyle behaviour of a smartphone user since we further make use of the detected scenes (OOIs in our case) for pattern searches. Apple claims that the app development and mobile data remain secure as the Core ML models run on the device so data does not need to leave the device to be analysed. That is also the aim of LPaMI: use of minimum (one-dimensional) local plus freely collected data in the smartphone.

We understand that the above cited studies may be providing more information about the lifestyle patterns of a smartphone user as compared to our proposed approach in LPaMI. However, our aim is to design an app which is light-weight in terms of using the minimum possible resources (only the photograph collection in this case) which are obtained without spending extra effort on data collection. Photograph collection is one such resource established in a smartphone, which does not require any extra effort by the user and does show intrinsic likes and interests at the same time.

5.2. Recognizing OOIs

Scene recognition and understanding is one conventional issue in the field of computer vision and machine learning. The key to identifying a scene type is to obtain its global information, which shows higher-level semantics. The global information can be obtained using various approaches like in [35]. Numerous works have already been done for scene recognition from images. However, HOG [20] and SIFT [13] features are still considered as state-of-the-art, as they are able to extract the local shape information of the images. In [36], the authors proposed topic model-based scene recognition, which is able to obtain both global and local features of the images. Gradient orientation consistency (GOC) is also able to obtain the local information from the images, and was introduced in [37]. However, the GOC is inspired from LBP and HOG. The main benefit of the GOC feature is the insensitiveness to illumination variance. In [38], the authors presented census transform-based feature extraction from the scenes. Moreover, a co-occurrence modelling framework is proposed in [39] to recognise the scenes, where multiple features are integrated in the semantic space. In our work, we also utilise state-of-the-art approaches to recognise OOIs with high accuracy.

5.3. Graph Modelling

LPaMI models the OOI data in the form of a graph to make use of graph pattern mining techniques. We observe that there exist many studies which also propose use of different data modelling strategies. For example, Ceccaroni et al. [40] obtained profile information initially from the user to analyse activity actions. The studies by Castillo et al. [41], Lee et al. [42], and Sebastia et al. [43] are also some examples in which the system is built by the combination of agents that corresponds to different operational components such as the user interface, preference and interest elicitation module, analysis and matching module, and route planning along with its generation module. Ant colony optimisation is used by Lee et al. [42] in the design of a route recommendation system that incorporates users. Optimal coverage of the attractions along a route is obtained here by approximating a solution of a travelling salesman problem by an autonomous entity set representing the ants that globally and indirectly communicate through pheromone mediation. These existing studies mine activity patterns based on statistics or sequential pattern models. However, these models are limited in their functionalities in that the statistics-based model cannot reflect all activity sequences and the sequential-pattern model can mine only daily activity patterns. On the other hand, our graph-based approach is very flexible in terms of supporting different types of graph mining algorithms.

We observe that Lee [44] proposed a graph-based approach to model and index video data. For this purpose, a a new graph-based video data structure is proposed, called the Spatio-Temporal Region Graph (STRG), which represents spatio-temporal features and the relationships among the video objects. The STRG is then used for various tasks like detecting object graph and background graph to facilitate searching process. Hence, we find that there exist some works which model video data in the form of a graph (where graph modelling from images can be considered as its simplest implementation) for certain tasks.

5.4. Graph Pattern Mining

Frequent subgraphs are patterns that have repeated or multiple instances within the given graph data. Frequent subgraph mining discovers common substructures which are used for data analysis, including graph classification, clustering, and indexing. For instance, this can be useful to find the association between chemical elements common to several chemical molecules or discovering frequent patterns in social networks to understand the functioning of society or the dissemination of diseases.

There exist numerous benefits of frequent subgraph mining (FSM), so many works have been proposed. They are divided into two categories: FSM on transactional graph data, and on a single graph. The transactional case allows the input as a set of small graphs, where each graph represents a transaction. A subgraph is frequent if it exists in at least t transactions, where t is a user-defined threshold. Some of the representative works in this domain are AGM [45], gSpan [23], CloseGraph [46], SPIN [47], Gaston [24], and Mofa [46]. On the other hand, in case of a single large graph, a subgraph is frequent if it has at least t appearances. The studies like SIGRAM [48], GREW [49], and GRAMI are some of the known studies. Since our aim is to utilise different pattern mining algorithms to learn lifestyle behaviour, we just provide a bird’s eye view of various algorithms in this section. Interested readers can refer to [50] for detailed study.

6. Conclusions

In this paper, we present LPaMI, which is an application to mine the lifestyle behaviour of a person from the photograph collection in his/her smartphone. It is a common practice to take photographs of things/places of interest. Therefore, we utilise this user data in order to learn lifestyle patterns. In this way, there is no dependency on external data sources or utilities. The uniqueness of our proposal lies in the graph-based modelling of related entities extracted from the photographs to make the data applicable for various graph pattern mining algorithms. We understand that the proposed model of LPaMI does not introduce any breach of privacy, as personal image data is kept with the user of the application and is locally mined for behaviour learning. Hence, LPaMI can be used as a plug-in for various services like recommendation systems. In the future, we plan to develop a smartphone application for LPaMI to directly operate on the existing photograph collections of users in smartphones.