Evaluating the Privacy and Utility of Time-Series Data Perturbation Algorithms

Abstract

Data collected from sensor-rich systems may reveal user-related patterns that represent private information. Sensitive patterns from time-series data can be protected using diverse perturbation methods; however, choosing the method that provides the desired privacy and utility level is challenging. This paper proposes a new procedure for evaluating the utility and privacy of perturbation techniques and an algorithm for comparing perturbation methods. The contribution is significant for those involved in protecting time-series data collected from various sensors as the approach is sensor-type-independent, algorithm-independent, and data-independent. The analysis of the impact of data integrity attacks on the perturbed data follows the methodology. Experimental results obtained using actual data collected from a VW Passat vehicle via the OBD-II port demonstrate the applicability of the approach to measuring the utility and privacy of perturbation algorithms. Moreover, important benefits have been identified: the proposed approach measures both privacy and utility, various distortion and perturbation methods can be compared (no matter how different), and an evaluation of the impact of data integrity attacks on perturbed data is possible.

1. Introduction

Time-series data collected from various sensor-rich systems (e.g., auto vehicles, wearable devices, industrial equipment) may not reveal tangible personal identifying information, such as name, physical address, or email addresses. However, such data may still reveal essential user-related information (e.g., geolocation, biometrics). For example, time-series data collected from automotive systems, wearable devices, or smart grids contain information that may lead to identifying the end-user [1,2,3]. Thus, sensitive information should be hidden before leaving the sensor-based device and reaching external data processing and analysis systems.

The state-of-the-art research proposes several time-series data perturbation algorithms capable of protecting sensitive data, while exposing useful data for aggregation and analysis purposes [3,4,5,6,7,8,9,10]. Furthermore, these algorithms aim to eliminate sensitive patterns that may lead to user identification, while introducing a minor utility loss for third-party processing systems. However, additional research is necessary to establish a proper (or desired) balance between data privacy and data utility.

This paper proposes a novel methodology for assessing the privacy and utility of time-series perturbation algorithms. It documents a systematic methodology to assess and compare existing data perturbation techniques from the perspective of data privacy and data utility.

The proposed technique is inspired by the cyber attack impact assessment (CAIA) methodology [11], an approach based on system dynamics research [12]. CAIA studies the behavior of complex physical processes in the presence and absence of deliberate or accidental interventions to evaluate cyber-assets’ significance in large-scale, hierarchical, and heterogeneous installations. The impact of the interventions on the system is measured, and decisions involving response adjustments are taken.

Our proposed approach considers a data protection system (from sensors to the perturbation method) similar to a physical process. It measures the impact of various user behavior (normal and sensitive) and data interventions on the time-series perturbation system, and evaluates the utility and privacy of the resulting perturbed data. Furthermore, the computed impact compares various perturbation algorithms and additionally identifies perturbation methods that preserve information on data interventions (e.g., from data integrity attacks). The approach is validated using real data captured from a VW Passat vehicle via the on board diagnostics 2 (OBD-II) port.

We consider the present work to be complementary to previous studies on measuring the utility and privacy of perturbation methods. Previous studies on time-series data perturbation techniques [3,8,9,13,14] focused on demonstrating their performance using specific datasets and conveniently selected parameters. However, more general approaches are necessary for enabling data perturbation on a large scale and for miscellaneous sensor-based data-collection scenarios.

The objectives of the current research were to: (i) define a testing procedure for measuring the data privacy and data utility provided by time series perturbation methods in the case of normal and sensitive behavior, as well as in the case of data interventions (attacks); (ii) describe an algorithm to compare two perturbation mechanisms; and (iii) propose a procedure to identify the types of attacks that can be detected after applying a specific perturbation mechanism.

Our main contributions are the following:

• A systematic procedure for evaluating the utility and privacy of perturbation algorithms. The approach is sensor-type-independent, algorithm-independent, and data-independent.
• A methodology for comparing data perturbation methods.
• We demonstrate applicability by assessing the impact of data integrity attacks on perturbed data.
• We analyze the approach on actual driving data and build the dataset following the stated requirements.

The remainder of the paper is organized as follows: Section 2 provides an overview of perturbation techniques of interest, explains our choices for the privacy and utility metrics, and briefly presents the CAIA methodology that inspired our research. The proposed approach is presented in Section 3. The experimental results are documented in Section 4, discussed in Section 5, and the conclusions are formulated in Section 6.

2. Background and Related Work

2.1. Time-Series Data Privacy Techniques

Time-series data collected continuously from selected data sources (e.g., sensors) are often regarded as time-domain signals. The signal characteristics (amplitude, average, peak and trough, trend, and periodicity) may reveal user-related patterns, and, as they represent private information, they should be subject to protection. Moreover, time-series data protection must consider both time and frequency domains.

Various techniques have been proposed to protect the privacy of potentially sensitive sensor data streams when transmitted over the Internet to third-party processing systems. The protection of private information from time-series sensor data is mainly achieved using encryption, de-identification, and perturbation [15]. Encryption is a standard privacy approach for protecting data from unauthorized access [16]. However, due to processing power constraints, encryption may be challenging to implement in sensor-based systems. Additionally, encryption methods assume that the entities that handle the personal information are trustworthy [17]. Anonymization (also called sanitation [18] or de-identification) consists of deleting, replacing, or hashing all the personally identifiable information (PII) within a dataset. This process does not reduce the data quality, and anonymized data can be safely transported over the Internet. Popular approaches for data anonymization include k-anonymity [19], l-diversity [20], and t-closeness [21]. However, anonymization is difficult to enforce on time-series data that do not contain explicit personal information.

Perturbation or random perturbation, the method of interest for the paper at hand, is a well-known approach for protecting time-series private data. It partially hides information, while maintaining the possibility of information extraction through data-mining techniques. The main advantage of perturbation is that it does not require additional knowledge of the data to be protected. Moreover, the computational complexity of the perturbation is low [18]. Data perturbation techniques include randomization-based methods (additive perturbation [4], multiplicative perturbation [5,6], geometric perturbation [6], nonlinear perturbation [22], differential privacy (DP) [23]) and transformation-based methods [3,7,8,9,10] (collected values are first translated from the original feature space into a lower-dimensional feature space where noise is added). Whether the perturbation preserves the essential characteristics of the data depends on (i) how much perturbation is added (i.e., $ϵ$ in differential privacy) and (ii) how the perturbation is added. Generally, the perturbation amount is predefined by the user.

Differential privacy (DP) or $ϵ$-differential privacy [24] is a widely accepted technique for implementing perturbation-based privacy protection, including for deep learning approaches [25]. Classical techniques (such as anonymization or aggregation) have been the subject of various privacy attacks, and even more so, modern techniques (such as k-anonymity [19]) have failed to protect data from certain types of attacks. Differential privacy protects data by adding a selected amount of noise to the original data using various mathematical algorithms (e.g., Laplace, Gaussian). Differential privacy, initially proposed to protect database queries, guarantees that changing the value of a single record has minimal effect on the statistical output of results [13]. However, obtaining the desired trade-off between privacy and accuracy may be challenging for time series, and it may reduce data utility [15,26].

2.2. Privacy and Utility Metrics for Data Perturbation

Data perturbation involves adding noise to data. The more noise is added, the more data privacy is achieved, and consequently, the more data is hidden, thus also reducing data utility. Many perturbation mechanisms, and, more precisely, those that use data transformation, do not only add noise, but also use other parameters for implementing privacy. Consequently, finding the proper perturbation is not only a matter of increasing or decreasing noise. Several methods for computing these parameters have been proposed [3,9,13], with limited success. In a recent survey, Dwork et al. [27], while referring to finding the optimum privacy budget for differential privacy-based algorithms, stated that there is no clear consensus on choosing privacy parameters for a practical scenario. Thus, to the best of our knowledge, this issue remains an open question.

After applying a perturbation mechanism, the usefulness of the resulting data needs to be measured to ensure that sufficient information is preserved for data analysis and other types of data processing (e.g., anomaly or tampering detection [14,28,29,30]). Utility metrics are essential to data analysts and data-processing applications. There is no universally accepted definition for data utility related to privacy mechanisms (i.e., differential privacy approach). However, generally speaking, data utility or usefulness is measured by the extent to which the chosen data privacy approach preserves aggregate statistical information [18].

Two main aspects are considered when measuring privacy and utility [6]: the privacy loss (PL) and the information loss (IL). Privacy loss measures the capacity of an attacker to extract relevant information about the original data from the perturbed data. In contrast, information loss measures the reduction in the statistical utility of the perturbed data compared to the original data. Therefore, the ideal approaches minimize privacy loss (maximize perturbation) and the loss of information (maximize utility).

The state-of-the-art literature lists several common metrics for measuring information loss (utility), such as variance [31,32], mean relative error (MRE) [33,34], mean squared error (MSE) [35] and mean absolute error (MAE) [13,36]. For all, a lower value implies better utility. To validate the proposed approach, we chose to compute the mean absolute error (MAE) to measure the utility (Equation (1)), a metric often utilized for comparing time-series perturbation methods [13,37]:

$$MAE=\frac{1}{N}\sum _{i\in N}|X_i-X_i^{\prime }|,$$

(1)

where N is the length of the time series, X is the original data vector, and $X^{\prime }$ is the perturbed data.

For quantifying privacy loss, three main approaches have been proposed in the literature [7]: measuring how closely the original values of a perturbed attribute can be estimated [4,13], using information theory [38], or, using the notion of privacy breach [39]. For our research, we calculate the probability density function (PDF) of queries executed on perturbed data and compute the probability of obtaining the original value (computed from the unperturbed data) from perturbed data. The lower the probability, the better the data privacy.

As previously shown, the scientific literature provides a rich palette of perturbation algorithms for time-series data and many metrics for measuring their performance. Despite the extensive research, currently, there needs to be a standardized approach for comparing these perturbation methods. The approach outlined in this paper stands out from previous works in the following ways: First, to the best of our knowledge, the presented procedure is the first approach to simultaneously measure both the privacy provided by the perturbation and the utility of the resulting data. Second, the comparison framework (including data generation) can be applied to diverse perturbation techniques without prior knowledge of the implemented algorithms. We note, however, that the methodology presented in this paper may be perceived as supplementary to the prevailing data privacy and utility metrics.

2.3. Cyber Attack Impact Assessment (CAIA) Methodology

The cyber attack impact assessment (CAIA) [11] builds on the behavioral analysis of physical processes proposed by Ford [12]. Additionally, the sensitivity assessment approach of Huang et al. [40] computes the relative variance between model behavior in the activated and deactivated control loop cases. Control loops rely on observed variables and cause changes to the physical process state via control variables. The objective of the sensitivity assessment is to quantify the contribution of the control loop to the behavior of a certain variable of interest.

The CAIA methodology computes the covariance of the observed variables before and after the execution of a specific intervention involving the control variables. Metrics quantify the impact of deliberate interventions on the control variables. The cross-covariance values, comprising the impact matrix, are computed between the observed variable with no intervention and the observed variable with intervention on the control variable. The impact matrix provides useful information on (i) the impact of cyber attacks on the studied system, (ii) the propagation of disturbances to remote assets, and (iii) equally significant observed variables.

3. Proposed Approach

The proposed methodology is inspired by research in system dynamics, sensitivity analysis, and the CAIA framework. The perturbation method is modeled as a dynamic system, and the perturbation is first analyzed in the absence and the presence of sensitive user behavior. The main symbols used throughout this research are described in

Table 1. Symbols and their description.

Symbol	Description
R	Set of collected regular (typical) user behavior data
S	Set of collected sensitive user behavior data
B	Set of collected user behavior data (regular and sensitive),
	$B=R\cup S$ and $R\cap S=\varnothing$
A	Set of intervention data (integrity attacks), $A\cap B=\varnothing$
$\sigma$	Standard deviation
${\sigma }_{r_{min}}$	Minimum standard deviation of regular user behavior data, $\forall r\in R$
${\sigma }_{r_{max}}$	Maximum standard deviation of regular user behavior data, $\forall r\in R$
${\sigma }_r$	Standard deviation of a regular user behavior data, $r\in R$
${\sigma }_s$	Standard deviation of a sensitive user behavior data, $s\in S$
${\sigma }_a$	Standard deviation of an intervention data (integrity attacks), $a\in A$
$X_r$	Regular user behavior data, $r\in R$
$X_0$	Landmark regular user behavior data
$X_s$	Sensitive user behavior data, $s\in S$
$X_b$	Regular or sensitive user behavior data, $b\in B$
$X_a$	Intervention (attack) data, $a\in A$
$\mathcal{M}$	Perturbation method
$Y_r$, $Y_0$, $Y_s$, $Y_b$, $Y_a$	Perturbed data, $Y_r=\mathcal{M}\left(X_r\right)$, $Y_0=\mathcal{M}\left(X_0\right)$, $Y_s=\mathcal{M}\left(X_s\right)$,
	$Y_b=\mathcal{M}\left(X_b\right)$, $Y_a=\mathcal{M}\left(X_a\right)$
C	Cross-covariance
$\mathcal{C}$	Relative impact of a behavior data on the observed variable (attribute)
$\overline{\mathcal{C}}$	Mean relative impact of a behavior data on the observed
	variable (attribute)
${\alpha }_p$	Behavior-privacy parameter
${\alpha }_u$	Behavior-utility parameter

.

3.1. Perturbation System Architecture and Design Consideration

Sensor data can either be protected by performing the perturbation locally, on the system that gathers the data, or to a remote system by securely transferring the data from the data source to the third-party processing systems. However, due to recent regulations, which explicitly stipulate that “wherever possible, use processes that do not involve personal data or transferring personal data outside of the vehicle (i.e., the data is processed internally)” [41], the local perturbation is preferred (see Figure 1).

Implementing a protection system for time-series data involves choosing the perturbation method, taking into account data and equipment restrictions, such as:

• the type of data leaving the system and the potentially sensitive information they carry;
• the amount of information to be hidden considering possible sensitive information or other external factors;
• utility restrictions (how much information about the data should still be available after perturbation);
• the processing power of the equipment.

Because many data protection mechanisms have been proposed and implemented during the last decade, choosing the most suitable one is challenging. Consequently, the main purpose of this research is to make such decisions simpler.

3.2. Formal Description

Consider time-series data X, collected from a sensor at time instants $T=\{1,2,\dots ,t,\dots \}$ for one observed variable (time series attribute), $X_r$ a vector containing measurements corresponding to regular (typical) user behavior $(r\in R)$, $X_s$ corresponding to sensitive behavior $(s\in S)$, $X_b$ corresponding to any type of regular or sensitive behavior ($b\in B$, $B=R\cup U$ and $R\cap S=\varnothing$), $X_a$ corresponding to data intervention $(a\in A)$. Moreover, consider $\sigma$ the standard deviation of X, $[{\sigma }_{min},{\sigma }_{max}]$ the feasible interval of minimum and maximum standard deviations (available from the sensor specifications), ${\sigma }_r$ the standard deviation of a regular (typical) user behavior, $[{\sigma }_{r_{min}},{\sigma }_{r_{max}}]$ the standard deviation interval corresponding to regular behavior, ${\sigma }_s$ the standard deviation of a sensitive behavior, and ${\sigma }_a$ the standard deviation of a data intervention.

Definition 1.

Time-series data X, collected by a sensor, contains information associated with regular behavior if its standard deviation ${\sigma }_r$ is in the interval of regular operation, ${\sigma }_r\in [{\sigma }_{r_{min}},{\sigma }_{r_{max}}]$.

The interval of regular operation $[{\sigma }_{r_{min}},{\sigma }_{r_{max}}]$ is obtained by computing the standard deviation for several time-series data collected during what is subjectively considered the typical operation of the device or equipment.

Definition 2.

Time-series data X, collected by a sensor, contains information associated with sensitive behavior (user-specific information) of the system user if the standard deviation of X, ${\sigma }_s$, is outside the regular operation interval, ${\sigma }_s\in [{\sigma }_{min},{\sigma }_{r_{min}})$ or ${\sigma }_s\in ({\sigma }_{r_{max}},{\sigma }_{max}]$.

From the privacy point of view, sensitive behavior corresponds to patterns that may lead to user identification or the identification of specific user behavior (e.g., aggressive driver behavior, nervous tics). Thus, such patterns should be recognized and protected by the perturbation system.

Data interventions are conducted either by altering the sensor or injecting erroneous data before the perturbation process occurs (see Figure 1) and, in this research, we associate them with integrity data attacks (e.g., pulse attacks, scaling attacks, random attacks). From the utility point of view, the information that may lead to an attack or anomaly detection should be maintained after data perturbation. The working hypotheses are that the impact of intervention data is more significant than the impact of the sensitive behavior data and that the impact of sensitive behavior data is higher, but reasonably close to, the regular behavior.

Definition 3.

An intervention is an intentional modification of the collected time-series data X, such that the standard deviation of X during the attack (${\sigma }_a$) is greater than the standard deviation of all collected sensitive behavior data, ${\sigma }_a>{\sigma }_s$ or it is smaller than the standard variation of all collected sensitive behavior data, ${\sigma }_a<{\sigma }_s$.

Consider $\mathcal{M}$ a perturbation mechanism that protects information from $X_b$, $b\in B$, while maintaining the possibility of partial information extraction through data-mining techniques such that:

$$Y_b=\mathcal{M}\left(X_b\right),\forall b\in B.$$

(2)

Let $X_0$ denote the reference data of regular operation, called landmark regular behavior, $X_{bt}$ the $t^{th}$ measurement of the observed variable for the b behavior data, and $Y_{bt}$ the perturbation of $X_{bt}$.

The mean of the observed values for the behavior b is defined by:

$${\overline{Y}}_b=\frac{1}{\left|T\right|}\sum _{t\in T}{Y}_{bt},\forall b\in B.$$

(3)

Further, let $C\left(Y_b\right)$ be the impact that behavior b has on the observed variable of the perturbation system, computed as the cross-covariance between the perturbed landmark regular behavior $Y_0=\mathcal{M}\left(X_0\right)$ and the collected behavior data b:

$$C\left(Y_b\right)=\frac{1}{\left|T\right|}\sum _{t\in T}\frac{(Y_{bt}-\overline{Y_b})(Y_{0t}-\overline{Y_0})}{\overline{Y_0}\overline{Y_b}},\forall b\in B.$$

(4)

The impact (Equation (4)) is a measure of how much the output of the system deviates from regular behavior.

The relative impact $\mathcal{C}$ of a behavior b on the observed variable is defined as:

$$\mathcal{C}(Y_b,X_b)=\frac{C\left(Y_b\right)}{C\left(X_b\right)}=\frac{{\sum }_{t\in T}\frac{(Y_{bt}-{\overline{Y}}_b)(Y_{0t}-{\overline{Y}}_0)}{{\overline{Y}}_0{\overline{Y}}_b}}{{\sum }_{t\in T}\frac{(X_{bt}-{\overline{X}}_b)(X_{0t}-{\overline{X}}_0)}{{\overline{X}}_0{\overline{X}}_b}},\forall b\in B.$$

(5)

As any perturbation method introduces a certain degree of uncertainty due to the added noise, the mean relative impact is used to quantify the impact of interventions under uncertainty:

$$\overline{\mathcal{C}}(Y_b,X_b)=\frac{1}{P}\sum _{p=1}^P\mathcal{C}(Y_b,X_b),\forall b\in B,$$

(6)

where P is the number of times perturbation is performed, i.e., $Y_0$ and $Y_b$ are computed. The larger the P, the more accurate the relative impact.

Definition 4.

Let $\mathcal{M}$ be a perturbation mechanism that takes as input time-series data $X_b$, $b\in B$, corresponding to a regular or sensitive behavior, such that $Y_b=\mathcal{M}\left(X_b\right)$, and let ${\alpha }_p$ be a real positive number. $\mathcal{M}$ satisfies $\alpha$-behavior-privacy for the observed variable if it holds that:

$$\overline{\mathcal{C}}(Y_b,X_b)\le {\alpha }_p,\forall b\in B.$$

(7)

where ${\alpha }_p$ is the behavior-privacy parameter. Definition 4 imposes that the relative impact on the perturbed data of any behavior should be less than a pre-defined value, ${\alpha }_p$, for the observed variable. The behavior-privacy parameter ${\alpha }_p$ in Equation (7) defines the desired level of privacy and it should be as small as possible for high data protection. In the case when the mean relative impact $\overline{\mathcal{C}}(Y_b,X_b)$ of a behavior data value b is higher than ${\alpha }_p$, we conclude that the perturbation method $\mathcal{M}$ does not provide sufficient protection, meaning that it does not hide enough information (sensitive patterns can be detected).

Definition 5.

Let $\mathcal{M}$ be a perturbation mechanism that takes as input time-series data $X_b$, $b\in B$, corresponding to a regular or sensitive behavior, such that $Y_b=\mathcal{M}\left(X_b\right)$, and let ${\alpha }_u$ be a real positive number. $\mathcal{M}$ satisfies $\alpha$-behavior-utility for the observed variable if it holds that:

$$\overline{\mathcal{C}}(Y_b,X_b)\ge {\alpha }_u,\forall b\in B.$$

(8)

where ${\alpha }_u$ is the behavior-utility parameter. Definition 5 states the condition to be met by the perturbation mechanism on any behavior data such that the perturbed result is useful. The behavior-utility parameter ${\alpha }_u$ in Equation (8) defines the desired level of utility, and it should be as large as possible for high data utility. When the mean relative impact $\overline{\mathcal{C}}(Y_b,X_b)$ of a behavior data value b is lower than ${\alpha }_u$, we conclude that the perturbation method $\mathcal{M}$ does not provide sufficient utility, meaning that it hides too much information.

An ideal perturbation mechanism for the observed variable satisfies both $\alpha$-behavior-privacy and $\alpha$-behavior utility conditions, such that the mean relative impact $\overline{\mathcal{C}}(Y_b,X_b)$ of any behavior b is in the interval $[{\alpha }_u,{\alpha }_p]$, ${\alpha }_u\le {\alpha }_p$.

3.3. Comparing Perturbation Methods

Consider ${\mathcal{M}}^1$ and ${\mathcal{M}}^2$, two perturbation mechanisms that satisfy $\alpha$-behavior-privacy and $\alpha$-behavior-utility with the targeted ${\alpha }_u$ and ${\alpha }_p$, the behavior-utility parameter and behavior-privacy parameter, respectively.

Definition 6.

For the perturbation methods ${\mathcal{M}}^1$ and ${\mathcal{M}}^2$ we define the following privacy-utility operators:

• ${\mathcal{M}}^1>{\mathcal{M}}^2$: if ${\mathcal{M}}^1$ provides higher utility than ${\mathcal{M}}^2$;
• ${\mathcal{M}}^1\gg {\mathcal{M}}^2$: if ${\mathcal{M}}^1$ provides higher privacy than ${\mathcal{M}}^2$;
• ${\mathcal{M}}^1⋙{\mathcal{M}}^2$: if ${\mathcal{M}}^1$ provides higher utility than ${\mathcal{M}}^2$ and ${\mathcal{M}}^1$ provides higher privacy than ${\mathcal{M}}^2$.

Next, consider $X_0$ the landmark regular operation data, $X_s$, a sensitive behavior data $s\in S$, $Y_0^1={\mathcal{M}}^1\left(X_0\right)$, $Y_s^1={\mathcal{M}}^1\left(X_s\right)$, $Y_0^2={\mathcal{M}}^2\left(X_0\right)$, $Y_s^2={\mathcal{M}}^2\left(X_s\right)$. Then, the mean relative impact is defined in Equations (9) and (10) as:

$$\overline{{\mathcal{C}}^1}(Y_s,X_s)=\frac{1}{P}\sum _{p=1}^P\frac{C\left(Y_s^1\right)}{C\left(X_s\right)}=\frac{{\sum }_{t\in T}\frac{(Y_{st}^1-{\overline{Y}}_s^1)(Y_{0t}^1-{\overline{Y}}_0^1)}{{\overline{Y}}_0^1{\overline{Y}}_s^1}}{{\sum }_{t\in T}\frac{(X_{st}-{\overline{X}}_s)(X_{0t}-{\overline{X}}_0)}{{\overline{X}}_0{\overline{X}}_s}},\forall s\in S.$$

(9)

and

$$\overline{{\mathcal{C}}^2}(Y_s,X_s)=\frac{1}{P}\sum _{p=1}^P\frac{C\left(Y_s^2\right)}{C\left(X_s\right)}=\frac{{\sum }_{t\in T}\frac{(Y_{st}^2-{\overline{Y}}_s^2)(Y_{0t}^2-{\overline{Y}}_0^2)}{{\overline{Y}}_0^2{\overline{Y}}_s^2}}{{\sum }_{t\in T}\frac{(X_{st}-{\overline{X}}_s)(X_{0t}-{\overline{X}}_0)}{{\overline{X}}_0{\overline{X}}_s}},\forall s\in S.$$

(10)

Let us denote $\overline{{\mathcal{C}}_s^1}=\overline{{\mathcal{C}}^1}(Y_s,X_s)$ and $\overline{{\mathcal{C}}_s^2}=\overline{{\mathcal{C}}^2}(Y_s,X_s)$.

Proposition 1.

If $min\left({\overline{\mathcal{C}}}_s^1\right)>min\left({\overline{\mathcal{C}}}_s^2\right)$, for any sensitive behavior s, then ${\mathcal{M}}^1>{\mathcal{M}}^2$.

Proof. 

For $s\in S$, $S\subset B$$\Rightarrow s\in B$. As ${\mathcal{M}}^1$ and ${\mathcal{M}}^2$ satisfy α-behavior-privacy and α-behavior-utility, $\overline{{\mathcal{C}}_s^1},\overline{{\mathcal{C}}_s^2}\in [{\alpha }_u,{\alpha }_p]$. If $min\left({\overline{\mathcal{C}}}_s^1\right)>min\left({\overline{\mathcal{C}}}_s^2\right)$ then ${\alpha }_u<min\left({\overline{\mathcal{C}}}_s^2\right)<min\left({\overline{\mathcal{C}}}_s^1\right)<{\alpha }_p$. Consequently, the impact that any sensitive behavior s has on data perturbed with ${\mathcal{M}}^1$ is higher than the impact of at least one sensitive behavior s on data perturbed with ${\mathcal{M}}^2$; thus, more information about the sensitive behavior is maintained in the data perturbed with ${\mathcal{M}}^1$ for all $s\in S$, providing higher overall data utility. Then, according to Definition 6, ${\mathcal{M}}^1>{\mathcal{M}}^2$.    □

Proposition 2.

If $max\left({\overline{\mathcal{C}}}_s^1\right)<max\left({\overline{\mathcal{C}}}_s^2\right)$, for any sensitive behavior s, then ${\mathcal{M}}^1\gg {\mathcal{M}}^2$.

Proof. 

For $s\in S$, $S\subset B$$\Rightarrow s\in B$. ${\mathcal{M}}^1$ and ${\mathcal{M}}^2$ satisfy α-behavior-privacy and α-behavior-utility; thus, $\overline{{\mathcal{C}}_s^1},\overline{{\mathcal{C}}_s^2}\in [{\alpha }_u,{\alpha }_p]$. If $max\left({\overline{\mathcal{C}}}_s^1\right)<max\left({\overline{\mathcal{C}}}_s^2\right)$ then ${\alpha }_u<max\left({\overline{\mathcal{C}}}_s^1\right)<max\left({\overline{\mathcal{C}}}_s^2\right)<{\alpha }_p$. The impact that any sensitive behavior s has on perturbed data with ${\mathcal{M}}^1$ is smaller than the impact of at least one sensitive behavior s on data perturbed with ${\mathcal{M}}^2$. Consequently, less information about the sensitive behavior is held in the data perturbed with ${\mathcal{M}}^1$ for all $s\in S$, providing higher overall data privacy. Then, according to Definition 6, ${\mathcal{M}}^1\gg {\mathcal{M}}^2$.    □

Proposition 3.

If $min\left({\overline{\mathcal{C}}}_s^1\right)>min\left({\overline{\mathcal{C}}}_s^2\right)$ and $max\left({\overline{\mathcal{C}}}_s^1\right)<max\left({\overline{\mathcal{C}}}_s^2\right)$, for any sensitive behavior s, then ${\mathcal{M}}^1⋙{\mathcal{M}}^2$.

Proof. 

According to Proposition 1, if $min\left({\overline{\mathcal{C}}}_s^1\right)>min\left({\overline{\mathcal{C}}}_s^2\right)$ then ${\mathcal{M}}^1>{\mathcal{M}}^2$, meaning that ${\mathcal{M}}^1$ provides higher utility than ${\mathcal{M}}^2$. If $max\left({\overline{\mathcal{C}}}_s^1\right)<max\left({\overline{\mathcal{C}}}_s^2\right)$ then ${\mathcal{M}}^1\gg {\mathcal{M}}^2$ (from Proposition 2), meaning that ${\mathcal{M}}^1$ provides higher privacy than ${\mathcal{M}}^2$. Then, according to Definition 6, ${\mathcal{M}}^1⋙{\mathcal{M}}^2$.    □

Taking into account Propositions 1–3, we propose the methodology described in Algorithm 1 for comparing two perturbation methods ${\mathcal{M}}^1$ and ${\mathcal{M}}^2$.

Before applying Algorithm 1, data preparation is required. Firstly, we describe the regular user behavior for the tested system, collect regular behavior data, compute standard deviations for all data, and find the interval of regular operation $[{\sigma }_{r_{min}},{\sigma }_{r_{max}}]$. Then, we choose the landmark regular behavior, $X_0$, selected from the collected regular behavior data such that the standard deviation $\sigma$ is the closest to $({\sigma }_{r_{max}}-{\sigma }_{r_{min}})/2$. Thirdly, we collect sensitive behavior data, $X_s$. The constituent steps are also illustrated in Figure 2.

Algorithm 1: Comparison of Perturbation Methods

3.4. Evaluation of the Utility of a Perturbation Method in Case of Data Interventions

External entities can alter sensor data, for instance, by modifying the sensor or changing the data after it is collected. Therefore, monitoring data interventions is essential for maintaining data integrity and detecting anomalies or attacks. We evaluate the impact data interventions have on the perturbed data and estimate the resulting data’s utility. Maintaining enough information after perturbation to detect anomalies/attacks is expected from the utility point of view. Our research focuses on a type of data intervention called an integrity attack, which consists of modifying data after it is collected using predefined patterns.

Consider A the set of possible data interventions. Let an intervention (attack) data $X_a$, $a\in A$, be the input of a perturbation method $\mathcal{M}$. Let $Y_a$ be the perturbed values, $Y_a=\mathcal{M}\left(X_a\right)$, and compute the mean relative impact of $X_a$ on the perturbed data $Y_a$ for an observed variable as:

$$\overline{\mathcal{C}}(Y_a,X_a)=\frac{1}{P}\sum _{p=1}^P\frac{C\left(Y_a\right)}{C\left(X_a\right)}=\frac{{\sum }_{t\in T}\frac{(Y_{at}-{\overline{Y}}_a)(Y_{0t}-{\overline{Y}}_0)}{{\overline{Y}}_0{\overline{Y}}_a}}{{\sum }_{t\in T}\frac{(X_{at}-{\overline{X}}_a)(Y_{0t}-{\overline{X}}_0)}{{\overline{X}}_0{\overline{X}}_a}}.$$

(11)

Denote $\overline{{\mathcal{C}}_a}=\overline{\mathcal{C}}(Y_a,X_a)$ and $\overline{{\mathcal{C}}_s}=\overline{\mathcal{C}}(Y_s,X_s)$, $\forall s\in S$.

Proposition 4.

If $\mathcal{M}$ satisfies the condition:

$$\overline{{\mathcal{C}}_a}>max\left(\overline{{\mathcal{C}}_s}\right),\forall s\in S,$$

(12)

then $\mathcal{M}$ preserves the intervention information such that the perturbed data $Y_a$ is useful for detecting data intervention a, $a\in A$.

Proof. 

For $\forall s\in S$ and $\forall b\in B$, $max\left(\overline{{\mathcal{C}}_s}\right)\ge \overline{{\mathcal{C}}_b}$. If $\overline{{\mathcal{C}}_a}>max\left(\overline{{\mathcal{C}}_s}\right)\Rightarrow \overline{{\mathcal{C}}_a}>max\left(\overline{{\mathcal{C}}_s}\right),\forall b\in B$. Then, the mean relative impact $\overline{{\mathcal{C}}_a}$ of a data intervention a, $a\in A$, is higher than the mean relative impact of any behavior b (regular or sensitive). Thus, the perturbed data resulted from applying $\mathcal{M}$ on a preserves information about the intervention/attack, maintaining data utility.    □

The consequence of Proposition 4 is that, if the impact of an intervention (attack) is higher than the impact of all defined sensitive behavior data, then the perturbed data is useful from the point of view of the attack or anomaly detection.

The proposed approach for evaluating the utility of a perturbation method in case of interventions is described in Algorithm 2. The same data preparation is necessary as in the case of Algorithm 1. Additionally, data intervention $X_a$, $a\in A$, is collected. If the result of the evaluation is positive, then the perturbation method provides utility for the considered intervention data. Otherwise, the usefulness of the perturbation method is low or uncertain.

Algorithm 2: Intervention Impact on Perturbed Data Algorithm

4. Experimental Results

The proposed framework is evaluated from several perspectives. Beforehand, the approach to collecting and generating the necessary data is described. Next, several standard perturbation methods are compared using the impact coefficients and the proposed algorithm for univariate time series. Finally, the method’s applicability is showcased for identifying the possibility of perturbation methods to detect specific types of integrity attacks. We consider the ability of the perturbation method to maintain information about the intervention as a measure of its utility.

The proposed framework is evaluated in the context of three time series distortion algorithms that leverage the discrete Fourier transform (DFT) as the transformation method: (i) a primary distortion method that consists of transforming the data in the frequency domain and filtering the first k coefficients (method denoted within this article as filtered FFT) and that does not introduce any noise perturbation (we use it for emphasizing the validity of the proposed method), (ii) the compressible perturbation algorithm (CPA) [8], based on the Fourier representation of the time series, it adds noise to a fraction of the frequencies, and (iii) the Fourier perturbation algorithm (FPA) [9], the first differentially private (DP) approach that offers practical utility for time-series data. The CPA and FPA algorithms are widely accepted as classical perturbation methods with many applications and variants. Thus, by demonstrating the validity of the proposed approach to these algorithms, we expect to have the framework’s utility applied to other similar perturbation techniques.

4.1. Data Collection and Intervention Generation

The first step in using the proposed framework is collecting data for both regular and sensitive behavior. Further, interventions are generated by simulating various integrity attacks based on regular behavior data.

This research uses data extracted from in-vehicle CAN data. Data was collected via the on board diagnostics 2 (OBD-II) port on a VW Passat vehicle using the OBD Fusion mobile app. Data were recorded every 1 second during driving, and 136 features were extracted through the OBD-II port. All driving data (regular behavior and sensitive behavior) were collected on the same route in similar traffic conditions.

The dataset preparation consists of the following steps:

• Step 1: Collect several normal behavior time-series data, compute the standard deviation $\sigma$ for each one, and identify $[{\sigma }_{r_{min}},{\sigma }_{r_{max}}]$, the interval of minimum and maximum standard deviation possible for normal behavior.
• Step 2: Choose the landmark normal behavior ($X_0$), the data further used for computing impact coefficients and for attack generation. For instance, choose the normal behavior that has the standard deviation closest to the middle of the $[{\sigma }_{r_{min}},{\sigma }_{r_{max}}]$ interval.
• Step 3: Identify possible sensitive behaviors and collect the corresponding data. The collected data qualifies as sensitive behavior if its standard deviation is outside the interval $[{\sigma }_{r_{min}},{\sigma }_{r_{max}}]$, according to Definition 2.

Intervention data is generated from the landmark regular behavior by simulating four integrity attacks commonly utilized in the literature for security studies [42,43], plus the step attack that can be associated with a defective sensor. The list of interventions is not aimed to be exhaustive, but is provided to showcase the methodology in the context of possible attack scenarios. Given the attack interval $[T_{start},T_{stop}]$, the following types of attacks on time-series data are considered:

• Pulse attack: In this case the altered value $X_j^{*}\left(t\right)$ is obtained by dividing the value of the attribute j at time t, $X_j\left(t\right)$, by an attack parameter $a_p$: $X_j^{*}\left(t\right)=X_j\left(t\right)/a_p$ with t in the attack interval $[T_{start},T_{stop}]$;
• Scaling attack: The value $X_j\left(t\right)$ is scaled by attack parameter $a_p$: $X_j^{*}\left(t\right)=a_p·X_j\left(t\right)$, for $t\in [T_{start},T_{stop}]$;
• Ramp attack: This type of attack adds values from a ramp function $X_j^{*}\left(t\right)=X_j\left(t\right)+ramp\left(t\right)$, for $t\in [T_{start},T_{stop}]$, where $ramp\left(t\right)=a_p·t$;
• Random attack: Here, a random value selected from a uniform distribution interval $(-a_p,a_p)$ is added: $X_j^{*}\left(t\right)=X_j\left(t\right)+random(-a_p,a_p)$, for $t\in [T_{start},T_{stop}]$;
• Step attack: This attack involves setting values to the attack parameter $a_p$ is added: $X_j^{*}\left(t\right)=a_p$, for $t\in [T_{start},T_{stop}]$.

Figure 3 displays a sample of the collected data, and the generated intervention data are illustrated in Figure 4. Interventions fulfill Definition 3 that states that the standard deviation of intervention data a is higher than the standard deviation of all defined sensitive behavior data s.

Table 2. Standard deviation for Vehicle speed collected data values and generated interventions/attacks.

Time-Series Data	Description	Standard Deviation ($\mathit{\sigma }$)
Regular behavior ($r_1$)	Vehicle usage under regular driving	0.0539
Regular behavior ($r_2$)	Vehicle usage under regular driving	0.0474
Regular behavior ($r_3$)	Vehicle usage under regular driving	0.0463
Regular behavior ($r_4$)	Vehicle usage under regular driving	0.0495
Regular behavior ($r_5$)—landmark ($X_0$)	Vehicle usage under regular driving	0.0500
Sensitive behavior ($s_1$)	Break pressed with a random intensity every 30 s	0.0376
Sensitive behavior ($s_2$)	Stop and go every 60 s	0.0598
Sensitive behavior ($s_3$)	Accelerate with a random intensity every 60 s	0.0557
Sensitive behavior ($s_4$)	Break and accelerate alternatively every 60 s	0.0678
Sensitive behavior ($s_5$)	Stop and go and accelerate alternatively every 60 s	0.0569
Intervention/attack ($a_1$)	Pulse attack (attack window size = 7, $a_p=20$)	0.0701
Intervention/attack ($a_2$)	Scaling attack (attack window size = 7, $a_p=2$)	0.0816
Intervention/attack ($a_3$)	Ramp attack (attack window size = 35, $a_p=0.8$)	0.0860
Intervention/attack ($a_4$)	Random attack (attack window size = 7, $a_p=50$)	0.0762
Intervention/attack ($a_5$)	Step attack (attack window size = 10, $a_p=70$)	0.0906

explains how sensitive behavior and attacks are generated and lists their corresponding standard deviations ($\sigma$).

4.2. Experiments

4.2.1. Compare Perturbation Methods

Auto vehicles, as well as many devices or industrial equipment, are enhanced with a large number of sensors. Only a fraction of the observed variable data collected from those sensors is sent to third-party processors. One or more collected or computed values are transmitted and must be protected.

To demonstrate the usage of the proposed approach, firstly consider a univariate time series consisting of the vehicle speed observed variable. According to the procedure described in Section 3.3, the regular and sensitive behavior time series are perturbed by applying selected distortion methods described in

Table 3. Distortion /perturbation methods and parameters.

Distortion/Perturbation Method	Notation	Number of Fourier Coefficients	Noise Size/Privacy Budget
No distortion/perturbation	${\mathcal{M}}^0$	-	-
Filtered FFT	${\mathcal{M}}^1$	$k=10$	-
Filtered FFT	${\mathcal{M}}^2$	$k=30$	-
CPA Algorithm	${\mathcal{M}}^3$	$k=50$	$discord=1.5$
CPA Algorithm	${\mathcal{M}}^4$	$k=65$	$discord=2$
FPA Algorithm	${\mathcal{M}}^5$	$k=50$	$ϵ=0.9$
FPA Algorithm	${\mathcal{M}}^6$	$k=65$	$ϵ=0.5$

. Figure 5 presents a sample of the perturbed data.

The perturbation is applied for all tested algorithms for each behavior data value, and the relative impact (Equation (5)) is computed. Finally, the process is repeated a significant number of times (e.g., larger than 100), and the mean relative impact is obtained (Equation (6)).

Table 4. Mean relative impact coefficients ${\overline{\mathcal{C}}}_b$ for tested perturbation methods applied on sensitive behavior data.

Sensitive Behavior	${\mathcal{M}}^0$	${\mathcal{M}}^1$	${\mathcal{M}}^2$	${\mathcal{M}}^3$	${\mathcal{M}}^4$	${\mathcal{M}}^5$	${\mathcal{M}}^6$
Break/30 s ($s_1$)	1.0	0.205	0.217	0.194	0.255	0.236	0.228
Stop/60 s ($s_2$)	1.0	0.266	0.205	0.353	0.202	0.235	0.259
Acceleration/60 s ($s_3$)	1.0	0.251	0.238	0.208	0.232	0.239	0.243
Break and acceleration
(alternatively)/60 s ($s_4$)	1.0	0.184	0.218	0.364	0.105	0.228	0.240
Stop and go and acceleration
(alternatively)/60 s ($s_5$)	1.0	0.257	0.256	0.334	0.216	0.245	0.249

summarizes the computed impact selected sensitive behavior has on various perturbation systems, and Figure 6a illustrates the minimum and maximum impact coefficients for all perturbation methods (highlighted in Table 4).

Furthermore, we investigate the proposed approach’s utility in the case of multiple observed variables data. We selected more data attributes (instant fuel consumption, CO${}_2$ flow, and magnetometer X) from the collected dataset, besides the already presented vehicle speed observed variable, and computed their mean relative impact coefficients (illustrated in Figure 7).

4.2.2. Evaluate the Utility of a Perturbation Module for Detecting Data Interventions/Attacks

Section 3.4 describes the procedure to evaluate the utility provided by data resulting from the considered perturbation methods in case of data intervention. Consider the integrity attacks presented in Table 2 and apply the proposed approach for computing the impact of attacks on normal behavior data.

Table 5. Mean relative impact coefficients ${\overline{\mathcal{C}}}_a$ for tested perturbation methods applied on intervention data (vehicle speed observed variable).

Intervention/Attack	${\mathcal{M}}^0$	${\mathcal{M}}^1$	${\mathcal{M}}^2$	${\mathcal{M}}^3$	${\mathcal{M}}^4$	${\mathcal{M}}^5$	${\mathcal{M}}^6$
Intervention/attack ($a_1$)	1.0	0.157	$0.222$	$0.249$	$0.254$	$0.244$	$0.240$
Intervention/attack ($a_2$)	1.0	0.140	$0.214$	$0.233$	$0.253$	$0.235$	$0.243$
Intervention/attack ($a_3$)	1.0	0.134	$0.209$	$0.231$	$0.254$	0.231	$0.237$
Intervention/attack ($a_4$)	1.0	0.138	$0.213$	$0.244$	0.263	$0.237$	$0.242$
Intervention/attack ($a_5$)	1.0	0.132	$0.208$	$0.229$	0.261	0.230	$0.237$
$min\left({\overline{\mathcal{C}}}_s\right)$	1.0	0.184	0.205	0.194	0.105	0.228	0.228
$max\left({\overline{\mathcal{C}}}_s\right)$	1.0	0.266	0.256	0.364	0.255	0.245	0.259

lists the mean relative impact coefficients (Equation (11)) for all integrity attacks and all tested perturbation methods. For all methods, $min\left({\overline{\mathcal{C}}}_s\right)$ and $max\left({\overline{\mathcal{C}}}_s\right)$ are extracted from Table 4.

5. Discussion

This paper proposes a comparison approach that identifies the minimum and maximum relative mean impact for each perturbation method. A suitable perturbation method is identified if it simultaneously holds the highest minimum and lowest maximum impact coefficients.

For one observed variable (vehicle speed), the computed minimum and maximum impact coefficients $\overline{\mathcal{C}}$, for all tested perturbation methods, are listed in Table 4 and illustrated in Figure 6a. As observed, the perturbation method ${\mathcal{M}}^5$ holds both the highest minimum impact ($\overline{\mathcal{C}}=0.228$) and the smallest maximum impact ($\overline{\mathcal{C}}=0.245$). Based on Propositions 1–3, we conclude that ${\mathcal{M}}^5$ provides the best privacy and utility from the tested algorithms, for the considered dataset, and the proposed sensitive behaviors. Perturbation ${\mathcal{M}}^6$ also provides good privacy and utility with impact coefficients close to those computed for ${\mathcal{M}}^5$ and it may be considered an alternative for ${\mathcal{M}}^5$.

The result is validated from the utility point of view by computing the mean absolute error (MAE) utility metric between the not perturbed sensitive behavior data and the perturbed version for each perturbation method (

Table 6. Mean absolute error (MAE) for tested perturbation methods ($\times {10}^{-2}$) applied on sensitive behavior data.

Sensitive Behavior	${\mathcal{M}}^0$	${\mathcal{M}}^1$	${\mathcal{M}}^2$	${\mathcal{M}}^3$	${\mathcal{M}}^4$	${\mathcal{M}}^5$	${\mathcal{M}}^6$
Break/30 s ($s_1$)	0.0	1.977	1.601	1.518	1.775	1.486	1.486
Stop/60 s ($s_2$)	0.0	3.977	2.832	2.668	2.650	2.568	2.577
Acceleration/60 s ($s_3$)	0.0	2.744	2.359	2.266	2.519	2.162	2.176
Break and acceleration
(alternatively)/60 s ($s_4$)	0.0	3.541	3.027	3.072	3.028	2.849	2.796
Stop and go and acceleration
(alternatively)/60 s ($s_5$)	0.0	3.218	2.454	2.197	2.276	2.273	2.248

). Figure 6b highlights perturbation methods ${\mathcal{M}}^5$ and ${\mathcal{M}}^6$ as the ones that provide the smallest information loss.

For measuring the privacy provided by the tested perturbation methods, we calculated the probability distribution function (PDF) of queries executed on perturbed data and computed the probability to obtain the actual query result from the original data (

Table 7. Probability of the real query result applied on sensitive behavior data.

Sensitive Behavior	${\mathcal{M}}^3$	${\mathcal{M}}^4$	${\mathcal{M}}^5$	${\mathcal{M}}^6$
Break/30 s ($s_1$)	$4.67\times {10}^{-11}$	$1.18\times {10}^{-8}$	$1.31\times {10}^{-112}$	$4.45\times {10}^{-64}$
Stop/60 s ($s_2$)	$2.05\times {10}^{-6}$	$3.83\times {10}^{-5}$	$3.03\times {10}^{-39}$	$4.91\times {10}^{-18}$
Acceleration/60 s ($s_3$)	$2.98\times {10}^{-10}$	$8.15\times {10}^{-9}$	$1.53\times {10}^{-122}$	$5.10\times {10}^{-57}$
Break and acceleration
(alternatively)/60 s ($s_4$)	$2.06\times {10}^{-9}$	$1.64\times {10}^{-8}$	$9.18\times {10}^{-78}$	$9.19\times {10}^{-37}$
Stop and go and
acceleration/60 s ($s_5$)	$2.83\times {10}^{-7}$	$8.87\times {10}^{-7}$	$4.43\times {10}^{-38}$	$4.30\times {10}^{-29}$

). Figure 6c shows the mean probability for all sensitive behaviors computed from PDFs generated from 1000 queries. The smaller the probability, the higher the privacy, as more information is hidden. Again, the computed probabilities emphasize the perturbation methods ${\mathcal{M}}^5$ and ${\mathcal{M}}^6$ as the ones providing the best privacy protection.

In addition, we tested the proposed framework for more observed variables with the objective of finding the best perturbation method that can be applied to all variables. Figure 7 illustrates the computed mean relative impact coefficients for attributes instant fuel consumption, CO${}_2$ flow, and magnetometer X. According to the stated requirements, the perturbation method ${\mathcal{M}}^6$ holds the highest minimum and lowest maximum impact coefficients for all variables, thus providing the best utility and privacy for the described scenario. Moreover, the result is confirmed from the utility and privacy points of view. Figure 8 shows that ${\mathcal{M}}^6$ has the minimum information loss (MAE) from all tested perturbation methods, and, moreover, the lowest maximum probability of the real query result (lower probability is better).

Further, we investigated the possibility of using the mean relative impact coefficients (Equation (11)) for detecting data integrity attacks. Based on Proposition 4, data from Table 5 indicate that all data protection methods hide important information about the attacks. Attack detection may be possible but challenging for most perturbation methods, as the impact is similar to those of sensitive behaviors. As the impact coefficients for method ${\mathcal{M}}^1$ are all smaller than the corresponding impact coefficients calculated for sensitive behavior, we conclude that ${\mathcal{M}}^1$ hides essential information about the attacks. Thus, ${\mathcal{M}}^1$ cannot be regarded as a suitable privacy protection method when attack or anomaly detection is an objective. In the case of ${\mathcal{M}}^4$, several attacks ($a_4$ and $a_5$) may be detected according to the proposed criteria, and the impact of the other attacks ($a_1$, $a_2$, and $a_3$) is also significant.

We demonstrated that the proposed methodology could be used to measure the utility and privacy of various perturbation algorithms, and the following advantages have been identified:

• Compared to the other mechanisms, the proposed approach measures both privacy and utility;
• Various distortion and perturbation methods can be compared, no matter how different they are;
• An evaluation of the impact of various data integrity attacks on perturbed data is possible.

However, a few observations on its limitations are necessary. Firstly, the accuracy of the evaluation depends on the set of collected normal behavior time series and the set of defined sensitive behaviors. The more accurately they cover the possible sensitive behaviors, the more accurate the comparison is.

Secondly, the experiments have shown that the proposed approach may only be able to identify a suitable perturbation method for some observed variables. For example, certain algorithms provide high privacy for some variables but lack utility or vice versa.

Additionally, when more observed variables are evaluated, it is possible to identify desired perturbation methods depending on the variable. This can be anticipated as the impact of sensitive behavior or intervention may not be the same on all variables. In this case, we propose an improvement to the perturbation system illustrated in Figure 1: Instead of using a perturbation method for all observed variables, add several perturbation methods and assign variables to the ones implementing the best privacy and utility (see Figure 9).

The approach proposed for comparing various perturbation methods on time-series data can be expanded to evaluate the utility of the perturbed data in the case of data interventions (e.g., integrity attacks). The computed impact coefficients show how much information about the intervention is hidden or preserved after perturbation. However, more research is necessary to test various attacks and anomaly detection techniques and assess their performance on perturbed data.

6. Conclusions

This paper addressed a new approach for measuring the privacy and utility provided by time-series perturbation algorithms. The main novelty and contribution to the state of the art is the exposed procedure for comparing various perturbation methods. The framework involved collecting data corresponding to sensitive behavior and measuring the impact this behavior had on the perturbation system. As shown, the presented metrics were helpful for simultaneously measuring the privacy and utility of the perturbed data. The research contribution is meaningful for those protecting time-series data collected from various sensors, as the approach is sensor-type-independent, algorithm-independent, and data-independent.

The experiments demonstrated that the approach had significant benefits. It could be applied to diverse perturbation algorithms and on various data, under the condition that sensitive behavior could be defined and corresponding data could be collected. Moreover, the research suggested evaluating the impact of integrity data attacks on perturbed data. Data was collected via the OBD-II port on a VW Passat auto vehicle for both regular/typical and sensitive behavior. The experiments showed that the approach was also promising in measuring the impact of sensitive behavior on the perturbed data regarding privacy and utility. Furthermore, by exemplifying the approach on two classical perturbation algorithms, we expect our method to be applied to other perturbation techniques.

In future work, we intend to test the proposed method on publicly available datasets and on more diverse perturbation algorithms. A key challenge will be the detection of sensitive user behavior on such data. As a result, further adjustments to the presented approach may be required. Lastly, additional evaluation of the impact of integrity attacks on perturbed data and, consequently, the impact on the accuracy of the anomaly and attack detection algorithms will be included in future research work.