*5.3. Attack Detection Models*

Many studies designed to detect attacks against water CPS use machine learningbased anomaly detection models, in which normal operational data is the primary (or sole) resource as there is often insufficient anomalous data to create models using supervised approaches. It is not readily possible to compare the performance of existing detection models, or to determine their generality or the reproducibility of their results. This is due both to a lack of datasets, leading to poor diversity in assumptions and plant models, and to a lack of common performance metrics. Where common datasets and performance metrics have been used, as in the case of, say, the SWaT and WADI datasets, reported results suggest that deep learning-based anomaly detection models perform better than conventional anomaly detection models. However, further studies are required to build confidence that such performance improvement is real.

As is usually the case with intrusion detection studies for CPS, the effectiveness of the proposed solutions were measured using conventional performance metrics, including accuracy, precision, recall, F-score, false positives and false negatives. These performance metrics were not designed for multivariate time-series datasets of CPS, in which anomalies usually occur in bursts [119]. Even when using these conventional performance metrics, some fail to report false positives and none of the studies reported detection latency, which is an important metric for critical systems [68] as early detection is critical for CPS.

Over the last decade, there has been an increase in number of CPS applying deep learning models to detect anomalous behaviour and datasets such as BATADAL, SWaT and WADi have contributed to some of these studies. However, studies from other fields have shown that machine learning-based approaches are rather vulnerable to accidental or intentional corruption of training data sets; thus, say, adversarial attacks can influence detection outcomes [120]. At the same time, there is a significant number of research studies that focus on improving the robustness of such models [121]. At present, however, such work is invariably targeted at other fields of study, most notably computer vision, and we are yet to understand the possible risks in the application of learning models to CPS.

The generation of attack or anomalous behaviour for testing detection models is often done manually. Typically, measurement values or control signals are modified, and performance data is collected both with and without these variations. However, such an approach assumes that the modifications are representative of those that will be experienced in reality, and this assumption is tenuous at best. Furthermore, over time, CPS actuators and sensors degrade as a result of ageing and become more prone to noise. As a result, normal behaviour is itself non-stationary and it will be necessary either to use richer training sets and models that capture temporal change, or to use online learning. The latter is again vulnerable to changes induced by an adversary that are intended to pervert the detection mechanism. There is therefore a pressing need to increase the attention paid to the practicalities associated with actual deployment, including the usability and maintainability of proposed detection models.

Identifying anomalous behaviour should ideally be followed by the raising of an alert that identifies the potential cause and so determines a strategy to be followed for mitigation. However, existing studies often stop at detection. Future work is therefore required to investigate approaches that identify the root cause of anomalous behaviour, locate compromised devices and respond and mitigate further damage in a timely manner.

#### *5.4. Collaboration with Industry*

Although several studies have consulted with engineers who have experience in dealing with water systems, we failed to identify any publications that were written by industry. There is currently a lack of collaborative work between industry and academia in this area. Securing water systems requires a multidisciplinary effort that involves both the designers and operators of these systems and academics working at the leading edge of technology to ensure that security research pushes the boundaries of the possible while remaining applicable and usable.
