Clustering of Handheld Thermal Camera Images in Volcanic Areas and Temperature Statistics

Abstract

Thermal camera use is becoming ever more widespread in volcanic and environmental research and monitoring activities. Depending on the scope of an investigation and on the type of thermal camera used, different software for thermal infrared (IR) images analysis is employed. The Osservatorio Vesuviano Sezione in Napoli of the Istituto Nazionale di Geofisica e Vulcanologia (INGV-OV) processes the images acquired during thermal monitoring activities acquired in the Neapolitan areas (Vesuvio, Ischia and Campi Flegrei) with different FLIR software that returns for each image, or for each selected area within the image, a series of parameters (maximum temperature, average temperature, standard deviation, etc.). An operator selects the area of interest and later “manually” inserts the relevant parameters in Excel sheets to generate graphs. Such a tedious, time- and resource-consuming procedure gave reason to implement a software able to automatically analyze sets of thermal images taken with a handheld thermal camera without any manual action. This paper describes the method and the software implemented to “automate” and refine the extrapolation process and the analysis of the relevant information. The employed method clusters thermal images by applying K-MEANS and DBSCAN techniques. After clustering a series of images, the software displays the necessary statistics to highlight possible fluctuations in temperature values. The software, “StaTistical Analysis clusteRed ThErmal Data” (STARTED), is already available. Although it has been developed mostly to support monitoring of the volcanoes in Campania, it is quite versatile and can be used for any activity that implies thermal data analysis. In this paper, we describe the workflow and the dataset used to develop the software, as well as the first result obtained from it.

1. Introduction

Permanent, handheld, unmanned aircraft system (UAS), plane-mounted, helicopter-mounted and satellite-sensor thermal cameras are increasingly employed in miscellaneous applications, amongst which are the study of active and quiescent volcanic areas, environmental monitoring, research and validation/calibration of satellite data [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29].

The study of temperature changes, especially in volcanic context, allows one to gain knowledge about volcanic behavior ([14] and reference therein). Thermal data, combined with geophysical and geochemical observations, may contribute to the detection of eruption precursors such as the change in fumarole temperature, the opening of new fractures and the different geochemical composition of the gases that rise to the surface [24].

The Osservatorio Vesuviano Sezione di Napoli of the Istituto Nazionale di Geofisica e Vulcanologia (INGV-OV) has the task of monitoring the volcanic areas (Ischia, Vesuvio and Campi Flegrei) in the Campania region (in the south of Italy; Figure 1) [30,31,32,33]. Ground temperature has been monitored and recorded since 2008 using handheld thermal cameras and since 2019 with the aid of UAVs. Currently, the acquired images are processed using FLIR software [34], which returns for each image, or for each manually selected area within an image, a series of parameters (average temperature, standard deviation, etc.), which are then “manually” inserted by an operator into Excel sheets to be analyzed.

This procedure is very time and resource consuming: since 2008 thermal monitoring of the Neapolitan volcanoes has produced a very high number of images (thousands for each area investigated): therefore, the need to implement a software able to automatically analyze sets of thermal images on its own without any manual action. Although there is a large bibliography on the use of thermal data for monitoring purposes, the release of groundbreaking software for data storage and analysis has only recently started to increase [19,35,36] and it is leading toward the application of machine learning techniques [37,38]. It is also worth mentioning the unavailability of free/commercial software that can automatically perform the previously mentioned operations.

The aim of this work is to present the software StaTistical Analysis clusteRed ThErmal Data (STARTED) [39] that we developed to automatically extract the desired information from a set of thermal images by means of clustering them. It analyzes one or more directories containing CSV files converted from FLIR thermal images, organized by date and area, and then creates the necessary statistics to highlight possible fluctuations in mean temperature values. The program has been written in Python and makes use of existing libraries and algorithms contained in the standard scikit-learn package. In particular, it uses the K-MEANS [40,41,42,43] algorithm to extract a given number of segments from the image and the DBSCAN [40,44,45] algorithm to extract clusters of near temperatures from the segment with the highest temperature.

The program has been tested on a subset of the thermal dataset acquired on Neapolitan volcanoes. Images were taken in a period circa between 2008 and 2017. As it has been written in Python it can run on any platform where a recent interpreter is available with the required standard packages and libraries (above all, scikit and matplotlib).

The software STARTED (Supplementary Materials) and the images analyzed can be found on the open-access repository Zenodo [39].

Figure 1. This image shows where the investigated volcanic areas are located in Italy. It also shows in detail both the three monitored areas (Ischia, Campi Flegrei and Vesuvio) and the specific points (in green—discrete IR thermal) where the monitoring is carried out. Finally, the image shows in red (Piz, Pan, Sf3a, Ps1a and Sv2) some specific areas referenced in the following paragraphs. Modified by [46].

Figure 1. This image shows where the investigated volcanic areas are located in Italy. It also shows in detail both the three monitored areas (Ischia, Campi Flegrei and Vesuvio) and the specific points (in green—discrete IR thermal) where the monitoring is carried out. Finally, the image shows in red (Piz, Pan, Sf3a, Ps1a and Sv2) some specific areas referenced in the following paragraphs. Modified by [46].

2. Materials and Methods

2.1. Handheld Thermal Images

Starting from 2008, the TTM group (Telecamere Termiche Mobili—Handheld Thermal Camera) at the Osservatorio Vesuviano performs monthly campaigns to acquire temperature data in a number of “stations” over the three Neapolitan volcanoes (Vesuvio, Campi Flegrei and Ischia); the map of these measurement stations is in Figure 1. Each station is in a different structural and volcanic context, but they are always linked to thermal anomalies related to the degassing structures of the three volcanoes: fumarolas, hot springs, resurgence of hot fluid, etc. The characteristic temperatures of the anomalies are therefore in a range under 100 °C, overlapping this value in a very few cases. This temperature range is mostly due to the interaction of hot fluid coming from the quiescent volcanoes with the upper water plant that acts as a temperature buffer on the hot fluid.

The relatively low value of the temperatures requires that data must be acquired during nightly hours to avoid the interference of solar irradiation. Thermal images are taken using a FLIR SC640 [21] camera with a resolution of 640 × 480 pixels and a temperature resolution of ±2 °C under 100 °C and ±2% over 100 °C. During each monthly campaign, for each station environmental data (air temperature and relative humidity) were first of all recorded with a thermo-hygrometer and then about 10 images of the thermal anomalies were shot. Most of the measures are taken at a distance ranging from 1 m to 5 m from the target; in some places (for example at Vesuvius and Ischia), such distances may also reach in the order of the kilometer. When possible, temperature was also recorded using a K thermocouple.

A typical example of an image shot during the campaigns is shown in Figure 2. Subsequently, an operator selected the best image of the shot set for each station: the one with a vision of the target not disturbed by gases or other external influences. For each of the selected images, a number of pieces of information were extracted through the use of the ThermaCam Research 2.10 software [34] (Figure 2): the maximum temperature, the average temperature with its standard deviation and, as the analysis requirement increased, the average temperature with the standard deviation of the hottest areas.

Selection of the hottest areas and recording of the extracted information is manually repeated for each of the acquisition stations and the results are inserted in an Excel sheet and then processed to deliver a graph of temperatures over time, as shown in Figure 3. Separating the hot zones in these images is an issue in itself: due to the nature and source of the thermal anomalies, their temperature is not so different from the environment (background) temperature. In case of hot lava, for example, the temperature difference between hot and cold areas is big enough to allow the use of a simple threshold at something around 100 °C to obtain a quite good separation. This is not the case of our thermal image as the hot zones are in the same temperature range of the surrounding environment and temperatures slowly decline and merge with cold ones, with the result that the use of a simple threshold is not effective at all. Moreover, the size and shape of thermal anomalies is heavily influenced by the external temperature, requiring a more sophisticated approach to the task.

2.2. Program Requirements and Operation

The described analysis method is, as said, very time consuming and, above all, lacks the possibility to analyze all the shot images as only one is selected for the task. We decided, then, to develop something to allow us to rapidly analyze the whole dataset of thermal images that has been accumulating over the years. As a starting point for the development, we selected a subset of the thermal images that had already been converted in the CSV format using the FLIR software and that were shot in the years between 2008 and 2017. In this way, we did not need to bother with the problem of converting the hideous FLIR thermal format into something readable.

Thermal images are made of many “pixels”, each containing an integrated temperature value. The described program has the ability to automate the identification and grouping processes of the pixels that have similar temperatures with subsequent extraction of their parameters. Segmentation and clustering were used as they are powerful methods of data analysis based on the idea of grouping similar elements in a set of data. The idea is then to apply clustering to the thermal images to split them into different regions (clusters) from which information is extracted and statistically analyzed.

The program performs a two-stage analysis of each image: during the first phase the K-MEANS [40,41,42,43] algorithm segments the thermal image in a given number of segments where each pixel has the minimum temperature distance to the center of the segment itself. The obtained segments are then used as the input for the DBSCAN [40,44,45] algorithm, which recognizes the elementary clusters and detects noise for each segment or only the hottest one. Afterward, the software stores all the cluster information in a text file. In this experimental implementation, once the clustering is done, the program processes the statistics and creates output images and graphics.

One or more directories holding a single or multiple CSV files serve as input to the software; the program supports both well-formed (using commas as field separators and dots as decimal separators) and hill-formed “Italian localized” CSV files using semicolons as field separators and commas as decimal separators. The names of the files must contain some extra metadata to identify the area of the image and the time of the shot. In Figure 4, there is shown how part of the temperature matrix that is contained in the CSV file appears (Figure 4).

Once the temperature matrix of the thermal image is loaded, the segmentation process takes place using a library within the OpenCV package (there is a similar routine in the scikit package that has proven to give the exact same result: its use has been implemented in a subsequent version of the program); a function of the mentioned library uses the K-MEANS [40,41,42,43] clustering technique that segments the image by executing an iterative algorithm until it reaches convergence: it separates the temperature distribution of the image in a given number of subintervals (segments) minimizing the difference between each point and the center of the interval itself. It is especially important to properly select the K parameter during this initial phase because it specifies the number of zones to return, or rather in how many regions the image has to be split (Figure 5).

The default setting is to separate the image into five segments, but the user can override the value of the parameter K. A higher value of K gives a greater precision in segment detection, requesting on the other hand a greater processing time while a lower value will give lower processing time and lower precision.

The segmentation phase splits the image into different areas, a sort of “macro-cluster” (segments) that needs yet another division process to find even multiple separated hot areas within the hot cluster: the final goal is to achieve clusters that are not linked between them. A second separation is then required to split eventual nonconnected areas that may be present in a segment (Figure 6).

The second step entails passing the hot segment detected via image segmentation to a function that applies DBSCAN clustering techniques (Figure 7). DBSCAN works by grouping together a number of “pieces” of a set when some “distance” between them is under a given threshold. In our case, the pieces of the set are the pixels of the thermal image (at least those contained in the hot segment) and, as distance, we used a simple Euclidean distance in the 3D space made by their Px and Py positions and their temperatures.

The DBSCAN algorithm is hence strongly dependent on a couple of parameters. ε is the most important one: it specifies the maximum distance between two pixels so that one can be considered close to the other (and belonging to the same cluster). If not set, the program will assume 20 as a default value. MinSamples specifies the number of pixels neighboring a point that will be considered a center point (if not specified the program will assume default value of 200).

When classifying the points inside a cluster, DBSCAN checks that a point has enough other adjacent points in a specified range. The points that do not satisfy the criteria to be included in a cluster (for example if a point does not have enough adjacent ones in a specified radius to be considered part of a cluster) are accounted as noise, hence ignored (it gets labeled as cluster −1).

The number and mean temperatures of the noise clusters may be reported in a separate file for each thermal image to avoid forgetting important features of the image that may get lost in the process. Anyway, we have developed a very simple algorithm to “recover” noise pixels as they occur. During the development of the STARTED (Supplementary Materials) program we also implemented another one focused on the analysis of larger thermal images obtained by drone flights. In such case, losing noise pixels may be a big issue as a single pixel may contain information for large areas. This algorithm simply calculates the “distance” (in the same Euclidean fashion seen before) between the noise pixels and all the identified clusters and then assigning them to the “nearest” cluster. This algorithm is not yet included in the current version of STARTED, but it can be very easily added to the workflow of the program. It is worth noting that, in any case, the occurrence of noise pixels can be completely avoided with a correct choice of the algorithm parameters.

An example of cluster identification is shown in Figure 7: here the segments identified in Figure 6 have been clusterized and each cluster is represented via a random color. Black dots represent noise pixels.

Once clusters are identified in the hot segment, four parameters are extracted or calculated for each of them: Tmin and Tmax (minimum and maximum temperature of the cluster) and Tmed with σTmed (average temperature with its standard deviation). This information is saved in a text file at the end of the process. The name of the file is created with an abbreviation of the name of the area where the images were taken: this abbreviation was in fact contained in the name of the input CSV files to easily identify them; each of these files contain a row for each clusterized image in the area (Figure 8). Cluster information are saved in the one row per input file with the following syntax:

YYYYMMDD NC [NP1 T1_min T1_max T1_med σT1_med] [NP2 T2_min T2_max T2_med σT2_med] [……]

where the format is as follows: YYYYMMDD is the date of the image acquisition (obtained from the input file name, by now) and NC is the number of clusters identified in the hot segment. For each of the clusters, the quintuplet NPn, Tnmin, Tnmax, Tnmed and σTnmed is stored in the following columns where NPn is the number of pixels of the nth cluster and the other are the four temperature parameters of said cluster described before. The clusters are listed in descending average temperature in each row.

The creation of these files may be considered the main output of the program: further statistical analysis may be deferred to more dedicated tools. In any case, as a bonus action, after detecting the clusters and saving the information of the hot clusters for each image, the software may process overall statistics to create some simple graphic showing the variation of some of the identified parameters over time.

3. Results

The program has been tested over a subset of 1103 CSV files extracted from the main thermal dataset of the TTM group. We have taken a single image for each campaign on the 31 stations in the period from July 2008 to December 2017. Here we present the results as time series of the weighted average temperatures of the first three hot clusters of two of the measurement stations: Sf3a at La Solfatara crater and Ps1a in the Pisciarelli area. An example of the thermal image appearance of these two stations has been provided in Figure 9 and Figure 10. Figure 11 and Figure 12 represent the time series of temperature averages of these two areas, while Figure 13 and Figure 14 represent the trend of the size in pixels of the three hottest clusters in the same measurement stations.

The program has been tested on a Dell workstation Precision 7820 with 32GB of RAM and a double Xeon Bronze 3106 CPU at 1.7GHz running Debian GNU/Linux 10.12 (codename “Buster”). The Python interpreter was the default 3.7.3 version shipped with Debian 10, and all the used libraries (Sklearn, OpenCV and Matplotlib) were the default ones installed with the operating system. Depending on the load of the machine, the program required from 2 to 4 h to load and analyze the 1103 CSV files of the test dataset and write the results. In all these tests, the program was instructed to also output images of the intermediate segmentation and clustering stages seen, for example, in Figure 4, Figure 6 and Figure 7. All the images and graphics were created using the standard matplotlib library of the Python interpreter.

4. Discussion and Conclusions

Portions having consistent temperatures are typical across thermal images of volcanic areas; out of these portions, parameters must be extracted and used for studying and characterizing thermally active volcanic features and detecting thermal changes in volcanic areas. These changes could be precursors of volcanic unrest and could be useful to forecast volcanic eruptions. FLIR “ThermaCAM Researcher Pro 2.10” software is usually used to extract data and it requires extensive manual analysis of the single images at great expense of time and work.

To solve this problem, a program has been developed to automate the identification and the grouping of pixels with similar temperatures values in a thermal image. It extracts parameters that speed up and ease the entire process of forecasting volcanic eruptions. In a nutshell, this program allows a user to automatically cluster large datasets of thermal images and, based on the information extracted, it processes statistics on the trends of temperatures and cluster sizes.

Open-source Python libraries have been used to develop the program. Specifically, K-MEANS is used to segment the image and, afterward, DBSCAN is used to recognize the elementary clusters and the noise pixels. Finally, the program saves the information of all the clusters (maximum and minimum temperature, average temperature of the cluster with its standard deviation) in a text file. Once the clustering is done, the desired statistics are processed.

The program has been developed mostly having in mind the specific task of the analysis of handheld thermal camera images. First runs over a subset of the image database has proven a nice identification of high temperature areas and their separation from the background “cool” pixels. It is quite simple to use and modify to address specific issues in other fields requiring thermal images analysis, as for example environmental pollution or building thermal check-ups.

Currently, its main limit is that it has to be fed with CSV matrices manually obtained from the FLIR thermal images. We are already planning, as the main improvement for the software, to implement the direct loading of thermal data from FLIR images using open-source libraries. Unfortunately, this may actually only be achieved using external Perl libraries (ExifTools, in particular) that are not yet available in Python. Once this improvement is implemented, it will be possible to analyze the whole thermal dataset in a single pass, gathering information directly from the data source, and to easily obtain statistically significant data when performing new thermal campaigns. We also plan to analyze all the images shot during a given campaign for a given station and not only the “best” one as done here. Another important modification to the current implementation will be borrowed by another program we are implementing for the analysis of larger thermal images obtained by drones: it will allow the program to “recover” possibly occurring noise pixels by stitching them to the “nearest” cluster (near in space and temperature).

Anyway, to make effective use of these enhancements we must rearrange the whole dataset in a way to easily access the correct metadata (shot area, above all: date and hour are contained inside the EXIF metadata of the FLIR image) of the raw thermal image. We are confident that we can obtain a modified version of the program with the dataset correctly metadated during the next year.