Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Featured Application

The methodology presented in this paper is directly intended for application in the area of turbojet engines. This paper presents the design of an automated system which can create diagnostic information using data from an infrared thermal camera, utilizing methods of computational intelligence and expert systems, which can be considered as a novel approach. The developed system segments infrared images use a self-organizing feature map and an expert system to automatically extract diagnostic information about the technical state of a turbojet engine. The diagnostic system was developed using a small iSTC-21v turbojet engine with a thrust of 500 N in laboratory conditions and can be simply modified for application to other engine types. The system is not limited to engines and can be considered for application in other technical systems which produce heat, such as generators, transmissions, and electronic circuit boards.

Abstract

There are only a few applications of infrared thermal imaging in aviation. In the area of turbojet engines, infrared imaging has been used to detect temperature field anomalies in order to identify structural defects in the materials of engine casings or other engine parts. In aviation applications, the evaluation of infrared images is usually performed manually by an expert. This paper deals with the design of an automatic intelligent system which evaluates the technical state and diagnoses a turbojet engine during its operation based on infrared thermal (IRT) images. A hybrid system interconnecting a self-organizing feature map and an expert system is designed for this purpose. A Kohonen neural network (the self-organizing feature map) is successfully applied to segment IRT images of a turbojet engine with high precision, and the expert system is then used to create diagnostic information from the segmented images. This paper represents a proof of concept of this hybrid system using data from a small iSTC-21v turbojet engine operating in laboratory conditions.

1. Introduction

Infrared thermography (IRT) is one of the many diagnostic methodologies currently applied in various types of industry and research [1]. It can be used to detect so-called invisible defects in a non-destructive way. Compared to traditional areas of IRT applications, such as biomedical applications [2], energetics, buildings, and other constructions [3,4,5,6], the application of this methodology in aviation is still at an early stage [1].

The advantages of IRT have been fully demonstrated in pre-flight and post-flight aircraft checks, where damage (e.g., water in honeycomb structures of composite parts), is monitored by thermal imaging cameras [7,8]. In aviation, extensive research and development is being conducted on the IRT diagnostics of corrosion in aluminum materials [9,10], and especially composite structures [11,12,13]. In aircraft operation, it is possible to use IRT to detect air leaks inside the cabin [14], abnormal strain in individual parts of aircraft casings, fuselage, and wheel attachments [15], damage to electrical wiring and insulation [16,17], etc. In aviation, IRT is also used for rescue, firefighting, and surveillance operations [18,19]. Other applications of IRT diagnostics can be found in the area of unmanned vehicles, whose thermal imaging cameras are used to inspect power lines [20], pipelines [21], buildings [22], constructions [23], vegetation [24], etc. Thermography has also recently found an application in the design of stealth aircraft [25].

For the diagnostics of aviation engines, IRT is mainly used to check their composite covers [26]. Another area where IRT is applied, which is closer to the topic of this paper, is the diagnostics of turbojet engines; infrared images of the diagnosed engines’ gas turbines are compared to standard IRT images of engines without defects, which allows temperature field anomalies to be detected and thus allows defects to be identified [27]. These defects can be damage to the turbine blades, combustion chamber disturbance, disturbance to the homogeneous field of fuel burning, etc. [28].

Aircraft engine diagnostics and health monitoring [29,30,31] can be divided into two basic processes: ground diagnostics and on-board diagnostics with their associated systems. On-board, or online, diagnostics involve the checking of engine components and thermodynamic processes during the operation of the engine [32]. These components include individual engine parts, such as compressors, combustion chambers, and turbines and associated accessories, while thermodynamic process checks include reviewing parameters such as temperature, pressure, rotational turbo-compressor speed, and vibrations. Ground, or offline, diagnostics [33,34] involve tribo-technical analysis, non-destructive checking methods, and checks by thermal cameras as mentioned earlier.

This article aims to develop a progressive method for the online evaluation of the technical state of a turbojet engine using infrared thermal imaging. Contrary to traditional methods, in which a comparative analysis of thermal images is performed on some normalized samples, this paper proposes an intelligent adaptive method which can discriminate anomalous areas of thermal images and create resulting diagnostic information automatically without any human input or previous training.

Since the output of a thermal imaging camera is a digital image, computer vision techniques and methodologies should be used to automatically process this output. In order to extract meaningful information from a thermal image for a diagnostic system, a pattern recognition/classification approach has to be applied. Many methodologies and an array of methods can be used for this purpose [35,36]. These include traditional methodologies, which perform pattern recognition using deterministic image-processing methods such as filtering, feature analysis, and support vector machines. [37]. However, the most successful methodologies for image classification come from the area of computational intelligence. These include artificial neural networks, which have been widely and successfully applied in image processing, pattern recognition, and diagnostic tasks [38,39]. The most complex neural networks, such as convolutional neural networks, are able to classify any pattern in an image with high precision, as well as detect objects [40]. However, these approaches are computationally complex and require large amounts of data and training [41]. In order to create a successful diagnostic method for pattern recognition in IRT images, a simpler methodology should be used, since such images are more homogeneous than traditional images; they have fewer colors, lower resolution, and are well structured, with well-defined edges. The motivation for this study was to develop a methodology which does not require a large amount of computationally intensive training to successfully segment infrared images using intelligent adaptive algorithms.

Accordingly, in this paper, the field of neural networks with uncontrolled learning is explored. One of the methods that has been successfully used in pattern recognition and image segmentation is self-organizing feature maps (SOFMs), which comply with the demand for simplicity, uncontrolled learning, and good classification results. SOFMs can be transformed and formalized into the form of a Kohonen neural network [42,43]. Kohonen neural networks have been successfully applied in image segmentation tasks, the detection of specific areas in geographic maps [44,45,46], and the detection of cracks and corrosion in materials [47]. The results obtained from these applications represent a solid foundation and an inspiration for this approach to be applied in the segmentation of infrared images for the detection of anomalies in the temperature fields of different objects.

The results of pattern recognition and image segmentation come in the form of classes present in the image. In order to extract knowledge from this information, the classes have to be interpreted. This can be done by simply labeling the classes; however, in the field of technical systems, this may not be sufficient. In technical objects, some areas are naturally hot and some are naturally cold, and therefore there needs to be some inference system which interprets the displacement and size of the classes from thermal images, and generates diagnostic information, which in the simplest case can be the determination of whether the diagnosed object is in a normal or an abnormal operational state [48,49]. The traditional approach is to consider a diagnosed system as healthy if it operates within some specific limits; otherwise, a failure is indicated [48,49]. This is a natural approach, and can be used when dealing with a technical system where parameters which determine the technical state (temperature, vibrations, etc.) can be simply measured, such as a jet engine. However, to extract diagnostic information from a classified image, a more complex approach needs to be used [49,50]. Diagnostic expert systems are a natural candidate for extracting diagnostic information from complex data, and have been applied in the diagnostics of turbojet engines [49,50]. The aim of the present study is to create a specialized approach for the creation of a rule-base of an expert system which can transform class information in an infrared image into useful diagnostic information, thus creating hybrid architecture.

The functionality of this approach has been evaluated and designed specifically for application to turbojet engines, and the approach can be considered as novel compared to other intelligent approaches in turbojet engine diagnostics, which are applied to data measured by the engines’ sensors and do not deal with visual/infrared image information [51,52,53]. The aim of this paper is to explore the design of this new approach and prove its functionality by applying thermal vision-based diagnostics in an automated adaptive system in the area of turbojet engine diagnostics. The approach can be expanded and applied to almost any technical system which has been observed by an infrared camera. The hybrid architecture of our approach, which combines a Kohonen neural network and a specialized expert system with a designed rule-base, can be considered as a novel and original contribution of our research. A proof of concept is presented using real data from an iSTC-21v turbojet engine tested and operated in laboratory conditions.

2. Methodology

2.1. Using Infrared Imaging for Diagnostics

Every object whose temperature is above absolute zero radiates thermal energy. IRT systems measure this energy to create thermal infrared images. These images can be used to identify heat radiation anomalies of an object, which are usually characterized by increases or decreases in the radiated temperature [8,9,13]. The basic principle of IRT diagnostics is shown in Figure 1, with the following general meaning:

• Image: an object to be captured by a thermal camera.
• Classification: monitoring the object, scanning it, obtaining infrared data, classifying hot and cold parts as well as possible problematic areas.
• Diagnostic information: evaluating infrared thermal images, for example, identifying defects or deficiencies, deciding about the technical state of the engine.

The aforementioned principle is a standard principle when using human expertise to obtain diagnostic information from infrared images. Examples of applications in aviation are pre-flight and post-flight engine checks, where the engine’s casing is checked for defects and cracks caused by freezing water or material deficiencies. In such cases, the evaluation of thermal images and the detection of anomalies is strictly dependent on human expert knowledge [49].

When a human expert inspects infrared images of a technical object, he or she basically performs two operations. First, he/she discriminates different areas of the thermal image and classifies them into categories according to their temperature. They next use their expert knowledge to inspect hot or cold areas for some anomalies according to their position and temperature. The traditional approach is to superimpose the obtained infrared image over a standardized infrared image of an object without defects and evaluate the differences between the two images [54]. The whole process can therefore be formalized into two basic parts—image processing and classification, and knowledge acquisition. A variety of methods exist to solve both problems, as mentioned in the introduction, and are suitable to identify and evaluate anomalies. As previously stated, one aim of this study was to develop an autonomous diagnostic system which is able to learn and locate anomalies without any prior knowledge (e.g., normal faultless operation IRT images). The other aim was to make the system autonomous.

The general concept of the intelligent diagnostic system can be expressed by the scheme shown in Figure 2. The data acquisition part of the approach is usually done using the infrared camera, and the image pre-processing can be done before the data are fed into a classifier. The effects of the image pre-processing and filtering are not directly investigated in this study; however, the methodology is designed to be relatively robust in order to use image data of reasonable quality with low noise levels. Infrared cameras usually pre-process the image data in order to compensate for air humidity and other environmental effects. The robustness is achieved by the application of an intelligent adaptive algorithm, and the design therefore consists mainly of the classifier part by finding a methodology and selecting its structure, input/output relations, and the training algorithm in the case of the application of neural networks. Information about classes in the thermal image can be further used to evaluate the technical state of the investigated object (in this case, a jet engine), and a diagnostic expert system is a natural candidate for the completion of this task. The basis of the proposed methodology is therefore centered around:

• Image processing classification system: involving a SOFM.
• Knowledge acquisition system: an expert system for the evaluation of the technical state of an engine.

The other advantage of the designed system is that it does not use the absolute temperature data obtained from the camera, which are dependent on many environmental and material factors, but rather only uses image data, which describe relative temperatures and relations as well as the displacement of the temperature field of the investigated object.

2.2. Self-Organizing Feature Maps in Image Segmentation and Classification

SOFMs are characterized by their outstanding properties in clustering (classification), data visualization, probability density estimation, etc. Kohonen artificial neural networks are one of the most widely used SOFMs [42].

Kohonen artificial neural networks (KNN) map a high-dimensional input space/layer (x_i for i = 1, 2, ..., n, where n is the number of inputs) onto an output space/layer (N_j = 1, 2, ..., m, where m is the number of neurons) with fewer or an equal number of dimensions. KNN contains only a single layer of neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is optional and usually equals the number of target classes. The inputs to the network are interconnected with all neurons in the Kohonen layer. This means that every neuron has information from all the inputs. Synaptic weights leading to each neuron (designated as a weight vector, w_ij, of the neuron) can be understood as coordinates, which indicate the location of the neuron in the array (Figure 3). Neurons are mostly organized in a regular two-dimensional grid/matrix or one-dimensional row/vector array.

There are lateral links (local connections) between the neurons in the Kohonen layer (dashed line in Figure 3), which means that any particular neuron affects other neurons in its neighborhood.

In order to segment an IRT image, a layer that has $1\times m$ neurons (Figure 4) is proposed. Using this type of array, it is possible to segment thermal infrared images into m different segments, which can be labeled as different areas of the temperature field (cold, hot, etc.). Thus, if a higher number of neurons is used, the classification will be more detailed.

In the most simplified way, the self-organization process (Kohonen learning) consists of four main steps: initialization, competition, cooperation, and adaptation [55]:

• Initialization: Random values are set for all neuron weights. The input vector from the input space is selected.
• Competition: From the inputs, the discriminant function (Equation (1), below) of a particular neuron is computed. A Kohonen unit computes the Euclidean distance between an input x_i and its weight vector w_ij. The winning neuron is the one with the smallest value of the discriminant function.
• Cooperation: The winning neuron influences its adjacent neurons (i.e., the weight vectors of adjacent neurons are influenced), however this influence decreases with increasing distance to other neurons.
• Adaptation: The excited neurons decrease their values of the discriminant function in relation to the input pattern through adjustment of the associated weights. The response of the winning neuron to the subsequent application of a similar input pattern is enhanced. The process will stop if the maximum number of iterations is reached.

$$d_j={{\displaystyle \sum }}_{i=1}^m{\left(x_i-w_{ij}\right)}^2$$

(1)

The self-organization process is graphically depicted in Figure 5, where blue dots represent neurons, green dots represent input vectors, and red lines represent connections between neighboring neurons. The temperature field has five areas, which are indicated by five neurons. In this simplistic example, the labeling is performed by coloring the neurons and the corresponding training data, which activates the neurons. In this manner, the blue area after adaptation represents the coldest area in the temperature field and the red area the hottest area.

2.3. Expert Diagnostic Systems

An expert system consists of a knowledge base, an inference engine (solving or deducing), an input/output data interface (user interface or links to other systems and tools), a working memory, and an explaining or communication module in the form of a user interface. These elements are shown in Figure 6. A very important part of every expert system is its inference engine, which uses data from the knowledge base (knowledge gained from an expert) and specific problem-oriented data [56].

The inference engine and the knowledge base represent the core of every expert system. The inference engine provides the mathematical means to evaluate the rules in the knowledge base to reach a conclusion about the state of the diagnosed object [58,59,60]. In applications for turbojet engines, the knowledge base contains rules defining the technical state of the turbojet engine based on the outputs from the segmentation algorithm (i.e., the Kohonen neural network). The design of the diagnostic system interconnected with a SOFM, which was used in the present study, is shown in Figure 7.

In this diagnostic system, the knowledge is represented by IF->THEN rules, which can be written in a simple way as IF E THEN H, as follows:

$$E\to H$$

(2)

where E represents an observation (evidence) and H represents a hypothesis.

Rule-based systems differ from classical logic systems in their non-monotonic “thinking” and their ability to process uncertainties. In rule-based systems, inference is based on the rule of modus ponens (direct thinking) [61], which is described as:

$$\underset{\_}{\frac{E,E\to H}{H}}$$

(3)

This means that, if the E assumption is true and the rule implies E→H, then the conclusion H is true. In the case of an inference network, the conclusions are represented by facts, which correspond to assumptions of other rules. These inference networks are mostly used in domains where the number of possible solutions is limited (e.g., diagnostic problems or classification problems). The benefit of inference networks is that they can be easily implemented. The following model rules (Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]:

A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X

(4)

2.4. Experimental Setup for Design of the Infrared Imaging-Based Diagnostics System

In order to design an infrared imaging-based diagnostic system for use with turbojet engines and to construct a working knowledge base, a small iSTC-21v turbojet engine, developed in the Laboratory of Intelligent Control Systems of Aircraft Engines at the Faculty of Aeronautics, Technical University of Košice (Figure 9), was used [62]. This is a single-stream, single-shaft turbojet engine with a single-stage one-sided radial compressor, an annular combustion chamber, and a single-stage non-cooled gas turbine with a variable exhaust nozzle with full authority intelligent electronic digital control system developed from the TS-21 engine (Ljulka-Saturn, Rybinsk, Russia). A schematic of the measurement and data acquisition system is shown in Figure 10, and the arrangement of the camera is shown in Figure 11 [62].

A digital data acquisition system was used to observe the state of the engine [62]. Various sensors and data acquisition (DAQ) devices serve this purpose and send all the data to the control personal computer (PC). This PC contains a control system created in the LabVIEW (National Instruments Corporation, Austin, TX, USA) and Matlab/Simulink (MathWorks, Natick, MA, USA) software which is able to control and visualize the engine’s performance, save and track all measured parameters, and update the control algorithms.

For the design of the IRT diagnostics system, the following parameters are utilized:

• n (rpm)—speed of the engine’s shaft.
• T₂ (°C)—air temperature at the output of the radial compressor.
• T₃ (°C)—gas temperature at the input to the gas turbine.
• T₄ (°C)—gas temperature at the output of the gas turbine.

These parameters represent the inner state and temperature of the engine. The interior temperature was measured by a thermal sensor contained in the engine. The temperature of the engine’s casing was measured by a thermal camera; this temperature can be correlated with the inner temperature although it has different dynamics.

To measure the thermal temperature field around the engine, a FLIR Thermovision A40 thermal imaging camera was used (FLIR Systems, Wilsonville, OR, USA), and a stationary IR camera was used for scanning during long measurement processes, where continuous measurement in real time is necessary. The IR detector of the camera is a 320 × 240 focal plane array uncooled microbolometer. Its spectral range is 7.5 to 13 µm, its instantaneous field of view (IFOV) is 1.3 mrad, its noise equivalent temperature difference at 60 Hz is 0.08 °C at 30 °C, its measurement uncertainty is ±2 °C or ± 2%, its temperature sampling frequency is 60 Hz, and its observable temperature range is from −40 to 1500 °C. The camera was controlled via the PC using the ThermaCAM Researcher software (FLIR Systems, Wilsonville, OR, USA) through the FireWire interface. One of the settings when capturing a video is the choice of color palette, which is used to convert the levels of the infrared spectrum into the visible spectrum. This selection is mainly important when there are small temperature differences between parts of an object or its surroundings. In the case of an object with a wide range of temperatures, such as a turbojet engine, the standard iron-bow palette is a suitable option [63]. As the engine is a dynamic object, the displacement of its thermal field needs to be evaluated continuously. The camera allows uncompressed video to be captured at 25 frames per second in audio video interleave (AVI) format using the selected color palette. The video can then be decomposed into single frames in uncompressed bitmap (BMP) format. The resulting pictures contain IR images of the engine with the assigned RGB pixel values from the selected color palette during its operation. Full color resolution of the camera can be obtained only when taking static images of the engine, which is not an option for the application, but a video with this resolution would definitely improve the resulting IR images. It can be concluded here that parameters of the camera are sufficient for application on an object operating in a wide temperature range, where fine temperature resolution is not that important.

The thermal fields were captured and observed during the whole engine operational process, from its start-up to the time of full thrust, as shown by the measured basic parameters displayed in Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h before the measurement in order to allow the temperature of its sensor to equalize with that of the environment. Before the measurement, the values of the background air temperature, air humidity, and the emissivity of the engine surface were determined and inputted in the camera. The environment in the laboratory had a relatively stable humidity (around 50%) and temperature (around 21 °C) during the experiments with no direct sunlight. These parameters can have an impact on the precision of the IRT measurements, but can be quite well-controlled in a laboratory environment [39]. The measured state parameters (n (rpm), T₂ (°C), T₃ (°C), T₄ (°C)) for a single run of the iSTC-21v engine in laboratory conditions are shown in Figure 12.

3. Thermal Image Segmentation and Classification

3.1. Design and Training of the Kohonen Self-Organizing Feature Map

The Kohonen SOFM was used for the segmentation of the IRT images obtained with the thermal camera at a resolution of $320\times 240$ pixels as described in Section 2.4. Each pixel was defined as a five-dimensional vector P_{_i} (t) = {R,G,B,x,y}, where R, G, and B represent the values of the color spectrum {R,G,B} = <0,255> and x and y represent the coordinates of each pixel where x = <0,320>; y = <0,240>. The engine’s temperature as evaluated and processed by the camera was not taken into account, since it may contain errors created by the camera compensating for environmental conditions and assigning different temperatures to a single color, thus confusing the classifier. In order to evaluate the technical state of the engine, the relative temperature-field displacement was used as it is holds the most important diagnostic information. The dynamic range of the camera was set in the interval <0,500> °C and was fixed. These parameters define the design of the SOFM. The pilot testing showed that these multi-dimensional vectors should be classified into five classes. As proposed in Section 2.2, a feature map with a single row is able to acceptably segment the IRT image, with low demands of computational time and complexity, which was one of the design goals.

The classes are labeled as N_i and represented by different colors as follows:

• N₁: the hottest and most critical part of the IRT image, red.
• N₂: a hot and critical part of the IRT image, yellow.
• N₃: a hot and non-critical part of the IRT image, green.
• N₄: a low-temperature and non-critical part, cyan.
• N₅: the temperature of the surrounding objects (not a part of the diagnosed object), blue.

The structure of the network can be visualized as shown in Figure 13. The training parameters which were found to be efficient were set as defined in

Table 1. Training parameters.

Topology function	Grid
Distance computation	Direct
Initial neighborhood size	3
Initial learning rate	0.9
Training iterations	500

, and the standard Kohonen network learning rule was used [42,43].

A training dataset was prepared to train the network as follows. The engine was run for 120 s and thermal images were recorded at two frames per second, as the pilot experiments showed that this timing is sufficient since the dynamics of the temperature of the engine casing have a time constant of 1.2 s. The training sample therefore contained a set of 240 thermal images. The run of the engine, which was used to obtain the IRT images is shown in Figure 14. To demonstrate the ability of the trained Kohonen SOFM, five operational points are shown in Figure 14, where the network was tested if it produced sensible results. The first operational point, O_p1, corresponds to the lowest evaluated rotational speed of the engine 42,058 rpm, and the last, O_p5, corresponds to the highest rotational speed (i.e., 50,289 rpm).

The classifications of the five example images are shown in Figure 15a, and the created segment areas are shown in Figure 15b. Following expert examination of the infrared images and heuristic evaluation of the temperature field, it can be concluded that the Kohonen network works as expected and is able to sensibly assign the hottest and coldest parts of the object to each of the five classes/segments. In order to quantify the results, the number of activations of each neuron are summed, giving the total segment areas S₁, …, S₅ corresponding to each of the classes N₁, …, N₅. This can be formalized as follows:

$$S_i={{\displaystyle \sum }}_{j=1}^p{x}_j;x=\{0;1\}$$

(5)

where p is the number of activations of the i-th neuron.

In this way, the size of each class and the total area of each temperature segment can be obtained. As can be seen in Figure 16, the total area of the segments in the three hottest classes increased as the engine speed and temperature increased. As it is not possible to define the exact areas which correspond to each class (as in traditional image segmentation where, for example, a river area is clearly defined), a system for the objective evaluation of the trained classifier performance was designed, and is described in the next section.

3.2. Classification System Testing and Results

To validate the ability of the trained Kohonen network to segment the thermal image, a system for its evaluation was designed as follows. Five artificial spheres, each with an area of 150 pixels, were created, each of which fell into only one of the classes. As can be seen in Figure 17, the network was able to correctly classify these artificial anomalies in this particular case.

In order to properly evaluate the functionality of the classifier, five testing infrared images were divided into 60 segments. The artificially created spherical temperature field anomaly for each class was moved across all segments, giving a total of 300 testing images for each class and 1500 testing images in total. The division of the image into segments when the artificial anomaly was included is shown in Figure 18.

The results are summarized in the confusion matrix shown in

Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM).

		Actual Class
		N1	N2	N3	N4	N5
Predicted Class	N1	286	5	2	0	0
	N2	10	284	15	1	0
	N3	4	10	272	4	6
	N4	0	1	10	275	34
	N5	0	0	1	20	260
	Accuracy	95.3%	94.6%	90.6%	91.6%	86%
	Total Accuracy	91.8% (1377/1500)

; the first row represents the real class of the anomaly and the first column represents the class output by the network. The contingency table gives a clear picture of the network’s ability to correctly segment the infrared images.

The network achieved a very good accuracy for the first two classes, and only misclassified objects for the three hottest classes. The lowest obtained accuracy was for the classification of class N₅, which was the coldest class and mostly represented the surroundings of the engine, which are of a low temperature. This class also contained misclassified objects belonging to class N₄, which only represented relatively cold parts of the engine or its surroundings. The most important classes for the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, that is, N₁, N₂, and N₃. The results could probably be improved by using a larger training dataset or different Kohonen layer topologies, however they can nevertheless be evaluated as very successful for the purpose of turbojet engine diagnostics.

4. Expert Diagnostic System Design Based on Infrared Images

4.1. Design of the Expert System

As presented in Figure 15, areas of the segments are not absolutely representative of the engine’s technical state, as their changes are quite slow as the engine heats up or cools down and the internal thermal energy is transferred to its casing, which has its own heating and cooling dynamics. The main task of the expert system is to evaluate the areas of the segments and their changes, and to then present to the operator some information about the technical state of the engine.

To fulfill this task, the designed expert system needs as an input the total area of the segments and some indication of how they evolve over time. Temporal changes in the areas of the segments can be predicted by the computation of derivations/differences between two segmented images in a set time interval $\Delta t$. The standard derivative equation can be used for this purpose for the i-th neuron and the total area of the i-th segment S_i:

$$d{S}_i\left(t\right)=\frac{S_i\left(t+\Delta t\right)-S_i\left(t\right)}{\Delta t}$$

(6)

From a diagnostic point of view, it makes sense to investigate only the three hottest states (levels S₁–S₃) and their derivations S_{1_d}–S_{3_d}, as they represent the most critical temperatures for the engine. The time derivation of the surface areas of the segments allows the magnitude of the change in the area of different temperature regions to be determined and early structural failures to be detected. The knowledge base of the proposed expert diagnostic system consists of a set of rules that describe normal and atypical engine operation based on the outputs of the Kohonen SOFM. Each rule R_i (i = 1, 2, ..., 8) is designed according to precisely defined intervals of the allowed amount of pixels (area sizes) of individual color levels and intervals of allowed values of their derivations, based on results for the normal state and atypical states of the object. The outputs of the rules R_i take Boolean logical values R_i Є {0; 1}. The rules R_i in

Table 3. A partial set of rules in the knowledge base of the proposed expert diagnostic system.

IF	S1	IS	<13,200; 13,300>	AND	S1_d is <−5; 50 >	=> R1
IF	S2	IS	<3600; 3700>	AND	S2_d is <−10; 30>	=> R2
IF	S3	IS	<7300; 7420>	AND	S3_d is <−5 ; 45>	=> R3
IF	S3_d	IS	<46; 160 >	–	–	=> R4
IF	S3_d	IS	<−85; −6>	–	–	=> R5
IF	S2_d	IS	<31; 140>	–	–	=> R6
IF	S2_d	IS	<−120; −11>	–	–	=> R7
IF	S1_d	IS	<51; 160>	–	–	=> R8

are defined as follows:

IF S_i is the area size <from;to> AND S_{i_d} is the derivative <from;to> => R_i, where S_i (i = 1, 2, 3)

(7)

The rules R_i are partial conclusions about the technical state of the investigated engine. In order to obtain some knowledge about the engine’s technical state, the hypotheses which are to be connected with these rules have to be defined. In order to keep the rule base simple at first, diagnostic signals (hypotheses) for a turbojet engine are proposed in

Table 4. Technical states of the turbojet engine.

D0	Normal condition
D1	Light overheating
D2	Moderate overheating
D3	Critical overheating
D4	Light structural defect
D5	Critical structural defect

.

These diagnostic signals (D_i; i = 0, 1, 2, ..., 5) are activated by combinations of the proposed rules. The rules to obtain the diagnostic signals are defined in

Table 5. Inference of the diagnostic signals.

IF	R1	AND	R2	AND	R3	=> D0
IF	R1	AND	R2	AND	R4	=> D1
IF	R1	AND	R5	AND	R6	=> D2
IF	R3	AND	R7	AND	R8	=> D3
IF	R2	AND	R5	AND	R8	=> D4
IF	R5	AND	R7	AND	R8	=> D5

.

The rules, hypotheses, and inputs can be formalized in the form of an oriented graph. The inference network is defined in Section 2.3. In the case of a turbojet engine with diagnostics based on the sizes and derivatives of segmented thermal images, the inference network shown in Figure 19 is proposed based on rules in Table 3 and Table 4, which represent the knowledge base of the expert system. The inference engine then performs a bottom–up search from data of the segments’ sizes S_i and their derivatives S_{i_d} through intermediate rules R_j up to evaluation of the hypotheses D_k.

4.2. Pilot Testing of the Expert System Using Infrared Thermography (IRT) Inputs

The idea behind the design of the presented expert system is to evaluate its possibilities and its ability to correctly present diagnostic signals to the engine’s operator or supervisory control system [64,65]. The designed rule base was therefore constructed to be very simple in order to evaluate the five pre-defined temperature states represented by hypotheses D. Pilot tests were conducted to prove that the system works as intended. For real-world engine operation, it was only possible to test the states D₀, D₁, D₄, and D₅. The resulting classifications for these tested situations are shown in

Table 6. Testing of the expert system for the evaluation of hypotheses D₀, D₁, D₄, and D₅.

		D0—normal condition R1: S1 = 13,270 AND S1_d = 20 R2: S2 = 3628 AND S2_d = 14 R3: S3 = 7374, AND S3_d = 24 R1 AND R2 AND R3 => D0
		D1—light overheating R1: S1 = 13,227 AND S1_d = 31 R2: S2 = 3643 AND S2_d = −5 R4: S3_d = 112 R1 AND R2 AND R4 => D1
		D4—light structural defect R2: S2 = 3671 AND S2_d = 3 R5: S3_d = −67 R8: S1_d = 92 R2 AND R5 AND R8 => D4
		D5—critical structural defect R5: S3_d = −72 R7: S2_d = −45 R8: S1_d = 118 R5 AND R7 AND R8 => D5

. States D₄ and D₅ were artificially induced by quickly adding heating material on the surface of the engine’s exhaust nozzle, which created anomalies in the temperature field of the engine. The condition of light overheating was achieved when the engine was running at a high speed setting, (i.e., at an operating speed of 50,000 rpm for around 20 s). The rules which were evaluated as true, and the way the diagnostic signal/hypothesis was obtained, are shown in Table 6. These results suggest that the diagnostic expert system is able to determine basic states represented by hypotheses D₀ , D₁, D₄, D₅ and that the designed rules work.

Additionally, another test was set up using the data from a single engine run, as shown in Figure 14. Normal and light overheating states were tested during standard operation of the engine, and are shown in Figure 20. It can be seen that the expert system selected the state D₁ (light overheating) even though the speed of the engine and the corresponding temperatures had decreased. This was the correct selection, since the temperature of the casing takes more time to cool down, whereas the inner gas temperature as measured by internal sensors decreases immediately. This set of pilot tests showed that the inference engine works as intended and that the expert system is able to produce some sensible results in the evaluation of the technical state of the engine during normal conditions and to detect artificially induced structural defects. In order to prove the correctness of the setting of the rules R_i, as well as the definition of intervals in these rules, a more systematic approach has to be taken. This will be designed in a follow-up study of the system.

5. Conclusions

This article defines a way to use infrared imaging directly in a new area of aviation, namely, in the evaluation of the technical state of turbojet engines. A concept of a hybrid system combining a SOFM, namely a Kohonen neural network, and an expert system, is shown to be a promising approach to solve the problem of thermal image segmentation and to extract knowledge about the technical state of the engine from the obtained segmented IRT images. The Kohonen neural network defined in this article is a computationally very simple algorithm, and was able to achieve a precision of higher than 90% in the segmentation of infrared images obtained by the FLIR A40 camera when introducing an artificial temperature anomaly. By using the total areas of segmented infrared images corresponding to different temperatures, and their time derivatives, the technical state of the engine can be extracted and evaluated. This can be considered as a novel approach, and it has shown promising results as a proof of concept, which can be further developed and investigated, for example, to take into account environmental or high noise conditions or the impact of application of different image filtering techniques. Moreover, accuracy of the classification could also be increased by utilizing a camera with the ability to record a video with 10 bit color resolution instead of the reduced iron-bow palette for applications, where fine temperature resolution is important. On the other hand, the developed methodology with an expert system seems to be robust enough to work well with the reduced color spectrum and the engine is working in a wide temperature range. The rule base of the expert system proposed in this article can be further expanded to contain more hypotheses, and can also be developed to use fuzzy logic to evaluate the rules to improve the precision of the system. It can be considered as a progressive methodology in the application of computational intelligence and infrared thermography to the area of aviation. The methodology can also be applied in other fields of technical practice to evaluate the temperature fields of different technical objects, such as electronic circuit boards, transmissions, rotating machinery, generators, and electric engines operating in laboratory conditions. The system is designed to be objective, and does not need any input or examination of the infrared images by humans. It can also be integrated into a complex situational control algorithm of a turbojet engine in laboratory conditions, where outputs of the expert system can be used as inputs for the situational classifier and also to tune adaptive controllers of the engine [62,64,65].