Next Article in Journal
Partition Behavior in Aqueous Two-Phase System and Antioxidant Activity of Flavonoids from Ginkgo biloba
Next Article in Special Issue
Classification of Heart Sound Signal Using Multiple Features
Previous Article in Journal
Influence of Interfacial Intermetallic Growth on the Mechanical Properties of Sn-37Pb Solder Joints under Extreme Temperature Thermal Shock
Previous Article in Special Issue
Thin Cap Fibroatheroma Detection in Virtual Histology Images Using Geometric and Texture Features
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Detection of Ventricular Fibrillation Using the Image from Time-Frequency Representation and Combined Classifiers without Feature Extraction

by
Azeddine Mjahad
1,†,
Alfredo Rosado-Muñoz
1,*,†,
Juan F. Guerrero-Martínez
1,†,
Manuel Bataller-Mompeán
1,†,
Jose V. Francés-Villora
1,† and
Malay Kishore Dutta
2,†
1
GDDP, Department Electronic Engineering, School of Engineering, Universitat de Valencia, 46100 Burjassot, Valencia, Spain
2
Centre for Advanced Studies, Dr. A.P.J. Abdul Kalam Technical University, Sec-11, Jankipuram, Vistar Yojna, Lucknow, Uttar Pradesh 226031, India
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Appl. Sci. 2018, 8(11), 2057; https://doi.org/10.3390/app8112057
Submission received: 19 September 2018 / Revised: 18 October 2018 / Accepted: 19 October 2018 / Published: 25 October 2018
(This article belongs to the Special Issue Deep Learning and Big Data in Healthcare)

Abstract

:
Due the fact that the required therapy to treat Ventricular Fibrillation ( V F ) is aggressive (electric shock), the lack of a proper detection and recovering therapy could cause serious injuries to the patient or trigger a ventricular fibrillation, or even death. This work describes the development of an automatic diagnostic system for the detection of the occurrence of V F in real time by means of the time-frequency representation ( T F R ) image of the ECG. The main novelties are the use of the T F R image as input for a classification process, as well as the use of combined classifiers. The feature extraction stage is eliminated and, together with the use of specialized binary classifiers, this method improves the results of the classification. To verify the validity of the method, four different classifiers in different combinations are used: Regression Logistic with L2 Regularization ( L 2 R L R ), adaptive neural network ( A N N C ), Bagging ( B A G G ), and K-nearest neighbor ( K N N ). The Hierarchical Method (HM) and Voting Majority Method (VMM) combinations are used. ECG signals used for evaluation were obtained from the standard MIT-BIH and AHA databases. When the classifiers were combined, it was observed that the combination of B A G G , K N N , and A N N C using the Hierarchical Method (HM) gave the best results, with a sensitivity of 95.58 ± 0.41%, a 99.31 ± 0.08% specificity, a 98.6 ± 0.04% of overall accuracy, and a precision of 98.25 ± 0.29% for V F . Whereas a sensitivity of 94.02 ± 0.58%, a specificity of 99.31 ± 0.08%, an overall accuracy of 99.14 ± 0.43%, and a precision of 98.59 ± 0.09% was obtained for V T with a run time between 0.07 s and 0.12 s. Results show that the use of T F R image data to feed the combined classifiers yields a reduction in execution time with performance values above to those obtained by individual classifiers. This is of special utility for V F detection in real time.

Graphical Abstract

1. Introduction

The most common causes of sudden death are cardiovascular diseases, which are among the leading causes of death worldwide. One of the cardiovascular diseases with the highest mortality is Ventricular Fibrillation ( V F ), which is a cardiac arrhythmia condition produced by a disorganized electrical activity in the ventricles. During V F , the ventricles contract with an absence of an effective beat causing a pumping failure which could lead to a sudden death if the patient is not adequately treated within a few minutes. Defibrillation is the only definitive treatment for V F . It consists of applying a high voltage electric shock on the patient’s chest, facilitating the restart of a normal electrical cardiac activity [1,2,3]. However, the success of defibrillation is inversely proportional to the interval of time lapsed from the beginning of the episode to the application of the discharge.
There are many difficulties in diagnosing V F : On the one hand, the intrinsic characteristics of the V F signal (lack of organization, irregularity, etc.) and, on the other hand, the great similarity between VF and other cardiac pathologies such as ventricular tachycardia ( V T ) [4], especially in early stages of V F . The differentiation between V T and V F is quite complex: The wrongful diagnosis of V F for a patient that really suffers of V T can cause serious complications at the time of applying the therapy corresponding to V F (high voltage electrical discharge), as it may cause V F to the patient. On the contrary, if V F is incorrectly interpreted as V T or any other cardiac rhythm, the result can also be dangerous for the patient’s life since the treatment would imply receiving less voltage than the appropriate level. Thus, an effective detection method for distinguishing V F from V T is critical in clinical research.
The electrocardiogram (ECG) is a non-invasive, low-cost examination tool that has been used as the basic method of diagnosing cardiac conduction disorders by studying the heart rate and morphology of different waves that constitute the cardiac cycle. ECG analysis is a good source of information from which different types of heart disease can be detected. Due to the fact that the ECG signal is a non-stationary random signal, the time domain analysis does not prove to be sufficiently sensitive to the distortions of the ECG waveforms. However, these methods do not always show all the information that can be extracted from the ECG signals [5,6], thus losing information on the frequency domain which shows additional information on the signal.
Diagnosis in the frequency domain [7] uses methods such as the Fourier transform. Therefore, the analysis in the frequency domain allows to determine the frequencies of the signal. On the other hand, the temporary-type information of the signal is lost, which is a very limited method and is not useful for the analysis of non-stationary signals. Several studies have used mathematical models that combine temporal and spectral information in the same representation. This technique of Time-Frequency Representation ( T F R ) is very important in the treatment of non-stationary signals such as the ECG signal, as it distributes the energy of the signal in a two-dimensional time-frequency space [8,9]. In addition, multiple factors might alter the acquisition and recording of the ECG signal: The influence of the environment, 50–60 Hz mains interference, variations of the base line of low-frequency interference in the range of 0 Hz to 0.5 Hz [10,11]. On the other hand, there are disturbances of physiological origin such as those of electromyography (EMG). ECG noise reduction has been one of the main fields of research in the last decades since an adequate noise reduction allows a good pre-processing of the signal, extracting the maximum amount of information possible and eliminating ECG signal contamination from other sources.
Usually, after the initial processing of the signal, several algorithms are applied to obtain characteristics, features, or parameters which are supposed to offer a difference in value depending on the pathology. Typically, these parameters can be redundant or remove relevant information, being necessary to apply different techniques to select the most adequate. After optimisation, selected parameters are intended to serve as input to a classifier responsible for separating classes (associated to a pathology or type of rhythm, in this case), i.e. identified signal types.
In order to improve the performance of individual classifiers, the combination of classifiers (multiclassifiers) can improve the performance in separating classes. It is based on constructing a global classifier built from a set of classifiers that can provide interesting information on the representation of data compared to the results achieved using individual classifiers. There are many examples in the literature that have used the combination of classifiers focused towards the field of bioinformatics and biomedical research, geophysical analysis and remote sensing, among others. Out of the most frequently used multi classifiers, Random Forests [12], Bagging [8], Boosting [13], or Random Subspaces are the most commonly employed multiclassifiers. In the case of Random Subspaces, different subsets of attributes are used to train each individual classifier. The Bagging type variety comes from using different subsets of instances to train each individual classifier. Random Forests is a substantial modification of bagging that uses Random Trees as individual classifiers. The Boosting type iteratively trains the individual classifiers, therefore, it modifies the weights of the instances that will use the next individual classifier. There are other methods such as cascading [14], Stacking [15], and Grading [16].
Other examples using a combination of classifiers for ECG signal analysis can be found in the literature as a multiple classifier system [17], a genetic ensembles of classifiers [18], or a classification approach that uses majority voting optimized by the taguchi method [19]. In some cases, a majority voting [20,21], or a combined stacking technique [22]. Other combinations are also applied, e.g., an application of the decision tree to integrate the results of a set of individual neural classifiers (MLP, TSK, and the SVM) working in parallel [23] or a majority voter determining the P-wave absence over seven beats [24].
This work proposes a new strategy for the detection of V F whose steps are the initial processing of the signal and obtaining its time-frequency representation ( T F R ) with its equivalent image ( T F R I ). The T F R or T F R I (both cases will be analysed) is directly entered into an individual classifier or combined, without calculating parameters or extracting features since time-frequency representation contains both temporal and spectral information from the ECG signal, allowing the classifier to have enough information for the detection of different types of cardiac pathologies in real time. Since the ECG is a temporal signal, it is not common to find works converting the temporal signal into an image and further analyse the image, some works used some geometrical features from the ECG in combination with other features entering the classification stage. Other works also extract features from a time-frequency or discrete wavelet transform, but they do not use it as an image.
In order to reach the objectives sought, the present work is structured as follows: Section 2 describes the materials and methods, followed by Section 3 which details the initial processing applied to the ECG signal. Section 4 shows the extraction of information. Section 5 presents the individual and combined classification algorithms. Section 6 shows the standard statistical indexes, and finally, Section 7 shows obtained results for individual and combined classifiers, and Section 8 and Section 9 give a comparison of results with other authors and conclusions, respectively.

2. Materials and Methods

Records of the ECG signals have been taken from the standard MIT-BIH Malignant Ventricular Arrhythmia Database [25,26] and AHA (2000 series) [27], generating both the training and the test sets from them. In total, 24 continuous monitoring records (22 MIT-BIH records plus two additional AHA records) were used, with a sampling frequency of 125 Hz. All records have cardiac events already labeled. The additional AHA records aim to increase the number of Ventricular Tachycardia ( V T ) episodes to improve the balance between recorded time of V T and V F episodes. With the episodes labeled, four groups (classes) of signals were created: Ventricular Fibrillation ( V F ) corresponds to the class V F represents all the sections of registers in which there has been ventricular fibrillation and ventricular flutter. ECG signals with the presence of ventricular tachycardia were assigned to the class V T , which, in many cases, appear as a prior stage to ventricular fibrillation (sometimes, V T sections have V F -like morphologies). Normal rhythms were assigned to the N o r m a l class that constitutes the segments labeled with sinus rhythm. Finally, the rest of signal types not labelled as the previous classes (other arrhythmias, noise, etc.) have also been considered and assigned to the class O t h e r s . In total, 20,040 s were generated for all ECG signal registers, 3600 s corresponded to the class V F , 1380 s to V T , 10,860 s to N o r m a l , and 4200 s to O t h e r s .

3. ECG Signal Processing

If we look for a classifier that can obtain a satisfactory result of detecting V F and its differentiation from V T , it is necessary that the data provided as input to this classifier are properly treated. For this reason, different stages of data conditioning are performed to both the temporal signal of ECG and its time-frequency representation T F R . Figure 1 shows the general scheme of the followed methodology, from the reading of the records of the database to the results obtained by the classifier.
The developed methodology is composed of three fundamental phases.
  • First phase: Data filtering in order to reduce the baseline that affects the ECG. Once filtered, obtain the Window Reference Mark ( W R M ) of the ECG signal. Each W R M indicates the beginning of a time window ( t w ) of the ECG signal.
  • Second phase: Extraction of information through the implementation of the Hilbert transform to each window t w obtained in the first phase, then, assesment of the T F R matrix using the Pseudo Wigner-Ville, and the Time-Frequency Representation Image ( T F R I ).
  • Third phase: The classification phase is carried out considering both the individual and combined classifiers used. In this phase, the previously obtained T F R I matrices are used as input. The success in the detection of V F depends on the processing of the signal and the structure of the classifiers used. In order to better adapt to the data, we must adjust the parameters of the classifier to obtain the best performance.

3.1. Reduction of Baseline Oscillations

The first step in the processing of the ECG signal is to use a baseline filter, reducing the variation of the baseline and thus obtaining a better quality and definition of the temporal signal that will result in better characteristics provided by the T F R . This processing consists of the implementation of an 8th order infinite impulse response filter (IIR) with a Butterworth bandpass type ranging from 1 Hz to 45 Hz [28,29]. Figure 2 represents the effect of applying this bandpass filter, showing a reduction of the baseline. Thus, all signal contribution not located in the mentioned frequency range, which corresponds to non-ECG source, are eliminated.

3.2. Reference Marks

Next, it is necessary to obtain a Window Reference Mark ( W R M ) to indicate the beginning of the t w ECG time window. Following [30], a value from 50 to 120 beats per minute (bpm) can be considered as a normal heart rate range, and thus, the minimum ( W R M m i n ) and the maximum ( W R M m a x ) distances between two consecutive W R M is 0.5 s and 1.2 s, respectively. Accordingly, these values were used in our analysis. The calculation of W R M reference marks was obtained by an already developed algorithm [8], where N L M C is the number of local maxima L M marks existing in the signal. From each previously generated W R M reference mark, a time window t w of 1.2 s in length (150 samples) was generated, starting at the corresponding W R M mark, t w = [ W R M , W R M + 1 . 2 s] as shown in (Equation (1)).
t w j = [ W R M j , W R M j + 1 . 2 s ] ; j = 1 , . . . , N L M C

4. Extraction of Information

For each window t w , the Hilbert transform (Ht) is calculated first and then T F R of the PWV (Pseudo Wigner-Ville) type is calculated. Once the T F R is obtained, the contributions of frequency over 45 Hz are canceled, thereby eliminating both the network interference (50 Hz or 60 Hz) as well as the electromyogram (EMG). After this process, a Data Matrix ( D M ) obtained from the T F R , being 45 × 150 in size is obtained. This is useful since the signal of interest is found in values below 45 Hz (see Figure 3a,b).
Once D M is obtained from the T F R for each t w window, this data matrix T F R is converted into an image T F R I L f × L t with size L f × L t pixels being L f = 45 and L t = 150 converting the energy levels of the signal into a pixel intensity range from 0 to 255. These values correspond to different levels of grey in an image as they are shown in Figure 3c,d. Each T F R I image is then stored in a Data Matrix ( D M ) of size 45 × 150 .
Once D M data matrix is obtained, the data matrix is directly fed into the classifier. By doing this, all ECG signal information in the temporal and spectral domains is contained in the data matrix, providing the classifier with maximal data information. Note that this method requires a large number of inputs to the classifiers ( 45 × 150 = 6750 ) since each D M i j data in a matrix coordinate corresponds to an input. It is important to note that there is no feature extraction from the data matrix D M as it contains the temporal and spectral information from the ECG signal.

5. Classification Algorithms

In this work, several classifiers are used to evaluate the efficiency of the V F detection algorithm. All D M data were separated into two subsets: One for training and one for test. The training subset is used so that the algorithms learn to discriminate among the various types of defined classes ( V F , V T , N o r m a l , and O t h e r s ). As soon as each classifier concludes its training, it generates a prediction function that is later used to evaluate new data. Each of the classifiers used in this work have parameters that must be optimized for the purpose of obtaining the maximum yield. The tuning of the classifiers is done on the basis of final classification performance.
Some algorithms propose the use of four classes [8,31]. However, we can combine different binary algorithms for two-class separation so that they can provide important complementary information about the representation of the data.

Combination Topologies

The parallel topology method is the most frequent in combination of classifiers. All the classifiers are run in parallel using the same input data, and the results achieved by all the classifiers (classifier1- C l a , classifier2- C l b , classifier3- C l c ) generate a multiclassifier result ( C l a _ C l b _ C l c ) that are combined with the objective of obtaining an appropriate decision using a combination rule, e.g., the voting method [32], as shown in Figure 4 which is called Voting Majority Method (VMM). For real time execution, this methodology has a high execution time since all classification algorithms must be executed to make the final decision. The voting method works in the same way as the humans when voting in political elections. In other words, depending on the number of votes reached in favor of each class, it is assigned to the one that obtains the majority.
In a hierarchical topology, parallel and cascaded topologies are combined (Figure 5). By joining these two approaches, better results than those achieved by using individual classifiers can be obtained. The first classifier ( C l a ) generates a binary output for signal to be V F V T or N o r m a l O t h e r s . Then, two specific classifiers C l b and C l c generate a new binary classification V F , V T for the signal classified as V F V T by C l a , and O t h e r s , N o r m a l for the signal classified as N o r m a l O t h e r s , respectively. By joining the classifiers, a multi-class cascaded algorithm is generated.

6. Performance Assessment

To evaluate the performance of the classifier, we use standard statistical indexes such as Sensitivity (Sens), Specificity (Spe), Accuracy (Acc), and Precision (Pre), as shown by Equations (2)–(5), where TP are the True Positives, FN the False Negatives, TN the True Negatives, and FP the False Positives [33].
S e n s i t i v i t y ( % ) = T P T P + F N × 100
S p e c i f i c i t y ( % ) = T N T N + F P × 100
A c c u r a c y ( % ) = T P + T N T P + F N + T N + F P × 100
P r e c i s i o n ( % ) = T P T P + F P × 100
The calculation of the value of the global specificity, accuracy and Precision of one of the types ( V F , V T , N o r m a l , O t h e r s ) is obtained by using the specificity, accuracy, and Precision of this type of pathology before the sum of the remaining pathology types. The execution time of each of the tests performed was measured using a Fujitsu AH544 (Tokyo, Japan) laptop computer with an Intel (R) Core (TM) [email protected] GHz processor with 8GB RAM, 64-bit operating system using Matlab (R).
For completion, the Receiver Operating Characteristic (ROC) curve and Area Under the Curve (AUC) values are also calculated.

7. Results

In total, 28,507 windows were generated for all M D obtained from the corresponding T F R I : 5309 corresponded to the class V F , 1987 to V T , 15,160 to N o r m a l , and 6051 to O t h e r s . For each class, 67% of the data were used for training, and the rest for testing. This approach is repeated by making a 5-fold cross validation: Individual and combined classifier algorithms are assessed by taking the average of these 5 iterations. A 5-fold validation was chosen amongst different z-fold possibilities after some trials, with 5-fold cross validation obtaining the lowest generalization error, thus minimizing the structural risk of classifiers. For 5-fold cross validation, each class was divided into five datasets, equal in size; four dataset were used for training and one for testing. After five iterations, all datasets served for training and testing, obtaining a more balanced result.
Different analyses were done, the first test is based on different types of individual classification algorithms, the second uses combined classification algorithms.

7.1. Results for Individual Classifiers

In this first test, individual classifiers are used. The results are obtained using four different classification algorithms: L 2 R L R , A N N C , B A G G , and K N N [8,31]. After several trials, the parameters for the classifiers were the following:
  • L 2 R L R : With regularization parameter λ = 10 9 . In this case, the λ value is very small to account for high values in regression coefficients.
  • A N N C : Two hidden layers, 20 neurons in each hidden layer. Two layers allow better classification in case of a high number of inputs, as in this case. In case of a single layer, a higher number of neurons should be used.
  • B A G G : 600 decision trees. This was an experimental value. A higher number of trees did not produce better results.
  • K N N : Euclidean distance showed a good performance, together with K = 1.
Table 1 and Table 2 summarize the results achieved by making comparisons between the values of sensitivity, specificity, accuracy, and precision. When analyzing the values shown in the tables, it is observed that the K N N classifier obtained the best result, with a sensitivity of 94.97 ± 0.70%, a global specificity of 99.27 ± 0.05%, accuracy of 98.47 ± 0.01%, and an overall precision of 97.09 ± 0.14% achieved for V F . For V T , a sensitivity of 93.47 ± 0.19%, specificity of 99.39 ± 0.15%, accuracy 98.97 ± 0.08%, and an overall precision of 92.11 ± 0.7% was obtained.
For a complete analysis, the confusion matrices for the classes and the used algorithms were calculated (Table 3 and Table 4). These tables show that the main conflicts exist in pairs: V F and V T , N o r m a l and O t h e r s . Actually, since the number of segments is lower for V F and V T compared to normal and Others, the proportion of confusion mainly resides in V F and V T S, which was expected according to other algorithm results and clinical practice.
Comparing the results offered by the different algorithms, there is an important variation in the results for the sensitivities of V F and V T depending on the algorithm. For instance, if the sensitivity level for V F is high, the sensitivity for V T decreases. It can be observed that the K N N classifier achieves the best performance for the proposed methodology due to the adequate detection and discrimination capacity of V F when compared against the rest of classes. However, it has a high execution time because the K N N algorithm requires many iterations to calculate the closest distances. For the A N N C algorithm, the classes are separated by means of a surface that maximizes the margin among them, with the least number of training errors, having a computational cost much lower than that obtained with K N N . The L 2 R L R algorithm has less time of execution because it is based on probabilities. The Bagging creates its individual classifiers by training a system of classification on different bootstrap samples of the training set, thus retrieving a higher run time than the rest of the classifiers (except for K N N ).

7.2. Comparative Study for the Method of Combined Classifiers

In this section, the results obtained by the combination methods are described: Voting Majority Method (VMM) and Hierarchical Method (HM) described above. In the classification tests performed, we show how the combination of classification algorithms behaves in relation to the results obtained in the previous test using individual classifiers. For proper comparison, the same D M data used for individual classifiers, and the same classifier parameters were used in these analyses.
In the first analysis, the Voting Majority Method (VMM) is applied using different combinations of three individual classification algorithms in parallel, the results obtained are shown in Table 5, Table 6, Table 7 and Table 8. When analyzing the results from the tables, it can be seen, in both cases, that the detection of V F has significantly improved when compared with individual classifiers. However, the detection of V T has decreased when compared with what those obtained by the K N N algorithm, which was the best individual classifier. It is concluded that the combination of classifiers do not exceed the results obtained in case of K N N .
In the second analysis, the Hierarchical Method (HM) is applied to three individual algorithms ( C l a , C l b , C l c ) getting the C l a _ C l b _ C l c multiclassifier where C l b and C l c are in parallel and both cascaded after C l a (Figure 5). The obtained results are shown in Table 9 and Table 10 with confusion matrices in Table 11, Table 12 and Table 13.
Since K N N was the best individual algorithm in the detection and discrimination between V F and V T , it was chosen as C l b . The A N N C , B A G G , L 2 R L R algorithms are taken as C l a for the discrimination between the classes V F V T and N o r m a l O t h e r s and the A N N C and L 2 R L R algorithms are used as C l c for N o r m a l and O t h e r s discrimination. Analyzing the results, it can be concluded that the combinational algorithms have a similar or better behavior than the individual K N N in the detection of V F in very large datasets and high dimensionality, with a reduced execution time.
With all the results obtained, the use of combined algorithms can be recommended as the best method of classification. In addition, results obtained using the combination B A G G _ K N N _ A N N C using HM showed better classification ratio when compared to those obtained using the algorithms individually, and other multi classifiers.
The B A G G _ K N N _ A N N C HM obtained a good behavior in the discrimination between the classes V F , V T , N o r m a l , and O t h e r s , with a sensitivity of 95.58 ± 0.4%, a global specificity of 99.31 ± 0.08%, an accuracy of 98.6 ± 0.04%, and an overall precision of 98.25 ± 0.29% for V F . For V T , a sensitivity of 94.02 ± 0.58%, a specificity of 99.31 ± 0.08%, an accuracy of 99.14 ± 0.43%, and a precision of 98.59 ± 0.09% was obtained.
It is interesting to note that the A N N C classifier obtained a good behavior in the discrimination between the classes N o r m a l and O t h e r s , and the B A G G classifier had a good behavior in the discrimination between the classes N o r m a l O t h e r s and V F V T with a fast execution time in comparison with the individual K N N algorithm.
Table 14 shows the average execution time of all the classification algorithms analyzed in this work. The execution time corresponds to the elapsed time between the input of a t w window from the ECG signal to the generation of a classification result of the algorithm. Concerning individual classifiers, it can be appreciated that L 2 R L R and A N N C have a lower computational cost than other individual algoithms, with a run time of t = 7 × 10 5 s and t = 5 × 10 4 s, respectively. For K N N and B A G G , t = 0 . 17 s and t = 0 . 05 s was attained, respectively. In case of the VMM combination methods, they are the slowest among HM and individual, ranging from t = 190 ms to t = 290 ms. This is normal since all three classifiers ( C l a , C l b , C l c ) must be computed, increasing the total computation time. Actually, any VMM combination method required more computation time than the slowest individual algorithm ( K N N ). In case of HM classification methods, we obtained different computation time depending on the executed classifier ( C l b or C l c ) depending on the results given by the first classifier ( C l a ). For this reason we obtained a minimum and maximum computation time, ranging from t = 50 ms to t = 130 ms. Thus HM combined methods provide a high classification, together with a reduced computation time, showing their feasibility for real-time classification systems.

8. Discussion

Table 15 and Figure 6 show the AUC values and ROC curves, respectively, Figure 6a for V F and Figure 6b for V T in case of the analyzed individual algorithms. Table 16 and Figure 7 show the AUC values and ROC curves, respectively, for V F (Figure 6a) and V T (Figure 6b) classification results for the VMM combination of classifiers. Table 17 and Figure 8 show the AUC values and ROC curves, respectively, for V F (Figure 6a) and V T (Figure 6b) classification results for the HM combination of classifiers. As shown, ROC curves are more adjusted in case of combined classifiers, especially in case of the VMM method.
Additionally, the structural risk of the classifier is important in order to determine the training robustness. A risk test is proposed by the A-test where multiple z-fold cross-validation are performed in order to assess how classification error evolves. In this case, we have also tested 9-fold cross validation for comparison purposes with 5-fold. Figure 9 shows that very similar results are obtained for the same classifier. Specially in case of HM combined classifers, z-5 provides a slightly higher classification ratio. In any case, differences between z-5 and z-9 in the same classifier do not exceed 1% in classification value.
Since correct detection and classification of V F and V T is of pivotal importance for an automatic external defibrillation and patient monitoring, they should be able to distinguish V F and V T accurately. If V T was misinterpreted as V F , a high-energy defibrillation would be delivered, which could damage the heart. If V T is misinterpreted as V F , the low-energy cardioversion may not return the heart to its normal sinus rhythm, which could be fatal [34]. However, clear distinction between ventricular arrhythmia rhythms and normal or other arrhythmias is required, preventing the patient to be unnecessarily exposed to an electrical cardioversion.
As previous results show, the proposed methods obtain a high accuracy, not only in V F and V T separation but also in N o r m a l and O t h e r s . This fact leads to further separate the O t h e r s class into other sub-classes where different heart pathologies could also be detected: Premature Ventricular Complex (PVC) in bigeminy or trigeminy, hypertrophy, idioventricular Rhythms, asystoles, etc. Thus, using a new classifier level, all rhythms detected as O t h e r s could enter into a new classification process in order to discern among other cardiac pathologies.
Table 18 compares results with different studies in the bibliography to check to what extent the obtained data support our hypothesis. Although different works are roughly comparable, we set two different groups for better comparison: those works aiming to distinguish between V F and V T , and those works classifying multiple rhythms.
For the first group, Xie et al. [39] used approximate entropy to distinguish between V F and V T with performance ratios of Sens = 91.84% to V F , Spe = 90.2%, Acc = 91.0%, using similar signal sources than our work. In addition, they also proposed a modified version using fuzzy similarity-based approximate entropy that, in turn, got high performance ratios (Sens = 97.98% to V F , Spe = 97.03%, Acc = 97.5%). Although we obtained higher values, to make a fair comparison between both analysis, it has to be taken into account that Xie used representative and clean episodes of V T and V F as input data, in front of our work that used a multiclass scheme, classifying four types of rhythms and considering complete patient’s registers as the input signal. The same happens for other studies distinguishing between V F and V T rhythms; Kaur and Singh [40] used approximate entropy with Empirical Mode Decomposition (EMD) and a more reduced dataset than Xie, having good performance values (Sens = 90.47 to V F , Spe = 91.66%, Acc = 91.2%). Later, Xia et al. [38] also used, in the same line, Lempel-Ziv Complexity and EMD in the same conditions that Xie did before, using a representative number of clean episodes of each pathology, and they also got high performance ratios (Sens = 98.15% to V F , Spe = 96.01%, Acc = 97.1%). The same occurred to Li et al. [37] using SVM where Sens = 96.20% to V F , Spe = 96.20%, and Acc = 96.3%, for a 2 s window, was obtained; in this case, a sensitive different set of source signals was used. Other works provides good performance ratios distinguishing between V F and V T when applied to compressed ECG signals [41]. In all cases, our performance results are slightly or sensitive better.
As a second group of comparable works we can find those aiming to distinguish normal sinus (N) apart from V T or V F . Within this group, Tan et al. [42] obtained good accuracy ratios (Acc( V F ) = 90.9%, Acc( V T ) = 84.0%, Acc(N) = 100%) using a type-2 fuzzy logic-based classifier for a three class multiclass classification ( V F , V T and N o r m a l ). Tan also described the results of using a SOM neural network with poor VT accuracy. Later, Phong et al. [43] followed the same line implementing another multiclass classifier using a type-2 TSK fuzzy system, with the same three classes than Tan used; in this case, with better accuracy ratios (Acc( V F ) = 93.3%, Acc( V T ) = 92.0%, Acc(N) = 100%). They also tried a a type-2 Mandami fuzzy system with lower values.
Other works analyse a binary distinction between V F and non- V F rhythms. Verma et al. [44] used 17 features: Morphological, spectral, and complexity. Here, the random forest classifier has been used for discrimination between V F category and non- V F category, with Acc = 94.79%, Sens = 95.04%, Spe = 94.78%. In [45], they used 13 parameters accounting for temporal (morphological), spectral, and complexity features of the ECG signal, using an SVM to distinguish between V F and non- V F categories with Sens = 95%, Spe = 99%. In another attempt [46], different heart rhythms were detected and classified into the V F and non- V F types using six features, four are derived from image-based phase plot analysis, one is derived in the frequency domain, and the last reflects the nonlinear characteristics of a data segment, values of (Acc = 95.3%, Sens = 94.5%, Spe = 94.2%) and (Acc = 90.4%, Sens = 91.6%, Spe = 89.3%) using binary decision tree (BDT) and the SVM, respectively. The algorithm proposed by Tripathy et al. [47], using digital Taylor-Fourier transform (DTFT) features of ECG signals and least square support vector machine (LS-SVM) with linear and radial basis function (RBF) kernels for detection of V F and non- V F arrhythmia episodes, obtained performance values of Acc = 83.75%, Sens = 85.20%, Spe = 82.46%.
Other authors have classified the ECG signal segments into V F V T and non- V F V T . These results are not directly comparable with those in the previous table since they provide a binary output. However, we include them since they are interesting to see how simpler two-class classification still provides similar results to those obtained in this work. Zhou et al. [48] classified the ECG signal segments into the normal sinus rhytnm (NSR) or arrhythmic shockable classes V F V T . The classification is based on Time-Delay Transform (TDT) of the signals and a neural network with Weight Fuzzy Membership Functions (NEWFM). They obtained Acc = 89.5%, Sens = 73.6%, and Spe = 93.5%. Xu et al. [49] detected V F V T using boosted classification and regression tree (Boosted-CART) obtaining Acc = 98.29%, Sens = 97.32%, and Spe = 98.95%. Other studies that have also used the class V F V T [1] have evaluated both time domain (e.g., energy, permutation entropy) and frequency domain (e.g., renyi entropy) features. The classification is done by using a Random Forest (RF) classifier aiming to identify shockable and non-shockable ventricular arrhythmia with CUDB and MITDB databases with results of Acc = 97.23%, Sens = 96.54%, Spe = 97.97% [50]. Thirteen time-frequency and statistical features were extracted and applied to the C4.5 classifier [51], resulting in Acc = 97.02%, Sens = 90.97%, Spe = 97.86% for V F V T detection (including ventricular flutter). In Kimmo et al. [52], gaussian processes were used to detect V T , V F L , and V F episodes (all three considered in the same class) extracting 15 metrics obtaining Acc = 91%, Sens = 89%, Spe = 88%.

9. Conclusions

As mentioned above, one of the main causes of sudden death is caused by the VF arrhythmia [3,53]. The rapid and correct detection of V F and V T is of fundamental importance both for the use of an automatic external defibrillator and for monitoring the patient. In order to obtain a reliable algorithm to discriminate between the different arrhythmias, an attempt was made to perform this detection task using the lowest computational load. The methodology uses the ECG to monitor biomedical signals that have different morphological and spectral characteristics.
We propose the analysis of the ECG signal for the real-time detection of the onset of ventricular fibrillation using a time-frequency method [7,54]. Reduction of network interference and other noises, which correspond to high frequency noises in these signals was carried out. After performing the steps above, the data matrix of each T F R is converted to an image ( T F R I ) corresponding to the different cardiac pathologies of the processed ECG signal, allowing to obtain an appropriate representation capable of providing useful information about the problem to be solved and allowing practical applications to the diagnosis in real time. The novelty of this work lies in the fact of using a reference mark W R M to establish an analysis window t w , obtaining a time-frequency representation and its associated image ( T F R and T F R I matrices, respectively) which are used as input to a combined classification algorithm without calculation of additional parameters for the classifier. This fact avoids the extraction of characteristics and thus, the loss of relevant information to discriminate between the different classes. Additionaly, we propose the use of combined specialized classifiers to improve classification. An analysis of several combination methodologies, and a comparative study between the individual performance of the K N N , A N N C , L 2 R L R , and B A G G algorithms was done. All of these individual and combined classifier algorithms were trained with the cross-validation method and evaluated based on sensitivity, specificity, accuracy, precision, and execution time.
Using the T F R I strategy, we concluded that, using z-5 cross validation, the individual K N N classifier achieves good results retrieving a sensitivity of 94.97 ± 0.70%, a specificity of 99.27 ± 0.05%, an accuracy of 98.47 ± 0.01%, and a precision of 97.09 ± 0.14% for V F . In case of V T , a sensitivity of 93.47 ± 0.19%, specificity of 99.39 ± 0.15%, accuracy of 98.97 ± 0.08%, and precision of 92.11 ± 0.7%, with a running time t = 0 . 17 s. Using the T F R I strategy with combined classifiers in hierarchical form (HM) achieved a sensitivity of 95.58 ± 0.40%, specificity of 99.31 ± 0.08%, 98.6 ± 0.04% accuracy, and a precision of 98.25 ± 0.29% for V F , with a sensitivity of 94.02 ± 0.58%, specificity of 99.31 ± 0.08%, accuracy of 99.14 ± 0.43%, and a precision of 98.59 ± 0.09% for V T , with execution time between 0 . 07 s and 0 . 12 s.
Different classifier robustness and classification analysis are performed to validate results: Sensibility, specificty, accuracy, precision, confusioin matrices, ROC, AUC, and A-test. All these analyses show that the used methodology is adequate and congruent results are obtained.
Taking into consideration the performed study, we have concluded that the use of combined classifiers is the best way to integrate the information since they provide stronger and efficient estimates than a single classifier. The proposed methodology provides useful information for the detection of V F in real time with a low computational time, discriminating V F from the rest of the cardiac pathologies satisfactorily. This fact significantly improves the possibilities of correct diagnosis of the patient when presenting an episode with any of these arrhythmias.

Author Contributions

Conceptualization, A.R.-M.; Data curation, A.M.; Formal analysis, A.M. and A.R.-M; Investigation, A.R.-M.; Methodology, A.R.-M.; Resources, J.F.G.-M.; Software, A.M.; Supervision, A.R.-M. and M.B.-M.; Validation, A.M., J.F.G.-M. and M.K.D.; Visualization, M.B.-M.; Writing—original draft, A.M. and J.V.F.-V.; Writing—review and editing, J.V.F.-V. and M.K.D.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Jekova, I. Shock advisory tool: Detection of life threatening cardiac arrhythmias and shock success prediction by means of a common parameter set. Biomed. Signal Process. Control 2007, 2, 25–33. [Google Scholar] [CrossRef]
  2. Othman, M.A.; Safri, N.M.; Ghani, I.A.; Harun, F.K.C.; Ariffin, I. A new semantic mining approach for detecting ventricular tachycardia and ventricular fibrillation. Biomed. Signal Process. Control 2013, 8, 222–227. [Google Scholar] [CrossRef]
  3. Jin, D.; Dai, C.; Gong, Y.; Lu, Y.; Zhang, L.; Quan, W.; Li, Y. Does the choice of definition for defibrillation and CPR success impact the predic-tability of ventricular fibrillation waveform analysis? Resuscitation 2017, 111, 48–54. [Google Scholar] [CrossRef] [PubMed]
  4. Mateo, J.; Torres, A.; Aparicio, A.; Santos, J. An efficient method for ECG beat classification and correction of ectopic beats. Comput. Electr. Eng. 2016, 53, 219–229. [Google Scholar] [CrossRef]
  5. Cohen, L. Time-frequency distributions-a review. Proc. IEEE 1989, 77, 941–981. [Google Scholar] [CrossRef] [Green Version]
  6. Mahmoud, S.S.; Hussain, Z.M.; Cosic, I.; Fang, Q. Time-frequency analysis of normal and abnormal biological signals. Biomed. Signal Process. Control 2006, 1, 33–43. [Google Scholar] [CrossRef] [Green Version]
  7. Yochum, M.; Renaud, C.; Jacquir, S. Automatic detection of P, QRS and T patterns in 12 leads ECG signal based on CWT. Biomed. Signal Process. Control 2016, 25, 46–52. [Google Scholar] [CrossRef] [Green Version]
  8. Mjahad, A.; Rosado-Munoz, A.; Bataller-Mompean, M.; Francés-Víllora, J.V.; Guerrero-Martńez, J.F. Ventricular Fibrillation and Tachycardia detection from surface ECG using time-frequency representation images as input dataset for machine learning. Comput. Methods Programs Biomed. 2017, 141, 119–127. [Google Scholar] [CrossRef] [PubMed]
  9. Poularikas, A.D. The Transforms and Applications Handbooks; 3 ed.; IEEE Press: Huntsville, AL, USA, 1999. [Google Scholar]
  10. Sornmo, L.; Laguna, P. Bioelectrical Signal Processing in Cardiac and Neurological Applications; Elsevier Academic Press: Amsterdam, The Netherlands, 2005; p. 688. [Google Scholar]
  11. Rangayyan, R.M. Biomedical Signal Analysis: A Case-Study Approach. Ann. Biomed. Eng. 2002, 30, 983. [Google Scholar]
  12. Masetic, Z.; Subasi, A. Congestive heart failure detection using random forest classifier. Comput. Methods Programs Biomed. 2016, 130, 54–64. [Google Scholar] [CrossRef] [PubMed]
  13. Johnson, W.; Vatterott, P.; Peterson, M.; Bagwe, S.; Underwood, R.; Bank, A.; Gage, R.; Ramza, B.; Foreman, B.; Splett, V.; Haddad, T.; et al. Body surface mapping using an ECG belt to characterize electrical heterogeneity for different left ventricular pacing sites during cardiac resynchronization: Relationship with acute hemodynamic improvement. Heart Rhythm 2017, 4, 385–391. [Google Scholar] [CrossRef] [PubMed]
  14. Gama, J.; Brazdil, P. Cascade generalization. Mach. Learn. 2000, 41, 315–343. [Google Scholar] [CrossRef]
  15. Wolpert, D. Stacked generalization. Neural Netw. 1992, 5, 241–259. [Google Scholar] [CrossRef] [Green Version]
  16. Hoffmann, F. Advances in Intelligent Data Analysis. In Proceedings of the 4th International Conference, IDA 2001, Cascais, Portugal, 13–15 September 2001; pp. 13–15. [Google Scholar]
  17. Woźniak, M.; Graña, M.; Corchado, E. A survey of multiple classifier systems as hybrid systems. Inf. Fusion 2014, 16, 3–17. [Google Scholar] [CrossRef]
  18. Pławiak, P. Novel genetic ensembles of classifiers applied to myocardium dysfunction recognition based on ECG signals. Swarm Evol. Comput. 2018, 39, 192–208. [Google Scholar] [CrossRef]
  19. Kim, Y.J.; Heo, J.; Park, K.S.; Kim, S. Proposition of novel classification approach and features for improved real-time arrhythmia monitoring. Comput. Biol. Med. 2016, 75, 190–202. [Google Scholar] [CrossRef] [PubMed]
  20. Huang, H.; Liu, J.; Zhu, Q.; Wang, R.; Hu, G. A new hierarchical method for inter-patient heartbeat classification using random projections and RR intervals. Biomed. Eng. Online 2014, 13, 90. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  21. da S. Luz, E.J.; Menotti, D.; Schwartz, W.R. Evaluating the use of ECG signal in low frequencies as a biometry. Expert Syst. Appl. 2014, 41, 2309–2315. [Google Scholar] [CrossRef]
  22. Teijeiro, T.; García, C.A.; Castro, D.; Félix, P. Arrhythmia Classification from the Abductive Interpretation of Short Single-Lead ECG Records. CoRR 2017. [Google Scholar] [CrossRef]
  23. Tran, H.L.; Pham, V.N.; Vuong, H.N. Multiple neural network integration using a binary decision tree to improve the ECG signal recognition accuracy. Int. J. Appl. Math. Comput. Sci. 2014, 24, 647–655. [Google Scholar] [CrossRef] [Green Version]
  24. Ladavich, S.; Ghoraani, B. Rate-independent detection of atrial fibrillation by statistical modeling of atrial activity. Biomed. Signal Process. Control 2015, 18, 274–281. [Google Scholar] [CrossRef]
  25. Goldberger, A.L.; Amaral, L.A.N.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.G.; Mietus, J.E.; Moody, G.B.; Peng, C.K.; Stanley, H.E. PhysioBank, PhysioToolkit, and PhysioNet. Circulation 2000, 101, e215–e220. [Google Scholar] [CrossRef] [PubMed]
  26. Physionet.org. The MIT-BIH. Database. Available online: https://physionet.org/physiobank/database/mitdb/ (accessed on 25 October 2018).
  27. Institute, E. American Heart Association ECG Database. Available online: http://ecri.org (accessed on 25 October 2018).
  28. Kaur, M.; Singh, B. Comparison of Different Approaches for Removal of Baseline Wander from ECG Signal. In Proceedings of the International Conference & Workshop on Emerging Trends in Technology, Mumbai, Maharashtra, India, 25–26 February 2011; ACM: New York, NY, USA, 2011; pp. 1290–1294. [Google Scholar] [CrossRef]
  29. Ravindra Pratap Narwaria, S.V.; Singhal, P.K. Removal of Baseline Wander and Power Line Interference from ECG Signal—A Survey Approach. Int. J. Electron. Eng. 2011, 3, 107–111. [Google Scholar]
  30. Viitasalo, M.; Karjalainen, J. Q T Intervals at Heart Rates From 50 to 120 Beats per Minute During 24 Hour Electrocardiographic Recordings in 100 Healthy Men Effects of Atenolol. Am. Heart Assoc. 1992, 86, 1439–1442. [Google Scholar] [CrossRef]
  31. Mjahad, A.; Rosado-Munoz, A.; Bataller-Mompeán, M.; Francés-Víllora, J.V.; Guerrero-Martńez, J.F. Ventricular Fibrillation detection using time-frequency and the KNN classifier without parameter extraction. Rev. Iberoam. Autom. Inform. Ind. 2018, 15, 124–132. [Google Scholar] [CrossRef]
  32. Maudes, J. Combinacion de Clasificadores: Construccion de Caracteristicas e Incremento de la Diversidad. Ph.D. Thesis, Universidad de Burgos, Burgos, Spain, 2010. [Google Scholar]
  33. Labatut, V.; Cherifi, H. Accuracy Measures for the Comparison of Classifiers. In The 5th International Conference on Information Technology; Ali, A.D., Ed.; Al-Zaytoonah University of Jordan: Amman, Jordan, 2011. [Google Scholar]
  34. Thakor, N.V.; Zhu, Y.S.; Pan, K.Y. Ventricular tachycardia and fibrillation detection by a sequential hypothesis testing algorithm. IEEE Trans. Biomed. Eng. 1990, 37, 837–843. [Google Scholar] [CrossRef] [PubMed]
  35. Mjahad, A.; Rosado-Muñoz, A.; Guerrero-Martinez, J.; Bataller-Mompean, M.; Frances-Villora, J.V. ECG Analysis for Ventricular Fibrillation Detection Using a Boltzmann Network. In VI Latin American Congress on Biomedical Engineering CLAIB 2014, Parana, Argentina, 29–31 October 2014; Braidot, A., Hadad, A., Eds.; Springer: Cham, Switzerland, 2015; Volume 49, pp. 532–535. [Google Scholar] [CrossRef]
  36. Jekova, I.; Krasteva, V. Real time detection of ventricular fibrillation and tachycardia. Physiol. Meas. 2004, 25, 1167. [Google Scholar] [CrossRef] [PubMed]
  37. Li, Q.; Rajagopalan, C.; Clifford, G.D. Ventricular fibrillation and tachycardia classification using a machine learning approach. IEEE Trans. Biomed. Eng. 2014, 61, 1607–1613. [Google Scholar] [PubMed]
  38. Xia, D.; Meng, Q.; Chen, Y.; Zhang, Z. Classification of Ventricular Tachycardia and Fibrillation Based on the Lempel-Ziv Complexity and EMD. In International Conference on Intelligent Computing; Springer: Cham, Switzerland, 2014; Volume 8590, pp. 322–329. [Google Scholar] [CrossRef]
  39. Xie, H.B.; Zhong-Mei, G.; Liu, H. Classification of Ventricular Tachycardia and Fibrillation Using Fuzzy Similarity-based Approximate Entropy. Expert Syst. Appl. 2011, 38, 3973–3981. [Google Scholar] [CrossRef]
  40. Kaur, L.; Singh, V. Ventricular Fibrillation Detection using Empirical Mode Decomposition and Approximate Entropy. Int. J. Emerg. Technol. Adv. Eng. 2013, 3, 260–268. [Google Scholar]
  41. Ibaida, A.; Khalil, I. Distinguishing between Ventricular Tachycardia and Ventricular Fibrillation from Compressed ECG Signal in Wireless Body Sensor Networks. In Proceedings of the 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, Buenos Aires, Argentina, 31 August–4 September 2010; pp. 2013–2016. [Google Scholar] [CrossRef]
  42. Tan, W.; Foo, C.L.; Chua, T.W. Type-2 Fuzzy System for ECG Arrhythmic Classification. In Proceedings of the 2007 IEEE International Fuzzy Systems Conference, London, UK, 23–26 July 2007; pp. 1–6. [Google Scholar] [CrossRef]
  43. Phong, P.A.; Thien, K.Q. Classification of Cardiac Arrhythmias Using Interval Type-2 TSK Fuzzy System. In Proceedings of the 2009 International Conference on Knowledge and Systems Engineering, Hanoi, Vietnam, 13–17 October 2009; pp. 1–6. [Google Scholar] [CrossRef]
  44. Verma, A.; Dong, X. Detection of Ventricular Fibrillation Using Random Forest Classifier. Biomed. Sci. Eng. 2016, 9, 259–268. [Google Scholar] [CrossRef]
  45. Alonso-Atienza, F.; Morgado, E.; Fernandez-Martínez, L.; Garcia-Alberola, A.; Rojo-Alvarez, J. Detection of Life-Threatening Arrhythmias Using Feature Selection and Support Vector Machines. IEEE Trans. Biomed. Eng. 2014, 61, 832–840. [Google Scholar] [CrossRef] [PubMed]
  46. Hajeb-Mohammadalipour, S.; Ahmadi, M.; Shahghadami, R.; Chon, K. Automated Method for Discrimination of Arrhythmias Using Time, Frequency, and Nonlinear Features of Electrocardiogram Signals. Sensors 2018, 18, 2090. [Google Scholar] [CrossRef] [PubMed]
  47. Tripathy, R.K.; Zamora-Mendez, A.; de la O Serna, J.A.; Paternina, M.R.A.; Arrieta, J.G.; Naik, G.R. Detection of Life Threatening Ventricular Arrhythmia Using Digital Taylor Fourier Transform. Front. Physiol. 2018, 9, 722. [Google Scholar] [CrossRef] [PubMed]
  48. Zhou, X.; Lim, J.A. A new Ventricular fibrillation/Tachycardia Detection Algorithm for Shockable Rhythm Detection. Biosci. Med. Res. Adv. Sci. Technol. Lett. 2015, 91, 113–116. [Google Scholar]
  49. Xu, Y.; Wang, D.; Zhang, W.; Ping, P.; Feng, L. Detection of ventricular tachycardia and fibrillation using adaptive variational mode decomposition and boosted-CART classifier. Biomed. Signal Process. Control 2018, 39, 219–229. [Google Scholar] [CrossRef]
  50. Tripathy, R.K.; Sharma, L.N.; Dandapat, S. Detection of shockable ventricular arrhythmia using variational mode decomposition. J. Med. Syst. 2016, 40, 79. [Google Scholar] [CrossRef] [PubMed]
  51. Mohanty, M.; Sahoo, S.; Biswal, P.; Sabut, S. Efficient classification of ventricular arrhythmias using feature selection and C4.5 classifier. Biomed. Signal Process. Control 2018, 44, 200–208. [Google Scholar] [CrossRef]
  52. Suotsalo, K.; Särkkä, S. Detecting malignant ventricular arrhythmias in electrocardiograms by Gaussian process classification. In Proceedings of the 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), Tokyo, Japan, 25–28 September 2017; pp. 1–5. [Google Scholar]
  53. Murakoshi, N.; Aonuma, K. Epidemiology of Arrhythmias and Sudden Cardiac Death in Asia. Circ. J. 2013, 77, 2419–2431. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  54. Martin, W.; Flandrin, P. Wigner-Ville spectral analysis of nonstationary processes. IEEE Trans. Acoust. Speech Signal Process. 1985, 33, 1461–1470. [Google Scholar] [CrossRef] [Green Version]
Figure 1. General diagram of the processing steps performed for the detection of Ventricular Fibrillation. After the reduction of the baseline oscillation and the calculation of the reference marks, the Hilbert transform (Ht) and the time-frequency representation ( T F R ) for a time window ( t w ) is obtained. Optionally, the possibility of obtaining the equivalent image from the T F R ( T F R I ) is used to serve as input to the combined classifier using three cascaded classifiers ( C l a , C l b , C l c ) where all of them are the same type ( C l a = C l b = C l c ) or combined C l a _ C l b _ C l c in case at least one classifier among them ( C l a , C l b , C l c ) is different.
Figure 1. General diagram of the processing steps performed for the detection of Ventricular Fibrillation. After the reduction of the baseline oscillation and the calculation of the reference marks, the Hilbert transform (Ht) and the time-frequency representation ( T F R ) for a time window ( t w ) is obtained. Optionally, the possibility of obtaining the equivalent image from the T F R ( T F R I ) is used to serve as input to the combined classifier using three cascaded classifiers ( C l a , C l b , C l c ) where all of them are the same type ( C l a = C l b = C l c ) or combined C l a _ C l b _ C l c in case at least one classifier among them ( C l a , C l b , C l c ) is different.
Applsci 08 02057 g001
Figure 2. IIR bandpass filter applied to a ’Normal’ type ECG. The input temporal signal is plotted in blue and the filtered output signal is plotted in red. Its frequency response is shown below.
Figure 2. IIR bandpass filter applied to a ’Normal’ type ECG. The input temporal signal is plotted in blue and the filtered output signal is plotted in red. Its frequency response is shown below.
Applsci 08 02057 g002
Figure 3. From top to bottom: (a,b) correspond to the original signal window and T F R ; (c,d) show the T F R I being 45 × 150 in size, for N o r m a l (c) and V F (d) types.
Figure 3. From top to bottom: (a,b) correspond to the original signal window and T F R ; (c,d) show the T F R I being 45 × 150 in size, for N o r m a l (c) and V F (d) types.
Applsci 08 02057 g003
Figure 4. Parallel classifier combination and final voting. All classifiers receive all data and must obtain a four-class decision through a majority voting among results generated by each classifier.
Figure 4. Parallel classifier combination and final voting. All classifiers receive all data and must obtain a four-class decision through a majority voting among results generated by each classifier.
Applsci 08 02057 g004
Figure 5. Hierarchical topology classifier. Multi-class classification is obtained by joining several binary classifiers.
Figure 5. Hierarchical topology classifier. Multi-class classification is obtained by joining several binary classifiers.
Applsci 08 02057 g005
Figure 6. ROC curve for V F (a) and V T (b) in L2 Regularization ( L 2 R L R ), adaptive neural network ( A N N C ), Bagging ( B A G G ), and K-nearest neighbor ( K N N ) individual algorithms.
Figure 6. ROC curve for V F (a) and V T (b) in L2 Regularization ( L 2 R L R ), adaptive neural network ( A N N C ), Bagging ( B A G G ), and K-nearest neighbor ( K N N ) individual algorithms.
Applsci 08 02057 g006
Figure 7. Receiver operating characteristic (ROC) curve for voting majority method (VMM) combination methods in detection of V F (a) and V T (b).
Figure 7. Receiver operating characteristic (ROC) curve for voting majority method (VMM) combination methods in detection of V F (a) and V T (b).
Applsci 08 02057 g007
Figure 8. ROC curve for heirarchical method (HM) classification combination methods in detection of V F (a) and V T (b).
Figure 8. ROC curve for heirarchical method (HM) classification combination methods in detection of V F (a) and V T (b).
Applsci 08 02057 g008
Figure 9. Classification rate for sensibility to each class, for the test dataset using z-5 and z-9 cross validation. Individual and combined VMM and HM classifiers are shown.
Figure 9. Classification rate for sensibility to each class, for the test dataset using z-5 and z-9 cross validation. Individual and combined VMM and HM classifiers are shown.
Applsci 08 02057 g009
Table 1. Results for the different algorithms adaptive neural network ( A N N C ), Bagging ( B A G G ), L2 Regularization ( L 2 R L R ), and K-nearest neighbor ( K N N ) using as input the time-frequency representation image ( T F R I ) used to characterize and detect V F and V T classes.
Table 1. Results for the different algorithms adaptive neural network ( A N N C ), Bagging ( B A G G ), L2 Regularization ( L 2 R L R ), and K-nearest neighbor ( K N N ) using as input the time-frequency representation image ( T F R I ) used to characterize and detect V F and V T classes.
ClassVFVT
Sens %Spe %Acc %Pre %Sens %Spe %Acc %Pre %
Classification AlgorithmsVFGlobalGlobalGlobalVTGlobalGlobalGlobal
L 2 R L R
Test91.54 ± 0.4698.45 ± 0.0697.15 ± 0.0994.74 ± 1.2788.15 ± 0.5998.81 ± 0.0498.07 ± 0.0789.67 ± 1.77
A N N C
Test95.56 ± 0.3398.80 ± 0.1398.19 ± 0.1595.26 ± 0.4588.80 ± 1.1399.52 ± 0.0698.87 ± 0.0394.05 ± 0.80
B A G G
Test98.46 ± 0.2698.43 ± 0.1298.44 ± 0.1497.09 ± 0.1484.8 ± 1.7099.86 ± 0.0498.82 ± 0,1297.81 ± 0.82
K N N
Test94.97 ± 0.7099.27 ± 0.0598.47 ± 0.0197.09 ± 0.1493.47 ± 0.1999.39 ± 0.1598.97 ± 0.0892.11 ± 0.70
Table 2. Results for individual algorithms A N N C , B A G G , L 2 R L R , and K N N using as input the T F R I used to characterize and detect O t h e r s and N o r m a l classes.
Table 2. Results for individual algorithms A N N C , B A G G , L 2 R L R , and K N N using as input the T F R I used to characterize and detect O t h e r s and N o r m a l classes.
ClassOthersNormal
Sens %Spe %Acc %Pre %Sens %Spe %Acc %Pre %
Classification AlgorithmsOthersGlobalGlobalGlobalNormalGlobalGlobalGlobal
L 2 R L R
Test94.80 ± 0.5598.79 ± 0.0397.91 ± 0.1296.29 ± 0.9897.63 ± 0.1296.61 ± 0.497.17 ± 0.2797.56 ± 0.68
A N N C
Test96.87 ± 1.0699.55 ± 0.0898.96 ± 0.2198.45 ± 0.3998.98 ± 0.1397.72 ± 0.5698.40 ± 0.2798.39 ± 0.22
B A G G
Test95.66 ± 0.3199.87 ± 0.0298.95 ± 0.0799.60 ± 0.1099.39 ± 0.1997.61 ± 0.0998.57 ± 1.3197.71 ± 0.04
K N N
Test97.69 ± 0.2199.40 ± 0.0499.03 ± 0.0597.92 ± 0.1699.07 ± 0.0898.29 ± 0.1098.71 ± 0.0698.66 ± 0.06
Table 3. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the L 2 R L R and A N N C algorithms.
Table 3. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the L 2 R L R and A N N C algorithms.
AlgorithmsL2RLRANNC
VFVTOthersNormalVFVTOthersNormal
V F 1627641348168934326
V T 5958321255589012
O t h e r s 90190979111194144
N o r m a l 328684895183274955
Table 4. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the B A G G and K N N algorithms.
Table 4. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the B A G G and K N N algorithms.
AlgorithmsBAGGKNN
VFVTOthersNormalVFVTOthersNormal
V F 172192201666521321
V T 855640727618011
O t h e r s 4119029080195435
N o r m a l 16364978141294949
Table 5. Results for the voting majority method (VMM) combination method using T F R I as data input. Results correspond to detection of V F and V T pathologies.
Table 5. Results for the voting majority method (VMM) combination method using T F R I as data input. Results correspond to detection of V F and V T pathologies.
ClassVFVT
Sens %Spe %Acc %Pre %Sens %Spe %Acc %Pre %
Combined Classification Algorithms (Cla,Clb,Clc)VFGlobalGlobalGlobalVTGlobalGlobalGlobal
A N N C _ K N N _ L 2 R L R
Test97.47 ± 0.1399.09 ± 0.1098.71 ± 0.0696.12 ± 0.2391.31 ± 0.8899.97 ± 0.0899.18 ± 0.0196.26 ± 0.90
A N N C _ K N N _ B A G G
Test98.46 ± 0.0698.98 ± 0.0298.89 ± 0.0196.04 ± 0.6089.63 ± 0.3099.86 ± 0.0199.14 ± 0.0297.64 ± 0.15
B A G G _ K N N _ L 2 R L R
Test98.43 ± 0.1698.98 ± 0.0898.87 ± 0.0495.79 ± 0.5091.01 ± 0.6899.83 ± 0.0399.16 ± 0.0297.51 ± 0.18
B A G G _ A N N C _ L 2 R L R
Test97.89 ± 0.0898.84 ± 0.0598.66 ± 0.0495.33 ± 0.3187.80 ± 1.2199.88 ± 0.0399.98 ± 0.0697.61 ± 0.50
Table 6. Results for the VMM combination method using T F R I as data input. Results correspond to detection of O t h e r and N o r m a l classes.
Table 6. Results for the VMM combination method using T F R I as data input. Results correspond to detection of O t h e r and N o r m a l classes.
ClassOthersNormal
Sens %Spe %Acc %Pre %Sens %Spe %Acc %Pre %
Combined Classification Algorithms (Cla,Clb,Clc)OthersGlobalGlobalGlobalNormalGlobalGlobalGlobal
A N N C _ K N N _ L 2 R L R
Test97.35 ± 0.1599.67 ± 0.0299.16 ± 0.0398.57 ± 0.0599.22 ± 0.0598.41 ± 0.0998.84 ± 0.0398.63 ± 0.02
A N N C _ K N N _ B A G G
Test97.20 ± 0.1199.84 ± 0.0499.27 ± 0.0199.21 ± 0.1499.52 ± 0.0698.29 ± 0.0598.95 ± 0.0198.64 ± 0.04
B A G G _ K N N _ L 2 R L R
Test96.95 ± 0.1199.77 ± 0.0499.16 ± 0.0199.25 ± 0.1899.4 ± 0.0598.30 ± 0.0398.89 ± 0.0198.51 ± 0.02
B A G G _ A N N C _ L 2 R L R
Test96.54 ± 0.0999.74 ± 0.0599.05 ± 0.0299.12 ± 0.1799.38 ± 0.0997.68 ± 0.2398.59 ± 0.0998.26 ± 0.17
Table 7. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l classes using the A N N C _ K N N _ L 2 R L R and A N N C _ K N N _ B A G G VMM combined classifier.
Table 7. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l classes using the A N N C _ K N N _ L 2 R L R and A N N C _ K N N _ B A G G VMM combined classifier.
Algorithms ANNC _ KNN _ L 2 RLR ANNC _ KNN _ BAGG
VFVTOthersNormalVFVTOthersNormal
V F 170623518172017213
V T 456001105359508
O t h e r s 8019484161194347
N o r m a l 140234965141154973
Table 8. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l classes using the B A G G _ K N N _ L 2 R L R and B A G G _ A N N C _ L 2 R L R VMM combined classifier.
Table 8. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l classes using the B A G G _ K N N _ L 2 R L R and B A G G _ A N N C _ L 2 R L R VMM combined classifier.
Algorithms BAGG _ KNN _ L 2 RLR BAGG _ ANNC _ L 2 RLR
VFVTOthersNormalVFVTOthersNormal
V F 1724140141717132
V T 55595066258608
O t h e r s 6119355570193060
N o r m a l 141144974151164971
Table 9. Results for different hierarchical method (HM) combined classifier methods to detect V F and V T pathologies.
Table 9. Results for different hierarchical method (HM) combined classifier methods to detect V F and V T pathologies.
ClassVFVT
Sens %Spe %Acc %Pre %Sens %Spe %Acc %Pre %
Combined Classification Algorithms (Cla,Clb,Clc)VFGlobalGlobalGlobalVTGlobalGlobalGlobal
A N N C _ K N N _ A N N C
Test95.13 ± 0.0699.10 ± 0.0498.41 ± 0.0496.38 ± 0.1992.32 ± 0.6699.33 ± 0.0798.91 ± 0.0291.38 ± 0.84
A N N C _ K N N _ L 2 R L R
Test94.89 ± 0.5699.08 ± 0.2098.32 ± 0.2796.02 ± 0.8792.37 ± 2.4699.27 ± 0.1098.78 ± 0.2390.71 ± 1.38
B A G G _ K N N _ A N N C
Test95.58 ± 0.4099.31 ± 0.0898.6 ± 0.0498.25 ± 0.2994.02 ± 0.5899.31 ± 0.0899.14 ± 0.4398.59 ± 0.09
B A G G _ K N N _ L 2 R L R
Test95.58 ± 0.4299.31 ± 0.0498.6 ± 0.0596.93 ± 0.1994.02 ± 0.5899.31 ± 0.0899.14 ± 0.4391.25 ± 0.91
L 2 R L R _ K N N _ A N N C
Test93.33 ± 0.3198.93 ± 0.0797.88 ± 0.1295.30 ± 0.9692.01 ± 0.5899.22 ± 0.0998.71 ± 0.0390.06 ± 1.37
L 2 R L R _ K N N _ L 2 R L R
Test93.54 ± 0.3198.01 ± 0.0797.98 ± 0.1295.65 ± 0.3592.95 ± 0.5899.26 ± 0.0898.82 ± 0.0390.70 ± 0.98
Table 10. Results for different HM combined classifier methods to detect N o r m a l and O t h e r s classes.
Table 10. Results for different HM combined classifier methods to detect N o r m a l and O t h e r s classes.
ClassOthersNormal
Sens %Spe %Acc %Pre %Sens %Spe %Acc %Pre %
Combined Classification Algorithms (Cla,Clb,Clc)OthersGlobalGlobalGlobalNormalGlobalGlobalGlobal
A N N C _ K N N _ A N N C
Test97.22 ± 0.1499.57 ± 0.1199.06 ± 0.1098.45 ± 0.3999.04 ± 0.1598.10 ± 0.2698.61 ± 0.1598.39 ± 0.22
A N N C _ K N N _ L 2 R L R
Test96.11 ± 0.6799.18 ± 0.3398.45 ± 0.3896.70 ± 1.1698.42 ± 0.4397.67 ± 0.2698.17 ± 0.3098.01 ± 0.22
B A G G _ K N N _ A N N C
Test97.42 ± 0.1299.52 ± 0.0799.07 ± 0.0898.25 ± 0.2999.02 ± 0.1498.36 ± 0.1098.72 ± 0.1198.59 ± 0.09
B A G G _ K N N _ L 2 R L R
Test96.92 ± 0.3898.92 ± 0.0598.28 ± 0.0496.10 ± 0.1798.22 ± 0.0797.73 ± 0.1997.99 ± 0.0598.06 ± 0.16
L 2 R L R _ K N N _ A N N C
Test97.22 ± 0.1199.50 ± 0.0699.01 ± 0.0798.20 ± 0.2498.61 ± 0.1497.39 ± 0.2798.05 ± 0.1897.79 ± 0.23
L 2 R L R _ K N N _ L 2 R L R
Test95.71 ± 0.3098.84 ± 0.0298.16 ± 0.0495.83 ± 0.0797.81 ± 0.0896.89 ± 0.3297.38 ± 0.1097.34 ± 0.27
Table 11. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the A N N C _ K N N _ A N N C and A N N C _ K N N _ L 2 R L R combined classifiers in HM topolgy.
Table 11. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the A N N C _ K N N _ A N N C and A N N C _ K N N _ L 2 R L R combined classifiers in HM topolgy.
Algorithms ANNC _ KNN _ ANNC ANNC _ KNN _ L 2 RLR
VFVTOthersNormalVFVTOthersNormal
V F 166756326166260723
V T 3361201138606111
O t h e r s 120194144120191966
N o r m a l 192274955182594924
Table 12. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the B A G G _ K N N _ A N N C and B A G G _ K N N _ L 2 R L R combined classifiers in HM topolgy.
Table 12. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the B A G G _ K N N _ A N N C and B A G G _ K N N _ L 2 R L R combined classifiers in HM topolgy.
Algorithms BAGG _ KNN _ ANNC BAGG _ KNN _ L 2 RLR
VFVTOthersNormalVFVTOthersNormal
V F 167456517167456715
V T 32617073261716
O t h e r s 5019454750191676
N o r m a l 163304954163704914
Table 13. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the L 2 R L R _ K N N _ A N N C and L 2 R L R _ K N N _ L 2 R L R combined classifiers in HM topolgy.
Table 13. Confusion matrix for classification of V F , V T , O t h e r s , and N o r m a l using the L 2 R L R _ K N N _ A N N C and L 2 R L R _ K N N _ L 2 R L R combined classifiers in HM topolgy.
Algorithms L 2 RLR _ KNN _ ANNC L 2 RLR _ KNN _ L 2 RLR
VFVTOthersNormalVFVTOthersNormal
V F 1635606511639561245
V T 3860401432610113
O t h e r s 9019414791191176
N o r m a l 327304934337694894
Table 14. Results of the average execution run time for each data window t w obtained using individual and combined classification algorithms.
Table 14. Results of the average execution run time for each data window t w obtained using individual and combined classification algorithms.
AlgorithmExecution Run Time (s)
Min (s)Max (s)
K N N 0.17
A N N C 5 × 10 4
L 2 R L R 7 × 10 5
B A G G 0.05
A N N C _ K N N _ L 2 R L R (VMM)0.27
A N N C _ K N N _ B A G G (VMM)0.29
B A G G _ K N N _ L 2 R L R (VMM)0.28
B A G G _ A N N C _ L 2 R L R (VMM)0.19
A N N C _ K N N _ A N N C (HM)0.050.10
A N N C _ K N N _ L 2 R L R (HM)0.050.10
B A G G _ K N N _ A N N C (HM)0.070.12
B A G G _ K N N _ L 2 R L R (HM)0.070.11
L 2 R L R _ K N N _ A N N C (HM)0.050.09
L 2 R L R _ K N N _ L 2 R L R (HM)0.050.13
Table 15. Area under the curve (AUC) values for V F and V T in L 2 R L R , A N N C , B A G G , and K N N individual algorithms.
Table 15. Area under the curve (AUC) values for V F and V T in L 2 R L R , A N N C , B A G G , and K N N individual algorithms.
Algorithms L 2 RLRC ( % ) ANNC ( % ) BAGG ( % ) KNN ( % )
AUC(VF)95.0297.1898.4497.12
AUC(VT)93.4894.1692.3396.43
Table 16. AUC for VMM combination methods in detection of V F and V T .
Table 16. AUC for VMM combination methods in detection of V F and V T .
Algorithm ANNC _ KNN _ L 2 RLR ( % ) ANNC _ KNN _ BAGG ( % ) BAGG _ KNN _ L 2 RLR ( % ) ANNC _ ANNC _ L 2 RLR ( % )
AUC(VF)98.2898.7298.7098.36
AUC(VT)95.0397.1898.4497.12
Table 17. AUC for HM classification combination methods in detection of V F and V T .
Table 17. AUC for HM classification combination methods in detection of V F and V T .
Algorithm ANNC _ KNN _ ANNC ( % ) ANNC _ KNN _ L 2 RLR ( % ) ANNC _ KNN _ L 2 RLR ( % ) ANNC _ KNN _ L 2 RLR ( % ) ANNC _ KNN _ L 2 RLR ( % ) BAGG _ KNN _ ANNC ( % )
AUC(VF)97.1196.9897.4497.4496.1395.77
AUC(VT)95.0097.1898.4497.1297.1298.44
Table 18. Performance evaluation of V F detection algorithms for comparison with the four analyzed classifiers in this work.
Table 18. Performance evaluation of V F detection algorithms for comparison with the four analyzed classifiers in this work.
TypesVFVTOtherNormalData Base
MethodSens %Spe %Acc %Sens %Spe %Acc %Sens %Spe %Acc %Sens %Spe %Acc %
This work, ANNC(t-f)95.5698.8098.1988.8099.5298.8796.8699.5599.5798.9897.7298.40AHA & MIT-BIH
This work, BAGG(t-f)98.4698.4398.4484.5899.8698.8295.6699.8798.9599.3997.6198.57AHA & MIT-BIH
This work, K N N (t-f)94.9799.2798.4793.4799.3998.9797.6999.4099.0399.0798.2998.71AHA & MIT-BIH
This work, B A G G _ K N N _ A N N C (t-f)95.5899.3198.694.0299.3199.1497.4299.5299.0799.0298.3698.72AHA & MIT-BIH
[35], using Boltzmann92.52 76.01 81.04 MIT-BIH
[1], using Discriminant Analysis94.1093.80 AHA & MIT-BIH
[36], using Filter and Counts94.4095.9094.7 AHA & MIT-BIH
[37], using SVM96.2096.2096.3 AHA & MIT-BIH
[38], using Lempel-Ziv and EMD98.1596.0197.1 CU & MIT-BIH
[39], using Fuzzy Simil App Entropy97.9897.0397.5 CU & MIT-BIH
[39], using Approximate Entropy91.8490.291.0 MIT-BIH
[40], using EMD & App Entropy90.4791.6691.2 MIT-BIH
[41], using KNN98.1088.0093.2 MIT-BIH
[41], using RBF91.5390.9191.3 MIT-BIH
[42], using Type 2 fuzzy 90.9 84.0 100MIT-BIH
[42], using Neural SOM 100.0 48.0 88MIT-BIH
[43], using Type 2 TSK Fuzzy 93.3 92.0 100MIT-BIH
[43], using Type 2 Mamdani Fuzzy 86.6 88.0 100MIT-BIH
[44], using Random Forest Classifier95.0494.7894.79 CU &MIT-BIH
[45], using SVM9599 CU &MIT-BIH
[46], using Binary Decision Tree (BDT)95.394.594.2 CU &MIT-BIH
[46], using SVM90.491.689.3 CU &MIT-BIH
[47], using LS-SVM classifier with RBF kernels85.2082.4683.75 CU &MIT-BIH

Share and Cite

MDPI and ACS Style

Mjahad, A.; Rosado-Muñoz, A.; Guerrero-Martínez, J.F.; Bataller-Mompeán, M.; Francés-Villora, J.V.; Dutta, M.K. Detection of Ventricular Fibrillation Using the Image from Time-Frequency Representation and Combined Classifiers without Feature Extraction. Appl. Sci. 2018, 8, 2057. https://doi.org/10.3390/app8112057

AMA Style

Mjahad A, Rosado-Muñoz A, Guerrero-Martínez JF, Bataller-Mompeán M, Francés-Villora JV, Dutta MK. Detection of Ventricular Fibrillation Using the Image from Time-Frequency Representation and Combined Classifiers without Feature Extraction. Applied Sciences. 2018; 8(11):2057. https://doi.org/10.3390/app8112057

Chicago/Turabian Style

Mjahad, Azeddine, Alfredo Rosado-Muñoz, Juan F. Guerrero-Martínez, Manuel Bataller-Mompeán, Jose V. Francés-Villora, and Malay Kishore Dutta. 2018. "Detection of Ventricular Fibrillation Using the Image from Time-Frequency Representation and Combined Classifiers without Feature Extraction" Applied Sciences 8, no. 11: 2057. https://doi.org/10.3390/app8112057

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop