Image Processing Techniques for Analysis of Satellite Images for Historical Maps Classification—An Overview

Abstract

Historical maps classification has become an important application in today’s scenario of everchanging land boundaries. Historical map changes include the change in boundaries of cities/states, vegetation regions, water bodies and so forth. Change detection in these regions are mainly carried out via satellite images. Hence, an extensive knowledge on satellite image processing is necessary for historical map classification applications. An exhaustive analysis on the merits and demerits of many satellite image processing methods are discussed in this paper. Though several computational methods are available, different methods perform differently for the various satellite image processing applications. Wrong selection of methods will lead to inferior results for a specific application. This work highlights the methods and the suitable satellite imaging methods associated with these applications. Several comparative analyses are also performed in this work to show the suitability of several methods. This work will help support the selection of innovative solutions for the different problems associated with satellite image processing applications.

1. Introduction

The integration of historical map classification and satellite images is a novel attempt to detect the everchanging land/water features in the earth. Satellite images captured over a period of time on a specific region are used to identify the increase/decrease in the area of the region. These data are further used to create accurate maps for all practical applications. Satellite image processing is one of the significant computational methods which finds application in military, agriculture, natural disaster prevention, natural resource identification and so forth. However, satellite image processing is extremely complex due to the large dimensions of the satellite images.

Remote sensing images contain large amount of information and if the image quality is not good or if the image analysis does not use optimum feature set, then the impact of remote sensing application for which the technique is used may not be fully utilized. The visual interpretation of remote sensing images utilize different elements of interpretation such as shape, hue, tone, texture and so forth. Manually interpretation is limited to analyzing only single image at a time due to the difficulty in multiple image interpretation. Manual interpretation is subjective and time for visual classification depends mainly on the image quality. This has paved way for automatic processing over visual analysis and is useful for simultaneous analysis of many spectral bands and can process large data sets much faster than humans. So image processing tools plays a major in applications pertaining to remote sensing. Remote sensing methods are considered as a complementary tool in various applications due to the wide range of area the images can cover. The usefulness of satellite images to remote sensing applications depends on the accuracy of the techniques. A good understanding of how well the image processing techniques perform will help in deciding the best choice of methods for various applications. With the enhancement of technology, demand for near-real-time monitoring and visual images for use in emergency services and the public in the case of a natural disaster is increasing. Recent advancements in earth monitoring satellites are making way towards use in such applications. Methods are thus being developed to utilize the available data effectively to make sure that the best possible intelligence is available to emergency services and decision makers on time.

While a vast amount of digital satellite and aerial imagery is available, the real challenge is in the analysis of raw image, extraction of useful information with high accuracy and applying it to real world decision making and applications. Even though there are various image processing techniques available, conventional methods need to be critically reviewed and image understanding with respect to features of interest is important. With the advances in artificial intelligence and machine learning, there is an improvement in extraction of detailed features from high resolution aerial imagery. The fin different layers are utilized for developing various image processing techniques.

Several methods have been already explored to process the satellite images efficiently. Still, there are plenty of scopes for the performance improvement of the conventional system. A detailed understanding of the conventional methods is extremely necessary to develop new automated systems for satellite image processing. In this work, an exhaustive survey is presented on the various aspects/applications of satellite image processing. Pros and cons of the methods are also presented which will aid in developing novel solutions for the current problems in the area of satellite image processing.

The reminder of the paper is organized as follows—Section 2 describes the various image processing techniques in remote sensing, Section 3 explains different performance evaluation metrics in image processing, Section 4 gives a brief discussion on the study with the conclusion given in Section 5.

2. Image Processing in Remote Sensing

There exist numerous methods and algorithms available for satellite image processing. The satellite image processing techniques discussed are enhancement, feature extraction, segmentation, fusion, change detection, compression, classification and feature detection which is represented in Figure 1.

2.1. Image Enhancement

The Satellite images comprise of low brightness level. This calls for the importance of image enhancement with the preservation of important details without loss of information. Contrast is an important parameter considered on subjectively evaluating an image in terms of its quality. Contrast, from human perception, is what differentiates object-to-object with background. In other words, it is the color and brightness difference between the objects and background.

Many algorithms have been designed and developed to accomplish contrast enhancement and solve various brightness related problems in image processing. Also it is one of the primary steps followed before proceeding with any of the other image processing techniques such as segmentation, object identification and so forth.

Image enhancement techniques categorized as spatial and frequency domain methods [1]. Histogram Equalization is one of the widely used image enhancement methods which can be performed on the complete image or some parts in it. It can enhance the overall image quality. But HE has the limitation that it cannot retain the average intensity in an image. Several modifications were developed over the HE such as Bi-Histogram Equalization (BHE), Recursive Mean Separate HE (RMSHE) and so forth. The enhancement methods and their evaluation metrics are shown in

Table 1. Image Enhancement methods and their performance metrics.

Method	Test Images	Performance Metrics
1. Histogram Planting [2]	Phobos Images of aircraft	Mean, Average Information
2. Modified differential evolution [3]	Images from NASA, Satpalda Geospatial Services and Satellite Imaging Corp.	Peak Signal to Noise Ratio (PSNR), Mean Square Error (MSE), Universal Quality Index (UQI), Normalized Absolute Error (NAE), Structural Contrast-Quality Index (SC-QI):
3. Fractional Differential (FD) unsharp masking [4]	Infrared Images	Average brightness (B), Average contrast (V), Average discrete information content (or entropy, E), Sharpness (S), Colorfulness (C), Correlation (R), Energy (E) and Homogeneity (M)

.

There were some unnatural occurrences in the resultant image after RMSHE which could be avoided using another method which is called the Histogram Planting (HP) [2]. It is a distribution based algorithm and it was seen to provide a reliable image enhancement technique. There are mainly two types of distribution techniques involved which are equality-spaced distribution (HP-ESD) and proportionality-spaced distribution (HP-PSD). Even though all the variations of HP show some acceptable level of enhancement, there are some unwanted artifacts introduced. To avoid these artifacts, a modification of the differential evolution algorithm combined with the Cuckoo Search (CS) algorithm for image enhancement is proposed [3].

From visual observation of the obtained results, the algorithm could clearly differentiate the minor details in the image without the presence of the artifacts. Also the average intensity of the image was also preserved. The main aim was to develop a transformation which can map intensity values to a new set of values effectively. Here the transform is such that maps the intensity by local feature extraction in the image by applying a sliding window whose size is predetermined. But since such a local feature extraction takes a lot of time, the Modified Differential Evolution (MDE) algorithm utilizes combination of global and local feature extractions. A technique for integrating multi-resolution satellite images from multiple sources is used for improving accuracy by avoiding Ground Control Points (GCPs) [5].

Here not all the rational polynomial coefficients are utilized. The tie points identified from the image are manually determined. The analysis is done by determining the root mean square error value and compared with different multi resolution images from different sensors. The future scope of the work can be extended such that instead of manually finding the tie points, an automated approach where tie points can be evaluated from different images could be done.

Some of the evaluation metrics for assessing the performance of image enhancement include Peak Signal-to-Noise Ratio (PSNR) which defines the standard of restructured image quality, Absolute Mean Brightness Error (AMBE) which is the difference between original and enhanced image brightness, Entropy which is the measure of the richness of information in the image. Mean Square Error (MSE) is the squared difference between input and the distorted image.

2.2. Feature Extraction

Feature extraction involves generating features for selection and classification. These are differentiable features and forwarded towards classification phase. Feature extraction plays a very critical role in deciding the efficiency of image classification process [6]. The various features can be categorized as—general and domain specific ones. General features are those which include color, shape and texture while domain specific features are those pertaining to specific applications such as conceptual features.

A feature matching algorithm involving graph theory is proposed in Reference [7]. This technique suffers from non-linear intensity differences. To overcome this, Mutual Information (MI) based similarity measure has been used for feature matching. But it has high computational complexity. Some robust descriptors such as Local Self-Similarity (LSS) have also been proposed which are much easier than the MI based method. To overcome the problem of quality of feature distribution, Uniform Robust (UR) Scale Invariant Feature Transform (SIFT) based algorithms is used. Recall is the ratio of correct match to the total match whereas precision is the ratio of the correct match to the sum of correct and false match.

Table 2. Feature extraction and their performance metrics.

Method	Test Images	Feature Extracted	Performance Metrics
1. Uniform Competency Feature Extraction [8]	Synthetic Images: ETM+, World View 2, Real Images: SPOT4, SPOT6, Quick Bird, GeoEye1, ASTER	Rotation and scale invariant Local features	Recall, Precision, Root Mean Square Error
2. RepTree, Machine Learning and Euclidean distance [9]	Envisat images	Continues features such as roads, rivers	Efficiency and processing time
3. Multi image saliency analysis [10]	SPOT 5 images	ROI extraction such as clouds	Recall = 0.8, precision = 0.75, F-measure = 0.75
4. Digital Surface Models [11]	ZY-3 satellite images	Pixel and feature level extraction of urban scenes	Overall accuracy = 96%, kappa coefficient = 0.862
5. Reversible jump Markov chain Monte Carlo sampler [12]	Synthetic image	Extraction of rivers, channels and roads	Completeness = 98.8, correctness = 94% and quality = 92.9%

shows the different feature extraction methods and their performance measures.

Local feature extraction based on particular filter designs were used mostly for extracting features such as roads which were slowly replaced by boundary detection methods due to its better performance. But Machine Learning is the most prominent technique in use today. A framework combining machine learning and Euclidean distance is used for feature extraction [9]. It utilizes the satellite image as input and extracts features such as river. The main goal is to identify the feature blocks from the image by applying machine learning. Here, REPTree is utilized to develop trees by taking multiple iterations.

Saliency analysis effectively suppresses the background information such that the features are extracted without any unwanted features in the extracted image. In multispectral images, k-means clustering is employed to get the region of interest and saliency maps are obtained for the same. The same procedure is repeated for the panchromatic images using co-occurrence histogram thus improving the saliency of the selected regions and the saliency maps are determined. Now a fusion method is used so as to combine the saliency maps. The fusion technique is such that the unwanted background information is suppressed and the region of interest is enhanced to improve the image quality. A method to classify urban area on the basis of angular properties is used as part of high resolution images [11]. Three distinct Angular Difference Features (ADFs) are used to understand the scene at pixel based, feature based levels. The main reason for using the angular features is that certain structures such as buildings can be easily differentiated from lands. But the main drawback faced is the underestimation of high structures because of certain image disparities. In high resolution images, they exhibit ambiguous features. Here some matching errors are observed as part of shadows in the images.

2.3. Image Segmentation

Image segmentation involves division of the image into different segments such that objects and boundaries can be located [13]. Cuckoo search (CS) algorithm is applied for color image segmentation. A modification of the CS method using McCulloch’s method is suggested in Reference [14]. This method uses Levy flight in CS method. Most commonly applied heuristic algorithms such as Particle Swarm Optimization (PSO), Artificial Bee Colony optimization (ABC) and CS suffers from the limitation of the exponential increase in execution time with the increasing number of thresholds for segmentation. The main difference from the other models is that it uses McCulloch’s method to form stable random numbers. This work can be extended to incorporate multi-level threshold for better segmentation. Remote sensing images being purely illuminated can contain many ambiguous areas. Pixel based clustering finds wide application in such cases. Hopfield Neural Network clustering strategy is one such method. This method gives better result when large number of clusters is used.

Table 3. Image segmentation methods and their performance metrics.

Method	Test Images	Performance Metrics
1. Cuckoo Search, McCulloch’s method [14]	Pleiades satellite images	PSNR = 21.9833, MSE = 411.8573, FSIM = 0.9595
2. Markov Random Filed method [15]	QuickBird bands with four spectral bands	F-measure = 0.7427, Precision = 0.7170, Recall = 0.7704
3. Deep convolutional Neural Network [16]	IR images	Accuracy = 89.9%, Confusion matrix
4. Levy flight firefly algorithm [17]	Multiband satellite images from NASA	FSIM = 0.66278, PSNR = 13.7037, MSE = 3376.1616, SSIM = 0.92448
5. Graph based segmentation, Gabor filter [18]	Quickbird satellite images	Completeness = 92.5%, Correctness = 91%, Quality = 84.7%

shows various segmentation algorithms and their performance measures.

One of the most widely used segmentation technique is the histogram based thresholding approach. Bi-level thresholding involves selection of a single threshold for segmenting into two classes while multilevel thresholding involves selection of more than one thresholds for segmenting into multiple classes [19]. To select the threshold, the method optimizes criteria such as maximum between class variance and minimum classification error [20]. The performance comparison of both these methods suggests that the multilevel thresholding outperforms the bi level thresholding. The classification can be enhanced using a Markov Random Field (MRF) as explained in Reference [15]. The method uses lesser clusters for quantization, clustering and finding the likelihood. The classification performance is evaluated using Random forest Method where it is seen that even if the features are minimum, the classification can be good. MRF finds the local pixel interactions which helps the minimum distance classification. Supervised Random Forest method is further applied on each of the extracted segments.

Deep Convolution Neural Network (DCNN) for segmentation in high resolution satellite images is presented in Reference [16]. Here, a novel framework is formed by combining boundary detection with SEGNET encoder. SEGNET is a neural network based on encoder-decoder architecture for multiclass segmentation. It performs pixel wise segmentation on image data. But it suffers from the limitation that it is complex in its size due to the excessive size of the decoder, which is an issue for researchers using this framework [21]. It has also the disadvantage of blurry nature of extracted boundaries. A firefly algorithm based on fuzzy entropy function is also used [17]. Fuzzy entropy function computes difference in adjacent entropies. The threshold levels are formed based on the minimum fitness function and all the regions will have same entropy values.

The firefly algorithm is an effective search algorithm based on the behavior of unisex fireflies and its efficiency is based on the light intensity and the attraction between flies. Extraction of roads from an urban area scene is indeed difficult due to the complex building geometry. The Gabor filter enhances the image features to distinguish the textures in the image. It is followed by segmentation of the scene based on a graphical method [18]. This mainly involves two major steps where a graph representation of the segmented image is developed first followed by splitting and merging of similar segments based on color and shape. A new method using CS for segmenting color images is presented in Reference [22]. It uses Tsallis entropy for multilevel thresholding. This entropy function uses global property of the histogram of the image which is easier to implement. Also the entropy index acts as an adjustable parameter for tuning the threshold values. This method suffers from the limitation that the algorithm has very high complexity.

The main aim in the case of object based analysis is to decide which technique will be useful for image segmentation which further decides the overall image performance. Object based analysis can be assessed using two stages. Firstly image segmentation is done followed by classification of the segmented image. The performance evaluation can be quantitative or qualitative. Compared to unsupervised methods, supervised methods are mostly used for quantitative analysis. Qualitative analysis is highly subjective but easier. An assessment technique using Under Segmentation Error (USE) and Over Segmentation Error (OSE) is used which is calculated for each reference in the image [23]. Finally once the OSE or USE of each reference in the image is computed, it is summed up to get the overall performance metric for segmentation problems.

Some of the evaluation metrics for assessing the performance of image segmentation include pixel accuracy which is the percentage of pixels which were correctly classified, precision which is the correctness of the positive detections to ground truth and recall which is the completeness of positive predictions to ground truth.

2.4. Image Fusion

Image fusion involves combining two or more images to form a new image. With the advancement of multiresolution analysis, it is possible to extract high-frequency components and then inject into multispectral image [24,25]. The wavelet transform is commonly used for pan-sharpening. Apart from the wavelet transform, the other prominent frequency filtering methods used are High Pass Filter Additive Method (HPFA) [26], High–Frequency-Addition Method (HFA), High Frequency Modulation Method (HFM) [27]. Another category is image fusion using statistical methods such as Local Mean Matching (LMM) [28], Local Mean and Variance Matching (LMVM) [29], Regression variable substitution (RVS) and Local Correlation Modeling (LCM) [30].

Image fusion can be broadly classified as probabilistic fusion, statistical fusion, unmixing based fusion, semiphysical fusion and hybrid fusion [31]. Probabilistic fusion involves fusion with probability based approach such as Bayesian and maximum likelihood method. The main application of using this method is to analyze the statistical relationship between images containing varying features. A Bayesian method to fuse multispectral and panchromatic images is mainly used in pan sharpening to enhance the spatial resolution of the image [32]. Bayesian fusion is also applied to predict the air pollutant concentrations [33] and fusion of Synthetic Aperture Radar (SAR) images for flood detection. Statistical fusion involves statistical relation between pixels. This method is based on pixel based approach and spatial and temporal adaptive reflectance fusion method (STARFM) is the most prominent among them. This fusion produces an output image with high spatial and temporal resolution. It is applied to flood mapping and to monitor water quality [34]. Unmixing based fusion is mainly applied to enhance the spatial resolution of coarse images. Spatial and Temporal Reflectance Unmixing Model (STRUM) is one technique used in applications involving surface reflectance such as cloud masking and gap filling. Semiphysical fusion involves models with physical structures for image fusion. These methods can accommodate temporal changes due to surface variations. It is used in applications such as ocean color observations and atmospherically correct the satellite images. Hybrid fusion involves integrating different fusion techniques improve fusion accuracy.

There are some geometric analysis tools for image fusion which are based on curvelet, contourlet, wedgelet and Non Subsampled Contourlet Transform (NSCT). In contourlet transform, the shift-invariance is not present. The NSCT can overcome this at the tradeoff of high time cost. Shearlet transform is mathematically similar to wavelet. Analysis of geometric structure is extremely important in satellite images and hence Shift-Invariant Shearlet transform (SIST) is proposed for remote sensing image fusion [35]. The proposed method involves the fusion of the high frequency sub bands based on their similarities. Finally the fused image is obtained by inverse SIST and inverse Entropy Component Analysis (ECA). Quantitative analysis is carried out by computing Spatial Correlation Coefficient (SCC) and mean gradient (G). Higher the value of G, better the quality. From visual analysis it is inferred that spectral distortion is extremely less.

A framework that combining the advantages of wavelet transform and sparse representation is another method which is being used presently. First the intensity components in multispectral images is separated using Intensity-Hue-Saturation(IHS) transform and the wavelet transform is applied to generate a multiscale representation which contains the low and high frequency sub-images which are processed using varying techniques depending on the applications. Figure 2 shows the block diagram of image fusion.

One of the major limitations in image fusion is the presence of some noise components in the fused image which makes it visually unappealing. A spider optimization based algorithm produces two enhanced images, one with high contrast and the other with high peak signal-to-noise ratio (PSNR) image [36].

Table 4. Image fusion methods and their performance metrics.

Method	Image Dataset	Performance Metrics
		Mutual Information	QAB/F	Visual Information Fidelity
1. Image cartoon texture decomposition and sparse coding [37]	NIR and visible IKONOS images	------	0.357	0.3470
2. Contourlet Transform [38]	SAR and panchromatic images	9.342	-----	------
3. Expectation Maximization algorithm [39]	Landsat image set	----	0.589	----
4. Dictionary learning method [40]	Visible and IR image set	5.778	0.660	0.578

represents the various image fusion techniques and their performance metrics.

SAR images are usually affected by speckle noises. The main advantage of NSCT is the effective edge capturing in the fused image [38]. Multi-dimensional expansion of the image is an added attraction in this method. A high efficiency is obtained on using Maximum A Posteriori estimation for removing the noise components in the SAR image. After denoising, the SAR image is fused with the PAN image using an edge based fusion. The most prevalent limitation in fusion is the failure in capturing the smoothness among different contours.

Analyzing the correlation between satellite images is important to attain a high accuracy of fusion. The correlation coefficient is evaluated on actual and synthetic time series images in a pixel-by-pixel basis [41]. Then the spatial and temporal fusion is done. The performance comparison was done using root mean square error calculation for different satellite images. This technique was mainly used for the validation of time series satellite images. It suffers from the drawback that as the land cover type changes, the accuracy of this method is seen to be decreasing. A method for the scene classification by fusion of local and global features in the image is proposed in Reference [42]. The image is differentiated into dense regions and then the k-means clustering method is applied. An image fusion based on multilevel morphological decomposition is proposed which decomposes the images into varying layers with varying morphological component so that the details in the source images are preserved [43]. Image fusion also finds application in radiometric normalization and there are various surface reflectance based fusion approaches. One such method that uses MODIS data to predict the Landsat reflectance for different viewing angle is described in Reference [44]. A fusion approach to understand the landscape changes at spatiotemporal resolutions using Spatial and Temporal Adaptive Reflectance Fusion Models is proposed in Reference [45]. But this method has the limitation of finding changes in sub-pixel ranges.

There are many commonly used fusion algorithms. One such algorithm is the Ehler method, which is developed on the Intensity-Hue-Saturation (IHS) transform along with adaptive filtering. It is mainly done to enhance the spatial resolution in multi spectral images. First the spectral and spatial components are separated and then the spatial content is modified for the better enhancement of the images. Following this, Fast Fourier Transform is performed on PAN image. Another one is the wavelet transform fusion algorithm. Here the high frequency components are extracted from the PAN image and combined with the spectral information from the MS image. Then the PAN image is divided into low resolution image. These images are then replaced by the MS bands. Finally inverse wavelet is applied to each of them. University of New Brunswick fusion method uses the least square method to find the relationship of the gray values in PAN, MS and fused images.

Some of the evaluation metrics for assessing the performance of image fusion include Root Mean Square Error which finds the difference between the source image and the fused image, Spectral Angle Mapper which finds the spectral angle between source and fused image, Signal to Noise Ratio which finds the ratio between information and noise of fused image and Mutual Information which measures the similarity of image intensity between source and fused image.

2.5. Change Detection

Change detection is done to understand the changes in a particular area given that satellite data over that area is available easily [46]. Change detection techniques can be broadly classified as algebra based, transform based, Geographical Information System (GIS) and based on advanced models. Algebra based method uses mathematical operations to find the change in images. Most commonly used techniques involve Image Differencing, Change vector analysis (CVA) and Principal Component Analysis (PCA). Image differencing is mainly used to obtain details about the vegetation growth in a particular area. PCA finds use to study changes in water quality and vegetation types in an area. Transform based models use the transform to find the image changes. One such method is Tasseled Cap Transformation (TCT). TCT is mainly seen to be useful to understand the effect of deforestation in an area. Image differencing captures the changes in tropical areas efficiently while TCT can monitor changes in urban area. The change detection techniques are explained in details in Reference [47]. GIS gives a base for integrating and analyzing image data. It can integrate past and current image maps for classification and change detection. It has helped in finding the image changes. A change detection method involving GIS and remote sensing data for vegetation cover change analysis is presented in Reference [48]. Land cover change detection involving GIS is presented in Reference [49]. Advanced models for change detection involve spectral mixture and reflectance models. Here the reflectance value in the image is converted to physical parameter. From the physical parameters, information concerning remote sensing is extracted. Temporal unmixing for vegetation cover analysis employing spectral mixture model is proposed in Reference [50]. Digital surface and elevation models are digital representation of Earth’s surface. DSM is used in urban planning and vegetation analysis.AN urban change detection using DSM is proposed in Reference [51].

Maximum likelihood supervised classification technique was initially done to get separate classes of each land cover type [52]. After ensuring that classified groups are indeed correct, image differencing, statistical change detection using a probability matrix is used to differentiate between the changes over the past and present images in the area. The maximum likelihood finds the different land types using a combination of past experience, the analysis of previous works and the satellite image maps. Change detection methods have the ability to predict the changes in time series images [53]. One such technique uses the Breaks For Additive Seasonal and Trend or BFAST method to find many changes in time series images by getting the magnitude and direction of change [54].

The two main parameters evaluated are mean and variance of the data to calculate the probability that a pixel belongs to a particular class. Main disadvantages of maximum likelihood is that the pixels remain misinterpreted and is placed in another class to which it does not belong. So the overall efficiency of the process can get reduced. A method called the Multi-Step Image Matching (MSIM) utilizes fast clustering, image detection and boundary matching for detecting the changes in disaster prone areas [55]. A method is developed to detect the spatial and temporal changes in a satellite image using a graph based method [56]. Usually classifier techniques are mainly used in these applications. But they cannot provide enough information regarding the temporal changes in the image. The graph based technique uses object based analysis. The quantitative analysis is done through the calculation of correlation coefficient.

Some of the evaluation metrics for assessing the performance of image change detection include Overall Accuracy (OA) which is the ratio of the changed and unchanged pixels which are correctly detected, False Alarm (FA) which is the ratio of the unchanged pixels which are found as changed ones and Missed Alarms (MA) is the ratio of changed pixels which are identified as unchanged ones.

2.6. Image Compression

The hardware resources available onboard in a satellite is limited and the satellite images are very large, which necessitates the need for satellite image compression [57,58,59]. This reduces onboard data storage and the bandwidth required to transmit the image from the satellite to the ground station as there is only a limited time during which the satellite passes over the ground station. Hence, it is an important technique which will reduce the dependence on the onboard resources without compromising the image quality and the information content which is required at the ground station for analysis. Image quality is a matter of concern since the compressed images need to be reconstructed at the ground station without missing relevant information.

Table 5. Image compression methods and their performance metrics.

Method	Test Images	Performance Metrics
1. DWT-DCT [60]	Landsat 8 images	PSNR, Compression Ratio
2. DWT, entropy encoder [61]	ETM+ satellite multispectral images	PSNR, Spectral Angle Mapper(SAM)
3. Content based compression [62]	SAR image	PSNR, Mean Absolute Error(MAE)
4. Adaptive dispersion clustering based compression [63]	AMSR-2 dataset	Mean Prediction Error Variance Ratio(MPEVR), Root Mean Square Prediction Error(RMSPE)

gives the various compression techniques and the performance metrics for each of the techniques.

Fractal compression methods have shown great potential in improving the compression ratio but they are not much preferred due to their computational complexity and distortions which affect the image quality. Another commonly used method of image compression is the differential method. Here the image pixels are predicted depending on the previous pixel values and then calculating the error in the pixel values. This method is very simple and less complex. But the main drawback is its very low compression ratio. A method of compression based on removing the sub bands in a multispectral image is proposed in Reference [61]. The main problem in multispectral image is based on how to store the different multispectral bands and how these are compressed effectively. The removed sub bands are found using the correlation coefficients between the various bands.

A lossless compression method where the spatial and spectral domains are decorrelated and the side information of the coefficients computed recursively are removed to improve compression [64]. Due to the huge size of the aurora spectral images, they are not transmitted as a single image but in the form of frames due to the limited bandwidth of the transmission channel. Here a Differential Pulse Code Modulation (DPCM) based scheme is proposed for the lossless compression of the image. Adaptive Spatial Dispersion Clustering (ASDC) is used in the spatial data compression [63]. This method outputs a new data which acts as the input to a spatial prediction method called Fixed Rank Kriging which uses only a subset of the entire data since using the entire data can increase the complexity of computation. In order to enhance compression rates and reduce the complexity of these methods, novel transform based compression method called Discrete Tchebichef transform or DTT is used [65]. It is an orthogonal transform. In this method, the DTT matrix obtained into a single-row elementary matrix such that there are minimum rounding errors.

Some of the evaluation metrics for analyzing compression include compression ratio which is the ratio of the number of bits in the original image to the number of bits used in the compressed image, mean square error for lossy compression which is mean square error between the source and the reconstructed image and structural similarity index which defines the variation in structure between the source and the distorted image.

2.7. Image Classification

Efficient image classification methods are necessary for categorizing the satellite images [66,67]. Image classification is a method of pattern recognition in which images/pixels are classified based on some similarity measures. It is divided into supervised, unsupervised and post-classification approaches. Supervised method needs the training data to teach the classifier and recognize the instances of necessary details in the image. It is divided into parametric and non-parametric methods where parametric method use algebraic method for assignment to classes. Bayesian and decision tree come under this category. Non-parametric method is used when density function is not present. K-Nearest Neighbor and Logical Regression come under this category [68].Unsupervised method finds group of similar instances with no information about class label. Unsupervised methods can be hierarchical and partition clustering. Image classification of satellite images suffer from two major drawbacks. The first is the issue of mixed pixels and second is the difficulty in handling large amount of data in the satellite images. Random Forest Classifier is popularly used for land use and land cover classification. It is an ensemble learning approach based on a combination of tree predictors. It combines the votes from different trees to decide the final class. A random forest classification of wetland covers is presented in Reference [69]. Geometric and spectral features were extracted from the multi-sensor images were optimized and a random forest model is created for classification. A random forest classifier with optimized parameters for land cover mapping is presented in Reference [67]. Here Genetic algorithm optimizes the number of features selected and decision trees to develop optimal land cover classification. Mixed pixel classification is a significant problem in remote sensing. Spectral unmixing finds the composition of each pixel to extract information which is also referred to as soft classification. Spectral unmixing in image classification to address the mixed pixel problem is proposed in Reference [70]. Here spectral unmixing is employed to find the fractional abundances of various classes. A hybrid unmixing for land cover classification is presented in Reference [71]. Supervised and unsupervised unmixing is employed for multispectral image classification. A Multiple End member Spectral Mixture Analysis (MESMA) for urban land cover analysis is explained in Reference [72]. A single class is described using multiple end members. Radial Basis Function Neural Network (RBFNN) is prominent neural network with many tunable parameters [73]. The main advantage of this method is its immunity to noise signals. A post classification algorithm for those pixels which are wrongly classified is also developed using the Classification Errors Correction (CEC) [74]. In the first stage the adjacent correlated bands are found and are merged so that the number of correlated bands are reduced. This is done using Discrete Wavelet Transform (DWT). Using the pixel averaging the bands are merged. A deep feature learning for satellite image classification is proposed in Reference [75]. Initially the image is separated into different scales and images in each scale are used to train DCNN. The training process is speeded up using spatial pyramid pooling (SPP). The SPP nets use different image scales and they have similar weight parameters.

A Bayesian network classifier model is done on the satellite image to obtain the presence of clouds [76]. For the cloud classification problem, a set of attributes corresponding to the cloud type are considered. In order to understand how effectively the classification can perform is analyzed using k-fold cross validation. It uses all the observations available both for learning and evaluation. It is necessary to assess how accurate the applied classification method is. This is done using an integration of fuzzy evaluation and Dempster-Shafer theory is proposed (FSE-DST) in Reference [77]. First the object based classification of the image is done. This technique is important for efficiently developing the object based image classification and helps in the further research process for obtaining more accurate results.

2.8. Image Feature Detection

There has been an increase in the satellite images which helps to understand and analyze various applications. Feature recognition is gaining importance with the advancement in deep learning.

Table 6. Feature Detection methods and their performance metrics.

Method	Test Images	Application	Performance Metrics
1. Fuzzy logic based detection	Hyperspectral image [78]	Edge detection	PSNR, MSE
2. Deep Convolutional Neural Network [79]	QuickBird VHR image	Informal settlement detection	Overall accuracy
3. Invariant pixel detection, PCA [80]	Landsat 7 ETM+ images	Cloud detection	Accuracy and F-measure
4. Active contour model [81]	Google Earth images	Shadow detection and height estimation	Precision and recall

represents the different feature detection methods and performance metrics.

The ability to differentiate features such as buildings, roads, vegetation and so forth is important in areas such as environmental analysis, urban monitoring and finding the disaster affected areas and so on. In order to find correspondences among a set of images, where feature correspondences between two or more images are needed, it is necessary to identify a set of salient points in each image. In a classification task, feature descriptors are matched with all trained image features and the trained image giving maximum correspondence is considered the best match. The use of global and texture features are successful for finding similar images in a database, while the local structure oriented features are considered adequate for classification. Non-linear features such as urban settlements, oil spills and so forth are difficult to be classified and identification of salient points in the no-linear features is essential for efficient classification. Hence feature detection plays a very significant role in image classification and proper detection of features guarantee proper image classification. Edge detection is very important in image processing. Different edge detection techniques used are Preweitt, Laplacian and Laplacian of Gaussian. The main limitation of this method is the fixed edge thickness and difficulty in threshold implementation. An alternate technique is linear time invariant filter which considers edge as a change in the intensity of the grey scale pixel and is computationally effective. Edge detection using fuzzy logic is developed in Reference [78]. For the cloud detection using Digital Elevation Model (DEM), a Digital Surface Model (DSM) is developed through image matching [82]. The developed DSM is compared with the DEM and the elevations are compared to get the seed point of the clouds. The cloudy areas are classified using the information about the image and height.

Another important factor which affects the image quality is the presence of jitter in the image. Hence it is necessary to detect them and compensate its presence in the image. A model based on the displacement data is used in the detection of the high altitude jitter in the satellite image [83]. Bidirectional Kalman filter and smoothing filter are used to reduce the effect of the detected jitter and a reimaging model is used to get the corrected image. Fully convolutional network (FCN) has the capability to learn information regarding pixel dependencies [79]. Distinguishing between the formal and informal settlements is done on the basis of morphological characteristics. To detect informal settlements, the spatial information contained in the image need to be extracted. Wavelet transforms are mainly used for the same.

An efficient oil spill detection in sea is used by the International Maritime Organization to evaluate the trade routes through sea. A feature selection using machine learning algorithm is employed for this [84]. The SAR images can provide images of oil spills on the surface of the ocean with wide coverage and good clarity. A novel transfer learning method is used for vehicle detection from aerial images [85]. This transfer learning method is based on a super resolution algorithm which learns the sparse coefficients which are highly correlated between the patches that have high and low resolution. A linear SVM based search technique is used to detect the vehicle positions. But the matching performance is very less since it is difficult to select similar image sections. This limitation is overcome using dictionary learning and sparse coding. It creates low-resolution sections with the help of most similar high-resolution sections utilizing shared coefficients obtained through dictionary learning.

On comparing the different image processing techniques, it can be assessed that the efficiency of the techniques depends on the application, type of noise present in the image and the type of image used. The accuracy of feature extraction, detection, fusion and change detection affect the extend of segmentation accuracy and hence the result. The presence of varying noise types decide the enhancement technique which affect the final outcome. The complexity of the satellite images in terms of edges, contours and textures paves way towards importance of general feature extraction. By adopting techniques suitable for application, the final image processing outcome can be modified to get the required result.

3. Performance Analysis

Quantitative and qualitative evaluation are available for analyzing the performance of different methods. Various performance metrics are calculated and compared with existing methods to evaluate the improvement in performance. A few of the computed parameters are explained in detail which include Peak Signal to Noise Ratio (PSNR), Mean Square Error (MSE), Feature Similarity Index (FSIM) and Structural Similarity Index (SSIM).

PSNR finds the quality of the final image. Equation (1) computes PSNR as:

$$PSNR={\log }_{10}\left(\frac{255\ast 255}{MSE}\right).,$$

(1)

where MSE is the Mean Square Error. Equation (2) calculates the MSE as:

$$MSE=\frac{1}{MN}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N{[I(i,j)-I^{\prime }(i,j)]}^2}},$$

(2)

where I(i,j) is the original image and I′(i,j) represents the thresholded image. PSNR and MSE are mainly used to analyze the quality of compressed and reconstructed image. SSIM measures the structural similarity between the source and final image while FSIM measures the feature similarity between the source and final image. There are two features that can be viewed in FSIM. They are Phase congruency and Gradient magnitude. Phase congruency (PC) is a dimensionless quantity which is significant in local structure map and is a primary feature in FSIM. PC is contrast invariant with no effect of contrast information on human visual system. Gradient magnitude (GM) is another important feature in FSIM. PC and GM are complementary to one another in describing the image local quality. It is computed using Equation (3) as

$$FSIM=\frac{{\displaystyle \sum _{x\in X}{S}_L(x)P{C}_m(x)}}{{\displaystyle \sum _{x\in X}P{C}_m(x)}},$$

(3)

where x represents the whole image, S_L(x) denotes the similarity in the two images and PC_m is the phase congruency map.

SSIM is calculated using Equation (4):

$$SSIM=\frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)},$$

(4)

where μ_x and μ_y represents the sample means of x and y, respectively; σ_x and σ_y give the sample variances of x and y, respectively; and σ_xy describes the sample correlation coefficient between x and y and x and y are local windows in the input images. SSIM and FSIM values vary between 0 and 1 with a value 1 representing best similarity. These values can analyze the extend by which the image has recovered after the image is distorted.

Following are the performance measures for classification problem with number of correct and incorrect predictions tabulated in terms of count values divided for each classes. A matrix based representation called the confusion matrix is developed and it can be used to analyze the performance of the classifier. Confusion matrix gives the performance of classifier with respect to a test data. It is a matrix which is indexed in a dimension by the true class of the object and in the other by the class assigned by classifier [86]. It gives the relation between observed and predicted values in classification. It is using this matrix it can be found whether mislabeling of classes has occurred and the predictions are correct or not [87]. On arranging in matrix form, correct predictions are given along the main diagonal while the incorrect predictions are along the non-diagonals. This makes it simpler to check where the predictions are wrong. The four main error metrics computed from the matrix are True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN). True Positive is when the sample is correctly classified as positive, TN is when the sample is correctly classified as negative, FP or False Positive is when the sample is incorrectly classified as positive and FN or False Negative is when the sample is incorrectly classified as negative [88]. Consider an example of confusion matrix for a classification problem with multiple classes. Here the land cover changes in a watershed area obtained using Landsat 7 ETM+ images of 2007 and 2018 are presented in Reference [89]. The matrix is developed for land cover change detection problem where there are six types of land cover classes identified namely—forest, water body, agricultural land, grassland, bare land and urban. Suppose that we need to assess the number of pixels that are correctly and incorrectly classified under each category. On classifying 1470 pixels, water body is correctly classified for 60 pixels, forest is correctly classified for 20 pixels, agricultural land correctly classified for 20 pixels, grass land correctly classified for 290 pixels, urban area correctly classified for 90 pixels and barren land is correctly classified for 750 pixels. Remaining pixels are incorrectly classified and responsible for the misclassification errors associated with the accuracy of land cover classification. In the case of water body, TP contains cases with water body correctly classified as water body, TN contains cases with all other classes correctly classified, FP contains cases with all other classes incorrectly classified as water body and FN contains cases with water body incorrectly classified as other categories. The example can be represented in a confusion matrix as shown in

Table 7. Confusion matrix example.

	Actual Class
	Water	Forest	Agriculture	Grass	Urban	Bareland	Total
Water	60	0	0	0	0	0	60
Forest	0	20	0	0	0	0	20
Agriculture	0	0	20	0	0	0	20
Grass	30	0	10	290	10	0	340
Urban	0	0	0	0	90	0	90
Bareland	10	20	60	50	50	750	940
Total	100	40	90	340	150	750	1470

.

Overall accuracy can be obtained by dividing the total number of correct pixels (diagonal elements in the matrix) by the total number of pixels in the confusion matrix. To compare two classified data products, the observed frequencies in a confusion matrix are assumed to be multinomial distribution with the complete random nature of the elements in the confusion matrix [90]. With the availability of true data reference, the randomness within fails. If the reference data are found along column and this data is considered as truth, the elements which belong to a particular type can be correctly classified or confused with other types and they will be located in same column but never in different columns. The randomness within the multinomial is not at all possible. For this, Quality Control Column Set (QCCS) is developed. The main advantage is that it enables to state quality specifications for each category and test if they are satisfied [91]. Accuracy, Recall, precision and F-measure are also derived from the confusion matrix [92]. Accuracy is measured as the ratio of the correctly predicted observations to the total ones. It is an efficient parameter mainly when the dataset is symmetric. Recall is a measure of the number of positive class predictions out of all positive examples in the database. For an imbalanced classification problem having just two classes, recall is the ratio of the number of true positives to the total number of true positives and false negatives. In an imbalanced classification problem with multiple classes, recall is computed as the ratio of sum of true positives across all classes to the sum of true positives and false negatives across all classes. Its value ranges between 0 and 1 with the best value considered to be 1.

Precision is a measure of the number of positive class predictions that actually belong to the positive class. For an imbalanced classification problem having two classes, precision is the ratio of the number of true positives to the total number of true positives and false positives. For an imbalanced classification problem with multiple classes, precision is computed as the ratio of the sum of true positives across all classes to the sum of true positives and false positives across all classes. In an imbalanced classification, the distribution of data across the known class is biased. It is extremely challenging in predictive modeling as most of the machine learning methods are developed considering the assumption that there are equal number of data in each class. Thus imbalanced classification is a problem when there is unequal class distribution in training data. Many real life classification problems such as spam and fraud detection involve imbalanced distribution. Equations (5) and (6) represent Recall and Precision respectively.

$$\mathrm{Re}call=\frac{TP}{TP+FN},$$

(5)

$$\mathrm{Pr}ecision=\frac{TP}{TP+FP},$$

(6)

where TP or True Positive is when the sample is correctly classified as positive, FP or False Positive is when the sample is incorrectly classified as positive and FN or False Negative is when the sample is incorrectly classified as negative.

F-measure is a parameter which decides the image accuracy. Equation (7) computes F-measure as:

$$F\_measure=\frac{2.\mathrm{Pr}ecision.\mathrm{Re}call}{\mathrm{Pr}ecision+\mathrm{Re}call}.$$

(7)

A high value of recall and precision guarantees a high F-measure. Precision, recall and F-measure are applicable mainly in pattern recognition and classification.

The significance of the confusion matrix lies in the fact that it can point out the errors made by the classifier.

ROC or Receiver Operating Characteristic curve is a graph representing the classification performance. Area Under Curve (AUC) is a measure of the two-dimensional area under ROC curve. ROC is plotted between True Positive Rate (TPR) and False Positive Rate (FPR). TPR is the ratio of the positive samples that are correctly considered as positive w.r.t all positive samples. FPR is the ratio of negative samples which are misclassified as positive w.r.t all negative samples. AUC ranges between 0 and 1 and closer the value to 1, better is the classification performance. Similar to ROC curve is precision-recall curve which is focused towards the performance of the minority class. It can analyze a single classifier but challenging for classifier comparisons. Precision-Recall AUC can also be computed and is useful for imbalanced classification.

4. Discussion

Remote sensing data comprises of spatial, spectral and temporal resolution. Spectral statistics is widely made use in remote sensing image classification. The main aspect in accuracy of ground object is spatial resolution. Temporal resolution helps to generate land cover maps for environmental planning, land use change detection, urban planning and so forth.

Image enhancement improves the image quality and information content before further processing is carried out. Some commonly used techniques include contrast enhancement, spatial filtering and so forth. The linear contrast enhancement is best applied to remote sensing images with Gaussian or near-Gaussian histograms where all the brightness values generally fall within a single narrow range of the histogram. But there occurs a situation mainly for scenes with large land and water bodies. Non-linear contrast enhancements can be applied on low-contrast imagery of which histogram equalization is a prominent technique. Non fusion based enhancement gives low spatial information but with high computational complexity. To overcome the limitation of high complexity, fusion based enhancement is used.

Pixel-based feature extraction methods are used to extract low-level features, which do not take into account the information about spatial relationships. Low-level features are extracted directly from the raw, noisy pixels with edge detection being the most commonly used algorithm. Object-based approaches are used to get high-level features, which represent shapes in images that are detected irrespective of illumination, translation, orientation and scale. High-level features are extracted depending on the information from low-level features. High-level feature extraction is mainly used for automated object detection and extraction. Depending on the spatial resolution of the source image, different segmentation algorithms are used. For low to medium resolution imagery, clustering algorithms are better choice, however for high resolution imagery, multiresolution segmentation offers better results. On a high spatial resolution image, detailed geometric features can easily be identified, while the multispectral images contain richer spectral information. The capabilities of the images can be enhanced if the advantages of both high spatial and spectral resolution can be integrated into one single image. The detailed features of such an integrated image thus can be easily recognized and will benefit many applications, such as urban and environmental studies.

In change detection, it is difficult to find the method which is suitable for finding the changes which have occurred. Selection of suitable technique for change detection is affected by various factors like variation of physical characteristics of features with time, improper registration of the images, effect of cloud/haze and so forth and it is quite a difficult task to find a single method since the nature of problem determines which method is best suited. Techniques like image differencing and image ratioing can be used only when the change and no-change information is necessary. If a detailed matrix is required, post classification change detection is a good choice. Image-classification techniques, either pixel-based or object-based, are used for converting an image into a thematic map. The creation of digital surface and terrain models (DSMs/DTMs) from satellite imagery has become a common method to analyze the structure and development of vegetation and geomorphological landforms.

Recently, image processing through quantum computation also called quantum image processing (QIP) and application of machine learning through quantum computation or quantum learning (QL) have provided a new perspective to large-scale image processing. QIP uses quantum properties to encode images for improving storage and time efficiency of certain operations like image rotation. Thus quantum algorithms are being widely extended to improving digital or conventional image-processing applications and missions.

5. Conclusions

In this study, a detailed review of the image processing techniques for satellite image analysis is presented. This work attempts to provide an insight into the importance of image processing techniques in remote sensing applications. A roadmap of the research activities in various image processing stages have been explained with relevance to remote sensing applications. With the emergence of technology and frequent availability of satellite imagery, image processing in remote sensing is a topic of growing interest. The survey explores various image processing techniques and points out the advantages and limitations associated with these methods. Further the techniques specific to various applications are discussed and the performance measures are explained. Low contrast of satellite imagery, improper threshold selection for image segmentation, misinterpretation of image pixels as part of change detection are some of the challenges in image processing. The complexity of satellite image further poses a difficulty in real world classification applications and can be solved by involving computationally intelligent paradigms like machine learning. In current scenario, machine learning is an evolving research area with its ability to make data driven decisions effectively and learn and perform intelligently. The quality of the input image and the complexity of image features are some factors responsible for deciding the image processing technique to be applied. Researches are currently being extended into hybridized image processing techniques to improve the robustness of the existing techniques. The future work can be extended to apply the explained techniques in various practical areas in remote sensing and extending the implementation of quantum algorithms for remote sensing applications.