Figure 1.
Diagram of TomoSAR result using very few tracks for building. The blue points represent the estimated scatterers using three tracks, and the orange line indicates the ideal surface. There will inevitably be large errors in elevation inversion, resulting in the fuzzy structure of the building and inaccuracy in estimation of height.
Figure 1.
Diagram of TomoSAR result using very few tracks for building. The blue points represent the estimated scatterers using three tracks, and the orange line indicates the ideal surface. There will inevitably be large errors in elevation inversion, resulting in the fuzzy structure of the building and inaccuracy in estimation of height.
Figure 2.
Flowchart of proposed method. The proposed method is composed of two main modules. The data generation module explains the generation of the super-resolution dataset that contains the paired low-quality and high-quality slice sets. The CGAN module illustrates the dominant compositions of the CGAN model and the data flowpath.
Figure 2.
Flowchart of proposed method. The proposed method is composed of two main modules. The data generation module explains the generation of the super-resolution dataset that contains the paired low-quality and high-quality slice sets. The CGAN module illustrates the dominant compositions of the CGAN model and the data flowpath.
Figure 3.
Diagram of TomoSAR imaging geometry. TomoSAR expands the spatial resolution in the elevation direction by coherent antenna phase centers (APCs), represented as a series of black points.
Figure 3.
Diagram of TomoSAR imaging geometry. TomoSAR expands the spatial resolution in the elevation direction by coherent antenna phase centers (APCs), represented as a series of black points.
Figure 4.
Flowchart of TomoSAR imaging procedure. The procedure contains a series of operations. Firstly, the image registration ensures the azimuth-range units in different coherent SAR images related to the same scatterers. Secondly, the channel imbalance calibration compensates the phase errors among channels. Thirdly, sparse recovery methods are applied to invert the elevation position of scatterers. Finally, the coordinate system is transformed from radar system to the geodetic system.
Figure 4.
Flowchart of TomoSAR imaging procedure. The procedure contains a series of operations. Firstly, the image registration ensures the azimuth-range units in different coherent SAR images related to the same scatterers. Secondly, the channel imbalance calibration compensates the phase errors among channels. Thirdly, sparse recovery methods are applied to invert the elevation position of scatterers. Finally, the coordinate system is transformed from radar system to the geodetic system.
Figure 5.
Flowchart of data generation. The final output is the paired super-resolution dataset composed of low-quality and high-quality slice sets. The low-quality set of low-SNR and low-resolution contains range-elevation 2D binary slices, which is generated from three tracks. In contrast, the high-quality set uses all tracks, and has high SNR and resolution. The binary operation means that the value set is 1 if a scatterer is estimated in this position. All slices are in radar coordinate system.
Figure 5.
Flowchart of data generation. The final output is the paired super-resolution dataset composed of low-quality and high-quality slice sets. The low-quality set of low-SNR and low-resolution contains range-elevation 2D binary slices, which is generated from three tracks. In contrast, the high-quality set uses all tracks, and has high SNR and resolution. The binary operation means that the value set is 1 if a scatterer is estimated in this position. All slices are in radar coordinate system.
Figure 6.
Flowchart of the CGAN module. The CGAN consists of two models named generator (G) and discriminator (D). The generator produces a fake result as similar as possible to the ground truth to make the discriminator believe the generation is the truth. On the contrary, the discriminator is used to distinguish the fake result and ground truth. After iterations, the generator will be able to generate a refined result which is hard to tell from the corresponding ground truth. Besides this, the content loss between the generated result and ground truth is also considered avoiding position bias.
Figure 6.
Flowchart of the CGAN module. The CGAN consists of two models named generator (G) and discriminator (D). The generator produces a fake result as similar as possible to the ground truth to make the discriminator believe the generation is the truth. On the contrary, the discriminator is used to distinguish the fake result and ground truth. After iterations, the generator will be able to generate a refined result which is hard to tell from the corresponding ground truth. Besides this, the content loss between the generated result and ground truth is also considered avoiding position bias.
Figure 7.
Network structure of the generator. The generator is composed of three dominant parts: downsampling compression, feature extraction, and upsampling reconstruction. The downsampling compression part compresses the data dimension and expands the feature dimension from 64 to 256. The feature dimension is indicated by the number on the left side of blocks, such as n128. The feature extraction part is composed of nine stacked blocks based on res-net structure, which has capability of digging out high-dimensional features of data. The upsampling part decreases the feature dimension and reconstructs the data dimension by deconvolution (TransposedConv) layers.
Figure 7.
Network structure of the generator. The generator is composed of three dominant parts: downsampling compression, feature extraction, and upsampling reconstruction. The downsampling compression part compresses the data dimension and expands the feature dimension from 64 to 256. The feature dimension is indicated by the number on the left side of blocks, such as n128. The feature extraction part is composed of nine stacked blocks based on res-net structure, which has capability of digging out high-dimensional features of data. The upsampling part decreases the feature dimension and reconstructs the data dimension by deconvolution (TransposedConv) layers.
Figure 8.
Network structure of the discriminator. The network contains four convolutional layers increasing the feature dimension from 64 to 512, and finally becomes 1 to indicate the truth possibility of a small area in the receptive field. In the first four layers, the BatchNorm model is inserted into layers to accelerate the convergence and the LeakyReLU model is used as the activation function.
Figure 8.
Network structure of the discriminator. The network contains four convolutional layers increasing the feature dimension from 64 to 512, and finally becomes 1 to indicate the truth possibility of a small area in the receptive field. In the first four layers, the BatchNorm model is inserted into layers to accelerate the convergence and the LeakyReLU model is used as the activation function.
Figure 9.
Optical and intensity SAR images of YunCheng data. Panel (a) is the optical image of target scene including nine buildings in total. Each of the buildings is indicated using rectangles and numbered in the top-left corner. Panel (b) is the SAR image covering the same scene. The buildings in the SAR image are indicated with rectangles and are numbered correspondingly. Besides, two buildings marked with red rectangles at the bottom-right of the images are selected as training set. The other buildings with white rectangles are the testing set. Moreover, buildings #3, #4, and #5 are strongly overlapped in the SAR image and buildings #6, #7, #8, and #9 are nonoverlapped buildings. Slice 1 and slice 2 are selected at two azimuth positions to explain the results by different methods of overlapped and nonoverlapped buildings.
Figure 9.
Optical and intensity SAR images of YunCheng data. Panel (a) is the optical image of target scene including nine buildings in total. Each of the buildings is indicated using rectangles and numbered in the top-left corner. Panel (b) is the SAR image covering the same scene. The buildings in the SAR image are indicated with rectangles and are numbered correspondingly. Besides, two buildings marked with red rectangles at the bottom-right of the images are selected as training set. The other buildings with white rectangles are the testing set. Moreover, buildings #3, #4, and #5 are strongly overlapped in the SAR image and buildings #6, #7, #8, and #9 are nonoverlapped buildings. Slice 1 and slice 2 are selected at two azimuth positions to explain the results by different methods of overlapped and nonoverlapped buildings.
Figure 10.
3D reconstruction results of all-track and three-track TomoSAR. Panel (a) is the 3D height distribution of all-track TomoSAR scatterers. The buildings can be easily distinguished. The structures of buildings are refined. Panel (b) is the normalized strength distribution of all-track TomoSAR. The dominant scatterers mainly locate at the surface of buildings. Panel (c) is the height distribution of three-track TomoSAR. The structures of buildings are fuzzy. Besides, there are also lots of artifacts and outliers. Panel (d) is the normalized strength distribution of three-track TomoSAR. Compared with all-track results, the strength distribution is worse with lots of powerful artifacts and outliers, which makes the height distribution of buildings fuzzy and declines the quality of reconstruction.
Figure 10.
3D reconstruction results of all-track and three-track TomoSAR. Panel (a) is the 3D height distribution of all-track TomoSAR scatterers. The buildings can be easily distinguished. The structures of buildings are refined. Panel (b) is the normalized strength distribution of all-track TomoSAR. The dominant scatterers mainly locate at the surface of buildings. Panel (c) is the height distribution of three-track TomoSAR. The structures of buildings are fuzzy. Besides, there are also lots of artifacts and outliers. Panel (d) is the normalized strength distribution of three-track TomoSAR. Compared with all-track results, the strength distribution is worse with lots of powerful artifacts and outliers, which makes the height distribution of buildings fuzzy and declines the quality of reconstruction.
Figure 11.
Results of nonoverlapped buildings #6, #7, #8, and #9 reconstructed by all-track and three-track TomoSAR at slice 1 position. Panel (a) is height map of all-track TomoSAR. The identity numbers of buildings are placed near the corresponding ones. Panel (b) is the normalized strength map of all-track TomoSAR. Panel (c) is the height map of three-track TomoSAR and (d) is the normalized strength map of three-track TomoSAR. The heights of four buildings are estimated from the strength maps and indicated by orange lines at the top of buildings. Compared with all-track TomoSAR results, the results of three-track TomoSAR have more artifacts and outliers with stronger power. The structures become blurry and are affected by the powerful multipath scattering (marked with red circles).
Figure 11.
Results of nonoverlapped buildings #6, #7, #8, and #9 reconstructed by all-track and three-track TomoSAR at slice 1 position. Panel (a) is height map of all-track TomoSAR. The identity numbers of buildings are placed near the corresponding ones. Panel (b) is the normalized strength map of all-track TomoSAR. Panel (c) is the height map of three-track TomoSAR and (d) is the normalized strength map of three-track TomoSAR. The heights of four buildings are estimated from the strength maps and indicated by orange lines at the top of buildings. Compared with all-track TomoSAR results, the results of three-track TomoSAR have more artifacts and outliers with stronger power. The structures become blurry and are affected by the powerful multipath scattering (marked with red circles).
Figure 12.
Results of nonoverlapped buildings #6, #7, #8, and #9 reconstructed by CGAN and nonlocal methods using three tracks at slice 1. Panel (a) is the height map of CGAN. Panel (b) is the normalized strength map of CGAN. Panel (c) is the height map of the nonlocal algorithm. Panel (d) is the normalized strength map of the nonlocal algorithm. The nonlocal algorithm can remove the artifacts and outliers by increasing the SNR. However, it is still affected by the multipath scattering, marked with red circles. In contrast, the proposed CGAN method generates a higher quality result by suppressing more artifacts and outliers. In addition, the multipath scattering is also well suppressed so that the structures are much clearer. The height estimations are also labeled with orange lines.
Figure 12.
Results of nonoverlapped buildings #6, #7, #8, and #9 reconstructed by CGAN and nonlocal methods using three tracks at slice 1. Panel (a) is the height map of CGAN. Panel (b) is the normalized strength map of CGAN. Panel (c) is the height map of the nonlocal algorithm. Panel (d) is the normalized strength map of the nonlocal algorithm. The nonlocal algorithm can remove the artifacts and outliers by increasing the SNR. However, it is still affected by the multipath scattering, marked with red circles. In contrast, the proposed CGAN method generates a higher quality result by suppressing more artifacts and outliers. In addition, the multipath scattering is also well suppressed so that the structures are much clearer. The height estimations are also labeled with orange lines.
Figure 13.
Results of overlapped buildings #3 and #4 reconstructed using three and all tracks at slice 2 position. Panel (a) is the height map of all-track TomoSAR. The identity number of buildings is placed near the corresponding ones. Panel (b) is the normalized strength map of all-track TomoSAR. Panel (c) is the height map of three-track TomoSAR. Panel (d) is the normalized strength map of three-track TomoSAR. The buildings #3 and #4 are overlapped in the SAR image. From results of three-track TomoSAR, the overlapped two buildings cannot be distinguished from each other; the structure of building #3 is too blurry to tell it apart from building #4. Moreover, the top of building #3 is hard to determine, and it is impossible to estimate its height. The height estimation is labeled with orange lines.
Figure 13.
Results of overlapped buildings #3 and #4 reconstructed using three and all tracks at slice 2 position. Panel (a) is the height map of all-track TomoSAR. The identity number of buildings is placed near the corresponding ones. Panel (b) is the normalized strength map of all-track TomoSAR. Panel (c) is the height map of three-track TomoSAR. Panel (d) is the normalized strength map of three-track TomoSAR. The buildings #3 and #4 are overlapped in the SAR image. From results of three-track TomoSAR, the overlapped two buildings cannot be distinguished from each other; the structure of building #3 is too blurry to tell it apart from building #4. Moreover, the top of building #3 is hard to determine, and it is impossible to estimate its height. The height estimation is labeled with orange lines.
Figure 14.
Results of overlapped buildings reconstructed by nonlocal algorithm and proposed CGAN methods. Panel (a) is the height map of the nonlocal algorithm. Panel (b) is the normalized strength map of the nonlocal algorithm. Panel (c) is the height map of the proposed CGAN method. Panel (d) is the normalized strength map of the proposed method. Generally, the nonlocal algorithm can divide two overlapped buildings with an obvious interval. However, there are still some artifacts and outliers. In contrast, the results of the proposed method can greatly distinguish between two buildings with a large interval. Additionally, the roofs of two buildings are clear. The height of building #3 estimated in the height map of the proposed method is closer to ground truth. The height estimation of building #3 in the nonlocal result is severely different from that of all-track results, which is probably affected by the multipath scattering. The height estimation is labeled with orange lines.
Figure 14.
Results of overlapped buildings reconstructed by nonlocal algorithm and proposed CGAN methods. Panel (a) is the height map of the nonlocal algorithm. Panel (b) is the normalized strength map of the nonlocal algorithm. Panel (c) is the height map of the proposed CGAN method. Panel (d) is the normalized strength map of the proposed method. Generally, the nonlocal algorithm can divide two overlapped buildings with an obvious interval. However, there are still some artifacts and outliers. In contrast, the results of the proposed method can greatly distinguish between two buildings with a large interval. Additionally, the roofs of two buildings are clear. The height of building #3 estimated in the height map of the proposed method is closer to ground truth. The height estimation of building #3 in the nonlocal result is severely different from that of all-track results, which is probably affected by the multipath scattering. The height estimation is labeled with orange lines.
Figure 15.
Comparison of entire scene 3D reconstruction between the nonlocal algorithm and proposed CGAN method. Panel (a) is the reconstruction result of the nonlocal algorithm. Panel (b) is the reconstruction result of the proposed CGAN method. Comparatively, the quality of the CGAN method is higher with fewer artifacts and outliers. In addition, the structures of buildings are much clearer, and are closer to the results of all-track TomoSAR.
Figure 15.
Comparison of entire scene 3D reconstruction between the nonlocal algorithm and proposed CGAN method. Panel (a) is the reconstruction result of the nonlocal algorithm. Panel (b) is the reconstruction result of the proposed CGAN method. Comparatively, the quality of the CGAN method is higher with fewer artifacts and outliers. In addition, the structures of buildings are much clearer, and are closer to the results of all-track TomoSAR.
Figure 16.
Optical and SAR intensity images of spaceborne data. Panel (a) is the SAR intensity image of the building. The red line is the slice selected to show details. Panel (b) is the corresponding optical image.
Figure 16.
Optical and SAR intensity images of spaceborne data. Panel (a) is the SAR intensity image of the building. The red line is the slice selected to show details. Panel (b) is the corresponding optical image.
Figure 17.
3D views of reconstruction by CS method using all tracks and three tracks. Panel (a) is the 3D view of reconstruction using three tracks. There are lots of outliers. Panel (b) is the 3D view of reconstruction using all tracks. There are few outliers and the surface of the building is clean and refined.
Figure 17.
3D views of reconstruction by CS method using all tracks and three tracks. Panel (a) is the 3D view of reconstruction using three tracks. There are lots of outliers. Panel (b) is the 3D view of reconstruction using all tracks. There are few outliers and the surface of the building is clean and refined.
Figure 18.
Reconstruction results of CS method using three and all tracks at the position indicated by the red line in the SAR intensity image. Panel (a) is the height map of three-track TomoSAR. Panel (b) is the normalized strength map of all-track TomoSAR. The orange circles indicate the outliers. Panel (c) is the height map of all-track TomoSAR. Panel (d) is the normalized strength map of all-track TomoSAR. The orange lines mark the height of the building. There are lots of outliers in the three-track reconstruction result.
Figure 18.
Reconstruction results of CS method using three and all tracks at the position indicated by the red line in the SAR intensity image. Panel (a) is the height map of three-track TomoSAR. Panel (b) is the normalized strength map of all-track TomoSAR. The orange circles indicate the outliers. Panel (c) is the height map of all-track TomoSAR. Panel (d) is the normalized strength map of all-track TomoSAR. The orange lines mark the height of the building. There are lots of outliers in the three-track reconstruction result.
Figure 19.
Reconstruction results of the nonlocal algorithm and the proposed CGAN methods. Panel (a) is the height map of the nonlocal algorithm. Panel (b) is the normalized strength map of the nonlocal algorithm. Panel (c) is the height map of the proposed CGAN method. Panel (d) is the normalized strength map of the proposed CGAN method. Generally, the nonlocal algorithm can compress the outliers. In contrast, the proposed CGAN method can further compress the outliers and generate more refined surface. The orange lines mark the height of the building.
Figure 19.
Reconstruction results of the nonlocal algorithm and the proposed CGAN methods. Panel (a) is the height map of the nonlocal algorithm. Panel (b) is the normalized strength map of the nonlocal algorithm. Panel (c) is the height map of the proposed CGAN method. Panel (d) is the normalized strength map of the proposed CGAN method. Generally, the nonlocal algorithm can compress the outliers. In contrast, the proposed CGAN method can further compress the outliers and generate more refined surface. The orange lines mark the height of the building.
Figure 20.
3D views of reconstruction results of the nonlocal algorithm and the proposed CGAN method. Panel (a) is the 3D result of the nonlocal algorithm using three tracks. There are also lots of outliers and the reconstructed surface is not smooth. Panel (b) is the 3D result of the proposed CGAN method using three tracks. There are few outliers and the surface of buildings is clean and refined. The CGAN, which is just trained on the airborne dataset, can effectively and robustly process the spaceborne dataset without any tuning.
Figure 20.
3D views of reconstruction results of the nonlocal algorithm and the proposed CGAN method. Panel (a) is the 3D result of the nonlocal algorithm using three tracks. There are also lots of outliers and the reconstructed surface is not smooth. Panel (b) is the 3D result of the proposed CGAN method using three tracks. There are few outliers and the surface of buildings is clean and refined. The CGAN, which is just trained on the airborne dataset, can effectively and robustly process the spaceborne dataset without any tuning.
Table 1.
Parameters of YunCheng airborne dataset.
Table 1.
Parameters of YunCheng airborne dataset.
| All-Track | Three-Track |
---|
Number of tracks | 8 | 3 |
Maximal elevation aperture | 0.588 m | 0.168 m |
Distance from the scene center | 1308 m |
Wavelength | 2.1 cm |
Incidence angle at scene center | 58° |
Table 2.
Configurations of nonlocal algorithm.
Table 2.
Configurations of nonlocal algorithm.
Parameter | Value |
---|
search window size | |
patch size | |
iterations | 10 |
posterior similarity coefficient (h) | 5.3 |
sparse prior KL similarity | |
minimum number of similar blocks () | 10 |
Table 3.
Height estimation results by different methods.
Table 3.
Height estimation results by different methods.
Building_Index | All Tracks | 3
Tracks\Error | Nonlocal\Error | Proposed\Error |
---|
#3 | 75 | 89\14 | 90\15 | 75\0 |
#4 | 40 | Hard to recognize | 42\2 | 42\2 |
#6 | 81 | 82\1 | 82\1 | 81\0 |
#7 | 74 | 87\13 | 87\13 | 73\1 |
#8 | 90 | 90\0 | 91\1 | 90\0 |
#9 | 93 | 93\0 | 93\0 | 93\0 |
Table 4.
Time consumption of different methods.
Table 4.
Time consumption of different methods.
Method | Time Consumption (s) |
---|
Nonlocal (10 iterations) | 14,492 |
Proposed GAN | 10 |
Table 5.
Parameters of TerraSAR-X spaceborne dataset.
Table 5.
Parameters of TerraSAR-X spaceborne dataset.
| All-Track | Three-Track |
---|
Number of tracks | 19 | 3 |
Maximal elevation aperture | 215 m | 42 m |
Distance from the scene center | 617 km |
Wavelength | 3.1 cm |
Incidence angle at scene center | 66° |