Figure 1.
(a–e) illustrate different frameworks of pansharpening.
Figure 1.
(a–e) illustrate different frameworks of pansharpening.
Figure 2.
Proposed SwinPAN framework for pansharpening.
Figure 2.
Proposed SwinPAN framework for pansharpening.
Figure 3.
Schematic illustration of the high-pass preservation module.
Figure 3.
Schematic illustration of the high-pass preservation module.
Figure 4.
Illustration of the self-attention module in Swin Transformer.
Figure 4.
Illustration of the self-attention module in Swin Transformer.
Figure 5.
Reduced-resolution results on the QuickBird imagery: (a) MS image. (b) PAN image. (c) Ground truth. (d–n) Pansharpening results obtained using (d) GSA, (e) MTF-GLP-HPM, (f) SFIM, (g) DiCNN, (h) MSDCNN, (i) DRPNN, (j) FusionNet, (k) PNN, (l) PanNet, (m) HyperTransformer and (n) the proposed approach.
Figure 5.
Reduced-resolution results on the QuickBird imagery: (a) MS image. (b) PAN image. (c) Ground truth. (d–n) Pansharpening results obtained using (d) GSA, (e) MTF-GLP-HPM, (f) SFIM, (g) DiCNN, (h) MSDCNN, (i) DRPNN, (j) FusionNet, (k) PNN, (l) PanNet, (m) HyperTransformer and (n) the proposed approach.
Figure 6.
Reduced-resolution results on the GaoFen2 imagery: (a) MS image. (b) PAN image. (c) Ground truth. (d–n) Pansharpening results obtained using (d) GSA, (e) MTF-GLP-HPM, (f) SFIM, (g) DiCNN, (h) MSDCNN, (i) DRPNN, (j) FusionNet, (k) PNN, (l) PanNet, (m) HyperTransformer and (n) the proposed approach.
Figure 6.
Reduced-resolution results on the GaoFen2 imagery: (a) MS image. (b) PAN image. (c) Ground truth. (d–n) Pansharpening results obtained using (d) GSA, (e) MTF-GLP-HPM, (f) SFIM, (g) DiCNN, (h) MSDCNN, (i) DRPNN, (j) FusionNet, (k) PNN, (l) PanNet, (m) HyperTransformer and (n) the proposed approach.
Figure 7.
Reduced-resolution results on the WorldView3 imagery: (a) MS image. (b) PAN image. (c) Ground truth. (d–n) Pansharpening results obtained using (d) GSA, (e) MTF-GLP-HPM, (f) SFIM, (g) DiCNN, (h) MSDCNN, (i) DRPNN, (j) FusionNet, (k) PNN, (l) PanNet, (m) HyperTransformer and (n) the proposed approach.
Figure 7.
Reduced-resolution results on the WorldView3 imagery: (a) MS image. (b) PAN image. (c) Ground truth. (d–n) Pansharpening results obtained using (d) GSA, (e) MTF-GLP-HPM, (f) SFIM, (g) DiCNN, (h) MSDCNN, (i) DRPNN, (j) FusionNet, (k) PNN, (l) PanNet, (m) HyperTransformer and (n) the proposed approach.
Figure 8.
Full-resolution results on the QuickBird imagery: (a) MS image. (b) PAN image. (c–m) Pansharpening results obtained using (c) GSA, (d) MTF-GLP-HPM, (e) SFIM, (f) DiCNN, (g) MSDCNN, (h) DRPNN, (i) FusionNet, (j) PNN, (k) PanNet, (l) HyperTransformer and (m) the proposed approach.
Figure 8.
Full-resolution results on the QuickBird imagery: (a) MS image. (b) PAN image. (c–m) Pansharpening results obtained using (c) GSA, (d) MTF-GLP-HPM, (e) SFIM, (f) DiCNN, (g) MSDCNN, (h) DRPNN, (i) FusionNet, (j) PNN, (k) PanNet, (l) HyperTransformer and (m) the proposed approach.
Figure 9.
Full-resolution results on the GaoFen2 imagery: (a) MS image. (b) PAN image. (c–m) Pansharpening results obtained using (c) GSA, (d) MTF-GLP-HPM, (e) SFIM, (f) DiCNN, (g) MSDCNN, (h) DRPNN, (i) FusionNet, (j) PNN, (k) PanNet, (l) HyperTransformer and (m) the proposed approach.
Figure 9.
Full-resolution results on the GaoFen2 imagery: (a) MS image. (b) PAN image. (c–m) Pansharpening results obtained using (c) GSA, (d) MTF-GLP-HPM, (e) SFIM, (f) DiCNN, (g) MSDCNN, (h) DRPNN, (i) FusionNet, (j) PNN, (k) PanNet, (l) HyperTransformer and (m) the proposed approach.
Figure 10.
Full-resolution results on the WorldView3 imagery: (a) MS image. (b) PAN image. (c–m) Pansharpening results obtained using (c) GSA, (d) MTF-GLP-HPM, (e) SFIM, (f) DiCNN, (g) MSDCNN, (h) DRPNN, (i) FusionNet, (j) PNN, (k) PanNet, (l) HyperTransformer and (m) the proposed approach.
Figure 10.
Full-resolution results on the WorldView3 imagery: (a) MS image. (b) PAN image. (c–m) Pansharpening results obtained using (c) GSA, (d) MTF-GLP-HPM, (e) SFIM, (f) DiCNN, (g) MSDCNN, (h) DRPNN, (i) FusionNet, (j) PNN, (k) PanNet, (l) HyperTransformer and (m) the proposed approach.
Figure 11.
(a–g) show the results of ablation study of high-pass preservation modules.
Figure 11.
(a–g) show the results of ablation study of high-pass preservation modules.
Figure 12.
(a–g) show the results of ablation study of static high-pass preservation modules and dynamic high-pass preservation modules.
Figure 12.
(a–g) show the results of ablation study of static high-pass preservation modules and dynamic high-pass preservation modules.
Figure 13.
(a–g) show the results of quantitative results with different values of feature dimension.
Figure 13.
(a–g) show the results of quantitative results with different values of feature dimension.
Figure 14.
(a–g) show the results of quantitative results with different numbers of DRBs.
Figure 14.
(a–g) show the results of quantitative results with different numbers of DRBs.
Table 1.
Parameter settings for the proposed approach on the experimental datasets.
Table 1.
Parameter settings for the proposed approach on the experimental datasets.
Paratmeters | DRB | DRL | Head | Dimension | Learning Rate | Batch Size |
---|
QuickBird | 6 | [2, 2, 2, 2, 2, 2] | 2 | 96 | | 32 |
GaoFen2 | 6 | [2, 2, 2, 2, 2, 2] | 6 | 60 | | 32 |
WorldView3 | 6 | [2, 2, 2, 2, 2, 2] | 2 | 60 | | 32 |
Table 2.
Quantative results at reduced resolution on the QuickBird dataset.
Table 2.
Quantative results at reduced resolution on the QuickBird dataset.
Methods | SAM↓ | REGAS↓ | Q4↑ | SCC↑ |
---|
SFIM | 8.1925 ± 1.7282 | 8.8807 ± 2.1295 | 0.8495 ± 0.0788 | 0.9315 ± 0.0150 |
MTF-GLP-HPM | 8.3063 ± 1.5742 | 10.4731 ± 0.9394 | 0.8411 ± 0.0138 | 0.8796 ± 0.0194 |
GSA | 8.3497 ± 1.6728 | 9.3289 ± 2.7366 | 0.8289 ± 0.1119 | 0.9284 ± 0.0140 |
DiCNN | 5.6262 ± 0.9368 | 5.4730 ± 0.3720 | 0.9488 ± 0.0103 | 0.9712 ± 0.0054 |
MSDCNN | 4.9896 ± 0.8182 | 4.1383 ± 0.2411 | 0.9720 ± 0.0057 | 0.9814 ± 0.0038 |
DRPNN | 5.0111 ± 0.8288 | 4.1363 ± 0.2487 | 0.9719 ± 0.0057 | 0.9794 ± 0.0040 |
FusionNet | 5.1158 ± 0.8432 | 4.3962± 0.2662 | 0.9678 ± 0.0071 | 0.9797 ± 0.0039 |
PNN | 5.4115 ± 0.8705 | 4.7185 ± 0.3218 | 0.9630 ± 0.0087 | 0.9763 ± 0.0044 |
PanNet | 5.5462 ± 1.0085 | 5.4995 ± 0.8098 | 0.9487 ± 0.0186 | 0.9687 ± 0.0082 |
HyperTransformer | 4.9931 ± 0.7630 | 4.1189 ± 0.3429 | 0.9715 ± 0.0078 | 0.9813 ± 0.0075 |
Ours | 4.8653 ± 0.7909 | 4.0546 ± 0.2531 | 0.9726 ± 0.0063 | 0.9826 ± 0.0035 |
Table 3.
Quantative results at reduced resolution on the GaoFen2 dataset.
Table 3.
Quantative results at reduced resolution on the GaoFen2 dataset.
Methods | SAM ↓ | REGAS ↓ | Q4 ↑ | SCC ↑ |
---|
SFIM | 6.2068 ± 1.1050 | 12.4050 ± 2.1028 | 0.5631 ± 0.1187 | 0.9691 ± 0.0120 |
MTF-GLP-HPM | 5.1642 ± 1.0338 | 10.5863 ± 3.2607 | 0.6186 ± 0.1613 | 0.9416 ±0.0142 |
GSA | 6.4668 ± 1.0011 | 12.8536 ±2.1755 | 0.5391 ± 0.1253 | 0.9659 ± 0.0127 |
DICNN | 1.1003 ± 0.2064 | 1.1222 ± 0.2184 | 0.9840 ± 0.0081 | 0.9862 ± 0.0059 |
MSDCNN | 0.9889 ± 0.1839 | 0.9679 ± 0.1777 | 0.9886 ± 0.0063 | 0.9901 ± 0.0043 |
DRPNN | 0.9118 ± 0.1634 | 0.8185 ± 0.1377 | 0.9916 ± 0.0045 | 0.9918 ± 0.0035 |
FusionNet | 1.0143 ± 0.1959 | 1.0551 ± 0.2079 | 0.9860 ± 0.0071 | 0.9889 ± 0.0048 |
PNN | 1.0907 ± 0.2105 | 1.1116 ± 0.2259 | 0.9842 ± 0.0085 | 0.9871 ± 0.0058 |
PanNet | 1.0248 ± 0.1724 | 0.9214 ± 0.1561 | 0.9893 ± 0.0056 | 0.9898 ± 0.0043 |
HyperTransformer | 0.9538 ± 0.1648 | 0.8506 ± 0.1283 | 0.9908 ± 0.0048 | 0.9905 ± 0.0038 |
Ours | 0.7996 ± 0.1441 | 0.7790 ± 0.1271 | 0.9922 ± 0.0041 | 0.9936 ± 0.0028 |
Table 4.
Quantative results at reduced resolution on the WorldView3 dataset.
Table 4.
Quantative results at reduced resolution on the WorldView3 dataset.
Methods | SAM ↓ | REGAS ↓ | Q8 ↑ | SCC ↑ |
---|
SFIM | 5.5385 ± 1.4737 | 5.7839 ± 1.7049 | 0.8704 ± 0.4548 | 0.9531 ± 0.0142 |
MTF-GLP-HPM | 5.7246 ± 1.5042 | 6.5285 ± 1.3622 | 0.8716 ± 0.3886 | 0.9237 ± 0.0211 |
GSA | 5.6828 ± 1.5025 | 6.6567 ± 1.8083 | 0.8732 ± 0.3973 | 0.9496 ± 0.0137 |
DICNN | 4.4534 ± 0.8643 | 3.2739 ± 0.8627 | 0.9228 ± 0.5535 | 0.9765 ± 0.0125 |
MSDCNN | 3.7875 ± 0.6942 | 2.7558 ± 0.6105 | 0.9373 ± 0.3181 | 0.9748 ± 0.0114 |
DRPNN | 3.5703 ± 0.6365 | 2.5916 ± 0.5484 | 0.8479 ± 0.4550 | 0.9778 ± 0.0119 |
FusionNet | 3.4672 ± 0.6286 | 2.5718 ± 0.5937 | 0.8516 ± 0.4376 | 0.9825 ± 0.0077 |
PNN | 3.9548 ± 0.7266 | 2.8655 ± 0.6668 | 0.8968 ± 0.4820 | 0.9750 ± 0.0109 |
PanNet | 3.7322 ± 0.6609 | 2.7974 ± 0.6270 | 0.8760 ± 0.4258 | 0.9729 ± 0.0128 |
HyperTransformer | 3.1275 ± 0.5250 | 2.6405 ± 0.5122 | 0.9210 ± 0.3970 | 0.9843 ± 0.1240 |
Ours | 2.9542 ± 0.5253 | 2.1558 ± 0.4439 | 0.9539 ± 0.4722 | 0.9890 ± 0.0046 |
Table 5.
Quantative results at full resolution on the QuickBird dataset.
Table 5.
Quantative results at full resolution on the QuickBird dataset.
Methods | | | QNR↑ |
---|
SFIM | 0.0512 ± 0.0113 | 0.1296 ± 0.0996 | 0.8243 ± 0.0974 |
MTF-GLP-HPM | 0.0506 ± 0.0234 | 0.1341 ± 0.1146 | 0.8217 ± 0.1095 |
GSA | 0.0465 ± 0.0207 | 0.2007 ± 0.1098 | 0.7614 ± 0.1033 |
DICNN | 0.0416 ± 0.0300 | 0.0910 ± 0.0514 | 0.8723 ± 0.0711 |
MSDCNN | 0.0604 ± 0.0390 | 0.0524 ± 0.0137 | 0.8903 ± 0.0391 |
DRPNN | 0.0394 ± 0.0327 | 0.0409 ± 0.0241 | 0.9219 ± 0.0513 |
FusionNet | 0.0402 ± 0.0341 | 0.0543 ± 0.0410 | 0.9088 ± 0.0676 |
PNN | 0.0399 ± 0.0342 | 0.0500 ± 0.0393 | 0.9133 ± 0.0665 |
PanNet | 0.0409 ± 0.0347 | 0.0418 ± 0.0334 | 0.9200 ± 0.0618 |
HyperTransformer | 0.0424 ± 0.0376 | 0.0412 ± 0.0195 | 0.9210 ± 0.0499 |
Ours | 0.0370 ± 0.0333 | 0.0398 ± 0.0257 | 0.9253 ± 0.0536 |
Table 6.
Quantative results at full resolution on the GaoFen2 dataset.
Table 6.
Quantative results at full resolution on the GaoFen2 dataset.
Methods | | | QNR↑ |
---|
SFIM | 0.0371 ± 0.0160 | 0.0647 ± 0.0460 | 0.9010 ± 0.0518 |
MTF-GLP-HPM | 0.0925 ± 0.0386 | 0.0805 ± 0.0531 | 0.8351 ± 0.0676 |
GSA | 0.0596 ± 0.0227 | 0.1027 ± 0.0542 | 0.8445 ± 0.0620 |
DICNN | 0.0179 ± 0.0145 | 0.0590 ± 0.0262 | 0.9244 ± 0.0371 |
MSDCNN | 0.0121 ± 0.0144 | 0.0387 ± 0.0198 | 0.9499 ± 0.0317 |
DRPNN | 0.0158 ± 0.0152 | 0.0319 ± 0.0168 | 0.9530 ± 0.0298 |
FusionNet | 0.0215 ± 0.0191 | 0.0546 ± 0.0262 | 0.9255 ± 0.0419 |
PNN | 0.0113 ± 0.0130 | 0.0333 ± 0.0175 | 0.9560 ± 0.0285 |
PanNet | 0.0115 ± 0.0118 | 0.0412 ± 0.0191 | 0.9486 ± 0.0285 |
HyperTransformer | 0.0174 ± 0.0170 | 0.0414 ± 0.0218 | 0.9422 ± 0.0355 |
Ours | 0.0110 ± 0.0099 | 0.0309 ± 0.0135 | 0.9585 ± 0.0210 |
Table 7.
Quantative results at full resolution on the WorldView3 dataset.
Table 7.
Quantative results at full resolution on the WorldView3 dataset.
Methods | | | QNR↑ |
---|
SFIM | 0.0353 ± 0.0106 | 0.0565 ± 0.0288 | 0.9075 ± 0.0351 |
MTF-GLP-HPM | 0.0389 ± 0.0229 | 0.0523 ± 0.0332 | 0.9113 ± 0.0482 |
GSA | 0.0325 ± 0.0131 | 0.0603 ± 0.0293 | 0.9062 ± 0.0381 |
DICNN | 0.0239 ± 0.0174 | 0.0575 ± 0.0339 | 0.9202 ± 0.0410 |
MSDCNN | 0.0267 ± 0.0146 | 0.0473 ± 0.0254 | 0.9275 ± 0.0351 |
DRPNN | 0.0266 ± 0.0168 | 0.0476 ± 0.0234 | 0.9274 ± 0.0369 |
FusionNet | 0.0320 ± 0.0253 | 0.0490 ± 0.0213 | 0.9207 ± 0.0354 |
PNN | 0.0249 ± 0.0139 | 0.0451 ± 0.0220 | 0.9313 ± 0.0312 |
PanNet | 0.0277 ± 0.0142 | 0.0603 ± 0.0237 | 0.9140 ± 0.0342 |
HyperTransformer | 0.0276 ± 0.0137 | 0.0487 ± 0.0212 | 0.9347 ± 0.0312 |
Ours | 0.0216 ± 0.0105 | 0.0326 ± 0.0194 | 0.9467 ± 0.0275 |