UAV image stitching method based on diffusion model and manifold gradient constraint

doi:10.7523/j.ucas.2024.061

Abstract

Abstract:

Image stitching is a crucial prerequisite step for unmanned aerial vehicle （UAV） remote sensing applications， while the stitched images using most of the current image stitching methods often suffer from large irregular boundaries and multiple stitching seams， which can seriously affect subsequent analysis and applications. Existing improved methods typically cannot simultaneously address these two issues， and integrating the two types of methods in sequence is a straightforward way to solve the two problems， while this often can not obtain satisfactory performance because of the inevitable error propagation problem. This paper proposes an inpainting method for the UAV image stitching task based on the denoising diffusion probability model （DDPM）. The method uniformly designs masks for irregular boundaries and stitching seams， and a diffusion model is then utilized with manifold gradient prior constraints to complete the masked regions. By doing so， both irregular boundaries and stitching seams are simultaneously eliminated， thereby improving the quality of the stitching results. Comparative experiments are conducted using four datasets established for different scenarios. The experimental results demonstrated the efficacy of the proposed method in effectively eliminating irregular boundaries and seams in the image stitching. Moreover， from patches to pictures quality （PaQ-2-PiQ） and multi-scale image quality （MUSIQ） scores increased by 4.36% and 15.37%， respectively. Furthermore， at the locations of irregular boundaries， the structural similarity （SSIM） and peak signal-to-noise ratio （PSNR） values improved by 20.22% and 33.69%， respectively. Compared with state-of-the-art methods and other conventional image stitching algorithms， the proposed method performs better in both subjective and objective quality metric scores， has good robustness and generalization， and can be widely applied to UAV image stitching scenarios.

Key words: UAV image, diffusion model, image stitching, irregular boundaries, stitching seams, manifold gradient constraint

CLC Number:

TP751.1

Jie WANG, Yongxi LUO, Jun CHEN, Yewei WU. UAV image stitching method based on diffusion model and manifold gradient constraint[J]. Journal of University of Chinese Academy of Sciences, 2026, 43(2): 252-264.

Figures/Tables 15

Fig.1 Irregular boundaries and stitching seams in the stitched images of the unmanned aerial vehicle

Fig.2 Overall diagram of the proposed method

Fig.3 U-Net with time embedding vectors

Fig.4 Comparison of results before and after restoration using the proposed method

Fig.5 Comparison of details before and after restoration using the proposed method

Table 1 Objective quality comparison of images before and after restoration using the proposed method

图像

PaQ-2-Piq

MUSIQ

修复前

修复后

0.734

0.766

0.631

0.728

Fig.6 Comparison between repaired values and ground truth for irregular boundaries

Table 2 Measurement of similarity with the reference value between images before and after restoration using the proposed method

评价指标

SSIM

PSNR

修复前

修复后

0.361

0.434

12.724

17.011

Fig.7 Comparison of stitching results with other rectangling methods

Table 3 Comparison of perceptual quality with other rectangling methods

方法

PaQ-2-PiQ

MUSIQ

Deep Rectangling

本文方法(仅使用不规则边界掩码)

0.742

0.751

0.648

0.708

Fig.8 Comparison of stitching results with seamless stitching methods

Table 4 Comparison of perceptual quality with seamless stitching methods

方法

PaQ-2-PiQ

MUSIQ

UDIS++

本文方法(仅使用拼接缝掩码)

0.695

0.743

0.629

0.676

Fig.9 Comparison of stitching results in pairwise stitching scenarios

Fig.10 Local detail comparison of stitching results in pairwise stitching scenarios

Table 5 Comparison of perceptual quality in pairwise stitching scenarios

方法

PaQ-2-PiQ

MUSIQ

SPW

LPC

APAP

AANAP

本文方法

0.691

0.711

0.643

0.652

0.728

0.686

0.706

0.682

0.621

0.711

References 29

[1]	He K M， Chang H W， Sun J. Rectangling panoramic images via warping［J］. ACM Transactions on Graphics， 2013， 32（4）：79. DOI： 10.1145/2461912.2462004 .
[2]	Li D P， He K M， Sun J， et al. A geodesic-preserving method for image warping［C］//2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Boston， MA. IEEE， 2015：213-221.DOI： 10.1109/CVPR.2015.7298617 .
[3]	Zhang Y， Lai Y K， Zhang F L. Content-preserving image stitching with piecewise rectangular boundary constraints［J］. IEEE Transactions on Visualization and Computer Graphics， 2021， 27（7）： 3198-3212. DOI： 10.1109/TVCG.2020.2965097 .
[4]	Nie L， Lin C Y， Liao K， et al. Deep rectangling for image stitching： a learning baseline［C］//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. New Orleans， LA， USA. IEEE， 2022： 5730-5738. DOI： 10.1109/CVPR52688.2022.00565 .
[5]	Kwatra V， Schödl A， Essa I， et al. Graphcut textures： image and video synthesis using graph cuts［J］. ACM Transactions on Graphics， 2003， 22（3）： 277-286. DOI： 10.1145/882262.882264 .
[6]	Eden A， Uyttendaele M， Szeliski R. Seamless image stitching of scenes with large motions and exposure differences［C］//2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition （CVPR'06）. New York， NY， USA. IEEE， 2006： 2498-2505. DOI： 10.1109/CVPR.2006.268 .
[7]	Li N， Liao T L， Wang C. Perception-based seam cutting for image stitching［J］. Signal， Image and Video Processing， 2018， 12（5）： 967-974. DOI： 10.1007/s11760-018-1241-9 .
[8]	Gao J， Li Y， Chin T J，et al.Seam-driven image stitching［C］//Eurographics （Short Papers），2013.DOI：10.2312/CONF/EG2013/SHORT/045-048 .
[9]	Zhang F， Liu F. Parallax-tolerant image stitching［C］//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus， OH， USA. IEEE， 2014： 3262-3269. DOI： 10.1109/CVPR.2014.423 .
[10]	Lin K M， Jiang N J， Cheong L F， et al. SEAGULL： seam-guided local alignment for parallax-tolerant image stitching［C］//European Conference on Computer Vision. Cham： Springer， 2016： 370-385.10.1007/978-3-319-46487-9_23.
[11]	Dai Q Y， Fang F M， Li J C， et al. Edge-guided composition network for image stitching［J］. Pattern Recognition， 2021， 118： 108019. DOI： 10.1016/j.patcog.2021.108019 .
[12]	Nie L， Lin C Y， Liao K， et al. Parallax-tolerant unsupervised deep image stitching［C］//2023 IEEE/CVF International Conference on Computer Vision （ICCV）. Paris， France. IEEE， 2023： 7365-7374. DOI： 10.1109/ICCV51070.2023.00680 .
[13]	Sohl-Dickstein J， Weiss E A， Maheswaranathan N， et al. Deep unsupervised learning using nonequilibrium thermodynamics［C］//Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37. July 6 - 11， 2015， Lille， France. ACM， 2015： 2256-2265. DOI： 10.5555/3045118.3045358 .
[14]	Ho J， Jain A， Abbeel P. Denoising diffusion probabilistic models［C］//Proceedings of the 34th International Conference on Neural Information Processing Systems. December 6 - 12， 2020，Vancouver， BC， Canada. ACM， 2020： 6840-6851. DOI： 10.5555/3495724.3496298 .
[15]	Nichol A， Dhariwal P. Improved denoising diffusion probabilistic models［EB/OL］. arXiv 2021：2102.09672. （2021-02-18）［2024-05-20］. DOI： 10.48550/arXiv.2102.09672 .
[16]	Zhu Y Z， Zhang K， Liang J Y， et al. Denoising diffusion models for plug-and-play image restoration［C］//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops （CVPRW）. Vancouver， BC， Canada. IEEE， 2023： 1219-1229. DOI： 10.1109/CVPRW59228.2023.00129 .
[17]	Lugmayr A， Danelljan M， Romero A， et al. RePaint： inpainting using denoising diffusion probabilistic models［C］//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. New Orleans， LA， USA. IEEE， 2022： 11451-11461. DOI： 10.1109/CVPR52688.2022.01117 .
[18]	Chung H， Sim B， Ryu D， et al. Improving diffusion models for inverse problems using manifold constraints［C］//Proceedings of the 36th International Conference on Neural Information Processing Systems. 28 November 2022， New Orleans， LA， USA. ACM， 2022： 25683-25696. DOI： 10.5555/3600270.3602132 .
[19]	Dhariwal P， Nichol A. Diffusion models beat GANs on image synthesis［EB/OL］. arXiv 2021： 2105.05233.（2021-06-01）［2024-05-20］. .
[20]	Lowe D G. Object recognition from local scale-invariant features［C］//Proceedings of the Seventh IEEE International Conference on Computer Vision. Kerkyra， Greece. IEEE， 1999： 1150-1157. DOI： 10.1109/ICCV.1999.790410 .
[21]	Fischler M A， Bolles R C. Random sample consensus： a paradigm for model fitting with applications to image analysis and automated cartography［M］//Readings in Computer Vision. Amsterdam： Elsevier， 1987： 726-740. DOI： 10.1016/b978-0-08-051581-6.50070-2 .
[22]	Chen Y B， Liu S F， Wang X L. Learning continuous image representation with local implicit image function［C］//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Nashville， TN， USA. IEEE， 2021： 8624-8634. DOI： 10.1109/CVPR46437.2021.00852 .
[23]	Lim B， Son S， Kim H， et al. Enhanced deep residual networks for single image super-resolution［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops （CVPRW）. Honolulu， HI， USA. IEEE， 2017： 1132-1140. DOI： 10.1109/CVPRW.2017.151 .
[24]	Ke J J， Wang Q F， Wang Y L， et al. MUSIQ： multi-scale image quality transformer［C］//2021 IEEE/CVF International Conference on Computer Vision （ICCV）. Montreal， QC， Canada. IEEE， 2021： 5128-5137. DOI： 10.1109/ICCV48922.2021.00510 .
[25]	Ying Z Q， Niu H R， Gupta P， et al. From patches to pictures （PaQ-2-PiQ）： mapping the perceptual space of picture quality［C］//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， WA， USA. IEEE， 2020： 3572-3582. DOI： 10.1109/CVPR42600.2020.00363 .
[26]	Jia Q， Li Z J， Fan X， et al. Leveraging line-point consistence to preserve structures for wide parallax image stitching［C］//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Nashville， TN， USA. IEEE， 2021： 12181-12190. DOI： 10.1109/CVPR46437.2021.01201 .
[27]	Liao T L， Li N. Single-perspective warps in natural image stitching［J］. IEEE Transactions on Image Processing， 2019，29：724-735. DOI： 10.1109/TIP.2019.2934344 .
[28]	Lin C C， Pankanti S U， Ramamurthy K N， et al. Adaptive as-natural-as-possible image stitching［C］//2015 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Boston， MA， USA. IEEE， 2015： 1155-1163. DOI： 10.1109/CVPR.2015.7298719 .
[29]	Zaragoza J， Chin T J， Brown M S， et al. As-projective-as-possible image stitching with moving DLT［C］//2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland， OR， USA. IEEE， 2013： 2339-2346. DOI： 10.1109/CVPR.2013.303 .