Collaborative Research: Unregistered Spectral Image Fusion in Remote Sensing: Foundations and Algorithms

About This Grant

Remotely sensed spectral images, such as hyperspectral images (HSIs) and multispectral images (MSIs), are widely used across science and engineering fields, including agriculture, oceanography, forest monitoring, mineral discovery, and space exploration. These image modalities involve an inherent trade-off between spatial and spectral resolution: HSIs provide fine spectral detail but coarse spatial resolution, whereas MSIs offer the reverse. Spectral image fusion techniques seek to combine the strengths of both by integrating an HSI and MSI of the same region to produce fused images with high-resolution information in both domains, supporting various tasks such as pixel classification, target identification, and change detection. However, many existing fusion methods operate under the assumption that the spectral images are co-registered (i.e., covering the same region and sharing the same coordinates), whereas in practice the data are often spatially misaligned by pixel shifts, rotations, and other distortions (collectively referred to as “unregistered”), typically arising from differences in sensors or imaging platforms. Despite its fundamental practical importance and considerable interest, the fusion of unregistered spectral images still lacks rigorous theoretical underpinnings and reliable algorithms. This project addresses these gaps by developing new analytical and computational methods to establish a solid theoretical and algorithmic foundation for this long-standing and practically significant problem, enabling performance-guaranteed fusion of unregistered spectral data in real-world scenarios. Beyond remote sensing, the outcomes are expected to benefit areas such as cross-platform medical imaging and domain adaptation/transfer in machine learning. The project also offers undergraduate research opportunities, providing students with training in machine learning, optimization, and image/signal processing. This project develops a unified, unsupervised framework for fusion of unregistered spectral images with provable guarantees, tackling key challenges including spatial misalignment, lack of training data, and nonrigid deformation. Thrust 1 focuses on establishing theoretical foundations by integrating spectral unmixing with adversarial learning through diversified distribution matching in a latent spatial domain, enabling provable spatio-spectral super-resolution under practical, unregistered scenarios. Thrust 2 extends this framework to more complex real-world cases such as those involving unknown and potentially large nonrigid deformations. Thrust 3 develops stable and efficient optimization algorithms for the proposed fusion formulations, tailored to adversarial learning in latent domains and addressing the limitations of standard optimizers. Validation on semi-realistic and real-world datasets is used to assess the robustness and generalizability of the proposed methods. Expected outcomes include new theoretical insights, practical algorithms with convergence guarantees, and reproducible benchmarks to advance unregistered spectral image fusion and its applications in machine learning, signal processing, and scientific imaging. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.