SIDNet: Learning Shading-aware Illumination Descriptor for Image Harmonization

Zhongyun Hu*, Ntumba Elie Nsampi*, Xue Wang, Qing Wang
School of Computer Science, Northwestern Polytechnical University
IEEE TETCI 2023
*Indicates Equal Contribution
Paper (High-Res) Supplementary Dataset
Code arXiv

SIDNet: Given a foreground image (a) and a background image (b) where the foreground image is to be placed, our model first learns to extract a shading-aware illumination descriptor (c) from the background image, and a set of shading bases (d) and albedo feature (e) from the foreground image. Then, the shading-aware illumination descriptor and shading bases are combined to render a new foreground shading (f) by a rendering equation (g). Last, the albedo feature and the foreground shading are used to render a new foreground image (h) that conforms to the illumination of the background image. Our model is designed with the physical principle of image formation.

Abstract

Image harmonization aims at adjusting the appearance of the foreground to make it more compatible with the background. Without exploring background illumination and its effects on the foreground elements, existing works are incapable of generating a realistic foreground shading. In this paper, we decompose the image harmonization task into two sub-problems: 1) illumination estimation of the background image and 2) re-rendering of foreground objects under background illumination. Before solving these two sub-problems, we first learn a shading-aware illumination descriptor via a well-designed neural rendering framework, of which the key is a shading bases module that generates multiple shading bases from the foreground image. Then we design a background illumination estimation module to extract the illumination descriptor from the background. Finally, the Shading-aware Illumination Descriptor is used in conjunction with the neural rendering framework (SIDNet) to produce the harmonized foreground image containing a novel harmonized shading. Moreover, we construct a large-scale photo-realistic synthetic image harmonization dataset (IllumHarmony-Dataset) that contains numerous shading variations. Extensive experiments on both synthetic and real data demonstrate the superiority of the proposed method, especially in dealing with foreground shadings.

IllumHarmony-Dataset

SIDNet

MY ALT TEXT

An overview of our proposed image harmonization method. Our method has two training stages: training the Neural Rendering Framework (NRF) and training the Background Illumination Estimation Module (BIEM). The key to the first stage is to learn a shading-aware illumination descriptor, which is then estimated from the background image in the second stage. During inference, our image harmonization pipeline (i.e., SIDNet) combines partial modules of the NRF {f,h,r} and the BIEM q to adjust the foreground appearance using the estimated background illumination descriptor.

Our Results

Limitations

MY ALT TEXT

We suggest that future work tackle the current challenges that SIDNet faces: (1) distortion and loss of texture details in albedo estimation module. (2) The lack of spatially-varying harmonization in indoor scenes.

BibTeX

@article{hu2024sidnet,
        title={{SIDNet}: Learning Shading-Aware Illumination Descriptor for Image Harmonization},
        author={Hu, Zhongyun and Nsampi, Ntumba Elie and Wang, Xue and Wang, Qing},
        journal={IEEE Transactions on Emerging Topics in Computational Intelligence},
        year={2024},
        volume={8},
        number={2},
        pages={1290-1302}
      }