OCC-NeRF: Occlusion-Free Scene Recovery via Neural Radiance Fields

Chengxuan Zhu1,2, Renjie Wan3, Yunkai Tang1,2, Boxin Shi1,2
1 Natl. Engineering Research Center of Visual Technology, School of Computer Science, Peking University 2 Natl. Key Lab. for Multimedia Information Processing, School of Computer Science, Peking University 3 Department of Computer Science, Hong Kong Baptist University
Paper With animation! Supplementary With animation! Code Data

OCC-NeRF removes the foreground occlusion in a scene, and refines the error-prone camera parameter estimation.

Abstract

Our everyday lives are filled with occlusions that we strive to see through. By aggregating desired background information from different viewpoints, we can easily eliminate such occlusions without any external occlusion-free supervision. Though several occlusion removal methods have been proposed to empower machine vision systems with such ability, their performances are still unsatisfactory due to reliance on external supervision. We propose a novel method, OCC-NeRF, for occlusion removal by adequately considering the benefits of multiple viewing angles, which directly builds a mapping between viewing angles and their corresponding scene details leveraging Neural Radiance Fields (NeRF). We also develop an effective scheme to jointly optimize camera parameters and scene reconstruction when occlusions are present. An additional depth constraint is applied to supervise the entire optimization without labeled external data for training. Our experimental results on existing and newly collected datasets validate the effectiveness of our method.

Animated Comparison

BibTeX

@inproceedings{zhu2023occlusion,
  author    = {Chengxuan Zhu and Renjie Wan and Yunkai Tang and Boxin Shi},
  title     = {Occlusion-Free Scene Recovery via Neural Radiance Fields},
  journal   = {CVPR},
  year      = {2023},
}