This paper introduces a new method for synthesizing 360° 3D scenes from a single image. The method generates panoramas and their corresponding depths in a coherent manner, addressing the limitations of existing state-of-the-art methods such as LucidDreamer and the WonderJourney. These methods add detail in turn following the generated trajectory, often resulting in visible seams as they loop back into the input image. In contrast, the method ensures consistency throughout the 360° scene, as shown in the figure. The yellow bar shows the area in each result that corresponds to the input.

Related link

• Paper: http://arxiv.org/abs/2412.04827v1
• Home page: https://people.engr.tamu.edu/nimak/Papers/PanoDreamer
• Code: https://github.com/avinashpaliwal/PanoDreamer

Paper introduction

PanoDreamer: Synthesizing 3D panoramas from a single image

The paper presents a novel method for generating a coherent 360° 3D scene from a single input image, PanoDreamer. Unlike existing methods of sequential scene generation, this method describes the problem as a single image panorama and depth estimation. Once a coherent panoramic image and its corresponding depth are obtained, the scene can be reconstructed by repairing small occluded areas and projecting them into 3D space. The main contribution of this paper is to make single image panorama and depth estimation into two optimization tasks, and introduce an alternate minimization strategy to effectively solve their objectives. The method is superior to existing techniques for single-image 360° scene reconstruction in terms of consistency and overall quality.

Single image panorama generation

In this paper, a repair diffusion model is used to solve the panorama generation problem of a single image, and it is framed as an optimization task solved by an alternate minimization strategy. During iteration, the input texture in the center gradually propagates outward.

Comparison of panoramic and multiple diffusion

Panoramic depth estimation

Similar to panorama generation, alternate minimization is used to align overlapping monocular depth blocks of cylindrical panormas, enabling estimates of a consistent 360° depth map.

Comparison of 3D scenes with other methods

Compare the render of this method PanoDreamer (right) with that of the baseline method (left). Try to choose different methods and scenarios!

conclusion

This paper presents a new method for generating 360° 3D scenes from a single input image. The method first generates a panoramic map and its corresponding depth map. After fixing the occluded areas, these images are used to optimize the 3DGS representation from which new views can be rendered. In order to create a coherent and globally consistent panorama, the paper designs the task as an optimization problem with two terms and solves it efficiently through an alternate minimization strategy. In addition, the problem of estimating the depth of panoramas using existing monocular depth estimation methods is proposed as an optimization, and alternate minimization is used to solve it. A large number of experiments show that this method is superior to the most advanced methods in both panorama generation and 3D scene reconstruction.