ViewCrafter

ViewCrafter is a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images. It is designed for professionals and researchers in the field of computer vision and 3D graphics who need to generate realistic views of objects or scenes from limited input data. ViewCrafter's method takes advantage of the powerful generation capabilities of video diffusion models and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control.

The process involves building a point cloud representation using a dense stereo model, which enables accurately moving cameras for free-view rendering. To address the large missing regions, geometric distortions, and point cloud artifacts exhibited in the point cloud render results, ViewCrafter trains a point-conditioned video diffusion model to serve as an enhanced renderer, facilitating the generation of high-fidelity and consistent novel views based on the coarse point cloud renders. An iterative view synthesis strategy is adopted to achieve long-range novel view synthesis, enabling a more complete point cloud reconstruction and benefiting downstream tasks such as 3D-GS optimization.

Professionals in fields requiring advanced 3D visualization and rendering, such as architects, product designers, and filmmakers, can greatly benefit from ViewCrafter. It allows them to create immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views. Additionally, ViewCrafter facilitates scene-level text-to-3D generation for more imaginative content creation, making it a valuable tool for enhancing creativity and productivity in various industries.