This AI paper introduces PaletteNeRF, a new method for modifying the photorealistic appearance of neural radiance fields (NeRF) based on 3D color decomposition

The ability of Neural Radiance Fields (NeRF) and its derivatives to accurately recreate real-world 3D scenes from 2D photos and enable new high-quality photorealistic synthesis has attracted increasing interest in recent years. However, because the look of the scene is implicitly recorded in neural features and network weights that do not allow for local manipulation or intuitive tampering, such volumetric representations are difficult to modify. Several methods have supported the modification of NeRF. One group of techniques recovers the material qualities of the scene so that they can be altered, such as surface roughness, or rendered again under new lighting conditions.

Such techniques depend on an accurate assessment of scene reflectance, which is often difficult for complicated real-world images taken in an open environment. Another class of methods involves discovering a latent code that NeRF can be trained to use to achieve the desired look. These techniques, however, don’t offer fine-grained editing and often have limited capabilities and flexibility. Also, while some other methods may tailor the look of NeRF to suit a certain type of image, they sometimes fail to preserve the same amount of photorealism in the original scene. They suggest PaletteNeRF in this work as an innovative way to facilitate flexible and easy editing of NeRF.

Their approach is influenced by earlier techniques for image editing that used color palettes, which use a condensed selection of hues to represent the full spectrum of hues in an image. They combine specular and diffuse components to describe the brightness of each point, and further divide the diffuse component into a linear combination of view-independent common color bases. To reduce the disparity between the produced images and the ground truth images, they jointly optimize the per-dot specular component, global chromatic bases, and per-dot linear weights during training.

To promote the sparsity and spatial consistency of the decomposition and create a more meaningful grouping, they also apply unique regularizers on the weights. By freely modifying the taught color bases, students can intuitively adjust the appearance of NeRF with the suggested structure (Fig. 1). Furthermore, they demonstrate how their system can be used in conjunction with semantic features to provide semantic modifications. Their technique offers more globally consistent and 3D consistent scene recoloring results across arbitrary viewpoints than previous palette-based image or video editing techniques. They show that their approach surpasses basic approaches numerically and subjectively, allowing for more precise local color change while faithfully maintaining the photorealism of the 3D scene.

In summary,

• Offers a unique facility to make altering NeRF easier by dividing the radiance field into a weighted blend of learned color bases.

• To produce intuitive decompositions, they devised a reliable optimization technique using unique regularizers.

• Their method allows for realistic palette-based customization of appearance, allowing even inexperienced users to interact with NeRF in an easy and manageable way on common hardware.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to subscribe our Reddit page and discord channelwhere we share the latest news on AI research, cool AI projects and more.