This AI paper introduces PaletteNeRF, a new method for modifying the photorealistic appearance of neural radiance fields (NeRF) based on 3D color decomposition

The ability of Neural Radiance Fields (NeRF) and its derivatives to accurately recreate real-world 3D scenes from 2D photos and enable new high-quality photorealistic synthesis has attracted increasing interest in recent years. However, because the look of the scene is implicitly recorded in neural features and network weights that do not allow for local manipulation or intuitive tampering, such volumetric representations are difficult to modify. Several methods have supported the modification of NeRF. One group of techniques recovers the material qualities of the scene so that they can be altered, such as surface roughness, or rendered again under new lighting conditions.

Such techniques depend on an accurate assessment of scene reflectance, which is often difficult for complicated real-world images taken in an open environment. Another class of methods involves discovering a latent code that NeRF can be trained to use to achieve the desired look. These techniques, however, don’t offer fine-grained editing and often have limited capabilities and flexibility. Also, while some other methods may tailor the look of NeRF to suit a certain type of image, they sometimes fail to preserve the same amount of photorealism in the original scene. They suggest PaletteNeRF in this work as an innovative way to facilitate flexible and easy editing of NeRF.

Figure 1: PaletteNeRFan innovative technique for effective modification of the appearance of the Neural Radiance Field (NeRF). The method reconstructs a NeRF and splits its appearance into a series of (b) 3D palette-based color bases using (a) multi-view images as training input. This makes it possible to (c) recolor the scene naturally and photos realistically with 3D coherence from any angle. Furthermore, they demonstrate that (d) the approach enables several palette-based editing applications, including lighting adjustment and 3D photorealistic style transfer.

Their approach is influenced by earlier techniques for image editing that used color palettes, which use a condensed selection of hues to represent the full spectrum of hues in an image. They combine specular and diffuse components to describe the brightness of each point, and further divide the diffuse component into a linear combination of view-independent common color bases. To reduce the disparity between the produced images and the ground truth images, they jointly optimize the per-dot specular component, global chromatic bases, and per-dot linear weights during training.

Introducing Hailo-8™: An AI Processor Using Computer Vision for Multi-Camera Multi-Person Re-Identification (Sponsored)

To promote the sparsity and spatial consistency of the decomposition and create a more meaningful grouping, they also apply unique regularizers on the weights. By freely modifying the taught color bases, students can intuitively adjust the appearance of NeRF with the suggested structure (Fig. 1). Furthermore, they demonstrate how their system can be used in conjunction with semantic features to provide semantic modifications. Their technique offers more globally consistent and 3D consistent scene recoloring results across arbitrary viewpoints than previous palette-based image or video editing techniques. They show that their approach surpasses basic approaches numerically and subjectively, allowing for more precise local color change while faithfully maintaining the photorealism of the 3D scene.

In summary,

• Offers a unique facility to make altering NeRF easier by dividing the radiance field into a weighted blend of learned color bases.

• To produce intuitive decompositions, they devised a reliable optimization technique using unique regularizers.

• Their method allows for realistic palette-based customization of appearance, allowing even inexperienced users to interact with NeRF in an easy and manageable way on common hardware.


Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to subscribe our Reddit page and discord channelwhere we share the latest news on AI research, cool AI projects and more.


Aneesh Tickoo is a Consulting Intern at MarktechPost. She is currently pursuing her BA in Data Science and Artificial Intelligence from Indian Institute of Technology (IIT), Bhilai. She spends most of her time working on projects that harness the power of machine learning. Her research interest is image processing and she is passionate about building solutions around it. She loves connecting with people and collaborating on interesting projects.


Add a Comment

Your email address will not be published. Required fields are marked *