RecolorNeRF: A new easy-to-use color modification approach for the neural radiance field

Neural Radiance Fields (NeRF) are a powerful representation of 3D scenes, making it possible that they may one day replace photos and film as a new type of media. Supporting the drafting of such a new representation is essential to achieve this goal. Recent publications on the subject have explored NeRF modification in terms of geometry deformation, appearance modification, and style transfer, among other things. Recoloring is changing appearance that often involves adjusting certain color tones in a scene for enhancement or correction. This process is crucial to the film making process. In the example of Fig. 1, the red car can be transformed into a blue one in a photorealistic way using a recolored edition.

Figure 1: The technique automatically separates an image into layers, then recolors each layer separately. The color of each layer can be chosen by the user and our approach automatically stitches the layers together.

Palette-based color editing (PCE), one of the methods of recoloring an image currently in use, offers the most natural means of engagement. The PCE foresees these three essential steps: 1) Extracting a palette. The first step is to choose a group of representative colors and create a palette based on the landscape. 2) Decomposition of layers. They specify a corresponding image layer with a consistent color value for each item in the palette. The main purpose of this stage is to choose the best method for blending these layers to recreate the original image. Color montage. By changing the color of each layer, the scene can be naturally recolored based on the previous two processes.

Tan and colleagues presented a convex hull simplification technique for palette extraction as one of PCE’s SOTA approaches for imaging. The layer decomposition is then expressed as an optimization problem using the captured palette. The assumption of sparing mix weights makes the problem manageable. In this study, they presented a brand new technique called RecolorNeRF, which, to their knowledge, is the first attempt to use a fully learnable palette for layer decomposition in photorealistic PCE for NeRF representation. While PosterNeRF has experimented with palette-based NeRF recoloring, the results may be more realistic since color editing can only be enabled after posterization. As known, multiview images are often used to reconstruct the NeRF of a scene.

Thus, another possible way to perform PCE of NeRF is to extract palettes from pixels in all input images, following the method of, and then conducting layer decomposition and color editing for each rendered view of the pre-trained NeRF . While easy to implement, this strategy suffers from three major problems: First, recoloring in this way becomes post-processing of NeRF rendering, which causes expensive computational costs. Second, because each view is processed independently, the results require more view consistency. Third, palette extraction is heuristic, which can make the palette color less representative and the layer decomposition not clean enough, interfering with color manipulation. Their main suggestion is to improve the palette, layer blending weights, and volumetric radiance fields into one framework to address the issues mentioned earlier. Then they employ “over” composition to address complex scenarios as the final formulation of the image. Specifically, the alpha blending of a collection of sorted layers, each corresponding to an alpha weight, is used to represent each pixel. Then, for each layer, they construct a volumetric alpha field, which, like the radiance field, can also be represented by an MLP. Keep in mind that different tiers employ different MLPs. Therefore, they must jointly optimize the MLPs for the density range and the MLPs for the mix weights.

As is known, each of the previous PCE systems performed palette extraction individually. The first attempt to tweak the palette is what they offered. Two new designs are proposed to help with the joint optimization problem: 1) An innovative convex hull regularization is proposed to allow a limited color palette to faithfully represent the entire scene. 2) Normally, scarcity on blend weights are employed to make the palette color more realistic. The scarcity constraint is given a unique order-weighting mechanism to improve the ability to simulate complicated situations. RecolorNeRF can create photorealistic images with adjustable color schemes and robustly deconstruct the implicit representation, according to experiments. To their knowledge, they are the first to consider jointly optimizing the palette and alpha blend weights, for which a new convex hull adjustment is designed to make it fixable. The entire RecolorNeRF framework is carefully designed, allowing you to perform color editing of NeRF representation in a photorealistic way using a fully learnable palette for layer decomposition. The code has yet to be released, but a video demo can be found on the project website


Check out the Paper And Project. All credit for this research goes to the researchers of this project. Also, don’t forget to subscribe our Reddit page, Discord channelAnd Email newsletterwhere we share the latest news on AI research, cool AI projects and more.


Aneesh Tickoo is a Consulting Intern at MarktechPost. She is currently pursuing her BA in Data Science and Artificial Intelligence from Indian Institute of Technology (IIT), Bhilai. She spends most of her time working on projects that harness the power of machine learning. Her research interest is image processing and she is passionate about building solutions around it. She loves connecting with people and collaborating on interesting projects.


Add a Comment

Your email address will not be published. Required fields are marked *