Art is a fascinating but extremely complex discipline. In fact, creating artistic images is often not only a time-consuming problem, but also requires a considerable amount of expertise. If this problem holds for 2D artworks, imagine extending it to dimensions beyond the picture plane, such as time (in animated content) or 3D space (with sculptures or virtual environments). This introduces new limitations and challenges that are addressed by this paper.
Previous results involving 2D stylization focus on video content divided frame by frame. The result is that the generated individual frames achieve high-quality stylization, but often lead to flickering artifacts in the generated video. This is due to the lack of temporal coherence of the produced frames. Moreover, it does not examine the 3D environment, which increases the complexity of the task. Other works that focus on 3D stylization suffer from geometrically inaccurate reconstructions of point clouds or triangle meshes and the lack of stylistic details. The reason lies in the different geometric properties of starting mesh and produced mesh, because the style is applied after a linear transformation.
The proposed method called Artistic Radiance Fields (ARF), can transfer the artistic features of a single 2D image to a real-world 3D scene, leading to artistic novel view renderings that are faithful to the input style image (Fig. 1).
For this purpose, the researchers exploited a photorealistic radiation field reconstructed from multiple images of real-world scenes and a new, stylized radiation field that supports high-quality stylized rendering from a new point of view. The results are shown in Fig.
As an example, in the input a number of real-world images of an excavator and a picture of the famous Van Gogh’s “starry night” paint as a “style” to be applied to it, the result is a colored bag with a smooth texture that resembles the image.
The ARF pipeline is presented in the figure below (Fig. 2).
The key point of this architecture is the coupling of the proposed Nearest Neighbor Featuring Matching (NNFM) loss and the color transfer.
The NNFM involves the comparison between the feature maps of rendered and style images, extracted using the notorious VGG-16 Convolutional Neural Network (CNN). In this way, the features can be used to guide the transfer of complex high-frequency visual details consistently across multiple viewpoints.
Color transfer is instead a technique used to avoid a noticeable color mismatch between the synthesized views and the style image. It involves a linear transformation of the pixels that make up the input images to match the mean and covariance of the pixels in the style image.
In addition, the architecture employs a deferred back-propagation method, which allows the calculation of losses on full-resolution images with reduced load on the GPU. The first step is rendering the image at full resolution and calculating the image loss and gradient with respect to the pixel colors, which produces a cached gradient image. Then these cache gradients are back-propagated patch-wise for the accumulation process.
The approach, ARF, presented in this paper brings several advantages. First, it leads to stunning creations of stylized images almost without artifacts. Second, the stylized images can be produced from new views with only a few input images, enabling artistic 3D reconstructions. Finally, with the closed backpropagation method, the architecture significantly reduces the GPU memory footprint.
This Article is written as a research summary article by Marktechpost Staff based on the research paper 'ARF: Artistic Radiance Fields'. All Credit For This Research Goes To Researchers on This Project. Check out the paper, github link and project.
Please Don't Forget To Join Our ML Subreddit
Daniele Lorenzi received his M.Sc. in ICT for Internet and Multimedia Engineering in 2021 from the University of Padua, Italy. He is a Ph.D. candidate at the Institute for Information Technology (ITEC) at the Alpen-Adria-Universität (AAU) Klagenfurt. He currently works at the Christian Doppler laboratory ATHENA and his research interests include adaptive video streaming, immersive media, machine learning, and QoS/QoE evaluation.