Neural Radiance Fields –the Baptistery of St. John in Florence reconstructed with crowd-sourced NeRF

CIPA2023 - Video Contest
Neural Radiance Fields –the Baptistery of St. John in Florence reconstructed with crowd-sourced NeRF
M. Häcki, L. Vandenabeele , M. Pfister

With the rapid development of digital technologies, new possibilities of representing objects and buildings are emerging. These opportunities can be used to document and communicate cultural heritage in new ways to a wider audience. The video explores the dome of the Baptistery of St. John in Florence using a neural radiance field (NeRF). NeRF are a novel method from the field of deep-learning-based computer vision, which enable to synthesize new views of complex three-dimensional scenes with the help of a neural network. The key difference to the Structure from Motion (SfM) pipeline is the use of deep-learning-based radiance fields to represent a three-dimensional scene and volume rendering instead of more common rendering techniques for visualization. As the radiance is direction-dependent in a NeRF, accurate rendering of transparency and reflections is possible.
To create the NeRF of the dome of the baptistery St. John shown in the video, only 27 touristic pictures crowd-sourced from the internet were used. The processing was done in Nerfstudio, an open-source project which offers a simple pipeline for the training and rendering of NeRF scenes as well as the export of renders, point clouds and meshes via its web viewer. The trained model of the dome of the Baptistery of St. John weights only about 150 megabytes but allows high-resolution renderings of the structure and its shiny mosaic from any perspective within a couple of minutes. In addition to the generation of high-resolution renderings, the NeRF approach offers a range of promising possibilities in various fields such as image processing or surface reconstruction.