Relief-type cultural heritage objects, prevalent in historical sites worldwide, often suffer from varying degrees of damage and deterioration. While traditional restoration methods demand extensive manual labor and specialized knowledge, a groundbreaking neural network model developed by researchers now enables the reconstruction of these reliefs as three-dimensional digital images from old photographs that capture their pre-damage condition.

Relief carvings, with figures that protrude from a background like a wall or slab, create a captivating sense of depth. However, many of these invaluable artworks have deteriorated over time. While modern 3D scanning and photogrammetry can preserve their current forms, they cannot restore the original appearance of these carvings. Conventional restoration techniques are laborious, further complicating the process.

To address this, researchers have turned to the 3D digital reconstruction of reliefs using historic photographs taken before the damage occurred. Unlike traditional 3D sculptures or 2D paintings, reliefs are designed to be viewed from the front or sides, allowing a single image to provide most of the information required for effective reconstruction.

In a significant advancement, a multinational research team led by Professor Satoshi Tanaka from Ritsumeikan University in Japan, alongside Dr. Jiao Pan from the University of Science and Technology Beijing, has developed an innovative multi-task neural network that facilitates this 3D reconstruction and digital preservation. "Previously, our team proposed a method based on monocular depth estimation from photos, achieving 95% reconstruction accuracy. However, finer details like human faces were often missing due to high compression of depth values in 2D relief images. Our new method enhances depth estimation, particularly around soft edges, using a novel edge-detection approach," explains Prof. Tanaka.

This collaborative effort also included Prof. Liang Li from Ritsumeikan University and Prof. Xiaojuan Ban from the University of Science and Technology Beijing. Their findings were presented at the ACM Multimedia 2024 conference in Australia and published in the conference proceedings.

The proposed multi-task neural network performs three interconnected tasks: semantic segmentation, depth estimation, and soft-edge detection. This trio works in concert to improve the accuracy of the 3D reconstruction. The core strength of the network lies in its depth estimation, achieved through an innovative soft-edge detector and an edge matching module. Rather than treating edge detection as a binary classification task, the soft-edge detector classifies edges in relief images as multi-class entities. This approach recognizes that edges represent not only changes in brightness but also variations in curvature, referred to as "soft edges."

By assessing the degree of softness of these edges, the network enhances depth estimation. The edge matching module employs two soft-edge detectors to extract multi-class soft-edge maps alongside a depth map from the input relief photo. This matching process hones in on soft-edge regions, leading to more detailed depth estimation. Additionally, the network optimizes a dynamic edge-enhanced loss function, incorporating loss from all three tasks to produce clear, detailed 3D images of reliefs.

The researchers applied this innovative model to reconstruct hidden reliefs at the Borobudur Temple, a UNESCO World Heritage Site in Indonesia. "The ground-level wall reliefs of Borobudur are obscured by stone walls due to reinforcement work conducted during the Dutch colonial period, rendering them invisible. Our multi-task neural network successfully reconstructed these hidden sections from surviving old photographs, enabling virtual exploration of these unseen treasures through computer visualization and virtual reality," highlights Prof. Tanaka, emphasizing the potential impact of their work.

Looking ahead, he states, "Our technology offers immense potential for preserving and sharing cultural heritage. It creates new opportunities not only for archaeologists but also for immersive virtual experiences through VR and metaverse technologies, ensuring the preservation of global heritage for future generations."