Since there is a lack of publicly available datasets containing multi-view, near-perfectly aligned RGB and thermal images, we share our collected dataset, called ThermalMix, here: zenodo.org/records/1106...
ThermalMix includes six common objects and a total of about 360 images.
5/7

A core challenge in building multi-sensory NeRFs is cross-modality calibration. We apply offline camera calibration prior to data capturing, leading to near-perfect alignments between images from different sensors. For thermal images, we chose a perforated aluminum plate.
4/7

We systematically compare four different strategies to learn multi-modal NeRFs from RGB + thermal, RGB + IR, and RGB + depth data: (1) Train from scratch on both modalities, leveraging camera poses computed from RGB images. (2) Pre-train on RGB, fine-tune on second modality.
2/7

How can we learn a multi-modal neural radiance field? What’s the best way to integrate images from a second modality, other than RGB, into NeRF? Check out our new paper!
Project page: mert-o.github.io/ThermalNeRF/
Paper: arxiv.org/abs/2403.11865
Dataset: zenodo.org/records/1106...
1/7