Data for: Iterating the transient light transport matrix for non-line-of-sight imaging
Abstract
Time-of-flight non-line-of-sight (NLOS) imaging measures transient light transport at a visible diffuse relay surface to recover the geometry of objects hidden from direct line of sight. These measurements are acquired by scanning a pulsed laser across the relay surface and recording the time-resolved response from visible patches with a single-photon detector. The complete set of such measurements forms the transient light transport matrix (TLTM), the time-resolved impulse response between illumination and detection points. Prior NLOS methods have mainly used only subsets of this matrix to reconstruct hidden geometry. Here, we show that light transport measured between visible relay-surface patches can be used to recover light transport between hidden-surface patches. Using a dense laser scan and a gated 16 × 16 single-photon avalanche diode array, we capture the complete first-order TLTM (TLTM-1) on a planar relay wall. We then process the measured TLTM-1 to compute TLTM-2, the time-resolved light transport between hidden-surface patches. This is achieved by computationally focusing virtual illumination and detection independently at different points in the hidden scene, effectively turning the relay surface into a synthetic time-of-flight camera. Experiments and simulations show that TLTM-2 reveals hidden-scene interactions beyond conventional geometry recovery, including indirect shadows, interreflections, and subsurface scattering. We further demonstrate direct-indirect transport separation, virtual relighting, and dual photography of occluded scenes. These results establish an iterative framework for recovering higher-order transport (TLTM-n).
https://doi.org/10.5061/dryad.47d7wm3qk
Description of the data and file structure
These datasets were collected using a time-of-flight non-line-of-sight imaging system with a gated 16 × 16 single-photon avalanche diode array. The system captures time-resolved measurements over two-dimensional illumination and detection grids on a planar relay surface. The measurements form first-order transient light transport matrices, which are processed using computational beamforming algorithms to recover spatiotemporal light transport between surfaces in the hidden scene.
The archive data.zip contains four datasets used to generate selected results in the main manuscript and supplementary materials.
Files and variables
File: data.zip
Description: Contains 4 datasets used to generate some results in our main paper.
- datasets/LFOV/14_1F: The SPAD array was focused on the relay surface over a large field of view. This dataset captures a hidden scene containing the number “2” and the letter “T” while sampling light transport from different regions of the relay surface. This dataset is used for large-field-of-view relighting and dual-photography results.
- datasets/MFOV/209_20F: The SPAD array was focused over a medium field of view on the relay surface. This dataset captures a hidden scene containing the letters “n” and “W,” a curved back wall, and an optical mirror. This dataset was split into training and test subsets to evaluate beamforming-based artifact suppression without overfitting to measurement noise.
- datasets/MFOV/209F_10F_Train: Training subset generated from
datasets/MFOV/209_20F. This dataset is used to compute beamforming coefficients by solving an additional linear inverse problem. - datasets/MFOV/209F_10F_Test: Test subset generated from
datasets/MFOV/209_20F. Beamforming coefficients computed from the training subset are applied to this test dataset to generate the final reconstruction results.
Code/software
All results were generated using MATLAB 2024A.
Representative analysis code and instructions for reproducing the manuscript figures are provided separately through the associated Code Ocean capsule.
The datasets were collected using a time-of-flight non-line-of-sight imaging system consisting of a pulsed laser, a two-mirror galvanometer for relay-wall scanning, and a gated 16 × 16 single-photon avalanche diode array focused onto a planar diffuse relay wall. For each dataset, the laser was scanned over a two-dimensional grid on the relay surface while the SPAD array recorded time-resolved photon counts from multiple relay-wall detection positions.
The resulting measurements form first-order transient light transport matrices that represent the time-resolved impulse response between relay-wall illumination and detection positions. These measurements were processed using phasor-field beamforming and Rayleigh–Sommerfeld propagation operators to reconstruct hidden-scene geometry and estimate second-order transient light transport between hidden-surface patches.
For datasets used in beamforming artifact suppression, the measurements were split into training and test subsets. The training subset was used to estimate beamforming coefficients, and the test subset was used to generate the final reconstructions.