Dataset and trained models for video denoising in fluorescence guided surgery
Data files
Jan 29, 2025 version files 34 GB
-
FGS_Data_and_Models.zip
34 GB
-
README.md
4.95 KB
Abstract
Fluorescence guided surgery (FGS) is a promising surgical technique that gives surgeons a unique view of tissue that is used to guide their practice by delineating tissue types and diseased areas. As new fluorescent contrast agents are developed that have low fluorescent photon yields, it becomes increasingly important to develop computational models to allow FGS systems to maintain good video quality in real time environments. To further complicate this task, FGS has a difficult bias noise term from laser leakage light (LLL) that represents unfiltered excitation light that can be on the order of the fluorescent signal. This dataset contains the data used to develop and train video denoising models for fluroescence guided surgery with LLL. This dataset contains bright fluorescence data in a mock chicken thigh surgery for FGS video simulation, non-fluorescent video for LLL simulation, as well as a number of calibration datasets for properly simulating a comercial system, and real noise video for testing. We also provide result videos of our denoising models trained with this data and the trained models.
README: Dataset and trained models for Video Denoising in Fluorescence Guided Surgery
https://doi.org/10.5061/dryad.8gtht76x9
This repository contains the data necessary for calibrating an accurate noise model for the OnLume Avata system, a fluorescence guided surgery commercial system. It also contains a number of low-noise fluorescent video of a mock chicken thigh surgery that can be used to simulate realistic noisy videos which can be used to train video denoising models for fluorescence guided surgery.
Description of the data and file structure
There are three main folders in our dataset:
/data : this folder contains our dataset videos. Each video is in "triple view" mode where the bottom left quadrant is the reference video, the top right is the fluorescence video and the top left is the fluorescence video overlayed in false color on the reference video. The full videos are 2048x1536 at 15 fps, each quadrant is 1024x768. Videos can be viewed with VLC video player (https://www.videolan.org/) or most other suitable video player.
/data has 4 subfolders
- /data/lowdose_chicken : contains 2 videos of a mock chicken thigh surgery with a low concentration of ICG injected. These videos make up our OL-Real dataset and have real system noise.
- /data/no-icg : contains 4 videos split into 3 training videos and 1 testing video make up the OL-LLL dataset. These videos are of a mock chicken thigh surgery with no icg injected, so the fluorescence video contains only laser leakage light.
- /data/quel_phantom : contains 2 videos that make up the OL-Phantom. These videos contain videos of the Quel phantom (RCS-ICG-ST01-QUEL0) which is used to calibrate our noise model.
- /data/train_test : contains 3 subfolders, our pretrained LLL-PN .pt files, and our train test split file.
- train_test.pt : this file represents a pytorch dictionary and is loaded in our training code. This file contains the indexes used to split the training/testing videos into 100 frame chunks in the network training loops. It is required to run the training and testing code in the referenced zenodo code base.
- NAF_32_Standard_LLv2_h256_w256_laser_leak_ep100_L1_loss.pt : this is our pre-trained laser leakage light prediction network (LLL-PN) used to simulate laser leakage light in our simulation pipeline. NAF_32_Standard_LLv2_h256_w256_laser_leak_ep100_L1_loss_reverse_test_train.pt is a pre-trained LLL-PN with different testing and training sets and is used to test the robustness of our models to changing LLL. These models are loaded by our simulation models in the zenodo code base, the code required to run and load these models is in the zenodo code base.
- /data/train_test/dark_frames : contains the OL-Dark dataset. This folder contains a number of dark frame videos that capture the read noise of our system that is used in our simulation method. This folder contains 2 unique videos which are each duplicated 3 times for training (train.mp4, train1.mp4 and OL-2023-09-15-143614-000002-record.mp4) and testing (test.mp4, test1.mp4, and OL-2023-09-15-142907-000001-record.mp4). These are duplicated to allow parallel training or testing of multiple models without creating read conflicts.
- /data/train_test/test : contains the triple view videos used in our test set
- /data/train_test/train: contains the triple view videos used in our training set
/models : this folder contains 5 pretrained models. BL_AM, BL_RNN, and BL_SW are our baseline models we created for this problem. NAF_32_Standard_h256_w256_OL24_ep3000_lowlr_1600.pt is a partially trained NafNet32 model used in the pretraining step of our BL_AM model. NAF_32_Standard_h256_w256_OL24_ep3000_lowlr_final.pt is the final NafNet32 model that can be used as a comparison to our baseline models. These models are pytorch .pt files and the python model code is in the referenced zenodo code base as well as examples on how to load these models into pytorch.
/results : this folder contains 4 result videos at different simulated noise levels. It also contains metric_results.pkl which contains 5 different model metrics on simulated data per test set video. This pkl file is a pandas dataframe and can be opened in pandas, the attached zenodo code contains and example jupyter notebook on how to read and process this pkl file as well as the code to generate the result videos and pkl file.
Sharing/Access information
Part of this dataset is derived from Seets, Trevor et al. (2024). Data for OFDVDnet: A sensor fusion approach for video denoising in fluorescence guided surgery [Dataset]. Dryad. https://doi.org/10.5061/dryad.v6wwpzh3w
Code/Software
The provided python code can be used to use this data to simulate noisy data and train our denoising models. The README provided with the code contains information on running the code.
Methods
This dataset contains fluorescence and reference (RGB) video of a mock chicken thigh surgery captured on the OnLume Avata commercial fluroescence guided surgery (FGS) system using indocyanine green (ICG) as the contrast agent. It contains a number of different conditions used to calibrate a realistic noise model and simulate real noisy data for the training of deep learning based video denoising models.
- First, this dataset contains a number of videos captured with high concentration of ICG that can act as a basis for simualtion and as a ground truth in training. This portion of the dataset is an expansion of the dataset in [1], and contains the videos from [1] along with another ~100 minutes of video.
- Second, our dataset contains 15 minutes of real noisy data of a mock chicken thigh surgery with low concentration of ICG.
- Third, our dataset contains 20 minutes of mock chicken thigh surgery with no ICG that can be used to simualte laser leakage light.
- Finally, our dataset contains a number of videos used to calibrate our noise model to properly simualte the OnLume Avata system noise.
[1] Seets, Trevor et al. (2024). Data for OFDVDnet: A sensor fusion approach for video denoising in fluorescence guided surgery [Dataset]. Dryad. https://doi.org/10.5061/dryad.v6wwpzh3w