Data and code from: Deep learning-based autonomous retinal vein cannulation in ex vivo porcine eyes
Data files
Dec 04, 2025 version files 7.51 GB
-
contact_network.zip
5.96 KB
-
dataset_for_all.zip
7.50 GB
-
navigation_network.zip
2.15 MB
-
puncture_network.zip
189.49 KB
-
README.md
3.12 KB
-
requirements.txt
303 B
-
SHER_visualiaztion_and_control.zip
7.73 MB
Abstract
Retinal Vein Cannulation (RVC) is an emerging method for treating Retinal Vein Occlusion (RVO). The success of this procedure depends on surgeon expertise and, recently, robotic assistance. This paper proposes an autonomous RVC workflow leveraging deep learning and computer vision. Two Steady Hand Eye Robots (SHERs) control a 100-micrometer metal needle and a medical spatula to execute precise tasks. Three convolutional neural networks are trained to predict needle movement direction and identify contact and puncture events. A surgical microscope with an intraoperative Optical Coherence Tomography (iOCT) system captures the surgical field through a microscope and cross-sectional images. The goal is to enable the robot to autonomously carry out the critical steps of the RVC procedure, especially those that are challenging and require expert knowledge. The less technically demanding tasks are assigned to the user, who also supervises the robot during these steps. Our method is tested on 20 ex vivo porcine eyes, achieving a success rate of 90 %. Additionally, we simulate eye movements caused by breathing on six other ex vivo porcine eyes. With the eyes moving in a sinusoidal pattern, we achieve a success rate of 83 %, demonstrating the robustness and stability of the proposed workflow. Our results demonstrate that the autonomous RVC workflow, incorporating deep learning and robotic assistance, achieves high success rates in both static and dynamic conditions, indicating its potential to enhance the precision and reliability of RVO treatment.
Introduction
This repository contains all necessary code and data for reproducing the training results and control algorithms for the paper titled "Deep Learning-Based Autonomous Retinal Vein Cannulation in ex vivo Porcine Eyes". Please unzip contact_network.zip, dataset_for_all.zip, navigation_network.zip, puncture_network.zip, and SHER_visualiaztion_and_control.zip first.
Installation
We recommend running this in a virtual environment:
# generate a virtual environment with name test_env and Python 3.8.16 installed
conda create -n test_env python=3.8.16
# activate the environment
conda activate test_env
# deactivate the environment
conda deactivate
# delete the virtual environment and all its packages
conda remove -n test_env --all
To install all necessary packages, please navigate to the cloned directory and run the following code in the terminal:
pip install -r requirements.txt
Navigation Network
Train the navigation network using navigation_network_train.ipynb. Validate and test accuracy using navigation_network_val_test.ipynb.
Contact Network
Train, validate, and test the contact network using contact_network_train_val_test.ipynb.
Puncture Network
Train the puncture network using puncture_network_train.ipynb. Validate and test accuracy using puncture_network_val_test.ipynb.
SHER Visualiaztion and Control
The code in this folder was developed for Steady Hand Eye Robot (SHER) at Johns Hopkins University only. If you need to use the code for your own robot system, please adjust the code accordingly. Following the below steps in the terminal to achieve autonomous retinal vein cannulation with fixed eyes and vertical sinusoidal motion and compensation.
- subscribe microscope image through HD SDI output and publish to /decklink/camera/image_raw ros topic using gscam package.
- run the keyboard contorller code:
sudo -s
cd ./keyboard_controller/
source devel/setup.bash
# publish the key commonds
rosrun key_publisher key_publisher.py
- open a new terminal:
cd ./keyboard_controller/
source devel/setup.bash
rosrun key_move key_move.py
For Autonomous RVC with Fixed Eyes
# publish B-scans
python3 visualization_b_scans.py
# visualization of the network output on microscope image and B-scan
python3 visualization.py
# control and inference
python3 visualization_inference.py
# data saving
python subscribe_all.py
Autonomous RVC with Vertical Sinusoidal Motion and Compensation
# publish B-scans
python3 publish_b_scans_for_breath_simulation.py
# visualization of the network output on microscope image and B-scan
python3 visualization.py
# initialize the XYZ linear stage
python xyz_stage_move_to_position.py
# simulate the breathing using the XYZ linear stage
python xyz_stage_rvc_z_axis_sinewave_motion_recording.py
# control and inference
python3 breath_simulation_inference.py
# data saving
python subscribe_all.py
For Breathing Simulation Only - Show How It Works
python3 breath_simulation.py
