Hummingbirds use compensatory eye movements to stabilize both rotational and translational visual motion
Data files
Nov 14, 2024 version files 14.70 GB
-
analysisScripts.zip
179.26 KB
-
eyeCalibrationImages.zip
4.56 MB
-
processedData.zip
842.87 MB
-
rawData.zip
3.50 GB
-
README.md
12.35 KB
-
testVideo.zip
583.62 MB
-
validationData.zip
3.58 MB
-
videos.zip
9.77 GB
Abstract
To maintain stable vision, behaving animals make compensatory eye movements in response to image slip, a reflex known as the optokinetic response (OKR). Although OKR has been studied in several avian species, eye movements during flight are expected to be minimal. This is because vertebrates with laterally placed eyes typically show weak OKR to nasal-to-temporal motion (NT), which simulates typical forward locomotion, compared to temporal-to-nasal motion (TN), which simulates atypical backward locomotion. This OKR asymmetry is also reflected in the pretectum, wherein neurons sensitive to global visual motion also exhibit a TN bias. Hummingbirds, however, stabilize visual motion in all directions through whole-body movements and are unique among vertebrates in that they lack a pretectal bias. We therefore predicted that OKR in hummingbirds would be symmetrical. We measured OKR in restrained hummingbirds by presenting gratings drifting across a range of speeds. OKR in hummingbirds was asymmetrical, though the direction of asymmetry varied with stimulus speed. Hummingbirds moved their eyes largely independent of one another. Consistent with weak eye-to-eye coupling, hummingbirds also exhibited disjunctive OKR to visual motion simulating forward and backward translation. This unexpected oculomotor behavior, previously unexplored in birds, suggests a potential role for compensatory eye movements during flight.
README
This folder contains all files and code related to the paper titled "Hummingbirds use compensatory eye movements to stabilize both rotational and translational visual motion".
Because of the size of some files (high-speed video, in particular), intermediate files are not included, as the total size of all files associated with this project is >5 tb. An example raw video is included to test pupilTracker.py. However, all videos are uploaded as high-quality MP4 files following consultation with journal staff.
The organization of these files is as follows:
analysisScripts -- contains major code for collection of pupil data (python), processing of pupil data (MATLAB), and analysis and visualization (R).
-- pupilTracker.py -- INPUT: video, DLC file, OUTPUT: file in "rawData". DESCRIPTION: extracts the position of the pupil using the video and a DeepLabCut file associated with that video. The resulting file combines the DeepLabCut output with the position of the pupil, tracked as a contour using openCV.
-- ocularAnalysis_autoProcess.mlx -- INPUT: kinematic data from "rawData" + stimulus file, OUTPUT: synced kinematic data in "processedData". DESCRIPTION: This is the major workhorse of the analysis. Raw position data, produced by pupilTracker.py is imported and processed. This includes synchronziation with the stimulus files, filtering, computing relative pupil position in degrees, segmenting into slow and rapid eye movements, and computing statistics for those segments of data.
-- ocularAnaylsis.Rmd -- INPUT: files from "processedData", OUTPUT: figures, summary results. DESCRIPTION: This script produces the summary stats and figures of the paper. Some minor touches to the panel layout were made using Adobe Illustrator.
rawData.zip -- contains all files output by the combination of DeepLabCut and pupilTracker.py. Files are organized according the the day of recording.
-- Folders -- The naming convention for each folder is hummingbird_recordingData_spatialFrequency. For example, the folder 'hummingbird_20240315_0.0625' contains all raw data recording on 15 March 2024. On this date, the spatial frequency of the stimulus was 0.0625 cycles per degree. Within each of these folders, you will find a series of .csv files. The data for recording sessions (a stimulus set presented to an individual bird) are contained across three files that share the same recording time in their file names - an L file, and R file, and an OKR_stimulus file. For example, 09-48-47.000L.csv, 09-48-47.000R.csv, OKR_stimulus_09-48-47.000.csv are a set and contain the data recorded at 09:48:47.000.
--- L and R files --- These contain the position data from the left and right side of the animal, respectively. For these files, Columm A (dt) contains the time in seconds relative to the start of the first stimulus presentation. Column B contains video frame number, starting from 0. Columns C through AL contain the x and y position (pixel units) along with the confidence (0-1) output by DeepLabCut for each tracked feature. Columns AM through AR contain the x position, y position, and area of the pupil tracking using OpenCV (pixel units). AM-AO are the result of blob detection. AP-AR are the result of contour detection. The latter method was used in all analysis.
--- OKR_stimulus files --- These describe the stimulus conditions from the recording session. These include the Left and Right Spatial Frequency CPD, Direction, Temporal Frequency, Speed PPS, dt, index, and Time. Spatial Frequency CPD is that of the stimulus in cycles per degree. Direction is 0, 180, blank, or station. 0 is nasalward for the monitor on the left side; temporalward for the right side. 180 is temporalward for the monitor on the left side; nasalward for the right side. Station indicates that the stimulus on that side remained stationary (no motion). Blank indicates that the monitor switched to 50:50 gray at the start of stimulus motion. See the paper for details. The dt column contains the time of motion onset relative to the start of the first stimulus presentation. In other words, the dt columns in these files are used to synchronize the stimulus data with the kinematic data in the L and R files. The Time column and index column are unused, but indicate global time and the order of stimulus presentation prior to randomization, respectively.
To see the association between recording times and indivdual bird identities, see ocularAnalysis_autoProcess.mlx.
processedData.zip -- contains the output from ocularAnalysis_autoProcess.mlx. These files are read into R for processing.
-- SFComp -- This folder contains the data output by ocularAnalysis_autoProcess.mlx for the original set of experiments - six individual's response to stimuli at five spatial frequencies and a range of speeds.
-- T1CPD -- This folder contains the data output by ocularAnalysis_autoProcess.mlx for the follow-up experiment, presented in Figure 4E - six individual's response to stimuli at 1 cycle per degree and an expanded range of speeds.
Files within these folders are named as follows: birdID_spatialFrequency_recordingTime (e.g. 280_0.25_15-38-54.000.csv).
--- .csv files --- Each csv file contains the following columns:
- frame - frame count of video
- dt - time (seconds) relative to start of first stimulus presentation
- periTime - time (seconds) relative to motion onset for that trial (-2s through 10s)
- trialID - a unique trial identifier generated randomly for plotting purposes
- SpatialFrequency - spatial frequency of stimulus (cycles per degree)
- Speed - speed of stimulus motion (degrees per second)
- Temporal Frequency - temporal frequency of stimulus (Hz)
- LeftDirection - direction of stimulus motion on the left monitor relative to the bird. TN: Temporal-Nasal, NT: Nasal-Temporal.
- RightDirection - direction of stimulus motion on the right monitor relative to the bird. TN: Temporal-Nasal, NT: Nasal-Temporal.
- blinkBreak - binary indicating whether a blink was detected at that frame in either eye.
- saccadeBreak - binary indicating whether a saccade was detected at that frame in either eye.
- break - union of blinkBreak and saccadeBreak.
- The remaining columns start with either 'Left' or 'Right', corresponding to data for left and right eyes (e.g. Left_angle_X).
- _angle_x - horizontal position of the eye (degrees). Data are replaced with NaN if break is 1.
- _angle_y - UNUSED - vertical position of the eye (degrees). Data are replaced with NaN if break is 1.
- _angle_xRAW - horizontal position of the eye (degrees).
- _angle_yRAW - UNUSED - vertical position of the eye (degrees).
- _pupil_xRAW_mm - horizontal position of the eye (millimeters).
- _pupil_yRAW_mm - UNUSED - vertical position of the eye (millimeters).
- _blink - binary indicating whether a blink was detected for that frame in that eye.
- _saccade - binary indicating whether a saccade was detected for that frame in that eye.
- _purk_rel_x - horizontal position of the corneal reflection (millimeters).
- _purk_rel_y - vertical position of the left corneal reflection (millimeters).
- _sp_trial - trial (1-3) for a given combination of stimulus conditions.
- _sp_startTime - start time of the slow phase (seconds) relative to trial start (periTime).
- _sp_cycle - running count of the number of slow phases occurring within a given trial.
- _sp_duration - duration of a given slow eye movement (seconds)
- _sp_slope - horizontal slope of a given slow eye movement (degrees per second)
- _sp_mag - horizontal magnitude of a given slow eye movement (degrees)
- _sp_startPosition - horizontal position of the eye at the start of a given slow phase (degrees)
- _sp_endPosition - horizontal position of the eye at the end of a given slow phase (degrees)
- _sp_RMSE - Root Mean Squared Error (degrees) of linear regression to a given slow phase
- _sp_fit - fitted values (degrees) from linear regression for each slow phase
- _sp_filt - median filtered version of horizontal position of the eye (degrees) for a given slow phase
- _sp_frac - fraction of rows for a given slow phase that contained raw data used for fitting
- _sp_aveVel - average velocity of the eye during a given slow phase calculated from median filtered data
- _sp_qp_mag - horizontal change in position (degrees) of the eye between successive fits -- i.e. the start_position of movement n - end_position of movement n-1
This also contains validation_results.csv, used by the R script to create panels A and B of Figure 4. See below. This file is included here for easier access.
validationData
-- contains csv files and a MATLAB script for created panels A and B of Figure 4. For this analysis, 10 features on the head and face of 2 birds were automatically tracked using DLTdv8 (https://biomech.web.unc.edu/dltdv/).
-- RAW_dltdv8Files -- contains the files output by DLTdv8 when digitizing. The files are named as Direction_Bird_Speed_SpatialFrequency. Only those that end in "xypts" contain any data, as this is a 2d analysis. The data contained are the frame in the video and the x and y position of the point (pixel units).
These xypts files are copied into the outer validationData folder with the xypts ending removed.
-- validationData_bootstrapping.m -- This script gathers the xypts and computes the mean and 95% confidence intervals for the position of the head using these 10 points and a bootstrapping function, after filtering to remove random jitter. The results are in millimeters.
-- all_results.csv -- Contains the output of validationData_bootstrapping.m. This file contains the following columns:
- frame - frame of video
- Direction - direction of stimulus motion (NT or TN)
- Bird - bird identity
- SpatialFrequency - spatial frequency of stimulus (cycles per degree)
- Mean_X - mean horizontal position of the head and face, calculated from 10 manually digitized points (millimeters)
- CI_Lower_X - Lower bound of 95% CI for horizontal head position (millimeters)
- CI_Upper_X - Upper bound of 95% CI for horizontal head position (millimeters)
- CI_Lower_Y - Lower bound of 95% CI for vertical head position (millimeters)
- CI_Upper_Y - Upper bound of 95% CI for vertical head position (millimeters)
-- validation_results.csv -- Contains the same data as all_results.csv plus the following columns:
- periTime - time relative to motion onset within a trial (seconds)
- Left_pupil_xRAW_mm - horizontal position of the left eye (millimeters).
- Left_pupil_yRAW_mm - vertical position of the left eye (millimeters).
- Right_pupil_xRAW_mm - horizontal position of the right eye (millimeters).
- Right_pupil_yRAW_mm - vertical position of the right eye (millimeters).
eyeCalibrationImages -- contains images and extract data for the conversion from mm to degrees of eye-in-head angle. Measurements for each eye, made using ImageJ, are contained with the calibrationValues.txt file along with the math for calculating the approximate distance between the pupil and the eye's center of rotation.
testVideo -- contains an example (30 second clip) from a head restrained animal along with the output from the DeepLabCut model. This is what is required to test pupilTracker.py. The default settings should be adjusted properly for this test video. Running just this 30 second clip takes ~6 minutes on my system. The one difference between the _combo file output and those in rawData.zip is the addition of the timecode column on the far left. This was added manually by viewing the videos to identify the first motion onset, and then setting the dt for that row to the first value of dt in the stimulus file from psychopy.
videos -- contains all videos organized by recording data and spatial frequency. These videos are high quality MP4 files.
stimulusCode_example.py -- An example python script used for generated stimulus. The included example is setup for a spatial frequency of 0.25 cycles per degree. We cannot confirm that this script will work out of the box on all systems.
Movie_1_10.1098_rspb.2024.2015 -- Animation showing eye position changes in Anna's hummingbird (Calypte anna) detected using video oculography. (This is uploaded as 'Supplemental Information')
Methods
Animals
Six adult male Anna’s hummingbirds (Calypte anna) were caught on the University of British Columbia campus (07 Feb 2022 – 04 March 2023). Animals were individually housed in 0.61 x 0.61 x 0.91-m cages and provided with ad libitum artificial nectar (Nektar-Plus, Nekton, Pforzheim, Germany & Nectar 9, Roudybush, Woodland, California, USA). All procedures were approved by the University of British Columbia Animal Care Committee in accordance with the guidelines set out by the Canadian Council on Animal Care. Animals were released back to the wild following the experiments described herein.
Method of restraint
To allow the release of the birds after our experiments, we developed a method for non-invasive restraint of hummingbirds. Head-restraint is required to study OKR in the absence of the vestibulo-ocular reflex. We restrained hummingbirds in a flight posture by wrapping them in an elastic “ACE Brand” bandage (3M, Maplewood, USA) and placing them on a plastic bed tilted 10° above horizontal (see Fig. 1A). To restrain the head, we fashioned a bill guide using the opening of a 30 mL Luer syringe (Henke Sass Wolf, Dudley, USA). The bill guide was oriented to hold the bill horizontally, as is typical in flight (Fig. 1B).
Birds were hand-fed a sucrose solution before each recording session. We then wrapped the bird in the elastic bandage, gently positioned the bill through the guide, and placed the animal on the plastic bed. We were careful to place the body close enough to the bill guide so that the animals were unable to retract their head. Extension of the head was prevented as the opening of the bill guide was smaller than the widest part of the bill. After each recording session, animals were returned to their home cage.
Eye tracking
An overview of the eye-tracking setup is provided in Figure 1. We used a bright pupil method to detect and track both eyes of each hummingbird [38]. Two cameras were positioned using tripods on the left- and right-side of the bird, at the height of the animal and along the optical axis of the eyes (~20 deg nasal from lateral). Bright pupil eye tracking uses an infrared light to increase the contrast between the pupil and the iris (analogous to the “red eye effect”). For illumination, we used 850 nm IR ring lights (Feyond FY76-72V4-F, Shenzhen City, China). These IR lights were attached to 50 mm Nikkor camera lenses (Kabushiki-gaisha Nikon, Tokyo, Japan). Videos were acquired using IR-sensitive cameras (Prosilicia GE680, Allied Vision, Stadtroda, Germany) operating at 120 Hz with recording performed using Streampix version 6 (NorPix Inc., Montreal, Canada). Recording occurred in the dark. The IR lights produce illumination outside of the visual range for hummingbirds [39].
Visual Stimuli
Stimuli were generated using PsychoPy (version 2023.1.3) and presented on two 144Hz LCD monitors (ASUS VG248QG, ASUSTeK Computer Inc., Taipai, Taiwan) set to sRGB color mode, resulting in gratings with a Michelson contrast of 0.99. The monitors were oriented perpendicular to the bird and positioned 30 cm away on either side (Fig. 1A). To allow a clear view of the bird for each camera, the monitors were height-adjusted so that the lower margin of the screen was just below that of the bird. Thus, visual stimuli were presented to the upper and lateral visual fields, occupying ~83 deg horizontally and ~45 deg vertically on either side of the animal.
Previous electrophysiology studies in hummingbirds used random dot fields to determine neuron direction preferences and then sinewave gratings to measure spatiotemporal tuning in the preferred direction [7,31,33]. We selected stimulus conditions to match these studies, but used sinewave gratings throughout. This allowed us to assess OKR at stimulus speeds beyond what we could produce with dots and replicate stimulus speeds across a consistent set of spatial and temporal frequencies (Table 1).
Within a recording session, we randomized the direction (NT or TN) and speed of stimuli presented on each monitor. Hummingbirds must feed frequently, limiting the maximum length of recording sessions to under 40 minutes. We therefore presented stimuli for each spatial frequency on separate days, except for spatial frequencies of 0.015625 cyc deg-1 and 4 cyc deg-1, which were presented on the same day but with a feeding break in between. Stimulus velocity was defined as the product of temporal and spatial frequency (Table 1). During binocular stimulation, stimulus speeds presented to both eyes were always matched (e.g., 1 deg s-1 TN to the right eye and 1 deg s-1 NT to the left eye).
Table 1: See manuscript
Each recording session consisted of a set of trials repeated three times and randomized within each set. Each trial began with a static pattern for 1 s, followed by a stationary sinewave grating for 1 s, followed by 10 s of motion of that same sinewave grating in either the TN or NT direction. As we did not wish to physically cover an eye for monocular trials and risk injury to the birds, two methods were used for monocular stimulation: 1) the monitor for the unstimulated eye was either turned gray after the static pattern, 2) or changed to the sinewave grating but held stationary. We analyzed data from the former method as the latter represents an unequal speed combination, though data from both methods were qualitatively similar. For binocular trials, the stimulus direction for each monitor was set independently to produce either regressive stimulation (TN & TN), progressive stimulation (NT & NT), or rotation (TN & NT) or (NT & TN). We refer to binocular NT & TN combinations as rotational stimulation; however, as our stimulus monitors did not curve around the animal, these conditions technically represent pseudo-rotation.
Analysis of oculomotor behavior
To extract the position of the pupil from each video, we used a combination of thresholding and contour segmentation in python (version 3.11.8). We trained a custom DeepLabCut [40] model (ResNet 152) to track the corneal reflection and eyelids. To track the pupil, we used data from the same DeepLabCut model to determine a region-of-interest around the eye, upscaled this portion of the video frame, and then used thresholding and contour detection in OpenCV (v. 4.9.0) for pupil detection.
Analysis of raw pupil position was performed in MATLAB R2023b (Mathworks, Natick, USA). We first removed positions if the confidence from DeepLabCut was low (< 95%). We next used the vertical distance between upper and lower eyelids to detect blinking, removing pupil position data 100 ms before and 400 ms after each detected blink to eliminate blink-artifacts. We then removed periods of pupil position data when the pupil area was changing rapidly, as asymmetrical dilation or constriction of the pupil could be erroneously interpreted as changes in position. Finally, we filled small gaps in the pupil position stream using linear interpolation to facilitate saccade detection.
To convert the filtered pupil position from pixels to degrees, we set the pupil position for each trial relative to the median position prior to motion onset. Pupil displacement was then converted to millimeters based on objects of known length in the video frame, digitized for each recording session using ImageJ 1.53k (NIH, USA). We removed data for trials in which the corneal reflection moved more than 1 mm, as this indicated that the bird rolled its head during the trial (6.7% of total trials). Finally, we converted displacement to degrees based on the assumption that the avian eye rotates around the midpoint between the pupil and the back of the eye [41]. We determined this value (2.181 ± 0.102 mm; mean ± sd) by measuring the half distance between the outer edge of the scleral ring and the medial margin of the eye for five eyes from four specimens available from previous lab projects. The eye-in-head position was then calculated as θ = tan-1 (rel. position mm / 2.181 mm). Importantly, this assumption does not impact the main conclusions of this work and resulted in a similar range of movement as determined using alternative methods [42].
We next automatically removed saccades using peak detection, analyzing the median-filtered velocity of the pupil for spikes above a speed threshold. This speed threshold was manually adjusted (between 5-15 deg s-1) for each session based on visual inspection of pupil position across trials.
Inspection of the parallel streams of data from each eye indicated that blinks and saccades often occurred simultaneously in both eyes, as in other birds [43,44]. For this reason, we considered a blink or saccade in either eye to constitute a break in OKR and computed the velocities of the eyes between each of these breaks. This allowed us to compare, for the same set of time windows, the motion occurring simultaneously in each eye. To compute OKR velocities, we fit each segment between breaks with a linear regression and extracted the slope (deg s-1), fitted values (deg), and quality of the fit (Root Mean Squared Error; RMSE). The magnitudes (deg) of rapid eye movements (blinks and saccades) were determined using these fitted values as the offset between the last and first elements in each adjacent segment.
Statistical analysis
We performed plotting and hypothesis testing in R version 4.4.0. For comparisons of OKR between conditions, we removed segments with poor fit quality (RMSE > 0.35) or with many (>10%) missing data points for either eye (together 12.7% of total fits). Significance was assessed by computing bootstrapped means and 95% confidence intervals (1000 iterations) using the R package boot [45]. Unless otherwise specified, bootstrapped values were computed by pooling fitted values from both eyes and bootstrapping across trials and individuals.