README: Odor source distance is predictable from time-histories of odor statistics for large scale outdoor plumes

This repository consist of the data analysis done for Odor Tracking experiment.

Abstract

Odor plumes in turbulent environments are intermittent and sparse. Lab-scaled experiments suggest that information about the source distance may be encoded in odor signal statistics, yet it is unclear whether useful and continuous distance estimates can be made under real-world flow conditions. Here we analyze odor signals from outdoor experiments with a sensor moving across large spatial scales in desert and forest environments to show that odor signal statistics can yield useful estimates of distance. We show that achieving accurate estimates of distance requires integrating statistics from 5-10 seconds, with a high temporal encoding of the olfactory signal of at least 20 Hz. By combining distance estimates from a linear model with wind-relative motion dynamics, we achieved source distance estimates in a 60x60 meter square search area with median errors of 3-8 meters, a distance at which point odor sources are often within visual range for animals such as mosquitoes.

Preprint: "Odor source location can be predicted from a time-history of odor statistics for a large-scale outdoor plume"

Data Files Walkthrough

Data Description:

This contains all datasets involved in the analysis of the preprint as available here: "Odor source location can be predicted from a time-history of odor statistics for a large-scale outdoor plume" and the analysis repository can be found in here in github. All the file format are in pandas .h5 or .hdf format, are compatible with 1.0.0 < pandas <= 1.5.3

Download the data from Data dryad. The data folder along with figure folder, can be placed in the home folder under ~/odor_anaylsis/ .

The datasets can be divided into:

1. *Interpolated Sensor and stationery wind sensor data data *: These datasets include data from mobile sensor stack that contains imu, gps data and odor sensor and data from stationery wind sensors that has ambient wind velocity, direction. The interpolation is done with respect to the odor sensor's sampling speed.

2. Derived dataframes: For different analysis example whiff statistics calculation, or lookback time analysis dataframes have been derived for easy to use plug and play experience.

Folder Structure

├── data                       # Contains mobile sensor data from desert and forest
├── derived_data
   ├── KF                      # datasets for kalman filter analysis 
   ├── stationery_wind_data    # contains ambient wind sensor data from desert and forest
   ├── wind_lag_analysis       # datasets for wind lag analysis 
├── Figure                     # contains svg files for reproducing the figure as seen in the paper

Below are the file descriptions under respective folders:

1. data
   • ForestMASigned.h5 : Contains interpolated sensor data and signed distance axis for general Whittell Forest data analysis.
   • NotWindyLR.h5 : Low resolution interpolated sensor data for wind speed < 3m/s for desert.
   • NotWindyMASigned.h5: Contains high resolution interpolated clean sensor data for wind speed < 3m/s for desert.
   • WindyMASigned.h5: Contains high resolution interpolated sensor data for wind speed > 3m/s for desert.
2. derived_data
   • NotWindyStatsTime_std.h5: Contains derived lookback time whiff statistics for wind speed < 3m/s for desert.
   • WindyStatsTime_std.h5: Contains derived lookback time whiff statistics for wind speed > 3m/s for desert.
   • ForestStatsTime_std.h5 : Contains derived lookback whiff statistics for forest.
   • 1hz.h5, 10hz.h5, 60hz.h5 : low pass filtered odor signal passed through 2nd order butterworth-filter and can be used with the script 8. Figure 8.
   • LpfForestFiltered.h5, LpfHWSFiltered.h5, LpfLWSFiltered.h5 : datasets containing effect on R-squared value when odor signal is low pass filtered. Use with Figure 8.
   • HWSLTall.h5, LWSLTall.h5, ForestLTall.h5, R2LtTime.h5 : datasets used in the script Figure 5 to analyse the effect of lookback window on r-squared when correlated with distance.
   • aic_filtered_model_params.h5 : Contains coefficients from the statsmodel of distance correlated aic filtered whiff statistics for all three wind scenarios. Run the script Figure S9.
   • lt_whiff_statistics.h5, All_AICDeltaTab.h5, all_Rsquared.h5, AllRsquaredAicCombinations.h5,: Contains all the 25 whiff statistics calculated across the different distance from source over a look back time of 10 seconds. Run the script Figure 5, in section Bootstrapped R2 and Boootstrapped R2 and AIC for filtered parameters to see the results.

Figure:

Figure folder is available for download. The following are the files that can be found in the Figure folder, which can be seen in the paper:

• method1.svg : (Figure1) Contains interpolated sensor data and signed distance axis for general Whittell Forest data analysis.
• method2.svg : (Figure2) Low resolution interpolated sensor data for wind speed < 3m/s for desert.
• fig3_revised.svg : (Figure3) Contains high resolution interpolated clean sensor data for wind speed < 3m/s for desert.
• fig4.svg : (Figure4) Contains high resolution interpolated sensor data for wind speed > 3m/s for desert.
• r2AicStatPlot.svg : (Figure5) Contains high resolution interpolated sensor data for wind speed > 3m/s for desert.
• clustering.svg : (Figure6) Contains high resolution interpolated sensor data for wind speed > 3m/s for desert.
• fig7.svg : (Figure7) Contains high resolution interpolated sensor data for wind speed > 3m/s for desert.
• lpf.svg : (Figure8) Contains high resolution interpolated sensor data for wind speed > 3m/s for desert.
• verticalMovement.svg : Figure S1: Contains high resolution interpolated sensor data for wind speed > 3m/s for desert.
• motionanalysis.svg : Figure S2: Motion analysis showing agent's movements similar to casting motions.
• normalityAnalysis.svg : Figure S3: Comprehensive residual analysis for model validation
• windLagAnalysis.svg :Figure S4: Wind characteristics from our LWS, HWS and Forest scenarios span a representative range of near surface wind characteristics from a larger dataset.
• time_spent.svg : Figure S5: Time spent - vs number of encounters
• individualDatasetWhiffStat.svg : Figure S6: Time spent - vs number of encounters
• mc_wsd.svg : Figure S7: Time spent - vs number of encounters
• windAicParams.svg : Figure S9: AIC filtered coefficiecnts across various wind scenarios
• klmsupplmental.svg : Figure S10: Kalman smoothed estimates of the distance to the odor source zoomed

Data Fields

Interpolated Sensor and stationery ambient sensor data

This section presents real-time and interpolated measurements from both mobile sensors on an agent and fixed ambient sensors, providing a comprehensive view of the environmental and agent-specific dynamics.

Derived Analysis Data

This section details analytics derived from the raw and interpolated data, aimed at understanding patterns, frequencies, and statistical measures related to odor encounters and agent navigation relative to the source.

Usage

This dataset is intended for use in developing and testing algorithms related to odor source localization in outdoor wind conditions in desert and forest terrains. Researchers are encouraged to utilize this data for exploring innovative approaches for odor source localization under varying environmental conditions.

Dependencies

To visualize the below figures and see the results and calculations, you will need to install the following:

1. FlyPlotLib
2. FigureFirst
3. Inkscape 1.2

Follow the setup of FigureFirst into inkscape.

Setup Environment

Folder Structure

├── data_exploration             # Contains all the analysis scripts
   ├── bag2h5.py                 # script for changing rosbag file to pandas dataframe
   ├── dataInterpolation.ipynb   # interpolation to match odor sensor sampling rate
   ├── odor_statitics_lib.py
   ├── figure                    # all script for generating figure
      └── supplemental           # contains scripts to generate supplemental figures
   └── demo                      # some analysis demo scripts
├── dataFileDescription.md       # readme for datafiles submitted in Data Dryad
└── README.md

Install Virtualenv

1. Create the virtualenv:

   virtualenv -p /usr/bin/python3.8 <env-name>  
   

2. Install Packages:

   pip install pandas
   pip install h5py
   pip install numpy
   pip install matplotlib
   pip install figurefirst
   pip install seaborn
   pip instal scikit-learn
   pip install h5py
   pip install tables
   python -m pip install statsmodels
   

Software/Scripts

The odor_analysis.zip folder contains the software which can also be found in the github with detailed description on how to use them.