Each of the 8 animals is summarized in a h5 file. All h5 files follow an identical structure.

Overview

The h5 files contain various measurements and analyses related to neural and behavioral data, including body angles, velocities, neural traces, and timestamps.

Data Structure

Behavioral Measurements

• velocity: Axial velocity measurements in mm/s
• pumping: Pharyngeal pumping rate measurements in Hz
• θh: Head angle measurements in radian
• θh_pos_is_ventral: Boolean flag indicating if a positive value in the head angle measurement corresponds to ventral
• ang_vel: Angular velocity measurements in rad/s
• body_angle: Angles between the first 30 regularly interspaced keypoints along the midline of the worm
• curve: Worm curvature, as defined by the standard deviation of body_angles at each time point
• s_velocity, s_θh, s_pumping, velocity_s, θh_s, pumping_s: normalized behavioral measurements that are irrelevant to this project

Neural Data

• n_neuron: Number of neurons tracked in this animal
• trace_original: Raw neural activity traces. Each row is a neuron, and each column is a confocal imaging time point
• trace_array: z-scored neural activity traces
• trace_array_F20: Neural activity traces normalized by the 20th percentile of raw neural activity of the same neuron
• trace_array_Fmean: Neural activity traces normalized by mean neural activity of the same neuron

Neuropal ID

• neuropal_id: Vector of genetic and anatomical identity of each neuron, ordered by their row number in trace_array
• neuropal_registration/roi_match: Vector of matches between immobilized ROI number and row number in trace_array. neuropal_id is generated from this.
• neuropal_registration/roi_match_confidence: Vector of confidence levels of neural identity annotation based on neuropal transgene expression. Low confidence annotations are excluded from the generation of neuropal_id.

Timing Information

• n_t: Total number of timepoints
• timestanp_confocal: timestamp of each sampled time point in neural traces (sampling frequency ~1.7 Hz)
• timestamp_nir: timestamp of each sampled time point in behavioral imaging (first sampled at 20 Hz, then matched to confocal time points)
• idx_splits: irrelevant to recordings shorter than or equal to 8 minutes, which are all the recordings in this project

Example dataset

Structure of dataset illustrated in Figure 4:

7_2023-05-05-22-data.h5:
├── ang_vel (Vector{Float64}) (800,)
├── body_angle (Matrix{Float64}) (30, 800)
├── body_angle_absolute (Matrix{Float64}) (55, 800)
├── body_angle_all (Matrix{Float64}) (55, 800)
├── curve (Vector{Float64}) (800,)
├── idx_splits (Vector{NamedTuple{(:start, :stop), Tuple{Int64, Int64}}}) (1,)
├── n_neuron (Int64) ()
├── n_t (Int64) ()
├── neuropal_id (Vector{String}) (159,)
├── neuropal_registration/
│   ├── roi_match (Vector{Int64}) (178,)
│   ├── roi_match_confidence (Vector{Float64}) (178,)
├── pumping (Vector{Float64}) (800,)
├── pumping_s (Vector{Float64}) (800,)
├── s_pumping (Float64) ()
├── s_velocity (Float64) ()
├── s_θh (Float64) ()
├── timestamp_confocal (Vector{Float64}) (800,)
├── timestamp_nir (Vector{Float64}) (9449,)
├── trace_array (Matrix{Float64}) (159, 800)
├── trace_array_F20 (Matrix{Float64}) (159, 800)
├── trace_array_Fmean (Matrix{Float64}) (159, 800)
├── trace_original (Matrix{Float64}) (159, 800)
├── velocity (Vector{Float64}) (800,)
├── velocity_s (Vector{Float64}) (800,)
├── θh (Vector{Float64}) (800,)
├── θh_pos_is_ventral (Bool) ()
├── θh_s (Vector{Float64}) (800,)

This means that a total of 159 neurons were tracked in this animal. The recording had 800 frames, which lasted around 8 minutes given our sampling frequency.
For specific datasets with shorter recordings:

if data_uid == "2023-03-08-03" 
    toi = 1:507
elseif data_uid == "2023-04-25-04"
    toi = 601:800
elseif data_uid == "2023-03-16-01"
    toi = 1:205
else
    toi = 1:800
end

Code snippet for extracting key measurements (in Julia)

Load the content of a h5 file

using HDF5
filepath = "./7_2023-05-05-22-data.h5" # please change to absolute path or reset your working directory first

dict = h5open(filepath, "r") do file
    dict = Dict()
    for key in keys(file)
        dict[key] = read(file[key])
    end
    return dict
end

Extract behavioral measurements:

velocity = dict["velocity"]
head_angle = dict["θh"]
pumping = dict["pumping"]

Extract neural trace by identity:

target = "AVAL"
if target in dict["neuropal_id"]
    neuron_idx = findall(x -> x == target, dict["neuropal_id"])[1]
    neuron_trace = dict["trace_array"][neuron_idx,:]
else
    print("$(target) is not identified with confidence in this animal.")
end

Code snippet for extracting key measurements (in Python)

Load the content of a h5 file to a dictionary:

import h5py
import numpy as np

filepath = "./7_2023-05-05-22-data.h5"  # please change to absolute path or reset your working directory first

dict_data = {}
with h5py.File(filepath, 'r') as file:
    for key in list(file.keys()):
        if isinstance(file[key], h5py.Dataset):
            dict_data[key] = file[key][()]

Extract behavioral measurements:

velocity = dict_data["velocity"]
head_angle = dict_data["θh"]
pumping = dict_data["pumping"]

Extract neural trace by identity:

target = "AVAL"

# Convert byte strings to regular strings if necessary
neuropal_ids = dict_data["neuropal_id"]
if isinstance(neuropal_ids[0], bytes):
    neuropal_ids = [id.decode('utf-8') for id in neuropal_ids]

# Find the neuron and extract its trace
if target in neuropal_ids:
    neuron_idx = neuropal_ids.index(target)
    neuron_trace = dict_data["trace_array"][neuron_idx, :]
else:
    print(f"{target} is not identified with confidence in this animal.")
    neuron_trace = None