Dataset DOI: 10.5061/dryad.sbcc2frf7

The Tassie BRUV dataset is an image dataset that we generated as a benchmark dataset using Baited Remote Underwater Videos (BRUVs) from chain moorings and environmentally friendly moorings in Tasmania. This data set is used in the paper "The Motion Picture: Leveraging Movement to Enhance AI Object Detection in Ecology", published in Ecology and Evolution. These videos were collected by CSIRO and OzFish volunteers as part of a separate study to observe if there are differences in the community abundance of fish species between chain and environmentally friendly moorings in this area. This dataset can be used to benchmark computer vision techniques, and, in conjunction with the surrounding video footage, techniques that include movement information. For code to fit such models, refer to https://github.com/BenMaslen/MCD. With camouflage, occlusion, schooling, poor visibility, and bio-mimicry, this underwater video dataset poses unique challenges for computer vision.

Description of how the data was generated

28 Baited Remote Underwater Videos were used to generate this benchmark data set. These BRUVs were placed next to chain moorings (16), control locations (6) (locations within a site with no moorings), and environmentally friendly moorings (6), being located at either North West Bay (7), Battery Point (4) or Sandy Bay (17) in Tasmania. The start and ends of these videos were cropped such that the camera being lowered and raised from the sea floor was removed and a time stamp in the top left hand corner of the video was blacked out to ensure it doesn't confound the effect of measurements of movement. Then, 50 frames were randomly selected from each of these videos and bounding boxes with fish id's to species level were labelled for each of them. Fish were detected using the videos as well as the frame image to help observe any camouflaged fish that were present. Then a further 100 frames were randomly selected from each of these videos and they were inspected if they contained any fish whose counts in the previous labelling process were less then 100 (referred to as 'rare fish'). Of the 2,800 frames, 512 had rare fish in them which were additionally labelled to boost the labels of rarer species in the dataset. This brought the total number of frames annotated to 1912, which were split into training (1340), test (380) and validation (192) sets (in the training_data_species_grouped folder). Unfortunately, there were still many species with low counts, so multiple species were grouped into higher order taxonomic levels with similar morphological attributes. In total 5,222 fish annotations were generated, with 3,834 of these annotations being Platycephalus bassensis (Southern Sand Flathead), a benthic fish that lies on the sandy ocean floor, camouflaging itself by burying in the sandy sediment to ambush prey. There was also a clear baiting effect present in the videos with majority of fish being annotated close to the bait in front of the camera.

Files, variables, and data structure

This dataset has the following folder structure (files not included):

Tassie_BRUV.zip
├── augmented_images
│   ├── abs_u_15_d_1
│   ├── back_sub_td_400_li_500_r
│   ├── diff_BS_bn_20_bs_120_u_15_d_1_r_pca
│   ├── dir_u_15_d_1_r
│   └── flow_1_r_pca
├── location_data
├── meta_data
├── training_data_species_grouped
├── training_data_species_high_counts
└── surrounding_videos

Within each augmented images folder (abs_u_15_d_1, back_sub_td_400_li_500_r, diff_BS_bn_20_bs_120_u_15_d_1_r_pca, dir_u_15_d_1_r, flow_1_r_pca) and training data folder (training_data_species_grouped, training_data_species_high_counts), we have the following sub file structure (in accordance with training of YOLO models):

labelled_image_folder
├── images
│   ├── test
│   ├── train
│   └── valid
├── labels
│   ├── test
│   ├── train
│   └── valid
├── .cache
├── data.yaml
├── explained_var
└── pca.pkl

Within each images/ andlabels/  sub folder, contains images and .txt labels (in YOLO format) used to test, train or validate an object detection algorithm. The .cache file is a caching file and can be ignored, and the data.yaml file is fed into a YOLO object detection algorithm and details the aforementioned training data structure and class label names (for the augmented image folders, this is using the training_data_species_grouped labels). This file will need to be edited appropriately to describe the parent directory structure before using in an object detection algorithm. Finally the pca.pkl file contains the principal components used for image augmentation, and the explained_var file details the proportion of explained variance using each principal component.

augmented_images

Within this folder we have some example augmented images that were used in the paper "The Motion Picture: Leveraging Movement to Enhance AI Object Detection in Ecology" published in Ecology and Evolution. Please refer to this paper for more details on the augmentation techniques. Each folder contains augmented images in YOLO txt format to be used for training a YOLO object detection algorithm. These images were augmented using code from https://github.com/BenMaslen/MCD, with the originals being found in the folder training_data_species_grouped.

Below details the augmentations used for each example folder:

abs_u_15_d_1 - Frame differencing using an absolute value method, scaled by 15, with a displacement distance of 1 frame and replacing all the image data.

back_sub_td_400_li_500_r - Background subtraction using the K-nearest Neighbour algorithm, replacing only the red image layers.

diff_BS_bn_20_bs_120_u_15_d_1_r_pca - Background subtraction using frame differencing based approach, with 20 background images, each 120 frames apart. The values were scaled by 15, with the three colour layers transformed into two colour layers using PCA (the explained variance of the PCA can also be found in the parent directory), and the third layer being the average of the resulting background subtraction across the three layers.

dir_u_15_d_1_r - Frame differencing with a direction based approach, replacing just the red layer.

flow_1_r_pca - Optical flow using using PCA to transform the three colour layers into two colour layers, and the third layer being the magnitude of the measured optical flow.

location_data

Example csv file demonstrating the format of the location_data.csv file that needs to be generated in order to use the augmentation code in https://github.com/BenMaslen/MCD.

This csv file has the following column headings:

image_name - Name of the image in the labelled image folder

split - Which training data split the image belongs to (either test, train or valid).

frame - Frame number the image was sourced from.

vid_location - Location of the video file that the labelled image was sourced from.

meta_data

Dataset metadata. This folder has two files; species_annotation_counts.csv and TB_species_all_metadata.csv.

The species_annotation_counts.csv file has the following column headers:

Species - The species class of each labelled fish species.

Family - The family class of each labelled fish species.

Order - The order class of each labelled fish species.

Annotation grouping - The annotation grouping for each labelled fish species from the training_data_species_grouped labelled images.

Annotation counts - The number of annotation counts per species class.

The TB_species_all_metadata.csv file has the following column headers:

Species - The species class of each labelled fish species.

Family - The family class of each labelled fish species.

Order - The order class of each labelled fish species.

Annotation counts - The number of annotation counts per species class.

Mobility - Mobility category (how quickly each species moves in and out of the view of the video).

Schooling - How much the species likes to form schools of fish.

Baiting_impact - How much the baiting effect impacts each species.

training_data_species_grouped

Labelled training data in YOLO txt format. Species have been grouped into higher order taxanomic or morphological groupings to avoid species groups with low counts. This is the training data used in the paper "The Motion Picture: Leveraging Movement to Enhance AI Object Detection in Ecology". You can find more details on how the species were grouped in this paper and by referencing the meta_data/species_annotation_counts.csv file.

training_data_species_high_counts

Labelled training data in YOLO txt format. Only species with high counts (>75 annotations) are included in this dataset, with others not labelled.

surrounding_videos

Baited Remote Underwater Videos in which the labelled training data was sourced from. These videos are required in conjunction with the location_data files in order to estimate movement to augment the labelled frames using the code from https://github.com/BenMaslen/MCD.

Accompanying code: MCD

All code used to generate the example augmented images in this dataset as well as produce the results in the paper "The Motion Picture: Leveraging Movement to Enhance AI Object Detection in Ecology" published in Ecology and Evolution can be found the MCD github repository:

https://github.com/BenMaslen/MCD

Access information

Data was derived from the following sources:

• These videos were collected by CSIRO and OzFish volunteers as part of a separate study to observe if there are differences in the community abundance of fish species between chain and environmentally friendly moorings in this area.