Data from: Humans outperform Merlin SoundID in field-based point-count surveys
Data files
Aug 28, 2025 version files 89.69 KB
-
merlin_analysis_dataset_doublesremoved.csv
67.61 KB
-
merlin_dataset_doublevisits_dryad.xlsx
11.93 KB
-
README.md
10.15 KB
Abstract
Point counts are integral to bird population monitoring, and multiple methods have been developed to improve their reliability. However, these improved protocols are time-consuming, expensive, and often require hiring surveyors with inconsistent abilities. Merlin Sound ID, an automated bird-sound classifier that can run in the field on mobile devices, could be a part of the solution to the problems associated with point counts by functioning as a no-cost observer with consistent performance. Here, we compare the accuracy of Merlin and human observers in 144 paired field-based point-counts. Our goals were to evaluate how the species identified, number of individual detections, and precision for humans and Merlin compared. We also examined the consistency of Merlin across different devices. Human observers reported 382 individual detections over the course of the study, 72% more than the 222 detections reported by Merlin. Precision was similar with 92% and 86% for humans and Merlin, respectively. Merlin detected four species missed by humans, but also detected 12 false positive species. Merlin disagreed with itself on 57% of point counts when running on two different devices at the same time. We propose Merlin not as a replacement for human point counters, but as an aid to increase detection probability of point-count surveys as secondary observers, with humans acting as reviewers and arbitrating disagreements between devices when needed.
This study required no ethical permits.
Dataset DOI: 10.5061/dryad.3ffbg79xb
Description of the data and file structure
Files and variables
File: merlin_dataset_doublevisits_dryad.xlsx
Description: This file contains the data from point counts where two observers ran Merlin at the same time on different devices.
Variables
- date: The date the survey took place. Ten point counts happened on each survey day except one which was cut short due to inclement weather.
- point: Point count location along route on Curtis Road, starting with point one closest to road entrance and point ten deepest into the forest parcel.
- what.species.1: What species did the Merlin user assign "user 1" for this survey day capture at this point/day combination. These species list include uncorrected Merlin output, so species labeled incorrect by the in-lab human reviewer are included.
- what.species.2: What species did the Merlin user assigned "user 2" for this survey day capture at this point/day combination. These species list include uncorrected Merlin output, so species labeled incorrect by the in-lab human reviewer are included.
- same = 1, diff = 0: If the species list was the same, a value of 1 is inout into the cell. If the species list differs in any way, a zero is input into the cell.
- species.misaligned: The number of species by which the uncorrected lists differed.
- what.species.corrected.1: What species did the Merlin user assigned "user 1" for this survey day capture at this point/day combination. These species list do not include uncorrected Merlin output, so species labeled incorrect by the in-lab human reviewer are removed.
- what.species.corrected.2: What species did the Merlin user assigned "user 2" for this survey day capture at this point/day combination. These species list do not include uncorrected Merlin output, so species labeled incorrect by the in-lab human reviewer are removed.
- same = 1, diff = 0 corrected: If the species list was the same, a value of 1 is inout into the cell. If the species list differs in any way, a zero is input into the cell.
- species.misaligned.corrected: The number of species by which the corrected lists differed.
- user.1.device: The device type used by Merlin user 1.
- user.2.device: The device type used by Merlin user 2.
File: merlin_analysis_dataset_doublesremoved.csv
Description: This file contains data from each paired human-Merlin point count and the review of the in-lab expert. If there were two users running Merlin at the same time, the results from one device was randomly chosen to use throughout the survey for comparison to the human observer and the other device was removed from this data sheet.
Variables
-
date: The date they survey took place. Ten point counts happened on each survey day except one which was cut short due to inclement weather.
-
observer.initials: This is the observer ID of the in-field human point-counter, the Merlin operator, or the in-lab audio reviewer.
-
obs.method: Merlin, hearing (in-field human), or playback (in-lab audio reviewer).
-
device: If Merlin was obs.method, what type of device was the app run on.
-
point.number: Point count location along route on Curtis Road, starting with point one closest to road entrance and point ten deepest into the forest parcel.
-
time: Time in EST the three-minute point count began.
-
sky: Sky, or cloud cover, was recorded on a scale of 0-6 (0 = clear skies, 1 = partly cloudy, 2 = overcast, 3 = fog that deters detection, 4 = light rain, 5 = rain, 6 = snow) with readings of ≥ 3 necessitating a pause or postponement of the survey.
-
wind: Wind was recorded on the Beaufort Scale, which ranks wind speed on a scale of 0-5 (0 = < 1 mph, 1 = 1-3 mph, 2 = 4-7 mph, 3 = 8-12 mph, 4 = 13-18 mph, 5 = 19-24 mph), with readings of ≥ 3 necessitating a pause or postponement of the survey.
-
noise: Background noise was measured by ear on a scale of 0-4 (0 = no background noise, 1 = faint background noise, 2 = moderate background noise, 3 = loud background noise, 4 = intense background noise) with readings of ≥ 3 necessitating a pause or postponement of the survey.
-
temp: Temperature measured in Fahrenheit as recorded from the phone of the in-field human observer. Temperature for points 1-5 was taken at at point one, and temperature for points 6-10 was taken at point 6.
-
species.code: The four-letter banding code derived from its common name. If species is none, UNK, or is a family name and not a species, that entry was discarded for analysis. None means that no species were observed for that point day combination, and a family name/UNK means the human observer had a guess on the identity of a bird they were hearing but could not ID to species. If species is none/UNK/a family name and not a species, that entry was discarded for analysis. Only entries with a species identification were used for the comparison of humans and Merlin.
Table S1. The banding code, common name, and scientific name of every bird detected over the course of the study.
Banding code Common name Scientific name AGOL American Goldfinch Spinus tristis AMCR American Crow Corvus brachyrhynchos AMRO American Robin Turdus migratorius BAOW Barred Owl Strix varia BAWW Black-and-white Warbler Mniotilta varia BCCH Black-capped Chickadee Poecile atricapillus BHVI Blue-headed Vireo Vireo solitarius BLJA Blue Jay Cyanocitta cristata BRCR Brown Creeper Certhia americana BTNW Black-throated Green Warbler Setophaga virens BWHA Broad-winged Hawk Buteo platypterus CANG Canada Goose Branta canadensis CEDW Cedar Waxwing Bombycilla cedrorum CHSP Chipping Sparrow Spizella passerina COHA Cooper's Hawk Astur cooperii CORA Common Raven Corvus corax COYE Common Yellowthroat Geothlypis trichas DEJU Dark-eyed Junco Junco hyemalis DOWO Downy Woodpecker Dryobates pubescens EAKI Eastern Kingbird Tyrannus tyrannus EAPH Eastern Phoebe Sayornis phoebe GCKI Golden-crowned Kinglet Regulus satrapa GRCA Gray Catbird Dumetella carolinensis HAWO Hairy Woodpecker Dryobates villosus HERG American Herring Gull Larus smithsonianus HETH Hermit Thrush Catharus guttatus MAWA Magnolia Warbler Setophaga magnolia MODO Mourning Dove Zenaida macroura NAWA Nashville Warbler Leiothlypis ruficapilla NOFL Northern Flicker Colaptes auratus NOPA Northern Parula Setophaga americana OSPR Osprey Pandion haliaetus OVEN Ovenbird Seiurus aurocapilla PAWA Palm Warbler Setophaga palmarum PISI Pine Siskin Spinus pinus PIWA Pine Warbler Setophaga pinus PIWO Pileated Woodpecker Dryocopus pileatus RBNU Red-breasted Nuthatch Sitta canadensis REVI Red-eyed Vireo Vireo olivaceus SOSP Song Sparrow Melospiza melodia SWTH Swainson's Thrush Catharus ustulatus WTSP White-throated Sparrow Zonotrichia albicollis YRWA Yellow-rumped Warbler Setophaga coronata -
count.number: If observation type was hearing (in-field human) how many individuals of that species did they detect. Not recorded for Merlin or playback as a count of individuals cannot be determined from audio, resulting in NA values.
-
type.a.v: If observation type was hearing (in-field human) did they detect the species visually or aurally. If detected both ways, aurally was recorded so it was noted that Merlin, in theory, could have also detected the bird.
-
distance.50.u.o: If observation type was hearing (in-field human) did they infer the species was under or over 50 meters from their position when it vocalized. If value is N/A or blank, distance was not recorded, likely by accident. Only the human, and not Merlin or playback, could estimate distance.
-
notes: Any notes added by the in-field observers or data reviewers.
-
correct.id: Was this identification correct as determined by the in-lab human reviewer. More information on how observations were deemed correct or incorrect for humans and Merlin is available in the methods section of the manuscript. NA for reviewer identifications as the reviewer was determining if Merlin and humans were correct.
This study took place at the Schoodic Forest in Winter Harbor, Maine, USA (Figure 1). This 687.97 hectare forest parcel is owned and managed by Maine Coast Heritage Trust and co-managed with Schoodic Institute. The Schoodic Forest property is a coastal spruce-fir forest with abundant Acer rubrum (Red Maple) that experienced partial harvest last in the late 1990s. The land is used by the local community for hunting, hiking, and other recreation. The survey was conducted along Curtis Road, a small gravel road that is now predominantly maintained as an ATV trail and access road and is the only road through the property.
The survey route consisted of 10 points along the unpaved Curtis Road, starting ~500 m off from the paved road (Figure 2). Due to turns in the road, not all points were separated by 500 m, but no two points were closer than 300 m. Surveys were conducted beginning at point one and continuing in increasing order along Curtis Road to point 10 over the course of one day, except one instance, where we began the survey at point 10 and continued in decreasing order to point one. We conducted these surveys from July to October in 2023, and visited all 10 points during each weekly survey, unless conditions deteriorated, to where detection probability would be impaired as indicated in the National Parks Service Breeding Landbird Monitoring Volunteer Resource Manual (Faccio et al. 2024). This happened on one survey day, when we only conducted counts at points 1-4 before ending the survey. Whenever possible, surveys were scheduled in accordance with ideal weather conditions. Each point count consisted of a one-minute settlement period, during which no data were collected, when observers stood silently to allow their presence to minimize disturbance on bird behavior. At the end of this one minute, we began a three-minute point count, where a human observer auditorily and visually identified birds alongside another observer operating Merlin on a handheld device (see Merlin Sound ID Software and Recorder Hardware for device make and models). Observers stood within 1-2 m of each other. The Merlin operator aimed the microphone of their handheld device away from themselves and remained still for the three minutes to minimize sound interference. The human observer did not consult with the Merlin operator during the count regarding birds identified by the software. Occasionally there were two Merlin operators on separate devices during the same count. There were two human observers over the course of the study, both deemed intermediately skilled as they were considered level two birders according to the standards outlined in the Birder Certification Online program (Howe & Giese 2021). All point counts were conducted between 1100 and 1800 UTC.
Six device types were used over the course of the study: Apple iPhone SE, Samsung Galaxy Tab A, Apple iPhone 12 Pro, Apple iPhone 14, Apple iPhone 13 Mini, and Apple iPhone 13. Three versions of the Merlin software (ver. 2.1.8, 3.0, and 3.0.1) were used during the four-month study period on Apple iPhones, and one version (ver. 2.1.4) on the Samsung Galaxy. To our knowledge, no changes to the classifier used in the Sound ID feature were made during the duration of the study. Over the course of the study (144 point counts), we generated 214 three-minute audio files, accounting for point counts where we had two devices running Merlin at the same time. Points 1-4 had 22 audio files each, while points 5-10 had 21 each, resulting in 642 minutes of audio in total.