1. Small scanning radars have been used for many years to track the movements of insects, birds, and bats. While the ability to track multiple flying animals simultaneously has numerous applications in basic ecology and applied conservation, translating radar tracks into accurate animal densities and fluxes requires estimates of detection and tracking probabilities. These can be challenging to determine, especially in environments with variable background clutter. 2. In order to assess radar tracking probabilities, we added echoes from simulated bird tracks to sequences of scans collected with an X-band marine radar at a colony of common and roseate terns (Sterna hirundo and S. dougallii) on Great Gull Island, New York, USA in the summers of 2014 and 2015. Automated detection, classification, and tracking algorithms were used to extract the trajectories of terns from the radar data. The proportion of simulated tracks recovered by these procedures could then be used to estimate the tracking probabilities for real birds. Stationary telescope transects provided visual ground-truth. 3. The radar could track individual birds up to 3 km away, performing best between 0.5-1.2 km, where 38% of simulated birds were correctly detected and tracked in each scan. Overall, 94% of all simulated birds were tracked over at least part of their trajectories. Tracking performance was limited by weak bird echoes, backscatter from the sea surface, and the inherent challenges of multi-target tracking. 4. This simulation-based method provides a low-cost, flexible approach for estimating radar tracking probabilities in complex, cluttered environments. Knowledge of these probabilities in turn allows the animal densities and fluxes to be corrected for imperfect detection. Despite their limitations, small scanning radars can track hundreds of birds simultaneously over 10s of km², giving a view of animal movement unavailable with other techniques.