OOS 10-4 - Unlocking general-purpose bioacoustic classification with open-source approaches

Ecologists are rapidly adopting machine learning classification to detect and identify species in audio data. One reason for this increasing use is the growing availability of general-purpose bioacoustic classifiers that can identify many species or sound types. However, these classifiers typically perform poorly when applied to new acoustic datasets. Here, we investigate obstacles preventing generalizable bioacoustic classification and suggest solutions to these challenges. We use a case study in bird sound classification, an area that has received considerable attention from ecologists and machine learning researchers alike. First, we assess species availability and geographic representation within all publicly available strongly-labeled bird sound datasets and a weakly-labeled online bioacoustic repository, Xeno-Canto. We find limited open annotated data for the vast majority of bird species and note geographic biases in the available data. Strongly labeling the publicly available Xeno-Canto dataset and releasing these data as an open-source resource could address these concerns. Second, we assess the performance of a general-purpose bird classification model, BirdNET, on two bioacoustic datasets, including a newly-collected one. We find that the general-purpose classifier performs poorly on the unpublished dataset and that its outputs violate assumptions of common techniques used to analyze classifier outputs, such as score thresholding. We recommend approaches to account for uncertainty when using these outputs. Ultimately, we suggest that to address these concerns, machine learning researchers in bioacoustics could release open-source, general-purpose classifiers that can be tuned to local data by non-expert users.