COS 311-3 - Making the most of scarce ecological ground truth with state of the art data augmentation methods

Abstract: By pairing computer vision systems with imagery gathered from unmanned aerial vehicles it is now possible to conduct large-scale ecological monitoring at high spatial and temporal resolution. Yet there is a substantial start-up cost associated with this approach, where practitioners must gather sufficient training data for computer vision systems to perform well. This often means that trained specialists must travel to remote field locations to, e.g., verify the presence of a target plant at many sites. Data augmentation, or altering training images by flipping, rotating, warping, and distorting color is one strategy to maximize the information gained from labor-intensive ground truthing efforts. This practice adds variability to the training data to improve predictions in new contexts, where plants may be imaged from different perspectives or under different lighting conditions. More recently, generative models, popularly employed for AI art, are the focus of data augmentation research. These models take as input text prompts, images, or combinations of both to generate novel content, including images. Generative models can also seamlessly transfer subjects from one context to another. For example, a plant imaged in a grassland can be transferred to a woodland. We explored the utility of Stable Diffusion, a text to image and image to image generative model, for augmenting a dataset of drone images in which we ground truthed the presence of the Euphorbia esula, or leafy spurge, an invasive species prevalent in North America. We systematically manipulated the quantity of real images for training a leafy spurge classifier, augmenting them with synthetic data generated by Stable Diffusion. We found that introducing synthetic data offered the most benefit with fewer real examples, and we achieved 0.82 accuracy with as few as four real examples per class, and 0.87 accuracy with eight real examples per class. These results indicate that state of the art augmentation techniques can be applied to train effective invasive species detection systems with little ground truth data. Enhancing data augmentation outcomes with Stable Diffusion and other generative models is an intense focus of machine learning research at present, with daily advances. We anticipate that gains from this study will greatly benefit the applications of species detection and ecological monitoring generally.