In recent years, there has been a lot of progress in deep learning. However, models often fail when deployed in new domains or distributions. Source-free domain adaptation (SFDA) is a type of research that focuses on adapting pre-trained models to new domains using only unlabeled data. This is important because training large models is computationally expensive and environmentally unfriendly.
Transfer learning is a popular area of research that focuses on adapting models to new tasks. SFDA is a practical area of transfer learning because it deals with real-world applications where labeled examples are not available. However, most SFDA research has focused on simple distribution shifts in image classification tasks.
We wanted to explore SFDA in the field of bioacoustics, where distribution shifts are common and present unique challenges. Bioacoustics deals with the study of sounds produced by living organisms, particularly animals. In this field, we often encounter distribution shifts characterized by insufficient labeled data and different environmental conditions.
We conducted a study using a bird species classifier that was pre-trained on a labeled dataset of bird songs. We then tested the classifier on different “soundscapes” from various geographical locations. Soundscapes are recordings obtained through omnidirectional microphones and often contain multiple birds vocalizing simultaneously.
Our findings were surprising. Existing SFDA methods performed poorly on the bioacoustics shifts. They were unable to consistently outperform the source model and often underperformed it. Even recent methods like Tent, which aim to reduce uncertainty in model predictions, failed to work effectively in our bioacoustics task.
However, we did find one promising approach called Noisy Student. This unsupervised approach encourages the model to reconstruct its own predictions on a target dataset using random noise. We modified this approach by using model dropout as the noise source and called it Dropout Student. While Dropout Student faced some challenges, it showed potential for improving SFDA in bioacoustics.
In conclusion, evaluating SFDA methods on commonly-used datasets and distribution shifts may not provide an accurate view of their performance and generalizability. We need to consider a wider range of distribution shifts, particularly those that occur naturally in real-world applications. This research has implications not only for the academic community but also for practitioners in fields like bioacoustics, where biodiversity preservation is a significant challenge.