Deep neural networks (DNNs) are widely used in various tasks, from image classification to meta-learning. The common approach for learning DNNs is empirical risk minimization (ERM), but it has some drawbacks. It treats all samples equally, ignoring the rare and difficult ones, which affects performance on unseen data.
To solve this problem, researchers have developed data re-weighting techniques, but they often require additional models and make training more complex. In a recent paper, we introduce a lightweight algorithm called Stochastic Re-weighted Gradient Descent (RGD) that re-weights data points based on their difficulty during optimization. It can be applied to any learning task with just two lines of code.
RGD improves the performance of various learning algorithms, including supervised learning and meta-learning. We show significant improvements over state-of-the-art methods in different tasks like DomainBed and tabular classification. It also boosts performance for BERT and ViT models.
Another important concept we utilize is Distributionally Robust Optimization (DRO). It optimizes the model’s loss for “worst-case” data distribution shifts, making the model more robust to changes in the data distribution. DRO focuses on useful features rather than noisy ones, leading to better generalization and performance on unseen data.
RGD is simple and doesn’t require an additional meta model. It can be easily implemented with just two lines of code and used with popular optimizers like SGD, Adam, and Adagrad.
We conducted empirical experiments to compare RGD with state-of-the-art techniques. RGD outperforms them in supervised learning tasks like language, vision, and tabular classification. It also performs better in domain generalization and class imbalance scenarios. However, it might not work well with highly corrupted training data.
In conclusion, RGD is a powerful algorithm that improves the performance of learning models. It is simple to implement and can be used in various tasks with impressive results.