Introducing the Never-ending UI Learner: Making Apps More Accessible and User-Friendly with AI
Machine learning models have revolutionized app development by predicting and improving user interface (UI) elements. By training these models to understand semantic information about UIs, we can create apps that are not only accessible, but also easier to test and automate. However, the traditional method of collecting and labeling datasets for these models, often performed by humans, can be costly and prone to errors.
The Challenge of UI Element Prediction
For instance, determining if a UI element is “tappable” can be based on visual cues from screenshots or unreliable metadata from the view hierarchy. However, the only foolproof way to know for certain is to programmatically tap the element and observe the result. To address this challenge, we developed the Never-ending UI Learner.
Introducing the Never-ending UI Learner
The Never-ending UI Learner is an innovative app crawler that automatically installs real apps from mobile app stores and explores them to find new and difficult training examples. It has crawled for more than 5,000 device-hours, conducting over half a million actions on 6,000 apps to train three computer vision models:
- Tappability Prediction Model: This model predicts if a UI element is “tappable.”
- Draggability Prediction Model: This model predicts if a UI element is draggable.
- Screen Similarity Model: This model measures the similarity between different app screens.
Through this process, the Never-ending UI Learner enhances the accuracy and reliability of training AI models, reducing the need for human intervention and the associated costs.
By leveraging its extensive experience and knowledge gained from crawling real-world apps, the Never-ending UI Learner significantly improves the prediction capabilities of machine learning models. This means apps can provide more seamless and intuitive user experiences, ultimately making them more user-friendly and accessible to a wider audience.
With the Never-ending UI Learner, the future of AI-driven app development is within reach, allowing us to create apps that adapt and improve effortlessly as technology advances.