Enhancing UI Display Issue Detection with Visual Understanding
The research presents a method utilizing visual understanding to detect UI display issues in mobile devices. By recruiting testers and employing visual techniques, the severity of issues like component occlusion, text overlap, and missing images was confirmed. CNN-based models aid in issue detection, while Grad-CAM is employed for localization. Data augmentation techniques enhance the process, ensuring effective identification and rectification of UI display issues.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Owl Eyes: Spotting UI Display Issues via Visual Understanding 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE) 1
Introduction Many display issues occur during the UI display rendered on different mobile devices Most of those issues are caused by different system settings in different devices 2
Goal Their goal is to detect those UI display issues Solution 1 : Recruit many testers for app GUI testing It requires significant human effort Some errors are difficult to spot manually Solution 2 : Detecting the UI display issues with the screenshots by visual understanding 3
Pilot study They carry out a pilot study to examine the prevalence of these issues 1. Data Collection -> get from crowd testing platforms 2. Manually check the display issue 3. Categorizing UI Display Issues Total of 4,470 screenshots have UI display issues (4470/10330) 4
Categorizing UI Display Issues 1. Component occlusion (47%) 2. Text overlap (21%) 3. Missing image (25%) 4. Null value (6%) 5. Blurred screen (1%) These findings confirm the severity of UI display issues 5
System Overview CNN-based model -> issues detection Grad CAM-based model -> localization 6
CNN-based UI Display Issues Detection CNN has proven to be effective in image classification and recognition 7
Grad CAM-based UI Localization Gradient weighted Class Activation Mapping (Grad-CAM) is a technique for visualizing the regions of input that are important for predictions on CNN-based models 8
Heuristic-based Data Augmentation They generating UI screenshots with display issues from bug-free images The data augmentation is based on the Rico 9
Experiment design RQ1: (Issues Detection Performance) How effective of our proposed OwlEye in detecting UI display issues RQ2: (Issues Localization Performance) How effective of our proposed OwlEye in localizing UI display issues RQ3: (Usefulness Evaluation) How does our proposed OwlEye work in real- world situations 10
Experimental Setup Dataset Metrics 11
Issues Detection Performance 85% (679/798) of the screenshots which are predicted as having UI display issues are truly buggy 84% (679/800) buggy screenshots can be found with OwlEye 12
Performance Comparison with Baselines 13
Performance Comparison among Model Configurations 14
Issues Localization Performance (RQ2) They conduct a user study to evaluate the localization performance Answer the question whether they agree with each of the localization results using 5-Likert scale 15
Usefulness Evaluation They randomly sample 1,500 Android applications from F-droid7 and 700 Android applications from Google Play They use DroidBot for exploring the mobile apps and take the screenshot of each UI pages 16
Conclusion This paper focuses on automatic detecting the UI display issues from the screenshots Their system has following benefits : -> High accuracy -> Generality across platforms -> Generality across languages 17
