Enhancing UI Display Issue Detection with Visual Understanding

The research presents a method utilizing visual understanding to detect UI display issues in mobile devices. By recruiting testers and employing visual techniques, the severity of issues like component occlusion, text overlap, and missing images was confirmed. CNN-based models aid in issue detection, while Grad-CAM is employed for localization. Data augmentation techniques enhance the process, ensuring effective identification and rectification of UI display issues.

• UI display issues
• Visual understanding
• CNN-based models
• Grad-CAM
• Data augmentation

bruc_162 Follow

Uploaded on Oct 08, 2024 | 2 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Owl Eyes: Spotting UI Display Issues via Visual Understanding 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE) 1

2. Introduction Many display issues occur during the UI display rendered on different mobile devices Most of those issues are caused by different system settings in different devices 2

3. Goal Their goal is to detect those UI display issues Solution 1 : Recruit many testers for app GUI testing It requires significant human effort Some errors are difficult to spot manually Solution 2 : Detecting the UI display issues with the screenshots by visual understanding 3

4. Pilot study They carry out a pilot study to examine the prevalence of these issues 1. Data Collection -> get from crowd testing platforms 2. Manually check the display issue 3. Categorizing UI Display Issues Total of 4,470 screenshots have UI display issues (4470/10330) 4

5. Categorizing UI Display Issues 1. Component occlusion (47%) 2. Text overlap (21%) 3. Missing image (25%) 4. Null value (6%) 5. Blurred screen (1%) These findings confirm the severity of UI display issues 5

6. System Overview CNN-based model -> issues detection Grad CAM-based model -> localization 6

7. CNN-based UI Display Issues Detection CNN has proven to be effective in image classification and recognition 7

8. Grad CAM-based UI Localization Gradient weighted Class Activation Mapping (Grad-CAM) is a technique for visualizing the regions of input that are important for predictions on CNN-based models 8

9. Heuristic-based Data Augmentation They generating UI screenshots with display issues from bug-free images The data augmentation is based on the Rico 9

10. Experiment design RQ1: (Issues Detection Performance) How effective of our proposed OwlEye in detecting UI display issues RQ2: (Issues Localization Performance) How effective of our proposed OwlEye in localizing UI display issues RQ3: (Usefulness Evaluation) How does our proposed OwlEye work in real- world situations 10

11. Experimental Setup Dataset Metrics 11

12. Issues Detection Performance 85% (679/798) of the screenshots which are predicted as having UI display issues are truly buggy 84% (679/800) buggy screenshots can be found with OwlEye 12

13. Performance Comparison with Baselines 13

14. Performance Comparison among Model Configurations 14

15. Issues Localization Performance (RQ2) They conduct a user study to evaluate the localization performance Answer the question whether they agree with each of the localization results using 5-Likert scale 15

16. Usefulness Evaluation They randomly sample 1,500 Android applications from F-droid7 and 700 Android applications from Google Play They use DroidBot for exploring the mobile apps and take the screenshot of each UI pages 16

17. Conclusion This paper focuses on automatic detecting the UI display issues from the screenshots Their system has following benefits : -> High accuracy -> Generality across platforms -> Generality across languages 17