Behavior Metrics for Prioritizing Investigations of Exceptions

This research paper introduces behavior metrics aimed at estimating user inconvenience caused by exceptions, which traditional metrics fail to account for. The study delves into analyzing patterns of exception occurrences and the impact on user experience, proposing a more refined approach to prioritizing investigations based on user-centric factors.

• Behavior Metrics
• Exception Prioritization
• User Experience
• Exception Patterns
• User Inconvenience

dsky Follow

Uploaded on Sep 25, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Behavior Metrics For Prioritizing Investigations of Exceptions Zack Coker Zack Coker, Kostadin Damevski, Claire Le Goues, Nicholas A. Kraft, David Shepherd, and Lori Pollock 1

2. RobotStudio 2

3. Sample log file with an exception OnlineControllerRemove AddVirtualController Exception|Services.RobApi.RobApiException ----EXCEPTION BEGIN---- at System.Runtime.CompilerServices.CompilerServices.TaskAwaiter .ThrowForNonSuccess(Task task) ----EXCEPTION END---- AddedController VirturalFlexPendant RapidEditorShow 3

4. ABB developers previously prioritized exceptions using measures of exception frequency 1. Number of exception instances 2. Number of users affected 4

5. The old metrics do not take into account the user inconvenience caused by an exception Input string not in the correct format exception may affect many users but many not have a significant effect on user experience 5

6. The old metrics do not take into account the user inconvenience caused by an exception Input string not in the correct format exception may affect many users but many not have a significant effect on user experience We created behavior metrics, which are designed to We created behavior metrics, which are designed to estimate user inconvenience caused by exceptions. estimate user inconvenience caused by exceptions. 6

7. Behavior patterns in the study Single Restart Multiple Restarts Repeat Action Repeat Exception Similar Exceptions 7

8. Behavior patterns in the study Single Restart 1. ResetConveyor 2. Exception|TargetInvocationException 3. ----BEGIN EXCEPTION---- 4. System.invokeMethod(Method m) 5. ----END EXCEPTION---- 6. ----LOG END---- Single Restart Multiple Restarts Repeat Action Repeat Exception Similar Exceptions 8

9. Behavior patterns in the study 1. ResetConveyor 2. Exception|TargetInvocationException 3. ----BEGIN EXCEPTION---- 4. System.invokeMethod(Method m) 5. ----END EXCEPTION---- 6. ----LOG END---- 7. ----LOG BEGIN---- 8. ----LOG END---- Single Restart Multiple Restarts Repeat Action Repeat Exception Similar Exceptions Multiple Restarts 9

10. Behavior patterns in the study 1. ResetConveyor 2. Exception|TargetInvocationException 3. ----BEGIN EXCEPTION---- 4. System.invokeMethod(Method m) 5. ----END EXCEPTION---- 6. ResetConveyor Single Restart Multiple Restarts Repeat Actions Repeat Exception Similar Exceptions Repeat Action 10

11. Behavior patterns in the study 1. ResetConveyor 2. Exception|TargetInvocationException 3. ----BEGIN EXCEPTION---- 4. System.invokeMethod(Method m) 5. ----END EXCEPTION---- 6. ControllerRequestWriteAccess 7. Exception|TargetInvocationException 8. ----BEGIN EXCEPTION---- 9. System.invokeMethod(Method m) 10.----END EXCEPTION---- Single Restart Multiple Restarts Repeat Actions Repeat Exception Similar Exceptions Repeat Exception 11

12. Behavior patterns in the study 1. ResetConveyor 2. Exception|TargetInvocationException 3. ----BEGIN EXCEPTION---- 4. System.invokeMethod(Method m) 5. ----END EXCEPTION---- 6. ControllerRequestWriteAccess 7. Exception|TargetInvocationException 8. ----BEGIN EXCEPTION---- 9. System.createInstance(Instance i) 10.----END EXCEPTION---- Single Restart Multiple Restarts Repeat Actions Repeat Exception Similar Exceptions Similar Exceptions 12

13. We verified the behavior metrics through a survey of ABB RobotStudio developers The survey was designed to determine if the behavior metrics were useful when prioritizing exceptions Contacted 12 developers of RobotStudio at ABB and asked them to take a survey Survey consisted of 12 exceptions from log files covering a 6 month period 13

14. 14

15. Participants changed their priority 30.6% (44/144) of the time after accounting for the behavior metrics 15

16. No priority changes were due to new metrics reinforcing old or information in stack traces For 67 out of 100 no priority changes were because participants felt that the new information reinforced their original assessment 22 of the 100 were because of information in the exception type and stack traces, which the metrics did not take into account NullReference exceptions are always programming errors and should be fixed immediately 16

17. Questions 17

18. Behavior Metrics Number of pattern instances for an exception Behavior metric percentage ?????? ?? ??????? ????????? ??? ?? ????????? ?????? ?? ????????? ????????? Rank of behavior metric percentage 18