Theory and Practice: A Case with Spectrum-Based Fault Localization

This study explores the alignment between theory and practice in the context of Spectrum-Based Fault Localization (SBFL). It delves into the analysis of execution traces, assigning suspiciousness scores to program elements, comparing SBFL formulas like Tarantula and Ochiai, and introducing new formulas like ER1 and ER5. The goal is to identify the theoretically best SBFL formula by Xie et al., evaluating its performance against popular SBFL formulas using benchmark programs and analyzing results from a dataset of 10 programs with 199 faulty versions. The study measures the efficacy of techniques like Tarantula, Ochiai, ER1a, ER1b, ER5a, ER5b, and ER5c and presents average percentages of inspected elements along with standard deviations.

• Fault Localization
• Spectrum-Based
• SBFL Formulas
• Theory and Practice
• Program Analysis

hettinger_s Follow

Uploaded on Oct 06, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Theory and Practice, Do They Match ? A Case with Spectrum-Based Fault Localization Tien-Duy B. Le, Ferdian Thung, and David Lo School of Information Systems Singapore Management University 1

2. Spectrum-Based Fault Localization Locating buggy program elements by Analyzing two sets of execution traces Normal traces and faulty traces Assigning suspiciousness scores to program elements Two well-known SBFL formulas Tarantula Ochiai 2

3. Spectrum-Based Fault Localization Xie et al. A Theoretical Analysis of the Risk Evaluation Formulas for Spectrum-based Fault Localization (TOSEM, 2013) Two families of SBFL formulas ER1 and ER5 (5 formulas in total) Theoretically proven to outperform Ochiai and Tarantula Under the assumption: test coverage is 100% 3

4. Our Goal Theoretically Best SBFL Formula by Xie et. al. ? Benchmark Programs vs. Popular SBFL Formula 4

5. Notations Notion ns nf (e ns nf f n Description Number of successful test cases that execute e Number of failing test cases that execute e Number of successful test cases that do not execute e Number of failing test cases that do not execute e ) ( ) ( e n e n n f f f + = ) ( ) ( e n e n n s s s + = (e (e ) ) ) (e ) sn 5

6. Popular SBFL Formulas Tarantula ( ) n e f n f = ( ) Tarantula e ( ) n e ( ) n e f n + s n s f Ochiai ( ) + n e f = ( ) Ochiai e ( ( ) ( )) n n e n e f f s 6

7. Theoretically Best SBFL Formulas 7

8. Dataset 10 programs, 199 faulty versions Siemens test suite Space, NanoXML, XML-Security Evaluation Metric Rank of the first faulty program element = score EXAM Total number of executable program elements The lower the EXAM score, the better the performance 8

9. Results Technique Tarantula Ochiai ER1a ER1b ER5a ER5b ER5c Average % Inspected 23.37% 21.02% 33.34% 21.09% 43.04% 43.04% 54.95% Standard Deviation 23.44% 21.96% 35.22% 19.48% 19.63% 19.63% 26.83% Ochiai has the lowest EXAM score (21.02%) 9

10. Results Technique Tarantula Ochiai ER1a ER1b ER5a ER5b ER5c Average % Inspected 23.37% 21.02% 33.34% 21.09% 43.04% 43.04% 54.95% Standard Deviation 23.44% 21.96% 35.22% 19.48% 19.63% 19.63% 26.83% Tarantula s EXAM score is lower than 4 out of the 5 theoretically best SBFL formulas 10

11. Results Technique Tarantula Ochiai ER1a ER1b ER5a ER5b ER5c Average % Inspected 23.37% 21.02% 33.34% 21.09% 43.04% 43.04% 54.95% Standard Deviation 23.44% 21.96% 35.22% 19.48% 19.63% 19.63% 26.83% Wilcoxon signed rank test (significance level of 0.05) Ochiai is statistically better than ER5a, ER5b, ER5c 11

12. 100% Test Coverage Assumption For 135 out of the 199 faulty versions Test coverage < 100% Average test coverage of the 199 versions 84.97% Theoretically best SBFL formulas cannot outperform popular SBFL formulas 12

13. Conclusion We conduct an empirical study on 10 programs with 199 versions Compare performance of 5 theoretically best SBFL formulas with Tarantula and Ochiai We find that: Ochiai outperforms all theoretically best formulas Tarantula outperforms 4 out of the 5 formulas Assumption of 100% test coverage is not valid in many cases 13

14. Future work In-depth study how test coverage and other factors affect effectiveness of SBFL formulas Theoretically analyze performance of SBFL formulas Assumption: test coverage < 100% 14

15. Thank you! Questions? Comments? Advice? {btdle.2012, ferdiant.2013, davidlo}@smu.edu.sg 15