Automated Unit Test Generation Using Mocking for Environment Dependencies

/ 32 Automated Unit Test Generation for Classes with Environment Dependencies Andrea Arcuri, Gordon Fraser, Juan Pablo Galeotti ASE 2014 Presented by Yongbae Park

1 / 32 Overview Motivation: low coverage & non-deterministic test results in automated test generation for a class interacting with environment 1. Branches depending on the environment that may not be covered by a sequence of method calls of a class under test 2. Passing test cases generated by a tool can fail as the environment changes The environment is inputs from outside of a class that a class cannot control such as network, file system, system time, and user inputs Approach: use a mock library interacting with the virtual environment to increase coverage and produce deterministic test result Users of a test generation tool do not have to build their own mock library The mock library provides deterministic inputs from the virtual environment separated from the real environment Evaluation: the mock library increases branch coverage and decreases the number of test cases making non-deterministic results EvoSuite with the mock library produces test cases whose branch coverage is increased by 183% and the number of tests with non-deterministic test results is reduced by 99.2%

2 / 32 Contents Background How EvoSuite generates test suites using a genetic algorithm Motivation The limitation of EvoSuite in a class with environment dependency Approach Mocking Java standard API to control the environment Empirical study Related Works Conclusion

3 / 32 Background EvoSuite Given class under test (CUT), EvoSuite automatically generates a test suite using a genetic algorithm 1. Create 2 initial test suites by adding method calls randomly and insert the test suites into current generation 2. Select two test suites from current generation 3. Create 2 new test suites by crossover (exchange test cases of the suites) 4. Modify two test suites (test drivers) from step 3 with mutation operators (remove, insert, change operators) 5. Insert the new two test suites from step 4 to next generation if coverage of the new two test suites are higher than that of parents 6. Repeat 2~5 until next generation has sufficient number of test suites 7. Repeat 2~6 until time limit is reached or all branches are covered 8. Select a test suit with the highest branch coverage and insert assertions by observing differences in execution traces between the original CUT and a mutated CUT

4 / 32 Example for EvoSuite Message class contains 2 members: created meaning message creation date and text meaning contents of a message The message creation date is set in the constructor of Message public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null? :text); } public String toString() { return created+ , +text; } public String getText() { return text; } } 1 2 3 4 5 6 7 8 9 10 11 12 13

5 / 32 Initial Test Suite Generation EvoSuite randomly inserts a new statement in empty test cases of two initial test suites until the length of test cases reaches the maximum length EvoSuite add a method call whose callee is a method of a class under test or a method call of an object that is available at the end A parameter of the created method call is selected from available values, null, or a random value The maximum length for each test case is set by random number public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null? :text); } public String toString() { return created+ , +text; } public String getText() { return text; } } 1 2 3 4 5 6 7 8 9 public class TestSuite1 { public void test0() { //length 3 Message v0 = new Message( e ); Message v1 = new Message( c ); String v2 = v1.toString(); } public void test1() { //length 2 Message v0 = new Message(null); String v1 = v0.getText(); 10 11 12 13 } }

6 / 32 Crossover EvoSuite selects test suites ? and ? from current population, and moves last ? test cases of ? to ? and first ? test cases of ? to ? where ? is a random number public class TestSuite1 { public void test0() { Message v0 = new Message( e ); Message v1 = new Message( c ); String v2 = v1.toString(); } public void test1() { Message v0 = new Message(null); String v1 = v0.toString(); } } public class TestSuite2 { public void test0() { Message v0 = new Message( a ); String v1 = v0.toString(); String v2 = v1.trim(); } public void test1() { Message v0 = new Message(null); String v1 = v0.getText(); String v2 = v0.toString(); } } public class TestSuite3 { public void test0() { Message v0 = new Message( e ); Message v1 = new Message( c ); String v2 = v1.toString(); } public void test1() { Message v0 = new Message( a ); String v1 = v0.toString(); String v2 = v1.trim(); } } public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); } public void test1() { Message v0 = new Message(null); String v1 = v0.getText(); String v2 = v0.toString(); } }

7 / 32 Mutation Insert Operator Insert operator adds a new statement at a random position by using one of the two ways 1. Add a method call statement whose callee is a method of a class under test 2. Add a method call of an object that is available at the random position A parameter of the created method call is selected from available values, null, or a random value public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); } } of a class under test Adding a method call Adding a method call of an available object public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); Message v2 = new Message( b ); } } public class TestSuite4 { public void test0() { Message v0 = new Message( a ); String v1 = v0.toString(); boolean v2 = v1.equals(null); } }

8 / 32 Mutation Remove & Change Operator Remove operator randomly selects a statement in a test case and remove the statement Change operator randomly changes a callee or a parameter of a method invocation statement in a test case Change operator selects a new callee method that whose return type is same as the original method Change operator changes a parameter of a method call into a value which is available from the previous statements or a random value public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); } } callee method change parameter change (random value) public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.getText(); } } public class TestSuite4 { public void test0() { Message v0 = new Message( a ); String v1 = v0.toString(); } }

9 / 32 Inserting Assertions EvoSuite adds assertions by observing differences between an execution trace of the original class under test and that of a mutated version of a class under test An assertion is added if values of a variable (primitive or String type) are different in the two execution traces EvoSuite mutates the original class under test using the insert, remove, change operators in test generation

10 / 32 Inserting Assertions - Example public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); } } public class Message { //original private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null? :text); } public String toString() { return created+ , +text; } } public class Message { //mutated private Date created; private String text; public Message(String text) { this.created=new Date(); 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 } public String toString() { return created+ , +text; } } 10 10 Execution result of Message Execution result of mutated Message v1 = 2014.10.02, v1 = 2014.10.02,null assertEquals(v1, 2014.10.02, )

11 / 32 Generated Test Case EvoSuite creates a test case with an assertion containing a string of the created date of the test case test0() uses a string 2014.10.2, However, the test case will fail when the test case is executed on a different day public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null? :text); } public String toString() { return created+ , +text; } } 1 2 3 4 5 6 7 8 9 public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); assertEquals(v1, 2014.10.02, ); } } 10

12 / 32 Motivation The test result of a class interacting with the environment is non-deterministic (not reproducible) The author defines the environment as inputs from outside of a class E.g. a state of virtual machine (free & total memory space), a state of operating system (system time, file system, user inputs) A test case is unstable if the test case passed when test case is generated but may fail in the next execution with different environment Operating System Virtual Machine Time & Date Class under test File system User inputs Memory state

13 / 32 Environment Mocking Mocking is the replacement real classes with modified classes that behave deterministically In the following example, Date is replaced by MockDate that always returns a fixed value public class Message { private Date created; private String text; public Message(String text) { this.created=new MockDate(); this.text=(text==null? :contents); } public String toString() { return created+ , +text; } } public class MockDate() { public String toString() { return 2014.10.02 ; } } 1 2 3 4 5 6 7 8 9 public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); assertEquals(v1, 2014.10.02, ); } } 10 11 12 13 14

14 / 32 Overview of Environment Mocking A test generation tool with a generic mock library create a stable test suite with higher branch coverage without user s (tester) efforts A mock library of the paper mocks console inputs, file I/O, general API class of Java standard API The mock library consist of a custom InputStream class for console inputs, 11 classes of file I/O (e.g. File), 12 general API classes (e.g. System, Runtime) The mock library have helper methods that set the environment in a test case The technique replaces standard library with the mock library using Java bytecode instrumentation Insert operator of EvoSuite inserts the helper methods in test cases to generate test suites that controls the initial environment

15 / 32 Console Inputs A customized InputStream called SystemInUtil has a helper method addInputLine(String) so that a test case can write contents of console inputs The console contents of SystemInUtil is reset before every test execution In instrumentation, System.io in a class under test is changed into SystemInUtil.io

16 / 32 File I/O The authors mocked 11 JVM file I/O API classes Mock classes are children classes of the mocked classes The authors overridden the methods of the mocked classes to access virtual file system instead of real file system of operating system For example, the authors overridden 37 methods among 52 methods of File class Test cases become independent from each other and there is no negative side-effect such as file system corruption The authors created helper methods such as appendLineToFile() to control the initial state of the virtual file system java.io.File java.io.PrintStream java.io.FileInputStream java.io.PrintWriter java.io.FileOutputStream java.util.logging.FileHandler java.io.RandomAccessFile javax.swing.JFileChooser java.io.FileReader java.io.FileWriter javax.swing.filechooser.FileSystemView

17 / 32 General JVM Calls The authors mocked 12 JVM general API classes to control the environment such as time and random number Class name Environment java.lang.Exception java.lang.Throwable Stack trace message java.util.logging.LogRecord java.lang.Thread Memory usage & the number of processors java.lang.Runtime java.lang.System java.util.Date Current system time & date java.util.Calendar java.util.GregorianCalendar Reflection (the order of Method objects) java.lang.Class java.lang.Math Random number java.util.Random

18 / 32 Stack Trace (1/2) StackTest has a method getStackTrace() which returns a stack trace string of Throwable object The message contains the name of a thread and the name of class that executes the test case The test case fails if the name of a thread or test runner is changed public class StackTest { public static String getStackTrace(Throwable t) { PureStringWriter sw = new PureStringWriter(); PrintWriter pw = new PrintWriter(); t.printStackTrace(pw); return sw.toString(); } } public class TestSuite { public void test0() { String result = StackTest.getStackTrace(new Throwable()); assertEqual("Exception in thread \ Thread-0\" java.lang.Throwable\n \tat TestRunner.run(TestRunner.java:120)\n \tat TestSuite.test0(Test.java:10)" , result); } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

19 / 32 Stack Trace (2/2) Stack trace message depends on the string representation of Thread object and JUnit runner Stack trace message contains the string representation with thread id depending on the number of created thread JUnit runner is a class that executes each test case and bottom of stack trace message contains the class name of JUnit runner class A tester can use custom JUnit runner to modify test execution process Mocked Thread, Exception, Throwable, and LogRecord create stack trace message deterministically The mock class of Thread print the string representation deterministically The mock class of Thread, Exception, Throwable, and LogRecord does not prints custom JUnit runner class in stack trace

20 / 32 Memory & Processor (1/2) Performance executes a certain task and measure execution time and memory usage getMemoryUsage() returns a string containing the amount of memory used by the task The test case fails if free memory space is changed by execution of a test runner or other test cases public class Performance { public static String getMemoryUsage() { performTask(); return "Memory: "+(Runtime.totalMemory()-Runtime.freeMemory()); } } public class TestSuite { public void test0() { String result = Performance.getMemoryUsage(); assertEqual( Memory: 341 , result); } } 1 2 3 4 5 6 7 8 9 10

21 / 32 Memory & Processor (2/2) Runtime provides system specific values such as the number of processors and memory usage availableProcessors() returns the number of available processors freeMemory(), totalMemory(), and maxMemory() return memory usage availableProcessors(), freeMemory(), totalMemory(), and maxMemory() of a mock class of Runtime return constant values

22 / 32 System Time (1/2) Message object contains the creation date of a message The generated test case contains a string of the day when the test case was created The test case fails if the test case is executed on 7next day public class Message { private Date created; private String msg; public Message(String msg) { created = new Date(); this.msg = msg; } public Date getCreated() { return created; } } public class TestSuite { public void test0() { Message msg = new Message(); assertEqual( 2014.10.02 , msg.getCreated().toString()); } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

23 / 32 System Time (2/2) Mock methods and classes of currentTimeMillis(), nanoTime(), Date, Calendar, and GeregorianCalendar returns default time The mock library has helper methods to set initial system time Each call of the system time methods increases the time by 1

24 / 32 Reflection (1/2) getAvailableMethods() returns a string of a list of public methods If test0() is executed before test1(), the execution result of test0() is pass Otherwise, the execution result of test0() is fail public class LoadLibrary { public static String getAvailableMethods(Class c) { String result = ; for (Method each : c.getMethods() ) { result += each.getName() + \n ; } return result; } } public class TestSuite { public void test0() { String methods = getAvailableMethods(TestSuite.class); assertEqual( test0\ntest1 , methods); public void test1() { } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

25 / 32 Reflection (2/2) The order of an array returned by reflection methods is non-deterministic The order of an array that is returned by reflection methods (getClasses(), getMethods()) depends on method invocation order or class loader implementation The reflection methods of a mock class of Class return sorted lists of available methods, classes, annotations, etc.

26 / 32 Random Number random() returns position data in 2D space randomly The content of the returned object can be different from the previous execution Mock classes of Math and Random return the next random number deterministically public class Position { public int x, y; public static Position random() { Position result = new Position(); result.x = (int)Math.random(); result.y = (int)Math.random(); return result; } } public class TestSuite { public void test0() { Position p = Position.random(); assertEqual(2, p.x); } } 1 2 3 4 5 6 7 8 9 10 11 12 13

27 / 32 Static Field Initialization A class under test that modifies static field may change behavior of test cases if the execution order of the test cases is changed EvoSuite creates unstable test because EvoSuite does not reset static field nor restart Java virtual machine The extended EvoSuite calls static initialization methods of classes whose static fields are modified to reset static field with low overhead Every writing to a static field is instrumented with a method call that records the modification A test suite generated by EvoSuite calls the static initialization methods before the execution of each test case

28 / 32 Empirical Study RQ1: does controlling the environment successfully increase coverage on known cases of environmental interaction? RQ2: does controlling the environment resolve known issues of unstable tests? RQ3: How do results generalize to the SF100 corpus? The authors applied extended EvoSuite to SF100 corpus SF100 corpus is 100 Java projects randomly selected from SourceForge SF100 corpus contains 11,219 Java classes EvoSuite created test cases up to 3 minutes for each class

29 / 32 RQ1: Coverage Setup The authors manually selected 30 classes from SF100 corpus that interact with the environment The classes that uses environment-related API are selected The authors compared coverage of 6 different configurations Base: the default EvoSuite Console: EvoSuite with console input mocking VFS: EvoSuite with virtual file system JVM: EvoSuite with general API call mocking Static: EvoSuite with static field initialization All: EvoSuite with console input, virtual file system, general API call mocking and static field initialization

30 / 32 RQ1: Coverage Experiment Results The technique increases branch coverage by 183%(=(0.82-0.29)/0.29) in average The blue numbers are statistically significant differences at ? = 0.05 Extended Extended No. Base No. Base Console VFS JVM Static All Console VFS JVM Static All 1 0.10 0.10 0.76 0.10 0.10 0.78 16 0.22 0.22 0.74 0.22 0.22 0.73 2 0.75 0.75 0.75 0.75 1.00 1.00 17 0.00 0.00 0.87 0.00 0.00 0.87 3 0.09 0.08 0.53 0.08 0.41 0.80 18 0.00 0.00 1.00 0.00 0.00 1.00 4 0.26 0.74 0.27 0.25 0.26 0.73 19 0.30 0.33 0.77 0.30 0.69 0.79 5 0.40 0.40 0.89 0.40 0.40 0.99 20 0.00 0.00 1.00 0.00 0.00 1.00 6 0.60 0.60 0.60 0.61 0.99 0.99 21 0.79 0.79 0.77 0.67 0.88 0.85 7 0.20 0.20 0.77 0.20 0.20 0.77 22 0.67 0.68 0.69 0.61 0.99 0.96 8 0.29 0.29 0.40 0.28 0.29 0.40 23 0.08 0.08 0.99 0.08 0.08 0.99 9 0.94 0.94 0.94 0.95 1.00 1.00 24 0.25 0.25 0.87 0.25 0.25 0.99 10 0.83 0.83 0.83 0.95 0.83 0.94 25 0.07 0.07 0.56 0.07 0.07 0.56 11 0.12 0.12 0.71 0.12 0.12 0.70 26 0.22 0.66 0.22 0.22 0.22 0.63 12 0.08 0.68 0.09 0.09 0.11 0.64 27 0.28 0.62 0.30 0.27 0.25 0.64 13 0.24 0.24 0.24 0.24 0.79 0.86 28 0.03 0.03 0.56 0.03 0.03 0.54 14 0.12 0.12 0.12 0.12 0.12 0.93 29 0.34 0.34 0.79 0.34 0.34 0.80 15 0.33 0.33 0.80 0.33 0.33 0.80 30 0.20 0.20 0.79 0.20 0.20 0.79 Avg. 0.29 0.35 0.65 0.29 0.37 0.82

31 / 32 RQ2: Unstable Tests Setup The authors manually selected 30 classes from SF100 corpus from which the default EvoSuite creates unstable tests There are 2 classes that are used in both RQ1 and RQ2 The authors compared coverage of 6 different configurations same as the experiments in RQ1

32 / 32 RQ2: Unstable Tests Experiment Results The technique reduces the number of unstable tests per class by 99.2%(=(2.43-0.02)/2.43) The blue numbers are statistically significant differences at ? = 0.05 Extended Extended No. Base No. Base Console VFS JVM Static All Console VFS JVM Static All 1 2.72 2.52 2.52 2.49 0.00 0.00 16 1.75 1.73 1.47 1.57 0.00 0.00 2 1.14 1.16 1.20 2.06 0.00 0.00 17 1.19 1.23 1.03 0.00 1.17 0.00 3 2.99 3.04 3.17 3.21 0.14 0.13 18 2.59 2.87 2.55 0.09 2.27 0.11 4 1.27 1.28 1.35 1.74 0.00 0.00 19 2.49 2.39 2.93 0.02 1.62 0.00 5 2.37 2.36 2.33 2.19 0.00 0.07 20 1.91 1.90 1.87 0.00 1.77 0.00 6 4.20 4.19 4.24 4.15 1.55 0.01 21 3.96 3.87 2.40 3.81 0.00 0.00 7 1.33 1.40 1.39 0.00 1.40 0.00 22 0.93 0.93 0.78 0.00 0.80 0.03 8 0.64 0.75 0.79 0.82 0.06 0.08 23 1.82 1.59 1.28 0.00 1.55 0.00 9 0.84 0.98 0.75 0.00 0.77 0.00 24 3.26 3.67 3.66 0.00 3.38 0.00 10 7.16 7.09 6.93 0.03 6.94 0.00 25 1.83 1.76 1.79 0.12 1.60 0.16 11 0.61 0.62 0.61 0.00 0.62 0.00 26 2.13 2.12 2.21 0.00 2.19 0.00 12 2.50 2.52 2.52 2.38 0.00 0.00 27 1.91 1.86 2.18 1.79 0.01 0.00 13 5.53 5.57 5.47 5.87 0.00 0.00 28 1.63 2.01 1.87 0.86 1.22 0.00 14 3.73 3.66 3.41 0.02 1.73 0.00 29 1.22 1.22 1.22 0.00 1.19 0.00 15 6.86 6.79 7.50 4.41 7.05 0.30 30 0.57 0.54 0.57 0.67 0.00 0.00 Avg. 2.43 2.45 2.40 1.27 1.30 0.02

33 / 32 RQ3: Generalization The authors compared the default EvoSuite (base) and the extended EvoSuite (all) by applying both of them to all 11,219 classes of SF100 corpus The branch coverage increased by 1.8% (=(77.9%-76.5)/76.5) The default EvoSuite achieved 76.5% of branch coverage and the extended EvoSuite achieved 77.9% The extended EvoSuite achieved 2,803 additional branches The number of classes from which EvoSuite creates unstable tests is reduced by 53.8% (=(1,452-671)/1,452) The default EvoSuite created unstable tests from 1,452 classes while the extended EvoSuite created unstable tests from 671 classes

34 / 32 Related Works Mocking frameworks help developers write mock classes of user classes, not class of the standard library The mocking frameworks such as Moles (C#), Jmock (Java) provide mechanisms to create mock class and replace the original classes with the mock classes The developers should change mock classes as the original class changes The paper aims mocking Java standard API which does not change frequently Once mock library is built, the mock library can be applied to any class under test

35 / 32 Conclusion Mocking library interacting with the virtual environment increases coverage and reduces unstable tests in automated unit test generation The modified EvoSuite creates test cases that controls initial environment to increase branch coverage The generated test cases are stable because interactions with the virtual environment is deterministic The authors shown the empirical results that using mocking library increase branch coverage and reduces unstable tests from EvoSuite for 100 Java projects