Evolution of Test Automation from Slow & Weak to Fast, Flaky, Blind to Smart & Effective

undefined
Test Automation:
From slow & weak …
to fast, flaky, & blind …
to smart & effective
Jeff Offutt
Jeff Offutt
George Mason University
George Mason University
cs.gmu.edu/~offutt
“Science can amuse and fascinate us, but
it is engineering that changes the world.”
-- Isaac Asimov
 
Test automation in the 1990s
 
Goals
 for automating test execution
Faster execution
Repeatable results
Fewer errors
Mostly 
system testing
Values generated by hand
Results checked by hand (oracle)
Capture-replay
 tools for GUIs
Early unit-level 
test frameworks
Assertions to check results
JUnit integrated the best ideas and simplified
 
2
Unit testing and automation
Unit tests are easier to automate
Unit tests are easier to automate
Thus … more automation
Thus … more automation
3
Faster to produce
Faster to produce
Cheaper
Cheaper
More predictable
More predictable
Less repetitive
Less repetitive
 work
 work
More modularity
More modularity
Less re-design
Less re-design
Re-verification
Re-verification
supports
supports
 evolution
 evolution
Better software
Better software
reduces support costs
reduces support costs
Then this—RoI for unit testing
4
6
0
Requirements
Prog / unit testing
Design
Integration testing
Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002
 
Assume $1000 unit cost per fault, 100 faults
System testing
Post-deployment
Fault origin (%)
Fault detection (%)
Unit cost (X)
Abstraction test design
Model-based testing
software
artifact
model /
structure
test
requirements
input
values
test
cases
test
scripts
test
results
pass /
fail
Concrete level
Concrete level
(implementation)
(implementation)
Abstract level
Abstract level
(design)
(design)
5
Raising our level of abstraction
Raising our level of abstraction
allows for m
allows for m
ore creative work
ore creative work
Ammann & Offutt, 
Introduction to Software Testing (
2nd), 2018
Offutt & Abdurazik, 
Generating tests from UML specifications
, UML (Models), 1999
Postfix values
Prefix (setup) values
Object obj = Min.min(list);
An automated model-based test
6
@Test
}
Test case values
public void testMutuallyIncomparable () {
Expected output
(expected = ClassCastException.class)
list.add(1);
List list = new ArrayList();
list.add("cat");
list.add("dog");
abstract
concrete
 
Test automation categories model
 
7
 
Test automation
 
how
 
what
 
human scripts
C/R (GUIs)
script languages
frameworks (JUnit)
continuous integration (DevOps)
 
Execution
 
Generation
 
Management
 
test values
 
new
test
 
change
test
 
delete
test
 
human-based
 
criteria-based
8
A
chievements in test
automation
(generation)
(1)
mapping problem
(abstract test to concrete test)
Mapping 
example—vending machine
Abstract test: [ 1, 3, 4, 1, 2, 4 ]
Refined abstract test:
Concrete test (mapping):
1.
2.
3.
4.
5.
Testers implement abstract tests by hand
But each transition 
maps
 to specific concrete actions
Automate by 
assembling
 concrete test components
9
1
2
3
4
Coin
 
GetChoc
 
AddChoc
Coin
 
GetChoc
Coin
 
AddChoc
 
AddChoc
 
AddChoc
or
 Coin
 
AddChoc, Coin, GetChoc, Coin, AddChoc
 
c
 = “m&ms”; addChocolate(
c
);
v
 = “$1”; addCoin(
v
);
chooseChocolate(
c
); dispense();
v
 = “$1”; addCoin(
v
);
c
 = “m&ms”; addChocolate(
c
);
Mapping 
example—vending machine
Abstract test: [ 1, 3, 4, 1, 2, 4 ]
Refined abstract test:
Concrete test (mapping):
1.
2.
3.
4.
5.
Testers implement abstract tests by hand
But each transition 
maps
 to specific concrete actions
Automate by 
assembling
 concrete test components
10
1
2
3
4
Coin
Coin
Coin
 
Add
Whiskey
, Coin, Get
Whiskey
, Coin, Add
Whiskey
 
w
 = Jameson; add
IrishWhiskey
(
w
);
v
 = “2 €”; addCoin(
v
);
choose
IrishWhiskey
(); dispense();
v
 = “2 €”; addCoin(
v
);
w
 = Jameson; add
IrishWhiskey
(
w
);
 
Get
Whiskey
 
Add
Whiskey
 
Get
Whiskey
 
Add
Whiskey
 
Add
Whiskey
 
Add
Whiskey
or
 Coin
Assembling test components
11
Test components
C1
C2
C3
C4
C5
C6
C7
Test 1
Test 3
Test 2
C1
C3
C4
C1
C5
C6
C2
C7
C4
C5
Each abstract
Each abstract
test component
test component
is mapped to real
is mapped to real
code in concrete
code in concrete
tests
tests
many
many
times
times
Li & Offutt, 
Test Automation Language
Framework for Behavioral Models
, A-MOST 2015
 
12
A
chievements in test
automation
(generation)
 
(2) test
 oracles
Briand, Di Penta, Labiche, 
Assessing and Improving State-Based Class Testing
, TSE 2004
Barr, Harman, McMinn, Shahbaz, & Yoo, 
The Oracle Problem in Software Testing: A Survey
, TSE 2015
 
1980s RIP model (fault & failure)
 
13
 
R
eachability
I
nfection
P
ropagation
Test
Fault
Error
program
state
 
Infects
 
Reaches
 
Propagates
Outputs
(final program state)
Morell, 
A Theory of Fault-Based Testing
, 1990 (PhD diss. 1984)
Offutt, 
Automatic Test Data Generation
 (PhD diss. 1988)
RIP model assumptions
14
R
eachability
I
nfection
P
ropagation
Implicit assumptions
1.
A human tester looked
 at the output state
2.
If the output was wrong, the tester
 saw it
        No
 
false positives
        No
 false negatives
assertions
assertions
If the assertion
If the assertion
looks
looks
common!
common!
 
RIP
R
 model
 
R
eachability
I
nfection
P
ropagation
Revealability
Test
Fault
Error
program
state
Test
oracles
Observed final
output 
state
 
Reaches
 
Infects
 
Propagates
 
Reveals
Incorrect
final output
 
15
 
Outputs
final program state
Li & Offutt, 
Test oracle strategies for model-based testing
, TSE 2017
 
What makes a good test oracle?
 
How do we model test oracles?
Do test oracles need to observe the entire output space?
Arbitrarily large
Do test oracles need to observe intermediate states?
Do test oracles always return the same answer?
How do we automate test oracles when we do not know
the correct output?
 
16
Hundreds of results on these questions!
 
Good test oracles are consistent
 
Sometimes tests behave differently on different runs
 
 
 
 
Google says 16% of their tests are 
flaky
What makes a test flaky?
 
17
 
Concurrency
Asynchronous behavior
Random inputs
Resource leaks
 
Test order dependency
Collection class assumptions
Relying on external systems
Fowler, 
Eradicating non-determinism in tests
, online 2011
Luo, Hariri, Eloussi, Marinov, 
An Empirical Analysis of Flaky Tests
, FSE 2014
Good test oracles look
Smoke
 (crash) tests miss about two-thirds of the failures
Why waste a good test?
Tests are expensive to design, to implement, and to run
18
@Test (expected = NullPointerException.class)
 public void addOneValue()
 {
     list.addFront(“cat”);
     Object obj = list.getFirst();
}
???
Should check: [ cat ]
 
Good test oracles (TO) can see
 
A 
blind test 
blind test 
does 
not
 check the portion of the output that
is incorrect
 
19
@Test
 public void testTwoValues()
 {
     list.addFront(“dog”);
     list.addFront(“cat”);
     Object obj = list.getFirst();
     assertTrue(“Two values”, obj.equals(“cat”));
 }
Passes:
[ cat ]
[ cat, cat ]
[ cat, null ]
Should check: [ cat, dog ]
Li & Offutt, 
Test oracle strategies for model-based testing
, TSE 2017
Up to 40% of industry automated tests are blind
Baral & Offutt, 
An Empirical Analysis of Blind Tests
, ICST 2020
 
20
A
chievements in test
automation
(management)
 
(3) test evolution
(smart tests)
Pre-automated tests
Scripts
 with human generated values are …
slow!
error prone!
have l
imited repeatability!
almost impossible to
integrate criteria
These tests are primitive
single-cell organisms
21
 
Modern not-smart automated tests
 
Test 
values
Created by a mix of humans and test data generators
Satisfy well-documented goals, test criteria, or specialized
domain needs
Integrated into 
automated test scripts
 (eg, JUnit)
Includes a small amount of brain power … these tests
know what 
results to expect
 (eg, assertions)
Fast
repeatable
These 
m
ulti-cellular
 tests show
preliminary signs of intelligence
 
22
Multicellular test model
Why is it there?
But a modern test
does not know …
When should it run?
When
 should it change?
When
 should it die?
23
Baral, Offutt, & Mulla, 
Self determination: A comprehensive strategy
for making automated tests more effective and efficient
, ICST 2021
 
Intelligent tests
Intelligent tests
Intelligent tests
need
need
self-awareness
self-awareness
and
and
self-determination!
self-determination!
Each test should encode
traceability
 … what it covers
Tests should check what 
changed
 
in
the software, and rerun if needed
Tests should 
alert tester and ask for a
change 
when they no longer match
Tests should 
delete themselves
when no longer needed
 
24
 
Code evolution—test changes
 
25
26
Seven challenges
 and
problems
in test automation
 
The Web changed everything
 
27
High quality become essential …
reliability, usability, maintainability, …
Distribution became free …
web deployment
Support became cheap …
we search the Web
Continuous updates …
“perfect out of the box” is outdated
The Web and agile processes resuscitated 
evolutionary design
Evolutionary design 
requires
 continuous (automated) testing
 
TA challenges and problems (1)
 
Examples:
Games, scientific modeling, non-deterministic, AI & ML, …
How to 
build
 automated tests?
Inputs
 are often very complicated
Test oracles
—JUnit assertions are not enough
Metamorphic
 testing is a great idea—but we need more
 
28
Non-determinant
 software
We cannot know the result
 
a priori
Instead of “correct behavior,”
we need
acceptable behavior
 
TA challenges and problems (2)
 
TOs are still usually created 
by hand
Old, current, and new approaches:
Formal specifications
—lots of research, but limited use
Aggregating
 unit-level TOs into system-level TOs
Impact analysis
 to identify which parts of output state to check
Screen capturing
 approaches
Machine learning
 to “guess” expected results?
Leverage 
TDD tests
 to create expected results for similar tests?
Mutate
 test values, then design corresponding TO mutations?
 
29
Test oracles
We need to automate
 generation
 
TA challenges and problems (3)
 
Test automation tools (execution) are very 
slow
Most 
inputs
 are through the screen with funny gestures
and auto-fills
Challenging to automate
Every mobile device has its own 
ecosystem
How to model these ecological communities?
How to test entanglements with other apps?
What does  a 
test oracle
 look like in a mobile app?
 
30
Testing mobile apps
Many details are very different
 
TA challenges and problems (4)
 
We need effective techniques to automate the testing of
non-behavioral properties
Timeliness
 … automatic execution of timely tests is complex
Accuracy
 … if software approximates a solution, how can we
evaluate how accurate its results are?
Usability
 … human studies are usually done by hand … can we
create a model that can empower automation?
Ethical 
and 
equitable 
behavior … can we define this generally
and specifically enough to empower automation?
 
31
Non-behavioral properties
 
TA challenges and problems (5)
 
Can the smart test approach be expanded to 
arbitrary
models
 and other 
coverage criteria
?
Can 
NLP
 be used to connect requirements to code?
How to build a 
full test framework 
where tests are truly
self-aware?
Agent-based software might be an effective model
Can such a framework be fully implemented into an 
IDE
?
 
32
Smart tests
We have only
 scratched the surface
 
TA challenges and problems (6)
 
When 
I learned 
to program …
compile … integrate … link … run … repeat
Each step took multiple iterations as automated tools (compilers,
IDEs, make) helped us find problems
IDEs should treat 
behavioral checking
 exactly the same as
syntax and semantics checking
Compile … integrate … 
generate tests
 … generate TOs … run tests
And the IDE should use automatic program repair to
automatically fix some faults
 
33
IDE-integrated testing
Tests should run when
 we compile
 
TA challenges and problems (7)
 
Tests identify 
failures
More failing tests help APR—especially if they are different
Can we automatically create more failing tests ?
Automatic 
localization
 is still challenging
How to localize faults that span multiple statements ?
How to localize faults that require structural changes ?
Automatic 
repair techniques
 are still rudimentary
Mutants are limited to simple changes
How can we model more complex changes ?
Are tests comprehensive enough to validate proposed repairs ?
 
34
Automatic program repair
 
Takeaways
 
35
(1)
Test automation has
 enormous impact
 
JUnit
 
test suite
management
 
agile &
TDD
 
continuous
integration
 
capture
replay
 
test
scripts
 
test
oracle
 
Model-
based
testing
 
Takeaways
 
36
(2)
Good models empower good research
 
impactful
 
^
Takeaways
37
(3)
We have barely started!
Every single research result
reveals two more problems
we need to solve
Jeff Offutt
Jeff Offutt
cs.gmu.edu/~offutt
“Computer science amuses and fascinates us,
but automation changes the world.”
(paraphrasing Isaac Asimov)
Slide Note

Biden said: “Your feet will bring you where your heart is. It’s an honor to return.”

Embed
Share

Test automation has evolved over the years, with goals shifting from faster execution and repeatable results to embracing unit testing and automation for more modularity and efficiency. Challenges and achievements in test automation demonstrate the progress towards smarter and more effective testing practices.


Uploaded on May 16, 2024 | 1 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Test Automation: From slow & weak to fast, flaky, & blind to smart & effective Jeff Offutt George Mason University cs.gmu.edu/~offutt Science can amuse and fascinate us, but it is engineering that changes the world. -- Isaac Asimov

  2. Test automation in the 1990s Goals for automating test execution Faster execution Repeatable results Fewer errors Mostly system testing Values generated by hand Results checked by hand (oracle) Capture-replay tools for GUIs Early unit-level test frameworks Assertions to check results JUnit integrated the best ideas and simplified test automation TA achievements major challenges & open problems takeaways of 37 2

  3. Unit testing and automation Unit tests are easier to automate Thus more automation More modularity Less re-design Faster to produce Re-verification supports evolution Cheaper Less repetitive work Better software reduces support costs More predictable test automation TA achievements major challenges & open problems takeaways of 37 3

  4. Then thisRoI for unit testing 60 Assume $1000 unit cost per fault, 100 faults 50 40 Fault origin (%) 30 Fault detection (%) 20 10 Unit cost (X) 0 Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002 test automation TA achievements major challenges & open problems takeaways of 37 4

  5. Abstraction test design Model-based testing model / structure test requirements Abstract level (design) Concrete level (implementation) software artifact input values pass / fail test results test scripts test cases Offutt & Abdurazik, Generating tests from UML specifications, UML (Models), 1999 Ammann & Offutt, Introduction to Software Testing (2nd), 2018 test automation TA achievements major challenges & open problems takeaways of 37 5

  6. An automated model-based test @Test Expected output (expected = ClassCastException.class) public void testMutuallyIncomparable () { List list = new ArrayList(); Prefix (setup) values list.add("cat"); list.add("dog"); Test case values list.add(1); Postfix values Object obj = Min.min(list); } abstract concrete test automation TA achievements major challenges & open problems takeaways of 37 6

  7. Test automation categories model Test automation Execution Generation Management how what new test change test delete test test oracle prefix postfix values test values human-based human scripts prefix & postfix criteria-based C/R (GUIs) assembled pieces script languages frameworks (JUnit) test oracle continuous integration (DevOps) test automation TA achievements major challenges & open problems takeaways of 37 7

  8. Achievements in test automation (generation) (1) mapping problem (abstract test to concrete test) test automation TA achievements major challenges & open problems takeaways of 37 8

  9. Mapping examplevending machine Abstract test: [ 1, 3, 4, 1, 2, 4 ] Refined abstract test: Concrete test (mapping): 1. 2. 3. 4. 5. Testers implement abstract tests by hand But each transition maps to specific concrete actions Automate by assembling concrete test components Coin Coin 1 2 AddChoc, Coin, GetChoc, Coin, AddChoc GetChoc AddChoc AddChoc c= m&ms ; addChocolate(c); v= $1 ; addCoin(v); chooseChocolate(c); dispense(); v= $1 ; addCoin(v); c= m&ms ; addChocolate(c); Coin 3 4 GetChoc AddChoc or Coin AddChoc test automation TA achievements major challenges & open problems takeaways of 37 9

  10. Mapping examplevending machine Abstract test: [ 1, 3, 4, 1, 2, 4 ] Refined abstract test: Concrete test (mapping): 1. 2. 3. 4. 5. Testers implement abstract tests by hand But each transition maps to specific concrete actions Automate by assembling concrete test components Coin Coin 1 2 AddWhiskey, Coin, GetWhiskey, Coin, AddWhiskey GetWhiskey AddWhiskey AddWhiskey w = Jameson; addIrishWhiskey(w); v= 2 ; addCoin(v); chooseIrishWhiskey(); dispense(); v= 2 ; addCoin(v); w = Jameson; addIrishWhiskey(w); Coin 3 4 GetWhiskey AddWhiskey or Coin AddWhiskey test automation TA achievements major challenges & open problems takeaways of 37 10

  11. Assembling test components Test components Test 1 Test 2 C1 C1 C1 C2 C2 C3 C3 Test 3 C4 C4 C4 C5 C5 C5 C6 C6 C7 C7 Li & Offutt, Test Automation Language Framework for Behavioral Models, A-MOST 2015 test automation TA achievements major challenges & open problems takeaways of 37 11

  12. Achievements in test automation (generation) (2) test oracles Briand, Di Penta, Labiche, Assessing and Improving State-Based Class Testing, TSE 2004 Barr, Harman, McMinn, Shahbaz, & Yoo, The Oracle Problem in Software Testing: A Survey, TSE 2015 test automation TA achievements major challenges & open problems takeaways of 37 12

  13. 1980s RIP model (fault & failure) Test Reaches Outputs Reachability (final program state) Fault Infection Infects Propagation Propagates Error program state Morell, A Theory of Fault-Based Testing, 1990 (PhD diss. 1984) Offutt, Automatic Test Data Generation (PhD diss. 1988) test automation TA achievements major challenges & open problems takeaways of 37 13

  14. RIP model assumptions Implicit assumptions 1. A human tester looked at the output state 2. If the output was wrong, the tester saw it No false positives No false negatives common! assertions Test Outputs (final program state) If the assertion looks Reaches Reachability Infection Propagation Fault Infects Error program state Propagates test automation TA achievements major challenges & open problems takeaways of 37 14

  15. RIPR model Outputs Test final program state Reaches Reachability Observed final output state Infection Fault Incorrect final output Propagation Infects Propagates Reveals Revealability Error program state Test oracles Li & Offutt, Test oracle strategies for model-based testing, TSE 2017 test automation TA achievements major challenges & open problems takeaways of 37 15

  16. What makes a good test oracle? How do we model test oracles? Do test oracles need to observe the entire output space? Arbitrarily large Do test oracles need to observe intermediate states? Do test oracles always return the same answer? How do we automate test oracles when we do not know the correct output? Hundreds of results on these questions! test automation TA achievements major challenges & open problems takeaways of 37 16

  17. Good test oracles are consistent Sometimes tests behave differently on different runs test A run 1 test A run 2 test A run 3 Google says 16% of their tests are flaky What makes a test flaky? Concurrency Asynchronous behavior Random inputs Resource leaks Test order dependency Collection class assumptions Relying on external systems Fowler, Eradicating non-determinism in tests, online 2011 Luo, Hariri, Eloussi, Marinov, An Empirical Analysis of Flaky Tests, FSE 2014 test automation TA achievements major challenges & open problems takeaways of 37 17

  18. Good test oracles look Smoke (crash) tests miss about two-thirds of the failures Why waste a good test? Tests are expensive to design, to implement, and to run @Test (expected = NullPointerException.class) public void addOneValue() { list.addFront( cat ); Object obj = list.getFirst(); } ??? Should check: [ cat ] test automation TA achievements major challenges & open problems takeaways of 37 18

  19. Good test oracles (TO) can see A blind test does not check the portion of the output that is incorrect @Test public void testTwoValues() { list.addFront( dog ); list.addFront( cat ); Object obj = list.getFirst(); assertTrue( Two values , obj.equals( cat )); } Passes: [ cat ] [ cat, cat ] [ cat, null ] Should check: [ cat, dog ] Up to 40% of industry automated tests are blind Li & Offutt, Test oracle strategies for model-based testing, TSE 2017 Baral & Offutt, An Empirical Analysis of Blind Tests, ICST 2020 test automation TA achievements major challenges & open problems takeaways of 37 19

  20. Achievements in test automation (management) (3) test evolution (smart tests) test automation TA achievements major challenges & open problems takeaways of 37 20

  21. Pre-automated tests Scripts with human generated values are slow! error prone! have limited repeatability! almost impossible to integrate criteria These tests are primitive single-cell organisms test automation TA achievements major challenges & open problems takeaways of 37 21

  22. Modern not-smart automated tests Test values Created by a mix of humans and test data generators Satisfy well-documented goals, test criteria, or specialized domain needs Integrated into automated test scripts (eg, JUnit) Includes a small amount of brain power these tests know what results to expect (eg, assertions) Fast repeatable These multi-cellular tests show preliminary signs of intelligence test automation TA achievements major challenges & open problems takeaways of 37 22

  23. Multicellular test model But a modern test does not know prefix values test values postfix values Why is it there? expected results When should it run? When should it change? When should it die? Baral, Offutt, & Mulla, Self determination: A comprehensive strategy for making automated tests more effective and efficient, ICST 2021 test automation TA achievements major challenges & open problems takeaways of 37 23

  24. Intelligent tests Intelligent tests need self-awareness and self-determination! Each test should encode traceability what it covers Tests should check what changed in the software, and rerun if needed Tests should alert tester and ask for a change when they no longer match Tests should delete themselves when no longer needed test automation TA achievements major challenges & open problems takeaways of 37 24

  25. Code evolutiontest changes Changed method Original method concatNames (s1, s2) fName = s1 lName = s2 return lName+ +fName concatNames (s1, s2) fName = s1 lName = s2 return fName+ +lName Changed test Original test testConcatNames (s1, s2) result = concatNames( Anita , Borg ) assertTrue(result == Borg Anita ) testConcatNames (s1, s2) result = concatNames( Anita , Borg ) assertTrue(resultContains( Anita ) && (result.contains( Borg )) test automation TA achievements major challenges & open problems takeaways of 37 25

  26. Seven challenges and problems in test automation test automation TA achievements major challenges & open problems takeaways of 37 26

  27. The Web changed everything High quality become essential reliability, usability, maintainability, Distribution became free web deployment Support became cheap we search the Web Continuous updates perfect out of the box is outdated The Web and agile processes resuscitated evolutionary design Evolutionary design requires continuous (automated) testing test automation TA achievements major challenges & open problems takeaways of 37 27

  28. TA challenges and problems (1) Non-determinant software We cannot know the result a priori Examples: Games, scientific modeling, non-deterministic, AI & ML, How to build automated tests? Inputs are often very complicated Test oracles JUnit assertions are not enough Metamorphic testing is a great idea but we need more Instead of correct behavior, we need acceptable behavior test automation TA achievements major challenges & open problems takeaways of 37 28

  29. TA challenges and problems (2) Test oracles We need to automate generation TOs are still usually created by hand Old, current, and new approaches: Formal specifications lots of research, but limited use Aggregating unit-level TOs into system-level TOs Impact analysis to identify which parts of output state to check Screen capturing approaches Machine learning to guess expected results? Leverage TDD tests to create expected results for similar tests? Mutate test values, then design corresponding TO mutations? test automation TA achievements major challenges & open problems takeaways of 37 29

  30. TA challenges and problems (3) Testing mobile apps Many details are very different Test automation tools (execution) are very slow Most inputs are through the screen with funny gestures and auto-fills Challenging to automate Every mobile device has its own ecosystem How to model these ecological communities? How to test entanglements with other apps? What does a test oracle look like in a mobile app? test automation TA achievements major challenges & open problems takeaways of 37 30

  31. TA challenges and problems (4) Non-behavioral properties We need effective techniques to automate the testing of non-behavioral properties Timeliness automatic execution of timely tests is complex Accuracy if software approximates a solution, how can we evaluate how accurate its results are? Usability human studies are usually done by hand can we create a model that can empower automation? Ethical and equitable behavior can we define this generally and specifically enough to empower automation? test automation TA achievements major challenges & open problems takeaways of 37 31

  32. TA challenges and problems (5) Smart tests We have only scratched the surface Can the smart test approach be expanded to arbitrary models and other coverage criteria? Can NLP be used to connect requirements to code? How to build a full test framework where tests are truly self-aware? Agent-based software might be an effective model Can such a framework be fully implemented into an IDE? test automation TA achievements major challenges & open problems takeaways of 37 32

  33. TA challenges and problems (6) IDE-integrated testing Tests should run when we compile When I learned to program compile integrate link run repeat Each step took multiple iterations as automated tools (compilers, IDEs, make) helped us find problems IDEs should treat behavioral checking exactly the same as syntax and semantics checking Compile integrate generate tests generate TOs run tests And the IDE should use automatic program repair to automatically fix some faults test automation TA achievements major challenges & open problems takeaways of 37 33

  34. TA challenges and problems (7) Automatic program repair Tests identify failures More failing tests help APR especially if they are different Can we automatically create more failing tests ? Automatic localization is still challenging How to localize faults that span multiple statements ? How to localize faults that require structural changes ? Automatic repair techniques are still rudimentary Mutants are limited to simple changes How can we model more complex changes ? Are tests comprehensive enough to validate proposed repairs ? test automation TA achievements major challenges & open problems takeaways of 37 34

  35. Takeaways (1) Test automation has enormous impact test scripts test oracle continuous integration JUnit capture replay agile & TDD Model- based testing test suite management test automation TA achievements major challenges & open problems takeaways of 37 35

  36. Takeaways (2) Good models empower good research ^ test model / structure requirements Execution Generation Management Abstract level how what new test change test delete test Concrete level software artifact values test oracle prefix postfix input values test values human-based human scripts pass / fail test results test scripts test cases prefix & postfix criteria-based C/R (GUIs) script languages assembled pieces frameworks (JUnit) test oracle continuous integration Outputs Test final program state prefix values Reaches Observed final output state Incorrect final state Fault test values expected results postfix values Infects Error program state Propagates Reveals Test oracles test automation TA achievements major challenges & open problems takeaways of 37 36

  37. Takeaways (3) We have barely started! Every single research result reveals two more problems we need to solve Computer science amuses and fascinates us, but automation changes the world. (paraphrasing Isaac Asimov) Jeff Offutt cs.gmu.edu/~offutt test automation TA achievements major challenges & open problems takeaways of 37 37

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#