Automatically Repairing Broken Workflows for GUI Applications
A
u
t
o
m
a
t
i
c
a
l
l
y
R
e
p
a
i
r
i
n
g
B
r
o
k
e
n
W
o
r
k
f
l
o
w
s
f
o
r
E
v
o
l
v
i
n
g
G
U
I
A
p
p
l
i
c
a
t
i
o
n
s
S
a
i
Z
h
a
n
g
University of Washington
Joint work with: Hao Lü, Michael D. Ernst
End-user’s workflow
A
w
o
r
k
f
l
o
w
=
A
s
e
q
u
e
n
c
e
o
f
U
I
a
c
t
i
o
n
s
f
o
r
a
s
p
e
c
i
f
i
c
t
a
s
k
2
A
3-action
workflow of creating a crossword puzzle:
1
2
3
1
. Click menu
i
tem
3
. Click OK
2
.Fill in textbox
1
2
E
x
a
m
p
l
e
:
GUI evolution can break workflows
3
V
e
r
s
i
o
n
0
.
3
V
e
r
s
i
o
n
0
.
3
5
?
T
h
e
w
o
r
k
f
l
o
w
i
s
b
r
o
k
e
n
!
(the
first
action in creating a puzzle)
the
first
action in creating a puzzle is
broken
.
Goal: repair a broken workflow
•
S
u
g
g
e
s
t
a
“
r
e
p
l
a
c
e
m
e
n
t
a
c
t
i
o
n
”
f
o
r
a
b
r
o
k
e
n
a
c
t
i
o
n
–
N
o
c
h
a
n
g
e
t
o
t
h
e
c
o
d
e
–
H
e
l
p
u
s
e
r
s
p
e
r
f
o
r
m
t
h
e
s
a
m
e
t
a
s
k
,
b
u
t
a
d
a
p
t
t
o
t
h
e
n
e
w
G
U
I
V
e
r
s
i
o
n
0
.
3
5
?
C
l
i
c
k
“
N
e
w
C
r
o
s
s
w
o
r
d
”
(Suggested by our technique:
FlowFixer,
since both
invoke method “
showCrosswordBuilder
”)
R
e
p
l
a
c
e
m
e
n
t
a
c
t
i
o
n
:
GUIs keep evolving all the time
5
GUIs keep evolving all the time
6
GUIs keep evolving all the time
7
GUIs keep evolving all the time
8
GUIs keep evolving all the time
9
GUIs keep evolving all the time
10
GUI evolution can
break
workflows!
Broken workflows in practice
•
A
f
f
e
c
t
u
s
e
r
e
x
p
e
r
i
e
n
c
e
(
f
o
c
u
s
o
f
t
h
i
s
t
a
l
k
)
•
I
m
p
a
c
t
a
u
t
o
m
a
t
e
d
t
e
s
t
i
n
g
- mimic workflows
-
3
0
–
7
0
%
o
f
t
h
e
m
a
r
e
b
r
o
k
e
n
i
n
G
U
I
e
v
o
l
u
t
i
o
n
[
Memon’03, Grechanik’09, Daniel’11
]
Tedious
and
challenging
to resolve them manually
1
0
0
+
p
o
s
t
s
Example:
the ribbon UI in Office 2007
•
A UI action’s effect cannot be observed statically
•
Repairing broken workflows needs to:
–
distinguish actions that
look
similar
but have
different
results
–
identify
different
UI actions that may perform the
same
task
The “action semantics” challenge
12
Requires knowing the “what the action does”
Outline
•
Problem
•
Technique
•
Evaluation
•
Related Work
•
Contributions
13
Key insights of FlowFixer
•
The
underlying
code
implementing the
same
functionality
stays relatively the same
between versions
•
“action semantics”
≈
the invoked methods
•
UI Actions invoking
similar
methods
are likely to perform
similar
tasks
14
An overview of the FlowFixer technique
15
G
U
I
c
h
a
n
g
e
actionPerformed()
showCrosswordBuilder()
...
O
l
d
v
e
r
s
i
o
n
N
e
w
v
e
r
s
i
o
n
User demonstration
Random testing
1. Click “New Crossword”
2. Click “Save Crossword”
3. Click “Solve New Crossword”
actionPerformed()
showCrosswordBuilder()
...
actionPerformed()
saveCrossword()
...
actionPerformed()
crosswordSolverPanel<init>()
...
Replacement actions:
1.
Click “New Crossword”
2.
…
Method matching
1
/
3
1
/
3
1
/
3
W
e
i
g
h
t
1
The FlowFixer technique
16
instrument
an execution trace
User demonstrates
the workflow
up to
the broken action
Record all methods invoked
by the broken action
(
t
h
e
f
i
r
s
t
a
c
t
i
o
n
i
s
b
r
o
k
e
n
)
The FlowFixer technique
17
instrument
an execution trace
Static Method
Matching
Match each method
invoked by the broken
action in the new version
The FlowFixer technique
18
instrument
an execution trace
instrument
Static Method
Matching
Random testing
Randomly execute
each applicable UI action, and
r
ecursively
explore UI actions on new screens
Matched
Methods
(in the new version)
The FlowFixer technique
19
instrument
an execution trace
instrument
Static Method
Matching
Random testing
Replacement
Action
Recommendation
For each invoked method, find all actions invoking it.
The weight of each action is
inversely proportional
to the
number of all possible invoking actions.
Matched
Methods
Outline
•
Problem
•
Technique
•
Evaluation
•
Related Work
•
Contributions
20
Research questions
•
How effective is FlowFixer in repairing broken workflows?
–
Accuracy
–
Efficiency
•
Comparison with a GUI-comparison-based technique
[
Grechanik’09
]
21
Subject programs and broken workflows
22
16
workflows
with
distinct
root causes.
Collected from user
manual.
Non-trivial
code changes
Popular software, being
actively developed for
3—12 years
•
Selection of broken workflows
–
3
5
6
d
o
c
u
m
e
n
t
e
d
w
o
r
k
f
l
o
w
s
,
7
0
a
r
e
b
r
o
k
e
n
,
1
6
h
a
v
e
d
i
s
t
i
n
c
t
r
o
o
t
c
a
u
s
e
s
–
E
x
c
l
u
d
e
t
r
i
v
i
a
l
U
I
c
h
a
n
g
e
s
,
e
.
g
.
,
•
swapping two neighboring menu items
•
move a button to a different location on the same panel
.
FlowFixer’s accuracy
•
M
e
a
s
u
r
e
d
b
y
t
h
e
a
b
s
o
l
u
t
e
r
a
n
k
o
f
t
h
e
c
o
r
r
e
c
t
a
c
t
i
o
n
s
23
F
l
o
w
F
i
x
e
r
c
a
n
r
e
p
a
i
r
1
5
b
r
o
k
e
n
w
o
r
k
f
l
o
w
s
FlowFixer’s efficiency
•
R
a
n
d
o
m
t
e
s
t
i
n
g
–
27 mins
per
application
(
A
o
n
e
-
t
i
m
e
c
o
s
t
,
s
h
a
r
e
d
b
y
d
i
f
f
e
r
e
n
t
w
o
r
k
f
l
o
w
s
)
•
U
s
e
r
d
e
m
o
n
s
t
r
a
t
i
o
n
–
<
1
m
i
n
p
e
r
w
o
r
k
f
l
o
w
(assuming the old version is installed)
•
A
c
t
i
o
n
r
e
c
o
m
m
e
n
d
a
t
i
o
n
–
4 mins
per
workflow
24
An example repair
25
S
a
v
e
c
u
r
r
e
n
t
s
t
a
t
e
?
F
i
l
l
t
h
e
t
e
x
t
b
o
x
t
o
s
a
v
e
t
h
e
c
u
r
r
e
n
t
s
t
a
t
e
An example repair
26
S
a
v
e
c
u
r
r
e
n
t
s
t
a
t
e
?
F
i
l
l
t
h
e
t
e
x
t
b
o
x
t
o
s
a
v
e
t
h
e
c
u
r
r
e
n
t
s
t
a
t
e
UndoableEditImpl.createTemporaryFile
Comparison with an existing technique
•
R
E
S
T
:
a
G
U
I
-
c
o
m
p
a
r
i
s
o
n
-
b
a
s
e
d
t
e
c
h
n
i
q
u
e
[
G
r
e
c
h
a
n
i
k
’
0
9
]
–
A
black-box
approach
–
Compare GUIs of two versions to identify modified UI elements
–
I
d
e
n
t
i
f
i
e
s
a
f
f
e
c
t
e
d
a
c
t
i
o
n
s
,
b
u
t
g
i
v
e
s
n
o
r
e
p
a
i
r
s
u
g
g
e
s
t
i
o
n
27
O
l
d
v
e
r
s
i
o
n
N
e
w
v
e
r
s
i
o
n
Comparison with an existing technique
•
R
E
S
T
:
a
G
U
I
-
c
o
m
p
a
r
i
s
o
n
-
b
a
s
e
d
t
e
c
h
n
i
q
u
e
[
G
r
e
c
h
a
n
i
k
’
0
9
]
–
A
black-box
approach
–
Compare GUIs of two versions to identify modified UI elements
–
I
d
e
n
t
i
f
i
e
s
a
f
f
e
c
t
e
d
a
c
t
i
o
n
s
,
b
u
t
g
i
v
e
s
n
o
r
e
p
a
i
r
s
u
g
g
e
s
t
i
o
n
•
E
x
t
e
n
d
R
E
S
T
f
o
r
w
o
r
k
f
l
o
w
r
e
p
a
i
r
–
Recommend actions on the
matched
UI element of the
new
version
28
O
l
d
v
e
r
s
i
o
n
N
e
w
v
e
r
s
i
o
n
REST vs. FlowFixer
29
R
E
S
T
F
l
o
w
F
i
x
e
r
1
5
w
o
r
k
f
l
o
w
s
f
i
x
e
d
6
w
o
r
k
f
l
o
w
s
f
i
x
e
d
F
a
i
l
t
o
f
i
x
1
w
o
r
k
f
l
o
w
F
a
i
l
t
o
f
i
x
1
0
w
o
r
k
f
l
o
w
s
Why REST did not work well?
•
R
E
S
T
o
n
l
y
r
e
p
a
i
r
s
6
w
o
r
k
f
l
o
w
s
w
h
e
r
e
a
U
I
e
l
e
m
e
n
t
i
s
moved
to a different location
•
F
l
o
w
F
i
x
e
r
r
e
p
a
i
r
s
1
5
b
r
o
k
e
n
w
o
r
k
f
l
o
w
s
–
Execute UI actions and observe their consequences
30
REST’s
black-box
approach is
not
aware of the “action semantics”
-
Ineffective
for non-trivial UI changes
UI label change
UI element change
UI action change
Experimental conclusions
•
FlowFixer is
accurate
and
efficient
in repairing broken
workflows
•
FlowFixer achieves
better
results than a
GUI-comparison-based technique
31
Outline
•
Problem
•
Technique
•
Evaluation
•
Related Work
•
Contributions
32
Related work
•
T
e
s
t
r
e
p
a
i
r
ReAssert [
Daniel’09
], REST [
Grechanik’09
], Guitar [
Memon’04
],
Genetic approach [
Huang’10
], WATER [
Choudhary’11
] …
Make obsoleted tests compilable
without
preserving its original semantics.
N
o
t
a
p
p
l
i
c
a
b
l
e
t
o
r
e
p
a
i
r
i
n
g
b
r
o
k
e
n
w
o
r
k
f
l
o
w
s
.
•
P
r
o
g
r
a
m
r
e
p
a
i
r
GenProg [
Weimer’09
], ClearView [
Perkins’09
], PAR [
Kim’13
]…
Search patches for bugs.
N
o
t
a
p
p
l
i
c
a
b
l
e
t
o
b
r
o
k
e
n
w
o
r
k
f
l
o
w
s
c
a
u
s
e
d
b
y
U
I
c
h
a
n
g
e
s
.
•
C
h
a
n
g
e
a
n
a
l
y
s
i
s
Chianti [
Ren’05
], SemDiff [
Dagenais’08
], RefactoringCrawler [
Dig’05
],
Hybrid approach [
Wang’12
] …
I
d
e
n
t
i
f
y
c
o
d
e
-
l
e
v
e
l
c
h
a
n
g
e
s
a
n
d
c
o
m
p
u
t
e
t
h
e
e
f
f
e
c
t
s
.
N
o
t
a
p
p
l
i
c
a
b
l
e
f
o
r
r
e
p
a
i
r
i
n
g
U
I
-
l
e
v
e
l
w
o
r
k
f
l
o
w
s
.
33
Outline
•
Problem
•
Technique
•
Evaluation
•
Related Work
•
Contributions
34
Future directions
•
User study
•
Extend FlowFixer to repair UI test scripts
–
Lift
syntax-correcting
repair to
semantics-preserving
repair
•
Integrate FlowFixer into software evolution
–
Proactively finding broken workflows
–
Summarize UI-level changes
–
Automatically update user manual
–
Help users learn new GUI features
35
Contributions
•
A technique to repair broken workflows
analyze method invocations and evolution to reason about fix actions
–
fully automated
–
handles non-trivial code changes
•
Experiments that demonstrate its usefulness
–
Accurate and efficient
•
Fixed
15
out of
16
broken workflows
–
Outperforms alternative techniques
•
The FlowFixer tool implementation:
http://workflow-repairer.googlecode.com
36
A
b
r
o
k
e
n
w
o
r
k
f
l
o
w
F
l
o
w
F
i
x
e
r
Fix suggestions
[Backup Slides]
37
What if multiple actions are broken?
•
Use FlowFixer in an interactive way
38
F
i
x
e
d
!
Might be a different broken action!
…
FlowFixer’s recommendation limitation
•
Recommends one replacement action for a broken action
•
Does not support recommending:
–
A
s
e
q
u
e
n
c
e
o
f
a
c
t
i
o
n
s
f
o
r
o
n
e
a
c
t
i
o
n
–
O
n
e
a
c
t
i
o
n
f
o
r
a
s
e
q
u
e
n
c
e
o
f
a
c
t
i
o
n
s
–
A
s
e
q
u
e
n
c
e
o
f
a
c
t
i
o
n
s
f
o
r
a
s
e
q
u
e
n
c
e
o
f
a
c
t
i
o
n
s
39
…
Why does this simple random testing work?
•
Goal:
–
I
d
e
n
t
i
f
y
“
s
i
g
n
a
t
u
r
e
”
m
e
t
h
o
d
f
o
r
e
a
c
h
U
I
a
c
t
i
o
n
–
NOT achieve good coverage
•
The “signature” method is often easy to reach:
•
Symbolic, model-based techniques might achieve
better results, but are more expensive to use
40
actionPerformed()
showCrosswordBuilder()
...
Event handler, shared by many actions
A “
signature
” method, only invoked by
“Clicking New Crossword”
Other methods. Requires certain states
Repairing broken workflows caused by GUI evolution in evolving applications is crucial for maintaining user experience and automated testing integrity. This research focuses on automatically suggesting replacement actions to adapt to changing interfaces without altering the underlying code, addressing manual resolution challenges commonly faced by developers.
Uploaded on Sep 21, 2024 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Automatically Repairing Broken Workflows for Evolving GUI Applications Sai Zhang University of Washington Joint work with: Hao L , Michael D. Ernst
End-users workflow A workflow = A sequence of UI actions for a specific task 2.Fill in textbox Example: A 3-action workflow of creating a crossword puzzle: 1. Click menu item 3. Click OK 12 1 2 2 3
GUI evolution can break workflows Version 0.3 (the first action in creating a puzzle) ? Version 0.35 The workflow is broken! 3
Goal: repair a broken workflow Suggest a replacement action for a broken action No change to the code Help users perform the same task, but adapt to the new GUI ? Version 0.35 the first action in creating a puzzle is broken. Replacement action: Click New Crossword (Suggested by our technique: FlowFixer, since both invoke method showCrosswordBuilder )
GUIs keep evolving all the time GUI evolution can break workflows! 10
Broken workflows in practice Affect user experience (focus of this talk) Example: the ribbon UI in Office 2007 100+ posts Impact automated testing - mimic workflows - 30 70% of them are broken in GUI evolution [Memon 03, Grechanik 09, Daniel 11] Tedious and challenging to resolve them manually
The action semantics challenge A UI action s effect cannot be observed statically Repairing broken workflows needs to: distinguish actions that look similar but have different results identify different UI actions that may perform the same task Requires knowing the what the action does 12
Outline Problem Technique Evaluation Related Work Contributions 13
Key insights of FlowFixer The underlying code implementing the same functionality stays relatively the same between versions action semantics the invoked methods UI Actions invoking similarmethods are likely to perform similar tasks 14
An overview of the FlowFixer technique Old version New version GUI change User demonstration Random testing Weight actionPerformed() showCrosswordBuilder() ... 1 1/3 1. Click New Crossword actionPerformed() showCrosswordBuilder() ... Method matching 1/3 2. Click Save Crossword actionPerformed() saveCrossword() ... 1/3 3. Click Solve New Crossword actionPerformed() crosswordSolverPanel<init>() ... Replacement actions: 1. Click New Crossword 2. 15
The FlowFixer technique Record all methods invoked by the broken action A broken workflow instrument --- --- an execution trace ------ ------ Old version Instrumented version --- - - --- User demonstrates the workflow up to the broken action New version (the first action is broken) 16
The FlowFixer technique A broken workflow instrument --- --- an execution trace ------ ------ Old version Instrumented version --- - - --- Static Method Matching New version Match each method invoked by the broken action in the new version 17
The FlowFixer technique A broken workflow instrument --- --- an execution trace ------ ------ Old version Instrumented version --- - - --- Static Method Matching Matched Methods New version (in the new version) Action method mapping f1(), f2(), f3() instrument --- ------ f1(), f4() Instrumented version Random testing Randomly execute each applicable UI action, and recursively explore UI actions on new screens 18
The FlowFixer technique A broken workflow instrument --- --- an execution trace ------ ------ Old version Instrumented version --- - - --- Static Method Matching Matched Methods New version Action method mapping f1(), f2(), f3() Replacement Action Recommendation instrument --- ------ f1(), f4() Instrumented version Random testing For each invoked method, find all actions invoking it. Ranked list of replacement actions The weight of each action is inversely proportional to the number of all possible invoking actions. 1. 2. 3. 19
Outline Problem Technique Evaluation Related Work Contributions 20
Research questions How effective is FlowFixer in repairing broken workflows? Accuracy Efficiency Comparison with a GUI-comparison-based technique [Grechanik 09] 21
Subject programs and broken workflows Subject Versions LOC #Broken workflows LOC Crossword JEdit Gantt Project JabRef Freemind 0.3 0.35 2.5 2.6 2.0.1 2.5.4 2.0 2.8.1 0.71 0.8 3,087 32,607 55,009 83,447 70,430 1,386 5,017 3,777 38,992 10,757 1 1 5 3 6 Popular software, being actively developed for 3 12 years Non-trivial code changes 16 workflows with distinct root causes. Collected from user manual. Selection of broken workflows 356 documented workflows, 70 are broken, 16 have distinct root causes Exclude trivial UI changes, e.g., swapping two neighboring menu items move a button to a different location on the same panel. 22
FlowFixers accuracy Measured by the absolute rank of the correct actions 1. 2. 2 workflows Correct action ranks second 3. 13 workflows Correct action ranks first 1 workflow FlowFixer outputs wrong result 16 broken workflows FlowFixer can repair 15 broken workflows 23
FlowFixers efficiency Random testing 27 mins per application (A one-time cost, shared by different workflows) User demonstration < 1 min per workflow (assuming the old version is installed) Action recommendation 4 mins per workflow 24
An example repair Save current state Fill the textbox to save the current state Gantt Project version 2.0 ? Gantt Project version 2.5 25
An example repair Save current state UndoableEditImpl.createTemporaryFile Fill the textbox to save the current state Gantt Project version 2.0 ? Gantt Project version 2.5 26
Comparison with an existing technique REST: a GUI-comparison-based technique [Grechanik 09] A black-box approach Compare GUIs of two versions to identify modified UI elements Identifies affected actions, but gives no repair suggestion Old version New version 27
Comparison with an existing technique REST: a GUI-comparison-based technique [Grechanik 09] A black-box approach Compare GUIs of two versions to identify modified UI elements Identifies affected actions, but gives no repair suggestion Extend REST for workflow repair Recommend actions on the matched UI element of the new version Old version New version 28
REST vs. FlowFixer Fail to fix 1 workflow Fail to fix 10 workflows 6 workflows fixed 16 broken workflows 16 broken workflows 15 workflows fixed REST FlowFixer 29
Why REST did not work well? REST only repairs 6 workflows where a UI element is moved to a different location - Ineffective for non-trivial UI changes UI label change UI element change UI action change FlowFixer repairs 15 broken workflows Execute UI actions and observe their consequences REST s black-box approach is not aware of the action semantics 30
Experimental conclusions FlowFixer is accurate and efficient in repairing broken workflows FlowFixer achieves better results than a GUI-comparison-based technique 31
Outline Problem Technique Evaluation Related Work Contributions 32
Related work Test repair ReAssert [Daniel 09], REST [Grechanik 09], Guitar [Memon 04], Genetic approach [Huang 10], WATER [Choudhary 11] Make obsoleted tests compilable without preserving its original semantics. Not applicable to repairing broken workflows. Program repair GenProg [Weimer 09], ClearView [Perkins 09], PAR [Kim 13] Search patches for bugs. Not applicable to broken workflows caused by UI changes. Change analysis Chianti [Ren 05], SemDiff [Dagenais 08], RefactoringCrawler [Dig 05], Hybrid approach [Wang 12] Identify code-level changes and compute the effects. Not applicable for repairing UI-level workflows. 33
Outline Problem Technique Evaluation Related Work Contributions 34
Future directions User study Extend FlowFixer to repair UI test scripts Lift syntax-correcting repair to semantics-preserving repair Integrate FlowFixer into software evolution Proactively finding broken workflows Summarize UI-level changes Automatically update user manual Help users learn new GUI features 35
Fix suggestions FlowFixer Contributions 1. 2. A broken workflow 3. A technique to repair broken workflows analyze method invocations and evolution to reason about fix actions fully automated handles non-trivial code changes Experiments that demonstrate its usefulness Accurate and efficient Fixed 15 out of 16 broken workflows Outperforms alternative techniques The FlowFixer tool implementation: http://workflow-repairer.googlecode.com 36
What if multiple actions are broken? Use FlowFixer in an interactive way Fix action 1. 2. 3. FlowFixer Might be a different broken action! Fix action 1. 2. 3. FlowFixer Fixed! 38
FlowFixers recommendation limitation Recommends one replacement action for a broken action Fix action 1. 2. 3. FlowFixer Does not support recommending: A sequence of actions for one action Fix action 1. 2. 3. FlowFixer Fix action 1. 2. 3. FlowFixer One action for a sequence of actions A sequence of actions for a sequence of actions Fix action 1. 2. 3. FlowFixer 39
Why does this simple random testing work? Goal: Identify signature method for each UI action NOT achieve good coverage The signature method is often easy to reach: Event handler, shared by many actions actionPerformed() showCrosswordBuilder() ... A signature method, only invoked by Clicking New Crossword Other methods. Requires certain states Symbolic, model-based techniques might achieve better results, but are more expensive to use 40
