Effective Strategies for Writing a Research Paper
undefinedTips on Writing a Research Paper
T
h
o
m
a
s
R
e
p
s
U
n
i
v
e
r
s
i
t
y
o
f
W
i
s
c
o
n
s
i
n
R
u
l
e
#
1
There is no universal rule for organizing your paper
However, there are some good guidelines out there
›
Simon Peyton-Jones, “
How to write a great research paper
”
›
Derek Dreyer, “
How to write papers so that people can read them
”
›
Simon Peyton-Jones, “
Other resources
”
Bear in mind that there will always be cases where
it is best to deviate from any single proposed
design
2
O
n
e
O
v
e
r
a
r
c
h
i
n
g
P
r
i
n
c
i
p
l
e
[
S
P
J
]
The purpose of your paper is to
convey your idea from your
head to your reader’s head
. All choices are made
in
support of this single goal
Explain it as if you were speaking to someone using a
whiteboard
Conveying the intuition is primary
, not secondary
Introduce the problem, and your idea, using
examples and
pictures
Once your reader has the intuition, she can follow the
details (but not vice versa)
Present the general case later in the paper
3
Spoon-feed the reader
Be
relentless
in working to achieve this goal
•
Everything you
write
should be in support of it
•
When you
re-read and self-critique
, your criterion
is: “Does this
material best support the goal of conveying my idea to the reader?”
T
w
o
O
v
e
r
a
r
c
h
i
n
g
C
h
a
l
l
e
n
g
e
s
4
Y
o
u
w
i
l
l
a
l
w
a
y
s
b
e
s
p
a
c
e
c
o
n
s
t
r
a
i
n
e
d
5
ACM two-column conference format
6
FSE (10 pages + references)
PLDI (12 pages + references)
ACM one-column Proc. ACM format
7
POPL (25 pages + references)
OOPSLA (23 pages + references)
PLDI (20 pages + references)
FSE (18 pages + 4 pages of refs.)
Springer LNCS
8
CAV (18 pages + references)
ETAPS (16-25 pages
+ references)
9
[SPJ]
10
[SPJ]
11
[SPJ]
S
t
r
u
c
t
u
r
e
(
c
o
n
f
e
r
e
n
c
e
p
a
p
e
r
)
[
S
P
J
]
Title (1000 readers)
Abstract (4 sentences, 100 readers)
Introduction (1 page, 100 readers)
The problem (1 page, 10 readers)
My idea (2 pages, 10 readers)
The details (5 pages, 3 readers)
Related work (1-2 pages, 10 readers)
Conclusions and further work (0.5 pages)
12
A
s
t
r
u
c
t
u
r
e
t
h
a
t
w
o
r
k
s
[
D
D
]
Abstract (1-2 paragraphs, 1000 readers)
Intro (1-2 pages, 100 readers)
Main ideas (2-3 pages, 50 readers)
Technical meat (4-6 pages, 5 readers)
Related work (1-2 pages, 100 readers)
13
A
M
o
r
e
C
o
m
p
l
e
t
e
S
t
r
u
c
t
u
r
e
[
R
e
p
s
]
Abstract
1.
Introduction
2.
Overview and Problem Statement
3.
[Terminology and Notation]
4.
Core Algorithm/Idea
5.
[Extensions]
6.
Experiments
7.
[Limitations/Threats to Validity]
8.
Related Work
9.
[Conclusion]
14
T
w
o
O
v
e
r
a
r
c
h
i
n
g
C
h
a
l
l
e
n
g
e
s
15
A
b
s
t
r
a
c
t
[
K
e
n
t
B
e
c
k
v
i
a
S
P
J
]
Four sentences
1.
State the problem
2.
Say why it’s an interesting problem
3.
Say what your solution achieves
4.
Say what follows from your solution
16
I
n
t
r
o
d
u
c
t
i
o
n
[
S
P
J
]
Describe the problem
State your contributions
... and that is all
ONE PAGE!
17
18
[SPJ]
19
[SPJ]
20
[SPJ]
21
[SPJ]
22
[SPJ]
23
[SPJ]
24
[SPJ]
Well, maybe . . .
•
If you can do it, great!
•
Sometimes just too awkward, or not natural for the introduction to refer to some section(s)
I probably include such a paragraph more often than not
25
[SPJ]
26
Often the no-related-work approach is the right one, but . . .
•
Sometimes it is better to discuss the 1-3 most important pieces of related work
•
The best way to introduce your idea might be to contrast it with some well-known concept
•
Sometimes you need to make sure that the reader understands some essential
background on which your work depends
•
Want to
ensure that
the reviewer does not dismiss your idea as “It’s just the same as XXX.”
[SPJ]
D
D
c
r
i
t
i
q
u
e
o
f
S
P
J
SPJ’s approach to the Introduction eliminates context
Start with a concrete example, e.g. “Consider this
Haskell code…”
If this works, it can be effective, but I find it often
doesn’t work
It assumes reader already knows context
27
T
h
e
C
G
I
M
o
d
e
l
f
o
r
a
n
A
b
s
t
r
a
c
t
/
I
n
t
r
o
[
D
D
]
C
ontext
›
Set the stage, motivate the general topic
G
ap
›
Explain your specific problem and why existing work does
not adequately solve it
I
nnovation
›
State what you’ve done that is new, and explain how it
helps fill the gap
Introduction
›
An expanded version of the abstract, still using CGI
28
S
ignificance
Why is the work important?
Might be part of C, G, or I, or entirely
separate
A
b
s
t
r
a
c
t
[
K
e
n
t
B
e
c
k
v
i
a
S
P
J
]
Four sentences
1.
State the problem (
C
)
2.
Say why it’s an interesting problem (
G
)
3.
Say what your solution achieves (
I
)
4.
Say what follows from your solution (
S
)
29
I
n
t
r
o
d
u
c
t
i
o
n
(
R
e
v
.
E
n
g
.
f
r
o
m
Z
e
l
d
o
v
i
c
h
)
Elevator story -- high-level statement of the problem
The problem in more technical terms
Conventional wisdom: sketch of previous techniques
and their shortcomings
Describe the solution to the problem as a black box,
so that it is clear what your solution offers over
previous work
Technical challenges to obtaining a solution (e.g., 1
paragraph for each)
State how you solve the challenges (at most a few
paragraphs)
Experimental highlights
Contributions
Organization of the paper
30
C
G
I
Zeldovich Example [SOSP09]
31
C: The problem
G: Conventional
wisdom
I: Solution
as black box
I: The solutions offered
I: Experimental
highlights
Organization
Contributions
I: Challenges
What If the Contribution is a Synthesis of Ideas?
32
A
M
o
r
e
C
o
m
p
l
e
t
e
S
t
r
u
c
t
u
r
e
[
R
e
p
s
]
Abstract
1.
Introduction
2.
Overview and Problem Statement
3.
[Terminology and Notation]
4.
Core Algorithm/Idea
5.
[Extensions]
6.
Experiments
7.
[Limitations/Threats to Validity]
8.
Related Work
9.
[Conclusion]
33
O
v
e
r
v
i
e
w
SPJ:
›
Conveying the intuition is primary; do it using examples and
pictures
›
Even if the reader skips the details, she still takes away
something valuable
DD:
1.
Forces you to have a “takeaway”
2.
Many readers only care about the takeaway, not the technical
details
3.
For those who want the technical details, the intuition is still
useful as “scaffolding”
The Gopan Principle:
“
By the time the reader reaches the end of the Overview, they
should feel like they don’t need to read the rest of the paper.”
Denis Gopan, c. 2006
34
Gopan Example [POPL05]
35
Somesh Jha: The problem statement is
not
“The goal is to produce the kind of answers that my
algorithm produces”
The problem should be stated in a way that is independent of
any particular technique or method
Specify
what
is to be obtained, not how it is obtained
›
Declarative
, not imperative
36
P
r
o
b
l
e
m
S
t
a
t
e
m
e
n
t
Problem Statement
37
Problem Statement
38
Problem Statement
39
A
M
o
r
e
C
o
m
p
l
e
t
e
S
t
r
u
c
t
u
r
e
[
R
e
p
s
]
Abstract
1.
Introduction
2.
Overview and Problem Statement
3.
[Terminology and Notation]
4.
Core Algorithm/Idea
5.
[Extensions]
6.
Experiments
7.
[Limitations/Threats to Validity]
8.
Related Work
9.
[Conclusion]
40
The technical meat is often the easiest part to write
•
It is similar to the f
ormalization
that you find in a
homework problem & solution
•
Gives a solution
to the problem, including
Pseudo-code for algorithms
Theorems [and proofs or proof-sketches]
A
M
o
r
e
C
o
m
p
l
e
t
e
S
t
r
u
c
t
u
r
e
[
R
e
p
s
]
Abstract
1.
Introduction
2.
Overview and Problem Statement
3.
[Terminology and Notation]
4.
Core Algorithm/Idea
5.
[Extensions]
6.
Experiments
7.
[Limitations/Threats to Validity]
8.
Related Work
9.
[Conclusion]
41
E
x
p
e
r
i
m
e
n
t
s
Experimental questions
›
What hypothesis are you trying to test?
“The experiments were designed to answer the following
questions:
1.
…
2.
…
3.
…”
Experimental setup
›
CPU, speed, amount of memory; OS version
›
Characteristics of the test suite
Experimental results
›
Table, graph, scatter-plot of X vs. Y, etc.
›
Don’t forget to say what the experiments
showed
about the
experimental questions
42
Experiments
43
Experiments
44
Our experiments were
designed to answer
the
following questions:
1.
…
2.
…
3.
…
Our experiments showed
that the answers to the
questions were
…
E
x
p
e
r
i
m
e
n
t
s
:
S
I
G
P
L
A
N
E
m
p
i
r
i
c
a
l
E
v
a
l
u
a
t
i
o
n
C
h
e
c
k
l
i
s
t
“This checklist is organized into seven
categories
, each with
associated
examples
for illustration. Categories are meant to be
comprehensive, applying to the breadth of possible empirical
evaluations. Examples in each category highlight specific, important
areas in which best practice was frequently not followed. These are
meant to be useful and illustrative, but they are neither comprehensive
nor applicable to every evaluation.
The goal of the checklist is to help authors produce stronger
scholarship, and to help reviewers evaluate such scholarship more
consistently. Importantly, the checklist is meant to support informed
judgment, not supplant it. The committee’s hope is that this checklist
can put all members of the community literally on the same page.”
Webpage
45
L
i
m
i
t
a
t
i
o
n
s
/
T
h
r
e
a
t
s
t
o
V
a
l
i
d
i
t
y
[
S
o
f
t
w
a
r
e
-
E
n
g
i
n
e
e
r
i
n
g
p
a
p
e
r
s
]
Construct validity
›
Are the parameters studied in the experiments relevant to
the research questions posed?
Internal validity
›
Is there selection bias in the examples chosen for study?
External validity
›
Do the conclusions hold in other scenarios
Ideally: explain how you addressed or controlled for
these issues
46
Example
47
R
e
l
a
t
e
d
W
o
r
k
Don’t just list related papers
›
Honest comparison: what is the delta?
Be generous with credit
Complimentary + complementary
48
Related Work
49
Related
Comparison 1
Comparison 2
Comparison 3
50
[SPJ]
51
[SPJ]
52
[SPJ]
C
o
n
c
l
u
s
i
o
n
S
e
c
t
i
o
n
Usually not needed (in my opinion)
›
First thing to be cut when I run out of space
›
But some reviewers complain
May be
similar to the abstract, but can use the
technical concepts
introduced in the paper’s body
Allows you to reiterate what the experimental
results showed
›
if your experiments are careful enough for you to claim
them as definitive (rare)
53
Conclusion Section
54
Recap of
experiments
More technical
recap of abstract
W
r
i
t
e
S
o
m
e
I
t
e
m
s
E
a
r
l
y
(
b
u
t
R
e
v
i
s
i
t
F
r
e
q
u
e
n
t
l
y
)
Abstract
›
Opinions vary
•
Do last
•
Do first, so you know what story you are trying to tell
A few key fragments
›
List of contributions (Introduction)
›
Running example (Overview)
›
Problem statement (Overview)
›
Research questions (Overview) & corresponding experimental
questions (Experiments)
›
Choose the form in which you plan to display experimental results
•
Create mock-ups of the tables/figures
›
Technical meat
•
Pseudo-code for the key algorithms
•
Key examples
The remaining text “writes itself”: you just explain the algorithms and
examples
55
Feature A
Feature B
Avoid 11
th
-hour frustration: “Shoot,
we never thought to implement the
+A/-B configuration”
As you write, your
outlook typically changes
•
You
might realize that YYY is really a
contribution of your work
•
You may need
to adjust the research
questions, problem statement, etc.
•
…
•
The final paper should have
no
inconsistencies
due to different parts
being written when you had not-fully-
formed views of the work
•
Settle on example figures, code,
illustrative diagrams, etc. early
•
The Overview then “writes itself”:
discuss/explain
the elements of
the example systematically
O
n
e
W
a
y
t
o
F
o
r
c
e
Y
o
u
r
s
e
l
f
…
G
i
v
e
a
T
a
l
k
Usual situation
›
Write paper and submit to conference
›
If accepted, make final revisions
›
Conference approaching! Make slides for talk
•
Hmm … how can I convey my idea without all the Greek letters?
•
Create appropriate diagrams
Often the diagrams in talks are superior to those in
the paper!!!
›
“Darn, I wish I had used this diagram in the paper . . .”
Avoid remorse: giving a talk in your group will force
you to make those diagrams early
56
•
Settle on example figures, code,
illustrative diagrams, etc. early
•
The Overview then “writes itself”:
discuss/explain
the elements of
the example systematically
S
t
o
r
y
b
o
a
r
d
i
n
g
:
P
l
a
n
H
o
w
t
o
U
s
e
t
h
e
A
l
l
o
w
e
d
S
p
a
c
e
57
A
d
o
p
t
a
D
i
f
f
e
r
e
n
t
i
a
l
P
e
r
s
p
e
c
t
i
v
e
(
w
h
e
n
a
p
p
r
o
p
r
i
a
t
e
)
Challenge: need to overcome reviewers’ bias to
think
“This work is just a minor increment on XXX.”
Introduction
•
Sometimes the best way to introduce your idea might be to
contrast it with some well-known concept
•
Want to
ensure that
the reviewer does not dismiss your idea as
“It’s just the same as something I already know about.”
Overview & Core Algorithm/Idea (next slide)
Related Work
•
“Complimentary + complementary”
58
A
d
o
p
t
a
D
i
f
f
e
r
e
n
t
i
a
l
P
e
r
s
p
e
c
t
i
v
e
(
w
h
e
n
a
p
p
r
o
p
r
i
a
t
e
)
Sometimes it is the best way to explain an idea
59
S
o
m
e
L
a
n
g
u
a
g
e
B
u
g
a
b
o
o
s
60
What is
shown in
this picture?
It’s a that!
Not a which
(or a witch)!
C
a
p
T
i
m
e
s
R
e
v
i
e
w
o
f
C
h
e
n
’
s
D
u
m
p
l
i
n
g
H
o
u
s
e
[
A
p
r
i
l
1
1
,
2
0
1
8
]
“[They went from] zero to 10 dumplings in 60
seconds.
For mere mortals, the technique is impressive.
(When I tried making dumplings with my friend’s
family in China, my friend looked at my dumplings
and said,
‘It’s OK, everyone has their own
dumpling style.’
)”
Rule #1: There is no universal rule for organizing
your paper
61
Tips for writing a successful research paper include organizing your paper effectively, focusing on conveying your ideas clearly to the reader, addressing challenges such as limited space and busy reviewers, and adhering to conference submission guidelines. Remember to prioritize conveying your main idea and make every part of your paper support this goal.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Tips on Writing a Research Paper Thomas Reps University of Wisconsin
Rule #1 Rule #1 There is no universal rule for organizing your paper However, there are some good guidelines out there Simon Peyton-Jones, How to write a great research paper Derek Dreyer, How to write papers so that people can read them Simon Peyton-Jones, Other resources Bear in mind that there will always be cases where it is best to deviate from any single proposed design 2
One Overarching One Overarching Principle Principle [SPJ] [SPJ] The purpose of your paper is to convey your idea from your head to your reader s head. All choices are made in support of this single goal Explain it as if you were speaking to someone using a whiteboard Conveying the intuition is primary, not secondary Introduce the problem, and your idea, using examples and pictures Once your reader has the intuition, she can follow the details (but not vice versa) Present the general case later in the paper Be relentless in working to achieve this goal Everything you write should be in support of it When you re-read and self-critique, your criterion is: Does this material best support the goal of conveying my idea to the reader? Spoon-feed the reader 3
Two Overarching Two Overarching Challenges Challenges Busy reviewers with 15-25 submissions to read Keep their interest! If they start to skim, game over!! Must overcome the general CS reviewing attitude, which is (almost) ??.??????" Length restrictions on conference submissions Human foible: You will use (more than) the space you are allowed Thus, you will always have to cut something at the end Learn to make wise decisions about using the space Do I use page or 1 page to cover this aspect? Do I use 1 or 4 to express this idea? (2 sentences or a full paragraph?) 4
You will always be space constrained You will always be space constrained ACM one-column Proc. ACM format ( acmsmall-conf ) POPL (25 pages + references) OOPSLA (23 pages + references) PLDI (20 pages + references) FSE (18 pages + 4 pages of references) Springer Lecture Notes in Computer Science (LNCS) CAV (18 pages + references + appendices) ETAPS (16-25 pages + references) Sometimes A submission can have appendices (e.g., for proofs) Allowed to submit supplementary material (e.g., for proofs) If accepted, you can buy 2-4 additional pages for the final paper (~$200/page, covered by your advisor s grant) 5
ACM one-column Proc. ACM format POPL (25 pages + references) OOPSLA (23 pages + references) PLDI (20 pages + references) FSE (18 pages + 4 pages of refs.) 7
Springer LNCS CAV (18 pages + references) ETAPS (16-25 pages + references) 8
[SPJ] 9
[SPJ] 10
[SPJ] 11
Structure (conference paper) [SPJ] Structure (conference paper) [SPJ] Title (1000 readers) Abstract (4 sentences, 100 readers) Introduction (1 page, 100 readers) The problem (1 page, 10 readers) My idea (2 pages, 10 readers) The details (5 pages, 3 readers) Related work (1-2 pages, 10 readers) Conclusions and further work (0.5 pages) 12
A structure that works [DD] A structure that works [DD] Abstract (1-2 paragraphs, 1000 readers) Intro (1-2 pages, 100 readers) Main ideas (2-3 pages, 50 readers) Technical meat (4-6 pages, 5 readers) Related work (1-2 pages, 100 readers) 13
A More Complete Structure [Reps] A More Complete Structure [Reps] Abstract 1. Introduction 2. Overview and Problem Statement 3. [Terminology and Notation] 4. Core Algorithm/Idea 5. [Extensions] 6. Experiments 7. [Limitations/Threats to Validity] 8. Related Work 9. [Conclusion] 14
Two Overarching Two Overarching Challenges Challenges Busy reviewers with 15-25 submissions to read Keep their interest! If they start to skim, game over!! Must overcome the general CS reviewing attitude, which is (almost) ??.??????" Length restrictions on conference submissions Human foible: You will use (more than) the space you are allowed Thus, you will always have to cut something at the end Learn to make wise decisions about using the space Do I use page or 1 page to cover this aspect? Do I use 1 or 4 to express this idea? (2 sentences or a full paragraph?) 15
Abstract [Kent Beck via SPJ] Abstract [Kent Beck via SPJ] Four sentences 1. State the problem 2. Say why it s an interesting problem 3. Say what your solution achieves 4. Say what follows from your solution 16
Introduction [SPJ] Introduction [SPJ] Describe the problem State your contributions ... and that is all ONE PAGE! 17
[SPJ] 18
[SPJ] 19
[SPJ] 20
Recommended: 3-5 bullet points 2: hmm, seems a little thin >5: may provide too large an attack surface [Aiken] more contributions more things to criticize [SPJ] 21
[SPJ] 22
[SPJ] 23
Well, maybe . . . If you can do it, great! Sometimes just too awkward, or not natural for the introduction to refer to some section(s) I probably include such a paragraph more often than not [SPJ] 24
[SPJ] 25
Often the no-related-work approach is the right one, but . . . Sometimes it is better to discuss the 1-3 most important pieces of related work The best way to introduce your idea might be to contrast it with some well-known concept Sometimes you need to make sure that the reader understands some essential background on which your work depends Want to ensure that the reviewer does not dismiss your idea as It s just the same as XXX. [SPJ] 26
DD critique of SPJ DD critique of SPJ SPJ s approach to the Introduction eliminates context Start with a concrete example, e.g. Consider this Haskell code If this works, it can be effective, but I find it often doesn t work It assumes reader already knows context 27
Significance The CGI Model for an Abstract/Intro [DD] The CGI Model for an Abstract/Intro [DD] Why is the work important? Might be part of C, G, or I, or entirely separate Context Set the stage, motivate the general topic Gap Explain your specific problem and why existing work does not adequately solve it Innovation State what you ve done that is new, and explain how it helps fill the gap Introduction An expanded version of the abstract, still using CGI 28
Abstract [Kent Beck via SPJ] Abstract [Kent Beck via SPJ] Four sentences 1. State the problem (C) 2. Say why it s an interesting problem (G) 3. Say what your solution achieves (I) 4. Say what follows from your solution (S) 29
Introduction (Rev. Eng. from Introduction (Rev. Eng. from Zeldovich Zeldovich) ) Elevator story -- high-level statement of the problem The problem in more technical terms Conventional wisdom: sketch of previous techniques and their shortcomings Describe the solution to the problem as a black box, so that it is clear what your solution offers over previous work Technical challenges to obtaining a solution (e.g., 1 paragraph for each) State how you solve the challenges (at most a few paragraphs) Experimental highlights Contributions Organization of the paper C G I 30
Zeldovich Example [SOSP09] I: Solution as black box I: The solutions offered C: The problem I: Experimental highlights I: Challenges Contributions G: Conventional wisdom Organization 31
A More Complete Structure [Reps] A More Complete Structure [Reps] Abstract 1. Introduction 2. Overview and Problem Statement 3. [Terminology and Notation] 4. Core Algorithm/Idea 5. [Extensions] 6. Experiments 7. [Limitations/Threats to Validity] 8. Related Work 9. [Conclusion] 33
Overview Overview SPJ: Conveying the intuition is primary; do it using examples and pictures Even if the reader skips the details, she still takes away something valuable DD: 1. 2. Forces you to have a takeaway Many readers only care about the takeaway, not the technical details For those who want the technical details, the intuition is still useful as scaffolding 3. The Gopan Principle: By the time the reader reaches the end of the Overview, they should feel like they don t need to read the rest of the paper. Denis Gopan, c. 2006 34
Problem Statement Problem Statement Somesh Jha: The problem statement is not The goal is to produce the kind of answers that my algorithm produces The problem should be stated in a way that is independent of any particular technique or method Specify what is to be obtained, not how it is obtained Declarative, not imperative 36
A More Complete Structure [Reps] A More Complete Structure [Reps] Abstract 1. Introduction 2. Overview and Problem Statement 3. [Terminology and Notation] 4. Core Algorithm/Idea 5. [Extensions] 6. Experiments 7. [Limitations/Threats to Validity] 8. Related Work 9. [Conclusion] Pseudo-code for algorithms Theorems [and proofs or proof-sketches] The technical meat is often the easiest part to write It is similar to the formalization that you find in a homework problem & solution Gives a solution to the problem, including 40
A More Complete Structure [Reps] A More Complete Structure [Reps] Abstract 1. Introduction 2. Overview and Problem Statement 3. [Terminology and Notation] 4. Core Algorithm/Idea 5. [Extensions] 6. Experiments 7. [Limitations/Threats to Validity] 8. Related Work 9. [Conclusion] 41
Experiments Experiments Experimental questions What hypothesis are you trying to test? The experiments were designed to answer the following questions: 1. 2. 3. Experimental setup CPU, speed, amount of memory; OS version Characteristics of the test suite Experimental results Table, graph, scatter-plot of X vs. Y, etc. Don t forget to say what the experiments showed about the experimental questions 42
Experiments 43
Our experiments showed that the answers to the questions were Experiments Our experiments were designed to answer the following questions: 1. 2. 3. 44
Experiments: SIGPLAN Empirical Evaluation Experiments: SIGPLAN Empirical Evaluation Checklist Checklist This checklist is organized into seven categories, each with associated examples for illustration. Categories are meant to be comprehensive, applying to the breadth of possible empirical evaluations. Examples in each category highlight specific, important areas in which best practice was frequently not followed. These are meant to be useful and illustrative, but they are neither comprehensive nor applicable to every evaluation. The goal of the checklist is to help authors produce stronger scholarship, and to help reviewers evaluate such scholarship more consistently. Importantly, the checklist is meant to support informed judgment, not supplant it. The committee s hope is that this checklist can put all members of the community literally on the same page. Webpage PDF 45
Limitations/Threats to Validity Limitations/Threats to Validity [Software [Software- -Engineering papers] Engineering papers] Construct validity Are the parameters studied in the experiments relevant to the research questions posed? Internal validity Is there selection bias in the examples chosen for study? External validity Do the conclusions hold in other scenarios Ideally: explain how you addressed or controlled for these issues 46
Example 47
Related Work Related Work Don t just list related papers Honest comparison: what is the delta? Be generous with credit Complimentary + complementary 48
Related Work Related Comparison 1 Comparison 2 Comparison 3 49
[SPJ] 50
[SPJ] 51
