Dynamically Switching Between Synergistic Workflows for Crowdsourcing
Dynamically Switching between
Synergistic Workflows for
Crowdsourcing
Christopher H. Lin, Mausam, Daniel S. Weld
University of Washington
AAAI-2012, Toronto, ON, Canada
1
2
3
Translate A
B
4
Translate A
B
5
Translate A
B
6
Translate A
B
7
Is this a face?
Yes
No
8
C
o
n
g
r
e
s
s
m
a
n
T
o
n
y
C
o
e
l
h
o
l
a
n
d
e
d
a
m
i
l
l
i
o
n
-
d
o
l
l
a
r
j
o
b
o
n
W
a
l
l
S
t
r
e
e
t
a
f
t
e
r
r
e
s
i
g
n
i
n
g
o
v
e
r
a
c
o
n
t
r
o
v
e
r
s
i
a
l
j
u
n
k
-
b
o
n
d
i
n
v
e
s
t
m
e
n
t
l
a
s
t
s
u
m
m
e
r
.
NER Tagging
9
C
o
n
g
r
e
s
s
m
a
n
T
o
n
y
C
o
e
l
h
o
l
a
n
d
e
d
a
m
i
l
l
i
o
n
-
d
o
l
l
a
r
j
o
b
o
n
W
a
l
l
S
t
r
e
e
t
a
f
t
e
r
r
e
s
i
g
n
i
n
g
o
v
e
r
a
c
o
n
t
r
o
v
e
r
s
i
a
l
j
u
n
k
-
b
o
n
d
i
n
v
e
s
t
m
e
n
t
l
a
s
t
s
u
m
m
e
r
.
NER Tagging
location
10
www.crowdsourcing.com
location
film
C
o
n
g
r
e
s
s
m
a
n
T
o
n
y
C
o
e
l
h
o
l
a
n
d
e
d
a
m
i
l
l
i
o
n
-
d
o
l
l
a
r
j
o
b
o
n
W
a
l
l
S
t
r
e
e
t
a
f
t
e
r
r
e
s
i
g
n
i
n
g
o
v
e
r
a
c
o
n
t
r
o
v
e
r
s
i
a
l
j
u
n
k
-
b
o
n
d
i
n
v
e
s
t
m
e
n
t
l
a
s
t
s
u
m
m
e
r
.
11
www.crowdsourcing.com
Wall Street
http://en.wikipedia.org/wiki/Wall_Street
W
a
l
l
S
t
r
e
e
t
i
s
t
h
e
f
i
n
a
n
c
i
a
l
d
i
s
t
r
i
c
t
o
f
N
e
w
Y
o
r
k
C
i
t
y
,
n
a
m
e
d
a
f
t
e
r
a
n
d
c
e
n
t
e
r
e
d
o
n
t
h
e
e
i
g
h
t
-
b
l
o
c
k
-
l
o
n
g
s
t
r
e
e
t
r
u
n
n
i
n
g
f
r
o
m
B
r
o
a
d
w
a
y
t
o
S
o
u
t
h
S
t
r
e
e
t
.
…
Wall Street (1987 film)
http://en.wikipedia.org/wiki/Wall_Street _(1987_film)
W
a
l
l
S
t
r
e
e
t
i
s
a
1
9
8
7
A
m
e
r
i
c
a
n
d
r
a
m
a
f
i
l
m
r
e
l
e
a
s
e
d
b
y
2
0
t
h
C
e
n
t
u
r
y
F
o
x
.
I
t
w
a
s
d
i
r
e
c
t
e
d
b
y
O
l
i
v
e
r
S
t
o
n
e
a
n
d
s
t
a
r
s
M
i
c
h
a
e
l
D
o
u
g
l
a
s
,
C
h
a
r
l
i
e
S
h
e
e
n
,
a
n
d
D
a
r
y
l
H
a
n
n
a
h
…
C
o
n
g
r
e
s
s
m
a
n
T
o
n
y
C
o
e
l
h
o
l
a
n
d
e
d
a
m
i
l
l
i
o
n
-
d
o
l
l
a
r
j
o
b
o
n
W
a
l
l
S
t
r
e
e
t
a
f
t
e
r
r
e
s
i
g
n
i
n
g
o
v
e
r
a
c
o
n
t
r
o
v
e
r
s
i
a
l
j
u
n
k
-
b
o
n
d
i
n
v
e
s
t
m
e
n
t
l
a
s
t
s
u
m
m
e
r
.
12
70%
90%
A/B Testing
Accuracy
13
70%
90%
A/B Testing
Accuracy
14
www.crowdsourcing.com
us_county
location
citytown
location
Washington State
Washington D.C
15
www.crowdsourcing.com
us_county
location
citytown
location
Washington State
Washington D.C
16
www.crowdsourcing.com
us_county
location
citytown
location
Washington State
Washington D.C
17
N
Create job using
best workflow
[Dai et al, 2010]
18
N
Create job using
best workflow
[Dai et al, 2010]
19
N
Create job using
best workflow
20
www.crowdsourcing.com
us_county
location
citytown
location
Washington State
Washington D.C
21
www.crowdsourcing.com
Washington
http://en.wikipedia.org/wiki/Washington
Washington, D.C., formally the District of Columbia and commonly
referred to as Washington, “the District,” or simply D.C., is the capital of
the United States….
Washington(state)
http://en.wikipedia.org/wiki/Washington_(state)
Washington () is a state in the Pacific Northwest region of the United
States located north of Oregon, west of Idaho, and south of the
Canadian province of British Columbia, on the coast of the Pacific
Ocean….
Washington State
Washington D.C
22
www.crowdsourcing.com
Washington
http://en.wikipedia.org/wiki/Washington
Washington, D.C., formally the District of Columbia and commonly
referred to as Washington, “the District,” or simply D.C., is the capital of
the United States….
Washington(state)
http://en.wikipedia.org/wiki/Washington_(state)
Washington () is a state in the Pacific Northwest region of the United
States located north of Oregon, west of Idaho, and south of the
Canadian province of British Columbia, on the coast of the Pacific
Ocean….
Washington State
Washington D.C
23
www.crowdsourcing.com
Washington
http://en.wikipedia.org/wiki/Washington
Washington, D.C., formally the District of Columbia and commonly
referred to as Washington, “the District,” or simply D.C., is the capital of
the United States….
Washington(state)
http://en.wikipedia.org/wiki/Washington_(state)
Washington () is a state in the Pacific Northwest region of the United
States located north of Oregon, west of Idaho, and south of the
Canadian province of British Columbia, on the coast of the Pacific
Ocean….
Washington State
Washington D.C
24
N
Create job using
best workflow
[Dai et al, 2010]
Before
25
26
Dynamically
27
States
Actions
Transition Function
Observation Probabilities
28
States
Actions
Transition Function
Observation Probabilities
29
30
2 + 2 = ?
31
2 + 2 = ?
d = 0
32
?
=
33
?
=
d = 1
34
Correct Answer
Workflow Difficulties
35
States
Actions
Transition Function
Observation Probabilities
36
K+1 actions
37
States
Actions
Transition Function
Observation Probabilities
38
States
Actions
Transition Function
Observation Probabilities
39
Difficulty
40
Worker Ability
41
Difficulty
Source:Wikipedia
42
[Figure courtesy Peng Dai]
43
4.0
2.0
1.0
0.5
0.25
44
45
[Dai et al, 2010]
46
AgentHunt beats SingleBestWorkflow
[Dai et al, 2010]
47
AgentHunt beats SingleBestWorkflow
48
AgentHunt beats SingleBestWorkflow
Contributions
•
A probabilistic model for multiple
workflows
•
A decision-theoretic agent that
dynamically switches between
workflows
•
Real experiments that show you
should not throw away your
workflows
49
50
http://cs.washington.edu/homes/chrislin
Repeated Labeling
Romney et al, 1986; Sheng et al, 2008; Raykar et al, 2010;
Whitehill et al, 2009; Dai et al, 2010, 2011; Kamar et al, 2012;
Parameswaran et al, 2010; Karger et al, 2011; Snow et al,
2008;
Complicated Workflows
Little et al, 2009; Bernstein et al, 2010; Liem et al, 2011;
Noronha et al, 2011; Lasecki et al, 2011; Kulkarni et al, 2012;
AI Control of Workflows
Dai et al, 2010,2011; Shahaf et al, 2010; Weld et al, 2011;
Kamar et al, 2012;
51
52
How many people are there
in this military base?
Source:Wikipedia
Christopher H. Lin, Mausam,
and Daniel S. Weld.
“Crowdsourcing Control:
Moving Beyond Multiple Choice.”
In UAI 2012
53
I’m going to tell you about why and how you can dynamically switch between workflows when crowdsourcing
I’m going to tell you about why two workflows is better than one.
This study explores the concept of dynamically switching between synergistic workflows for crowdsourcing, presented at AAAI-2012 in Toronto, Canada. It covers various translation techniques, human and machine involvement, NER tagging examples, and the application of crowdsourcing in determining if an image contains a face. Additionally, it delves into Congressman Tony Coelho's career shift and A/B testing accuracy comparisons.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Dynamically Switching between Synergistic Workflows for Crowdsourcing Christopher H. Lin, Mausam, Daniel S. Weld University of Washington AAAI-2012, Toronto, ON, Canada 1
Translate AB HumanTr A B Choose HumanTr A B 4
Translate AB MachineTr A B ProofRd 5
Translate AB Translate A C Translate C B 6
www.crowdsourcing.com Is this a face? Yes No 7
NER Tagging Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer. 8
NER Tagging Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer. location 9
www.crowdsourcing.com Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer. location film 10
www.crowdsourcing.com Congressman Tony Coelho landed a million - dollar job on Wall Street after resigning over a controversial junk - bond investment last summer. Wall Street http://en.wikipedia.org/wiki/Wall_Street Wall Street is the financial district of New York City, named after and centered on the eight-block-long street running from Broadway to South Street. Wall Street (1987 film) http://en.wikipedia.org/wiki/Wall_Street _(1987_film) Wall Street is a 1987 American drama film released by 20th Century Fox. It was directed by Oliver Stone and stars Michael Douglas, Charlie Sheen, and Daryl Hannah 11
A/B Testing Accuracy 70% 90% 12
A/B Testing Accuracy 70% 90% 13
Washington State Washington D.C www.crowdsourcing.com us_county location citytown location 14
Washington State Washington D.C www.crowdsourcing.com us_county location citytown location 15
Washington State Washington D.C www.crowdsourcing.com us_county location citytown location 16
N Create job using best workflow [Dai et al, 2010] 17
SUBOPTIMAL N Create job using best workflow [Dai et al, 2010] 18
N Create job using best workflow 19
Washington State Washington D.C www.crowdsourcing.com us_county location citytown location 20
Washington State Washington D.C www.crowdsourcing.com Washington http://en.wikipedia.org/wiki/Washington Washington, D.C., formally the District of Columbia and commonly referred to as Washington, the District, or simply D.C., is the capital of the United States . Washington(state) http://en.wikipedia.org/wiki/Washington_(state) Washington () is a state in the Pacific Northwest region of the United States located north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean . 21
Washington State Washington D.C www.crowdsourcing.com Washington http://en.wikipedia.org/wiki/Washington Washington, D.C., formally the District of Columbia and commonly referred to as Washington, the District, or simply D.C., is the capital of the United States . Washington(state) http://en.wikipedia.org/wiki/Washington_(state) Washington () is a state in the Pacific Northwest region of the United States located north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean . 22
Washington State Washington D.C www.crowdsourcing.com Washington http://en.wikipedia.org/wiki/Washington Washington, D.C., formally the District of Columbia and commonly referred to as Washington, the District, or simply D.C., is the capital of the United States . Washington(state) http://en.wikipedia.org/wiki/Washington_(state) Washington () is a state in the Pacific Northwest region of the United States located north of Oregon, west of Idaho, and south of the Canadian province of British Columbia, on the coast of the Pacific Ocean . 23
Before N Create job using best workflow [Dai et al, 2010] 24
AGENTHUNT 25
AGENTHUNT Dynamically 26
POMDP States Actions Transition Function Observation Probabilities 27
POMDP States Actions Transition Function Observation Probabilities 28
www.crowdsourcing.com 2 + 2 = ? 30
www.crowdsourcing.com 2 + 2 = ? d = 0 31
www.crowdsourcing.com ? = 32
www.crowdsourcing.com ? = d = 1 33
Workflow Difficulties Correct Answer 34
POMDP States Actions Transition Function Observation Probabilities 35
K+1 actions 36
POMDP States Actions Transition Function Observation Probabilities 37
POMDP States Actions Transition Function Observation Probabilities 38
Difficulty 40
Worker Ability Difficulty 41
42 Source:Wikipedia
4.0 2.0 1.0 0.5 0.25 [Figure courtesy Peng Dai] 43
AgentHunt beats SingleBestWorkflow [Dai et al, 2010] 46
AgentHunt beats SingleBestWorkflow [Dai et al, 2010] 47
AgentHunt beats SingleBestWorkflow AgentHunt Single Best Workflow 85.85 Average Accuracy (%) 92.45 Average Net Utility -13.36 -18.35 48
Contributions A probabilistic model for multiple workflows A decision-theoretic agent that dynamically switches between workflows Real experiments that show you should not throw away your workflows 49
INTERESTED? http://cs.washington.edu/homes/chrislin 50
