Textual Entailment in Natural Language Processing
Recognizing Partial Textual Entailment
Torsten Zesch
Iryna Gurevych
Ubiquitous Knowledge Processing Lab
Technische Universität Darmstadt
Omer Levy
Ido Dagan
Natural Language Processing Lab
Bar-Ilan University
1
Textual Entailment
2
T: muscles move bones
3
T: muscles move bones
H: muscles generate movement
4
T: muscles move bones
H: muscles generate movement
5
T: muscles move bones
H: the muscles’ main job is to move bones
?
6
T: muscles move bones
H: the muscles’ main job is to move bones
7
Complete
Textual Entailment
8
Complete
Textual Entailment
9
Complete
Textual Entailment
.
Partial
.
10
Complete
Textual Entailment
.
Partial
.
(Nielsen et al, 2009)
11
T: muscles move bones
H: the muscles’ main job is to move bones
12
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the muscles’ main job is to move bones
13
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the muscles’ main job is to move bones
14
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the muscles’ main job is to move bones
15
Student Response Analysis
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the muscles’ main job is to move bones
16
Student Response Analysis
Summarization
Information Extraction
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the muscles’ main job is to move bones
Facets
17
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the
muscles’ main job
is to move bones
(main job, muscles)
(muscles, move) (move, bones)
Facets
18
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the
muscles
’ main job is to
move
bones
(main job, muscles)
(muscles, move)
(move, bones)
Facets
19
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved)
H: the muscles’ main job is to
move bones
(main job, muscles) (muscles, move)
(move, bones)
Facets
20
Facet: a pair of terms
and their implied semantic relation.
21
Facet: a pair of
terms
and their implied semantic relation.
(move, bones)
22
Facet: a pair of
terms
and their implied semantic
relation
.
(move, bones)
23
Recognizing
Faceted
Entailment
24
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing
Faceted
Entailment
25
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing
Faceted
Entailment
26
T: muscles move bones
H: the
muscles
’ main job is to
move
bones
Recognizing
Faceted
Entailment
27
T: muscles move bones
H: the
muscles
’ main job is to
move
bones
Recognizing
Faceted
Entailment
28
What did we do?
29
Approach
: Leverage
Existing Methods
30
Approach: Leverage Existing Methods
31
Approach: Leverage Existing Methods
32
Approach: Leverage Existing Methods
33
Approach: Leverage Existing Methods
34
Approach: Leverage Existing Methods
35
Approach: Leverage Existing Methods
36
Bar-Ilan University Textual Entailment Engine
(Stern and Dagan, 2011)
37
BIUTEE
38
BIUTEE
39
T
H
BIUTEE
40
T H
BIUTEE
41
T H
42
43
44
Facets to Dependencies
pneumonia
is
by
45
treated
penicillin
Facets to Dependencies
penicillin
pneumonia
is
by
46
treated
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR Lexical Inference
Exact Match OR Syntactic Inference
Lexical Inference
Exact Match OR AND
Syntactic Inference
47
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR Lexical Inference
Exact Match OR Syntactic Inference
Lexical Inference
Exact Match OR AND
Syntactic Inference
48
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR WordNet
Exact Match OR BIUTEE
Lexical Inference
Exact Match OR AND
Syntactic Inference
49
Baseline
BaseLex
BaseSyn
Majority
Decision Mechanisms
Exact Match
Exact Match OR WordNet
Exact Match OR BIUTEE
WordNet
Exact Match OR AND
BIUTEE
50
SemEval 2013 Task 7
51
Student Response Analysis
52
What is the muscles’ main job?
Muscles move bones.
The muscles’ main job is to move bones.
Question:
Student Answer:
Reference Answer:
53
What is the muscles’ main job?
Muscles move bones.
The muscles’ main job is to move bones.
Question:
Student Answer:
Reference Answer:
54
What is the muscles’ main job?
Muscles move bones.
The muscles’ main job is to move bones.
Question:
Student Answer:
Reference Answer:
55
What is the muscles’ main job?
Muscles move bones.
The muscles’ main job is to move bones.
Question:
Text:
Hypothesis:
56
What is the muscles’ main job?
Muscles move bones.
The muscles’ main job is to move bones.
Question:
Text:
Hypothesis:
57
SciEntsBank
(Djikovska et al, 2012)
15 Domains
5,106 Student Responses
16,263 Facets
Dataset
58
SciEntsBank
(Djikovska et al, 2012)
15 Domains
5,106 Student Responses
16,263 Facets
Dataset
59
SciEntsBank
(Djikovska et al, 2012)
15 Domains
5,106 Student Responses
16,263 Facets
Dataset
60
SciEntsBank
(Djikovska et al, 2012)
15 Domains
13,145 Train Facets
16,263 Test Facets
Dataset
61
SciEntsBank
(Djikovska et al, 2012)
15 Domains
13,145 Train Facets
16,263 Test Facets
Dataset
62
Unseen Answers
9%
Unseen Questions
12%
Unseen Domains
79%
Scenarios
63
Unseen Answers
9%
Unseen Questions
12%
Unseen Domains
79%
Scenarios
64
Unseen Answers
9%
Unseen Questions
12%
Unseen Domains
79%
Scenarios
65
Unseen Answers
9%
Unseen Questions
12%
Unseen Domains
79%
Scenarios
66
Results
67
Performance (Weighted F1)
68
Summary
69
Problem:
Need fine-grained entailment
Solution:
Partial Textual Entailment
Approach:
Leverage existing methods
Results:
Significant improvement
70
Problem:
Need fine-grained entailment
Solution:
Partial
Textual Entailment
Approach:
Leverage existing methods
Results:
Significant improvement
71
Problem:
Need fine-grained entailment
Solution:
Partial Textual Entailment
Approach:
Leverage existing methods
Results:
Significant improvement
72
Problem:
Need fine-grained entailment
Solution:
Partial Textual Entailment
Approach:
Leverage existing methods
Results:
Significant improvement
73
Future
:
Automatic Facet Extraction
74
Future:
Automatic
Facet
Extraction
75
Partial Textual Entailment is
interesting
76
Partial Textual Entailment is
useful
77
Partial Textual Entailment is
practical
78
Partial Textual Entailment is
interesting
79
Questions?
80
Good morning. I’m Omer Levy, and I’ll be talking to you today about what we did in the pilot task, from a textual entailment point of view. This is joint work with my advisor, Ido Dagan, and our colleagues Torsten Zesch and Iryna Gurevych from the UKP lab in Darmstadt.
This collection of images showcases various aspects of textual entailment, where one text can entail another. It explores the relationship between statements like "muscles move bones" and "muscles generate movement". Different levels of entailment are depicted, culminating in complete textual entailment examples. The images depict scenarios where the statement "the muscles' main job is to move bones" can be fully or partially entailed by related statements.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Recognizing Partial Textual Entailment Omer Levy Natural Language Processing Lab Bar-Ilan University Ido Dagan Torsten Zesch Ubiquitous Knowledge Processing Lab Technische Universit t Darmstadt Iryna Gurevych 1
T: muscles move bones H: muscles generate movement 4
T: muscles move bones H: muscles generate movement 5
T: muscles move bones ? H: the muscles main job is to move bones 6
T: muscles move bones H: the muscles main job is to move bones 7
Complete Textual Entailment . Partial . 10
Complete Textual Entailment . Partial . (Nielsen et al, 2009) 11
T: muscles move bones H: the muscles main job is to move bones 12
T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones 13
T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones 14
T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones 15
T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones 16
Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones 17
Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones (main job, muscles) (muscles, move) (move, bones) 18
Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones (main job, muscles) (muscles, move) (move, bones) 19
Facets T: muscles move bones (main job of muscles) (muscles move) (bones are moved) H: the muscles main job is to move bones (main job, muscles) (muscles, move) (move, bones) 20
Facet: a pair of terms and their implied semantic relation. 21
(move, bones) Facet: a pair of terms and their implied semantic relation. 22
(move, bones) Facet: a pair of terms and their implied semantic relation. theme is to move bones 23
Recognizing Faceted Entailment T: muscles move bones H: the muscles main job is to move bones 25
Recognizing Faceted Entailment T: muscles move bones H: the muscles main job is to move bones 26
Recognizing Faceted Entailment T: muscles move bones H: the muscles main job is to move bones 27
Recognizing Faceted Entailment T: muscles move bones H: the muscles main job is to move bones 28
Approach: Leverage Existing Methods Exact Match penicillin cures pneumonia (penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillin Lexical-Syntactic Inference P: 96%, R: 30% 32
Approach: Leverage Existing Methods Exact Match penicillin cures pneumonia (penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillin Lexical-Syntactic Inference P: 96%, R: 30% 33
Approach: Leverage Existing Methods Exact Match penicillin cures pneumonia (penicillin, cure) Lexical Inference pneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference WordNet 34
Approach: Leverage Existing Methods Exact Match penicillin cures pneumonia (penicillin, cure) Lexical Inference pneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference WordNet 35
Approach: Leverage Existing Methods Exact Match penicillin cures pneumonia (penicillin, cure) Lexical Inference pneumonia is treated by penicillin (penicillin, cure) Lexical-Syntactic Inference penicillin cures pneumonia pneumonia is treated by penicillin 36
Bar-Ilan University Textual Entailment Engine (Stern and Dagan, 2011) 37
BIUTEE 38
BIUTEE T H cures penicillin pneumonia 39
BIUTEE T H cures treated penicillin pneumonia pneumonia is by penicillin 40
BIUTEE T H cures treated penicillin pneumonia pneumonia is by penicillin 41
cures penicillin pneumonia 42
cures penicillin pneumonia cure treat (lexical) treats penicillin pneumonia 43
cures penicillin pneumonia cure treat (lexical) treats penicillin pneumonia active passive (syntactic) treated pneumonia is by penicillin 44
Facets to Dependencies treated pneumonia is by penicillin 45
Facets to Dependencies treated pneumonia is by penicillin 46
Decision Mechanisms Baseline Exact Match Exact Match OR Lexical Inference BaseLex BaseSyn Exact Match OR Syntactic Inference Lexical Inference Exact Match OR AND Syntactic Inference Majority 47
Decision Mechanisms Baseline Exact Match Exact Match OR Lexical Inference BaseLex BaseSyn Exact Match OR Syntactic Inference Lexical Inference Exact Match OR AND Syntactic Inference Majority 48
Decision Mechanisms Baseline Exact Match Exact Match OR WordNet BaseLex BaseSyn Exact Match OR BIUTEE Lexical Inference Exact Match OR AND Syntactic Inference Majority 49
Decision Mechanisms Baseline Exact Match Exact Match OR WordNet BaseLex BaseSyn Exact Match OR BIUTEE WordNet Exact Match OR AND BIUTEE Majority 50
