Entity-based Memory Network for Text Comprehension

Entity-based Memory Network
for Text Comprehension
Wang Xun
2016/10/07
Text Comprehension
Open domain
QA
Closed
domain
QA
 
 
feature
Raw text
model
feature
model
feature
model
Unified Feature
Representation
Distributed Representation
Memory Network
Memory Models
LSTM/GRU, neuron level memory
 
Memory Network, sentence level memory
(with attentions)
Entity-based Memory Network
 
States of
entities
(Memories)
Input sentences
Extracted
entities
Current Sentence
Question
Update
states
 
 of
entities
Retrieve related
memories
Generate output
feature
Predict the
answer
Answer
Input Modular
Generation Modular
Output Feature Modular
Response Modular
Entity-based Memory Network
Mary
moved
to
the
bathroom
.
S
Mary
moved
to
the
bathroom
.
Mary
bathroom
S’
Autoencoder
Reconstruct Sentence Using Entities
Pre
-trained
Word
 
Vectors
Entity-based Memory Network
Mary
moved
to
the
bathroom
.
S
Mary
bathroom
Pre
-trained
Word
 
Vectors
Entity-based Memory Network
Entity states
Scores -- 1
st
 iteration
Scores – 2
nd
 iteration
Scores – m
th
 iteration
Question
Output feature vector
Update
Update
Update
Entity-based Memory Network
 
Full supervision
Weak supervision
Experiments
bAbI
20 topics
Toy data, text advanture
Path finding/position reasoning
.36/.65 (MNN)
.35/.60 (DMNN)
.53/.67 (EMNN)
.53/.67 (EMNN)
 
Large Movie Review Dataset
50,000 reviews from IMDB, about 30 reviews per movie.
Scores from 1(---) to 10 (+++)
“FUTZ is the only show preserved from the experimental theatre
movement in New York in the 1960s (the origins of Off Off Broadway).
Though it‘s not for everyone, it is a genuinely brilliant, darkly funny,
even more often deeply disturbing tale about love, sex, personal
liberty, and revenge, a serious morality tale even more relevant now in
a time when Congress wants to outlaw gay marriage by trashing our
Constitution. The story is not about being gay, though -- it’s about love
and sex that don‘t conform to social norms and 
…”
(-):1-4 / (+):7-10
Q: What does the reviewer think about the movie?
A: Positive/Negative.
97.2 
97.2 
(previous work: 89/93.4/95)
 
Machine Comprehension Test
500 stories and 2000 questions
Most
 
options
 
contain
 
one
 
or
 
two
 
words.
No
 
world
 
knowledge
 
is
 
required.
multiple choices questions as QA
options
 
as
 
related
 
entities/full
 
supervision
683 questions Acc.=.73
+
  (.60)
 
Once upon a time there was a princess who lived in a high tower and she was not allowed to
 
leave
because of her mean mother. One day she chose to leave but her mother would not let
 
her. The
princess climbed out the window of the high tower and climbed down the south wall
 
when her
mother was sleeping. She wandered out a good ways. Finally she went into the
 
forest where there
are no electric poles but where there are some caves.  
1: Where did the princess wander to after escaping?
 A) 
Mountain
*B) 
Forest
 C) 
Cave
 D) 
Castle
2: Who escaped from the tower?
 A) 
Mother
*B) 
Princess
 C) 
Man
 D) 
John
 
bAbI
Acc of >95 for 18 of 20 topics
Toy data, text adventure game
,
 
not
 
convincing
 
Large Movie Review Dataset
Closed
 
domain
Machin
e
 
Comprehension
 
Test
Children
 
Stories
Open
 
domain
 
 
Slide Note
Embed
Share

Entity-based Memory Network is a model designed for text comprehension, focusing on question-answering tasks in both open and closed domain QA. It incorporates features like distributed representation, feature modeling, and memory models at various levels to generate output features, predict answers, and update entity states. The network utilizes pre-trained word vectors and modular components for sentence reconstruction and entity-related tasks through iterations, aiming to enhance comprehension and information retrieval in natural language processing.

  • Memory Network
  • Text Comprehension
  • Natural Language Processing
  • Entity-based
  • Question-Answering

Uploaded on Aug 21, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Entity-based Memory Network for Text Comprehension Wang Xun 2016/10/07

  2. Text Comprehension Text Comprehension The ability to read text, process it, and understand its meaning Question answering ?: ????,???????? {?????} Open domain QA / Closed domain QA Formulating NLP tasks as a QA problem POS tagging Q: What are the parts of speech? Co-reference resolution Q: What does XX refer to? Xxx xxx Closed domain QA Open domain QA Closed domain QA

  3. answer Memory Network model Distributed Representation answer Unified Feature Representation feature model feature feature model answer Raw text

  4. Memory Models LSTM/GRU, neuron level memory

  5. Memory Network, sentence level memory (with attentions)

  6. Entity-based Memory Network

  7. Entity-based Memory Network Generate output feature Predict the answer Extracted entities Answer Update states of entities States of entities (Memories) Response Modular Question Input sentences Generation Modular Retrieve related memories Current Sentence Output Feature Modular Input Modular

  8. Entity-based Memory Network Autoencoder Mary moved to the bathroom . Mary moved to the bathroom . S S Reconstruct Sentence Using Entities Pre-trained Word Vectors Mary bathroom ?:? = ??????????? ? While (1): ? = ? ??? ? , ???? ? = ?(??? ? , ??? ???? ) ???????? ||? ? ||2 Return [ Mary , bathroom ],f

  9. Entity-based Memory Network Mary moved to the bathroom . S Pre-trained Word Vectors Mary bathroom

  10. Entity-based Memory Network ?(??)?= ?( ?,??) Scores mth iteration Scores -- 1st iteration Scores 2nd iteration Entity states ? = ( ,?) Output feature vector Update Update Update = ? = (???( ,? ?)) Question

  11. Full supervision Weak supervision

  12. Experiments bAbI 20 topics Toy data, text advanture Path finding/position reasoning .36/.65 (MNN) .35/.60 (DMNN) .53/.67 (EMNN)

  13. Large Movie Review Dataset 50,000 reviews from IMDB, about 30 reviews per movie. Scores from 1(---) to 10 (+++) FUTZ is the only show preserved from the experimental theatre movement in New York in the 1960s (the origins of Off Off Broadway). Though it s not for everyone, it is a genuinely brilliant, darkly funny, even more often deeply disturbing tale about love, sex, personal liberty, and revenge, a serious morality tale even more relevant now in a time when Congress wants to outlaw gay marriage by trashing our Constitution. The story is not about being gay, though -- it s about love and sex that don t conform to social norms and (-):1-4 / (+):7-10 Q: What does the reviewer think about the movie? A: Positive/Negative. 97.2 (previous work: 89/93.4/95)

  14. Machine Comprehension Test 500 stories and 2000 questions Most options contain one or two words. No world knowledge is required. multiple choices questions as QA options as related entities/full supervision 683 questions Acc.=.73+ (.60)

  15. Once upon a time there was a princess who lived in a high tower and she was not allowed to leave because of her mean mother. One day she chose to leave but her mother would not let her. The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there are no electric poles but where there are some caves. 1: Where did the princess wander to after escaping? A) Mountain *B) Forest C) Cave D) Castle 2: Who escaped from the tower? A) Mother *B) Princess C) Man D) John

  16. bAbI Acc of >95 for 18 of 20 topics Toy data, text adventure game, not convincing Large Movie Review Dataset Closed domain Machine Comprehension Test Children Stories Open domain

More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#