Incremental Neural Coreference Resolution: Constant Memory Approach

Slide Note
Embed
Share

This research delves into Incremental Neural Coreference Resolution using a Limited-memory algorithm for efficient processing while addressing memory constraints. It explores techniques such as neural components and explicit entity representations, making advancements in resolving coreference in long documents without overwhelming GPU memory. Discover the motivation and solutions provided for handling large documents efficiently.

elio
elio Follow

Uploaded on Aug 28, 2024 | 2 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

  1. Incremental Neural Coreference Resolution in Constant Memory Patrick Xia, Jo o Sedoc, Benjamin Van Durme

  2. Outline 1. Background and Motivation 2. Algorithm and Model 3. Experiments and Results

  3. Background Span Detection Mention Pair Scoring (Lee et al., 2017)

  4. Background Span Detection Mention Pair Scoring (Lee et al., 2017)

  5. Extensions Higher-order resolution: re-score against cluster average (Lee et al., 2018) ELMo/BERT/SpanBERT: Fine-tune pretrained encoders (Joshi et al., 2019) Machine reading comprehension scorer (Wu et al., 2020)

  6. Motivation For long documents (roughly >3000 tokens), GPUs run out of memory The encoder is not always the bottleneck Some documents (books) exceed 100K tokens Model Loading Encoder final layer + concat Pairwise scoring Memory Profile for a long document with model by Joshi et al., 2019

  7. Some solutions Some fixes: Sparse Transformer Sequential pairwise scoring but they do not resolve: Span ranking is linear in document size All spans are needed in decoding Even sparse Transformers need (document size)

  8. Outline 1. Background and Motivation 2. Algorithm and Model 3. Experiments and Results

  9. Approach Limited-memory incremental algorithm for coreference resolution (Webster and Curran, 2014) Similar to shift-reduce algorithms Neural components + explicit entity representations

  10. Entity List Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  11. Entity List Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  12. Entity List Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  13. Entity List Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  14. Entity List {Christmas} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  15. Entity List {Christmas} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  16. Entity List {Christmas, Christmas} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  17. Entity List {Christmas, Christmas} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  18. Entity List {Christmas, Christmas}, {any presents} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  19. Entity List {Christmas, Christmas}, {any presents} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  20. Entity List {Christmas, Christmas}, {any presents}, {Jo} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  21. Entity List {Christmas, Christmas}, {any presents}, {Jo} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  22. Entity List {Christmas, Christmas}, {any presents}, {Jo}, {the rug} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  23. Entity List {Christmas, Christmas}, {any presents}, {Jo}, {the rug} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  24. Entity List {Christmas, Christmas}, {any presents}, {Jo}, {the rug} Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  25. Entity List {Christmas, Christmas}, {Jo}, Document "Christmas won't be Christmas without any presents," grumbled Jo, lying on the rug. "It 's so dreadful to be poor!" sighed Meg, looking down at her old dress.

  26. Implementation Encoder: SpanBERT Spans: top-k spans based on learned scorer PairScore: FFNN(m, e) Update: learned average of spans Evict: all old entities

  27. Outline 1. Background and Motivation 2. Algorithm and Model 3. Experiments and Results

  28. Experiments OntoNotes 5.0 (English) Evaluated with average F1 (MUC, B3, and CEAF 4) Goal: compare performance between incremental algorithm vs. full- document model

  29. Results 79.6 F1 79.4 F1 Virtually no loss in performance Constant space implementation at inference (2GB) and training (<10GB) Our algorithm has O(1) space complexity A fixed-sized set of entities kept across time

  30. Entity Analysis Coreference is well-suited for (online) clustering: coreferent mentions are close in embedding space

  31. Conclusions Constant memory algorithm + model for coreference resolution Can be applied to future SOTA models Detailed analysis of document and segment lengths in paper! Thanks! Code and models at github.com/pitrack/incremental-coref

Related


Automatic Incremental View Maintenance in DBSP: A Comprehensive Overview

Automatic Incremental View Maintenance in DBSP: A Comprehensive Overview

adine
Understanding Memory Allocation in Operating Systems

Understanding Memory Allocation in Operating Systems

mae
Incremental Borrowing Cost Analysis in Mortgage Decision Making

Incremental Borrowing Cost Analysis in Mortgage Decision Making

dagger
Understanding Memory Organization in Computers

Understanding Memory Organization in Computers

capri
Understanding Memory Organization in Computers

Understanding Memory Organization in Computers

manisa
Understanding Cache and Virtual Memory in Computer Systems

Understanding Cache and Virtual Memory in Computer Systems

katy
Understanding Artificial Neural Networks From Scratch

Understanding Artificial Neural Networks From Scratch

inara
Dynamic Memory Allocation in Computer Systems: An Overview

Dynamic Memory Allocation in Computer Systems: An Overview

kashton
Understanding Memory Management in Operating Systems

Understanding Memory Management in Operating Systems

asriel
Exploring Neural Quantum States and Symmetries in Quantum Mechanics

Exploring Neural Quantum States and Symmetries in Quantum Mechanics

adelinda
Introduction to Neural Networks in IBM SPSS Modeler 14.2

Introduction to Neural Networks in IBM SPSS Modeler 14.2

anouk
Understanding Memory Management and Swapping Techniques

Understanding Memory Management and Swapping Techniques

alijah
Incremental Development Scheme: Khuda-ki-Basti

Incremental Development Scheme: Khuda-ki-Basti

tripp
Overcoming Memory Constraints in Deep Neural Network Design

Overcoming Memory Constraints in Deep Neural Network Design

ariy_9
Advancing Coreference Resolution: Transfer Model by Patrick Xia and Benjamin Van Durme

Advancing Coreference Resolution: Transfer Model by Patrick Xia and Benjamin Van Durme

kima_580
Russian Anaphora and Coreference Resolution Evaluation

Russian Anaphora and Coreference Resolution Evaluation

rhol
Coreference Resolution System Architecture and Inference Methods

Coreference Resolution System Architecture and Inference Methods

tre_a
Event Coreference Resolution Task and Features Analysis

Event Coreference Resolution Task and Features Analysis

ali_eah
Stream Processing for Incremental Sliding Window Analytics

Stream Processing for Incremental Sliding Window Analytics

ellanor
Understanding Incremental Analysis and Business Decision Making

Understanding Incremental Analysis and Business Decision Making

nicc_439
Tampa Bay Summer Campaign Marketing Analysis

Tampa Bay Summer Campaign Marketing Analysis

dombrowski_j
Understanding Shared Memory, Distributed Memory, and Hybrid Distributed-Shared Memory

Understanding Shared Memory, Distributed Memory, and Hybrid Distributed-Shared Memory

benjami

More Related Content