Language Model and Architecture for RSVP-iconCHAT Workshop
A
B
C
I
W
o
r
k
s
h
o
p
2
0
1
3
:
L
a
n
g
u
a
g
e
M
o
d
e
l
a
n
d
A
r
c
h
i
t
e
c
t
u
r
e
f
o
r
R
S
V
P
-
i
c
o
n
C
H
A
T
K
a
r
l
W
i
e
g
a
n
d
N
o
r
t
h
e
a
s
t
e
r
n
U
n
i
v
e
r
s
i
t
y
B
o
s
t
o
n
,
M
A
U
S
A
January 14, 2013
M
y
B
a
c
k
g
r
o
u
n
d
•
Computer science Ph.D. student
•
Natural language processing (NLP)
•
Artificial intelligence (AI)
•
Applications to augmentative and alternative
communication (AAC)
O
u
t
l
i
n
e
1.
Constraints and approach
2.
Interface and demo
3.
Language model
4.
Current architecture
5.
Application to Unlock
C
o
n
s
t
r
a
i
n
t
s
a
n
d
A
p
p
r
o
a
c
h
Constraints:
1.
Single input signal (P300)
2.
Icon-based AAC
Approach:
1.
Event timer
2.
Semantic frames
S
e
m
a
n
t
i
c
F
r
a
m
e
s
•
Actions are central to messages
(Fillmore, 1976)
•
Verbs have "frames" with semantic roles:
Give (
Agent
,
Object
,
Beneficiary
)
•
WordNet, FrameNet, "Read the Web"
•
Verb-first message construction
(Patel et al, 2004)
•
Any order in RSVP-iconCHAT
I
n
t
e
r
f
a
c
e
D
e
m
o
n
s
t
r
a
t
i
o
n
L
a
n
g
u
a
g
e
M
o
d
e
l
•
predict(role, state): listof([word, probability])
•
Semantic grams
(Wiegand and Patel, 2012)
"
I
like
to
play
chess
with my
brother
."
L
M
T
r
a
i
n
i
n
g
1.
Choose a corpus:
"Blog Authorship
Corpus""Crowdsourced AAC-Like Corpus"
2.
Split sentences and remove stop words
3.
Count sentence lengths
4.
Stem and count sem-grams
L
M
A
l
g
o
r
i
t
h
m
1.
Tag closed vocabulary with possible roles
2.
Select statistics for closed vocabulary
3.
Get words from target role
4.
Generate sem-grams from current roles
5.
Convert sem-gram counts into probabilities
6.
Reorder and return
M
o
r
e
L
M
•
Semantic frames have syntactic forms
•
First-word prediction is based on 1-grams
•
Roles have uniform selection probability
•
How do we detect wrong selections of a
role? Of a word?
C
u
r
r
e
n
t
A
r
c
h
i
t
e
c
t
u
r
e
A
r
c
h
i
t
e
c
t
u
r
e
D
e
t
a
i
l
s
BCI/Host Controller ("The Brain")•
Control loop and signal processing
Presentation Layer (Client)
•
User interaction -- images and keyboard
Language Model (Client)
•
Oval and word prediction
•
Semantic selections to syntactic phrase
R
u
n
t
i
m
e
P
r
o
c
e
s
s
:
L
M
1.
Cache vocabulary statistics
2.
Connect to the host controller
3.
Wait for a request header:
"Oval probabilities" -- None, current_state
"Icon probabilities" -- oval, current_state
"Syntactic utterance" -- current_state
R
u
n
t
i
m
e
P
r
o
c
e
s
s
:
P
r
e
s
e
n
t
a
t
i
o
n
1.
Connect to host controller
2.
Wait for a request header:
"Start event loop" -- [oval/icon, bitcode]
"Pause event loop"
"Stop event loop"
"Made decision" -- [oval/icon, bitcode]
"Reset event loop" -- [oval/icon, bitcode]
R
u
n
t
i
m
e
P
r
o
c
e
s
s
:
H
o
s
t
1.
Initialize gTec hardware
2.
Initialize BCI modules
3.
Receive connections from Client modules
4.
Do:
a.
Query LM for oval probabilities
b.
Reorganize display order of ovals
c.
Send display order to Presentation
d.
Detect P300
e.
Query LM for icon probabilities
f.
Reorganize display order of icons
g.
Send decision to Presentation
h.
Repeat 4a - 4g until user selects Speak...
R
u
n
t
i
m
e
P
r
o
c
e
s
s
:
H
o
s
t
(
c
o
n
t
.
)
i.
Query LM for syntactic utterance
j.
Send utterance to Presentation
k.
Reset Presentation
l.
Go to 4a...
P
r
o
j
e
c
t
M
a
n
a
g
e
m
e
n
t
•
Git repository on BitBucket
•
Task management via Asana
•
Schedules in Google Calendar
•
Meeting notes in Google Drive
•
Code backups, relevant papers, and meeting
board photographs in CSLftp
I
m
p
l
e
m
e
n
t
a
t
i
o
n
D
e
t
a
i
l
s
•
IPC is via TCP/IP packets
•
Shared network packet structure
•
Controller uses Matlab
•
Presentation and LM use Python, Twisted,
and either Pygame+SDL or Pyglet+OpenGL
•
Test Controller uses Python+Kivy+Twisted
A
p
p
l
i
c
a
t
i
o
n
t
o
U
n
l
o
c
k
•
P300 design is different than SSVEP design
•
Semantic frames:
o
Divide sentences
o
Free order construction
o
Semantic to syntactic mapping
•
Semantic grams:
o
Free order prediction
o
Require applicable corpus
•
IPC is nice with a standard packet structure
Thanks to Dr. Rupal Patel, Dr. Deniz
Erdogmus, and the National Science
Foundation (Grant #0914808).
T
h
a
n
k
y
o
u
f
o
r
l
i
s
t
e
n
i
n
g
!
Join the workshop to explore language models and architecture for RSVP-iconCHAT presented by Karl Wiegand from Northeastern University. Learn about semantic frames, interface design, LM training, and more in the field of artificial intelligence and natural language processing.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
ABCI Workshop 2013: Language Model and Architecture for RSVP-iconCHAT Karl Wiegand Northeastern University Boston, MA USA January 14, 2013
My Background Computer science Ph.D. student Natural language processing (NLP) Artificial intelligence (AI) Applications to augmentative and alternative communication (AAC)
Outline 1. Constraints and approach 2. Interface and demo 3. Language model 4. Current architecture 5. Application to Unlock
Constraints and Approach Constraints: 1. Single input signal (P300) 2. Icon-based AAC Approach: 1. Event timer 2. Semantic frames
Semantic Frames Actions are central to messages (Fillmore, 1976) Verbs have "frames" with semantic roles: Give ( Agent, Object, Beneficiary ) WordNet, FrameNet, "Read the Web" Verb-first message construction (Patel et al, 2004) Any order in RSVP-iconCHAT
Language Model predict(role, state): listof([word, probability]) Semantic grams (Wiegand and Patel, 2012) "I like to play chess with my brother." brother, chess brother, i brother, like brother, play chess, i ... brother, chess, i brother, chess, like brother, chess, play chess, i, like chess, i, play ...
LM Training 1. Choose a corpus:"Blog Authorship Corpus""Crowdsourced AAC-Like Corpus" 2. Split sentences and remove stop words 3. Count sentence lengths 4. Stem and count sem-grams
LM Algorithm 1. Tag closed vocabulary with possible roles 2. Select statistics for closed vocabulary 3. Get words from target role 4. Generate sem-grams from current roles 5. Convert sem-gram counts into probabilities 6. Reorder and return
More LM Semantic frames have syntactic forms First-word prediction is based on 1-grams Roles have uniform selection probability How do we detect wrong selections of a role? Of a word?
Architecture Details BCI/Host Controller ("The Brain") Control loop and signal processing Presentation Layer (Client) User interaction -- images and keyboard Language Model (Client) Oval and word prediction Semantic selections to syntactic phrase
Runtime Process: LM 1. Cache vocabulary statistics 2. Connect to the host controller 3. Wait for a request header: "Oval probabilities" -- None, current_state "Icon probabilities" -- oval, current_state "Syntactic utterance" -- current_state
Runtime Process: Presentation 1. Connect to host controller 2. Wait for a request header: "Start event loop" -- [oval/icon, bitcode] "Pause event loop" "Stop event loop" "Made decision" -- [oval/icon, bitcode] "Reset event loop" -- [oval/icon, bitcode]
Runtime Process: Host 1. Initialize gTec hardware 2. Initialize BCI modules 3. Receive connections from Client modules 4. Do: a. Query LM for oval probabilities b. Reorganize display order of ovals c. Send display order to Presentation d. Detect P300 e. Query LM for icon probabilities f. Reorganize display order of icons g. Send decision to Presentation
Runtime Process: Host (cont.) i. Query LM for syntactic utterance j. Send utterance to Presentation k. Reset Presentation l. Go to 4a...
Project Management Git repository on BitBucket Task management via Asana Schedules in Google Calendar Meeting notes in Google Drive Code backups, relevant papers, and meeting board photographs in CSLftp
Implementation Details IPC is via TCP/IP packets Shared network packet structure Controller uses Matlab Presentation and LM use Python, Twisted, and either Pygame+SDL or Pyglet+OpenGL Test Controller uses Python+Kivy+Twisted
Application to Unlock P300 design is different than SSVEP design Semantic frames: o Divide sentences o Free order construction o Semantic to syntactic mapping Semantic grams: o Free order prediction o Require applicable corpus IPC is nice with a standard packet structure
Thank you for listening! Thanks to Dr. Rupal Patel, Dr. Deniz Erdogmus, and the National Science Foundation (Grant #0914808).
