SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language

 
SmartSynth
: Synthesizing
Smartphone Automation Scripts
from Natural Language
 
 
Vu Le 
(UC Davis)
Sumit Gulwani (MSR Redmond)
Zhendong Su (UC Davis)
 
motivation
 
programming
 
 
App Inventor
 
Tasker
 
natural language
 
 
When I receive a new SMS, if the phone is
connected to my car’s bluetooth, it reads the
message content and replies the sender “I’m
driving.”
 
script
 
 
when
 (
number
, 
content
) := 
MessageReceived
()
 
if
 (
IsConnectedToBTDevice
(
Car_BT
) 
then
  
Speak
(
content
);
  
SendMessage
(
number
, 
"I'm driving"
);
 
SmartScript
 
Script 
P
 
  
::= 
 
I  E  T  C  M
Parameter 
I
 
 
::= 
 
input 
(i
1
, …, i
n
)  |  
ε
Event 
E
 
  
::= 
 
(
r
1
, …,
 r
n
) := 
when
 
Event
()  |  
ε
Conversions 
T
 
 
::= 
 
F
1
; …; F
n
;
Condition 
C
 
 
::= 
 
if
 (
Π
1
 ∧ … ∧ 
Π
n
) 
then 
 |  
ε
Body 
M
 
  
::= 
 
Stmt
1
; … ; Stmt
n
;
Conversion 
F
 
 
::= 
 
x := Convert(a)
Predicate 
Π
 
 
::= 
 
Predicate
 (a
1
, …, a
n
)
Statement 
Stmt
 
 
::= 
 
S  |  
foreach
 x 
in
 a 
do
 S1; … ; Sn; 
od
Atomic Stmt. 
S
 
::= 
 
A  |  F
Action 
A
 
 
::= 
 
(
r
1
, …, 
r
n
) := 
Action
 (a
1
, …, a
n
)
Argument 
a
 
 
::= 
 
x  |  i  |  
r
  |  
l
 
approach
 
 
 
Script = Components + Relations
 
approach
 
 
 
Script = 
Components
 + Relations
 
approach
 
 
 
Script = Components + 
Relations
 
approach
 
 
 
Script = Components 
+
 Relations
 
definition
 
o
Component
API: MessageReceived
Entity
Literal: “I’m driving”
API return value: MessageReceived.Text
O
o
Relation
 = <Entity, API parameter>
E.g., < “I’m driving”, SendMessage.Text
I 
>
 
 
component discovery
 
mapping features
 
o
Regular expressions
o
Bag of words
o
Phrase length
o
Punctuation
o
Parse tree (Stanford NLP parser)
 
relation discovery
 
o
Rule-based relation detection
Relative locations of APIs and entities
 
script generation
 
when
 (
number
, 
content
) := 
MessageReceived
()
 
if
 (
IsConnectedToBTDevice
(
Car_BT
) 
then
  
Speak
(
content
);
  
SendMessage
(
number
, 
"I'm driving"
);
 
ambiguity
 
incompleteness
 
when
 (number, content) := 
MessageReceived
()
 
if
 (
IsConnectedToBTDevice
(Car_BT) 
then
  
Speak
(content);
  
SendMessage
(number, "I'm driving");
 
insight
 
 
NLP + Program Synthesis
 
ambiguity
 
{
 MessagedReceived, 
IsConnectedToBTDevice, Car_BT, Speak,
MessageReceived.Text
O, 
(SendMessage OR SendEmail), 
"I'm driving“ 
}
Ranking
 
Script 1 (SendMessage)
 
SendMessage
 
SendEmails
 
Script 1
 
Script 2
Synthesizer
 
incompleteness
 
o
Search for most likely missing relations
 
q & a
 
o
Distinguishing multiple choice questions
Question
: API parameter
Answers
: Equally-likely related entities
 
What do you want the phone to speak?
A.
The received message content
B.
“I’m driving”
 
architecture
 
evaluation
 
o
50 scripts collected from user forums
o
User study: give NL descriptions from scripts
11 students participated
Each student was given 25 problems
640 correct descriptions (725 total)
 
 
component mapping
 
relation detection
 
completing ratio
 
synthesis time
 
related work
 
o
General purpose programming using natural
language
o
Natural language interfaces
o
Specification extraction from natural
language
 
thank you
 
 
 
Script = Components + Relations
 
NLP
 
Program Synthesis
 
Identify
 
Detect
 
Resolve
 
Complete
 
v.s. Siri
 
Conversation
 
Single phone API
 
Composition
 of APIs
 
Siri
 
SmartSynth
 
v.s. Tasker
Slide Note

Hello.

I’m Vu Le from UC Davis.

Today I will talk about SmartSynth, a system that allows users synthesize smartphone automation scripts from natural language descriptions.

Embed
Share

SmartSynth is a cutting-edge project by Vu Le, Sumit Gulwani, and Zhendong Su that aims to automate smartphone tasks by converting natural language instructions into executable scripts. The project involves programming in Java and C#, utilizing tools like Tasker and App Inventor to create automation scripts based on user-defined conditions and actions. The system processes natural language input, generates corresponding programming scripts, and executes tasks such as reading SMS messages while driving. SmartSynth's approach consists of defining components, relationships, and conversions to streamline the scripting process effectively.


Uploaded on Oct 08, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language Vu Le (UC Davis) Sumit Gulwani (MSR Redmond) Zhendong Su (UC Davis)

  2. motivation

  3. programming http://xedicorporation.com/images/java_icon.png http://jtmgroup.in/images/csharp_logo.png http://www.trademarkia.com/services/logo.ashx?sid=77108817 Tasker App Inventor

  4. natural language When I receive a new SMS, if the phone is connected to my car s bluetooth, it reads the message content and replies the sender I m driving.

  5. script when when (number, content) := MessageReceived() if if (IsConnectedToBTDevice(Car_BT) then then Speak(content); SendMessage(number, "I'm driving");

  6. SmartScript Script P P Parameter I I Event E E Conversions T T Condition C C Body M M Conversion F F Predicate Statement Stmt Atomic Stmt. S S ::= Action A A Argument a a ::= ::= ::= ::= ::= ::= ::= ::= I E T C M input input (i1, , in) | (r1, , rn) := when F1; ; Fn; if if ( 1 n) then Stmt1; ; Stmtn; x := Convert(a) Predicate (a1, , an) S | foreach foreach x in in a do A | F (r1, , rn) := Action (a1, , an) x | i | r | l when Event() | then | Stmt ::= do S1; ; Sn; od od ::= ::=

  7. approach Script = Components + Relations

  8. approach Script = Components Components + Relations

  9. approach Script = Components + Relations Relations

  10. approach Script = Components + + Relations

  11. definition o Component Component API: MessageReceived Entity Literal: I m driving API return value: MessageReceived.TextO o Relation Relation = <Entity, API parameter> E.g., < I m driving , SendMessage.TextI >

  12. component discovery Description Description Component Component When I receive a new SMS MessageReceived MessageReceived IsConnectedToBTDevice IsConnectedToBTDevice if the phone is connected to my car s bluetooth reads the message content replies the sender I m driving Car_BT Car_BT Speak Speak MessageReceived.Text MessageReceived.TextO O SendMessage SendMessage MessageReceived.Number MessageReceived.NumberO O "I'm driving" "I'm driving"

  13. mapping features o Regular expressions o Bag of words o Phrase length o Punctuation o Parse tree (Stanford NLP parser)

  14. relation discovery o Rule-based relation detection Relative locations of APIs and entities Return Value Return Value or Literal Car_BT Car_BT MessageReceived.Text MessageReceived.TextO O MessageReceived.Number MessageReceived.NumberO O I m driving I m driving or Literal API Parameter API Parameter IsConnectedToBTDevice.Text IsConnectedToBTDevice.TextI Speak.Text Speak.TextI I SendMessage.Number SendMessage.NumberI I SendMessage.Text SendMessage.TextI I

  15. script generation when when (number, content) := MessageReceived() if if (IsConnectedToBTDevice(Car_BT) then then Speak(content); SendMessage(number, "I'm driving");

  16. ambiguity Description Description Possible Component Mappings Possible Component Mappings MessageReceived IsConnectedToBTDevice IsConnectedToBTDevice IsConnectedToWifiNetwork IsConnectedToWifiNetwork IsConnectedToDataService IsConnectedToDataService Car_BT Speak MessageReceived.TextO SendMessage SendMessage SendEmail SendEmail "I'm driving" When I receive a new SMS if the phone is connected to if the phone is connected to my car s bluetooth reads the message content replies replies I m driving

  17. incompleteness when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

  18. insight NLP + Program Synthesis

  19. ambiguity { MessagedReceived, IsConnectedToBTDevice, Car_BT, Speak, MessageReceived.TextO, (SendMessage OR SendEmail), "I'm driving } SendMessage SendEmails Synthesizer Synthesizer Script 1 Script 2 Ranking Ranking Script 1 ( Script 1 (SendMessage SendMessage) )

  20. incompleteness o Search for most likely missing relations

  21. q & a o Distinguishing multiple choice questions Question: API parameter Answers: Equally-likely related entities Question Answers What do you want the phone to speak? A. The received message content B. I m driving

  22. architecture User 1 6 Natural Language Description Intended Script 5 2 Natural Language Q&A (Optional) Feedback on Description (Optional) 3 Mapping Components Script Discovery Component Discovery Feedback on Comp. Mapping 4

  23. evaluation o 50 scripts collected from user forums o User study: give NL descriptions from scripts 11 students participated Each student was given 25 problems 640 correct descriptions (725 total)

  24. component mapping [1] Regex + Bags-of-Words [3] Punctuation [2] Phrase length [4] Parse tree Series1 100 90 80 70 60 50 40 30 20 10 0 [1] [1] [2] [1] [2] [3] [1][2][3][4] SmartSynth

  25. relation detection 6 5 # Detected Relations 4 3 2 1 0 1 2 3 4 7 8 # Relations

  26. completing ratio Detected by NLP Engine Completed by Synthesis Engine 250 200 # Descriptions 150 100 50 0 0 1 2 3 4 7 8 # Relations

  27. synthesis time Parsing time Total time 7 6 5 Time (s) 4 3 2 1 0 2 3 4 5 6 7 8 10 11 12 # Components

  28. related work o General purpose programming using natural language o Natural language interfaces o Specification extraction from natural language

  29. thank you NLP NLP Identify Identify Detect Detect Script = Components + Relations Resolve Resolve Complete Complete Program Synthesis

  30. v.s. Siri Single phone API Composition of APIs Conversation SmartSynth Siri

  31. v.s. Tasker Productivity Tasker SmartSynth Time

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#