SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language

Slide Note

SmartSynth is a cutting-edge project by Vu Le, Sumit Gulwani, and Zhendong Su that aims to automate smartphone tasks by converting natural language instructions into executable scripts. The project involves programming in Java and C#, utilizing tools like Tasker and App Inventor to create automation scripts based on user-defined conditions and actions. The system processes natural language input, generates corresponding programming scripts, and executes tasks such as reading SMS messages while driving. SmartSynth's approach consists of defining components, relationships, and conversions to streamline the scripting process effectively.

ptri Follow

Uploaded on Oct 08, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript

SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language Vu Le (UC Davis) Sumit Gulwani (MSR Redmond) Zhendong Su (UC Davis)

motivation

programming http://xedicorporation.com/images/java_icon.png http://jtmgroup.in/images/csharp_logo.png http://www.trademarkia.com/services/logo.ashx?sid=77108817 Tasker App Inventor

natural language When I receive a new SMS, if the phone is connected to my car s bluetooth, it reads the message content and replies the sender I m driving.

script when when (number, content) := MessageReceived() if if (IsConnectedToBTDevice(Car_BT) then then Speak(content); SendMessage(number, "I'm driving");

SmartScript Script P P Parameter I I Event E E Conversions T T Condition C C Body M M Conversion F F Predicate Statement Stmt Atomic Stmt. S S ::= Action A A Argument a a ::= ::= ::= ::= ::= ::= ::= ::= I E T C M input input (i1, , in) | (r1, , rn) := when F1; ; Fn; if if ( 1 n) then Stmt1; ; Stmtn; x := Convert(a) Predicate (a1, , an) S | foreach foreach x in in a do A | F (r1, , rn) := Action (a1, , an) x | i | r | l when Event() | then | Stmt ::= do S1; ; Sn; od od ::= ::=

approach Script = Components + Relations

approach Script = Components Components + Relations

approach Script = Components + Relations Relations

approach Script = Components + + Relations

definition o Component Component API: MessageReceived Entity Literal: I m driving API return value: MessageReceived.TextO o Relation Relation = <Entity, API parameter> E.g., < I m driving , SendMessage.TextI >

component discovery Description Description Component Component When I receive a new SMS MessageReceived MessageReceived IsConnectedToBTDevice IsConnectedToBTDevice if the phone is connected to my car s bluetooth reads the message content replies the sender I m driving Car_BT Car_BT Speak Speak MessageReceived.Text MessageReceived.TextO O SendMessage SendMessage MessageReceived.Number MessageReceived.NumberO O "I'm driving" "I'm driving"

mapping features o Regular expressions o Bag of words o Phrase length o Punctuation o Parse tree (Stanford NLP parser)

relation discovery o Rule-based relation detection Relative locations of APIs and entities Return Value Return Value or Literal Car_BT Car_BT MessageReceived.Text MessageReceived.TextO O MessageReceived.Number MessageReceived.NumberO O I m driving I m driving or Literal API Parameter API Parameter IsConnectedToBTDevice.Text IsConnectedToBTDevice.TextI Speak.Text Speak.TextI I SendMessage.Number SendMessage.NumberI I SendMessage.Text SendMessage.TextI I

script generation when when (number, content) := MessageReceived() if if (IsConnectedToBTDevice(Car_BT) then then Speak(content); SendMessage(number, "I'm driving");

ambiguity Description Description Possible Component Mappings Possible Component Mappings MessageReceived IsConnectedToBTDevice IsConnectedToBTDevice IsConnectedToWifiNetwork IsConnectedToWifiNetwork IsConnectedToDataService IsConnectedToDataService Car_BT Speak MessageReceived.TextO SendMessage SendMessage SendEmail SendEmail "I'm driving" When I receive a new SMS if the phone is connected to if the phone is connected to my car s bluetooth reads the message content replies replies I m driving

incompleteness when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

insight NLP + Program Synthesis

ambiguity { MessagedReceived, IsConnectedToBTDevice, Car_BT, Speak, MessageReceived.TextO, (SendMessage OR SendEmail), "I'm driving } SendMessage SendEmails Synthesizer Synthesizer Script 1 Script 2 Ranking Ranking Script 1 ( Script 1 (SendMessage SendMessage) )

incompleteness o Search for most likely missing relations

q & a o Distinguishing multiple choice questions Question: API parameter Answers: Equally-likely related entities Question Answers What do you want the phone to speak? A. The received message content B. I m driving

architecture User 1 6 Natural Language Description Intended Script 5 2 Natural Language Q&A (Optional) Feedback on Description (Optional) 3 Mapping Components Script Discovery Component Discovery Feedback on Comp. Mapping 4

evaluation o 50 scripts collected from user forums o User study: give NL descriptions from scripts 11 students participated Each student was given 25 problems 640 correct descriptions (725 total)

component mapping [1] Regex + Bags-of-Words [3] Punctuation [2] Phrase length [4] Parse tree Series1 100 90 80 70 60 50 40 30 20 10 0 [1] [1] [2] [1] [2] [3] [1][2][3][4] SmartSynth