Challenges and Solutions in Mobile App Data Synchronization

Reliable, Consistent, and Efficient
Data Sync for Mobile Apps
 
Younghwan Go*, Nitin Agrawal,
Akshat Aranya, and Cristian Ungureanu
 
NEC Labs. America     KAIST*
Increase in Data-centric Mobile Apps
 
Massive growth in mobile data traffic 
[Cisco VNI Mobile 2014]
24.3 Exabytes per month by 2019
190 Exabytes of mobile traffic generated globally by 2018
= 42 trillion images, 4 trillion video clips
 
2
Difficulty in Building Data-centric Apps
 
Reliability: transparent failure handling
 
 
 
Consistency: concurrent updates, sync atomicity
 
 
 
Efficiency: minimize traffic/battery usage
3
Look!
My data is
corrupted!
010111001
110110111
Mobile App Study on Reliability
 
Study mobile app recovery under failures
Network disruption, local app crash, device power loss
Analyze recovery when failed during write/update
Test 15 apps that use tables and objects
Independent or existing sync services (e.g., Dropbox, Parse, Kinvey)
Test process
4
1.
Activate airplane mode
2.
Manually kill app
3.
Pull the battery out
Current Mobile Apps are not Reliable!
 
Disruption recovery
Loss of data if app/notification closed during disruption
No notification of sync failure
Manual re-sync creates multiple copies of same note
Crash recovery
Partial object created locally without sync
Corrupted object synced and spread to second client
Additional observations
No app correctly recovered from crash at object update
Many apps simply disable object update capability altogether
5
More details of the study can be found in our paper 
Goals of Sync as a Service
 
Reliability
User can always sync to the latest data
User’s update is guaranteed to be synced to server
 
Consistency
Data can always return to a consistent state even after failures
Inter-dependent structured/unstructured data are synced atomically
 
Efficiency
Minimum mobile data traffic is generated for sync/recovery
Device’s overall network radio usage is reduced to save battery
6
Outline
 
Introduction
Mobile app study on reliability
Simba Client Design
Evaluation
Conclusion
 
7
Simba: Data-sync Service for Mobile Apps
 
High-level programming abstraction
CRUD-like interface for easy development
Unify tabular and object data
 
Transparent handling of data syncs and failures
Failure detection & recovery at network disruption and crash
Guarantee atomic sync of tabular and object data
 
Resource frugality with delay-tolerance and coalescing
Delay sync messages to be clustered
Reduce number of network messages & radio usage
8
Writing a Photo App with Simba
 
Create a photo album
 
Register read/write sync
 
Add a new photo
 
 
Retrieve stored photo
9
registerReadSync(“album”,600,0,3G);   
// period=10min, pref=3G
registerWriteSync(“album”,300,0,WIFI);
// period=5min, pref=WiFi
createTable(“album”, “name VARCHAR, photo OBJECT”, FULL_SYNC);
objs = writeData(“album”, {“name=Snoopy”}, {“photo”});
objs[0].write(photoBuffer); 
// write object data
cursor = readData(“album”, {“photo”}, “name=?”, {“Snoopy”});
mis = cursor.getInputStream().get(0); 
// inputstream for object
mis.read(buffer); 
// read object data into buffer
Writing a Photo App with Simba
Conflict resolution
10
beginCR(“album”);
rows = getConflictedRows(“album”);
for (row; rows; next row) {
    
// choice = MINE, THEIRS, OTHERS
    resolveConflict(“album”, row, MINE);
}
endCR(“album”);
Overall Architecture
11
Simba Cloud paper to be presented at EuroSys 2015!
“Simba: Tunable End-to-End Data Consistency for Mobile Apps”
sClient: Simba Content Service
 
Simba Client API (sClientLib)
Interface to access table
and object data for apps
Upcall alerts for events
(new data, conflict) to apps
SimbaSync
Manage fault-tolerance,
data consistency,
row-level atomicity
N/W Manager
Send/receive sync messages, receive notifications
Simba Client Data Store
12
Simba Client Data Store
 
We don’t want half-formed data to appear on our phone!
Simba Table (sTable)
Unified table store for tabular and object data
13
 
Logical sTable
 
Physical sTable
SQLite
LevelDB
 
Subdivide object into chunks
 
Map by object_id
Simba Local States
14
 
Include additional local states to determine:
Health of data (latest vs. updated)
Sync readiness (object closed after update)
Failure state (sync in progress after network disruption)
Recovery actions (retry, reset, recover corrupted objects, etc.)
Simba local states
 
 
 
 
Dirty Chunk Table (DCT): updated chunk ids per object
 
Update in
tab | obj data
 
End of
obj update
 
Sync in
progress
 
Row in
conflict
Handling Network Failures
15
Handling App/Device Failures
16
Evaluation
 
Evaluation goals
Does Simba provide transparency to apps?
Does Simba perform well for sync and local I/O?
Evaluation setup
sClient
Galaxy Nexus (Android 4.2)
Nexus 7 (Android 4.2)
sCloud
2 Intel Xeon servers: 16-core (2.2GHz), 64GB DRAM, 8 7200RPM 2TB disk
4 VMs on each sCloud: 4 core, 8GB DRAM, one disk
WiFi: 802.11n (WPA)
Cellular: 4G LTE (KT, LGU+, AT&T)
17
App Development with Simba
 
Simple and easy app development with Simba
 
 
 
 
 
 
Building a photo app with existing sync service (Dropbox)
No inter-operation of table and object
No support for row-level atomicity (only column-level!)
No detection & recovery of torn rows
18
Sync Performance
 
End-to-end sync latency for “
1
B col” & “
1
B col + 
1
KB obj”
Test method
Client 1 updates for sync 
 client 2 receives update
Clients (Korea), sCloud (Princeton), Dropbox server (California)
 
 
 
 
Results
Network latency: small component of total sync latency
Simba performs well compared to Dropbox in all cases
19
Local I/O Performance
 
Time to write/read/delete one row with 
1
MB object
 
 
 
 
 
 
~10% slower than Dropbox for write/read
IPC overhead between Simba-app and sClient
Better than Dropbox for delete
Lazy deletion: marked for delete 
 delete after sync completion
20
Conclusions
 
Building data-centric mobile app should be transparent
Mobile app developers should focus on implementing app core logic
Require service that handles complex network and data management
Simba: reliable, consistent, and efficient data-sync service
Unified sTable and API for managing tabular and object data
Transparent handling of data syncs and failures
Resource frugality with delay-tolerant coalescing of sync messages
Practical for real-world usage
Easy app development/porting with CRUD-like API
Sync performance comparable to existing services
Minimum local I/O overhead
21
Thank you!
Simba source: 
https://github.com/SimbaService/Simba
Project homepage: 
http://www.nec-labs.com/~nitin/Simba
 
Related Works
 
Data sync services
Parse, Kinvey, Bayou, Mobius [MobiSys’12]: support table sync
LBFS [SOSP’01]: support file sync
Do not provide sync service for both tables and objects
Failure tolerance
ViewBox [FAST’14]: guarantee consistency of local data at crash
Works for files in desktop FS
Storage unification
TableFS [ATC’13]: separate storage pools for metadata and files
KVFS [FAST’13]: store file and metadata in a single key-value store
Consider integration without network sync or a unified API
23
Balancing Sync Efficiency & Transparency
 
In-memory vs. persistent DCT
Sync only updated chunks for each object during sync
In-memory DCT lost after crash: s
end entire object 
 inefficient!
Persist DCT to prevent re-syncing entire, potentially large objects
In-place vs. out-of-place update
Recover a torn (corrupted) row with data from the consistent state
Out-of-place: local state + I/O overhead for common-case operation
In-place: retrieve consistent version of row from sCloud
24
 
69%
Slide Note

Hi, I am Younghwan from KAIST.

In this talk, I’ll be introducing a sync service we built called Simba, which provides reliable, consistent, and efficient data sync service for mobile apps.

This is a joint work with Nitin, Akshat and Cristian from NEC Labs.

Embed
Share

This informative content discusses the challenges faced in building data-centric mobile apps, such as reliability, consistency, and efficiency. It highlights the increase in data-centric mobile apps and the difficulty in ensuring transparent failure handling, concurrent updates, and minimizing traffic/battery usage. The study on mobile app reliability reveals issues in disruption and crash recovery, emphasizing the importance of reliable sync services and structured data handling for user satisfaction.


Uploaded on Oct 06, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Reliable, Consistent, and Efficient Data Sync for Mobile Apps Younghwan Go*, Nitin Agrawal, AkshatAranya, and Cristian Ungureanu NEC Labs. America KAIST*

  2. Increase in Data-centric Mobile Apps Massive growth in mobile data traffic [Cisco VNI Mobile 2014] 24.3 Exabytes per month by 2019 190 Exabytes of mobile traffic generated globally by 2018 = 42 trillion images, 4 trillion video clips 2

  3. Difficulty in Building Data-centric Apps Reliability: transparent failure handling Look! My data is corrupted! Consistency: concurrent updates, sync atomicity Structured data Col Obj Row ID Unstructured data name file ????? Efficiency: minimize traffic/battery usage 010111001 110110111 3

  4. Mobile App Study on Reliability Study mobile app recovery under failures Network disruption, local app crash, device power loss Analyze recovery when failed during write/update Test 15 apps that use tables and objects Independent or existing sync services (e.g., Dropbox, Parse, Kinvey) Test process Client 1 (RECOVER) 1. Activate airplane mode 2. Manually kill app 3. Pull the battery out Client 2 (READ) Client 1 (WRITE/UPDATE) 4

  5. Current Mobile Apps are not Reliable! Disruption recovery Loss of data if app/notification closed during disruption No notification of sync failure Manual re-sync creates multiple copies of same note Crash recovery Partial object created locally without sync Corrupted object synced and spread to second client Additional observations No app correctly recovered from crash at object update Many apps simply disable object update capability altogether More details of the study can be found in our paper 5

  6. Goals of Sync as a Service Reliability User can always sync to the latest data User s update is guaranteed to be synced to server Consistency Data can always return to a consistent state even after failures Inter-dependent structured/unstructured data are synced atomically Efficiency Minimum mobile data traffic is generated for sync/recovery Device s overall network radio usage is reduced to save battery 6

  7. Outline Introduction Mobile app study on reliability Simba Client Design Evaluation Conclusion 7

  8. Simba: Data-sync Service for Mobile Apps High-level programming abstraction CRUD-like interface for easy development Unify tabular and object data Transparent handling of data syncs and failures Failure detection & recovery at network disruption and crash Guarantee atomic sync of tabular and object data Resource frugality with delay-tolerance and coalescing Delay sync messages to be clustered Reduce number of network messages & radio usage 8

  9. Writing a Photo App with Simba Create a photo album createTable( album , name VARCHAR, photo OBJECT , FULL_SYNC); Register read/write sync registerReadSync( album ,600,0,3G); // period=10min, pref=3G registerWriteSync( album ,300,0,WIFI);// period=5min, pref=WiFi Add a new photo objs = writeData( album , { name=Snoopy }, { photo }); objs[0].write(photoBuffer); // write object data Retrieve stored photo cursor = readData( album , { photo }, name=? , { Snoopy }); mis = cursor.getInputStream().get(0); // inputstream for object mis.read(buffer); // read object data into buffer 9

  10. Writing a Photo App with Simba Conflict resolution beginCR( album ); rows = getConflictedRows( album ); for (row; rows; next row) { // choice = MINE, THEIRS, OTHERS resolveConflict( album , row, MINE); } endCR( album ); 10

  11. Overall Architecture Reliable data sync between sClient Simba Cloud (sCloud) Manage data across multiple apps, tables, and clients Respond to sClient s sync request Push notifications to sClient Version-based Sync Protocol Row-level consistency Unique id per row, ????? One version per row, ???? sCloud Simba Cloud paper to be presented at EuroSys 2015! Simba: Tunable End-to-End Data Consistency for Mobile Apps 11

  12. sClient: Simba Content Service Simba Client API (sClientLib) Interface to access table and object data for apps Upcall alerts for events (new data, conflict) to apps SimbaSync Manage fault-tolerance, data consistency, row-level atomicity N/W Manager Send/receive sync messages, receive notifications Simba Client Data Store Simba Cloud 12

  13. Simba Client Data Store We don t want half-formed data to appear on our phone! Simba Table (sTable) Unified table store for tabular and object data Logical sTable Physical sTable Map by object_id SQLite Subdivide object into chunks LevelDB 13

  14. Simba Local States Include additional local states to determine: Health of data (latest vs. updated) Sync readiness (object closed after update) Failure state (sync in progress after network disruption) Recovery actions (retry, reset, recover corrupted objects, etc.) Simba local states Row in conflict End of obj update Sync in progress Update in tab | obj data Row ID Version Name Snoopy Photo object_id ?????? ?????? ????????????? ?????? 0/1 0/1 0/1/../n 0/1 0/1 ????? ???? Dirty Chunk Table (DCT): updated chunk ids per object 14

  15. Handling Network Failures Move to a consistent state after network disruption Detect & recover in the middle of sync Consult state upon network disruption Recovery policy dependent on server response (???, ???, ???, ???) No op, normal operation, retry, reset & retry, roll forward Upstream sync example State at network disruption Implication Recovery Policy Action [SP=1] before sync response Missed response Reset & retry SP=0,TD=1, OD=1 if DCT Downstream sync example TD OD Recovery Action ??? State at network disruption Implication Delete entry, resend downstream sync request * * * [???=1] after sync response Partial response 15

  16. Handling App/Device Failures Roll back/forward to a consistent state after crash Recovery policy dependent on local states ??????, ??????, ???????, ??????, ??????, ??? Recover from a crash during sync TD OD OO SP CF Recovery Action 0 0 =0 1 - Restart upstream sync (SP = 0, TD = 1, OD = 1 if DCT) Recover from a crash at update TD OD OO SP CF Recovery Action 1 0 >0 0 0 Start upstream sync (OO = 0) Torn row! Retrieve consistent row version from sCloud (TD = 0, OD = 0, OO = 0) * 1 >0 0 0 16

  17. Evaluation Evaluation goals Does Simba provide transparency to apps? Does Simba perform well for sync and local I/O? Evaluation setup sClient Galaxy Nexus (Android 4.2) Nexus 7 (Android 4.2) sCloud 2 Intel Xeon servers: 16-core (2.2GHz), 64GB DRAM, 8 7200RPM 2TB disk 4 VMs on each sCloud: 4 core, 8GB DRAM, one disk WiFi: 802.11n (WPA) Cellular: 4G LTE (KT, LGU+, AT&T) 17

  18. App Development with Simba Simple and easy app development with Simba App Description Total LoC 4,178 Simba LoC 367 Simba-Notes Rich note-taking with embedded images and media Monitor and record a person s heart rate, cadence and altitude using Zephyr heartbeat sensor Record car engine s RPM,speed, engine load, and etc using Soliport OBD2 sensor Photo-sync app with write/update/read/delete operations on tabular and object data HbeatMonitor 2,472 384 CarSensor 3,063 384 Simba-Photo 527 170 Building a photo app with existing sync service (Dropbox) No inter-operation of table and object No support for row-level atomicity (only column-level!) No detection & recovery of torn rows 18

  19. Sync Performance End-to-end sync latency for 1B col & 1B col + 1KB obj Test method Client 1 updates for sync client 2 receives update Clients (Korea), sCloud (Princeton), Dropbox server (California) Results Network latency: small component of total sync latency Simba performs well compared to Dropbox in all cases 19

  20. Local I/O Performance Time to write/read/delete one row with 1MB object ~10% slower than Dropbox for write/read IPC overhead between Simba-app and sClient Better than Dropbox for delete Lazy deletion: marked for delete delete after sync completion 20

  21. Conclusions Building data-centric mobile app should be transparent Mobile app developers should focus on implementing app core logic Require service that handles complex network and data management Simba: reliable, consistent, and efficient data-sync service Unified sTable and API for managing tabular and object data Transparent handling of data syncs and failures Resource frugality with delay-tolerant coalescing of sync messages Practical for real-world usage Easy app development/porting with CRUD-like API Sync performance comparable to existing services Minimum local I/O overhead 21

  22. Thank you! Simba source: https://github.com/SimbaService/Simba Project homepage: http://www.nec-labs.com/~nitin/Simba

  23. Related Works Data sync services Parse, Kinvey, Bayou, Mobius [MobiSys 12]: support table sync LBFS [SOSP 01]: support file sync Do not provide sync service for both tables and objects Failure tolerance ViewBox [FAST 14]: guarantee consistency of local data at crash Works for files in desktop FS Storage unification TableFS [ATC 13]: separate storage pools for metadata and files KVFS [FAST 13]: store file and metadata in a single key-value store Consider integration without network sync or a unified API 23

  24. Balancing Sync Efficiency & Transparency In-memory vs. persistent DCT Sync only updated chunks for each object during sync In-memory DCT lost after crash: send entire object inefficient! Persist DCT to prevent re-syncing entire, potentially large objects In-place vs. out-of-place update Recover a torn (corrupted) row with data from the consistent state Out-of-place: local state + I/O overhead for common-case operation In-place: retrieve consistent version of row from sCloud 69% 24

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#