Practically Adopting Human Activity Recognition
P
r
a
c
t
i
c
a
l
l
y
A
d
o
p
t
i
n
g
H
u
m
a
n
A
c
t
i
v
i
t
y
R
e
c
o
g
n
i
t
i
o
n
M
o
b
i
C
o
m
2
0
2
3
1
Huatao Xu, Pengfei Zhou, Rui Tan, Mo Li
Nanyang Technological University, University of Pittsburgh,
Hong Kong University of Science and Technology
H
uman Activity Recognition
2
[1] Xu, Huatao, et al. "Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications." Sensys 2021.
H
uman Activity Recognition
P
r
a
c
t
i
c
a
l
a
d
o
p
t
i
o
n
a
t
s
c
a
l
e
.
3
Unlabeled
Data
Labeled Data
H
uman Activity Recognition
A
u
n
i
v
e
r
s
a
l
H
u
m
a
n
A
c
t
i
v
i
t
y
R
e
c
o
g
n
i
t
i
o
n
(
H
A
R
)
s
c
e
n
a
r
i
o
.
4
Massive
and
heterogeneous
Unlabeled
Data
T
arget
U
sers
Source
U
sers
Limited
and
biased
Labeled
Data
H
uman Activity Recognition
A
u
n
i
v
e
r
s
a
l
H
u
m
a
n
A
c
t
i
v
i
t
y
R
e
c
o
g
n
i
t
i
o
n
(
H
A
R
)
s
c
e
n
a
r
i
o
.
5
T
arget
U
sers
Source
U
sers
Prior Works
6
T
arget
U
sers
Source
U
sers
1. Supervised learning models.
Overfitted models
Massive
and
heterogeneous
Unlabeled Data
Limited and biased
Labeled Data
Prior Works
7
T
arget
U
sers
Source
U
sers
2. Foundation models,
e.g., LIMU-BERT
.
Overfitted classifiers
Limited and biased
Labeled Data
Massive
and
heterogeneous
Unlabeled Data
Prior Works
8
T
arget
U
sers
Source
U
sers
3
. Unsupervised domain
adaptation models.
Poor performance
Limited and biased
Labeled Data
There is still a gap in designing general HAR models!
There is still a gap in designing general HAR models!
Massive
and
heterogeneous
Unlabeled Data
Key Challenge
H
o
w
t
o
c
o
m
b
a
t
d
a
t
a
h
e
t
e
r
o
g
e
n
e
i
t
y
a
n
d
e
x
t
r
a
c
t
g
e
n
e
r
a
l
i
z
a
b
l
e
f
e
a
t
u
r
e
s
f
o
r
a
c
t
i
v
i
t
y
r
e
c
o
g
n
i
t
i
o
n
?
9
Solution1:
E
xploiting all
available
data with a two-stage framework.
Key Challenge
H
o
w
t
o
c
o
m
b
a
t
d
a
t
a
h
e
t
e
r
o
g
e
n
e
i
t
y
a
n
d
e
x
t
r
a
c
t
g
e
n
e
r
a
l
i
z
a
b
l
e
f
e
a
t
u
r
e
s
f
o
r
a
c
t
i
v
i
t
y
r
e
c
o
g
n
i
t
i
o
n
?
10
Solution2:
Data augmentations with support of underlying physics
.
Physics-based
Data
Augmentation
Data Augmentation
D
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
i
s
a
t
e
c
h
n
i
q
u
e
t
h
a
t
e
n
r
i
c
h
e
s
t
h
e
d
a
t
a
s
e
t
b
y
c
r
e
a
t
i
n
g
m
o
d
i
f
i
e
d
c
o
p
i
e
s
o
f
e
x
i
s
t
i
n
g
d
a
t
a
.
11
[1] https://medium.com/@tagxdata/data-augmentation-for-computer-vision-9c9ed474291e.
Data Augmentation
A
p
p
l
y
i
n
g
e
x
i
s
t
i
n
g
d
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
s
t
o
a
c
t
i
v
i
t
y
r
e
c
o
g
n
i
t
i
o
n
.
12
It is non-trivial to adopt data augmentation
It is non-trivial to adopt data augmentation
for
for
activity recognition.
activity recognition.
Data Augmentation
W
h
y
d
o
m
o
s
t
d
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
s
f
a
i
l
t
o
i
m
p
r
o
v
e
H
A
R
p
e
r
f
o
r
m
a
n
c
e
?
13
How have underlying physical states changed?
How have underlying physical states changed?
Data Augmentation
W
h
y
d
o
m
o
s
t
d
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
s
f
a
i
l
t
o
i
m
p
r
o
v
e
H
A
R
p
e
r
f
o
r
m
a
n
c
e
?
14
Acc.
Gyr.
Gyr.
Acc.
No Valid States
Unrealistic
Unrealistic
Observation
Observation
s
s
!
!
Physics-Informed Data Augmentation
D
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
s
h
o
u
l
d
h
a
v
e
a
c
o
r
r
e
s
p
o
n
d
i
n
g
t
r
a
n
s
i
t
i
o
n
i
n
p
h
y
s
i
c
a
l
s
t
a
t
e
s
!
15
Acc.
Gyr.
Gyr.
Acc.
R
R
ealistic
ealistic
Observation
Observation
s
s
!
!
Physics-Informed Data Augmentation
D
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
s
h
o
u
l
d
h
a
v
e
a
c
o
r
r
e
s
p
o
n
d
i
n
g
t
r
a
n
s
i
t
i
o
n
i
n
p
h
y
s
i
c
a
l
s
t
a
t
e
s
!
16
Acc.
Gyr.
Gyr.
Acc.
Physical
Physical
Embedding
Embedding
17
Has a physical
embedding
Has deterministic
observations
Complete Data
Augmentation
Yes
No
Approximate Data
Augmentation
Flaky Data
Augmentation
Yes
No
Physics-Informed
Physics-Informed
Data Augmentation
Data Augmentation
e.g., local rotation
e.g., linear upsampling
e.g., flipping
Data
Augmentation
Data Augmentation Adoption
18
I
n
t
e
g
r
a
t
i
n
g
p
h
y
s
i
c
s
-
i
n
f
o
r
m
e
d
d
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
s
i
n
t
o
t
h
e
f
r
a
m
e
w
o
r
k
.
Complete data augmentation
Approximate data augmentation
Flaky data augmentation
Complete data augmentation
Approximate data augmentation
Flaky data augmentation
Our Solution
19
Data-decentralized
scenario
raw data
UniHAR + Federated learning
raw data
UniHAR + Domain adaptation
U
n
i
H
A
R
:
a
n
u
n
i
v
e
r
s
a
l
h
u
m
a
n
a
c
t
i
v
i
t
y
r
e
c
o
g
n
i
t
i
o
n
f
r
a
m
e
w
o
r
k
.
Data-centralized
scenario
Implementation and System Evaluation
20
U
n
i
H
A
R
i
s
f
u
l
l
y
p
r
o
t
o
t
y
p
e
d
i
n
m
o
b
i
l
e
p
l
a
t
f
o
r
m
.
Comparative Evaluation
21
Open datasets: HHAR, UCI, Motion, and Shoaib.
Baseline models:
•
Data-decentralized scenario: MM’15-DCNN, IMWUT’19-TPN, and SenSys’21-LIMU-BERT.
•
Data-centralized scenario: PerCom’18-HDCNN, SECON’20-XHAR, and IMWUT’21-FM.
Cross-dataset experiments:
•
Merge the data with the same activity labels from four datasets.
•
T
r
a
n
s
f
e
r
m
o
d
e
l
s
f
r
o
m
o
n
e
d
a
t
a
s
e
t
t
o
o
t
h
e
r
t
h
r
e
e
d
a
t
a
s
e
t
s
w
i
t
h
o
u
t
a
n
y
l
a
b
e
l
e
d
d
a
t
a
.
Comparative Evaluation
22
O
v
e
r
a
l
l
p
e
r
f
o
r
m
a
n
c
e
.
Comparative Evaluation
23
E
f
f
e
c
t
i
v
e
n
e
s
s
o
f
p
h
y
s
i
c
s
-
i
n
f
o
r
m
e
d
d
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
.
Conclusion
24
•
We consider a practical and challenging HAR
scenario at scale.
•
We present a thorough analysis of IMU data
augmentation methodology and characterize
physics-informed data augmentations.
•
We integrates different data augmentation
methods into a learning framework to address
data heterogeneity.
•
We prototype UniHAR on the standard mobile
platform and evaluate its generalization with
practical settings across different datasets.
Scan to visit UniHAR
Thank
you!
Q&A
25
Backup Slides
26
Prior Works
P
r
e
l
i
m
i
n
a
r
y
e
x
p
e
r
i
m
e
n
t
s
f
o
r
u
n
s
u
p
e
r
v
i
s
e
d
d
o
m
a
i
n
a
d
a
p
t
a
t
i
o
n
m
e
t
h
o
d
s
.
27
Data Visualization
Dataset 1
Dataset 2
Prior Works
P
r
e
l
i
m
i
n
a
r
y
e
x
p
e
r
i
m
e
n
t
s
f
o
r
u
n
s
u
p
e
r
v
i
s
e
d
d
o
m
a
i
n
a
d
a
p
t
a
t
i
o
n
m
e
t
h
o
d
s
.
28
Data Visualization
There is still a gap in designing general HAR models!
There is still a gap in designing general HAR models!
Data Augmentation
D
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
i
s
t
e
c
h
n
i
q
u
e
t
h
a
t
e
n
r
i
c
h
e
s
t
h
e
d
a
t
a
s
e
t
b
y
c
r
e
a
t
i
n
g
m
o
d
i
f
i
e
d
c
o
p
i
e
s
o
f
e
x
i
s
t
i
n
g
d
a
t
a
.
29
[1] Iwana, Brian Kenji, and Seiichi Uchida. "Time series data augmentation for neural networks by time warping with a discriminative teacher." ICPR2021.
Physics-Informed Data Augmentation
30
D
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
m
o
d
e
l
a
n
d
p
h
y
s
i
c
a
l
e
m
b
e
d
d
i
n
g
.
G(·) is a
physical embedding
of F(·)
if observations of the transformed
physical states equal the augmented
observations of F(·).
UniHAR
31
Data-decentralized scenario
(
no
raw data transmission)
Data-centralized scenario
(raw data transmission
is allowed
)
UniHAR Design
32
D
a
t
a
-
d
e
c
e
n
t
r
a
l
i
z
e
d
s
c
e
n
a
r
i
o
(
n
o
r
a
w
d
a
t
a
t
r
a
n
s
m
i
s
s
i
o
n
)
•
F
e
a
t
u
r
e
e
x
t
r
a
c
t
i
o
n
:
s
e
l
f
-
s
u
p
e
r
v
i
s
e
d
+
f
e
d
e
r
a
t
e
d
l
e
a
r
n
i
n
g
w
i
t
h
a
u
g
m
e
n
t
e
d
u
n
l
a
b
e
l
e
d
d
a
t
a
.
•
A
c
t
i
v
i
t
y
r
e
c
o
g
n
i
t
i
o
n
:
s
u
p
e
r
v
i
s
e
d
l
e
a
r
n
i
n
g
w
i
t
h
a
u
g
m
e
n
t
e
d
l
a
b
e
l
e
d
d
a
t
a
.
UniHAR Design
33
D
a
t
a
-
c
e
n
t
r
a
l
i
z
e
d
s
c
e
n
a
r
i
o
(
r
a
w
d
a
t
a
t
r
a
n
s
m
i
s
s
i
o
n
i
s
a
l
l
o
w
e
d
)
•
A
c
t
i
v
i
t
y
r
e
c
o
g
n
i
t
i
o
n
:
s
u
p
e
r
v
i
s
e
d
l
e
a
r
n
i
n
g
w
i
t
h
a
u
g
m
e
n
t
e
d
l
a
b
e
l
e
d
d
a
t
a
+
a
d
v
e
r
s
a
r
i
a
l
t
r
a
i
n
i
n
g
w
i
t
h
u
s
e
r
l
a
b
e
l
s
.
Comparative Evaluation
34
V
a
r
y
i
n
g
n
u
m
b
e
r
o
f
l
a
b
e
l
s
.
Comparative Evaluation
35
V
a
r
y
i
n
g
d
a
t
a
a
u
g
m
e
n
t
a
t
i
o
n
a
d
o
p
t
i
o
n
s
t
r
a
t
e
g
i
e
s
.
Cutting-edge research in Human Activity Recognition (HAR) focuses on practical adoption at scale, leveraging labeled and unlabeled data for inference and adaptation. Prior works explore models like LIMU-BERT and address challenges in combating data heterogeneity for feature extraction.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Practically Adopting Human Activity Recognition MobiCom 2023 Huatao Xu, Pengfei Zhou, Rui Tan, Mo Li Nanyang Technological University, University of Pittsburgh, Hong Kong University of Science and Technology 1
Human Activity Recognition Inference 2 [1] Xu, Huatao, et al. "Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications." Sensys 2021.
Human Activity Recognition Practical adoption at scale. Labeled Data Unlabeled Data 3
Human Activity Recognition A universal Human Activity Recognition (HAR) scenario. Source Users Limited and biased Labeled Data Massive and heterogeneous Unlabeled Data Target Users 4
Human Activity Recognition A universal Human Activity Recognition (HAR) scenario. Source Users Adaptation Target Users 5
Prior Works 1. Supervised learning models. Source Users Limited and biased Labeled Data Overfitted models Target Users Massive and heterogeneous Unlabeled Data 6
Prior Works 2. Foundation models, e.g., LIMU-BERT. Source Users Limited and biased Labeled Data Overfitted classifiers Target Users Massive and heterogeneous Unlabeled Data 7
Prior Works 3. Unsupervised domain adaptation models. Source Users Limited and biased Labeled Data Poor performance Target Users Massive and heterogeneous Unlabeled Data 8 There is still a gap in designing general HAR models!
Key Challenge How to combat data heterogeneity and extract generalizable features for activity recognition? Solution1: Exploiting all available data with a two-stage framework. 9
Key Challenge How to combat data heterogeneity and extract generalizable features for activity recognition? Solution2: Data augmentations with support of underlying physics. Physics-based Data Augmentation 10
Data Augmentation Data augmentation is a technique that enriches the dataset by creating modified copies of existing data. 11 [1] https://medium.com/@tagxdata/data-augmentation-for-computer-vision-9c9ed474291e.
Data Augmentation Applying existing data augmentations to activity recognition. It is non-trivial to adopt data augmentation for activity recognition. 12
Data Augmentation Why do most data augmentations fail to improve HAR performance? Gyr. Acc. Gyr. Acc. 13 How have underlying physical states changed?
Data Augmentation Why do most data augmentations fail to improve HAR performance? Unrealistic Observations! Gyr. Acc. Gyr. Acc. No Valid States 14
Physics-Informed Data Augmentation Data augmentation should have a corresponding transition in physical states! Realistic Observations! Gyr. Acc. Gyr. Acc. 15
Physics-Informed Data Augmentation Data augmentation should have a corresponding transition in physical states! Gyr. Acc. Gyr. Acc. Physical Embedding 16
Data Augmentation Has a physical embedding Yes No Flaky Data Augmentation e.g., flipping Has deterministic observations No Yes Complete Data Augmentation e.g., local rotation Approximate Data Augmentation e.g., linear upsampling Physics-Informed Data Augmentation 17
Data Augmentation Adoption Integrating physics-informed data augmentations into the framework. Complete data augmentation Approximate data augmentation Flaky data augmentation Complete data augmentation Approximate data augmentation Flaky data augmentation 18
Our Solution UniHAR: an universal human activity recognition framework. Data-centralized scenario Data-decentralized scenario 2 1 raw data raw data UniHAR + Federated learning UniHAR + Domain adaptation 19
Implementation and System Evaluation UniHAR is fully prototyped in mobile platform. 20
Comparative Evaluation Open datasets: HHAR, UCI, Motion, and Shoaib. Baseline models: Data-decentralized scenario: MM 15-DCNN, IMWUT 19-TPN, and SenSys 21-LIMU-BERT. Data-centralized scenario: PerCom 18-HDCNN, SECON 20-XHAR, and IMWUT 21-FM. Cross-dataset experiments: Merge the data with the same activity labels from four datasets. Transfer models from one dataset to other three datasets without any labeled data. 21
Comparative Evaluation Overall performance. 22
Comparative Evaluation Effectiveness of physics-informed data augmentation. 23
Conclusion We consider a practical and challenging HAR scenario at scale. We present a thorough analysis of IMU data augmentation methodology and characterize physics-informed data augmentations. We integrates different data augmentation methods into a learning framework to address data heterogeneity. We prototype UniHAR on the standard mobile platform and evaluate its generalization with practical settings across different datasets. Scan to visit Scan to visit UniHAR UniHAR 24
Thank you! Q&A 25
Prior Works Preliminary experiments for unsupervised domain adaptation methods. Dataset 1 Data Visualization Dataset 2 27
Prior Works Preliminary experiments for unsupervised domain adaptation methods. Data Visualization There is still a gap in designing general HAR models! 28
Data Augmentation Data augmentation is technique that enriches the dataset by creating modified copies of existing data. 29 [1] Iwana, Brian Kenji, and Seiichi Uchida. "Time series data augmentation for neural networks by time warping with a discriminative teacher." ICPR2021.
Physics-Informed Data Augmentation Data augmentation model and physical embedding. G( ) is a physical embedding of F( ) if observations of the transformed physical states equal the augmented observations of F( ). 30
UniHAR Data-centralized scenario (raw data transmission is allowed) Data-decentralized scenario (no raw data transmission) 31
UniHAR Design Data-decentralized scenario (no raw data transmission) Feature extraction: self-supervised + federated learning with augmented unlabeled data. Activity recognition : supervised learning with augmented labeled data. 32
UniHAR Design Data-centralized scenario (raw data transmission is allowed) Activity recognition : supervised learning with augmented labeled data + adversarial training with user labels. 33
Comparative Evaluation Varying number of labels. 34
Comparative Evaluation Varying data augmentation adoption strategies. 35
