What to Expect of Classifiers: Reasoning about Logistic Regression with Missing Features
Computer Science Department, UCLA
Pasha Khosravi, Yitao Liang, YooJung Choi and Guy Van den Broeck
.
W
h
a
t
t
o
E
x
p
e
c
t
o
f
C
l
a
s
s
i
f
i
e
r
s
?
R
e
a
s
o
n
i
n
g
a
b
o
u
t
L
o
g
i
s
t
i
c
R
e
g
r
e
s
s
i
o
n
w
i
t
h
M
i
s
s
i
n
g
F
e
a
t
u
r
e
s
C
o
m
p
u
t
e
r
S
c
i
e
n
c
e
August 15, 2019
2
M
o
t
i
v
a
t
i
o
n
Train
Classifier
(ex. Logistic Regression)
Predict
}
Test samples with
Missing Features
August 15, 2019
3
C
o
m
m
o
n
A
p
p
r
o
a
c
h
e
s
•
Common approach is to fill out the missing features, i.e.
doing imputation.
•
They make unrealistic assumptions (mean, median, etc).
•
More sophisticated methods such as MICE don’t scale to
bigger problems (also have assumptions).
•
We want a more principled way of dealing with this.
August 15, 2019
4
G
e
n
e
r
a
t
i
v
e
v
s
D
i
s
c
r
i
m
i
n
a
t
i
v
e
M
o
d
e
l
s
August 15, 2019
5
G
e
n
e
r
a
t
i
v
e
v
s
D
i
s
c
r
i
m
i
n
a
t
i
v
e
M
o
d
e
l
s
August 15, 2019
6
G
e
n
e
r
a
t
i
v
e
v
s
D
i
s
c
r
i
m
i
n
a
t
i
v
e
M
o
d
e
l
s
Now what? How to do predictions using the two models?
August 15, 2019
7
E
x
p
e
c
t
e
d
P
r
e
d
i
c
a
t
i
o
n
M
:
M
i
s
s
i
n
g
f
e
a
t
u
r
e
s
y
:
O
b
s
e
r
v
e
d
F
e
a
t
u
r
e
s
August 15, 2019
8
E
x
p
e
c
t
e
d
P
r
e
d
i
c
a
t
i
o
n
I
n
t
u
i
t
i
o
n
•
I
m
p
u
t
a
t
i
o
n
T
e
c
h
n
i
q
u
e
s
:
R
e
p
l
a
c
e
t
h
e
m
i
s
s
i
n
g
-
n
e
s
s
u
n
c
e
r
t
a
i
n
t
y
w
i
t
h
o
n
e
o
r
m
u
l
t
i
p
l
e
p
o
s
s
i
b
l
e
i
n
p
u
t
s
,
a
n
d
e
v
a
l
u
a
t
e
t
h
e
m
o
d
e
l
s
.
•
E
x
p
e
c
t
e
d
P
r
e
d
i
c
t
i
o
n
:
C
o
n
s
i
d
e
r
s
a
l
l
p
o
s
s
i
b
l
e
i
n
p
u
t
s
a
n
d
c
o
m
p
u
t
e
w
e
i
g
h
t
e
d
a
v
e
r
a
g
e
o
f
t
h
e
e
v
a
l
u
a
t
i
o
n
s
.
August 15, 2019
9
H
a
r
d
n
e
s
s
o
f
T
a
k
i
n
g
E
x
p
e
c
t
a
t
i
o
n
s
•
In general, it is intractable for arbitrary pairs of discriminative and
generative models.
•
We proved that even for the simple case where F is Logistic
Regression and P is Naïve Bayes, the task is NP-Hard.
•
How can we compute the expected prediction?
August 15, 2019
10
W
h
a
t
t
o
d
o
i
n
s
t
e
a
d
?
What if we only have a generative model that is both good at:
1) Classifying 2) Missing features.
C
o
n
f
o
r
m
a
n
t
l
e
a
r
n
i
n
g
•
Given a discriminative classifier, learn a generative model that
1.
Conforms
to the classifier.
2.
Maximizes the likelihood of joint feature distribution P(X)
August 15, 2019
11
N
a
ï
v
e
C
o
n
f
o
r
m
a
n
t
L
e
a
r
n
i
n
g
(
N
a
C
L
)
We focus on of Conformant Learning involving Logistic Regression and Naïve Bayes
•
Given a NB model there is unique LR model that conform to it
•
Given a LR model there is many NB models that conform to it
August 15, 2019
12
N
a
ï
v
e
C
o
n
f
o
r
m
a
n
t
L
e
a
r
n
i
n
g
(
N
a
C
L
)
August 15, 2019
13
N
a
ï
v
e
C
o
n
f
o
r
m
a
n
t
L
e
a
r
n
i
n
g
(
N
a
C
L
)
Logistic Regression
Weights
“Best” Conforming
Naïve Bayes
}
NaCL
G
i
t
H
u
b
:
g
i
t
h
u
b
.
c
o
m
/
U
C
L
A
-
S
t
a
r
A
I
/
N
a
C
L
August 15, 2019
14
E
x
p
e
r
i
m
e
n
t
s
:
F
i
d
e
l
i
t
y
t
o
O
r
i
g
i
n
a
l
C
l
a
s
s
i
f
i
e
r
Using Cross Entropy to compare
- probabilities of the original classifier vs probabilities of NaCL’s learned model
August 15, 2019
15
E
x
p
e
r
i
m
e
n
t
s
:
C
l
a
s
s
i
f
i
c
a
t
i
o
n
A
c
c
u
r
a
c
y
More experimental results in the paper.
August 15, 2019
16
O
t
h
e
r
A
p
p
l
i
c
a
t
i
o
n
s
We saw
Expected Prediction
is very effective with handling missing features.
What else can we do?
•
Explanations
•
Feature Selection
•
Fairness
August 15, 2019
17
L
o
c
a
l
E
x
p
l
a
n
a
t
i
o
n
s
u
s
i
n
g
M
i
s
s
i
n
g
-
n
e
s
s
Remove maximum number of supporting
features until expected classification is about
to change, then show the remaining support
features.
S
u
f
f
i
c
i
e
n
t
E
x
p
l
a
n
a
t
i
o
n
s
August 15, 2019
18
C
o
n
c
l
u
s
i
o
n
•
Expected Prediction is an effective tool for several applications such as
missing data, generating explanations
•
We introduced NaCL, which given a logistic regression model, learns a
Naïve Bayes model that
conforms
with the LR model and also
maximizes joint feature likelihood.
•
Future work would be looking at more expressive pair of models, and
potentially choose models that make the expected prediction tractable
.
August 15, 2019
19
What to Expect of Classifiers? Reasoning about Logistic Regression with Missing Features
GitHub:
github.com/UCLA-StarAI/NaCL
This research discusses common approaches in dealing with missing features in classifiers like logistic regression. It compares generative and discriminative models, exploring the idea of training separate models for feature distribution and classification. Expected Prediction is proposed as a principled way to reason about classifier outcomes. The study aims to provide a more structured approach to handling missing data in classifiers.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Computer Science What to Expect of Classifiers? Reasoning about Logistic Regression with Missing Features Pasha Khosravi, Yitao Liang, YooJung Choi and Guy Van den Broeck. Computer Science Department, UCLA Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Motivation Predict } Classifier Train (ex. Logistic Regression) Test samples with Missing Features August 15, 2019 2 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Common Approaches Common approach is to fill out the missing features, i.e. doing imputation. They make unrealistic assumptions (mean, median, etc). More sophisticated methods such as MICE don t scale to bigger problems (also have assumptions). We want a more principled way of dealing with this. August 15, 2019 3 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Generative vs Discriminative Models Notation: Generative Model: A model that defines joint features and class probability distribution, ? ?,? . For example, Na ve Bayes. Discriminative Model: A model that defines class conditional probability distribution, ? ? ?). For example, Logistic Regression. August 15, 2019 4 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Generative vs Discriminative Models Generative Models (ex. Na ve Bayes) Discriminative Models (ex. Logistic Regression) Classification Accuracy Missing Features August 15, 2019 5 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Generative vs Discriminative Models What if we train two models: 1. Generative model for feature distribution ?(?). 2. Discriminative model for the classifier ? ? = ? ? ?). Now what? How to do predictions using the two models? August 15, 2019 6 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Expected Predication Expected Prediction is a principled way to reason about outcome of a classifier, ?(?), w.r.t. a feature distribution ?(?). M: Missing features y: Observed Features August 15, 2019 7 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Expected Predication Intuition Imputation Techniques: Replace the missing-ness uncertainty with oneor multiplepossible inputs, and evaluate the models. Expected Prediction: Considers all possible inputs and compute weighted average of the evaluations. August 15, 2019 8 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Hardness of Taking Expectations How can we compute the expected prediction? In general, it is intractable for arbitrary pairs of discriminative and generative models. We proved that even for the simple case where F is Logistic Regression and P is Na ve Bayes, the task is NP-Hard. August 15, 2019 9 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
What to do instead? What if we only have a generative model that is both good at: 1) Classifying 2) Missing features. Conformant learning Given a discriminative classifier, learn a generative model that 1. Conforms to the classifier. 2. Maximizes the likelihood of joint feature distribution P(X) No missing features Same quality of classification Has missing features No problem, do inference August 15, 2019 10 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Nave Conformant Learning (NaCL) We focus on of Conformant Learning involving Logistic Regression and Na ve Bayes Given a NB model there is unique LR model that conform to it Given a LR model there is many NB models that conform to it August 15, 2019 11 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Nave Conformant Learning (NaCL) We showed that we can write the Na ve Conformant Learning Optimization task as a Geometric Program. Solving geometric programs has an exact and efficient algorithm, and modern GP solvers can handle large problems. For NaCL, we have O(??) number of constraints. ? is the number of features, and ? is the number of classes. August 15, 2019 12 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Nave Conformant Learning (NaCL) } NaCL Logistic Regression Weights Best Conforming Na ve Bayes GitHub: github.com/UCLA-StarAI/NaCL August 15, 2019 13 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Experiments: Fidelity to Original Classifier Using Cross Entropy to compare - probabilities of the original classifier vs probabilities of NaCL s learned model August 15, 2019 14 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Experiments: Classification Accuracy More experimental results in the paper. August 15, 2019 15 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Other Applications We saw Expected Prediction is very effective with handling missing features. What else can we do? Explanations Feature Selection Fairness August 15, 2019 16 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Local Explanations using Missing-ness Goal: To explain an instance of classification Support Features: Making them missing probability goes down Opposing Features: Making them missing probability goes up Sufficient Explanations Remove maximum number of supporting features until expected classification is about to change, then show the remaining support features. August 15, 2019 17 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Conclusion Expected Prediction is an effective tool for several applications such as missing data, generating explanations We introduced NaCL, which given a logistic regression model, learns a Na ve Bayes model that conforms with the LR model and also maximizes joint feature likelihood. Future work would be looking at more expressive pair of models, and potentially choose models that make the expected prediction tractable. August 15, 2019 18 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
Thank You Thank You Thank You What to Expect of Classifiers? Reasoning about Logistic Regression with Missing Features GitHub: github.com/UCLA-StarAI/NaCL August 15, 2019 19 Computer Science Department What to Expect of Classifiers? Reasoning about Logistic Regression with missing features
