Understanding Convolutional Neural Networks: Architectural Characterizations for Accuracy Inference

undefined
 
Duc Hoang – Rhodes College
Supervisor: Dr. Gabriel N. Perdue
SIST – Final Presentation
1 August 2019
 
I
n
f
e
r
r
i
n
g
 
C
o
n
v
o
l
u
t
i
o
n
a
l
 
N
e
u
r
a
l
 
N
e
t
w
o
r
k
s
a
c
c
u
r
a
c
y
 
f
r
o
m
 
i
t
s
 
a
r
c
h
i
t
e
c
t
u
r
a
l
 
c
h
a
r
a
c
t
e
r
i
z
a
t
i
o
n
s
 
 
I. MINERvA experiment
a.
Neutrinos and MINERvA detector
b.
Vertex finding and hadron multiplicity problem
O
u
t
l
i
n
e
8/01/2019
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
2
 
 
IV. Summary & Outlook
 
III. Inferring CNNs’ accuracy before training time
a.
Architectural characterizations
b.
Predicting CNNs’ accuracy based on characterizations – why
is it useful?
 
II. Deep Learning
a.
Deep Neural Networks
b.
Convolutional Neural Networks (CNNs)
c.
CNN’s design difficulties
 
N
e
u
t
r
i
n
o
s
8/17/2024
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
3
 
N
e
u
t
r
i
n
o
s
 
a
r
e
 
f
u
n
d
a
m
e
n
t
a
l
.
T
h
e
y
 
a
r
e
 
e
l
e
c
t
r
i
c
a
l
l
y
 
n
e
u
t
r
a
l
 
"
p
a
r
t
n
e
r
s
"
 
o
f
 
t
h
e
 
f
a
m
i
l
i
a
r
 
c
h
a
r
g
e
d
l
e
p
t
o
n
s
 
(
e
.
g
.
,
 
e
l
e
c
t
r
o
n
s
)
.
T
h
e
y
 
a
r
e
 
v
e
r
y
 
l
i
g
h
t
,
T
h
e
y
 
v
e
r
y
 
r
a
r
e
l
y
 
i
n
t
e
r
a
c
t
 
w
i
t
h
 
o
t
h
e
r
 
p
a
r
t
i
c
l
e
s
 
M
I
N
E
R
v
A
8/17/2024
4
 
N
u
c
l
e
a
r
 
e
f
f
e
c
t
s
 
w
i
t
h
 
a
v
a
r
i
e
t
y
 
o
f
 
t
a
r
g
e
t
 
m
a
t
e
r
i
a
l
s
r
a
n
g
i
n
g
 
f
r
o
m
 
H
e
l
i
u
m
 
t
o
L
e
a
d
.
F
i
n
e
-
g
r
a
i
n
e
d
 
r
e
s
o
l
u
t
i
o
n
f
o
r
 
e
x
c
e
l
l
e
n
t
 
k
i
n
e
m
a
t
i
c
m
e
a
s
u
r
e
m
e
n
t
s
.
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
V
e
r
t
e
x
 
F
i
n
d
i
n
g
 
a
n
d
 
H
a
d
r
o
n
 
M
u
l
t
i
p
l
i
c
i
t
y
 
p
r
o
b
l
e
m
8/17/2024
5
Hadron
Showers
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
Fully-connected architecture
E
a
c
h
 
i
n
p
u
t
 
m
u
l
t
i
p
l
i
e
d
 
b
y
 
a
w
e
i
g
h
t
.
W
e
i
g
h
t
e
d
 
v
a
l
u
e
s
 
a
r
e
s
u
m
m
e
d
,
 
B
i
a
s
 
i
s
 
a
d
d
e
d
.
N
o
n
-
l
i
n
e
a
r
 
a
c
t
i
v
a
t
i
o
n
f
u
n
c
t
i
o
n
 
i
s
 
a
p
p
l
i
e
d
T
r
a
i
n
e
d
 
b
y
 
v
a
r
y
i
n
g
 
t
h
e
p
a
r
a
m
e
t
e
r
s
 
t
o
 
 
 
m
i
n
i
m
i
z
e
 
a
l
o
s
s
 
 
 
f
u
n
c
t
i
o
n
 
(
q
u
a
n
t
i
f
i
e
s
h
o
w
 
m
a
n
y
 
m
i
s
t
a
k
e
s
 
 
 
t
h
e
n
e
t
w
o
r
k
 
m
a
k
e
s
)
D
e
e
p
 
N
e
u
r
a
l
 
N
e
t
w
o
r
k
s
8/17/2024
6
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
S
i
m
i
l
a
r
 
c
o
n
c
e
p
t
 
t
o
 
D
e
e
p
 
N
e
u
r
a
l
 
N
e
t
w
o
r
k
s
,
 
b
u
t
 
h
i
g
h
l
y
 
e
f
f
e
c
t
i
v
e
 
f
o
r
 
i
m
a
g
e
i
n
p
u
t
s
,
 
a
n
d
 
m
o
d
e
r
n
 
n
e
u
t
r
i
n
o
 
d
e
t
e
c
t
o
r
s
 
a
r
e
 
i
m
a
g
i
n
g
 
d
e
t
e
c
t
o
r
s
.
C
o
n
v
o
l
u
t
i
o
n
a
l
 
N
e
u
r
a
l
 
N
e
t
w
o
r
k
s
 
(
C
N
N
s
)
8/17/2024
7
 
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
X
U
V
 
 
There is no universal CNN design for every tasks.
And designing an appropriate structure/architecture of CNN
takes a lot of time and effort even for the experts.
There is no systematic way to design CNNs: mainly rely on
human intuition and random/grid search.
Computationally expensive to train a CNN model.
D
i
f
f
i
c
u
l
t
y
8/17/2024
Presenter | Presentation Title or Meeting Title
8
 
 
 
I.
P
r
o
p
o
s
e
 
a
 
s
y
s
t
e
m
a
t
i
c
l
a
n
g
u
a
g
e
 
t
o
 
c
h
a
r
a
c
t
e
r
i
z
e
C
N
N
s
 
a
r
c
h
i
t
e
c
t
u
r
e
,
 
a
n
d
d
e
m
o
n
s
t
r
a
t
e
 
t
h
a
t
 
t
h
e
y
 
c
a
n
 
b
e
p
r
e
d
i
c
t
i
v
e
 
o
f
 
a
 
C
N
N
s
a
c
c
u
r
a
c
y
.
II.
S
u
g
g
e
s
t
 
a
r
c
h
i
t
e
c
t
u
r
a
l
c
h
a
n
g
e
s
 
t
o
 
C
N
N
s
 
f
o
r
d
i
f
f
e
r
e
n
t
 
p
h
y
s
i
c
s
 
t
a
s
k
s
 
(
v
e
r
t
e
x
f
i
n
d
i
n
g
 
a
n
d
 
h
a
d
r
o
n
m
u
l
t
i
p
l
i
c
i
t
y
)
O
b
j
e
c
t
i
v
e
s
8/17/2024
9
 
Examples of architectural
attributes we extracted (32 in
total):
Number of convolutional
layers.
Number of rectified linear
unit (ReLU) activated
convolutional layers.
Average depth
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
M
e
t
h
o
d
8/17/2024
Presenter | Presentation Title or Meeting Title
10
 
Architectural
characterizations
 
CNN performance (accuracy)
Machine
Learning models
(classification
and regression)
 
Important architecture
of CNN for physics task
 
Interpret the models
 
 
D
i
v
i
d
e
 
d
a
t
a
 
s
e
t
 
i
n
t
o
b
r
o
k
e
n
 
a
n
d
 
 
h
e
a
l
t
h
y
n
e
t
w
o
r
k
s
.
U
s
e
 
R
a
n
d
o
m
 
F
o
r
e
s
t
 
a
n
d
E
x
t
r
e
m
e
l
y
 
R
a
n
d
o
m
i
z
e
d
T
r
e
e
 
t
o
 
p
r
e
d
i
c
t
 
e
a
c
h
C
N
N
s
 
c
l
a
s
s
.
 
C
l
a
s
s
i
f
i
c
a
t
i
o
n
8/17/2024
11
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
M
a
c
h
i
n
e
 
L
e
a
r
n
i
n
g
 
m
o
d
e
l
s
 
p
e
r
f
o
r
m
 
s
i
g
n
i
f
i
c
a
n
t
l
y
 
b
e
t
t
e
r
 
t
h
a
n
r
a
n
d
o
m
 
g
u
e
s
s
i
n
g
 
(
5
0
%
 
w
h
e
n
 
t
h
e
r
e
 
i
s
 
n
o
 
c
l
a
s
s
 
o
v
e
r
f
l
o
w
)
:
 
C
l
a
s
s
i
f
i
c
a
t
i
o
n
 
r
e
s
u
l
t
s
8/17/2024
12
 
Were also able to extract important features to suggest
architectural changes to CNNs. 
More details in paper.
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
F
i
t
t
e
d
 
a
n
 
n
o
n
-
l
i
n
e
a
r
 
O
r
d
i
n
a
r
y
 
L
e
a
s
t
 
S
q
u
a
r
e
 
m
o
d
e
l
 
 
o
n
 
j
u
s
t
 
t
h
e
h
e
a
l
t
h
y
 
n
e
t
w
o
r
k
s
.
 
R
e
g
r
e
s
s
i
o
n
 
o
n
 
j
u
s
t
 
h
e
a
l
t
h
y
 
n
e
t
w
o
r
k
s
 
8/17/2024
 
13
 
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
L
i
m
i
t
a
t
i
o
n
s
:
 
S
t
i
l
l
 
n
o
t
 
e
n
o
u
g
h
 
p
a
r
a
m
e
t
e
r
s
 
t
o
 
c
h
a
r
a
c
t
e
r
i
z
e
 
t
h
e
d
e
t
a
i
l
e
d
 
r
e
l
a
t
i
o
n
s
h
i
p
 
b
e
t
w
e
e
n
 
C
N
N
s
 
a
r
c
h
i
t
e
c
t
u
r
e
 
a
n
d
 
i
t
s
p
e
r
f
o
r
m
a
n
c
e
.
 
▶︎
 
P
l
a
n
n
i
n
g
 
t
o
 
e
x
t
e
n
d
 
a
t
t
r
i
b
u
t
e
 
s
e
t
 
i
n
 
t
h
e
 
f
u
t
u
r
e
.
R
e
g
r
e
s
s
i
o
n
 
r
e
s
u
l
t
s
8/17/2024
14
 
Models fitted on two populations of vertex finding CNNs:
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
Proposed a systematic language to characterize
convolutional neural networks architecture.
Successfully demonstrated that we can use those parameters
to predict whether a network is “good”.
There are limitations to predict the exact accuracy, but initial
results are promising. Extension of the attributes set might
help in the future.
One of the early studies about relationship between CNN’s
architecture and its performance.
More details in our up-coming paper.
O
u
t
l
o
o
k
 
&
 
S
u
m
m
a
r
y
8/17/2024
15
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
Special thanks to my supervisor, Dr. Gabriel N. Perdue.
To members of MINERvA’s machine learning group for their
mentorship and useful conversations:
Anushree
Steven (Oak Ridge)
Nafis
Luis
To members of SIST committee and my mentor group and
other awesome interns:
Laura Fields
Sandra Charles
Judy Nunez
 
A
c
k
n
o
w
l
e
d
g
e
m
e
n
t
s
 
8/17/2024
 
16
 
 
 
Alexander Martinez
 Raul Campos
 Matthew Alvarez
 
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
R
e
f
e
r
e
n
c
e
s
 
8/17/2024
 
17
 
Neural networks and Convolutional Networks visualization:
Source
 
Random Forest: 
Source
 
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
B
a
c
k
-
u
p
s
 
8/17/2024
 
18
 
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
K
-
f
o
l
d
s
 
c
r
o
s
s
 
v
a
l
i
d
a
t
i
o
n
 
8/17/2024
 
19
 
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
Different deep learning
problems require
different network
architecture.
However, selecting an
appropriate architecture
for CNNs is usually done
by human intuition or
random search
If we have a way to
uniformly characterize a
network architecture,
then it would be
particularly useful.
A
r
c
h
i
t
e
c
t
u
r
a
l
 
c
h
a
r
a
c
t
e
r
i
z
a
t
i
o
n
s
 
o
f
 
C
N
N
s
8/17/2024
20
Good
for the
task?
 
?
Duc Hoang | Inferring Convolutional Neural Network’s accuracy
 
 
R
a
n
d
o
m
 
F
o
r
e
s
t
 
a
n
d
 
E
x
t
r
e
m
e
l
y
 
R
a
n
d
o
m
i
z
e
d
 
T
r
e
e
 
8/17/2024
 
Presenter | Presentation Title or Meeting Title
 
21
Slide Note
Embed
Share

This presentation by Duc Hoang from Rhodes College explores inferring the accuracy of Convolutional Neural Networks (CNNs) based on their architectural characterizations. The talk covers the MINERvA experiment, deep learning concepts including CNNs, and the significance of predicting CNN accuracy before training. It delves into the challenges of designing CNNs and emphasizes the importance of architectural characterizations in optimizing network performance.


Uploaded on Aug 17, 2024 | 0 Views


Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

E N D

Presentation Transcript


  1. Inferring Convolutional Neural Networks accuracy from its architectural characterizations Duc Hoang Rhodes College Supervisor: Dr. Gabriel N. Perdue SIST Final Presentation 1 August 2019

  2. Outline I. MINERvA experiment a. Neutrinos and MINERvA detector b. Vertex finding and hadron multiplicity problem II. Deep Learning a. Deep Neural Networks b. Convolutional Neural Networks (CNNs) c. CNN s design difficulties III. Inferring CNNs accuracy before training time a. Architectural characterizations b. Predicting CNNs accuracy based on characterizations why is it useful? IV. Summary & Outlook 8/01/2019 Duc Hoang | Inferring Convolutional Neural Network s accuracy 2

  3. Neutrinos Neutrinos are fundamental. They are electrically neutral "partners" of the familiar charged leptons (e.g., electrons). They are very light, They very rarely interactwith other particles Proton UU D Quarks Protons Neutrons Electrons 3 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  4. MINERvA Nuclear effects with a variety of target materials ranging from Helium to Lead. Fine-grained resolution for excellent kinematic measurements. Target 5 Target 4 Water Target Target 3 Target 2 Target 1 He Target 4 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  5. Vertex Finding and Hadron Multiplicity problem strip number Hadron Showers True vertex plane number 5 meters 8/17/2024 5 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  6. Deep Neural Networks Fully-connected architecture Each input multiplied by a weight. Weighted values are summed, Bias is added. Non-linear activation function is applied Trained by varying the parameters to minimize a loss function (quantifies how many mistakes the network makes) 8/17/2024 6 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  7. Convolutional Neural Networks (CNNs) Similar concept to Deep Neural Networks, but highly effective for image inputs, and modern neutrino detectors are imaging detectors. U X V 7 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  8. Difficulty There is no universal CNN design for every tasks. And designing an appropriate structure/architecture of CNN takes a lot of time and effort even for the experts. There is no systematic way to design CNNs: mainly rely on human intuition and random/grid search. Computationally expensive to train a CNN model. 8/17/2024 Presenter | Presentation Title or Meeting Title 8

  9. Objectives Examples of architectural attributes we extracted (32 in total): Number of convolutional layers. Number of rectified linear unit (ReLU) activated convolutional layers. Average depth I. Propose a systematic language to characterize CNN s architecture, and demonstrate that they can be predictive of a CNN s accuracy. II. Suggest architectural changes to CNNs for different physics tasks (vertex finding and hadron multiplicity) 8/17/2024 9 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  10. Method Important architecture of CNN for physics task Interpret the models Machine Learning models (classification and regression) Architectural characterizations CNN performance (accuracy) 10 8/17/2024 Presenter | Presentation Title or Meeting Title

  11. Classification Divide data set into broken and healthy networks. Healthy Broken Use Random Forest and Extremely Randomized Tree to predict each CNN s class. 11 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  12. Classification results Machine Learning models perform significantly better than random guessing (50% when there is no class overflow): Average 5-fold cross- validation scores Model Accuracy on test set 70.3 0.006 % 70.6 % Random Forest 70.2 0.003 % 70.5 % Extremely Randomized Tree Were also able to extract important features to suggest architectural changes to CNNs. More details in paper. 12 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  13. Regression on just healthy networks Fitted an non-linear Ordinary Least Square model on just the healthy networks. 8/17/2024 13 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  14. Regression results Models fitted on two populations of vertex finding CNNs: ?? of OLS Population Number of Healthy CNNs First 0.426 49276 Second 0.295 21415 Combined 0.961 70691 Limitations: Still not enough parameters to characterize the detailed relationship between CNN s architecture and its performance. Planning to extend attribute set in the future. 14 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  15. Outlook & Summary Proposed a systematic language to characterize convolutional neural networks architecture. Successfully demonstrated that we can use those parameters to predict whether a network is good . There are limitations to predict the exact accuracy, but initial results are promising. Extension of the attributes set might help in the future. One of the early studies about relationship between CNN s architecture and its performance. More details in our up-coming paper. 15 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  16. Acknowledgements Special thanks to my supervisor, Dr. Gabriel N. Perdue. To members of MINERvA s machine learning group for their mentorship and useful conversations: Anushree Steven (Oak Ridge) Nafis Luis To members of SIST committee and my mentor group and other awesome interns: Laura Fields Sandra Charles Judy Nunez Alexander Martinez Raul Campos Matthew Alvarez 8/17/2024 16 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  17. References Neural networks and Convolutional Networks visualization: Source Random Forest: Source 8/17/2024 17 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  18. Back-ups 18 8/17/2024 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  19. K-folds cross validation 8/17/2024 19 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  20. Architectural characterizations of CNNs Different deep learning problems require different network architecture. However, selecting an appropriate architecture for CNNs is usually done by human intuition or random search If we have a way to uniformly characterize a network architecture, then it would be particularly useful. Good for the task? ? 8/17/2024 20 Duc Hoang | Inferring Convolutional Neural Network s accuracy

  21. Random Forest and Extremely Randomized Tree 8/17/2024 Presenter | Presentation Title or Meeting Title 21

Related


More Related Content

giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#giItT1WQy@!-/#