Enhancing Cyber Performance Through STAT Techniques
I
n
s
t
i
t
u
t
e
f
o
r
D
e
f
e
n
s
e
A
n
a
l
y
s
e
s
730
East
Glebe
Road
Alexandria,
Virginia
22305
Applying
Scientific
Test
and
Analysis
Techniques
(STAT)
to
Testing
and
Evaluating
Performance
Across
the
Cyber
Domain
Mike
Gilmore,
Kelly
Avery,
Matt
Girardi,
John
Hong,
Rebecca
Medlin
July
2022
CLEARED
For
Open
Publication
Jun
22,
2022
Department
of
Defense
OFFICE
OF
PREPUBLICATION
AND
SECURITY
REVIEW
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Evaluating
Performance
Across
the
Cyber
Domain
1
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
The
core
of
the
emerging
National
Defense
Strategy
will
include
“integrated
deterrence,…
a
framework
for
working
across
warfighting
domains,
theaters
and
the
spectrum
of
conflict.”*
Thus,
successfully
executing
Multi-
Domain
Operations
(MDO)
will
remain
key
to
the
strategy.
Enabling
successful
execution
will
include
robust
performance
across
the
Cyber
Domain,
an
important
and
increasingly
contested
part
of
MDO.
Comprehensive
and
efficient
cybersecurity
testing
will
be
needed
to
rigorously
evaluate
mission
assurance
across
the
Cyber
Domain.
*See
.defense
national-defense-
strategy/
,
accessed
March
7,
2022.gov/News/News-Stories/Article/Article/2954945/integrated-deterrence-at-center-of-upcoming-https://www
How
Can
STAT
Benefit
Cyber
Testing
and
Evaluation?
T
h
e
D
o
D
C
y
b
e
r
s
e
c
u
r
i
t
y
T
&
E
G
u
i
d
e
b
o
o
k
“
p
r
o
m
o
t
e
s
d
a
t
a
-
d
r
i
v
e
n
m
i
s
s
i
o
n
-
i
m
p
a
c
t
-
b
a
s
e
d
a
n
a
l
y
s
i
s
a
n
d
a
s
s
e
s
s
m
e
n
t
m
e
t
h
o
d
s
f
o
r
c
y
b
e
r
s
e
c
u
r
i
t
y
t
e
s
t
a
n
d
e
v
a
l
u
a
t
i
o
n
.
.
.
”
I
n
t
h
a
t
r
e
g
a
r
d
,
S
c
i
e
n
t
i
f
i
c
T
e
s
t
a
n
d
A
n
a
l
y
s
i
s
T
e
c
h
n
i
q
u
e
s
o
f
f
e
r
s
:
Efficient
coverage
of
operational
space
and
potential
vulnerabilities
consistent
with
limited
resources
and
time
Objective
and
quantitative
determination
of
how
much
testing
is
enough
and
risks
of
insufficient
testing
Identification
and
statistical
quantification
of
significant
factors/vulnerabilities
Quantitative
evaluation
of
what
is
lost
if
rules
of
engagement
(ROE)
are
too
constraining
and/or
time
is
too
short
Addition
of
structure
to
previously
ad
hoc
test
events,
thereby
aiding
comprehensive
evaluation,
while
not
eliminating
free
play
2
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Framework
for
Applying
STAT
(or
for
Planning
any
Test
and
Evaluation)
D
e
t
e
r
m
i
n
e
s
c
o
p
e
o
f
t
e
s
t
•
Questions
you
can
ask
about
the
system
I
d
e
n
t
i
f
y
a
p
p
r
o
p
r
i
a
t
e
m
e
t
r
i
c
s
•
How
you
should
measure
system
performance
I
d
e
n
t
i
f
y
f
a
c
t
o
r
s
t
h
a
t
a
f
f
e
c
t
p
e
r
f
o
r
m
a
n
c
e
•
Types
of
data
to
collect,
operational
envelope
D
e
v
e
l
o
p
T
e
s
t
D
e
s
i
g
n
•
Quantity
of
data
necessary,
best
resource
allocation,
objective
plans
C
o
n
d
u
c
t
t
h
e
t
e
s
t
•
Adjust
test
execution
if
necessary
A
n
a
l
y
z
e
t
h
e
d
a
t
a
•
Structured
mathematical
data
analysis
plan
appropriate
for
the
design
D
r
a
w
c
o
n
c
l
u
s
i
o
n
s
•
Defensible
risk
assessments
based
on
test
results
Subject
Matter
Expertise
Analytical
Expertise
T
e
s
t
&
E
v
a
l
u
a
t
i
o
n
r
e
q
u
i
r
e
s
c
o
l
l
a
b
o
r
a
t
i
o
n
STAT
tools
can
be
applied
at
each
step
3
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
D
e
t
e
r
m
i
n
e
s
c
o
p
e
o
f
t
e
s
t
W
h
e
r
e
/
w
h
a
t
a
r
e
t
h
e
p
o
t
e
n
t
i
a
l
v
u
l
n
e
r
a
b
i
l
i
t
i
e
s
?
Example
1
–
Using
STAT
to
Help
Structure
a
Systematic
Cyber
Assessment
of
a
Hypothetical
Processing
System
(PS)
4
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Hypothetical
PS—
Comprises
15
Subsystems;
2
Operations
Consoles
3
Subsystem
3
4
Subsystem
4
5
Subsystem
5
6
Subsystem
6
7
Subsystem
7
8
Subsystem
8
9
Subsystem
9
10
Subsystem
10
11
Subsystem
11
12
Subsystem
12
13
Subsystem
13
14
Subsystem
14
15
Subsystem
15
16
Operations
Console
1
17
Operations
Console
2
STAT
can
be
used
to--
-
•
Initially
guide
systematic
assessments
in
narrowing
the
number
of
subsystems
to
be
tested*
•
Aid
structuring
the
“final”
tests
•
Aid
analysis
of
test
results
*Potential
venues
include
Cyber
Table
Tops
(CTTs)
and
other
Mission-
Based
Cyber
Risk
Assessments
(MBCRAs)
How
can
STAT
help?
1
Subsystem
1
2
Subsystem
2
5
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Structuring
a
Systematic
Cyber
Assessment
of
a
Hypothetical
Processing
System
(PS)
-
-
Attacks
on
Single
Subsystems—
Narrow
the
Number
of
Potential
Vulnerabilities
-
-
Attacks
Spanning
Multiple
Subsystems—
6
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
1
Subsystem
1
2
Subsystem
2
3
Subsystem
3
4
Subsystem
4
5
Subsystem
5
6
Subsystem
6
7
Subsystem
7
8
Subsystem
8
9
Subsystem
9
10
Subsystem
10
11
Subsystem
11
12
Subsystem
12
13
Subsystem
13
14
Subsystem
14
15
Subsystem
15
16
Operations
Console
1
17
Operations
Console
2
Consider
entry
using
Operations
Consoles---2-
level
factor
(Entry)
Remaining
subsystems
are
targets---15-level
factor
(Target)
PS
Option
1
:
Operations
Console
1,
Operations
Console
2
for
Entry
(2)
Remaining
Subsystems
are
Targets
(15)
Nearsider
and
Insider
Attack
Postures
(2)
Native,
Foreign
Tools
(2)
120
Total
Combinations
Consider
68
percent
(minimal)
and
80
percent
power
to
correctly
assess/identify
vulnerabilities
to
subsystems
(true
positive)
Consider
80
percent
confidence
of
correctly
excluding
vulnerabilities
(true
negative)
Options
for
Design
of
PS
Cyber
Assessment--
-
Single
Subsystem
Attacks
7
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
PS
Design
Options
for
Assessment--
-
Single
Subsystem
Attacks
15
Subsystems
15
Subsystems
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
s
Assessing
45
potential
vulnerabilities
covers
120
combinations
with
68%
power
and
80%
confidence;
65
assessments
required
for
80%
power
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
8
A
t
t
a
c
k
P
o
s
t
u
r
e
9
Structuring
a
Systematic
Cyber
Assessment
of
a
Hypothetical
Processing
System
(PS)
-
-
Attacks
on
Single
Subsystems—
Narrow
the
Number
of
Potential
Vulnerabilities
-
-
Attacks
Spanning
Multiple
Subsystems—
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
10
Software
Faults
versus
Number
of
Interacting
Parameters
Source:
Kuhn,
D.,
et
al,
Practical
Combinatorial
Testing,
October
2010,
available
at
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-
142.pdf
,
accessed
January
14,
2022.
PARAMETER
=
Input
Data
OR
Configuration
T
r
e
a
t
S
u
b
s
y
s
t
e
m
s
s
p
a
n
n
e
d
a
s
a
C
o
n
f
i
g
u
r
a
t
i
o
n
~87%
to
99%
of
faults
involve
3
parameters
~60%
to
96%
of
faults
involve
2
parameters
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
11
S
u
p
p
o
s
e
:
A
s
s
e
s
s
m
e
n
t
o
f
s
i
n
g
l
e
s
u
b
s
y
s
t
e
m
s
d
e
s
c
r
i
b
e
d
p
r
e
v
i
o
u
s
l
y
n
a
r
r
o
w
s
f
o
c
u
s
t
o
8
s
u
b
s
y
s
t
e
m
s
f
o
r
i
n
i
t
i
a
l
i
n
s
i
d
e
r
(
o
n
l
y
)
p
e
n
e
t
r
a
t
i
o
n
/
a
t
t
a
c
k
t
h
r
o
u
g
h
O
p
e
r
a
t
i
o
n
s
C
o
n
s
o
l
e
1
o
r
2
;
b
u
t
-
-
-
Concern
exists
regarding
attacks
spanning
more
than
one
subsystem
Consider
attacks
spanning
those
8
subsystems
and
any
one
of
the
other
15-
1
with
the
tool(s)
used
unspecified,
but
assumed
to
be
those
most
applicable
in
each
case
as
determined
by
prior
assessment
(e.g.,
specific
native
or
foreign)
PS
Option
2
:
Operations
Console
1,
Operations
Console
2
for
Entry
8
Subsystems
are
first
Targets
(Target
Subsystem
1)
14
Subsystems
are
second
targets
(Target
Subsystem
2)
Insider
Attack
Posture
Most
Applicable
Tool
224
Total
Combinations
(2x8x14)
Options
for
Design
of
PS
Cyber
Assessment--
-
Attacks
Spanning
Two
Subsystems
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
PS
Design
Options
for
Assessment--
-
Attacks
Spanning
Two
Subsystems
1
5
S
u
b
s
y
s
t
e
m
s
Assessing
50
potential
vulnerabilities
covers
224
combinations
with
68%
power
8
S
u
b
s
y
s
t
e
m
s
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
1
and
80%
confidence;
65
assessments
for
80%
power
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
12
13
S
u
p
p
o
s
e
F
u
r
t
h
e
r
:
A
s
s
e
s
s
m
e
n
t
o
f
t
w
o
-
s
u
b
s
y
s
t
e
m
c
o
m
b
i
n
a
t
i
o
n
s
n
a
r
r
o
w
s
f
o
c
u
s
t
o
6
s
u
b
s
y
s
t
e
m
s
a
s
s
e
c
o
n
d
t
a
r
g
e
t
s
;
b
u
t
-
-
-
Concern
exists
regarding
attacks
spanning
up
to
three
subsystems
Consider
attacks
spanning
the
identified
8
first
targets,
6
second
targets,
and
any
one
of the
remaining
15-2
subsystems
PS
Option
3
:
Operations
Console
1,
Operations
Console
2
for
Entry
8
Subsystems
as
first
Targets
(Target
Subsystem
1)
6
Subsystems
as
second
targets
(Target
Subsystem
2)
13
Subsystems
as
third
targets
(Target
Subsystem
3)
Insider
Attack
Posture
Most
Applicable
Tool
1248
Total
Combinations
(2x8x6x13)
PS
Design
Options
for
Assessment--
-
Attacks
Spanning
Three
Subsystems
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
14
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
3
1
5
S
u
b
s
y
s
t
e
m
s
e
a
c
h
v
e
r
t
i
c
a
l
b
a
n
d
Assessing
55
potential
vulnerabilities
covers
1248
combinations
with
68%
power
and
80%
confidence;
70
assessments
for
80%
power
PS
Design
Options
for
Assessment--
-
Attacks
Spanning
Three
Subsystems
6
S
u
b
s
y
s
t
e
m
s
8
S
u
b
s
y
s
t
e
m
s
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
1
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Framework
for
Applying
STAT
(or
for
Planning
any
Test
and
Evaluation)
D
e
t
e
r
m
i
n
e
s
c
o
p
e
o
f
t
e
s
t
•
Questions
you
can
ask
about
the
system
I
d
e
n
t
i
f
y
a
p
p
r
o
p
r
i
a
t
e
m
e
t
r
i
c
s
•
How
you
should
measure
system
performance
I
d
e
n
t
i
f
y
f
a
c
t
o
r
s
t
h
a
t
a
f
f
e
c
t
p
e
r
f
o
r
m
a
n
c
e
•
Types
of
data
to
collect,
operational
envelope
D
e
v
e
l
o
p
T
e
s
t
D
e
s
i
g
n
•
Quantity
of
data
necessary,
best
resource
allocation,
objective
plans
C
o
n
d
u
c
t
t
h
e
t
e
s
t
•
Adjust
test
execution
if
necessary
A
n
a
l
y
z
e
t
h
e
d
a
t
a
•
Structured
mathematical
data
analysis
plan
appropriate
for
the
design
D
r
a
w
c
o
n
c
l
u
s
i
o
n
s
•
Defensible
risk
assessments
based
on
test
results
Subject
Matter
Expertise
Analytical
Expertise
T
e
s
t
&
E
v
a
l
u
a
t
i
o
n
r
e
q
u
i
r
e
s
c
o
l
l
a
b
o
r
a
t
i
o
n
STATtools
can
be
applied
at
each
step
D
e
m
o
n
s
t
r
a
t
e
d
H
o
w
m
i
g
h
t
t
h
i
s
w
o
r
k
?
15
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Applying
the
Framework
to
Cyber
T&E
(Steps
2
-
3)
Objectives
-
-
-
C
o
o
p
e
r
a
t
i
v
e
t
e
s
t
–
a
t
t
e
m
p
t
t
o
c
o
m
p
r
e
h
e
n
s
i
v
e
l
y
i
d
e
n
t
i
f
y
v
u
l
n
e
r
a
b
i
l
i
t
i
e
s
a
n
d
v
a
l
i
d
a
t
e
e
x
p
o
s
u
r
e
s
i
n
s
y
s
t
e
m
A
d
v
e
r
s
a
r
i
a
l
t
e
s
t
–
u
s
i
n
g
t
h
e
r
e
s
u
l
t
s
o
f
t
h
e
c
o
o
p
e
r
a
t
i
v
e
t
e
s
t
i
n
a
s
r
e
a
l
i
s
t
i
c
s
e
t
t
i
n
g
a
s
a
p
p
r
o
p
r
i
a
t
e
,
a
s
s
e
s
s
s
y
s
t
e
m
/
u
s
e
r
s
t
o
p
r
o
t
e
c
t
,
m
i
t
i
g
a
t
e
,
a
n
d
r
e
s
t
o
r
e
w
h
e
n
f
a
c
e
d
w
i
t
h
various
types
of
cyber
threats
Potential
response
variables
--
-
A
t
t
a
c
k
t
h
r
e
a
d
l
e
n
g
t
h
/
n
u
m
b
e
r
o
f
s
t
e
p
s
L
e
v
e
l
o
f
t
h
r
e
a
t
c
a
p
a
b
i
l
i
t
y
r
e
q
u
i
r
e
d
t
o
a
c
h
i
e
v
e
a
c
t
i
o
n
(
N
a
s
c
e
n
t
,
L
i
m
i
t
e
d
,
M
o
d
e
r
a
t
e
,
A
d
v
a
n
c
e
d
)
S
e
v
e
r
i
t
y
o
f
m
i
s
s
i
o
n
e
f
f
e
c
t
s
(
N
o
n
e
,
L
o
w
,
M
e
d
,
H
i
g
h
)
(
A
A
o
n
l
y
)
T
i
m
e
t
o
d
e
t
e
c
t
/
m
i
t
i
g
a
t
e
/
r
e
s
t
o
r
e
T
i
m
e
t
o
p
e
n
e
t
r
a
t
e
/
a
c
h
i
e
v
e
e
f
f
e
c
t
Potential
factors
--
-
P
r
o
t
o
c
o
l
o
r
o
b
j
e
c
t
i
v
e
(
W
e
b
a
p
p
l
i
c
a
t
i
o
n
,
s
e
r
v
e
r
s
,
i
n
t
e
r
f
a
c
e
s
w
i
t
h
o
t
h
e
r
s
y
s
t
e
m
s
,
e
t
c
.
)
T
y
p
e
o
f
c
y
b
e
r
e
f
f
e
c
t
(
C
o
n
f
i
d
e
n
t
i
a
l
i
t
y
,
I
n
t
e
g
r
i
t
y
,
A
v
a
i
l
a
b
i
l
i
t
y
)
S
t
a
r
t
i
n
g
p
o
s
t
u
r
e
(
O
u
t
s
i
d
e
r
,
N
e
a
r
-
s
i
d
e
r
,
I
n
s
i
d
e
r
)
T
o
o
l
T
y
p
e
(
N
a
t
i
v
e
,
F
o
r
e
i
g
n
)
S
y
s
t
e
m
l
o
a
d
/
N
u
m
b
e
r
o
f
u
s
e
r
s
(
L
o
w
,
H
i
g
h
)
L
e
v
e
l
o
f
d
e
f
e
n
d
e
r
p
a
r
t
i
c
i
p
a
t
i
o
n
(
U
s
e
r
s
o
n
l
y
,
U
s
e
r
s
+
l
o
c
a
l
d
e
f
e
n
d
e
r
s
,
U
s
e
r
s
+
l
o
c
a
l
+
C
S
S
P
)
Examples
of
many
possibilities
16
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
•
Consider
a
sequential approach
–
–
First
stage
-
-
screen
for
potential
vulnerabilities
–
Second
stage
–
refine
test,
characterize
significance
of
factors
and
interactions
in
greater
detail
•
Cyber/system
SMEs
should
determine
which
interaction
effects
are
likely/interesting,
which
specific
response
variables
are
most
meaningful
•
Create
design
first,
then
update
based
on
specifics,
such
as
rules
of
engagement
(ROE)
and
disallowed
combinations,
while
considering
tradeoffs
–
Enables
effects/constraints
of
ROE
to
be
understood
•
Could include ability
to
control
for
learning effects
over
time
–
Would
need
to
randomize
to
the
extent
possible
and
collect
enough
data
to
be
able
to
include
coefficients
for
time
and
person
in
the
model
17
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Applying
the
Framework
to
Cyber
T&E
(Steps
2
–
3)
Applying
the
Framework
to
Cyber
T&E
(Steps
2
–
3)
One
such
model
could
be:
y
S
D
f
E
A
model
is
fit to data
to form an
empirical
relationship
between
the
response
variable
and
factor
settings
for
the
purposes
of:
--
Determining
which
factors
have
a
large
effect
on
the
response
--
Making
predictions
across
the
factor
space
(including
combinations
that
were
not
explicitly
tested)
--
Quantifying
uncertainty
in
test
results
Responses
:
Time
to
get
in/achieve
effect,
Thread
length,
Level
of
threat
required,
Time
to
detect/mitigate/restore,
Severity
of
mission
effects
Normally-
distributed
error
Estimated
model
coefficients
While
the
model
is
linear
in
its
parameters,
the
factors/responses
are
not
necessarily
linear
or
normal:
Time-
based
responses
are
likely
right-
skewed,
so
lognormal
regression
or
a
survival
model
may
be
appropriate
The
mission
effects
response
is
categorical
so
a
multinomial
logistic
regression
is
one
appropriate
modeling
choice
The
test
could
be
designed
to
allow
the
ability
to
include
additional
recorded
factors
(e.g.
tool/method,
time)
in
the
model
and
estimate
their
effects
18
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Example
2
–
Hypothetical
Command
and
Control
(C
2
)
System
D
e
v
e
l
o
p
T
e
s
t
D
e
s
i
g
n
19
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Web
Server
Application
Server
Data
Base
Directly
Connected
External
Systems
D-
System
1
D-
System
2
D-
System
3
D-
System
4
D-
System
5
D-
System
6
External
Systems
E-
System
1
E-
System
2
E-
System
3
E-
System
4
E-
System
5
E-
System
6
E-
System
7
E-
System
8
E-
System
9
E-
System
10
E-
System
11
E-
System
12
E-
System
13
E-
System
14
User
1
User
2
=
Protocol/Entry
Point
=
Objective
P
1
P
2
P
3
P
=
Protocol
P
4
P
5
P
6
P
7
Hypothetical
C2
System
20
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Maintenance
Protocols
Design
for
Cooperative
Test
(1
of
2)
•
Create
a
design
using
the
5
varied
factors
presented
earlier
•
For
the
cooperative
test,
cover
the
space
of
all
entry
point/protocol
combinations
(an
8-
level
factor)
•
Focus
on
main
effects
•
Can
choose
more
than
the
minimum
number
of
runs
enabling
additional
covariates
to
be
included
in
the
statistical
model
during
analysis
•
Forty
runs
(attempted
penetrations)
chosen
as
an
example,
but
more
usually
better
21
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Design
for
Cooperative
Test
(2
of
2)
•
The
resulting
40
run
design
provides
coverage
(albeit
sparse)
of
the
8
X
3
X
3
X
4
=
288
factor
space
22
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Distribution
Statement
A:
Approved
for
public
release.
Distri
b
Cooperative
Test
Measures
of
Merit
•
The
design
is
sufficient
to
provide
high
power
to
detect
large
differences
(SNR=2)
in
main
effects
with
80%
confidence
•
There
is
necessarily
some
aliasing
in
the
design,
but
it
is
mostly
among
higher
order
terms.
Correlations
between
main
effects
are
very
low
and
not
a
concern
No
major
confounding
between
factors
23
ution
is
unlimited.
Analysis—
How
it
Might
Work
A
n
a
l
y
z
e
t
h
e
d
a
t
a
24
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Outsider
Nearsider
Insider
Starting
Posture
Level
of
Defender
Part.
P1
P2
P4
P3
P5
P6
P7
Maintainence
Protocol
Native
Foreign
Low
High
Example
Analysis
of
a
Continuous
Response
Variable
Protocol
/
Entry
Point
Level
of
Defender
Participation
Test
Point
to
Execute
E
x
e
c
u
t
e
t
h
e
T
e
s
t
25
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
C
a
p
t
u
r
e
t
h
e
D
a
t
a
Notional
distribution
of
the
continuous
response
variable
collected
from
the
40
test
points
Example
Analysis
of
a
Continuous
Response
Variable
Tool
Type
8
Protocols
A
f
t
e
r
e
x
e
c
u
t
i
n
g
t
h
e
t
e
s
t
,
w
e
c
a
n
p
e
r
f
o
r
m
a
n
e
x
p
l
o
r
a
t
o
r
y
a
n
a
l
y
s
i
s
.
O
b
s
e
r
v
a
t
i
o
n
s
c
o
n
s
i
d
e
r
i
n
g
t
h
r
e
e
o
f
t
h
e
f
a
c
t
o
r
s
i
n
c
l
u
d
e
N
a
t
i
v
e
T
o
o
l
s
a
p
p
e
a
r
t
o
h
a
v
e
h
i
g
h
e
r
r
e
s
p
o
n
s
e
s
t
h
a
n
F
o
r
e
i
g
n
T
o
o
l
s
,
a
s
d
o
I
n
s
i
d
e
r
A
t
t
a
c
k
s
.
T
h
e
r
e
a
l
s
o
a
p
p
e
a
r
t
o
b
e
s
o
m
e
d
i
f
f
e
r
e
n
c
e
s
i
n
r
e
s
p
o
n
s
e
s
a
c
r
o
s
s
t
h
e
P
r
o
t
o
c
o
l
s
.
Observed
Response
Response
Legend
High
Notional
Continuous
Response
Variable
Low
Protocol
/
Entry
Point
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
26
O
u
r
t
e
s
t
d
e
s
i
g
n
e
n
a
b
l
e
s
u
s
f
i
t
t
i
n
g
t
h
e
s
t
a
t
i
s
t
i
c
a
l
m
o
d
e
l
a
s
a
f
u
n
c
t
i
o
n
o
f
t
h
e
d
e
s
i
g
n
f
a
c
t
o
r
s
Example
Analysis
of
a
Continuous
Response
Variable
Native
Foreign
Tool
Type
8
Protocols
Protocol
/
Entry
Point
W
e
c
a
n
s
u
m
m
a
r
i
z
e
t
h
e
r
e
s
u
l
t
s
u
s
i
n
g
t
h
e
p
o
i
n
t
e
s
t
i
m
a
t
e
a
n
d
c
o
n
f
i
d
e
n
c
e
i
n
t
e
r
v
a
l
s
Statistical
difference
between
Native and
Foreign
tools
---and--
- Starting
Postures
Statistical
differences also
exist
between
some
of
the
Protocols
Notional
Results
Estimated
Mean
Notional
Results
Estimated
Mean
y
S
D
f
E
Observed
Response
F
r
o
m
t
h
e
m
o
d
e
l
f
i
t
,
w
e
s
e
e
t
h
a
t
s
o
m
e
f
a
c
t
o
r
s
h
a
v
e
a
n
e
f
f
e
c
t
o
n
t
h
e
N
o
t
i
o
n
a
l
C
o
n
t
i
n
u
o
u
s
R
e
s
p
o
n
s
e
V
a
r
i
a
b
l
e
O
u
t
s
i
d
e
r
N
e
a
r
s
i
d
e
r
I
n
s
i
d
e
r
27
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
Starting
Posture
Back-
up
28
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
PS
Design
Options
for
Assessment--
-
Single
Subsystem
Attacks
15
Subsystems
15
Subsystems
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
s
Assessing
65
potential
vulnerabilities
covers
120
combinations
with
80%
power
and
80%
confidence
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
29
A
t
t
a
c
k
P
o
s
t
u
r
e
PS
Design
Options
for
Assessment--
-
Attacks
Spanning
Two
Subsystems
1
5
S
u
b
s
y
s
t
e
m
s
8
S
u
b
s
y
s
t
e
m
s
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
1
Assessing
65
potential
vulnerabilities
covers
120
combinations
with
80%
power
and
80%
confidence
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
30
PS
Design
Options
for
Assessment--
-
Attacks
Spanning
Three
Subsystems
6
S
u
b
s
y
s
t
e
m
s
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
3
1
5
S
u
b
s
y
s
t
e
m
s
e
a
c
h
v
e
r
t
i
c
a
l
b
a
n
d
Assessing
70
potential
vulnerabilities
covers
1248
combinations
with
31
80%
power
and
80%
confidence
Distribution
Statement
A:
Approved
for
public
release.
Distribution
is
unlimited.
8
S
u
b
s
y
s
t
e
m
s
T
a
r
g
e
t
S
u
b
s
y
s
t
e
m
1
Exploring the application of Scientific Test and Analysis Techniques (STAT) for evaluating cyber performance, this publication highlights the benefits of data-driven analysis in cybersecurity testing. The guidebook emphasizes efficient coverage of vulnerabilities, quantitative risk assessment, and structured evaluation processes to enhance mission assurance in the Cyber Domain.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
CLEARED For Open Publication Jun 22, 2022 Department of Defense OFFICE OF PREPUBLICATION AND SECURITY REVIEW Applying Scientific Test and Analysis Techniques (STAT) to Testing and Evaluating Performance Across the Cyber Domain Mike Gilmore, Kelly Avery, Matt Girardi, John Hong, Rebecca Medlin July 2022 Institute for Defense Analyses 730 East Glebe Road Alexandria, Virginia 22305 Distribution Statement A: Approved for public release. Distribution is unlimited.
Evaluating Performance Across the Cyber Domain The core of the emerging National Defense Strategy will include integrated deterrence, a framework for working across warfighting domains, theaters and the spectrum of conflict. * Thus, successfully executing Multi-Domain Operations (MDO) will remain key to the strategy. Enabling successful execution will include robust performance across the Cyber Domain, an important and increasingly contested part of MDO. Comprehensive and efficient cybersecurity testing will be needed to rigorously evaluate mission assurance across the Cyber Domain. *See https://www.defense.gov/News/News-Stories/Article/Article/2954945/integrated-deterrence-at-center-of-upcoming- national-defense-strategy/, accessed March 7, 2022 1 Distribution Statement A: Approved for public release. Distribution is unlimited.
How Can STAT Benefit Cyber Testing and Evaluation? The DoD Cybersecurity T&E Guidebook promotes data-driven mission- impact-based analysis and assessment methods for cybersecurity test and evaluation... In that regard, Scientific Test and Analysis Techniques offers: Efficient coverage of operational space and potential vulnerabilities consistent with limited resources and time Objective and quantitative determination of how much testing is enough and risks of insufficient testing Identification and statistical quantification of significant factors/vulnerabilities Quantitative evaluation of what is lost if rules of engagement (ROE) are too constraining and/or time is too short Addition of structure to previously ad hoc test events, thereby aiding comprehensive evaluation, while not eliminating free play 2 Distribution Statement A: Approved for public release. Distribution is unlimited.
Framework for Applying STAT (or for Planning any Test and Evaluation) Test & Evaluation requires collaboration Determine scope of test Questions you can ask about the system Subject Matter Expertise Identify appropriate metrics Analytical Expertise How you should measure system performance Identify factors that affect performance STAT tools can be applied at each step Types of data to collect, operational envelope Develop Test Design Quantity of data necessary, best resource allocation, objective plans Conduct the test Adjust test execution if necessary Analyze the data Structured mathematical data analysis plan appropriate for the design Draw conclusions Defensible risk assessments based on test results 3 Distribution Statement A: Approved for public release. Distribution is unlimited.
Determine scope of test Where/what are the potential vulnerabilities? Example 1 Using STAT to Help Structure a Systematic Cyber Assessment of a Hypothetical Processing System (PS) 4 Distribution Statement A: Approved for public release. Distribution is unlimited.
Hypothetical PSComprises 15 Subsystems; 2 Operations Consoles How can STAT help? 1 Subsystem 1 2 Subsystem 2 STAT can be used to--- 3 Subsystem 3 4 Subsystem 4 5 Subsystem 5 6 Subsystem 6 7 Subsystem 7 8 Subsystem 8 9 Subsystem 9 10 Subsystem 10 11 Subsystem 11 12 Subsystem 12 13 Subsystem 13 14 Subsystem 14 15 Subsystem 15 16 Operations Console 1 17 Operations Console 2 Initially guide systematic assessments in narrowing the number of subsystems to be tested* Aid structuring the final tests Aid analysis of test results *Potential venues include Cyber Table Tops (CTTs) and other Mission-Based Cyber Risk Assessments (MBCRAs) 5 Distribution Statement A: Approved for public release. Distribution is unlimited.
Structuring a Systematic Cyber Assessment of a Hypothetical Processing System (PS) --Attacks on Single Subsystems Narrow the Number of Potential Vulnerabilities --Attacks Spanning Multiple Subsystems 6 Distribution Statement A: Approved for public release. Distribution is unlimited.
Options for Design of PS Cyber Assessment--- Single Subsystem Attacks 1 Subsystem 1 2 Subsystem 2 3 Subsystem 3 4 Subsystem 4 5 Subsystem 5 6 Subsystem 6 7 Subsystem 7 8 Subsystem 8 9 Subsystem 9 10 Subsystem 10 11 Subsystem 11 12 Subsystem 12 13 Subsystem 13 14 Subsystem 14 15 Subsystem 15 16 Operations Console 1 17 Operations Console 2 Consider entry using Operations Consoles---2-level factor (Entry) Remaining subsystems are targets---15-level factor (Target) PS Option 1: Operations Console 1, Operations Console 2 for Entry (2) Remaining Subsystems are Targets (15) Nearsider and Insider Attack Postures (2) Native, Foreign Tools (2) 120 Total Combinations Consider 68 percent (minimal) and 80 percent power to correctly assess/identify vulnerabilities to subsystems (true positive) Consider 80 percent confidence of correctly excluding vulnerabilities (true negative) 7 Distribution Statement A: Approved for public release. Distribution is unlimited.
PS Design Options for Assessment--- Single Subsystem Attacks Attack Posture 15 Subsystems 15 Subsystems Target Subsystems Assessing 45 potential vulnerabilities covers 120 combinations with 68% power and 80% confidence; 65 assessments required for 80% power Distribution Statement A: Approved for public release. Distribution is unlimited. 8
Structuring a Systematic Cyber Assessment of a Hypothetical Processing System (PS) --Attacks on Single Subsystems Narrow the Number of Potential Vulnerabilities --Attacks Spanning Multiple Subsystems 9 Distribution Statement A: Approved for public release. Distribution is unlimited.
Software Faults versus Number of Interacting Parameters ~87% to 99% of faults involve 3 parameters ~60% to 96% of faults involve 2 parameters Source: Kuhn, D., et al, Practical Combinatorial Testing, October 2010, available at https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-142.pdf, accessed January 14, 2022. PARAMETER = Input Data OR Configuration Treat Subsystems spanned as a Configuration 10 Distribution Statement A: Approved for public release. Distribution is unlimited.
Options for Design of PS Cyber Assessment--- Attacks Spanning Two Subsystems Suppose: Assessment of single subsystems described previously narrows focus to 8 subsystems for initial insider (only) penetration/attack through Operations Console 1 or 2; but--- Concern exists regarding attacks spanning more than one subsystem Consider attacks spanning those 8 subsystems and any one of the other 15-1 with the tool(s) used unspecified, but assumed to be those most applicable in each case as determined by prior assessment (e.g., specific native or foreign) PS Option 2: Operations Console 1, Operations Console 2 for Entry 8 Subsystems are first Targets (Target Subsystem 1) 14 Subsystems are second targets (Target Subsystem 2) Insider Attack Posture Most Applicable Tool 224 Total Combinations (2x8x14) 11 Distribution Statement A: Approved for public release. Distribution is unlimited.
PS Design Options for Assessment--- Attacks Spanning Two Subsystems 15 Subsystems Target Subsystem 1 8 Subsystems Assessing 50 potential vulnerabilities covers 224 combinations with 68% power and 80% confidence; 65 assessments for 80% power 12 Distribution Statement A: Approved for public release. Distribution is unlimited.
PS Design Options for Assessment--- Attacks Spanning Three Subsystems Suppose Further: Assessment of two-subsystem combinations narrows focus to 6 subsystems as second targets; but--- Concern exists regarding attacks spanning up to three subsystems Consider attacks spanning the identified 8 first targets, 6 second targets, and any one of the remaining 15-2 subsystems PS Option 3: Operations Console 1, Operations Console 2 for Entry 8 Subsystems as first Targets (Target Subsystem 1) 6 Subsystems as second targets (Target Subsystem 2) 13 Subsystems as third targets (Target Subsystem 3) Insider Attack Posture Most Applicable Tool 1248 Total Combinations (2x8x6x13) 13 Distribution Statement A: Approved for public release. Distribution is unlimited.
PS Design Options for Assessment--- Attacks Spanning Three Subsystems 6 Subsystems Target Subsystem 1 8 Subsystems Target Subsystem 3 15 Subsystems each vertical band Assessing 55 potential vulnerabilities covers 1248 combinations with 68% power and 80% confidence; 70 assessments for 80% power Distribution Statement A: Approved for public release. Distribution is unlimited. 14
Framework for Applying STAT (or for Planning any Test and Evaluation) Test & Evaluation requires collaboration Determine scope of test Demonstrated Questions you can ask about the system Subject Matter Expertise Identify appropriate metrics Analytical Expertise How you should measure system performance Identify factors that affect performance STATtools can be applied at each step Types of data to collect, operational envelope Develop Test Design How might this work? Quantity of data necessary, best resource allocation, objective plans Conduct the test Adjust test execution if necessary Analyze the data Structured mathematical data analysis plan appropriate for the design Draw conclusions Defensible risk assessments based on test results 15 Distribution Statement A: Approved for public release. Distribution is unlimited.
Applying the Framework to Cyber T&E (Steps 2 - 3) Objectives--- Cooperative test attempt to comprehensively identify vulnerabilities and validate exposures in system Adversarial test using the results of the cooperative test in as realistic setting as appropriate, assess system/users to protect, mitigate, and restore when faced with various types of cyber threats Potential response variables--- Attack thread length/number of steps Level of threat capability required to achieve action (Nascent, Limited, Moderate, Advanced) Severity of mission effects (None, Low, Med, High) (AA only) Time to detect / mitigate / restore Time to penetrate / achieve effect Examples of many possibilities Potential factors--- Protocol or objective (Web application, servers, interfaces with other systems, etc.) Type of cyber effect (Confidentiality, Integrity, Availability) Starting posture (Outsider, Near-sider, Insider) Tool Type (Native, Foreign) System load/Number of users (Low, High) Level of defender participation (Users only, Users + local defenders, Users + local + CSSP) 16 Distribution Statement A: Approved for public release. Distribution is unlimited.
Applying the Framework to Cyber T&E (Steps 2 3) Consider a sequential approach First stage -- screen for potential vulnerabilities Second stage refine test, characterize significance of factors and interactions in greater detail Cyber/system SMEs should determine which interaction effects are likely/interesting, which specific response variables are most meaningful Create design first, then update based on specifics, such as rules of engagement (ROE) and disallowed combinations, while considering tradeoffs Enables effects/constraints of ROE to be understood Could include ability to control for learning effects over time Would need to randomize to the extent possible and collect enough data to be able to include coefficients for time and person in the model 17 Distribution Statement A: Approved for public release. Distribution is unlimited.
Applying the Framework to Cyber T&E (Steps 2 3) A model is fit to data to form an empirical relationship between the response variable and factor settings for the purposes of: --Determining which factors have a large effect on the response --Making predictions across the factor space (including combinations that were not explicitly tested) --Quantifying uncertainty in test results Responses: Time to get in/achieve effect, Thread length, Level of threat required, Time to detect/mitigate/restore, Severity of mission effects One such model could be: y S D f E Normally-distributed error Estimated model coefficients While the model is linear in its parameters, the factors/responses are not necessarily linear or normal: Time-based responses are likely right-skewed, so lognormal regression or a survival model may be appropriate The mission effects response is categorical so a multinomial logistic regression is one appropriate modeling choice The test could be designed to allow the ability to include additional recorded factors (e.g. tool/method, time) in the model and estimate their effects 18 Distribution Statement A: Approved for public release. Distribution is unlimited.
Develop Test Design Example 2 Hypothetical Command and Control (C2) System 19 Distribution Statement A: Approved for public release. Distribution is unlimited.
Hypothetical C2 System P 1 P 2 P 3 P = Protocol User 1 User 2 Application Server Data Base External Systems E-System 1 E-System 2 E-System 3 E-System 4 E-System 5 E-System 6 E-System 7 E-System 8 E-System 9 E-System 10 E-System 11 E-System 12 E-System 13 E-System 14 Web Server P 4 P 5 Directly Connected External Systems D-System 1 D-System 2 D-System 3 D-System 4 D-System 5 D-System 6 P 6 P 7 Maintenance Protocols = Protocol/Entry Point = Objective 20 Distribution Statement A: Approved for public release. Distribution is unlimited.
Design for Cooperative Test (1 of 2) Create a design using the 5 varied factors presented earlier For the cooperative test, cover the space of all entry point/protocol combinations (an 8-level factor) Focus on main effects Can choose more than the minimum number of runs enabling additional covariates to be included in the statistical model during analysis Forty runs (attempted penetrations) chosen as an example, but more usually better 21 Distribution Statement A: Approved for public release. Distribution is unlimited.
Design for Cooperative Test (2 of 2) The resulting 40 run design provides coverage (albeit sparse) of the 8 X 3 X 3 X 4 = 288 factor space 22 Distribution Statement A: Approved for public release. Distribution is unlimited.
Cooperative Test Measures of Merit The design is sufficient to provide high power to detect large differences (SNR=2) in main effects with 80% confidence There is necessarily some aliasing in the design, but it is mostly among higher order terms. Correlations between main effects are very low and not a concern Term Power 0.77 0.99 0.99 1.00 1.00 Protocol/Entry Point Starting Posture Level of Defender Participation Tool Type Network Load/Traffic No major confounding between factors 23 ution is unlimited. Distribution Statement A: Approved for public release. Distrib
Analyze the data Analysis How it Might Work 24 Distribution Statement A: Approved for public release. Distribution is unlimited.
Example Analysis of a Continuous Response Variable Native Foreign Low High Starting Posture Outsider Nearsider Insider Maintainence Protocol P7 Protocol / Entry P6 Execute the Test P5 Point P4 P3 P2 Capture the Data P1 Test Point to Execute Level of Defender Part. Level of Defender Participation Notional distribution of the continuous response variable collected from the 40 test points 25 Distribution Statement A: Approved for public release. Distribution is unlimited.
Example Analysis of a Continuous Response Variable After executing the test, we can perform an exploratory analysis. Observations considering three of the factors include Native Tools appear to have higher responses than Foreign Tools, as do Insider Attacks. There also appear to be some differences in responses across the Protocols. Response Legend High Notional Continuous Response Variable Tool Type Low 8 Protocols Observed Response Protocol / Entry Point 26 Distribution Statement A: Approved for public release. Distribution is unlimited.
Example Analysis of a Continuous Response Variable Our test design enables us fitting the statistical model as a function of the design factors y S D f E Observed Response From the model fit, we see that some factors have an effect on the Notional Continuous Response Variable Statistical difference between Statistical differences also exist between some of the Protocols Native and Foreign tools ---and--- Starting Postures Notional Results Notional Results Estimated Mean Estimated Mean We can summarize the results using the point estimate and confidence intervals Native Foreign Outsider Nearsider Insider 8 Protocols Protocol / Entry Point Tool Type Starting Posture 27 Distribution Statement A: Approved for public release. Distribution is unlimited.
Back-up 28 Distribution Statement A: Approved for public release. Distribution is unlimited.
PS Design Options for Assessment--- Single Subsystem Attacks Attack Posture 15 Subsystems 15 Subsystems Target Subsystems Assessing 65 potential vulnerabilities covers 120 combinations with 80% power and 80% confidence Distribution Statement A: Approved for public release. Distribution is unlimited. 29
PS Design Options for Assessment--- Attacks Spanning Two Subsystems 15 Subsystems Target Subsystem 1 8 Subsystems Assessing 65 potential vulnerabilities covers 120 combinations with 80% power and 80% confidence Distribution Statement A: Approved for public release. Distribution is unlimited. 30
PS Design Options for Assessment--- Attacks Spanning Three Subsystems 6 Subsystems Target Subsystem 1 8 Subsystems Target Subsystem 3 15 Subsystems each vertical band Assessing 70 potential vulnerabilities covers 1248 combinations with 80% power and 80% confidence 31 Distribution Statement A: Approved for public release. Distribution is unlimited.
