The Difference Between LLVM Profile-Instr-Generate and Profile-Generate Options
EXERCISE #28
1
LLVM INSTRUMENTATION REVIEW
Write your name and answer the following on a piece of paper
Describe the difference between the profile-instr-generate and profile-generate options
for LLVM instrumentation?
ADMINISTRIVIA
AND
ANNOUNCEMENTS
Free exercises from last week
FUZZING
EECS 677: Software Security Evaluation
Drew Davidson
WHERE WE’RE AT
DYNAMIC INSTRUMENTATION
4
Use the execution of a program to find
(security) bugs
Necessarily dependent on encountered
execution behavior
5
PREVIOUSLY: LLVM INSTRUMENTATION
REVIEW: LAST LECTURE
Described commands to use PGO for line
coverage analysis
S
ETUP
FOR
A
C
USTOM
LLVM
A
NALYSIS
U
SAGE
OF
LLVM
BUILT
-
IN
INSTRUMENTATION
A
NALYSIS
Described the basic infrastructure necessary to
craft a custom instrumentation
6
THIS LESSON: FUZZING
OUTLINE / OVERVIEW
G
ENERATING
GOOD
TEST
CASES
Cases that exercise unexpected behavior
Cases tha
t increase coverage of program behaviors
P
REVIOUS
STABS
AT
THIS
TOPIC
The random “fuzz” of white noise
Consider testing as an intrinsic part of the SSDLC
methodology
Test-driven development
Post-hoc evaluation via coverage metrics
T
ODAY
: J
UST
GUESS
7
HISTORY OF FUZZING
OUTLINE / OVERVIEW
1988: I
T
WAS
A
D
ARK
AND
S
TORMY
N
IGHT
Professor Bart Miller attempts to work from home…
Telnet
Connection
noise
Nonsense
Commands
Program
Crash!
Well-formed
Commands
8
BREAKING CIRCULAR LOGIC
OUTLINE / OVERVIEW
A
UTOMATED
TEST
CASE
GENERATION
RESOLVES
A
FUNDAMENTAL
CONFLICT
IN
TESTING
…
Tautologically
impossible to predict u
npredictable
behavior
Apply a technique that obviated the need for
expectations
9
GRACEFUL FAILURE
OUTLINE / OVERVIEW
Any error should be anticipated and handled by the
system, with an informative error message should
recovery become impossible
A
KEY
PRINCIPLE
IN
THE
VALIDITY
OF
FUZZING
“The user should never see a seg fault”
10
THE SIMPLEST FUZZER
FUZZ TESTING
T
HE
M
OST
B
ASIC
F
ORM
OF
F
UZZING
cat /dev/random | program
A study in the 90s basically did this, finding bugs in…
adb, as, bc, cb, col, diction, emacs, eqn, ftp, indent, lex,
look, m4, make, nroff, plot, prolog, ptx, refer!, spell, style,
tsort, uniq, vgrind, vi
11
EXPLORING UNEXPECTED BEHAVIOR
FUZZING
R
ANDOM
INPUT
IS
SURPRISINGLY
EFFECTIVE
Numerous bugs found in practice via fuzzing…
Busybox utilities
Windows bugs
Linux Kernel bugs
B
ENEFITS
OF
FUZZING
Very easy to run
Instant results
Highly scalable
12
PRIORITIZING INPUT
FUZZING
T
HE
CHALLENGE
OF
FUZZERS
IS
(
USUALLY
)
GETTING
PAST
THE
FIRST
VALIDATION
CHECK
if (!sane_input()){exit 1;
}
//The rest of the program
13
SIMPLE TESTING STRATEGY
FUZZING
C
ONSIDER
“I
NTERESTING
” I
NPUT
Values close to the maximum, minimum, middle, etc
C
ASE
S
TUDY
: C
ARD
READER
INPUT
:
[FRISBY ET AL., 2012]
14
MUTATION-BASED FUZZERS
FUZZING
E
XPLORE
DEVIATIONS
FROM
KNOWN
INPUT
Example mutations:
Binary input
– Bit flips
- Byte flips
- Change random bytes
- Insert random byte chunks
- Delete random byte chunks
- Set randomly chosen byte chunks to interesting
values e.g. INT_MAX, INT_MIN, 0, 1, -1, … §
Text input
- Insert random symbols or keywords from a dictionary
15
REPRESENTATIVE TOOL: AFL
FUZZING
AFL (A
MERICAN
F
UZZY
L
OP
)
Maintained by Google
S
TATE
OF
THE
ART
Generally considered the best, state-of-the-art fuzzer
16
REPRESENTATIVE TOOL: AFL
OUTLINE / OVERVIEW
E
XAMPLE
COMMAND
“
TRADITIONAL
F
UZZING
”
mkdir in_dir
echo “hello” > in_dir/hello
afl-fuzz -n -i in_dir -o out_dir cat
17
REPRESENTATIVE TOOL: AFL
FUZZING
I
NSTRUMENTATION
MODE
1)
Compile the program with
coverage probes
2)
Attempt to prioritize / mutate
test cases that extend
coverage
afl-clang++ <build command>
18
FUZZING ORACLES
FUZZING
B
EYOND
GRACEFUL
F
AILURE
In C/C++ there are a lot of violations of proper behavior that are invisible
“Seems fine until it’s a huge problem”
S
ANITIZERS
UBSan – Undefined behavior sanitizer
ASan – Address sanitizer
TSan – Thread sanitizer
19
RESEARCH DIRECTION: “GUNKING”
FUZZING
F
UZZING
AS
ADVERSARIAL
RECON
Fuzzing is so good at finding bugs that even the bad guys do it
P
ERHAPS
A
PROGRAM
SHOULD
DEPLOY
A
NTI
-F
UZZING
TECH
What would that look like?
WRAP-UP
INTRODUCED THE CONCEPT AND THE
“INDUSTRY STANDARD” TOOL OF FUZZING
20
A simple, elegant idea
The profile-instr-generate and profile-generate options in LLVM instrumentation serve distinct purposes. Profile-instr-generate generates instrumentation based on profiling data during compilation, aiding in performance optimization. In contrast, profile-generate is used to generate a profile based on instrumented code execution, facilitating feedback for further optimizations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
EXERCISE #28 LLVM INSTRUMENTATION REVIEW Write your name and answer the following on a piece of paper Describe the difference between the profile-instr-generate and profile-generate options for LLVM instrumentation? 1
Free exercises from last week ADMINISTRIVIA AND ANNOUNCEMENTS
FUZZING EECS 677: Software Security Evaluation Drew Davidson
4 WHERE WE RE AT DYNAMIC INSTRUMENTATION Use the execution of a program to find (security) bugs Necessarily dependent on encountered execution behavior
5 PREVIOUSLY: LLVM INSTRUMENTATION REVIEW: LAST LECTURE USAGEOF LLVM BUILT-IN INSTRUMENTATION ANALYSIS Described commands to use PGO for line coverage analysis SETUPFORA CUSTOM LLVM ANALYSIS Described the basic infrastructure necessary to craft a custom instrumentation
6 THIS LESSON: FUZZING OUTLINE / OVERVIEW GENERATINGGOODTESTCASES Cases that increase coverage of program behaviors Cases that exercise unexpected behavior PREVIOUSSTABSATTHISTOPIC Consider testing as an intrinsic part of the SSDLC methodology Test-driven development Post-hoc evaluation via coverage metrics The random fuzz of white noise TODAY: JUSTGUESS
7 HISTORY OF FUZZING OUTLINE / OVERVIEW 1988: ITWASA DARKAND STORMY NIGHT Professor Bart Miller attempts to work from home noise Program Telnet Connection Well-formed Commands Nonsense Commands Crash!
8 BREAKING CIRCULAR LOGIC OUTLINE / OVERVIEW AUTOMATEDTESTCASEGENERATIONRESOLVESA FUNDAMENTALCONFLICTINTESTING Tautologically impossible to predict unpredictable behavior Apply a technique that obviated the need for expectations
9 GRACEFUL FAILURE OUTLINE / OVERVIEW Any error should be anticipated and handled by the system, with an informative error message should recovery become impossible A KEYPRINCIPLEINTHEVALIDITYOFFUZZING The user should never see a seg fault
10 THE SIMPLEST FUZZER FUZZ TESTING THE MOST BASIC FORMOF FUZZING cat /dev/random | program A study in the 90s basically did this, finding bugs in adb, as, bc, cb, col, diction, emacs, eqn, ftp, indent, lex, look, m4, make, nroff, plot, prolog, ptx, refer!, spell, style, tsort, uniq, vgrind, vi
11 EXPLORING UNEXPECTED BEHAVIOR FUZZING RANDOMINPUTISSURPRISINGLYEFFECTIVE Numerous bugs found in practice via fuzzing Busybox utilities Windows bugs Linux Kernel bugs BENEFITSOFFUZZING Very easy to run Instant results Highly scalable
12 PRIORITIZING INPUT FUZZING THECHALLENGEOFFUZZERSIS (USUALLY) GETTINGPASTTHEFIRSTVALIDATION CHECK if (!sane_input()){ exit 1; } //The rest of the program
13 SIMPLE TESTING STRATEGY FUZZING CONSIDER INTERESTING INPUT Values close to the maximum, minimum, middle, etc CASE STUDY: CARDREADERINPUT: [FRISBY ET AL., 2012]
14 MUTATION-BASED FUZZERS FUZZING EXPLOREDEVIATIONSFROMKNOWNINPUT Example mutations: Binary input Bit flips - Byte flips - Change random bytes - Insert random byte chunks - Delete random byte chunks - Set randomly chosen byte chunks to interesting values e.g. INT_MAX, INT_MIN, 0, 1, -1, Text input - Insert random symbols or keywords from a dictionary
15 REPRESENTATIVE TOOL: AFL FUZZING AFL (AMERICAN FUZZY LOP) Maintained by Google STATEOFTHEART Generally considered the best, state-of-the-art fuzzer
16 REPRESENTATIVE TOOL: AFL OUTLINE / OVERVIEW EXAMPLECOMMAND TRADITIONAL FUZZING mkdir in_dir echo hello > in_dir/hello afl-fuzz -n -i in_dir -o out_dir cat
17 REPRESENTATIVE TOOL: AFL FUZZING INSTRUMENTATIONMODE 1) Compile the program with coverage probes 2) Attempt to prioritize / mutate test cases that extend coverage afl-clang++ <build command>
18 FUZZING ORACLES FUZZING BEYONDGRACEFUL FAILURE In C/C++ there are a lot of violations of proper behavior that are invisible Seems fine until it s a huge problem SANITIZERS UBSan Undefined behavior sanitizer ASan Address sanitizer TSan Thread sanitizer
19 RESEARCH DIRECTION: GUNKING FUZZING FUZZINGASADVERSARIALRECON Fuzzing is so good at finding bugs that even the bad guys do it PERHAPSAPROGRAMSHOULDDEPLOY ANTI-FUZZINGTECH What would that look like?
20 WRAP-UP INTRODUCED THE CONCEPT AND THE INDUSTRY STANDARD TOOL OF FUZZING A simple, elegant idea
