Optimizing Bazel Sandboxing with FUSE File System
Optimizing Bazel Sandboxing
with a FUSE File System
Overview of Bazel Sandboxing and sandboxfs
By Julio Merino (@jmmv) for FOSDEM on 2020-02-01
What is an Action in Bazel?
A
c
t
i
o
n
cc -I. -c parser.c -o parser.o
I
n
p
u
t
s
/usr/bin/cc
./parser.h
./parser.c
O
u
t
p
u
t
s
./parser.o
Bazel In-Memory Data Structures
Why Do We Sandbox Actions?
A
c
t
i
o
n
cc -I. -c parser.c -o parser.o
I
n
p
u
t
s
/usr/bin/cc
./parser.h
./parser.c
O
u
t
p
u
t
s
./parser.o
W
o
r
k
s
p
a
c
e
./lexer.h
./lexer.c
./parser.h
./parser.c
Bazel In-Memory Data Structures
File System
Action Sandboxing: Process Isolation
S
a
n
d
b
o
x
P
r
o
c
e
s
s
clever-cc input.c -o output.o
Mock hostname
Network
input.c
/dev/random
output.o
Real hostname
On Linux: user namespaces – On macOS: sandbox-exec
https://blog.bazel.build/2015/09/11/sandboxing.html
,
https://jmmv.dev/2019/11/macos-sandbox-exec.html
Action Sandboxing: File System Preparation
A
c
t
i
o
n
cc -I. -c parser.c -o parser.o
I
n
p
u
t
s
/usr/bin/cc
./parser.h
./parser.c
O
u
t
p
u
t
s
./parser.o
S
a
n
d
b
o
x
D
i
r
e
c
t
o
r
y
./parser.h (ro?)
./parser.c (ro?)
./ (rw)
W
o
r
k
s
p
a
c
e
./lexer.h
./lexer.c
./parser.h
./parser.c
W
o
r
k
s
p
a
c
e
./lexer.h
./lexer.c
./parser.h
./parser.c
./parser.o
Create
sandbox
Extract
outputs
Bazel In-Memory Data Structures
File System
Action Sandboxing: File System Preparation
A
c
t
i
o
n
cc -I. -c parser.c -o parser.o
I
n
p
u
t
s
/usr/bin/cc
./parser.h
./parser.c
O
u
t
p
u
t
s
./parser.o
S
a
n
d
b
o
x
D
i
r
e
c
t
o
r
y
./parser.h (ro?)
./parser.c (ro?)
./ (rw)
W
o
r
k
s
p
a
c
e
./lexer.h
./lexer.c
./parser.h
./parser.c
W
o
r
k
s
p
a
c
e
./lexer.h
./lexer.c
./parser.h
./parser.c
./parser.o
One symlink syscall per input
Scales poorly: 1ms extra per 20,000 actions = 20s extra wall time
Create
sandbox
Extract
outputs
Bazel In-Memory Data Structures
File System
sandboxfs: Using FUSE to Avoid Symlinks
https://github.com/bazelbuild/sandboxfs/
sandboxfs
Bazel
CreateSandbox(
action1
, {/
→ .../scratch/action1 (rw),
/src/file1.h
→
.../src/file1.h
(ro),
/src/file1.c
→
.../src/file1.c
(ro),
})
Mount Point
sandboxfs: Using FUSE to Avoid Symlinks
https://github.com/bazelbuild/sandboxfs/
sandboxfs
Bazel
CreateSandbox(
action1
, {/
→ .../scratch/action1 (rw),
/src/file1.h
→
.../src/file1.h
(ro),
/src/file1.c
→
.../src/file1.c
(ro),
})
Mount Point
action1/ (rw)
action1/src/file1.h (ro)
action1/src/file1.c (ro)
In-memory
operations
sandboxfs: Using FUSE to Avoid Symlinks
https://github.com/bazelbuild/sandboxfs/
sandboxfs
Bazel
CreateSandbox(
action1
, {/
→ .../scratch/action1 (rw),
/src/file1.h
→
.../src/file1.h
(ro),
/src/file1.c
→
.../src/file1.c
(ro),
})
Mount Point
action1/ (rw)
action1/src/file1.h (ro)
action1/src/file1.c (ro)
In-memory
operations
Same idea as:
mount --bind
(Linux)
mount -t null
(BSD)
sandboxfs: Using FUSE to Avoid Symlinks
https://github.com/bazelbuild/sandboxfs/
sandboxfs
Bazel
CreateSandbox(
action1
, {/
→ .../scratch/action1 (rw),
/src/file1.h
→
.../src/file1.h
(ro),
/src/file1.c
→
.../src/file1.c
(ro),
})
CreateSandbox(
action2
, {/
→
.../scratch/action2
(rw),
/gfx/libgfx.a
→
.../libgfx.a
(ro),
/main.a
→
.../main.a
(ro),
})
Mount Point
action1/ (rw)
action1/src/file1.h (ro)
action1/src/file1.c (ro)
action1/ (rw)
action1/src/file1.h (ro)
action1/src/file1.c (ro)
action2/ (rw)
action2/gfx/libgfx.a (ro)
action2/main.a (ro)
In-memory
operations
In-memory
operations
No remount!
Performance Results (macOS, circa 2018)
https://blog.bazel.build/2018/04/13/preliminary-sandboxfs-support.html
Performance Results (macOS, circa 2018)
https://blog.bazel.build/2018/04/13/preliminary-sandboxfs-support.html
Performance Results (macOS, circa 2018)
https://blog.bazel.build/2018/04/13/preliminary-sandboxfs-support.html
Rewriting sandboxfs from Go to Rust
Go
●
Easy to write (intern project!)
●
VSCode has great support
●
Hit scalability problems
●
Started to become hard to maintain (few
annotations in the code)
https://jmmv.dev/2018/07/rust-vs-go.html
Rewriting sandboxfs from Go to Rust
Go
●
Easy to write (intern project!)
●
VSCode has great support
●
Hit scalability problems
●
Started to become hard to maintain (few
annotations in the code)
Rust
●
Harder to write
●
VSCode has support but is very slow
●
Much more confident in the code
●
In translating the old Go code, Rust
uncovered many concurrency issues in it
https://jmmv.dev/2018/07/rust-vs-go.html
Rewriting sandboxfs from Go to Rust
Go
●
Easy to write (intern project!)
●
VSCode has great support
●
Hit scalability problems
●
Started to become hard to maintain (few
annotations in the code)
Rust
●
Harder to write
●
VSCode has support but is very slow
●
Much more confident in the code
●
In translating the old Go code, Rust
uncovered many concurrency issues in it
Commonalities
●
pprof for finding performance issues
●
FUSE bindings not first-class
https://jmmv.dev/2018/07/rust-vs-go.html
Future Work
●
Optimize further
○
Current Bazel ↔ sandboxfs protocol very inefficient
●
pkg_comp
●
Other sandboxing approaches (Microsoft's BuildXL)
●
But... beware of OSXFUSE and kexts on Mac
Thank You!
https://bazel.build/
https://github.com/bazelbuild/sandboxfs/
https://jmmv.dev/
https://twitter.com/jmmv/
Overview of Bazel sandboxes and the implementation of sandboxfs using FUSE to enhance sandboxing capabilities. Details include the concept of actions in Bazel, the rationale behind sandboxing actions, process isolation, file system preparation, and the usage of sandboxfs to avoid symlinks.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Optimizing Bazel Sandboxing with a FUSE File System Overview of Bazel Sandboxing and sandboxfs By Julio Merino (@jmmv) for FOSDEM on 2020-02-01
What is Bazel? https://bazel.build/
What is an Action in Bazel? Bazel In-Memory Data Structures Inputs /usr/bin/cc ./parser.h ./parser.c Action Outputs ./parser.o cc -I. -c parser.c -o parser.o
Why Do We Sandbox Actions? Bazel In-Memory Data Structures Inputs /usr/bin/cc ./parser.h ./parser.c Action Outputs ./parser.o cc -I. -c parser.c -o parser.o File System Workspace ./lexer.h ./lexer.c ./parser.h ./parser.c
Action Sandboxing: Process Isolation Sandbox Mock hostname Real hostname Process Network clever-cc input.c -o output.o input.c output.o /dev/random On Linux: user namespaces On macOS: sandbox-exec https://blog.bazel.build/2015/09/11/sandboxing.html, https://jmmv.dev/2019/11/macos-sandbox-exec.html
Action Sandboxing: File System Preparation Bazel In-Memory Data Structures Inputs /usr/bin/cc ./parser.h ./parser.c Action Outputs ./parser.o cc -I. -c parser.c -o parser.o File System Workspace ./lexer.h ./lexer.c ./parser.h ./parser.c ./parser.o Workspace ./lexer.h ./lexer.c ./parser.h ./parser.c Sandbox Directory ./parser.h (ro?) ./parser.c (ro?) ./ (rw) Create sandbox Extract outputs
Action Sandboxing: File System Preparation Bazel In-Memory Data Structures Inputs /usr/bin/cc ./parser.h ./parser.c Action Outputs ./parser.o cc -I. -c parser.c -o parser.o File System Workspace ./lexer.h ./lexer.c ./parser.h ./parser.c ./parser.o Workspace ./lexer.h ./lexer.c ./parser.h ./parser.c Sandbox Directory ./parser.h (ro?) ./parser.c (ro?) ./ (rw) Create sandbox Extract outputs One symlink syscall per input Scales poorly: 1ms extra per 20,000 actions = 20s extra wall time
sandboxfs: Using FUSE to Avoid Symlinks Bazel sandboxfs Mount Point CreateSandbox(action1, { / .../scratch/action1 (rw), /src/file1.h .../src/file1.h (ro), /src/file1.c .../src/file1.c (ro), }) https://github.com/bazelbuild/sandboxfs/
sandboxfs: Using FUSE to Avoid Symlinks Bazel sandboxfs Mount Point CreateSandbox(action1, { / .../scratch/action1 (rw), /src/file1.h .../src/file1.h (ro), /src/file1.c .../src/file1.c (ro), }) In-memory operations action1/ (rw) action1/src/file1.h (ro) action1/src/file1.c (ro) https://github.com/bazelbuild/sandboxfs/
sandboxfs: Using FUSE to Avoid Symlinks Bazel sandboxfs Mount Point CreateSandbox(action1, { / .../scratch/action1 (rw), /src/file1.h .../src/file1.h (ro), /src/file1.c .../src/file1.c (ro), }) In-memory operations action1/ (rw) action1/src/file1.h (ro) action1/src/file1.c (ro) Same idea as: mount --bind (Linux) mount -t null (BSD) https://github.com/bazelbuild/sandboxfs/
sandboxfs: Using FUSE to Avoid Symlinks Bazel sandboxfs Mount Point CreateSandbox(action1, { / .../scratch/action1 (rw), /src/file1.h .../src/file1.h (ro), /src/file1.c .../src/file1.c (ro), }) In-memory operations action1/ (rw) action1/src/file1.h (ro) action1/src/file1.c (ro) CreateSandbox(action2, { / .../scratch/action2 (rw), /gfx/libgfx.a .../libgfx.a (ro), /main.a .../main.a (ro), }) No remount! action1/ (rw) action1/src/file1.h (ro) action1/src/file1.c (ro) action2/ (rw) action2/gfx/libgfx.a (ro) action2/main.a (ro) In-memory operations https://github.com/bazelbuild/sandboxfs/
Performance Results (macOS, circa 2018) Target Machine Local Bazel MacBook Pro 2017 581s Bazel Mac Pro 2013 247s iOS app Mac Pro 2013 1235s https://blog.bazel.build/2018/04/13/preliminary-sandboxfs-support.html
Performance Results (macOS, circa 2018) Target Machine Local Symlinked sandbox Bazel MacBook Pro 2017 581s 621s (+6%) Bazel Mac Pro 2013 247s 265s (+7%) iOS app Mac Pro 2013 1235s 4572s (+270%) https://blog.bazel.build/2018/04/13/preliminary-sandboxfs-support.html
Performance Results (macOS, circa 2018) Target Machine Local Symlinked sandbox sandboxfs sandbox Bazel MacBook Pro 2017 581s 621s (+6%) 612s (+5%) Bazel Mac Pro 2013 247s 265s (+7%) 250s (+1%) iOS app Mac Pro 2013 1235s 4572s (+270%) 1922s (+55%) https://blog.bazel.build/2018/04/13/preliminary-sandboxfs-support.html
Rewriting sandboxfs from Go to Rust Go Easy to write (intern project!) VSCode has great support Hit scalability problems Started to become hard to maintain (few annotations in the code) https://jmmv.dev/2018/07/rust-vs-go.html
Rewriting sandboxfs from Go to Rust Go Rust Easy to write (intern project!) VSCode has great support Hit scalability problems Started to become hard to maintain (few annotations in the code) Harder to write VSCode has support but is very slow Much more confident in the code In translating the old Go code, Rust uncovered many concurrency issues in it https://jmmv.dev/2018/07/rust-vs-go.html
Rewriting sandboxfs from Go to Rust Go Rust Easy to write (intern project!) VSCode has great support Hit scalability problems Started to become hard to maintain (few annotations in the code) Harder to write VSCode has support but is very slow Much more confident in the code In translating the old Go code, Rust uncovered many concurrency issues in it Commonalities pprof for finding performance issues FUSE bindings not first-class https://jmmv.dev/2018/07/rust-vs-go.html
Future Work Optimize further Current Bazel pkg_comp Other sandboxing approaches (Microsoft's BuildXL) But... beware of OSXFUSE and kexts on Mac sandboxfs protocol very inefficient
Thank You! https://bazel.build/ https://github.com/bazelbuild/sandboxfs/ https://jmmv.dev/ https://twitter.com/jmmv/
