Exploring Virtually Networked FreeBSD Jails
Virtually-Networked
FreeBSD Jails
Shawn Webb
shwebb@wayfair.com
BSDCan 2012
Who Am I
•
S
o
f
t
w
a
r
e
E
n
g
i
n
e
e
r
a
n
d
S
e
c
u
r
i
t
y
A
n
a
l
y
s
t
f
o
r
W
a
y
f
a
i
r
L
L
C
•
I
n
d
e
p
e
n
d
e
n
t
s
e
c
u
r
i
t
y
r
e
s
e
a
r
c
h
e
r
•
T
e
c
h
b
l
o
g
g
e
r
•
D
i
s
c
l
a
i
m
e
r
:
a
n
y
b
e
l
i
e
f
s
,
o
p
i
n
i
o
n
s
,
e
t
c
.
a
r
e
m
i
n
e
a
n
d
d
o
n
o
t
n
e
c
e
s
s
a
r
i
l
y
r
e
f
l
e
c
t
t
h
o
s
e
o
f
m
y
e
m
p
l
o
y
e
r
What’s Covered
•
Q
u
i
c
k
h
i
s
t
o
r
y
1.
Jails, virtual networking, and ZFS
•
S
e
t
t
i
n
g
u
p
v
i
r
t
u
a
l
n
e
t
w
o
r
k
i
n
g
•
B
a
s
i
c
j
a
i
l
i
n
g
•
C
o
m
b
i
n
i
n
g
v
i
r
t
u
a
l
n
e
t
w
o
r
k
,
j
a
i
l
i
n
g
,
a
n
d
Z
F
S
•
F
u
t
u
r
e
w
o
r
k
History of Jails
•
I
n
t
r
o
d
u
c
e
d
i
n
F
r
e
e
B
S
D
4
.
x
b
y
P
o
u
l
-
H
e
n
n
i
n
g
K
a
m
p
•
C
o
n
t
i
n
u
o
u
s
l
y
b
e
i
n
g
i
m
p
r
o
v
e
d
•
S
e
c
u
r
e
r
e
p
l
a
c
e
m
e
n
t
f
o
r
c
h
r
o
o
t
•
O
S
-
b
a
s
e
d
v
i
r
t
u
a
l
i
z
a
t
i
o
n
•
I
n
s
p
i
r
e
d
S
o
l
a
r
i
s
’
Z
o
n
e
s
/
C
o
n
t
a
i
n
e
r
s
History of Virtual Networking
•
C
a
l
l
e
d
V
I
M
A
G
E
(
o
r
v
n
e
t
)
•
W
o
r
k
s
t
a
r
t
e
d
i
n
7
-
C
U
R
R
E
N
T
•
O
f
f
i
c
i
a
l
f
e
a
t
u
r
e
i
n
9
-
R
E
L
E
A
S
E
/
9
-
S
T
A
B
L
E
•
A
n
a
l
o
g
o
u
s
t
o
S
o
l
a
r
i
s
C
r
o
s
s
b
o
w
1.
Not as feature-complete as Crossbow
•
R
e
a
s
o
n
s
t
o
u
s
e
V
I
M
A
G
E
1.
Network security
2.
NATing jails
History of ZFS
•
T
h
e
G
o
d
o
f
f
i
l
e
s
y
s
t
e
m
s
•
Z
F
S
f
i
r
s
t
i
n
t
e
g
r
a
t
e
d
o
n
0
6
A
p
r
i
l
2
0
0
7
•
z
p
o
o
l
v
2
8
m
e
r
g
e
d
i
n
t
o
8
-
S
T
A
B
L
E
a
n
d
i
n
9
-
R
E
L
E
A
S
E
/
9
-
S
T
A
B
L
E
•
M
a
n
y
w
o
n
d
e
r
f
u
l
f
e
a
t
u
r
e
s
1.
New, powerful features coming from Delphix and Joyent
Reasons to Combine all Three
•
B
a
s
i
c
c
l
o
u
d
-
l
i
k
e
i
n
f
r
a
s
t
r
u
c
t
u
r
e
1.
ZFS for instant snapshots and clones
2.
vnet for VLANs and private networks
3.
Jails for VMs
•
G
o
t
c
h
a
’
s
1.
No pf or ipf support
2.
Must use IPFW
•
Special kernel config
1.
Enable VIMAGE, IPFW
•
Set up firewall
•
Set up NAT
1.
Not required, but useful
# Kernel Config
options VIMAGE
options IPFIREWALL
options IPDIVERT
# rc.conf NAT
gateway_enable=“YES”
firewall_enable=“YES”
firewall_type=“OPEN” # Change!
natd_enable=“YES”
natd_interface=“em0” # Change!
natd_flags=“”
Setting up Virtual Networking
•
Create bridge
•
epair devices
1.
Pair of two ifconfig-able devices
(epair[n]{a,b})2.
Two ends of an ethernet cable
3.
Plug one end into bridge
4.
Plug other end into jail
ifconfig bridge0 create
ifconfig epair0 create
ifconfig bridge0 inet
192.168.2.1
ifconfig bride0 addm epair0a
ifconfig epair0a up
ifconfig epair0b vnet [jail]
jexec [jail] ifconfig epair0b
inet 192.168.2.2
jexec [jail] route add default
192.168.2.1
Setting up Virtual Networking
•
Use ZFS to create template jail dataset
1.
Create snapshot
2.
Attack of the clones
•
Install world/distribution
•
Install ports tree
•
Install ports
# Initial installation
D=/jails/template
zfs create –omountpoint=/jails
tank/jails
zfs create tank/jails/template
cd /usr/src
make installworld DESTDIR=$D
make distribution DESTDIR=$D
portsnap –p $D/usr/ports fetch
extract
# Set default route, DNS
resolution
echo ‘nameserver 4.4.4.4’ > \
$D/etc/resolv.conf
# Set up temporary vnet
ifconfig bridge0 create
ifconfig bridge0 inet 192.168.2.1
ifconfig epair0 create
ifconfig bridge0 addm epair0
ifconfig epair0 up
Setting up Basic Jailing
# Start the jail and set up networking in it
jail –c vnet host.hostname=template name=template
path=/jails/template
ifconfig epair0b vnet template
jexec template ifconfig epair0b inet 192.168.2.2
jexec template route add default 192.168.21.1
# Install ports
jexec template sh
*** NOW IN JAIL ***
cd /usr/ports/security/sudo
make install clean distclean
*** EXIT JAIL ***
# Snapshot for clones
zfs snapshot tank/jails/template@date
# New jail:
zfs clone tank/jails/template@date tank/jails/newjail
So Many Commands!
•
A
l
o
t
o
f
i
n
i
t
i
a
l
w
o
r
k
•
T
a
k
e
s
a
l
o
t
o
f
t
i
m
e
•
P
r
o
b
l
e
m
s
:
1.
FreeBSD’s rc.d does not support vnet jails
2.
People reporting kernel panics destroying epair devices
•
I have had one or two kernel panics
Making it Easy
•
I
’
v
e
w
r
i
t
t
e
n
a
D
r
u
p
a
l
m
o
d
u
l
e
t
o
a
d
m
i
n
v
n
e
t
j
a
i
l
s
•
S
h
o
u
l
d
s
u
p
p
o
r
t
I
P
v
6
o
u
t
-
o
f
-
t
h
e
-
b
o
x
•
P
l
a
n
s
:
1.
epair ifconfig aliases
2.
Reporting
3.
Privilege separation
4.
External API
5.
Make vnet optional
•
W
i
l
l
n
o
t
s
u
p
p
o
r
t
n
o
n
-
Z
F
S
s
e
t
u
p
s
•
h
t
t
p
s
:
/
/
g
i
t
h
u
b
.
c
o
m
/
l
a
t
t
e
r
a
/
d
r
u
p
a
l
-
j
a
i
l
a
d
m
i
n
•
W
i
l
l
r
e
l
e
a
s
e
a
n
e
w
v
e
r
s
i
o
n
a
t
t
h
e
e
n
d
o
f
t
o
d
a
y
Demo
•
D
e
m
o
o
f
c
r
e
a
t
i
n
g
a
t
e
m
p
l
a
t
e
j
a
i
l
f
r
o
m
s
c
r
a
t
c
h
•
D
e
m
o
o
f
u
s
i
n
g
j
a
i
l
a
d
m
i
n
D
r
u
p
a
l
m
o
d
u
l
e
Future Work
•
O
b
v
i
o
u
s
r
c
.
d
s
u
p
p
o
r
t
•
D
t
r
a
c
e
s
u
p
p
o
r
t
1.
Like Solaris Zones
2.
Metrics
3.
Debugging
•
C
o
m
p
l
e
t
e
v
i
r
t
u
a
l
i
z
a
t
i
o
n
1.
Certain resources still shared (i.e. 127.0.0.1)
•
K
V
M
i
n
a
j
a
i
l
?
(
N
e
e
d
K
V
M
f
i
r
s
t
)
Delve into the history and setup of FreeBSD Jails, Virtual Networking, and ZFS. Understand the reasons to combine them for cloud-like infrastructure and learn to set up virtual networking for improved security and network configurations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
BSDCan 2012 Virtually-Networked FreeBSD Jails Shawn Webb shwebb@wayfair.com
Who Am I Software Engineer and Security Analyst for Wayfair LLC Independent security researcher Tech blogger Disclaimer: any beliefs, opinions, etc. are mine and do not necessarily reflect those of my employer 2
Whats Covered Quick history 1. Jails, virtual networking, and ZFS Setting up virtual networking Basic jailing Combining virtual network, jailing, and ZFS Future work 3
History of Jails Introduced in FreeBSD 4.x by Poul-Henning Kamp Continuously being improved Secure replacement for chroot OS-based virtualization Inspired Solaris Zones/Containers 4
History of Virtual Networking Called VIMAGE (or vnet) Work started in 7-CURRENT Official feature in 9-RELEASE/9-STABLE Analogous to Solaris Crossbow 1. Not as feature-complete as Crossbow Reasons to use VIMAGE 1. Network security 2. NATing jails 5
History of ZFS The God of filesystems ZFS first integrated on 06 April 2007 zpool v28 merged into 8-STABLE and in 9-RELEASE/9-STABLE Many wonderful features 1. New, powerful features coming from Delphix and Joyent 6
Reasons to Combine all Three Basic cloud-like infrastructure 1. ZFS for instant snapshots and clones 2. vnet for VLANs and private networks 3. Jails for VMs Gotcha s 1. No pf or ipf support 2. Must use IPFW 7
Setting up Virtual Networking # Kernel Config options VIMAGE options IPFIREWALL options IPDIVERT Special kernel config 1. Enable VIMAGE, IPFW Set up firewall Set up NAT 1. Not required, but useful # rc.conf NAT gateway_enable= YES firewall_enable= YES firewall_type= OPEN # Change! natd_enable= YES natd_interface= em0 # Change! natd_flags= 8
Setting up Virtual Networking ifconfig bridge0 create ifconfig epair0 create ifconfig bridge0 inet 192.168.2.1 ifconfig bride0 addm epair0a ifconfig epair0a up Create bridge epair devices 1. Pair of two ifconfig-able devices (epair[n]{a,b}) 2. Two ends of an ethernet cable 3. Plug one end into bridge 4. Plug other end into jail ifconfig epair0b vnet [jail] jexec [jail] ifconfig epair0b inet 192.168.2.2 jexec [jail] route add default 192.168.2.1 9
Setting up Basic Jailing # Initial installation D=/jails/template zfs create omountpoint=/jails tank/jails zfs create tank/jails/template cd /usr/src make installworld DESTDIR=$D make distribution DESTDIR=$D portsnap p $D/usr/ports fetch extract Use ZFS to create template jail dataset 1. Create snapshot 2. Attack of the clones Install world/distribution Install ports tree Install ports # Set default route, DNS resolution echo nameserver 4.4.4.4 > \ $D/etc/resolv.conf # Set up temporary vnet ifconfig bridge0 create ifconfig bridge0 inet 192.168.2.1 ifconfig epair0 create ifconfig bridge0 addm epair0 ifconfig epair0 up 10
# Start the jail and set up networking in it jail c vnet host.hostname=template name=template path=/jails/template ifconfig epair0b vnet template jexec template ifconfig epair0b inet 192.168.2.2 jexec template route add default 192.168.21.1 # Install ports jexec template sh *** NOW IN JAIL *** cd /usr/ports/security/sudo make install clean distclean *** EXIT JAIL *** # Snapshot for clones zfs snapshot tank/jails/template@date # New jail: zfs clone tank/jails/template@date tank/jails/newjail 11
So Many Commands! A lot of initial work Takes a lot of time Problems: 1. FreeBSD s rc.d does not support vnet jails 2. People reporting kernel panics destroying epair devices I have had one or two kernel panics 12
Making it Easy I ve written a Drupal module to admin vnet jails Should support IPv6 out-of-the-box Plans: 1. epair ifconfig aliases 2. Reporting 3. Privilege separation 4. External API 5. Make vnet optional Will not support non-ZFS setups https://github.com/lattera/drupal-jailadmin Will release a new version at the end of today 13
Demo Demo of creating a template jail from scratch Demo of using jailadmin Drupal module 14
Future Work Obvious rc.d support Dtrace support 1. Like Solaris Zones 2. Metrics 3. Debugging Complete virtualization 1. Certain resources still shared (i.e. 127.0.0.1) KVM in a jail? (Need KVM first) 15
