OpenStack Upgrade Process Overview
J
u
n
o
t
o
O
c
a
t
a
Jacob Estelle
●
Multiple control planes across multiple clusters
●
Juno Nova, Keystone, Glance, and Horizon
●
RHEL 6
○
Only supports old version of Libvirt
●
MySQL DB
●
User Syncer
2
W
h
e
r
e
w
e
s
t
a
r
t
e
d
●
Upgrade the HVs from RHEL 6 to 7 for Libvirt
○
This can be done months before Ocata upgrade
●
Re-image the control plane to RHEL 7
●
Migrate the DB
●
Deploy Ocata to the control plane and HVs
●
Repeat steps for each cluster
3
H
i
g
h
-
L
e
v
e
l
S
t
r
a
t
e
g
y
●
Non-destructive
○
Nova instances stored on separate partition
●
Automated
○
Important for clusters with thousands of HVs
●
Main source of downtime
○
Re-imaging can take a couple of hours
○
Possible to avoid this by migrating instances to other HVs
during upgrade
4
H
V
U
p
g
r
a
d
e
F
o
r
L
i
b
v
i
r
t
●
Stop all OpenStack-related services
○
Keystone, Nova, Glance, Horizon, and user syncer
●
Clean up the DB
○
Remove lazy-deleted instances (for speed)
○
Remove instances stuck in transition state (required)
●
Do the DB migration to Ocata
5
C
o
n
t
r
o
l
P
l
a
n
e
U
p
g
r
a
d
e
P
a
r
t
1
●
Re-image control plane to RHEL7
●
Deploy Ocata OpenStack components
○
API components on freshly re-imaged host
○
Nova-compute on top of Juno HV
●
Start all OpenStack-related services
●
Ensure everything works
●
Add cluster to Horizon
6
C
o
n
t
r
o
l
P
l
a
n
e
U
p
g
r
a
d
e
P
a
r
t
2
●
Simple at a high-level
○
N → N+1 is doable
○
By induction, upgrade to any version
●
For release in range(Kilo, Ocata)
○
Download and install $release
○
Run DB migration command
7
D
a
t
a
b
a
s
e
M
i
g
r
a
t
i
o
n
:
S
t
r
a
t
e
g
y
●
Old versions of python packages are either broken or missing
●
Commands to upgrade change with versio
n
○
Order in which you run those commands matter
●
Expectation to bring up a cluster after upgrade
○
Know which extra, special commands you need to run
○
Kilo Nova forgot to add one of these special commands
8
D
B
M
i
g
r
a
t
i
o
n
:
C
o
m
p
l
i
c
a
t
i
o
n
s
●
Migrating customizations
○
Configuration, plugins, and patches
●
The Nova Network to Neutron migration
○
Ain’t nobody got time for that!
○
Force Nova Network to work with CellsV2
9
O
t
h
e
r
M
i
g
r
a
t
i
o
n
s
●
Downtime was minimal
○
Time to re-image and deploy Ocata
●
What upstream might improve
○
DB migration quirks
○
Backwards compatibility in general
●
Future work
○
Migrating Nova Network to Neutron
○
Ironic cluster migration (includes Neutron)
○
wiki.openstack.org/wiki/Fast_forward_upgrades
10
10
R
e
t
r
o
s
p
e
c
t
11
11
Q
u
e
s
t
i
o
n
s
JacobEstelle@oath.com
Detailed overview of the OpenStack upgrade process from RHEL 6 to RHEL 7, including strategies for upgrading control planes, high-level upgrade steps, database migration strategies, complications, and more.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Juno to Ocata Jacob Estelle
Where we started Multiple control planes across multiple clusters Juno Nova, Keystone, Glance, and Horizon RHEL 6 Only supports old version of Libvirt MySQL DB User Syncer 2
High-Level Strategy Upgrade the HVs from RHEL 6 to 7 for Libvirt This can be done months before Ocata upgrade Re-image the control plane to RHEL 7 Migrate the DB Deploy Ocata to the control plane and HVs Repeat steps for each cluster 3
HV Upgrade For Libvirt Non-destructive Nova instances stored on separate partition Automated Important for clusters with thousands of HVs Main source of downtime Re-imaging can take a couple of hours Possible to avoid this by migrating instances to other HVs during upgrade 4
Control Plane Upgrade Part 1 Stop all OpenStack-related services Keystone, Nova, Glance, Horizon, and user syncer Clean up the DB Remove lazy-deleted instances (for speed) Remove instances stuck in transition state (required) Do the DB migration to Ocata 5
Control Plane Upgrade Part 2 Re-image control plane to RHEL7 Deploy Ocata OpenStack components API components on freshly re-imaged host Nova-compute on top of Juno HV Start all OpenStack-related services Ensure everything works Add cluster to Horizon 6
Database Migration: Strategy Simple at a high-level N N+1 is doable By induction, upgrade to any version For release in range(Kilo, Ocata) Download and install $release Run DB migration command 7
DB Migration: Complications Old versions of python packages are either broken or missing Commands to upgrade change with version Order in which you run those commands matter Expectation to bring up a cluster after upgrade Know which extra, special commands you need to run Kilo Nova forgot to add one of these special commands 8
Other Migrations Migrating customizations Configuration, plugins, and patches The Nova Network to Neutron migration Ain t nobody got time for that! Force Nova Network to work with CellsV2 9
Retrospect Downtime was minimal Time to re-image and deploy Ocata What upstream might improve DB migration quirks Backwards compatibility in general Future work Migrating Nova Network to Neutron Ironic cluster migration (includes Neutron) wiki.openstack.org/wiki/Fast_forward_upgrades 10
Questions JacobEstelle@oath.com 11
