The POWERful Choice: Carrier Ethernet or MPLS for Power Utilities
undefined
The POWERful Choice –
Carrier Ethernet or MPLS
For Power Utilities
Yaakov (J) Stein
CTO
SONET/SDH is being phased out
SONET technology is widely deployed, but
•
SONET technology is aging
•
SONET equipment is becoming obsolete and hard to find
•
SONET is hard to maintain (parts hard to obtain and expensive)
•
finding staff with SONET expertise is becoming ever more difficult
•
no new rates/functionality/standards/applications are being developed for SONET
Modern packet-based networks (based on Ethernet, MPLS, and IP)
•
are the present and future
•
are broadband and becoming even more so
•
are less expensive (both CAPEX and OPEX) and more flexible
•
are being actively extended (e.g., migration to 61850)
But there are open questions
•
can all the relevant services be migrated to packet
(e.g., teleprotection, synchrophasors)
?
•
which packet-based network to choose ?
The options
•
Carrier Ethernet
–
Based on most popular technology in the world
–
Look and feel similar to SONET/SDH networks
–
Mature carrier-grade technology
–
Support for synchronization
–
Network security mechanisms available
•
MPLS
–
Core
network technology
–
Inherits rich IP control plane
–
Deterministic paths available (MPLS-TE)
–
Has no inherent network security
•
MPLS-TP
–
Based on MPLS, but adds mechanisms patterned after Carrier Ethernet
•
OAM and protection switching (including rings)
–
Look and feel similar to SONET/SDH networks
–
Does not require IP forwarding or control plane
–
Has no inherent network security
What is
Carrier
Ethernet
?
(1)
Ethernet
started out as a
LAN
technology
LAN networks are small
and operated by consumer
and hence are easily managed
When
Ethernet
left the LAN environment
new mechanisms were needed, e.g.
–
scalability
(to reach 100s of thousands of end-points)
–
OAM (
F
ault
M
anagement,
P
erformance
M
onitoring)
–
deterministic (Connection-Oriented) connections
–
support for various topologies (e.g., point-point, rings, trees)
–
resilience mechanisms (e.g., Automatic Protection Switching)
–
support for synchronization
Carrier Ethernet (CE)
adds
carrier-grade
features to Ethernet
so that it can replace
SONET/SDH as
a transport network
Metcalf’s original sketch of Ethernet
Blue
means Ethernet
What is
Carrier Ethernet
?
(2)
•
Mature Technology
–
widely deployed by service providers
–
promoted and maintained by Metro Ethernet Forum (MEF)
•
Deterministic and
C
onnection
O
riented
(unlike connectionless IP)
–
provisioning through management system (not routing)
–
support for point-point, multipoint-multipoint, ring, tree, … topologies
•
Support for
Q
uality
o
f
S
ervice
(
up to 8
C
lasses
o
f
S
ervice)
–
enforcement of bandwidth profiles (dual token bucket shaping/policing)
–
color (conformance) marking
•
Carrier-grade operations mechanisms
:
–
service activation testing (Y.1564)
–
F
ault
M
anagement (802.1ag, Y.1731)
–
P
erformance
M
onitoring (Y.1731)
–
A
utomatic
P
rotection
S
witching (G.8031, G.8032)
–
Synchronization <timing distribution> (SyncE, 1588)
•
Network security mechanisms
:
–
access authorization (802.1X)
–
source authentication, integrity and optional encryption (MACSec)
What is
MPLS
?
(1)
MPLS
started out as a technology to accelerate IP forwarding
by setting up tunnels to transport IP
other traffic can be transported via
pseudowires
MPLS
defined by the IETF, and inherits the rich IP protocol suite
like all IETF protocols, MPLS does not define layer 2 or below
MPLS
is a mature technology for
core
IP networks
full
T
raffic
E
ngineering is available, but not
traffic conditioning
(policing/shaping)
supports mesh topologies
uses local Fast ReRoute (not protection switching) for resilience
no network security mechanisms (since core elements are trusted)
A new MPLS version (
MPLS-TP
)
takes MPLS out of the core network into the transport domain
WARNING: there are two non-interoperable versions (from IETF and ITU-T)
Red
means MPLS
What is
MPLS
?
(2)
We can now distinguish four distinct flavors of MPLS:
1.
best effort
MPLS
(usually with LDP, perhaps with RSVP-TE for FRR)
not true CO – pinned to route not to Network Elements
used in Internet core
2.
MPLS for
L3VPN
services
(RFC 4364 <ex-2547> using BGP)
used to deliver VPN services to business users
3.
traffic engineered
MPLS-TE
(currently with RSVP-TE)
true CO with resource reservation
used when strict SLA guarantees must be given
(banks, government, …)
4.
transport profile -
MPLS-TP
(with management or RSVP-TE)
–
does not assume the existence of IP forwarding plane
–
does not require the IP control plane
(can work with management systems)
–
implements OAM and APS functionality
(based on Carrier Ethernet)
–
supports ring topologies
–
still in initial phases of deployment
(little
interop
testing has been performed)
–
does not add network security features
(still susceptible to attack)
The battlefront
•
Ethernet
started in the local network (LAN)
and for many years has moved into transport networks
•
MPLS
started in the core network (WAN)
and is now trying to conquer transport networks with MPLS-TP
Technical
Comparison
Features in common
Both
Ethernet
and
MPLS
(all flavors)
:
•
can natively transport IP traffic
–
Ethernet can natively transport other traffic types (EtherType)
–
MPLS can transport other traffic types via pseudowire technology
•
can be transported over
SONET/SDH
and
OTN
•
are being actively developed (by multiple standards organizations)
–
Ethernet by the IEEE, MEF, ITU, …
–
MPLS by the IETF, ITU-T, …
•
may exhibit
very high
or
very low
transit delays
(and everything in-between)
(unlike
SONET/SDH
which has constant switching latency)
–
very high delay when packets need to wait in a queue
–
very low delay (much lower than
SONET/SDH
) for prioritorized traffic
Both
CE
and
MPLS-TP
:
•
typically use network management systems for configuration
•
define FM/PM OAM and diagnostic tests
•
support rings and define APS
1
st
reason for differences – format
Ethernet packet headers are
self-describing
•
a globally unique source address
•
a globally unique destination address
•
an optional connection identifier (VLAN)
•
optional Class of Service and Drop Eligibility Indicator
•
a payload protocol type identifier (EtherType)
MPLS packet headers are only locally meaningful
•
no unique addresses
•
a locally meaningful label (stack)
•
a TTL field (to avoid packet looping)
•
optionally a Traffic Class (TC) field
2
nd
reason for differences – control
•
Ethernet was
zero-touch
in broadcast domain LANs
•
CE uses
network management
to support large networks
•
Ethernet does define L2 control protocols
(STP, LACP, LLDP, …)
but does not define a
routing
protocol
(neglecting TRILL, E-VPN, etc.)
•
Best effort MPLS tunnels according to topology found by IP
routing protocols
•
So
best effort
MPLS:
–
does not require sophisticated management system
–
does requires the full
logistics
of an IP network
•
MPLS-TE requires
both
IP routing
and
a sophisticated management system
•
MPLS-TP is the only flavor of MPLS that does not require IP routing
but when routing is not used, configuration management is required
(basically equivalent to Carrier Ethernet)
Additional differences
•
Ethernet defines physical (L1) layers (but may run over MPLS as a PW)
MPLS requires a server layer to transport it (which is usually Ethernet)
•
Ethernet can not tolerate forwarding loops
Carrier Ethernet supports rings with G.8032
and Industrial Ethernet supports them with
H
igh-availability
S
eamless
R
edundancy
MPLS can (since it contains a TTL field)
•
Carrier Ethernet supports bandwidth profiles (bucketing)
•
Ethernet supports IEEE 1588 timing distribution over packet
and defines a physical layer to support Synchronous Ethernet
MPLS may obtain support for 1588
(work ongoing in IETF)
but since MPLS does
not a physical layer it can not provide physical layer synchronization support
•
Ethernet has network security mechanisms (MACsec, 802.1X, SNMPv3)
MPLS does not define any standardized network security mechanisms
and since MPLS has no source address
it can not provide source authentication
The new trend – SDN
Distributed routing protocols are limited to
•
finding simple connectivity
•
minimizing number of hops
but can not perform more sophisticated operations
•
optimizing paths under constraints (e.g., delay, security)
•
setting up backup paths
•
integrating networking functionalities (e.g., NAT, firewall) into paths
Lately, a new paradigm has arisen –
S
oftware
D
erived
N
etworking, which:
•
removes control protocols from network elements
•
replaces distributed routing with centralized path computation
•
configures the forwarding actions of the switches from a central site
SDN sees the IP/MPLS control plane as a disadvantage
and adopts the
Carrier Ethernet
/
MPLS-TP
approach
New SDN tools can optimally manage operational networks
•
SDN services can be added and modified at the
speed of software
•
SDN should lead to significant OPEX reductions
Why not use both ? (1)
We have seen that
MPLS
is missing several critical features
in particular, synchronization and network security
So, why not use both
Ethernet
and
MPLS
taking the best features of each ?
In fact,
MPLS
does
not
define its own physical layer
and the most common physical layer supporting
MPLS
is
Ethernet
although
MPLS
can be transported over other physical layers, e.g.,
SDH
or
OTN
So the real question is whether to maintain an Ethernet network
or an MPLS network
in addition
to an Ethernet network !
Why not use both ? (2)
How many networks are there ?
Ethernet defines its own physical layer
although Ethernet
can
be transported over other physical layers
When transporting IP over Ethernet there are actually 2 or 3 networks
3
IP
2
Ethernet
1
Ethernet or optionally SONET/SDH or OTN
MPLS does
not
define its own physical layer
When transporting IP over MPLS there are actually 3 or 4 networks
3
IP
2
.5
MPLS
2
Ethernet
1
Ethernet or optionally SONET/SDH or OTN
Do we care how many networks there are ?
Why not use both ? (3)
Yes, because maintaining networks is never trivial or expense-free!
•
Attempts to design a network to use Ethernet as a
dumb pipe
under MPLS
usually end up using a large number of Ethernet mechanisms
•
For example, when running MPLS over Ethernet, one usually needs :
–
staff trained in Ethernet technologies
and
staff trained in IP/MPLS technologies
–
to be able to run Ethernet OAM
and
MPLS diagnostic tools
–
to maintain an Ethernet NMS
and
MPLS management screens
•
Network management is the core business of a network service provider
and for them it may be reasonable to maintain
duplicate staff, tools, operations centers, etc.
Network maintenance is
not
the core business of a power utility and the
duplication and added complexity is usually not justifiable
Operational
Comparison
Utilities network requirements
•
Traffic types
(not an exhaustive list)
–
SCADA operational traffic
–
teleprotection traffic
–
synchrophasor traffic
–
surveillance video
–
general TCP/IP
and there is a growing demand for bandwidth
•
Determinism
(CO behavior)
–
best effort / nondeterministic (Internet-like) behavior is not acceptable
•
Resilience
(critical infrastructures must be highly reliable)
•
Low (and constant) end-end delay
(for SCADA and teleprotection applications)
•
Management
–
networks presently employ centralized management
–
end-to-end provisioning and maintenance are
must
s
•
Synchronization
•
Network security
(merits discussion in a separate section)
–
cyber security is a growing concern
–
regulatory requirements are appearing
Traffic types
•
SONET/SDH
was designed to transport certain traffic types and rates
–
mapping new traffic types is difficult and complex
–
transport of most traffic rates is inefficient
–
no higher rates are being defined for SONET/SDH
•
Ethernet
was designed to transport arbitrary traffic types and rates
–
EtherType mechanism to indicate payload types
–
pseudowire technology may also be used
–
no rate constraints
–
higher rates being defined (presently 100Gbps)
•
MPLS
was designed to transport IP traffic
–
pseudowire technology enables transport of arbitrary traffic types
–
MPLS imposes no rate constraints or limitations
So, regarding traffic,
SONET/SDH
is reaching End-of-Life
while
Ethernet
and
MPLS
are future proof!
Determinism
Networks are deterministic
when traffic consistently flows through the network in the same way
With nondeterministic networks (e.g., IP and
best effort MPLS
)
each packet may take a different route through the network, thus
•
enabling intermittent faults
(only when the packets happen to go there)
•
complicating troubleshooting
(where did the packets go?)
•
excluding the reservation of resources or specific processing
at particular network elements
(you can’t be sure the packets will go where you want …)
SONET/SDH
networks are
C
ircuit
S
witched, and thus completely deterministic
CE
and some types of
MPLS
(TE, TP) are
C
onnection
O
riented
and thus relatively deterministic
traffic consistently takes the same path through the network
but does not always take precisely the same time to traverse
So, due to lack of determinism,
best effort MPLS
is
not
a reasonable candidate
for a power utility operational network
Resilience
•
SONET/SDH
is well-known for its Automatic Protection Switching
–
gold standard
1:1 APS supports < 50 millisecond protection switching time
–
1+1 APS can provide hitless switching (at the cost of increased bandwidth)
•
Best effort
MPLS
relies on slow
rerouting
for recovery
•
MPLS
with
F
ast
R
e
R
oute performs local detours around failures
–
at the expense of loss of determinism
•
CE
and
MPLS-TP
support several types of APS
–
CE’s G.8031 and G.8032 and MPLS-TP’s RFC 6378, 6974, ITU-T G.8131
–
1+1 pseudowire redundancy achieves hitless switching
at the cost of increased bandwidth consumption
So, from the point of view of resilience
CE
and
MPLS-TP
are as good as
SONET/SDH
!
End-end delay and delay consistency
Some operational traffic require low and consistent delay
For example, teleprotection’s end-end delay budget may be 6 milliseconds
•
SONET/SDH
latency is typically sufficiently low
(e.g., under 2 msec.)
–
is constant
–
is independent of SONET/SDH rate (whether OC3 or OC192)
•
Carrier Ethernet
and
MPLS
may have much lower transit latencies
prioritorized
packets only wait for the packet already exiting the switch for the worst case (1500B
packet that just started) this latency is:
–
1
2
3
sec at 100 Mbps (about the same as a SONET/SDH frame)
–
12.3
sec at 1 Gbps
–
1.23
sec at 10 Gbps
•
TDM pseudowire traffic requires a jitter buffer
–
eliminates delay variation
–
adds additional latency (under 1 msec for prioritorized, low PDV, traffic)
So, delay considerations actually favor
CE
and
MPLS
over
SONET/SDH
!
What about delay asymmetry?
For some bi-directional applications
the delay must be
symmetric
(the same in both directions)
•
SONET/SDH
–
ADM rings have constant delay asymmetry
(without “spatial reuse” management)
–
teleprotection mechanisms compensate for this
•
CE
and
MPLS
–
CE is always
co-routed
and thus symmetric
–
best effort MPLS may not be co-routed
–
but MPLS-TE and MPLS-TP can be
•
TDM pseudowire
–
may introduce buffer asymmetry
–
correct implementation keeps this very low
So, delay asymmetry considerations
actually favor
CE
and
MPLS-TP
over
SONET/SDH
!
Management
•
SONET/SDH
networks typically are typically supported
by sophisticated management platforms
(
O
peration
S
upport
S
ystems,
N
etwork
M
anagement
S
ystems)
developed by vendors or users over decades
•
Carrier Ethernet
was developed
to replace SONET/SDH in service provider networks
and thus borrowed heavily from existing SONET/SDH management
architecture, terminology, and look-and-feel
•
MPLS-TP
was developed
to be functionally equivalent to previously developed
CE
and thus borrowed heavily from existing
SONET/SDH
management
architecture, terminology, and look-and-feel
So, from the point of view of management
SONET/SDH
,
CE
and
MPLS-TP
are exceptionally similar
while best-effort MPLS is completely different
Synchronization
•
Synchronization (AKA timing)
the ability to transfer highly accurate frequency or time
over a network (obviating reliance on GPS)
While timing may not be a requirement in present-day utilities networks
it is crucial to support some imminent applications
such as new teleprotection mechanisms and synchrophasors
•
SONET/SDH
has
native support
for frequency transfer
as it requires highly accurate frequency for its own operation
but
does not support
time transfer
•
Ethernet
fully supports
both time and frequency transfer
by use of Synchronous Ethernet (ITU-T G.8261/2/4) for physical layer support
and support for IEEE 1588 Precision Time Protocol for packet layer distribution
•
MPLS
does not currently support
timing at all
work in IETF-TICTOC is progressing to provide some support for IEEE 1588
having no physical layer,
MPLS
will never support physical layer frequency distribution
So, regarding synchronization
CE
is the best alternative
followed by
SONET/SDH
(
and
MPLS
has no support
)
Summary
(so far)
So far we have compared
CE
,
MPLS
, and
MPLS-TP
to
SONET/SDH
, and found
Traffic types
and
growing demand for bandwidth
•
Determinism
–
SONET/SDH
,
CE
and
MPLS-TP
are all acceptable
–
best effort MPLS
is unacceptable for critical operational networks
•
Resilience
–
CE
and
MPLS-TP
(but not non-TP
MPLS
) are as good as
SONET/SDH
•
Delay
(including consistency and asymmetry)
–
favors
CE
and
MPLS
(for asymmetry only
MPLS-TP
) over
SONET/SDH
•
Management
–
CE
and
MPLS-TP
(but not non-TP
MPLS
) are equivalent to
SONET/SDH
•
Synchronization
–
CE
has full support,
SONET/SDH
supports frequency,
MPLS
is deficient
In the final section we will discuss
Network Security
and discover further differences between
Carrier Ethernet
and
MPLS
Network Security
for
Power Utilities
Security highlights
•
MPLS
was invented for
core
networks
where network elements are in secure locations, and therefore trusted
and was thus designed without any security mechanisms
•
In particular, the
MPLS
forwarding plane
–
can not be
source authenticated
(no source address!)
–
has
no standardized integrity
mechanism
and t
he MPLS control plane
uses
soft-state protocols
•
Ethernet
was designed for untrusted network elements
•
CE
does not suffer from most of these ailments since Ethernet ports can be:
–
Authorized
(by 802.1X)
and
Ethernet
packets can be
–
Source authenticated
(by MACsec)
–
Integrity
(and replay)
tested
(by MACsec)
and
CE
uses a security-enabled management plane
(instead of a control plane)
Let’s see why this is important !
MPLS data plane DoS
(injection)
attack
•
Once a packet is inside an
MPLS
network it can not be blocked (no authentication)
•
If an attacker gains physical access to an MPLS network node
(e.g., by using a free port)
he/she can
inject
fake MPLS packets (guessing until a valid label is found)
•
At high rates this injection can overwhelm forwarding resources
MPLS
Core
Substation
Central Site
Connect to any free
MPLS
port
PE
C
E
c
a
n
b
l
o
c
k
t
h
i
s
a
t
t
a
c
k
u
s
i
n
g
8
0
2
.
1
X
a
u
t
h
o
r
i
z
a
t
i
o
n
LSP
LSP
PE
PE
PE
Data Plane
Central Site
MPLS man in the middle attack
•
Tampering means falsifying SCADA RTU/IED <-> control station data
•
Can be implemented by
owning
the switch or by inserting an
evil SFP
into a port
•
MPLS
has no integrity mechanisms to detect tampering
•
Result can be power disruption and/or physical damage to equipment
MPLS
Core
Substation
LSP
LSP
PE
Data Plane
C
E
c
a
n
b
l
o
c
k
t
h
i
s
a
t
t
a
c
k
u
s
i
n
g
M
A
C
S
e
c
’
s
i
n
t
e
g
r
i
t
y
c
h
e
c
k
Central Site
MPLS LSP swap attack
•
The attacker exchanges the internal labels belonging to 2 substations
•
Implemented by
owning
the switch or via an
Evil SFP
•
MPLS
has no source authentication mechanisms
•
The Central Site control systems now believe that indications from substation A
belong to substation B (and vice versa)
MPLS
Core
Substation A
LSP
PE
Data Plane
CE can block this attack
using MACSec’s source authentication
Data A
Substation B
PE
Data B
Data A
Data B
Data A
Data B
MPLS control plane attack
Not relevant for MPLS-TP w/o control plane
•
MPLS
control protocols (e.g., LDP and RSVP-TE) are
soft-state
(when contact with a peer is lost, LSPs are withdrawn)
•
Intermittently deleting consecutive few heartbeat packets
causes massive denial of service
•
A more complex attack can poison the
Label Information Base
Substation
Central Site
MPLS
Core
LDP or RSVP-TE
LSP
PE
Control Plane
A
t
t
a
c
k
i
s
n
o
t
a
p
p
l
i
c
a
b
l
e
t
o
C
E
w
h
i
c
h
d
o
e
s
n
’
t
u
s
e
a
C
o
n
t
r
o
l
P
l
a
n
e
Summary (final this time)
In our previous summary we saw that
•
Carrier Ethernet
and
MPLS-TP
(but not
MPLS
)
were as good as, or even better than,
SONET/SDH
on most accounts
and had the further advantage of being
future proof
•
Best effort MPLS
is
nondeterministic
and should not be considered for operational networks
•
Concerning
synchronization
(crucial for up-and-coming applications)
Carrier Ethernet
has full support
while
MPLS
has none (thus diminishing its status as being
future proof
)
Now we have seen that
•
Regarding
Network Security
MPLS
is highly vulnerable
while
Carrier Ethernet
possesses mechanisms to fight off attacks
These facts should be taken into account
when planning future transport networks
Yaakov (J) Stein
CTO
yaakov_s@rad.com
Carrier Ethernet and MPLS are modern packet-based network technologies replacing aging SONET/SDH systems in the power utility sector. Learn about the features, benefits, and considerations of choosing between Carrier Ethernet and MPLS for your power utility network infrastructure.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
The POWERful Choice Carrier Ethernet or MPLS For Power Utilities Yaakov (J) Stein CTO
SONET/SDH is being phased out SONET technology is widely deployed, but SONET technology is aging SONET equipment is becoming obsolete and hard to find SONET is hard to maintain (parts hard to obtain and expensive) finding staff with SONET expertise is becoming ever more difficult no new rates/functionality/standards/applications are being developed for SONET Modern packet-based networks (based on Ethernet, MPLS, and IP) are the present and future are broadband and becoming even more so are less expensive (both CAPEX and OPEX) and more flexible are being actively extended (e.g., migration to 61850) But there are open questions can all the relevant services be migrated to packet (e.g., teleprotection, synchrophasors)? which packet-based network to choose ? 2 The POWERful Choice - Carrier Ethernet or MPLS
The options Carrier Ethernet Based on most popular technology in the world Look and feel similar to SONET/SDH networks Mature carrier-grade technology Support for synchronization Network security mechanisms available MPLS Core network technology Inherits rich IP control plane Deterministic paths available (MPLS-TE) Has no inherent network security MPLS-TP Based on MPLS, but adds mechanisms patterned after Carrier Ethernet OAM and protection switching (including rings) Look and feel similar to SONET/SDH networks Does not require IP forwarding or control plane Has no inherent network security 3 The POWERful Choice - Carrier Ethernet or MPLS
What is Carrier Ethernet ? (1) Blue means Ethernet Ethernet started out as a LAN technology LAN networks are small and operated by consumer and hence are easily managed When Ethernet left the LAN environment new mechanisms were needed, e.g. scalability (to reach 100s of thousands of end-points) OAM (Fault Management, Performance Monitoring) deterministic (Connection-Oriented) connections support for various topologies (e.g., point-point, rings, trees) resilience mechanisms (e.g., Automatic Protection Switching) support for synchronization Metcalf s original sketch of Ethernet Carrier Ethernet (CE) adds carrier-grade features to Ethernet so that it can replace SONET/SDH as a transport network 4 The POWERful Choice - Carrier Ethernet or MPLS
What is Carrier Ethernet ? (2) Mature Technology widely deployed by service providers promoted and maintained by Metro Ethernet Forum (MEF) Deterministic and Connection Oriented (unlike connectionless IP) provisioning through management system (not routing) support for point-point, multipoint-multipoint, ring, tree, topologies Support for Quality of Service (up to 8 Classes of Service) enforcement of bandwidth profiles (dual token bucket shaping/policing) color (conformance) marking Carrier-grade operations mechanisms: service activation testing (Y.1564) Fault Management (802.1ag, Y.1731) Performance Monitoring (Y.1731) Automatic Protection Switching (G.8031, G.8032) Synchronization <timing distribution> (SyncE, 1588) Network security mechanisms: access authorization (802.1X) source authentication, integrity and optional encryption (MACSec) 5 The POWERful Choice - Carrier Ethernet or MPLS
What is MPLS ? (1) Red means MPLS MPLS started out as a technology to accelerate IP forwarding by setting up tunnels to transport IP other traffic can be transported via pseudowires MPLS defined by the IETF, and inherits the rich IP protocol suite like all IETF protocols, MPLS does not define layer 2 or below MPLS is a mature technology for core IP networks full Traffic Engineering is available, but not traffic conditioning (policing/shaping) supports mesh topologies uses local Fast ReRoute (not protection switching) for resilience no network security mechanisms (since core elements are trusted) A new MPLS version (MPLS-TP) takes MPLS out of the core network into the transport domain WARNING: there are two non-interoperable versions (from IETF and ITU-T) 6 The POWERful Choice - Carrier Ethernet or MPLS
What is MPLS ? (2) We can now distinguish four distinct flavors of MPLS: 1. 2. 3. 4. best effort MPLS (usually with LDP, perhaps with RSVP-TE for FRR) not true CO pinned to route not to Network Elements used in Internet core MPLS for L3VPN services (RFC 4364 <ex-2547> using BGP) used to deliver VPN services to business users traffic engineered MPLS-TE(currently with RSVP-TE) true CO with resource reservation used when strict SLA guarantees must be given(banks, government, ) transport profile - MPLS-TP(with management or RSVP-TE) does not assume the existence of IP forwarding plane does not require the IP control plane (can work with management systems) implements OAM and APS functionality (based on Carrier Ethernet) supports ring topologies still in initial phases of deployment (little interop testing has been performed) does not add network security features (still susceptible to attack) 7 The POWERful Choice - Carrier Ethernet or MPLS
The battlefront ETHERNET first mile core network TRANSPORT NETWORK local network MPLS last mile Ethernet started in the local network (LAN) and for many years has moved into transport networks MPLS started in the core network (WAN) and is now trying to conquer transport networks with MPLS-TP 8 The POWERful Choice - Carrier Ethernet or MPLS
Technical Comparison
Features in common Both Ethernet and MPLS (all flavors) : can natively transport IP traffic Ethernet can natively transport other traffic types (EtherType) MPLS can transport other traffic types via pseudowire technology can be transported over SONET/SDH and OTN are being actively developed (by multiple standards organizations) Ethernet by the IEEE, MEF, ITU, MPLS by the IETF, ITU-T, may exhibit very high or very low transit delays (and everything in-between) (unlike SONET/SDH which has constant switching latency) very high delay when packets need to wait in a queue very low delay (much lower than SONET/SDH) for prioritorized traffic Both CE and MPLS-TP : typically use network management systems for configuration define FM/PM OAM and diagnostic tests support rings and define APS 10 The POWERful Choice - Carrier Ethernet or MPLS
1st reason for differences format Ethernet packet headers are self-describing DA(6B) SA(6B) VT(2B) VLAN(2B) T/L(2B) a globally unique source address a globally unique destination address an optional connection identifier (VLAN) optional Class of Service and Drop Eligibility Indicator a payload protocol type identifier (EtherType) MPLS packet headers are only locally meaningful Label (20b) TC(3b)S(1b) TTL(8b) no unique addresses a locally meaningful label (stack) a TTL field (to avoid packet looping) optionally a Traffic Class (TC) field 11 The POWERful Choice - Carrier Ethernet or MPLS
2nd reason for differences control Ethernet was zero-touch in broadcast domain LANs CE uses network management to support large networks Ethernet does define L2 control protocols (STP, LACP, LLDP, ) but does not define a routingprotocol(neglecting TRILL, E-VPN, etc.) Best effort MPLS tunnels according to topology found by IP routing protocols So best effort MPLS: does not require sophisticated management system does requires the full logistics of an IP network MPLS-TE requires both IP routing and a sophisticated management system MPLS-TP is the only flavor of MPLS that does not require IP routing but when routing is not used, configuration management is required (basically equivalent to Carrier Ethernet) 12 The POWERful Choice - Carrier Ethernet or MPLS
Additional differences Ethernet defines physical (L1) layers (but may run over MPLS as a PW) MPLS requires a server layer to transport it (which is usually Ethernet) Ethernet can not tolerate forwarding loops Carrier Ethernet supports rings with G.8032 and Industrial Ethernet supports them with High-availability Seamless Redundancy MPLS can (since it contains a TTL field) Carrier Ethernet supports bandwidth profiles (bucketing) Ethernet supports IEEE 1588 timing distribution over packet and defines a physical layer to support Synchronous Ethernet MPLS may obtain support for 1588 (work ongoing in IETF) but since MPLS does not a physical layer it can not provide physical layer synchronization support Ethernet has network security mechanisms (MACsec, 802.1X, SNMPv3) MPLS does not define any standardized network security mechanisms and since MPLS has no source address it can not provide source authentication 13 The POWERful Choice - Carrier Ethernet or MPLS
The new trend SDN Distributed routing protocols are limited to finding simple connectivity minimizing number of hops but can not perform more sophisticated operations optimizing paths under constraints (e.g., delay, security) setting up backup paths integrating networking functionalities (e.g., NAT, firewall) into paths Lately, a new paradigm has arisen Software Derived Networking, which: removes control protocols from network elements replaces distributed routing with centralized path computation configures the forwarding actions of the switches from a central site SDN sees the IP/MPLS control plane as a disadvantage and adopts the Carrier Ethernet / MPLS-TP approach New SDN tools can optimally manage operational networks SDN services can be added and modified at the speed of software SDN should lead to significant OPEX reductions 14 The POWERful Choice - Carrier Ethernet or MPLS
Why not use both ? (1) We have seen that MPLS is missing several critical features in particular, synchronization and network security So, why not use both Ethernet and MPLS taking the best features of each ? In fact, MPLS does not define its own physical layer and the most common physical layer supporting MPLS is Ethernet although MPLS can be transported over other physical layers, e.g., SDH or OTN So the real question is whether to maintain an Ethernet network or an MPLS network in addition to an Ethernet network ! MPLS ETHERNET 15 The POWERful Choice - Carrier Ethernet or MPLS
Why not use both ? (2) How many networks are there ? Ethernet defines its own physical layer although Ethernet can be transported over other physical layers When transporting IP over Ethernet there are actually 2 or 3 networks 3 IP 2 Ethernet 1 Ethernet or optionally SONET/SDH or OTN MPLS does not define its own physical layer When transporting IP over MPLS there are actually 3 or 4 networks 3 IP 2.5MPLS 2 Ethernet 1 Ethernet or optionally SONET/SDH or OTN Do we care how many networks there are ? 16 The POWERful Choice - Carrier Ethernet or MPLS
Why not use both ? (3) Yes, because maintaining networks is never trivial or expense-free! Attempts to design a network to use Ethernet as a dumb pipe under MPLS usually end up using a large number of Ethernet mechanisms For example, when running MPLS over Ethernet, one usually needs : staff trained in Ethernet technologies and staff trained in IP/MPLS technologies to be able to run Ethernet OAM and MPLS diagnostic tools to maintain an Ethernet NMS and MPLS management screens Network management is the core business of a network service provider and for them it may be reasonable to maintain duplicate staff, tools, operations centers, etc. Network maintenance is not the core business of a power utility and the duplication and added complexity is usually not justifiable 17 The POWERful Choice - Carrier Ethernet or MPLS
Operational Comparison
Utilities network requirements Traffic types (not an exhaustive list) SCADA operational traffic teleprotection traffic synchrophasor traffic surveillance video general TCP/IP and there is a growing demand for bandwidth Determinism (CO behavior) best effort / nondeterministic (Internet-like) behavior is not acceptable Resilience(critical infrastructures must be highly reliable) Low (and constant) end-end delay (for SCADA and teleprotection applications) Management networks presently employ centralized management end-to-end provisioning and maintenance are musts Synchronization Network security (merits discussion in a separate section) cyber security is a growing concern regulatory requirements are appearing 19 The POWERful Choice - Carrier Ethernet or MPLS
Traffic types SONET/SDH was designed to transport certain traffic types and rates mapping new traffic types is difficult and complex transport of most traffic rates is inefficient no higher rates are being defined for SONET/SDH Ethernet was designed to transport arbitrary traffic types and rates EtherType mechanism to indicate payload types pseudowire technology may also be used no rate constraints higher rates being defined (presently 100Gbps) MPLS was designed to transport IP traffic pseudowire technology enables transport of arbitrary traffic types MPLS imposes no rate constraints or limitations So, regarding traffic, SONET/SDH is reaching End-of-Life while Ethernet and MPLS are future proof! 20 The POWERful Choice - Carrier Ethernet or MPLS
Determinism Networks are deterministic when traffic consistently flows through the network in the same way With nondeterministic networks (e.g., IP and best effort MPLS) each packet may take a different route through the network, thus enabling intermittent faults (only when the packets happen to go there) complicating troubleshooting (where did the packets go?) excluding the reservation of resources or specific processing at particular network elements(you can t be sure the packets will go where you want ) SONET/SDH networks are Circuit Switched, and thus completely deterministic CE and some types of MPLS (TE, TP) are Connection Oriented and thus relatively deterministic traffic consistently takes the same path through the network but does not always take precisely the same time to traverse So, due to lack of determinism, best effort MPLS is not a reasonable candidate for a power utility operational network 21 The POWERful Choice - Carrier Ethernet or MPLS
Resilience SONET/SDH is well-known for its Automatic Protection Switching gold standard 1:1 APS supports < 50 millisecond protection switching time 1+1 APS can provide hitless switching (at the cost of increased bandwidth) Best effort MPLS relies on slow rerouting for recovery MPLS with Fast ReRoute performs local detours around failures at the expense of loss of determinism CE and MPLS-TP support several types of APS CE s G.8031 and G.8032 and MPLS-TP s RFC 6378, 6974, ITU-T G.8131 1+1 pseudowire redundancy achieves hitless switching at the cost of increased bandwidth consumption So, from the point of view of resilience CE and MPLS-TP are as good as SONET/SDH ! 22 The POWERful Choice - Carrier Ethernet or MPLS
End-end delay and delay consistency Some operational traffic require low and consistent delay For example, teleprotection s end-end delay budget may be 6 milliseconds SONET/SDH latency is typically sufficiently low (e.g., under 2 msec.) is constant is independent of SONET/SDH rate (whether OC3 or OC192) Carrier Ethernet and MPLS may have much lower transit latencies prioritorized packets only wait for the packet already exiting the switch for the worst case (1500B packet that just started) this latency is: 123 sec at 100 Mbps (about the same as a SONET/SDH frame) 12.3 sec at 1 Gbps 1.23 sec at 10 Gbps TDM pseudowire traffic requires a jitter buffer eliminates delay variation adds additional latency (under 1 msec for prioritorized, low PDV, traffic) So, delay considerations actually favor CE and MPLS over SONET/SDH ! 23 The POWERful Choice - Carrier Ethernet or MPLS
What about delay asymmetry? For some bi-directional applications the delay must be symmetric (the same in both directions) SONET/SDH ADM rings have constant delay asymmetry (without spatial reuse management) teleprotection mechanisms compensate for this CE and MPLS CE is always co-routed and thus symmetric best effort MPLS may not be co-routed but MPLS-TE and MPLS-TP can be TDM pseudowire may introduce buffer asymmetry correct implementation keeps this very low So, delay asymmetry considerations actually favor CE and MPLS-TP over SONET/SDH ! SONET/SDH Delay asymmetry CE or MPLS Symmetric delay 24 The POWERful Choice - Carrier Ethernet or MPLS
Management SONET/SDH networks typically are typically supported by sophisticated management platforms (Operation Support Systems, Network Management Systems) developed by vendors or users over decades Carrier Ethernet was developed to replace SONET/SDH in service provider networks and thus borrowed heavily from existing SONET/SDH management architecture, terminology, and look-and-feel MPLS-TP was developed to be functionally equivalent to previously developed CE and thus borrowed heavily from existing SONET/SDH management architecture, terminology, and look-and-feel So, from the point of view of management SONET/SDH, CE and MPLS-TP are exceptionally similar while best-effort MPLS is completely different 25 The POWERful Choice - Carrier Ethernet or MPLS
Synchronization Synchronization (AKA timing) the ability to transfer highly accurate frequency or time over a network (obviating reliance on GPS) While timing may not be a requirement in present-day utilities networks it is crucial to support some imminent applications such as new teleprotection mechanisms and synchrophasors SONET/SDH has native support for frequency transfer as it requires highly accurate frequency for its own operation but does not support time transfer Ethernet fully supports both time and frequency transfer by use of Synchronous Ethernet (ITU-T G.8261/2/4) for physical layer support and support for IEEE 1588 Precision Time Protocol for packet layer distribution MPLS does not currently support timing at all work in IETF-TICTOC is progressing to provide some support for IEEE 1588 having no physical layer, MPLS will never support physical layer frequency distribution So, regarding synchronization CE is the best alternative followed by SONET/SDH (and MPLS has no support) 26 The POWERful Choice - Carrier Ethernet or MPLS
Summary (so far) So far we have compared CE, MPLS, and MPLS-TP to SONET/SDH, and found Traffic types and growing demand for bandwidth Determinism SONET/SDH, CE and MPLS-TP are all acceptable best effort MPLS is unacceptable for critical operational networks Resilience CE and MPLS-TP (but not non-TP MPLS) are as good as SONET/SDH Delay (including consistency and asymmetry) favors CE and MPLS (for asymmetry only MPLS-TP) over SONET/SDH Management CE and MPLS-TP (but not non-TP MPLS) are equivalent to SONET/SDH Synchronization CE has full support, SONET/SDH supports frequency, MPLS is deficient In the final section we will discuss Network Security and discover further differences between Carrier Ethernet and MPLS 27 The POWERful Choice - Carrier Ethernet or MPLS
Network Security for Power Utilities
Security highlights MPLS was invented for core networks where network elements are in secure locations, and therefore trusted and was thus designed without any security mechanisms In particular, the MPLS forwarding plane can not be source authenticated (no source address!) has no standardized integrity mechanism and the MPLS control plane uses soft-state protocols Ethernet was designed for untrusted network elements CE does not suffer from most of these ailments since Ethernet ports can be: Authorized (by 802.1X) and Ethernet packets can be Source authenticated (by MACsec) Integrity (and replay) tested (by MACsec) and CE uses a security-enabled management plane (instead of a control plane) Let s see why this is important ! 29 The POWERful Choice - Carrier Ethernet or MPLS
MPLS data plane DoS (injection) attack CE can block this attack using 802.1X authorization Central Site DMS/EMS PE Substation RTU Data Center TPR MPLS Core PE LSP LAN LSP PE PE http://tiwibzone.tiwib.netdna-cdn.com/images/guy-fawkes-mask.jpg Connect to any free MPLS port Once a packet is inside an MPLS network it can not be blocked (no authentication) If an attacker gains physical access to an MPLS network node (e.g., by using a free port) he/she can inject fake MPLS packets (guessing until a valid label is found) At high rates this injection can overwhelm forwarding resources 30 The POWERful Choice - Carrier Ethernet or MPLS
MPLS man in the middle attack Central Site CE can block this attack using MACSec s integrity check DMS/EMS Substation RTU Data Center TPR PE MPLS Core LAN LSP http://tiwibzone.tiwib.netdna-cdn.com/images/guy-fawkes-mask.jpg Tampering means falsifying SCADA RTU/IED <-> control station data Can be implemented by owning the switch or by inserting an evil SFP into a port MPLS has no integrity mechanisms to detect tampering Result can be power disruption and/or physical damage to equipment 31 The POWERful Choice - Carrier Ethernet or MPLS
MPLS LSP swap attack Central Site CE can block this attack using MACSec s source authentication DMS/EMS Substation A RTU Data Center TPR PE LAN MPLS Core Substation B RTU TPR PE http://tiwibzone.tiwib.netdna-cdn.com/images/guy-fawkes-mask.jpg LAN The attacker exchanges the internal labels belonging to 2 substations Implemented by owning the switch or via an Evil SFP MPLS has no source authentication mechanisms The Central Site control systems now believe that indications from substation A belong to substation B (and vice versa) 32 The POWERful Choice - Carrier Ethernet or MPLS
MPLS control plane attack Not relevant for MPLS-TP w/o control plane Attack is not applicable to CE which doesn t use a Control Plane Central Site DMS/EMS Substation RTU Data Center TPR MPLS Core PE LAN http://tiwibzone.tiwib.netdna-cdn.com/images/guy-fawkes-mask.jpg MPLS control protocols (e.g., LDP and RSVP-TE) are soft-state (when contact with a peer is lost, LSPs are withdrawn) Intermittently deleting consecutive few heartbeat packets causes massive denial of service A more complex attack can poison the Label Information Base 33 The POWERful Choice - Carrier Ethernet or MPLS
Summary (final this time) In our previous summary we saw that Carrier Ethernet and MPLS-TP (but not MPLS) were as good as, or even better than, SONET/SDH on most accounts and had the further advantage of being future proof Best effort MPLS is nondeterministic and should not be considered for operational networks Concerning synchronization (crucial for up-and-coming applications) Carrier Ethernet has full support while MPLS has none (thus diminishing its status as being future proof) Now we have seen that Regarding Network Security MPLS is highly vulnerable while Carrier Ethernet possesses mechanisms to fight off attacks These facts should be taken into account when planning future transport networks 34 The POWERful Choice - Carrier Ethernet or MPLS
Yaakov (J) Stein CTO yaakov_s@rad.com
