Distributed File Systems with NVMM and RDMA
Distributed File Systems
with NVMM and RDMA
Shared by Xinyang Shao
N
e
x
t
R
e
a
d
i
n
g
G
r
o
u
p
•
Metadata
decouple
•
Metadata
partition
•
Metadata
disaggregation
•
Paper
sharing:
InfiniFS
•
DFS
architecture
with
new
hardware
C
o
n
t
e
n
t
s
•
New hardwares
•
NVM: Non-Volatile Memory
•
RDMA (Remote Direct Memory Access)
•
State-of-the-art DFSs with NVM and RDMA
N
V
M
c
o
n
c
e
p
t
f
a
m
i
l
y
•
NVMM
•
NVM
•
PCM
•
ReRAM/Memristor
•
STT-MRAM
•
PM / PMEM / SCM
•
NVDIMM
N
V
M
t
e
c
h
s
•
PCM=Phase-Change Memory
（相变存储器）
•
ReRAM=Resistive RAM
（可变电阻式存储器），也叫
Memristor
（忆阻器）
•
STT-MRAM=Spin Transfer Torque - Magnetic Random
Access Memory
（自旋扭矩转换随机存储器）
•
FeRAM=Ferroelectric RAM
（铁电随机存储器）
•
MRAM=Magnetoresistive RAM
（磁阻式随机存储器）
•
Flash
（闪存）
I
n
t
e
l
O
p
t
a
n
e
D
C
m
e
m
o
r
i
e
s
N
V
M
c
o
n
c
e
p
t
f
a
m
i
l
y
•
NVMM=Non-Volatile Main Memory
（统称）
•
NVM=Non-Volatile Memory=NVMM
•
PM/PMEM=Persistent Memory
（持久化内存），一般等
同于
SCM=Storage Class Memory
（存储级内存）
•
NVDIMM=Non-Volatile Dual In-line Memory Module
（非
易失性双列直插式内存模块）
H
i
e
r
a
r
c
h
y
l
a
y
e
r
o
f
N
V
M
•
DRAM: ~100ns
•
Optane: ~350ns
I
n
c
o
r
p
o
r
a
t
i
o
n
o
f
N
V
M
•
As memory
•
Replace DRAM
•
Co-exist with DRAM
•
Replace SWAP
•
As storage
N
e
w
s
o
f
t
w
a
r
e
s
t
a
c
k
s
f
o
r
l
o
c
a
l
F
S
•
PMFS (EuroSys’14): Skip page cache for traditional FS
N
e
w
s
o
f
t
w
a
r
e
s
t
a
c
k
s
f
o
r
l
o
c
a
l
F
S
•
NOVA (FAST’16): Log-structured FS for DRAM+NVMM
S
u
m
m
a
r
y
o
f
N
V
M
M
•
Low latency and high bandwidth
•
Expensive
•
Excellent crash consistency
•
Byte-addressable
R
D
M
A
•
CPU offload
•
Kernel bypass
•
Zero copy
图来源：
https://img-blog.csdnimg.cn/20190117125615125.png?x-oss-
process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIxMTI1MTgz,s
ize_16,color_FFFFFF,t_70
C
o
n
t
e
n
t
s
•
New hardwares
•
NVM: Non-Volatile Memory
•
RDMA (Remote Direct Memory Access)
•
State-of-the-art DFSs with RDMA and/or NVM
•
Traditional DFS with RDMA (CephFS, GlusterFS, etc)
•
New archs: NVFS, Octopus, Orion, Assise, …
T
r
a
d
i
t
i
o
n
a
l
D
F
S
w
i
t
h
R
D
M
A
•
CephFS with Accelio
•
GlusterFS with its own RDMA library.
•
Lustre with RoCEv2
N
V
F
S
(
I
C
S
’
1
6
)
•
NVFS=
NV
M- and RDMA-aware HD
FS
O
c
t
o
p
u
s
(
A
T
C
’
1
7
)
O
c
t
o
p
u
s
+
(
T
o
S
’
2
1
)
•
Metadata and data of single file is coupled.
•
Consistent hashing partition.
O
c
t
o
p
u
s
(
A
T
C
’
1
7
)
O
c
t
o
p
u
s
+
(
T
o
S
’
2
1
)
•
Eliminating memory copies:
O
r
i
o
n
(
F
A
S
T
’
1
9
)
•
Orion exposes as a log-structured file system across
only one MDS and clients.
A
s
s
i
s
e
(
O
S
D
I
’
2
0
)
•
Full Client-side NVM implementation of DFS.
L
i
n
e
F
S
(
S
O
S
P
’
2
1
)
•
Offload DFS operations to smart NICs.
Q
&
A
"This content delves into the integration of Non-Volatile Memory Modules (NVMM) and Remote Direct Memory Access (RDMA) in Distributed File Systems. It covers various aspects such as new hardware architectures, NVM concepts and technologies, Intel Optane DC memories, incorporation of NVM in memory and storage systems, and new software stacks for local file systems. The significance of NVMM is highlighted for its low latency, high bandwidth, crash consistency, and byte-addressable nature, making it an expensive yet excellent solution. A detailed study on RDMA and CPU usage complements the exploration of these cutting-edge technologies."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Distributed File Systems with NVMM and RDMA Shared by Xinyang Shao
Next Reading Group Metadata decouple Metadata partition Metadata disaggregation Paper sharing: InfiniFS DFS architecture with new hardware 2
Contents New hardwares NVM: Non-Volatile Memory RDMA (Remote Direct Memory Access) State-of-the-art DFSs with NVM and RDMA 3
NVM concept family NVMM NVM PCM ReRAM/Memristor STT-MRAM PM / PMEM / SCM NVDIMM 4
NVM techs PCM=Phase-Change Memory ReRAM=Resistive RAM Memristor STT-MRAM=Spin Transfer Torque - Magnetic Random Access Memory FeRAM=Ferroelectric RAM MRAM=Magnetoresistive RAM Flash 5
NVM concept family NVMM=Non-Volatile Main Memory NVM=Non-Volatile Memory=NVMM PM/PMEM=Persistent Memory SCM=Storage Class Memory NVDIMM=Non-Volatile Dual In-line Memory Module 7
Hierarchy layer of NVM DRAM: ~100ns Optane: ~350ns 8
Incorporation of NVM As memory Replace DRAM Co-exist with DRAM Replace SWAP As storage 9
New software stacks for local FS PMFS (EuroSys 14): Skip page cache for traditional FS 10
New software stacks for local FS NOVA (FAST 16): Log-structured FS for DRAM+NVMM 11
Summary of NVMM Low latency and high bandwidth Expensive Excellent crash consistency Byte-addressable 12
RDMA CPU offload Kernel bypass Zero copy https://img-blog.csdnimg.cn/20190117125615125.png?x-oss- process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIxMTI1MTgz,s ize_16,color_FFFFFF,t_70 13
Contents New hardwares NVM: Non-Volatile Memory RDMA (Remote Direct Memory Access) State-of-the-art DFSs with RDMA and/or NVM Traditional DFS with RDMA (CephFS, GlusterFS, etc) New archs: NVFS, Octopus, Orion, Assise, 14
Traditional DFS with RDMA CephFS with Accelio GlusterFS with its own RDMA library. Lustre with RoCEv2 15
NVFS (ICS16) NVFS=NVM- and RDMA-aware HDFS 16
Octopus (ATC17) Octopus+(ToS21) Metadata and data of single file is coupled. Consistent hashing partition. 17
Octopus (ATC17) Octopus+(ToS21) Eliminating memory copies: 18
Orion (FAST19) Orion exposes as a log-structured file system across only one MDS and clients. 19
Assise (OSDI20) Full Client-side NVM implementation of DFS. 20
LineFS (SOSP21) Offload DFS operations to smart NICs. 21
Q & A 22
