The Google File System (GFS)
undefined
THE GOOGLE
FILE SYSTEM
S. Ghemawat, H. Gobioff, and S.‐T. Leung.
SOSP 2003
An unusual environment
Component failures are the norm, not the exception
Scale and component quality
Files are huge by traditional standards
Most files contain many application objects (web pages)
Most file updates are append-only
Very few random writes
Once written, files are only read
GFS was co-designed with the applications using it
Design Assumptions (I)
System is built from many inexpensive commodity components
Must constantly monitor itself
Must quickly recover from component failures
System will store a modest number of large files
Over a million files
Typically 100MB or more
Design Assumptions (II)
Workload primarily consists of
Large streaming reads (1MB or more)
Small sequential reads (a few KBs)
Many large sequential writes
Append data to files
Very few random updates
Design Assumptions (III)
Many concurrent appends to the same file by multiple clients
Need efficient implementation of well-defined semantics
High sustained bandwidth is more important than latency
A note
G
F
S
w
a
s
d
e
s
i
g
n
e
d
t
o
b
e
u
s
e
d
b
y
m
o
s
t
l
y
c
o
-
d
e
s
i
g
n
e
d
a
p
p
l
i
c
a
t
i
o
n
s
Not by regular users
Explains many of its features
User interface (I)
Quite familiar but non-POSIX
Files organized in directories
Usual primitives for
Creating, deleting, opening, closing files
Writing to and reading from files
User interface (II)
Two new operations
S
n
a
p
s
h
o
t
s
Create copies of files and directories
R
e
c
o
r
d
a
p
p
e
n
d
s
Allow multiple clients to concurrently append data
to the same file
Useful for implementing
Multi-way merge results
Producer-consumer queues
GFS clusters
GFS Cluster
A
m
a
s
t
e
r
M
u
l
t
i
p
l
e
c
h
u
n
k
s
e
r
v
e
r
s
Concurrently accessed by many clients
M
a
s
t
e
r
C
h
u
n
k
s
e
r
v
e
r
C
h
u
n
k
s
e
r
v
e
r
C
h
u
n
k
s
e
r
v
e
r
The files
F
i
l
e
s
a
r
e
d
i
v
i
d
e
d
i
n
t
o
f
i
x
e
d
-
s
i
z
e
c
h
u
n
k
s
o
f
6
4
M
B
Similar to clusters or sectors in other file systems
E
a
c
h
c
h
u
n
k
h
a
s
a
u
n
i
q
u
e
6
4
-
b
i
t
l
a
b
e
l
Assigned by the master node at time of creation
G
F
S
m
a
i
n
t
a
i
n
l
o
g
i
c
a
l
m
a
p
p
i
n
g
s
o
f
f
i
l
e
s
t
o
c
o
n
s
t
i
t
u
e
n
t
c
h
u
n
k
s
Chunks are replicated
A
t
l
e
a
s
t
t
h
r
e
e
t
i
m
e
s
More for critical or heavily used files
Architecture
The master server (I)
Single master server
Stores chunk-related metadata
Tables mapping the 64-bit labels to chunk locations
The files they make up
Locations of chunk replicas
What processes are reading or writing to a particular chunk, or
taking a snapshot of it
The master server (II)
Communicates with its chunkservers through
heartbeat messages
Also controls
Lease management
Garbage collection of orphaned chunks
Chunk migration between chunk servers
T
h
e
m
e
t
a
d
a
t
a
s
e
r
v
e
r
The chunk servers
Store chunks as Linux files
T
r
a
n
s
f
e
r
d
a
t
a
d
i
r
e
c
t
l
y
t
o
/
f
r
o
m
c
l
i
e
n
t
s
Neither the clients nor the chunk servers cache files
Little benefits in a streaming environment
Omitting it results in a simpler design
Linux I/O buffers already keep in RAM frequently accessed
chunks
Accessing a file
1.
Client converts (
file name, file offset
) into
(
file name, chunk index
)
2.
Sends (
file name, chunk index
) to master
3.
Master replies with
chunk handle
and
replica locations
4.
Client caches this information
5.
Client selects a chunk server and sends
(
chunk handle
,
byte range
within the chunk)
Optimization
Clients typically send requests for multiple chunks to the master
Master can add to their reply information about chunks
immediately following the requested chunks
Avoid many client requests to the master
At almost no cost!
Same idea as
readdirplus()
in NFS
Chunk size
Large chunk sizes
Reduce the number of interactions between clients and master
As clients are more likely to perform many operations on the
same chunk, they reduce the number of TCP connection
requests
Reduce the size of the metadata stored on the master
A
l
s
o
i
n
c
r
e
a
s
e
t
h
e
l
i
k
e
l
i
h
o
o
d
o
f
o
b
s
e
r
v
i
n
g
h
o
t
s
p
o
t
s
.
Not a real problem and replication helps
Metadata
M
a
s
t
e
r
s
t
o
r
e
s
i
n
m
e
m
o
r
y
File and chunk namespaces
Mapping from files to chunks
Locations of each chunk's replica
F
i
r
s
t
t
w
o
t
y
p
e
s
o
f
m
e
t
a
d
a
t
a
a
r
e
k
e
p
t
p
e
r
s
i
s
t
e
n
t
b
y
l
o
g
g
i
n
g
m
u
t
a
t
i
o
n
s
t
o
a
n
o
p
e
r
a
t
i
o
n
l
o
g
s
t
o
r
e
d
o
n
t
h
e
m
a
s
t
e
r
'
s
H
D
Not true for the locations of chunk replicas
Obtained from the chunkservers themselves
Chunk locations
Obtained from chunkservers
At startup time
Maintained up to date because master
Controls all chunk placement
Monitors chunkserver status though heartbeats
Simplest solution
Operation log
Contains historical record of critical metadata changes
Acts a logical time line for the order of all concurrent operations
Replicated on multiple remote machines
U
s
i
n
g
b
l
o
c
k
i
n
g
w
r
i
t
e
s
,
b
o
t
h
l
o
c
a
l
l
y
a
n
d
r
e
m
o
t
e
l
y
Consistency model
A
l
l
f
i
l
e
n
a
m
e
s
p
a
c
e
m
u
t
a
t
i
o
n
s
a
r
e
a
t
o
m
i
c
Handled exclusively by the master
Status of a file region can be
C
o
n
s
i
s
t
e
n
t
:
a
l
l
c
l
i
e
n
t
s
s
e
e
t
h
e
s
a
m
e
d
a
t
a
D
e
f
i
n
e
d
:
a
l
l
c
l
i
e
n
t
s
s
e
e
t
h
e
s
a
m
e
d
a
t
a
,
w
h
i
c
h
i
n
c
l
u
d
e
t
h
e
e
n
t
i
r
e
t
y
o
f
t
h
e
l
a
s
t
m
u
t
a
t
i
o
n
U
n
d
e
f
i
n
e
d
b
u
t
c
o
n
s
i
s
t
e
n
t
:
a
l
l
c
l
i
e
n
t
s
s
e
e
t
h
e
n
s
a
m
e
d
a
t
a
b
u
t
i
t
m
a
y
n
o
t
r
e
f
l
e
c
t
w
h
a
t
a
n
y
o
n
e
m
u
t
a
t
i
o
n
h
a
s
w
r
i
t
t
e
n
I
n
c
o
n
s
i
s
t
e
n
t
Data mutations
W
r
i
t
e
s
:
Cause data to be written at a specific offset
R
e
c
o
r
d
a
p
p
e
n
d
s
:
Cause data to be automatically appended at least once at an
offset of GFS choosing
C
o
n
s
i
s
t
e
n
c
y
i
s
e
n
s
u
r
e
d
b
y
Applying mutations to a chunk in the same order
Using chunk version numbers
Dealing with stale chunk locations
Not covered
Numbers refer to
the steps in the
two previous slides
Mutations (I)
1.
Client requests a lease from master server :
2.
Master server grants update permission to a client for a finite
period of time (60 seconds)
3.
Client pushes data to all the replicas
Data end in internal LRU buffer cache of each chunkserver
Once the replicas have all ACKed receiving the data,
client sends a write request to the primary replica.
Primary assigns a serial number to the mutation and applies it to
its local state
Mutations (II)
5.
Primary replicas forwards the write request to all secondary
replicas, which apply the mutation in the same serial order.
6.
Secondary replicas reply to the primary once they have
completed the operation
7.
Primary notifies to the client the mutation is completed
Atomic record appends
GFS appends the new data
At least once
Atomically
At an offset of GFS choosing
Returns that offset to the client
Widely used to implement concurrent access
Snapshots
Copies file and directories in parallel with regular operations
U
s
e
c
o
p
y
-
o
n
-
w
r
i
t
e
a
p
p
r
o
a
c
h
Temporarily make copied data read-only
To detect changes taking place while the snapshot is being
taken
Implementation
As a user-level library
Easiest solution
Performance
Performance
When used with relatively small number of servers (15),
GFS achieves
Reading performance comparable to that of a single disk
(80–100 MB/s)
Reduced write performance (30 MB/s)
Even lower performance (5 MB/s) in appending data to
existing files
Performance
Read rate increases significantly with the number of chunk
servers
583 MB/s for 342 nodes
The Google File System (GFS) introduced in 2003 by Ghemawat, Gobioff, and Leung, is designed for an unusual environment where component failures are common. GFS is specifically tailored for storing large files with primarily read-heavy workloads, emphasizing high sustained bandwidth over low latency. The system is built from inexpensive components, intended to recover quickly from failures, and is optimized for handling concurrent appends efficiently. GFS was co-designed with its applications, offering unique features like snapshots and record appends, making it ideal for large-scale data storage needs.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
THE GOOGLE FILE SYSTEM S. Ghemawat, H. Gobioff, and S. T. Leung. SOSP 2003
An unusual environment Component failures are the norm, not the exception Scale and component quality Files are huge by traditional standards Most files contain many application objects (web pages) Most file updates are append-only Very few random writes Once written, files are only read GFS was co-designed with the applications using it
Design Assumptions (I) System is built from many inexpensive commodity components Must constantly monitor itself Must quickly recover from component failures System will store a modest number of large files Over a million files Typically 100MB or more
Design Assumptions (II) Workload primarily consists of Large streaming reads (1MB or more) Small sequential reads (a few KBs) Many large sequential writes Append data to files Very few random updates
Design Assumptions (III) Many concurrent appends to the same file by multiple clients Need efficient implementation of well-defined semantics High sustained bandwidth is more important than latency
A note GFS was designed to be used by mostly co-designed applications Not by regular users Explains many of its features
User interface (I) Quite familiar but non-POSIX Files organized in directories Usual primitives for Creating, deleting, opening, closing files Writing to and reading from files
User interface (II) Two new operations Snapshots Create copies of files and directories Record appends Allow multiple clients to concurrently append data to the same file Useful for implementing Multi-way merge results Producer-consumer queues
GFS clusters GFS Cluster A master Multiple chunkservers Concurrently accessed by many clients Chunkserver Chunkserver Master Chunkserver
The files Files are divided into fixed-size chunks of 64MB Similar to clusters or sectors in other file systems Each chunk has a unique 64-bit label Assigned by the master node at time of creation GFS maintain logical mappings of files to constituent chunks Chunks are replicated At least three times More for critical or heavily used files
The master server (I) Single master server Stores chunk-related metadata Tables mapping the 64-bit labels to chunk locations The files they make up Locations of chunk replicas What processes are reading or writing to a particular chunk, or taking a snapshot of it
The master server (II) Communicates with its chunkservers through heartbeat messages Also controls Lease management Garbage collection of orphaned chunks Chunk migration between chunk servers The metadata server
The chunk servers Store chunks as Linux files Transfer data directly to/from clients Neither the clients nor the chunk servers cache files Little benefits in a streaming environment Omitting it results in a simpler design Linux I/O buffers already keep in RAM frequently accessed chunks
Accessing a file 1. Client converts (file name, file offset) into (file name, chunk index) 2. Sends (file name, chunk index) to master 3. Master replies with chunk handle and replica locations 4. Client caches this information 5. Client selects a chunk server and sends (chunk handle, byte range within the chunk)
Optimization Clients typically send requests for multiple chunks to the master Master can add to their reply information about chunks immediately following the requested chunks Avoid many client requests to the master At almost no cost! Same idea as readdirplus() in NFS
Chunk size Large chunk sizes Reduce the number of interactions between clients and master As clients are more likely to perform many operations on the same chunk, they reduce the number of TCP connection requests Reduce the size of the metadata stored on the master Also increase the likelihood of observing hot spots. Not a real problem and replication helps
Metadata Master stores in memory File and chunk namespaces Mapping from files to chunks Locations of each chunk's replica First two types of metadata are kept persistent by logging mutations to an operation log stored on the master's HD Not true for the locations of chunk replicas Obtained from the chunkservers themselves
Chunk locations Obtained from chunkservers At startup time Maintained up to date because master Controls all chunk placement Monitors chunkserver status though heartbeats Simplest solution
Operation log Contains historical record of critical metadata changes Acts a logical time line for the order of all concurrent operations Replicated on multiple remote machines Using blocking writes, both locally and remotely
Consistency model All file namespace mutations are atomic Handled exclusively by the master Status of a file region can be Consistent: all clients see the same data Defined: all clients see the same data, which include the entirety of the last mutation Undefined but consistent: all clients see then same data but it may not reflect what any one mutation has written Inconsistent
Data mutations Writes: Cause data to be written at a specific offset Record appends: Cause data to be automatically appended at least once at an offset of GFS choosing Consistency is ensured by Applying mutations to a chunk in the same order Using chunk version numbers
Dealing with stale chunk locations Not covered
Numbers refer to the steps in the two previous slides
Mutations (I) 1. Client requests a lease from master server : 2. Master server grants update permission to a client for a finite period of time (60 seconds) 3. Client pushes data to all the replicas Data end in internal LRU buffer cache of each chunkserver Once the replicas have all ACKed receiving the data, client sends a write request to the primary replica. Primary assigns a serial number to the mutation and applies it to its local state
Mutations (II) 5. Primary replicas forwards the write request to all secondary replicas, which apply the mutation in the same serial order. 6. Secondary replicas reply to the primary once they have completed the operation 7. Primary notifies to the client the mutation is completed
Atomic record appends GFS appends the new data At least once Atomically At an offset of GFS choosing Returns that offset to the client Widely used to implement concurrent access
Snapshots Copies file and directories in parallel with regular operations Use copy-on-write approach Temporarily make copied data read-only To detect changes taking place while the snapshot is being taken
Implementation As a user-level library Easiest solution
Performance When used with relatively small number of servers (15), GFS achieves Reading performance comparable to that of a single disk (80 100 MB/s) Reduced write performance (30 MB/s) Even lower performance (5 MB/s) in appending data to existing files
Performance Read rate increases significantly with the number of chunk servers 583 MB/s for 342 nodes
