Advancing High Throughput Computing: A Revolution in Job Handling
Throughput
Computing
Miron Livny
Vials Research Professor
John P. Morgridge Professor of Computer Science
Director UW Center for High Throughput Computing
Technical Director of the OSG
Welcome to
HTC23
•
1986
– First deployment of (HT)Condor
•
1996
– High Throughput Computing (HTC)
formulated
•
2005
– OSG Consortium established
•
2006
– Center for High Throughput
Computing (CHTC) established
•
2020
– Partnership for Advanced
Throughput Computing (PATh)
funded by NSF
•
2023
– First Throughput Computing
event
“The Partnership to Advance Throughput
Computing (PATh) project will expand
Distributed High Throughput Computing
(dHTC) technologies and methodologies
through innovation, translational effort, and
large-scale adoption to advance the Science &
Engineering goals of the broader community.”
P
A
T
h
P
r
o
p
o
s
a
l
0
4
/
2
1
/
2
0
2
0
T
r
a
n
s
l
a
t
i
o
n
a
l
F
l
o
w
www.zonkafeedback.com/blog/positive-feedback-loop
HTC Services
NSF-funded Advanced CI Ecosystem
(Highly Accessible Computing)
Throughput Commuting is all about
scaling out and therefore about the
rate of job handling
During the month of June 2023, the OSPool
handled close to
18,000,000
jobs
•
600,000
jobs per day
•
25,000
jobs per hour
•
410
jobs per minute
•
7
jobs per second
NSF
Awarded
Individual
Allocation
I
I
n
d
i
v
i
d
u
a
l
Personal
NSF
Awarded
FAIR-Share
Access
Point
U
s
e
i
t
R
e
m
o
t
e
l
y
D
e
p
l
o
y
i
t
L
o
c
a
l
l
y
PATh services and technologies enable
federation of computing capacity
•
More than
50
institutions contribute to the
OSPool capacity provided by more than
70
sites
PATh services and technologies enable
effective access to remote datasets
•
More than
160
datasets federated by the
Open Science Data Federation (OSDF)
Scaling out with federated
capacity and datasets means a
lot of file transfers
•
In 2022, the OSDF executed more than
1B
file transfers (
32
transfers per second)
•
In June 2023, Jobs executed by the OSPool
required
320M
file transfers (
120
transfers
per second)
PATh Executive Summary
“
B
r
o
a
d
e
r
I
m
p
a
c
t
–
W
e
f
i
r
m
l
y
b
e
l
i
e
v
e
i
n
d
H
T
C
a
s
a
n
a
c
c
e
s
s
i
b
l
e
c
o
m
p
u
t
i
n
g
p
a
r
a
d
i
g
m
w
h
i
c
h
s
u
p
p
o
r
t
s
t
h
e
d
e
m
o
c
r
a
t
i
z
a
t
i
o
n
o
f
r
e
s
e
a
r
c
h
c
o
m
p
u
t
i
n
g
t
o
i
n
c
l
u
d
e
r
e
s
e
a
r
c
h
e
r
s
a
n
d
o
r
g
a
n
i
z
a
t
i
o
n
s
o
t
h
e
r
w
i
s
e
u
n
d
e
r
r
e
p
r
e
s
e
n
t
e
d
i
n
t
h
e
n
a
t
i
o
n
a
l
C
I
e
c
o
s
y
s
t
e
m
.
O
u
r
w
o
r
k
i
s
f
o
u
n
d
e
d
o
n
u
n
i
v
e
r
s
a
l
p
r
i
n
c
i
p
l
e
s
l
i
k
e
s
h
a
r
i
n
g
,
a
u
t
o
n
o
m
y
,
u
n
i
t
y
o
f
p
u
r
p
o
s
e
,
a
n
d
m
u
t
u
a
l
t
r
u
s
t
.
”
P
A
T
h
P
r
o
p
o
s
a
l
0
4
/
2
1
/
2
0
2
0
In June 2023
236
researchers
from
90
projects
placed OSPool Jobs.
What can we do
more/different to
scale out adoption of
Throughput Computing?
Explore the evolution of High Throughput Computing (HTC) through milestones like the formation of the OSG Consortium and the Partnership to Advance Throughput Computing (PATh). Discover how the PATh project aims to innovate and expand Distributed HTC technologies for Science & Engineering goals. Dive into the world of job handling and scaling out in the HTC ecosystem, where the OSPool processes millions of jobs efficiently. Unveil the array of NSF-funded Advanced CI Ecosystem resources that power cutting-edge HTC services across various institutions and testbeds.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Throughput Computing Miron Livny Vials Research Professor John P. Morgridge Professor of Computer Science Director UW Center for High Throughput Computing Technical Director of the OSG
Welcome to HTC23
1986 First deployment of (HT)Condor 1996 High Throughput Computing (HTC) formulated 2005 OSG Consortium established 2006 Center for High Throughput Computing (CHTC) established 2020 Partnership for Advanced Throughput Computing (PATh) funded by NSF 2023 First Throughput Computing event
The Partnership to Advance Throughput Computing (PATh) project will expand Distributed High Throughput Computing (dHTC) technologies and methodologies through innovation, translational effort, and large-scale adoption to advance the Science & Engineering goals of the broader community. PATh Proposal 04/21/2020
Translational Flow Feedback Loop Laboratory (Innovate) Locale (Deploy) Community (Use) www.zonkafeedback.com/blog/positive-feedback-loop
NSF-funded Advanced CI Ecosystem (Highly Accessible Computing) ACCESS PIs Leadership-class Innovative systems HTC Services NCAR Cloud Resources Shared Campus Resources PAWR Testbeds Chameleon Lab (Chicago) CloudLab (Salt Lake City) PATh/OSG CloudBank (San Diego) Cheyenne (Cheyenne) (Madison) Frontera (Austin) Jetstream, JetStream-2 (Bloomington) Delta (Urbana-Champaign) Ookami (Stonybrook) Anvil (W. Lafayette) Bridges-2, Neocortex (Pitt) ACES (College Station) Expanse, Voyager, National Research Platform (San Diego) Stampede 2, Wrangler (Austin)
Throughput Commuting is all about scaling out and therefore about the rate of job handling During the month of June 2023, the OSPool handled close to 18,000,000 jobs 600,000 jobs per day 25,000 jobs per hour 410 jobs per minute 7 jobs per second
OSPool Com. Clouds RAMPS PATh Facility OSG My Campus Bring Your Own Capacity BYOC Access Point I Individual My Collaboration Use it Remotely Deploy it Locally
PATh services and technologies enable federation of computing capacity More than 50 institutions contribute to the OSPool capacity provided by more than 70 sites PATh services and technologies enable effective access to remote datasets More than 160 datasets federated by the Open Science Data Federation (OSDF)
Scaling out with federated capacity and datasets means a lot of file transfers In 2022, the OSDF executed more than 1B file transfers (32 transfers per second) In June 2023, Jobs executed by the OSPool required 320M file transfers (120 transfers per second)
PATh Executive Summary Broader Impact We firmly believe in dHTC as an accessible computing paradigm which supports the democratization of research computing to include researchers and organizations otherwise underrepresented in the national CI ecosystem. Our work is founded on universal principles like sharing, autonomy, unity of purpose, and mutual trust. PATh Proposal 04/21/2020
In June 2023 236 researchers from 90 projects placed OSPool Jobs. What can we do more/different to scale out adoption of Throughput Computing?
