Introduction to High Throughput Computing Notebooks with HTCondor
H
i
g
h
T
h
r
o
u
g
h
p
u
t
C
o
m
p
u
t
i
n
g
N
o
t
e
b
o
o
k
s
1
HTCondor Week 2019
Todd Tannenbaum
Center for High Throughput Computing
›
Open source browser-based
application to create and share
interactive documents
that contain
h
Live code
•
Python
, R, Julia, Scala, Bash, …
h
Equations
h
Visualizations
h
Narrative Text
›
Also has a console window and file
mover
J
u
p
y
t
e
r
N
o
t
e
b
o
o
k
2
3
›
Install Jupyter via Anaconda or PIP, e.g.
% pip install jupyter
›
And fire it up
% jupyter notebook
›
This command will start a small embedded
web server on your laptop; point your
browser at
http://localhost:8888
and go.
C
a
n
s
t
a
r
t
a
J
u
p
y
t
e
r
i
n
s
t
a
n
c
e
o
n
y
o
u
r
l
a
p
t
o
p
4
›
Point your browser at a URL where a
JupyterHub server is listening
C
a
n
s
t
a
r
t
a
J
u
p
y
t
e
r
i
n
s
t
a
n
c
e
o
n
a
r
e
m
o
t
e
s
e
r
v
e
r
5
Allow users to easily
develop/test
using a
small/responsive pool (eg their laptop!), and
then easily
run
using all the cores in an
HTCondor cluster
G
o
a
l
:
B
r
i
n
g
d
i
s
t
r
i
b
u
t
e
d
H
i
g
h
T
h
r
o
u
g
h
p
u
t
C
o
m
p
u
t
i
n
g
i
n
t
o
t
h
e
s
c
i
e
n
t
i
f
i
c
P
y
t
h
o
n
e
n
v
i
r
o
n
m
e
n
t
6
H
i
g
h
T
h
r
o
u
g
h
p
u
t
C
o
m
p
u
t
i
n
g
N
o
t
e
b
o
o
k
7
Docker container with
h
Python 3
h
Jupyter
h
Popular Python science packages
h
HTCondor Python bindings
h
HTMap
h
Personal HTCondor pool
•
Started if no _condor_SCHEDD_HOST
environment variable set
H
i
g
h
T
h
r
o
u
g
h
p
u
t
C
o
m
p
u
t
i
n
g
N
o
t
e
b
o
o
k
›
Run it on your laptop
h
Container will start up a personal
HTCondor pool, and then Jupyter
h
HTMap uses the personal pool
›
Run it on a server that has both
JupyterHub and a HTCondor Schedd
connected to your site's pool
h
Container will start Jupyter
h
HTMap uses the entire site pool
8
R
u
n
h
t
c
-
n
o
t
e
b
o
o
k
o
n
y
o
u
r
l
a
p
t
o
p
w
i
t
h
a
p
e
r
s
o
n
a
l
p
o
o
l
9
h
Container will start up a personal
HTCondor pool, and then Jupyter
h
HTMap uses the personal pool
R
u
n
h
t
c
-
n
o
t
e
b
o
o
k
o
n
a
r
e
m
o
t
e
s
e
r
v
e
r
w
i
t
h
a
p
e
r
s
o
n
a
l
p
o
o
l
10
11
R
u
n
h
t
c
-
n
o
t
e
b
o
o
k
o
n
a
r
e
m
o
t
e
s
e
r
v
e
r
w
i
t
h
a
p
e
r
s
o
n
a
l
p
o
o
l
R
u
n
o
n
a
r
e
m
o
t
e
s
e
r
v
e
r
u
s
i
n
g
e
n
t
i
r
e
s
i
t
e
p
o
o
l
12
13
R
u
n
o
n
a
r
e
m
o
t
e
s
e
r
v
e
r
u
s
i
n
g
e
n
t
i
r
e
s
i
t
e
p
o
o
l
14
R
u
n
o
n
a
r
e
m
o
t
e
s
e
r
v
e
r
u
s
i
n
g
e
n
t
i
r
e
s
i
t
e
p
o
o
l
›
Docker Hub (images) / GitHub Repos (src)
https://hub.docker.com/u/htcondor
https://github.com/htcondor/htc-notebook
›
Run it on your laptop
docker run –p 8888:8888 htcondor/htc-base-notebook
(then open your web browser as instructed)
›
Pick a software environment!
h
htcondor/htc-scipy-notebook
h
htcondor/htc-tensorflow-notebook
h
htcondor/htc-r-notebook
h
htcondor/htc-pyspark-notebook
h
htcondor/htc-datascience-notebook
15
T
h
a
n
k
Y
o
u
I
n
t
e
r
e
s
t
e
d
?
T
a
l
k
t
o
u
s
!
Docker Hub (images) / GitHub Repos (src)
https://hub.docker.com/u/htcondor
https://github.com/htcondor/htc-notebook
"Explore the world of high throughput computing through Jupyter Notebooks, capable of running on laptops or remote servers. Learn to set up personal HTCondor pools, utilize Python bindings, and integrate with JupyterHub for efficient scientific computing."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
High Throughput Computing Notebooks HTCondor Week 2019 Todd Tannenbaum Center for High Throughput Computing 1
Jupyter Notebook Open source browser-based application to create and share interactive documents that contain Live code Python, R, Julia, Scala, Bash, Equations Visualizations Narrative Text Also has a console window and file mover 2
Can start a Jupyter instance on your laptop Install Jupyter via Anaconda or PIP, e.g. % pip install jupyter And fire it up % jupyter notebook This command will start a small embedded web server on your laptop; point your browser at http://localhost:8888 and go. 4
Can start a Jupyter instance on a remote server Point your browser at a URL where a JupyterHub server is listening 5
Goal: Bring distributed High Throughput Computing into the scientific Python environment Allow users to easily develop/test using a small/responsive pool (eg their laptop!), and then easily run using all the cores in an HTCondor cluster 6
High Throughput Computing Notebook Docker container with Python 3 Jupyter Popular Python science packages HTCondor Python bindings HTMap Personal HTCondor pool Started if no _condor_SCHEDD_HOST environment variable set 7
High Throughput Computing Notebook Run it on your laptop Container will start up a personal HTCondor pool, and then Jupyter HTMap uses the personal pool Run it on a server that has both JupyterHub and a HTCondor Schedd connected to your site's pool Container will start Jupyter HTMap uses the entire site pool 8
Run htc-notebook on your laptop with a personal pool condor_schedd Container will start up a personal HTCondor pool, and then Jupyter HTMap uses the personal pool 9
Run htc-notebook on a remote server with a personal pool 10
Run htc-notebook on a remote server with a personal pool condor_schedd 11
Run on a remote server using entire site pool condor_schedd 12
Run on a remote server using entire site pool condor_schedd 13
Run on a remote server using entire site pool condor_schedd 14
Docker Hub (images) / GitHub Repos (src) https://hub.docker.com/u/htcondor https://github.com/htcondor/htc-notebook Run it on your laptop docker run p 8888:8888 htcondor/htc-base-notebook (then open your web browser as instructed) Pick a software environment! htcondor/htc-scipy-notebook htcondor/htc-tensorflow-notebook htcondor/htc-r-notebook htcondor/htc-pyspark-notebook htcondor/htc-datascience-notebook 15
Thank You Interested? Talk to us! Docker Hub (images) / GitHub Repos (src) https://hub.docker.com/u/htcondor https://github.com/htcondor/htc-notebook
