Efficient Job Scheduling and Runtime Management in DLWorkspace Cloud Computing and Storage Group

Explore the intricate system of job scheduling and runtime management in DLWorkspace, involving SQL server, K8s Master API, Web Portal, Restful API, Cluster Manager, NVIDIA driver plugins, and shared storage. Learn about the process flow from job submission to approval, status monitoring, and device mapping for efficient job execution.

• Cloud Computing
• Job Scheduling
• Runtime Management
• DLWorkspace
• NVIDIA Driver

maje_986 Follow

Uploaded on Oct 03, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Job Scheduling and Runtime in DLWorkspace Cloud Computing and Storage Group July. 7th, 2017

2. System Diagram SQL server Job K8s Master API Web Portal RestfulAPI Cluster Manager

3. SQL server Job K8s Master API Web Portal RestfulAPI Cluster Manager Web Portal: Authentication Get job parameters from users and submit the request to RestfulAPI Browse and manage the existing jobs Monitor the cluster status etc

4. SQL server Job K8s Master API Web Portal RestfulAPI Cluster Manager RestfulAPI: Process the request from web portal SubmitJob ListJobs KillJob GetJobDetail GetClusterStatus ApproveJob etc

5. SQL server Job K8s Master API Web Portal RestfulAPI Cluster Manager

6. SQL server Job K8s Master API Web Portal RestfulAPI Cluster Manager Cluster Manager: Job manager Get new submitted jobs from SQL server, generate k8s pod description file and submitted to k8s master api. The pod description file is generated from templates. Query job status from k8s api and update the job status to SQL server etc Log manager Node manager User manager etc

7. SQL server Job K8s Master API Web Portal RestfulAPI Cluster Manager

8. DLWorkspace Job Runtime Nvidia driver plugin Shared storage Special permission Special device mapping

9. DLWorkspace Job Runtime - Nvidia driver plugin Install nvidia driver on the host machine CoreOS: use privileged Docker to insert kernel module Ubuntu: apt-get install nvidia-*** Official Kubernetes: Put driver libraries to a folder e.g. /opt/nvidia-driver/ Map the driver folder to container (the Docker image should be inherited from nvidia/cuda) Our customized Kubernetes: Call nvidia-docker-plugin to create a Docker volume for nvidia driver libraries Mount the Docker volume to container

10. DLWorkspace Job Runtime - Shared Storage All the shared storage are mounted on the host and then mapped to the container Storage mount point DLWorkspace system folder storage, work, jobfiles Soft link from storage mount point to system folder Samba interface to allow users access their home folder (work folder) and data folder from windows machines (domain machines)

11. DLWorkspace Job Runtime - Special permission E.g. run privileged Docker Special approval work flow is supported (On going ) If the cluster is configured to allow special permission, it may require additional approval from the system admin