Network-Enabled Optimization System for Job Solver Categories

The content discusses neos, a Network-Enabled Optimization System, its mathematical formulation, and job solver categories such as bco, co, cp, go, kestrel, lno, ndo, and more. It covers optimization, management of servers, specialized solvers, and usage reports in a detailed manner.

• Optimization
• Solver Categories
• Server Management
• Job Solvers
• Mathematical Formulation

geen880 Follow

Uploaded on Sep 06, 2024 | 1 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. neos & Optimizing Your World

2. neos: Network-Enabled Optimization System Mathematical Formulation Maximize j Vbjyj (i,j) Adijxij Subject to exiting node on tour (i,j) Axij yi=0, i V Subject to entering node on tour (i,j) Axij yi=0, j V neos AMPL Model set V; set LINKS := {i in V, j in V: i <> j}; param alpha >= 0; param d{LINKS} >= 0; param b{V} >= 0; #default benefit of visiting a bar param c{V} >= 0; # default cost of one drink param B default 30; # default maximum budget for drinks http://www.neos-guide.org/content/bar-crawl-optimization neos &H HT TC Condor

3. neos Workflow Optimization Job Solver Categories bco co cp go kestrel lno ndo neos lp sdp sio slp socp uco Results milp minco miocp nco Browser Web Solver Names csdp ddsip DICOPT Domino DSDP feaspump FilMINT filter filterMPEC Gurobi icos Ipopt KNITRO LANCELOT L-BFGS-B LINDOGlobal AlphaECP ASA BARON BDMLP BiqMac BLMVM bnbs Bonmin bpmpd Cbc Clp concorde CONDOR CONOPT Couenne CPLEX LOQO LRAMBO MILES MINLP MINOS MINTO MOSEK MUSCOD-II NLPEC NMTR nsips OOQP PATH PATHNLP PENBMI PENSDP PGAPack proxy PSwarm QSopt_EX RELAX4 SBB scip SD SDPA SDPLR SDPT3 SeDuMi SNOPT SYMPHONY TRON XpressMP Email Email Solver Pool at UW-Madison Custom XMLRPC Off-Site Solvers Argonne National Lab Arizona State University University of Klagenfurt University of Minho Solver Inputs jpg LP MATLAB_BINARY MOSEL MPS Kestrel (AMPL or GAMS) AMPL C CPLEX Fortran GAMS OSIL RELAX4 SDPA SDPLR SMPS SPARSE SPARSE_SDPA TSP ZIMPL

4. Jobs Per Month 140000 120000 100000 Minho 80000 Klagenfurt 60000 Arizona State Argonne 40000 UW Madison 20000 0 neos &H HT TC Condor

5. Our Pool Rackmount Dell Servers running RedHat Managed via puppet: http://puppet.com/ Usage Reports in Ganglia: http://ganglia.info/ 1 Central Manager 1 Submit Node 5 Execute Nodes with specialized solvers neos &H HT TC Condor

6. Started Homegrown neos scip ASU Incoming Which Solver? neos neos Solver-1 CPLEX CPLEX CPLEX Solver Pool at UW-Madison CPLEX Queue (March 2014) neos Solver-2 neos &H HT TC Condor

7. Problems Solves Resource Allocation 3G memory per job 8 hour maximum 1-4 cores dependent on solver Adjust priority by length of job Rapid solver additions with load balancing New node has over 2x CPU and memory Smaller servers were overloaded # Original don t overload one by one # APPEND_RANK = ( -1 * TARGET.TotalLoadAvg ) # Updated (slightly overloads largest) APPEND_RANK = (TARGET.TotalCpus TARGET.TotalLoadAvg) neos &H HT TC Condor

8. User Authentication Wishlist Provide additional resources and priority to some Allow users to track, easily access jobs Better analytics, tracking, problem resolution Monetize? Integrated user authentication code Name, Username, Password, Email LDAP, MySQL backend PHP/JavaScript handler headers on existing pages Server hooks to connect user to job, give priority neos &H HT TC Condor

9. Solutions and Diminishing Returns 1 0.5 0 0 8 16 24 8 hour job limit Partial Results have value neos &H HT TC Condor

10. Problem: Partial Results 1. Streaming results Only on stdout, stderr Doesn t pass files created by solvers stream_output = true stream_error = true 2. Spool on evict Undocumented debugging option for evicted jobs Didn t work through multiple HTCondor version updates SpoolOnEvict = true 3. Transfer Output First solution of Research Computing Facilitators Works! when_to_transfer_output = ON_EXIT_OR_EVICT want_graceful_removal = true +SpoolOnEvict = false neos &H HT TC Condor

11. Ongoing Projects HTCondor Migrate to Python bindings Tie job to neosjob General Adding Authenticated User to XMLRPC neos &H HT TC Condor

12. Acknowledgements http://www.neos-guide.org/content/domino-art neos &H HT TC Condor