Python3 Migration Readiness at J.P. Morgan: Strategies and Challenges
Explore the comprehensive steps taken by J.P. Morgan and Athena to prepare for a massive Python3 migration. Learn about reducing code size, analyzing runtime, assessing external libraries, and implementing preventive measures in this insightful journey towards Python3 adoption on a large scale.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Readiness for Python3 on a Massive Scale Yehuda Lavy yehuda.lavy@jpmorgan.com PyCon Israel 2018
We all know what may happen ! Image result for indiana jones switch skull
Agenda J.P. Morgan and Athena Getting ready for Python3 migration Q&A
J.P. Morgan 250,000+ employees globally 40,000+ employees in Technology $9.5B+ annual investment in technology Offices all around the globe, including Herzlia
Athena Pricing, risk management, analysis and trade management platform Rapid development and deployment model that puts developers at the heart of the business ~800 deliveries to production daily, in a continuous commit flow, with no versioning 95% Python 1500+ Python developers ~30M lines of code, ~30K of files
Readiness Steps for Python3 Migration Reduce code size Readiness of external libs Preventive measures Plan changes by dependency SDLC (Software Development Life Cycle) responsibilities
Reduce Code Size - Static Analysis Use vulture package: Unused Import or Variable can be found by pyflake\pylint too Extra full search in code is advised for findings of unused class or function
Reduce code size - Runtime Analysis Finding unused modules: We could not use coverage module due to the size of the code, and as unused apps may call the unused code Our solution: Before each app terminates, it logs sys.modules.keys() and some metadata Find unused code by comparing logs with modules in our code
Readiness of external libs Use of caniusepython3 package Can be added also as a pylint rule
Preventive measures C bindings: Reduce number of different binding techniques (e.g. CFFI and SWIG). Stay with mainly Cython and pybind Improve test coverage by priority of functionality Pay attention to Byte strings and pickle implementation Implementing new linters for new code, (e.g. pylint, pyflake, 2to3)
Plan changes by dependency Use snakefood package to produce dependency graph. Start changes for readiness from the bottom (no dependencies) Mark ready files, change files one level up Continue to change the code to the next level recursively
SDLC responsibilities Automated: Run readiness (for example, use futurize or modernize package) Run tests Manual: Approve the changes after the change Fix failures of code and tests
Summary Reduce the code size to be migrated Better safety net(Tests, Early detection of possible failures) Incremental changes by dependency Automate as much as possible
Q&A http://www.jpmorgan.com/techcareers
