Python is on the rise. Driven by the vastly growing number of machine learning and data analytics applications, Python has become one of the most popular and widely used programing languages. Even though, the HPC community is traditionally rather sceptical regarding the usage of interpreted script languages, growing user demands are hard to ignore - especially in the context of the current machine learning hype.
Fig. [1]: Projection of questions posted on Stackoverflow, by programming language.
In this talk, we will discuss some of the main challenges providing Python services on HPC systems at production level. This includes:
- security issues
- containment and control of Python processes in multi-user environments
- maintenance of the Python software stack
- GPU integration for Python machine learning workflows
- solutions for interactive workflows.
Towards a practical solution of these issues, we will focus on the containerization of Python environments and their integration into running HPC systems.
About the Authors:
- Dominik Straßel is the senior operator of Fraunhofers GPU-cluster and core developer of Fraunhofer’s Python software stack.
- Philipp Reusch is currently working on his masters thesis at ITWM. One aspect of his work is the optimization of the Python software stack on HPC systems.
- Janis Keuper leads the “Large Scale Machine Learning” Group at ITWM. His research is focusing at ML systems and HPC scale ML algorithms.
[1] Figure by hacckernoon.com, https://hackernoon.com/top-3-most-popular-programming-languages-in-2018-and-their-annual-salaries-51b4a7354e06
WATCH VIDEO
