Debugging and Optimization of PyTorch Models

Deep learning models are often viewed as uninterpretable "black boxes". As researchers, we often extend this thinking to the memory and compute utilization of such models. Using PyTorch Profiler, we can identify model bugs and bottlenecks to understand how to improve model performance from an efficiency perspective. This will improve training scaling and allow completion of large hyperparameter optimizations more efficiently. Here we will dicuss the usage of PyTorch Profiler, including some case studies of real training examples, and discuss possible optimizations based on profiler results. _______________________________________­________ This webinar was presented by Collin Wilson (SHARCNET) on September 11th, 2024, as a part of a series of weekly Compute Ontario Colloquia. The webinar was hosted by SHARCNET. The colloquia cover different advanced research computing (ARC) and high performance computing (HPC) topics, are approximately 45 minutes in length, and are delivered by experts in the relevant fields. Further details can be found on this web page: www.computeontario.ca/training-colloquia . Recordings, slides, and other materials can be found here: helpwiki.sharcnet.ca/wiki/Online_Seminars SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers (www.sharcnet.ca/). SHARCNET is a part of Compute Ontario (computeontario.ca/) and Digital Research Alliance of Canada (alliancecan.ca/).