Accelerating Model Training: Performance Antipatterns Eliminator Framework

In the realm of ML/DL training pipelines, the training-specific data preparation of complex models may consume up to 87% of the total training time. A data scientist may build training pipelines using Python data structures on GPU while being unaware of the performance antipatterns that arise due to communication between CPU and GPU during model training, etc. These antipatterns may not be easily identifiable using traditional profiling tools alone. In this paper, we propose Performance Antipatterns Eliminator Framework (PAEF), a framework to identify six performance antipatterns occurring due to data movements between CPU and GPU during training. Our framework co-relates profiles of CPU and GPU executions of the pipeline along with the static analysis of the code to identify the performance antipatterns. We further replace these antipatterns with their performant versions. We evaluate the benefits of PAEF for two industrial recommendation models, where we showcase up to 7X speedup by using PAEF over the original pipeline.