Avatar: Large Scale Entity Resolution of Heterogeneous User Profiles

Entity Resolution (ER), also known as record linkage or de-duplication, has been a long-standing problem in the data management space. Though an ER system follows an established pipeline involving the Blocking -> Matching -> Clustering components, the Matching forms the core element of an ER system. At CareerBuilder, we perform de-duplication of massive datasets of people profiles collected from disparate sources with varying informational content. In this paper, we discuss the challenges of de-duplicating inherently heterogeneous data and illustrate the end-to-end process of building a functional and scalable machine learning-based matching platform. We also provide an incremental framework to enable differential ER assimilation for continuous de-duplication workflows.