Empirical Evaluation of Word Representation Methods in the Context of Candidate-Job Recommender Systems

Abstract

In this paper, we have evaluated our hybrid two-way recommendation system with expert-ranked resumes and job descriptions. The aim of the paper is to compare the lists produced by the recommendation system with human-ranked lists for candidate and job descriptions. Firstly, we set up four scenarios such as the matching of resume to resumes, job to jobs, resume to jobs, and job to resumes, and prepared a human ranking based on the content similarity on a total of 400 documents. Based on this annotated corpus we tested our system to calculate the cosine-similarity-based ranking for each scenario using the Global Vectors for Word Embeddings and Term Frequency-Inverse Document Frequency representations. Finally, we compared the similarities of human ranked lists and system-ranked lists by using the Rank Biased Overlap (RBO) similarity score. In both methods, GloVe and TF-IDF, the median RBO between human-ranked lists and system ranked are greater than 0.5. The highest median score is achieved on TF-IDF with a slight difference compared to GloVe apart from the ranking of resume-to-resume scenario where the variation between the two methods is considerable. This is due to the similarity between human-ranked lists and program-generated lists.