SMIAltmetric: A comprehensive metric for evaluating social media impact of scientific papers on Twitter (X)

Abstract

The rise of social media has significantly influenced scholarly communication, knowledge dissemination, and research evaluation, leading to the enrichment of alternative metrics (altmetrics) for evaluating academic papers’ social impact, which assesses the social impact of academic papers through online activities, including reading, bookmarking, downloading, and commenting. However, these altmetrics often focus on the number of mentions on social media rather than thoroughly evaluating the source, content, and dissemination of these mentions. To address this gap, this study introduces the social media impact altmetric (SMIAltmetric), which is based on 44,087 publications and 860,680 tweets (now “posts”), a comprehensive scoring system for evaluating scientific papers on Twitter (now “X”), using diverse features, including literature-related, social media engagement-related, user-related, and content-related features. Employing Altmetric Attention Acores (AAS) as labels, we tested eight machine learning algorithms, with XGBoost demonstrating the highest accuracy at 0.8672. Crucial factors influencing SMIAltmetric, as identified by the SHAP value, were followers, retweets, mentions, and citation. Furthermore, consistency analysis and convergent validation between the proposed SMIAltmetric and AAS confirm the reliability and finer differentiation of SMIAltmetric. The proposed SMIAltmetric provides a more comprehensive understanding of a paper’s social media impact, enhancing the evaluation of scientific discourse and its engagement with society.