Automated quality assessment of online medical science popularization Videos: A framework based on large language models

Abstract

Background

Unverified information in online medical science popularization videos poses significant public health risks. In the era of user-generated content, the sheer volume of daily video uploads overwhelms conventional human moderation, necessitating scalable automated assessment technologies. This study aims to develop and validate a deterministic large language model (LLM) framework for automated assessment of online medical science popularization videos.

Methods

A dataset of 121 Chinese dietary weight loss videos was curated from TikTok, Bilibili, and Kwai. Audio tracks were transcribed using OpenAI's Whisper-1 toolkit, excluding videos with excessive background noise or no spoken content. Videos were annotated by physicians using mDISCERN, a validated tool assessing clarity, reliability, fairness, references, and rigor. Thirteen LLMs—spanning general-purpose, medically tuned, and Chinese-optimized architectures—were evaluated via a novel likelihood selection strategy. This approach converted mDISCERN scoring into binary classification tasks, calculating answer likelihoods in a single forward pass to ensure deterministic outputs. Performance was measured using accuracy, F1 score, and Matthews correlation coefficient, with statistical significance assessed via paired Chi-square tests.

Results

Llama3-based models (Meta-Llama-3-8B-Instruct and Llama3-8B-Chinese-Chat) achieved the highest macro-average accuracy (79.50%), closely matching expert ratings in clarity (95.87%), references (97.52%), and rigor (99.17%). However, fairness assessment revealed systemic limitations, with 12/13 models scoring 19.83% accuracy, except Deepseek-R1-Distill-Llama-70B (70.25%).

Conclusion

Our study validates the potential of LLM-driven quality assessment for weight loss videos. Performance comparison emphasizes architecture over scale or medical specialization for reliable assessments. By open-sourcing the pipeline, we provide a foundational framework that holds promise for broader application in medical science popularization.