Leveraging LLMs for action item identification in Urdu meetings: Dataset creation and comparative analysis

Abstract

In response to the increasing number of online meetings, automation of action items identification in online Urdu meetings, has become crucial. To serve this purpose, this research presents the first ever dataset and guidelines for annotating action items in code-mixed Urdu-English language. Collected dataset comprises of 240 recorded meetings, 600 fabricated action items, and 250 real meeting action items, totaling 2948 action items. We evaluated the efficiency and accuracy of various deep learning and machine learning models through a comparative analysis on a balanced dataset being discussed in Section 4.2. Additionally, three Large Language Models (LLMs) BLOOMZ, LLaMA, and GPT-3.5 were tested using zero-shot and few-shot configurations. BLOOMZ and LLaMA were specifically fine-tuned to enhance their performance in recognizing Urdu meeting action items. The fine-tuned model, ur_BLOOMZ-1b1, achieved the highest average F1 score of 0.94, surpassing all other traditional models. This study lays a solid foundation for future research in multilingual environments and advances our understanding of action item identification in Urdu meetings.