Automated identification of discourse markers using NLP approach: The case of okay

Abstract

This study aimed to examine the potentials of the NLP approach in detecting discourse markers (DMs), namely okay, in transcribed spoken data. One hundred thirty-eight concordance lines were presented to human referees to judge the functions of okay in them as a DM or Non-DM. After that, the researchers used a Python script written according to the POS tagging scheme of the NLTK library to set rules for identifying cases where okay is used as non-DM. The output of the script was compared to the reference human-annotated data. The results showed that the script could accurately identify the function of okay as DM or non-DM in 92% of the cases. The inaccuracy of detecting the rest was found to be caused by a lack of proper and detailed punctuations. The main implications of the results are that new NLP approaches can detect DMS; however, proper punctuation is required to enable the proper identification of DMs. In accordance with the findings, the researcher recommended adopting the approach after conducting further comprehensive studies.