Pre-annotating Clinical Notes and Clinical Trial Announcements for Gold Standard Corpus Development: Evaluating the Impact on Annotation Speed and Potential Bias

In this study our aim was to present a series of experiments to evaluate the impact of pre-annotation: (1) on the speed of manual annotation of clinical notes and clinical trial announcements; and (2) test for potential bias if pre-annotation is utilized. The gold standard was 900 clinical trial announcements from clinicaltrials.gov website and 1655 clinical notes annotated for diagnoses, signs, symptoms, UMLS CUI and SNOMED CT codes. Two dictionary-based methods were used to pre-annotate the text. Annotation time savings ranged from 2.89% to 29.1% per entity. The pre-annotation did not reduce the IAA or annotator performance but reduced the time to annotate in every experiment. Dictionary-based pre-annotation is a feasible and practical method to reduce cost of annotation without introducing bias in the process.