An accurate and customizable text classification algorithm: Two applications in healthcare

Text classification is an important step in many data analysis procedures. The demand on text classification algorithm is booming due to the increase of the amount of digital data, especially in the healthcare field. A customizable and accurate algorithm is expected to produce positive impact on the efficiency of many data analysis procedures. In this work, we proposed a novel algorithm for accurately classifying data entries in huge text files into several pre-determined categories. We built the algorithm with multiple rules according to text similarity, frequency, and weight. For different classification tasks, the algorithm can be conveniently adjusted to process the corresponding data sets. Data sets related to healthcare cost analysis (hospital discharge summary) and medical classification systems (ICD-9) are used to evaluate the algorithm. When the algorithm is used on the ICD-9 data, the overall accuracy of the algorithm was 100%. After the algorithm was used on 7480 healthcare cost entries, the results were then compared with the ones processed manually by a physician, and the accuracy was between 86%–91.6%, and the difference is from different classification of ambiguous entries, which is hard to determine the correct category even when it is done manually because those entries were documented improperly. This new classification algorithm is 3 to 5 times faster than the manual process on the same data set. Therefore, this customizable and accurate text classification algorithm is effective in saving time compared to the manual classification methods.