Can We Trust Privacy Policy: Privacy Policy Classification Using Machine Learning

Mobile applications frequently request privacy information from users to supposedly use to improve online service and applications. The collected data, such as personally identifiable information, raises users’ concerns since some applications actually have malicious intentions to leak personal data. Privacy policies are an important resource as they are the sole source of information users can easily gain access in order to determine how applications plan to collect and use their data prior to downloading and using the application. However, users tend to ignore or gloss over privacy policies as they are often written in the complicated hard-to-understand language. Thus, users often miss crucial privacy-related information after reading such documents. In this paper, we experimentally determine how much we can trust an application’s privacy policy by looking at the language used in more than 9,000 privacy policies and compare them to what the applications actually do. We attempt to classify whether or not applications transmit privacy-related information using machine learning with three classifiers, support vector machines (SVMs), k- nearest neighbors (KNN), logistic regression (LR). The best results show the average recall and precision of 0.81 and 0.31, respectively. High recall indicates that we are able to correctly identify most of the applications that transmit personally identifiable information. But, low precision indicates that we often over-identify applications as ones that transmit personally identifiable information when in reality they do not.