MolAR: Bringing Chemical Structures to Life with Augmented Reality and Machine Learning

Abstract

Visualizing three-dimensional molecular structures is crucial to understanding and predicting their chemical behavior. Existing visualization software, however, can be cumbersome to use, and, for many, hand-drawn skeletal structures remain the preferred method of chemical communication. Although convenient, the static, two-dimensional nature of these drawings can be misleading in conveying the molecule’s 3D structure, not to mention that dynamic movement is completely disregarded. Here, we combine machine learning and augmented reality (AR) to develop MolAR, an immersive mobile application for visualizing molecules in real-world scenes. The application uses deep learning to recognize hand-drawn hydrocarbons structures which it converts into interactive 3D molecules in AR. Users can also “hunt” for chemicals in food and drink to uncover molecules in their real-life environment. A variety of interesting molecules are pre-loaded into the application, and users can visualize molecules in PubChem by providing their name or SMILES string and proteins in the Protein Data Bank by providing their PDB ID. MolAR was designed to be used in both research and education settings, providing an almost barrierless platform to visualize and interact with 3D molecular structures in a uniquely immersive way.