Extraction of Sound Signal from Tiny Vibrations in Motion Magnified Video Using Optical Flow

Abstract

Sound is a wave of pressure and displacement through the air or a different medium. When sound waves hit an object, they cause tiny vibrations which are impossible to see with the naked eye on the objects’ surface. A few techniques exist in the literature that can extract sound partially from the high-speed video. In this paper, it is shown that how using the only normal mobile camera, those vibrations can detect and partially recover the sound that produced them. Through this technique, any kind of lighter object like a glass of water, a box of tissues, or a bag of chips is turned into a visual microphone which allows to convert the subtle motions into a sound signal. This technique includes computing local motion signal which is based on the computation of optical flow velocity components for determining vibration signal and then computing global motion signal by squared average values of the local motion signals.