Analyzing Human Observer Ability in Morphing Attack Detection—Where Do We Stand?

Abstract

Morphing attacks are based on the technique of digitally fusing two (or more) face images into one, with the final visage resembling the contributing faces. Morphed images not only pose a challenge to Face-Recognition Systems (FRS) but also challenge experienced human observers due to high quality, postprocessing eliminating any visible artifacts, and further the printing and scanning process. Few studies have concentrated on examining how human observers can recognize morphing attacks, even as several publications have examined the susceptibility of automated FRS to morphing attacks and offered morphing attack detection (MAD) approaches. MAD approaches base their decisions either on a single image with no reference to compare against (Single-Image MAD (S-MAD)) or using a reference image (Differential MAD (D-MAD)). One prevalent misconception is that an examiner’s or observer’s capacity for facial morph detection depends on their subject expertise, experience, and familiarity with the issue. No works have reported the specific results of observers who regularly verify identity (ID) documents for their jobs. As human observers are involved in checking ID documents having facial images, a lapse in their competence can result in significant societal challenges. To assess the observers’ proficiency, this research first builds a new benchmark database of realistic morphing attacks from 48 different subjects, resulting in 400 morphed images. Unlike the previous works, we also capture images from Automated Border Control (ABC) gates to mimic realistic border-crossing scenarios in the D-MAD setting with 400 probe images, to study the ability of human observers to detect morphed images. A new dataset of 180 morphing images is also produced to research human capacity in the S-MAD environment. In addition to creating a new evaluation platform to conduct S-MAD and D-MAD analysis, the study employs 469 observers for D-MAD and 410 observers for S-MAD who are primarily governmental employees from more than 40 countries, along with 103 control group members who are not examiners. The analysis offers intriguing insights and highlights the lack of expertise and failure to recognize a sizable number of morphing attacks by experienced observers. Human observers tend to detect morphed images to a lower accuracy as compared to the four automated MAD algorithms evaluated in this work. The results of this study are intended to aid in the development of training programs that will prevent security failures while determining whether an image is bona fide or altered.