Frame Selection for Producing Recipe with Pictures from an Execution Video of a Recipe

Abstract

In cooking procedure instruction, text format plays an important role in conveying quantitative information accurately, such as time and quantity. On the other hand, image format can smoothly convey qualitative information (e.g., the target food state of a procedure) at a glance. Our goal is to produce multimedia recipes, which have texts and corresponding pictures, for chefs to better understand the procedures. The system takes a procedural text and its unedited execution video as the input and outputs selected frames for instructions in the text. We assume that a frame suits to an instruction when they share key objects. Under this assumption, we extract the information of key objects using named entity recognizer from the text and object detection from the frame, and we convert them into feature vectors and calculate their cosine similarity. To enhance the measurement, we also calculate the scene importance based on the latest changes in object appearance, and aggregate it to the cosine similarity. Finally we align the instruction sequence and the frame sequence using the Viterbi algorithm referring to this suitability and get the frame selection for each instruction. We implemented our method and tested it on a dataset consisting of text recipes and their execution videos. In the experiments we compared the automatic alignment results with those by human annotators. The precision, recall, and F-measure showed that the proposed approach made a steady improvement in this challenging problem of selecting pictures from an unedited video.