Constructing query-biased summaries: a comparison of human and system generated snippets

Modern search engines display a summary for each ranked document that is returned in response to a query. These summaries typically include a snippet -- a collection of text fragments from the underlying document -- that has some relation to the query that is being answered. In this study we investigate how 10 humans construct snippets: participants first generate their own natural language snippet, and then separately extract a snippet by choosing text fragments, for four queries related to two documents. By mapping their generated snippets back to text fragments in the source document using eye tracking data, we observe that participants extract these same pieces of text around 73% of the time when creating their extractive snippets. In comparison, we notice that automated approaches for extracting snippets only use these same fragments 10% of the time. However, when the automated methods are evaluated using a position-independent bag-of-words approach, as typically used in the research literature for evaluating snippets, they are scored much more highly, seemingly extracting the "correct" text 24% of the time. In addition to demonstrating this large scope for improvement in snippet generation algorithms with our novel methodology, we also offer a series of observations on the behaviour of participants as they constructed their snippets.