sbv IMPROVER Diagnostic Signature Challenge

Abstract

The sbv IMPROVER (systems biology verification—Industrial Methodology for Process Verification in Research) process aims to help companies verify component steps or tasks in larger research workflows for industrial applications. IMPROVER is built on challenges posed to the community that draws on the wisdom of crowds to assess the most suitable methods for a given research task. The Diagnostic Signature Challenge, open to the public from Mar. 5 to Jun. 21, 2012, was the first instantiation of the IMPROVER methodology and evaluated a fundamental biological question, specifically, if there is sufficient information in gene expression data to diagnose diseases. Fifty-four teams used publically available data to develop prediction models in four disease areas: multiple sclerosis, lung cancer, psoriasis, and chronic obstructive pulmonary disease. The predictions were scored against unpublished, blinded data provided by the organizers, and the results, including methods of the top performers, presented at a conference in Boston on Oct. 2–3, 2012. This paper offers an overview of the Diagnostic Signature Challenge and the accompanying symposium, and is the first article in a special issue of Systems Biomedicine, providing focused reviews of the submitted methods and general conclusions from the challenge. Overall, it was observed that optimal method choice and performance appeared largely dependent on endpoint, and results indicate the psoriasis and lung cancer subtypes sub-challenges were more accurately predicted, while the remaining classification tasks were much more challenging. Though no one approach was superior for every sub-challenge, there were methods, like linear discriminant analysis, that were found to perform consistently well in all.