Reliability in design science research

Reliability, which is concerned with whether something performs as it should, is well-accepted as an important part of any scientific research, with recognised methods for assessing it in the natural sciences. Reliability is a prerequisite for validity, which assesses knowledge production. Design science research is unique in that it produces an artefact and makes contributions to a design knowledge base and is tasked with the duality of design and science. Therefore, reliability should be reconsidered to address the assumptions underlying this type of information systems research.

This paper proposes a Design Science Reliability Framework. The framework identifies the part of design science research that should be evaluated (artefact, methods, measures and design knowledge). It also considers when reliability should be assessed as a project progresses by differentiating synchronic versus diachronic evaluation. The framework is used to derive a set of techniques for establishing reliability that can be applied during the evaluation process to both a project's knowledge production and its artefacts. Application of the framework should assist design science researchers when considering how to assess their results.