Towards BPM Skill Assessment using Computerized Adaptive Testing

The assessment of modeling skills is a topic that has hardly been discussed in literature. Being able to assess a modeler's skill level, however, provides valuable input for planning further educational interventions or allows for upfront estimations of capabilities to successfully perform a real-world modeling task. Literature indicates that skill levels can be estimated based on the capability to identify mistakes in given models. We propose to use the metrics specified in the 3QM framework as a foundation for assessing technical modeling skills. Based on these metrics, we have developed testing items for assessing modeler's capabilities to recognize syntactic, semantic and pragmatic issues in models, which serve as an indicator for their modeling skill level. These items are embedded in a computerized adaptive testing platform, which uses item response theory to estimate skill levels. We introduce the testing platform and finally describe the implementation of our test system prototype, which demonstrated the technical feasibility of the chosen approach.