Developing competency frameworks using natural language processing: An exploratory study

Abstract

Background

Competency models provide useful frameworks for organizing learning and assessment programs, but their construction is both time intensive and subject to perceptual biases. Some aspects of model development may be particularly well-suited to automation, specifically natural language processing (NLP), which could also help make them more generalizable and thus more learner-centric.

Aims

In this study, we sought to evaluate the potential for NLP techniques be applied to competency framework development.

Materials & Methods

Using NLP, we developed a set of new competency frameworks from a sample of existing leadership competency models from the health professions (e.g. nursing, medicine, healthcare management, social work, spiritual care). We then arranged for a human reviewer who was blind to the frameworks’ sources to evaluate their relative coherence.

Results

The human-developed frameworks tended to be viewed as more coherent than the NLP-generated frameworks, however the coherence advantage was greatest for the least complex models, and there was no apparent advantage in the most complex model we tested.

Discussion

Although NLP did not consistently outperform the human-developed model structures, the pattern of results suggested directions for further model refinement and future study.

Conclusion

Replicating this research with a broader sample of competency models will be important for establishing whether the observed relationship between NLP performance and model size is a more widely generalizable principle.