A novel framework for evaluating developers’ code comprehension proficiency through technical and non-technical skills

Context:

Code comprehension is an essential software maintenance skill, where technical skills are often considered the primary benchmark for evaluating developers’ proficiency, overlooking the significant role of non-technical skills.

Objective:

Our work aims to propose a generalized framework for measuring developers’ code comprehension proficiency by integrating technical and non-technical skills, inspired by cognitive attraction networks, and conducting an empirical study to evaluate code comprehension proficiency based on selective skills.

Methods:

The generalized framework evaluates developers’ technical and non-technical skills separately using collected data and computes their respective indices to derive an overall measure of code comprehension ability, represented as the comprehension measure index (CMI). Additionally, an empirical study with 158 participants assessed technical skills, including code understanding, debugging, and completion, alongside non-technical skills such as problem-solving, emotions, long-term memory, belief, desire, intention, and commitment to compute their overall code comprehension proficiency.

Results:

Based on the obtained indices values related to technical and non-technical parameters, the study identifies multiple factors affecting participants’ performance, including lack of technical knowledge, reliance on guesswork, stress intolerance, lack of commitment and desire, difficulty understanding logic, inability to recall concepts, and check other contributing factors. To enhance our results K-means clustering is done to group the participants into three clusters according to their performance.

Conclusion:

Integrating technical and non-technical skills enables a more accurate assessment by addressing factors beyond technical expertise. The framework can help managers and tutors identify strengths and weaknesses, allowing task assignments that align with strengths of developers while addressing areas for improvement.