Measuring program understanding

Abstract

A number of recent investigations have been concerned with the study of programming languages and practices from a human factors perspective (Green, 1977; Shneiderman, 1977; Sime et al, 1973). These research efforts have examined differences in programming languages, statement alternatives and programming aids in an attempt to increase programmer productivity by suggesting language designs and guidelines that should result in more efficient program development efforts. Four measures of programming ability have been suggested for use in the psychological study of programming languages and practices: composition, which reflects an ability to write a program; comprehension, which reflects a capability to understand an existing program; debugging, which reflects a capability to detect errors in an existing program; and modification, which reflects an ability to insert changes into an existing program (Shneiderman, 1976). Of the four measures, comprehension is believed the more basic and hence of the most interesting from a human factors viewpoint (Sime et al, 1973). Program comprehension, or understanding, occurs at two levels (Shneiderman and Mayer, 1976). The first is termed semantic understanding and refers to the realization of the purpose of a program or a program module. The second is termed syntactic understanding and refers to the knowledge of a language's grammar. Semantic understanding is considered a higher level of understanding as it is believed that programs are not comprehended in a line-byline fashion based on syntax but in terms of modules relating to semantics. Also, semantic understanding is of prime importance when modification, composition and non-syntax debugging activity are attempted. Three measurement techniques are suggested for ascertaining degree of program understanding: questionnaires, program memorization/ recall and program bracketing. Questionnaires can require subjects to indicate the results of a program or program module given certain conditions ("tracing") or to indicate which conditions need to be set to provide a given result ("taxon identification"). It is believed the second type of question better ascertains program understanding (Green, 1977). Memorization/ recall refers to observing how many lines of a program can be recalled by a subject. It is believed that semantic modules are memorized and are then converted back to the original program via syntactic knowledge (Shneiderman, 1976, 1977; Shneiderman and Mayer, 1976). Scoring can either be objective (perfect recall) or subjective (functionally correct recall). Finally, it is believed that the identification of semantic modules by bracketing logicallyrelated program segments also may indicate program understanding (Shneiderman and Mayer, 1976). Little work has been reported tha~ validates these measures of program understanding. The major intent of the two studies to be reported was to provide some information on the validity of memorization as a measure of program understanding. Study I