Investigation of a gambling approach to the subjective entropy test

Abstract

Researchers currently arc investigating a number of possible practical solutions for the problem of measuring program comprehension on a pragmatic level. Two well known tests arc the comprehension test and the Cloze lest [1], Perhaps less well known is the Subjective Entropy Test (SET) proposed by Zunde [2], This paper describes an investigation into a new version of SET called SETBET. We wished lo determine whether adding a betting scheme to the SET would change its performance. The conventional comprehension lest is probably the most intuitive of the tests. Such tests generally ask the subject to "play computer". For example, he is given a program and asked to determine what it does, or he is given a program containing a bug and asked to correct it. However, the problems with the comprehension test are that it is very time consuming lo prepare and that it re­ quires subjective grading. Because of these problems, other tests, such as the Cloze test, have been investigated. The Cloze lest consists of filling in missing parts in a program for which the respondent has to supply the correct word. One advantage of the Cloze lest over the comprchensi n type test is that the Cloze tests are simpler lo pre­ pare. Another advantage is grading, which can be totally objective if exact answers are required, or it can be some­ what more subjective if elements such as variable names can be changed. A problem with the Cloze test is that it is sensitive to bias and to the stimulus materials [3] and |1], Problems associated with this test motivated more re­ search. One result of the new research by Zunde [2] is a lest called SET.