Formative evaluation: a technique for attaining people-oriented systems

Let me tell you what I mean by formative evaluation. For example, in the design of something like Sesame Street or FreeStyle, social science researchers have gotten away from the role they're usually cast in - that is, to come in afterwards and explain why things failed. Instead, they get in right near the beginning and try to make sure that things turn out to be successes. You've all heard of the success of Sesame Street, and you're probably beginning to hear of the success of FreeStyle. When you have a big project and there is a great disparity between the people who are doing it and the people they're supposed to be impacting or who are going to be their users, the possibilities of misunderstanding and potential failure are great. This is a very common problem with large systems that are being designed today. Even though people think they are talking the same language, they find out they aren't. We have a lot of systems that are designed now where the system designers or the system analysts come up with specifications, let out bids, someone gets the bid, and that supposedly is it. Although misunderstandings between the systems analysts and the contractors may exist, that doesn't guarantee that the system analysts have a very good understanding of the end users. The idea in formative evaluation is that as you're getting the design put together and putting the concepts in place, you go out and start gathering representative data from the end user population to see whether in fact it is possible to attain the goals. Everybody knows that design is an iterative process, and evaluation should be a component of it. We give it lip service but very rarely do we do it, and the cliches are, "Well, you know the users don't know what they're going to want, they're going to keep on changing." To some extent that is true, but that is very often used as an excuse to skip the data gathering completely. By focusing on the misunderstandings, by trying to clarify the goals, one can early on learn whether in fact a system can be developed that is useable and satisfactory to those users.