Techniques for evaluating collaboration toolkits

The most reliable approach to evaluating a collaboration toolkit is to carry out field studies. This approach has the problem that it requires each project to be interdisciplinary and create a complete, working system, which seems too costly given the budget for a typical research project and the complexity of a collaboration toolkit; and more important, does not allow a project to converge incrementally towards a complete solution. Based on the lessons learned from our work and that of others on collaboration toolkits, we have identified several lower-cost techniques for evaluating collaboration toolkits including inspecting the design to see if requirements have been met, simulating other systems, implementing complete solutions to standard problems, and performing self and lab studies. This paper discusses these techniques, points out their pros and cons, identifies the influential projects in which they have been used, and shows how they fit together.