Distributed versus face-to-face meetings for architecture evalution: a controlled experiment

Scenario-based methods for evaluating software architecture require a large number of stakeholders to be collocated for evaluation sessions. Collocating stakeholders is often an expensive exercise. To reduce expense, we have proposed a framework for supporting software architecture evaluation process using groupware systems. This paper presents a controlled experiment that we conducted to assess the effectiveness of scenario profile construction using distributed meetings. We used a cross-over experiment involving 32 teams of three 3rd and 4th year undergraduate students. We found that the quality of scenarios produced by distributed teams using a groupware tool were significantly better than the quality of scenarios produced by face-to-face teams (p<0.001). However, questionnaires indicated that most participants preferred the face-toface arrangement (82%) and 60% thought the distributed meetings were less efficient. We conclude that distributed meetings are extremely effective but that tool support must be of a high standard or participants will not find distributed meetings acceptable.