The Rashomon effect: which features of a speaker’s talk do listeners notice?

ABSTRACT The same L2 speaking performance may be analysed and evaluated in very different ways by different teachers or raters. We present a new, technology-assisted research design which opens up to investigation the trajectories of convergence and divergence between raters. We tracked and recorded what different raters noticed when, whilst grading a speaker’s videoed talk in the IELTS Speaking Test. We devised graphs of the ‘noticing trajectories’ of raters to compare the relative degrees of convergence and divergence between all raters at any point. The possible reasons for convergence/divergence were elicited using stimulated recall interviews. We found that raters’ noticing and rating tended to converge in relation to easily identifiable features such as idioms. However, for some stretches of speaker talk, there was clear divergence in terms of features noticed and rated by raters; in real IELTS tests, a background score-monitoring system deals with divergence. The research design enables delicate depiction of how the process of rater convergence/divergence unfolds over time. Applications are suggested in terms of examiner training, as well as for classroom discourse research and CA data workshops.