The Effects of Tablet-Based Electronic Grading on Airline Evaluator Performance

ABSTRACT Objective: This experimental study assessed the effects of an Apple iPad (Apple and iPad are trademarks of Apple Inc., registered in the U.S. and other countries) grading instrument on airline evaluator performance. Background: Extant research exploring the effects of grading formats on evaluator performance focused on pen and paper modalities. This research investigated how a contemporary iPad-based grading format affected airline evaluator performance. Method: Forty-five evaluators from a major U.S. airline graded a Boeing 767 (Boeing is a wordmark of The Boeing Company, registered in the U.S. and other countries) flight crew experiencing an engine failure during takeoff. Three grading formats were used: pen and paper, a paper-based equivalent of the iPad grading format designed to replicate the function of the iPad, and the iPad. The effects of the grading formats were measured using the following dependent variables: recorded technical criteria, recorded non-technical criteria, and correlation and inter-rater agreement with a referent score. Results: Members of the iPad paper-based equivalent and iPad groups recorded significantly more technical and non-technical criteria compared to participants using pen and paper (p < .001). Members of the iPad paper-based equivalent and iPad groups had significantly higher correlation (p = .01) and inter-rater agreement levels (p < .01) compared to participants using pen and paper. Conclusion: The results of the study failed to support the iPad as a superior grading format. Research outcomes suggested structured grading may be a more important factor in predicting airline evaluator performance than the type of grading format used.