IORapp: An R tool for Inter-Observer Reliability Assessment of Time and Motion data

Abstract

Direct observational time and motion studies, in which one or more observers continuously observe and record an individual's activities over a certain period of time, can provide valuable insights into work and communication patterns and assist in identification of areas to intervene to support safe and effective work. These studies have increased in number and sophistication, also due to the availability of software tools for PDA to support the observers. A key methodological issue for these studies is the assessment of Inter-Observer Reliability (IOR). IOR is crucial for ensuring reliability of data collection in which multiple observers are involved. In workflow time studies the multivariate, time-stamped and ordered nature of the data limits the applicability of traditional inter-rater reliability measures, and makes this assessment challenging. Measures such as Cohen's Kappa, in fact, are only applicable to one variable at the time, so that high K scores for one aspect can be achieved even if two observers disagree substantially on other variables which are the object of their observation. Secondly, computing these measures first requires either matching pairs of datapoints from different observers viewing the same entity, a problem that cannot be done with perfect certainty, or restructuring the data into fixed-length time window sequences. No single method can address all the different aspects on which observers in time and motion studies can disagree, so one should adopt a composite method. We developed a set of functions and a Shiny app to assist this process for data collected with the Work Observation Method By Activity Timing (WOMBAT) method. The app allows the loading of data from pairs/groups of observers and computes a wide set of agreement measures on both matched data and on time window data. These measures are presented in a dashboard along with interactive visualizations of the observers’ data, and can be saved and plotted over time in a different section.