Observer agreement on the morphology of porous cranial lesions: Results from a workshop at the 2019 meeting of the Paleopathology Association

Objective

This paper presents the results of a 2019 Paleopathology Association workshop that tested observer agreement on porous cranial lesion morphology and presence using multiple sets of existing guidelines for data collection.

Materials

Sixteen conference attendees of varying osteological experience served as observers. Three crania were assigned to each of four published guidelines for identifying and categorizing lesion morphology, for a total of twelve well-preserved human crania from the National Museum of Natural History Biological Anthropology Collections.

Methods

Observers assessed each cranium macroscopically according to its assigned set of guidelines.

Results

Observer concordance was higher using scoring guidelines with higher-quality photographs, such as the 2019 guidelines from Rinaldo and colleagues.

Conclusions

Data collection guidelines with high-quality color photos may support greater reliability of researcher-generated data on macroscopic skeletal features.

Significance

The conclusions of any research study are only as reliable as the data on which they are based. This work highlights the need for ongoing practices of quality control in a field in which much data results from individual judgement calls.

Limitations

Observer concordance is not a measure of observer accuracy. Sample size is insufficient to draw broadly generalizable conclusions on the reliability of data collected using the guidelines tested, and conference environments are not a facsimile of research settings.

Suggestions for further research

Iterative testing of methodological consistency using larger sample sizes and more non-pathological crania is advised to identify the factors that influence observer discordance and to improve guidelines for qualitative assessments.