From the Daily Peer Review of Abnormal Pap Test Slides to the Monitoring of Individual and Laboratory Performances: 5 Years of Data Collection and New Potential (Key) Performance Indicators

Abstract

Objectives

The Peer Review (PR) consists of the daily examination, by all cytologists, of Pap slides that resulted abnormal/difficult, in order to reach a consensus on the final diagnosis (FD). We explore data from 5 years (2017–2021) of PR to: (i) evaluate the agreement (both inter-observer and versus FD) over time; (ii) identify new quality indicators.

Methods

5673 slides were submitted to PR and examined by an average of 8 cytologists (range: 4–13). The agreement between cytologists and between the individual diagnosis with FD were evaluated by Kappa (k) and weighted Kappa (wK) and compared between ‘experts’ and ‘less experienced’ readers.

Results

The inter-observer agreement showed a moderate agreement among readers (whole team k = 0.44; experts k = 0.48). The highest and the lowest agreement was reported in HSIL and ASC-H, respectively. In 2018 and 2021, a significant reduction of kappa was observed, likely attributable to team turnover. The laboratory agreement versus FD resulted in significantly higher scores in experts (wk = 0.73, 95% CI 0.73–0.74) compared to less experienced individuals (wk = 0.65, 95% CI 0.64–0.66), with a general reduction of wk recorded in 2021. The individual agreement versus FD (calculated for 16 cytologists) achieved a moderate/substantial level of agreement (wK range: 0.57–0.80), with a shift toward higher wk in experts.

Conclusion

The levels of agreement are influenced by cytologist experience and team turnover. We propose new potential (key) performance indicators to strictly monitor the occurrence of systematic differences in interpretation criteria among cytologists. The proposed reference values are based on preliminary data and should be validated prospectively over a longer monitoring period.