Interlaboratory study to evaluate background databases for the calculation of likelihood ratios in the interpretation of vehicle glass evidence using LA-ICP-MS data

Abstract

Glass samples were analyzed by 13 laboratories participating in an interlaboratory study that used laser ablation inductively coupled plasma mass spectrometry (LA-ICP-MS) with a standard test method (ASTM E2927–23) for the forensic analysis and comparison of vehicle glass. The aim of this study was to explore the performance of the application of a match criterion described in the standard test method and from likelihood ratio (LR) calculations when reporting the significance of glass evidence comparisons. Five (5) databases populated in different countries and combinations of the databases were used as background data to calculate LRs for two (2) casework scenarios involving vehicle glass comparisons. When the ASTM E2927–23 was used to compare vehicle glass samples that originated from the same source, all laboratories (except one) correctly reported the samples to be indistinguishable thus concluding that the possibility that the glass originated from the same source could not be eliminated. When the LR was calculated for the same comparison, most laboratories obtained large LR values (≈ 10,000) interpreted as “strong support” for same-source proposition. The LR rate of misleading evidence for the same-source (ROME-ss) comparisons was < 2 % for scenario 1. Comparing vehicle glass samples known to originate from different sources resulted in most laboratories reporting the glass to be “distinguishable” when using the ASTM standard method criterion or produced very small LR values (≈ 0.0001) when using the LR comparison criteria, interpreted as “strong (or very strong) support” for different-source proposition. The LR rate of misleading evidence for different-source (ROME-ds) comparisons for scenario 1 was < 21 %, which was due to the number of comparisons of glass samples that are chemically similar (different vehicles but same source of manufacturing). If the chemically similar glass comparisons from the same manufacturer were not treated as “different source”, the ROME-ds was reduced to zero. Glass samples that were chemically similar (those that originated from different vehicles but were collected from the same make, model, and year or originated from the same vehicle but a different pane of glass) sometimes resulted in an LR value (≈ 1) interpreted as no support of either proposition or that the possibility that the glass originated from the same source could not be eliminated when using the ASTM match criterion. The laboratories reported approximately 20 % false support for same-source proposition (or “false inclusion”) and 7 % false support for different-source proposition (or “false exclusion”) when using the ASTM match criterion in the first scenario. All “false inclusions” were derived from the comparison of chemically similar samples, such as inner and outer panes from the same windshield, thus “error rates” on this dataset should not be generalized outside of the context of this study. A database composed of about 2000 background samples originating from different countries and analyzed in different laboratories, produced consistent results. When examined for calibration, all databases and their combinations had “false exclusion” rates below 5 % as well as “false inclusion” rates below 0.5 % for the ASTM calculation. The rate of misleading evidence of LR for same-source comparisons for the databases and their combinations was below 2 % and the rate of misleading evidence for different-source comparisons was below 2 %. An empirical cross entropy (ECE) plot was used to evaluate the calibration of all the databases and their combinations, which resulted in the log-likelihood ratio cost (Cllr) of less than 0.02.