Development of a brief test protocol for assessing cognitive change in the peri-operative period in older adults undergoing elective surgery

Many patients are afraid of permanent cognitive impairment after general anaesthesia [1]. However, there is no uniform method to diagnose postoperative neurocognitive disorders extending beyond 1 month after surgery [2]. The gold standard is a time-consuming neuropsychological assessment [3]. We aimed to identify and validate internally the best predictive subset of neuropsychological assessment tests, balancing accuracy and utility suitable for both research and clinical purposes.

This study used data from a previous study [3]. After ethical approval and informed consent, patients aged ≥ 65 y undergoing elective surgery were enrolled. Patients with hearing impairment, multiple procedures under anaesthesia or pre-existent cognitive impairment were not studied. Neuropsychological assessment was done pre-operatively and 4–8 weeks postoperatively. It covered five cognitive domains, with outcomes for 17 individual subtests and four combined scores (online Supporting Information Appendix S1). Outcomes were reported in T-scores, corrected for age and educational level compared with a healthy Dutch patient group [4]. Missing outcomes were imputed.

Our previous study used a definition based on composite cognitive domain scores, yielding an 18% incidence of postoperative neurocognitive disorder [3]. However, as predictors would be based on the results of individual tests, we defined postoperative neurocognitive disorder in this analysis as a decline of ≥ 1 SD on ≥ two tests in at least one cognitive domain, or a decline of ≥ 1 SD in total cognitive domain score, aligning with the most used research definition [5]. After checking for multicollinearity, we aimed to create a logistic regression model (online Supporting Information Appendix S2). All 17 individual delta test scores were included, and backward selection based on the Akaike Information Criterion was performed, using different cut-offs. When backward selection included the middle or last subtest from a test, we deemed it necessary to also include the other subtest (e.g. Stroop 1, 2 and 3). When this would make the test protocol too time-consuming, we selected the tests with the lowest p values and their subtests, while also checking AUROC, so time of administration would be acceptable. Internal validation was done using bootstrapping, where AUROC and regression coefficients were corrected uniformly for measured optimism. Calibration was assessed and thresholds determined. Predictions were made using the formula $\frac{1}{1+e^{-\left({\beta }_0+{\beta }_1*x1+\dots \right)}}$. Sensitivity and specificity of the final selected subset were tested against the composite domain definition. Full statistical analyses can be found in online Supporting Information Appendix S3.

In total, 77 patients had complete data and all baseline characteristics can be found in online Supporting Information Appendix S4. Neuropsychological assessment diagnosed 32 (42%) patients with postoperative neurocognitive disorder. After backward selection the selected test protocol was approximately 30 min and deemed too time-consuming (Table 1). Therefore, we included the three tests with the lowest p values, as this gave the best balance between administration time and AUROC. The final model included the Digit Span Test (forward, backward and sorting), Stroop (1, 2, 3), and Auditory Verbal Learning Test with an approximate duration of 20–24 min and yielded an AUROC of 0.91 (95%CI 0.83–0.98) before and 0.86 (95%CI 0.77–0.97) after internal validation (Table 1). Entering the delta T-scores into the model gave a predicted probability on the presence of postoperative neurocognitive disorder. Calibration is shown in Fig. 1. Thresholds and comparison with the other definition can be found in online Supporting Information Appendix S5.

Table 1. Three considered subsets of tests (model A, B, C) and the final testprotocol before and after validation. Predictors are delta test scores of each test (postoperative – pre-operative T-scores).

Model 1: only tests, lowest AIC			Model 2: only tests, until AIC did not drop with one point			Model 3: including cardiac surgery, lowest AIC			Final model: before internal validation			After internal validation
Test	Coefficient	p value	Test	Coefficient	p value	Test	Coefficient	p value	Test	Coefficient	p value	Coefficient	p value
Intercept	0.280	0.566	Intercept	0.526	0.357	Intercept	0.280	0.566	Intercept	0.356	0.3973	0.356	0.3973
TMT A	-0.063	0.105	TMT A	-0.064	0.115	TMT A	-0.063	0.105	Stroop 1	0.044	0.4546	0.031	0.6058
TMT B	-0.097	0.066	TMT B	-0.106	0.056	TMT B	-0.097	0.066	Stroop 2	-0.056	0.3155	-0.038	0.4885
Stroop 3	-0.176	0.011	Stroop 1	0.105	0.253	Stroop 3	-0.176	0.012	Stroop 3	-0.159	0.0074	-0.101	0.0645
Digit span sorting	-0.250	0.000	Stroop 3	-0.192	0.011	Digit span sorting	-0.250	0.000	Digit span forward	-0.030	0.4916	-0.021	0.6351
Category fluency professions	-0.068	0.150	Digit span sorting	-0.271	0.000	Category fluency professions	-0.068	0.150	Digit span backward	0.012	0.7665	0.008	0.8376
Letter fluency COWAT	0.080	0.089	Category fluency professions	-0.075	0.121	Letter fluency COWAT	0.081	0.089	Digit span sorting	-0.195	0.0002	-0.135	0.0112
VAT	-0.165	0.126	Letter fluency COWAT	0.097	0.063	VAT	-0.165	0.126	AVLT immediate	-0.021	0.6769	-0.015	0.7736
AVLT delayed	-0.143	0.005	VAT	-0.194	0.098	AVLT delayed	-0.143	0.005	AVLT delayed	-0.117	0.0060	-0.081	0.0580
			AVLT delayed	-0.164	0.006
Performance
AUROC	0.95 (0.90–0.99)		AUROC	0.949 (0.904–0.993)		AUROC	0.945 (0.898–0.992)		AUROC	0.908 (0.835–0.981)		0.86 (0.777–0.967)
Pseudo R2	0.533		Pseudo R2	0.543		Pseudo R2	0.533		Pseudo R2	0.429		0.515 (0.316–0.808)
Duration	23–32 min		Duration	25–35 min		Duration	23–32 min		Duration	20–24 min

• AIC, Akaike information criterion; AUROC, area under the receiver operating characteristics curve; AVLT, auditory verbal learning test; COWAT, controlled oral word association test; TMT, trail making test; VAT, visual association test.

Figure 1

Open in figure viewerPowerPoint

Calibration plot of the final selected test protocol using bootstrapping (500 repetitions). Final subset, solid line; ideal subset, dashed line.

We believe we have identified a short test protocol with good properties providing a promising alternative for the current neuropsychological assessment, which can be administered before and after surgery.

Interestingly, we found a postoperative neurocognitive disorder incidence of 42%, which is relatively high [6, 7]. This might be attributed to the inclusion of patients undergoing cardiac surgery and the definition used in this study. This again emphasises the need for a uniform definition for postoperative neurocognitive disorder and the corresponding diagnostic tool.

Limitations include the small sample size which might have led to possible overfitting of the data, and part of this study was conducted during the COVID-19 pandemic which may have led to a higher dropout rate as procedures were cancelled in our vulnerable population. However, this does not appear to be an expression of underlying frailty, as dropouts did not have lower neuropsychological assessment scores or higher ASA physical status classifications.

In conclusion, this short test protocol is a promising alternative to the extensive neuropsychological assessment. This is initially of interest for research, and we recommend future studies to externally validate these results. Consensus on the precise definition of postoperative neurocognitive disorder and the recommended test protocol should be generated in a multidisciplinary working group.