Development and Validation of the ACP-CAT for Assessing the Quality of Advance Care Planning Communication

Context: High-quality advance care planning (ACP) discussions are important to ensure patient receipt of goal-concordant care; however, there is no existing tool for assessing ACP communication quality. Objectives: The objective of this study was to develop and validate a novel instrument that can be used to assess ACP communication skills of clinicians and trainees. Methods: We developed a 20-item ACP Communication Assessment Tool (ACP-CAT) plus two summative items. Randomized rater pairs assessed residents' performances in video-recorded standardized patient encounters before and after an ACP training program using the ACP-CAT. We tested the tool for its 1) discriminating ability, 2) interrater reliability, 3) concurrent validity, 4) feasibility, and 5) raters' satisfaction. Results: Fifty-eight pre/post-training video recordings from 29 first-year internal medicine residents at Mount Sinai Hospital were evaluated. ACP-CAT reliably discriminated performance before and after training (median score 6 vs. 11, P < 0.001). For both pre/post-training encounters, interrater reliability was high for ACP-CAT total scores (intraclass correlation coefficient or ICC = 0.83 and 0.82) and the summative items Overall impression of ACP communication skills (ICC = 0.73 and 0.80) and Overall ability to respond to emotion (ICC = 0.83 and 0.82). Concurrent validity was shown by the strong correlation between ACP-CAT total score and both summative items. Raters spent an average of 4.8 minutes to complete the ACP-CAT, found it feasible, and were satisfied with its use. Conclusion: ACP-CAT provides a validated measure of ACP communication quality for assessing video-recorded encounters and can be further studied for its applicability with clinicians in different clinical contexts.