OR in the News

Mathur, Piyush, Cywinski, Jacek B., Maheshwari, Kamal, Niezgoda, Julie, Mathew, Jibin, do Nascimento, Claudio Cesar, Abdelmalak, Basem B., Papay, Francis A. Automated analysis of ambulatory surgery patient experience comments using artificial intelligence for quality improvement: A patient centered approach. Intelligence-Based Medicine 2021: 5

October 16, 2021

Background: Quality improvement in healthcare is limited by both the quality and quantity of data available in the electronic health records or self reported by clinicians. Appropriate and timely reporting can help identify opportunities for quality improvement and deployment of mitigation steps but many of these process remain manual. Negative patient experience comments can potentially be used in identifying opportunities for quality improvement. Artificial intelligence (AI) techniques, like natural language processing (NLP) including sentiment analysis and topic modeling can be used to characterize patient experience comments in near real time, thus making the process timely, efficient and measurable.
Methods: We analyzed 15,453 de-identified patient experience comments from the Press Ganey™ survey of adult patients undergoing outpatient surgeries with anesthesia at Cleveland Clinic from 01/01/2012-05/03/2016. We used open source NLP including sentiment analysis and topical modeling to analyze post-discharge patient experience survey comments and feedback verbatim. For sentiment analysis we used an open source sentiment analyzer called VADER. For topic modeling we used Latent Dirichlet Allocation (LDA) algorithm.
Results: Sentiment analysis of patient comments using VADER was highly accurate with F1 score ranging from 0.83 to 0.84 for positive comments and 0.71–0.73 for negative comments compared to clinician’s assessment. Two clinicians reviewed 1955 random comments as positive or negative with good agreement, with Cohen’s k test score of k = 0.92 (p < 0.001). Six thematic areas were identified based on LDA algorithms topic analysis with interpretable and measurable weighable of words extracted from the comments associated with poor patient satisfaction, thus defining targeted opportunities for improvement. Conclusion: We conclude that artificial intelligence can help in near real-time analysis of patient experience surveys. Commonly used open source algorithms can possibly be utilized in healthcare for quality improvement negating the need for significant development resources with the opportunity to generalize and scale.