Classification of Emergency Department CT Imaging Reports

using Natural Language Processing and Machine Learning

Efsun Sarioglu, Kabir Yadav, Meaghan Smith, Hyeong-Ah Choi

This project supported by the NIH National Center for Research Resources(UL1RR031988 and KL2RR031987)

Background, Objective & Methods Use of electronic medical record data for

clinical research and quality improvement requires free-text data interpretation for outcomes of interest. Natural language processing has shown

promise for this purpose To demonstrate real-world performance of

a hybrid NLP-machine learning system for automated classification of radiology reports

Approach Overview Multicenter review of consecutive CT

reports obtained for facial trauma using a trained reference standard Medical Language Extraction and Encoding (MedLEE) WEKA 3.7.5 Salford Systems CART 6.6

Results

Total reports: 3710 Positive cases: 460 (12.4%) Manual coding had high

reliability Kappa=0.97 [95% CI 0.94-0.99]

CART Decision Trees (50:50)Raw Text (8-node) NLP (9-node)

Classification Performance

Raw Text

NLP

Precision

0.949 0.968

Recall 0.932 0.964

F-score 0.940 0.966

Unexpectedly high performance of machine learning without NLP

Comparable to inter-rater performance and prior studies of inter-physician agreement

Comparable to prior real-world and simulation studies

Concluding Remarks How’s it novel?

One of only a handful of real-world NLP studies using validated reference standard

Translating existing NLP and machine learning technologies to support CER

Next step: validation Test approach using other clinical cases Evaluate different features or

classification algorithms