multimodal user interface with natural language classification for clinicians at point of care...
TRANSCRIPT
![Page 1: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/1.jpg)
Multimodal User Interface with Natural Language Classification for
Clinicians At Point of Care
Health Informatics Showcase
Peter Budd
Sponsors: NCCH - Donna Truran Microsoft - Steven Edwards
![Page 2: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/2.jpg)
A Language Model of Health Information
>80% of information of interest is language. Patients and clinicians use language for
> 90% of information exchange. An EMR should be more like a document
than a database record. Data Capture is a language processing
problem more than a form filling problem
![Page 3: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/3.jpg)
Purpose of Hospital Information Systems
Retrieve patient records for clinicians Provide data to answer research questions Provide data to answer Management
questions Provide clinical alerts for critical incidents Provide decision support for patient care
management plan Provide auditing of patient care
![Page 4: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/4.jpg)
Data Analytics
The primary purpose of HIS is to provide extensive support for patient care,
It is not for the medico legal protection of clinicians interests
Data Analytics should be the fundamental objective of a HIS
The storage repository has more in common with a Content Management System than a relational database IS.
Language should be reduced to a canonical form – SNOMED CT
![Page 5: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/5.jpg)
Data Entry - Objectives
Mimic the workplace processing as closely as possible
Identify text as the primary content Make canonical encoding as automatic as
possible Make canonical encoding as hidden from
view as necessary Maximise flexibility in data entry modes
![Page 6: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/6.jpg)
Technology Strategy
Multimodal Interface– Developed on the Tablet PC
Handwriting & Drawing Capabilities Sub-vocal microphones for speech input
– Designed to closely mimic “real” paper forms Generic Form Generation Able to be localised for individual hospitals
– Automatically classify Natural Language Classify free text into SNOMED-CT ontology
![Page 7: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/7.jpg)
Top Level Overview
InterfaceInterface
Form Generator
Form Generator
Augmented Lexicon &
Standard Lexicon
Augmented Lexicon &
Standard Lexicon
Token MatcherToken
MatcherClinicianClinician
DatabaseDatabase
![Page 8: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/8.jpg)
Token Matching
Phase 1– Currently implemented– Matching based off sequence runs of medical
terms– Adjacent words compared against each other– Match with most words used chosen as optimal
match– SNOMED-CT Description table used; Multiple
descriptions map to the same concept
![Page 9: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/9.jpg)
Token Matching
Phase 2– To be implemented as future work– If bad matches are found, words close in spelling
may be used to accommodate mistakes in the handwriting or speech recognition
– Matching algorithm allows inconsistencies/ missing elements in the input
– Uses language knowledge to fill in the gaps
![Page 10: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/10.jpg)
Token Matching
Phase 3– Also not yet implemented– Uses sophisticated Natural Language Processing
techniques to break sentences into “clumps”– Token Matching is then run on the clumps– Allows the negation of SNOMED terms based off
sentence clumps
![Page 11: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/11.jpg)
Form Generation
Necessary attributes of the form are extracted out into an XML format
Form generated “on-the-fly” at program runtime
Allows hospitals to have non-technical staff use interface generator software to localize standard forms or create their own
Output into standard XML for saving into Database
![Page 12: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/12.jpg)
Form Generation
Next Phase of Implementation– Form can be loaded pre-filled or seeded with data
based off statistically average usage– Allow multiple clinicians (Doctors and Nurses)
access to the same form at the same time (from multiple Tablet PCs) to speed up data entry and reduce duplication
– Add speech recognition and video capture to the interface
![Page 13: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/13.jpg)
Conclusion
Project outcomes– User Interface was created which closely mimics actual
forms currently used in the workplace– Automatically classifies natural language into a medical
ontology
Performance issues– Classification runs in acceptable time as a background
process– Form Generation runs in pseudo-real time– Time for form generation well inside time required to pick up
a real paper form
![Page 14: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/14.jpg)
Current Progress
Building an ED information system based on this model
Using Process diagram collated from 3 month study at Westmead ED
Subject of ARC Linkage grant application with Sydney West Area Health Service
![Page 15: Multimodal User Interface with Natural Language Classification for Clinicians At Point of Care Health Informatics Showcase Peter Budd Sponsors: NCCH -](https://reader036.vdocument.in/reader036/viewer/2022083004/56649f005503460f94c15f87/html5/thumbnails/15.jpg)
Questions