auditing emr system usage - vanderbilt...
TRANSCRIPT
![Page 2: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/2.jpg)
Anomalous Usage 2 © You Chen, 2011
Health data being accessed by hackers, lost with laptop
computers, or simply read by curious employees
![Page 3: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/3.jpg)
Jail Time for Malicious Accesses
$675,000 for Privacy Violation
Current HIPAA Security Rules
are not enough
![Page 4: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/4.jpg)
HIPAA Security Rules • Administrative Safeguards
– Administrative actions, and policies and procedures, to manage the selection, development, implementation, and maintenance of security measures to protect electronic protected health information (PHI)
• Physical Safeguards ― Physical measures, policies and procedures to protect
a covered entity's electronic information systems and related buildings and equipment, from natural and environmental hazards, and unauthorized intrusion
• Technical Safeguards ― The technology and the policy and procedures for its use
that protect electronic protected health information [PHI] and control access to it
![Page 5: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/5.jpg)
Current State
Monitor VIPs (the Clooney effect- finding more attractive man)
Monitor employee-employee access
Follow-up on external suspicion
Spot checks
![Page 6: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/6.jpg)
Technical Safeguards
• Access Control • Audit controls: Implement systems to record
and audit access to protected health information within information systems – Track & audit employees access to patient records
– Store logs for ≥ 6 years
![Page 7: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/7.jpg)
Access Control? • “We have *-Based Access Control.”
• “We have a mathematically rigorous access policy logic!”
• “We can specify {context, team, temporal} policies!” (Georgiadis et al, 2001; Park et al. 2001;)
• “We can control your access at a fine-grained level!”
• “Isn’t that enough?”
C. Georgiadis, I. Mavridis, G. Pangalos, and R. Thomas. Flexible team-based access control using contexts. Proceedings of ACM Symposium on Access Control Model and Technology. 2001: 21-27.
J. Park, R. Sandhu, and G. Ahn. Role-based access control on the web. ACM Transactions on Information and System Security. 2001; 4(1): 37-71.
![Page 8: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/8.jpg)
Why is the Problem So Hard?
• Hospital system is inherently dynamic
- Multiple responsibilities - Fuzzy “ - Changing “
Roles
- Situation dependent -Constrained by availability “Can you work today?”
Teams
- Who defines the policies? - Often Vague - Cluttered by “multi-use” systems
Reason A G I L I T Y
Formal Security
A lack of availability can
harm someone!
![Page 9: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/9.jpg)
But If You Let Them, They Will Come • In March 2006, researchers carried out an
investigation on hospitals in the Central Norway Health Region
• Users were assigned to an initial set of privileges and could invoke actualization, temporarily escalating their rights as necessary
• Such an access control system is feasible when the number of actualizations is small
Role Users Invoked Actualization in Past Month
Nurse 5633 36%
Doctor 2927 52%
Health Secretary 1876 52%
Physiotherapist 382 56%
Psychologist 194 58%
L. Røstad and N. Øystein. Access control and integration of health care systems: an experience report and future challenges. Proceedings of the 2nd International Conference on Availability, Reliability and Security (ARES). 2007: 871-878.
This case: •53,650 of 99,352 patients actualized •5,310 of 12,258 users invoked actualization •Over 295,000 actualizations in one month
![Page 10: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/10.jpg)
Aim of Access Logs Auditing
Expected Model (EM)
Access Log (AL)
High Efficiency of Resource Allocation
Improve Quality of Patient Treatment
Patient Information Protection
![Page 11: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/11.jpg)
Examples of Accesses Encounter number Department
Patient position Reason Relationship
User Date
Location
![Page 12: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/12.jpg)
Patient
Encounter number Diagnose codes
Examples of Patient Diagnose Codes
![Page 13: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/13.jpg)
Various Ways of Access Logs Auditing
Attributes of Users and Patients
Accesses of EHR
Social Network
Workflows
Treatment/ Access Patterns
Usable Rules
Models Analysis
Time Series
![Page 14: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/14.jpg)
Logs
Anomalous Users Detection-CADS
(Community based Anomaly Detection
System) and MetaCADS
Anomalous Accesses Detection-SNAD
(Specialized Network Anomaly Detection)
Predicting Diseases
Relation Rules of Departments
User view
Patient view
Interaction Network of
Users
Interaction Network of
Departments
Social Analysis
Department view
Accesses of Users on Patients
User level
Access level
Recommending Community of Departments
Community of Departments
Alerting of Anomalous Treatment or Accesses
![Page 15: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/15.jpg)
Uncovering Anomalous Usage of Medical Records via Social Network Analysis
![Page 16: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/16.jpg)
Two Typical Attacks
Anomalous users
Anomalous Access
(1) Anomalous users detection –user level (2) Anomalous accesses detection –access level
Intruders have little knowledge of the system and the anticipated behavior
Intruders have complete knowledge of the system and its policies
![Page 17: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/17.jpg)
S3
S5
S1
Behavioral Modeling
S(ubjects) U(sers)
Accesses
Two general objects of health information system
![Page 18: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/18.jpg)
Where are We Going? User Level Anomaly Detection
Community Anomaly Detection System (CADS) and Its Extension MetaCADS
(IEEE TDSC)
Access Level Anomaly Detection
Specialized Network Anomaly Detection (SNAD)
(Security Informatics) You Chen, Steve Nyemba and Bradley Malin. Detecting Anomalous Insiders in Collaborative Information Systems. IEEE Transaction on Dependable and Secure Computing. Vol.9.No 3, p332-344.
You Chen, Steve Nyemba, Wen Zhang and Bradley Malin. Specializing Network Analysis to Detect Anomalous Insider Actions. Security Informatics. 1:5, 2012, p1-24.
You Chen and Bradley Malin. Detection of Anomalous Insiders in Collaborative Environments via Relational Analysis of Access Logs. Proceedings of ACM Conference on Data and Application Security and Privacy. 2011, p63-74
![Page 19: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/19.jpg)
Social Networks are a Novel Approach to Discovery of Electronic Medical Record Misuse
![Page 20: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/20.jpg)
Deviation
# of
Acc
esse
s
CADS on Vanderbilt
Dataset
![Page 21: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/21.jpg)
CADS on Northwestern
Dataset
Deviation
# of
Acc
esse
s
![Page 22: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/22.jpg)
Example Environments Electronic Health
Records (EHR)
• Vanderbilt University Medical Center “StarPanel” Logs
• 3 months in 2010 • Arbitrary Day
≈ 4,208 users ≈ 1,006 patients ≈ 1,482 diagnoses ≈ 22,014 accesses of subjects ≈ 4,609 assignments of diagnoses
![Page 23: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/23.jpg)
Where are We Going?
• User Level: CADS and MetaCADS – Framework of CADS and MetaCADS – An Example of CADS – Experimental Evaluation – Limitation
• Access Level: Specialized Network Anomaly Detection (SNAD)
(SI)
![Page 24: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/24.jpg)
CADS and MetaCADS
![Page 25: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/25.jpg)
Where are We Going?
• User Level: CADS and MetaCADS – Framework of CADS and MetaCADS – An Example of CADS – Experimental Evaluation – Limitation
• Access Level: Specialized Network Anomaly Detection (SNAD)
(SI)
![Page 26: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/26.jpg)
Distance via Weighted Euclidean Distance
Nearest Neighbor Network
Bipartite Graph->Access Network of Users
Communities via Singular Value Decomposition
Deviation Scores Calculation
![Page 27: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/27.jpg)
How Do We Set “k”-NN?
• Conductance- a measure of community quality (Kannan et al)
![Page 28: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/28.jpg)
Min
imum
cond
ucta
nce
for v
alue
k
k
Minimum conductance at k=6
![Page 29: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/29.jpg)
The average cluster coefficient for this network is 0.48, which is significantly larger than 0.001 for random networks
Users exhibit collaborative behavior in the health information system
Example 6-Nearest Neighbor Network (1 day of accesses)
![Page 30: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/30.jpg)
• Every user is assigned a radius r: – the distance to his kth nearest neighbor
• Smaller the radius higher density in user’s network
Measuring Deviation from k-NN
k
rruDev
knniju ji
∑ −=
∈2)(
)(
k
rr
knnijuj∑
=∈
Radius for these points are larger than 10, and every r values significantly different
Radius for these points are nearly 2, and for q1 is 3
If we set threshold of radius as 10, than q1 is a normal user, who in fact is anomalous
42.05
)2.23(4)2.22()(22
1 =−+×−
=qDev
5 nearest
2.25
32222=
++++=r
![Page 31: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/31.jpg)
Where are We Going?
• User Level: CADS and MetaCADS – Framework of CADS and MetaCADS – An Example of CADS – Experimental Evaluation – Limitation
• Access Level: Specialized Network Anomaly Detection (SNAD)
(SI)
![Page 32: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/32.jpg)
Experimental Design
• Datasets are not annotated for illicit behavior • We simulated users in several settings to test:
– Sensitivity to number of records accessed of a specific users
• Range from 1 to 120
– Sensitivity to number of anomalous users • simulated users correspond to 0.5% to 5% of total users • Number of records accessed fixed to 5
– Sensitivity to diversity • Random number of users(0.5%~5%) and records accessed (1~150)
![Page 33: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/33.jpg)
Exp1: False Positive Rate Decreases, when the Number of Subjects Accessed Increases
False Positive Rate
Number of patients accessed per user
MetaCADS achieves a smaller false positive rate than CADS. This is because the assignment network facilitates a stronger portrayal of real users’ communities than the access network in isolation
![Page 34: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/34.jpg)
Exp2: Detection Rate With Various Mix Rates of Real and Simulated Users
when the number of simulated users is low (i.e., 0.5 percent), MetaCADS yields a slightly higher AUC than CADS (0.92 versus 0.91)
As the number of simulated users increases, CADS clearly dominates MetaCADS. The performance rate of CADS increases from 0.91 to 0.94, while MetaCADS decreases from 0.92 to 0.87.
Because when the number of simulated users increases, they have more frequent categories in common. In turn, these categories enable simulated users to form more communities than those based on patients alone, thus lowering their deviation scores.
![Page 35: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/35.jpg)
Exp3: MetaCADS dominates when the mix rate is low (mix rate = 0.5%)
True
Pos
itive
Rat
e
False Positive Rate
![Page 36: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/36.jpg)
MetaCADS deviation scores of real and simulated users as a function of the number of subjects accessed. This system was generated with a mix rate of 0.5 percent and a random number of subjects accessed per simulated user
# of
pat
ient
s acc
esse
d
Deviation
# of
use
rs
![Page 37: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/37.jpg)
Where are We Going?
• User Level: CADS and MetaCADS – Framework of CADS and MetaCADS – An Example of CADS – Experimental Evaluation – Limitation
• Access Level: Specialized Network Anomaly Detection (SNAD)
(SI)
![Page 38: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/38.jpg)
• Simulated users are indicative of misuse of the system… …but actual illicit behavior may be more directed.
• “False positives” are not necessarily false! (Adjudication by EHR privacy experts under way)
• Need to specialize tool to account for semantics of users and subjects – User: {Role, Department, Residence} – Patient: {Diagnosis, Procedure, Demographics, Residence}
• Anomalous users… not anomalous accesses – Need to account for insiders that deviate by only a couple of actions
Some Limitations
![Page 39: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/39.jpg)
Where are We Going?
• User Level: CADS and MetaCADS • Access Level: Specialized Network Anomaly
Detection (SNAD) (SI)
– Framework of SNAD – An Example of CADS – Experimental Evaluation – Limitation
![Page 40: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/40.jpg)
SNAD Framework
![Page 41: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/41.jpg)
Where are We Going?
• User Level: CADS and MetaCADS
• Access Level: Specialized Network Anomaly Detection (SNAD)
(SI) – Framework of SNAD – An Example of SNAD – Experimental Evaluation – Limitation
![Page 42: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/42.jpg)
User Modeling
![Page 43: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/43.jpg)
Access Network Construction
![Page 44: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/44.jpg)
Access Network Measurement
![Page 45: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/45.jpg)
Measuring Accesses for Changes in Network Similarity
Access: u1->s3
![Page 46: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/46.jpg)
Where are We Going?
• CADS and MetaCADS
• Access Level: Specialized Network Anomaly Detection (SNAD)
(SI) – Framework of SNAD – An Example of SNAD – Experimental Evaluation – Limitation
![Page 47: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/47.jpg)
Experimental Design
• Datasets are not annotated for illicit behavior • We simulated users in several settings to test:
– Sensitivity to number of subjects accessed • Range from 1 to 1,00
– Sensitivity to number of anomalous users • Range from 2 to 20 • Number of subjects accessed fixed to 5
– Sensitivity to diversity • Random number of users and subjects accessed
![Page 48: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/48.jpg)
SNAD: Detection Rate Increase with Number of Subjects Accessed
Number of Subjects the Intruder Accesses
AUC
![Page 49: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/49.jpg)
SNAD: Detection Rate Increases with Number of Intruders
Number of Intruders
AUC
![Page 50: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/50.jpg)
SNAD Outperforms Competitors When the Number of Intruders & Accessed Subjects is Random
False positive rate
True
pos
itive
rate
![Page 51: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/51.jpg)
Where are We Going?
• CADS and MetaCADS
• Access Level: Specialized Network Anomaly Detection (SNAD)
(SI) – Framework of SNAD – An Example of SNAD – Experimental Evaluation – Limitation
Anomalous Usage © You Chen, 2011 51
![Page 52: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/52.jpg)
Limitations • SNAD has high performance in Vanderbilt’s EHR system because
– organization is collaborative – access networks have high network similarity
• SNAD may not be appropriate for large access network with low
network similarity – Absence of a user has little influence on the similarity.
52 Size of network Sim
ilarity of network
![Page 53: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/53.jpg)
Conclusions
• It is an effective way by using social network analysis to detect anomalous usages of electronic health records, such as CADS and SNAD
• Adding semantic information of users and subjects will make social network analysis be more understandable
![Page 54: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/54.jpg)
Protecting Patients through Dynamic Network Analysis of Hospital Department
Relationships
You Chen, Steve Nyemba and Bradley Malin. Auditing Medical Records Accesses via Healthcare Interaction Networks. AMIA 2012 Annual Symposium. p93-102
![Page 55: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/55.jpg)
Patient information needs to be protected from insiders
• Traditional security practices (e.g., role-based access control) are insufficient to ensure EMR security
– Common for >100 employees to access a patient’s medical record during their visit
– Often difficult to determine who the members of a care team are and who will need access to what information at which time
![Page 56: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/56.jpg)
EHRs have adopted collaborative capabilities to facilitate interaction between
teammates and coordinate care
• We hypothesize that HCO departments will exhibit predictable interaction behavior
• Our goals: 1. Investigate if such behavior exists 2. If so, determine if it is stable
• If stable interactions become unstable associated patients will be anomalous
![Page 57: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/57.jpg)
Expected behavior: Pediatric housestaff; pediatric cardiology; Vanderbilt
Children’s hospital
Utilized behavior: Mental health center; burn center; breast cancer center;
pediatric housestaff; pediatric cardiology; Vanderbilt Children’s hospital
The dependent relations between green departments and red departments are very low
Our goal is to retrieve the dependent relations of departments and determine whether the dependencies among departments touching that patient are expected?
![Page 58: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/58.jpg)
Healthcare Interaction Networks
Local view for p6
Global view
Tripartite graph of departments, users
and patients
Bipartite graph of departments and
patients
Health interaction network
![Page 59: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/59.jpg)
Where are We Going?
A Global Network of Departments
Two metrics: certainty and reciprocity Stable status in terms of the two metrics
Local Network-for a specific patient Two metrics: local network score and reciprocity
Application of the Networks Detecting patients with anomalous medical records accesses
![Page 60: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/60.jpg)
Certainty to Model Relationship of Global Network
Cert(Lifeflight event medicine (d3)->Emergency medicine (d1)) = 4/4
Cert(Inpatient medicine(d2)->Inpatient medicine(d2)) = 6/7
p2 p3
p7 p5
p4 p6
p1
p2 p3
p7 p5
Health interaction network
Bipartite graph of departments and patients
p4 p6
p1
p2 p3
p7 p5
Lifeflight event medicine
Emergency medicine
Inpatient medicine
![Page 61: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/61.jpg)
Using reciprocity to characterize the mutual interaction between all pairs of departments in the global network
Reciprocity=1
Pediatric Emergency Dept -> Peds Respiratory Care =0.57
Peds Respiratory Care -> Pediatric Emergency Dept = 0.037
![Page 62: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/62.jpg)
Where are We Going?
A Global Network of Departments Two metrics: certainty and reciprocity
Stable status in terms of the two metrics
Local Network-for a specific patient Two metrics: local network score and reciprocity
Application of the Networks Detecting Patients with Anomalous Medical Records Accesses
![Page 63: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/63.jpg)
• Vanderbilt University Medical Center “StarPanel” • 3 months in 2010 • Arbitrary Week
≈ 9,200 users ≈ 99,000 patient records ≈ 400,000 accesses ≈ 450 departments
Dataset used for this study
![Page 64: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/64.jpg)
Although the relations of the network are very unbalanced, the unbalance is stable over time
Time Week 1 Week 2 Week 3 Week 4
Reciprocity 0.267 0.2814 0.2858 0.2871
05.0267.0/)267.02814.0( =−
Week 1 to week 2
![Page 65: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/65.jpg)
The changes become smaller over time (centralization: green > blue > red)
Degree of relations between departments changes little over time >82.5% of the change resides in [-0.25, 0.25]
Certainty Change
Prop
ortio
n of
Rel
atio
ns
![Page 66: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/66.jpg)
Strong relations between VUMC departments over a four week period
![Page 67: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/67.jpg)
Where are We Going?
A Global Network of Departments Two metrics: certainty and reciprocity
Stable status in terms of the two metrics
Local Network-for a specific patient
Two metrics: local network score and reciprocity
Application of the Networks Detecting patients with anomalous medical records accesses
![Page 68: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/68.jpg)
Healthcare Interaction Networks
Local view for p6
Tripartite graph of departments, users
and patients
Bipartite graph of departments and
patients
Health interaction network
![Page 69: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/69.jpg)
Evolution of Local Networks in Terms of Local Network Score and Local Network Reciprocity
Each point in Pstart corresponds to a local network
![Page 70: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/70.jpg)
© You Chen, 2012
Over 98% of patients are normal because they exhibit a score change <0.05
Local Network Score Change
Prop
ortio
n of
Pat
ient
s
![Page 71: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/71.jpg)
Approximately 99% of patients are normal because they have a change of reciprocity <0.1
Reciprocity Change Prop
ortio
n of
pat
ient
s
![Page 72: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/72.jpg)
Where are We Going?
A Global Network of Departments Two metrics: certainty and reciprocity
Stable status in terms of the two metrics
Local Network-for a specific patient Two metrics: local network score and reciprocity
Application of the Networks
Detecting patients with anomalous medical records accesses
![Page 73: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/73.jpg)
© You Chen, 2012 73
p2 has -0.93 change of local network score and -0.79 change of local reciprocity from the 1st to the 2nd week
week
Loca
l net
wor
k sc
ore
# of
dep
artm
ents
Cancer Infusion Center
Breast Center, [Anonymized Street Location], Care/Eskind Diab Acces, Disease Management Service, Eskind Diabetes - Adult, Free Stipends, Internal Medicine, VIM, VMG Physician Billing Services, Vanderbilt Home Care Primary
![Page 74: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/74.jpg)
Conclusions
• We hypothesized an HCO would exhibit strong stability confirmed by our experiments
• We can characterize how strange a patient’s local network appears – Two groups of patients; those with small changes in local network score
and reciprocity score and those with significant changes
– The changes in the latter group do not justify the claim that the patient has been intruded upon, but may provide a reason for an investigation that incorporates more nuanced domain knowledge
![Page 75: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/75.jpg)
• Global and local networks appear to represent the business processes of HCO departments – however, such claims must be confirmed with employees knowledge
about the working of the medical center and its affiliated clinics
• Need to specialize tool to account for semantics of patients – Patient: {Diagnosis, Procedure, Demographics, Residence, physical
location in a hospital}
– Incorporating semantics about the patient, p2 in the last figure may have no intrusion; rather it is likely a complex cancer patient, which could be confirmed by inspection of clinical documents in the medical record
Some Limitations
![Page 76: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/76.jpg)
![Page 77: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/77.jpg)
SNAD assumes that access scores are approximately distributed around a well-centered mean.
Access score
Num
ber o
f acc
esse
s
![Page 78: Auditing EMR System Usage - Vanderbilt Universityhiplab.mc.vanderbilt.edu/~ychen/lecture_auditing.pdf · Auditing EMR System Usage You Chen Jan, 17, 2013 . ... You Chen, Steve Nyemba](https://reader031.vdocument.in/reader031/viewer/2022030613/5ade2e607f8b9a1a088e4e9f/html5/thumbnails/78.jpg)
Access score
Perc
enta
ge o
f Acc
esse
s
The correlation coefficient between real and Laplace distribution is 0.886