Distributed Knowledge-Based Abstraction, Visualization, and

Exploration of Time-Oriented Clinical Data

Yuval Shahar, M.D., Ph.D.

Medical Informatics Research CenterBen Gurion University, Beer Sheva, Israel

&Stanford University, CA, USA

The Need for Intelligent Integration of Multiple Time-Oriented Clinical Data

• Many medical tasks, especially those involving chronic patients, require extraction of clinically meaningful concepts from multiple sources of raw, longitudinal, time-oriented data– Example: “Modify the standard dose of the drug, if during treatment,

the patient experiences a second episode of moderate anemia that has persisted for at least two weeks”

• Examples of clinical tasks:– Diagnosis

• Searching for “a gradual increase of fasting blood-glucose level”– Therapy

• Following a treatment plan based on a clinical guideline– Quality assessment

• Comparing observed treatments with those recommended by a guideline– Research

• Detection of hidden dependencies over time between clinical parameters

The Need for Intelligent Mediation:The Gap Between Raw Clinical Data and Clinically Meaningful Concepts

• Clinical databases store raw, time-stamped data• Care providers and decision-support applications

reason about patients in terms of abstract, clinically meaningful concepts, typically over significant time periods

• A system that automatically answers queries or detects patterns regarding either raw clinical data or concepts derivable from them over time, is crucial for effectively supporting multiple clinical tasks

The Temporal-Abstraction Task

• Input: time-stamped clinical data and relevant events (interventions)

• Output: interval-based abstractions

• Identifies past and present trends and states

• Supports decisions based on temporal patterns, such as: “modify therapy if the patient has a second episode of Grade II bone-marrow toxicity lasting more than 3 weeks”

• Focuses on interpretation, rather than on forecasting

Temporal Abstraction:The Bone-Marrow Transplantation

Domain

.

•

0 40020010050

•

1000

2000( )

100K

150K

( )

•••

• • • ••

• •

•••

Granu-locytecounts

• • •

•

Time (days)

Plateletcounts

PAZ protocol

M[0] M[1] M[2] M[3] M[1] M[0]

BMT

Expected CGVHD

M[i]= Myelotoxicity (bone-marrow toxicity) of severity Grade i

The Bone-Marrow Transplantation Example, Revisited

Uses of Temporal Abstractions• Therapy planning and monitoring (e.g., to support guideline-based

care)

• Creating high-level summaries of time-oriented medical records

• Supporting explanation modules for a medical DSS

• Representing goals of therapy guidelines for quality assurance at runtime and quality assessment retrospectively; guideline intentions

regarding both the (care provider) process and (patient) outcomes can be captured as temporal patterns to be achieved or avoided

• Visualization and exploration of time-oriented clinical data: the KNAVE project

The Temporal-Abstraction Ontology• Events (interventions) (e.g., insulin therapy) - part-of, is-a relations

• Parameters (measured raw data and derived concepts) (e.g., hemoglobin values; anemia levels) - abstracted-into, is-a relations

• Patterns (e.g., crescendo angina; chronic GVHD) - component-of, is-a relations

• Abstraction goals (user views)(e.g., therapy of diabetes) - is-a relations

• Interpretation contexts (effect of regular insulin) - subcontext, is-a relations

• Interpretation contexts are induced by all other entities

Temporal-Abstraction Output Types

• State abstractions (LOW, HIGH)• Gradient abstractions (INC, DEC)• Rate Abstractions (SLOW, FAST) • Pattern Abstractions (CRESCENDO)

- Linear patterns- Periodic patterns

Temporal-Abstraction Knowledge Types

• Structural (e.g., part-of, is-a relations) - mainly declarative/relational

• Classification (e.g., value ranges; patterns) - mainly functional

• Temporal-semantic (e.g., “concatenable” property) - mainly logical • Temporal-dynamic (e.g., interpolation functions) - mainly probabilistic

Dynamic Induction of Contexts

+2W+4W

AZT-toxicity interpretation context

AZT administration

ss

ee

sees

Induction of Interpretation Contexts

+2W+4W

CCTG-522_AZT-toxicity interpretation context

AZT-administration event

(a)

+4W+-6W

-2W

Hepatitis B

HB prodrome Chronic active hepatitis

(b)

CCTG-522 protocol

The Benefits of Interpretation Contexts

• Context intervals serve as a frame of reference for interpretation of clinical data; abstractions are meaningful only in a particular clinical context

• Context intervals focus and limit the computations to only those relevant to a particular context

• Contexts enable the use of context-specific medical knowledge• contexts support maintenance of several concurrent views (or

interpretations) of the data• Facilitate maintenance of clinical knowledge

– The same context-forming entity (e.g., a hepatitis episode) can induce several clinical contexts

– The same context (e.g., “a chemotherapy effect”) might be induced by several entities (e.g., multiple medication types)

Local and Global Persistence Functions

1

0

I1 I2

t

1 2

th

Time

Bel()

Temperature

Hemoglobin Level

Linear Component

Week 2 Week 3Week 1

Anemia

Fever Fever

Anemia Anemia

FeverFever

Anemia

Fever

Linear ComponentLinear Component Linear Component

Periodic Pattern

Abstraction of Periodic Patterns

The RÉSUMÉ System Architecture

Temporal-abstraction mechanisms

Temporal fact baseE v e n t s

C o n t e x t s

A b s t r a c t e d i n t e r v a l s

P r i m i t i v e d a t a •

Domain knowledge base

Event ontology

Parameter ontology

Primitive data

Events

••

Context ontology

External patient database

+ +

+

•+

Test Domains for the RÉSUMÉ System

• Medical domains:– Guideline-based care

• AIDS therapy

• Oncology

– Monitoring of children’s growth

– Therapy of insulin-dependent diabetes patients

• Non-medical domains:– Evaluation of traffic-controllers actions

– summarization of meteorological data

Acquisition of Temporal-Abstraction Knowledge

Editing The Ontology Using Protégé Tools

Knowledge-Based Visualization andExploration of Time-Oriented Data:

The KNAVE-I and KNAVE-II Projects

(Shahar and Cheng, 1999, 2000; Shahar et al., 2003)• KNAVE = Knowledge-Based Navigation of Abstractions for Visualization and Explanation

• Interactive queries regarding both raw data and multiple levels of time-oriented abstractions derivable from these data

• Visualization and manipulation of query results• Dynamic exploration of the results using the domain’s temporal-

abstraction ontology• The semantics of all operators do not depend on any specific

domain, but the interface uses each domain’s ontology to compute and display specific terms and explore their relations

• KNAVE accesses the data through the IDAN temporal-abstraction mediator, which uses the ALMA system, a constraint-based re-implementation of RESUME, for temporal-reasoning

The IDAN Temporal-Abstraction Mediator(Boaz and Shahar, 2003)

Temporal-Abstraction

Controller

Knowledge- acquisition tool

Standard Medical Vocabularies Service KNAVE-II

Knowledge Service

Temporal-Abstraction

Service (ALMA)

Data Access Service

Medical Expert

Clinical User

The KNAVE-II Browsing and Exploration Interface[Shahar et al., AIM 2006]

Overall pattern

Raw clinical data

Intermediate abstractionsMedical

knowledge browser

Concept search

Moving Data Panels Around

Global Temporal-Granule Zoom (I)

Global Temporal-Granule Zoom (II)

Global Calendar-Based Zoom

Global Content-Based Zoom (I)

Global Content-Based Zoom (II)

Local Time-Sensitive Zoom

Exploration Operators• Motion across semantic links in the domain’s knowledge base by

using the semantic explorer; in particular, relations such as:

- part-of - is-a - abstracted-from - subcontext • Motion across abstraction types: state, gradient, rate, pattern

• Application of aggregation operators such as mean and distribution

• Dynamic change of temporal-granularity (e.g., days, months)

• Explanation by context-sensitive display of relevant knowledge

• “What- if” queries allow hypothetical assertion or retraction of data and examination of resultant patterns

Semantic Exploration of Temporal Abstractions

Explanation: A Classification Function

Explanation: A Persistence Function

Functionality and Usability Evaluation of KNAVE-II(Palo Alto Veterans Administration Health Care System)

Eight clinicians with varying medical/computer use backgrounds A second study used six additional clinicians and more difficult queries

Each user was given a 15 minute demonstration of the interface and two warm-up queries to answer

The evaluation used an online database of more than 1000 bone-marrow transplantation patients followed for 2 to 4 years

Each user was asked to answer 10 queries common in oncology protocols, about individual patients, at increasing difficulty levels

A cross-over study design compared the KNAVE-II module versus two existing methods (in the 2nd study, control-group users chose which one): Paper charts An electronic spreadsheet (ESS)

Measures: Quantitative: time to answer and accuracy of responses Qualitative: the Standard Usability Score (SUS) and comparative ranking

The KNAVE-II Evaluation Results(Martins et al., MEDINFO 2004; AIM 2008)

• Direct Ranking comparison: KNAVE-II ranked first in preference by all users • Detailed Usability Scores: The Standard Usability Scale (SUS) mean scores:

KNAVE-II 69, ESS 48, Paper 46 (P=0.006) (more than 50 is user friendly)• Time to answer:

– In the first study, users were significantly faster using KNAVE-II as the level of difficulty increased, up to a mean of 93 seconds difference versus paper, and 27 seconds versus Excel, for the hardest query (p = 0.0006)

– The second evaluation, using more difficult queries and more advanced features of KNAVE-II, emphasized the differences even further: The comparison with Excel showed a similar trend for moderately difficult queries (P=0.007) and for hard queries (p=0.002); on the average, study participants answered each of the two hardest queries 277 seconds faster using KNAVE than when using the ESS

• Correctness: – In the first study, using KNAVE-II significantly enhanced correctness versus

using paper, especially as level of difficulty increased (P=0.01) – In the second study, 91.6% (110/120) of all of the questions asked within

queries of all levels produced correct answers using KNAVE-II, versus only 57.5% (69/120) using the ESS (p<0.0001); the correctness scores for KNAVE-II versus the ESS were significantly higher for all queries

The VISITORS System:Interactive Exploration of Multiple Patients

• Supports, among other types, aggregation queries:• Who [Patients] Query

– Find all male patients who were older than 50 and were treated by Dr. Johnson; and, starting within the two weeks following a BMT procedure, had a one week episode of WBC_State whose value was lower than “Normal”

– Show all patients whose HGB_State can be interpreted as “moderate anemia” during more than 5% of February 2006

• When [Time intervals] Query– Select time intervals during which more than 10% of the specific

patients have HGB state value greater than “normal”

• Demographic Constraints:

– Young (age≤20) or Old (age≥70) Male patients

The VISITORS Query-Builder InterfaceThe VISITORS Query-Builder Interface

• Knowledge-based constraints– Hemoglobin [HGB] State was abstracted as Normal or higher for at least seven

days after the two weeks period starting from the allogenic bone marrow transplantation,

– WBC counts were abstracted as Gradient = Increasing during the same period

The VISITORS Main Display InterfaceThe VISITORS Main Display Interface

Multiple-subjects raw dataMedical knowledge

browser

Subject groups

Distribution of derived patterns over time

Concept search

The VISITORS System: Zooming into a PanelThe VISITORS System: Zooming into a Panel

Modifying Display Parameters for a Raw-Data Concept

Displaying the Distribution of an Abstract Concept over Time

Interactive Temporal Data Mining:Temporal Association Charts

[Klimov & Shahar, IDAMP 2007]Abstractions for the same subject group are connected; support and confidence indicated by width and hue

The data of each subject are

connected by a line

Temporal Association Charts: Support and Confidence

Links

Three data mining parameters displayed for each temporal association link:

support = 55.60% of patient group

confidence = 61.00% probability

actual number of patients = 25

Temporal Association Charts:Direct Manipulation Using a Time-Value Lens

Current WBC minimal value 1.4 103 cells/ml

91% of patients have the “moderately_low” value of the HGB_STATE_BMT

New WBC minimal value 5.43 103

cells/ml

Now only 44.4% of patients have the “moderately_low” value of the HGB_STATE_BMT

Temporal Association Charts:Direct Manipulation Using a Time-Value Lens (Cont)

Temporal Association Charts: Using a Relative Time Line

Time line is

relative to the BMT_Al procedure

Temporal Association Charts: Using a Relative Time Line (Cont)

During the first month after BMT_Al

During the second month after the BMT_Al procedure

Temporal Association Charts: Using a Relative Time Line (Cont)

1st month1st month2nd month 2nd month

Adding a New Clinical Database to The IDAN Temporal Mediator Architecture

• Due to local variations in terminology and data structure, linking to a new clinical database requires creation of– A term-mapping table– A unit-mapping table– A schema-mapping table

• The mapping tools use a vocabulary-server search engine that organizes and searches within several standard controlled medical vocabularies (ICD-9-CM , LOINC, CPT, SNOMED, NDF)

• Clinical databases are mapped into the standard terms and structure that are used by the clinical knowledge base, thus making the knowledge base(s) highly generic

The LOINC Server Search Engine

LOINC Search Results

Summary:Abstraction,Visualization, & Exploration

of Time-Oriented Clinical Data

• Automated, intelligent interpretation of longitudinal clinical data is useful for monitoring, therapy planning, data summarization and visualization, explanation, and quality assessment

• Interactive query, visualization, and exploration requires runtime access to the medical domain’s ontology

• The visualization and exploration semantics are specific to the intelligent-exploration task, but not to any particular medical domain