Knowledge-Based Interpretation, Visualization,

and Exploration of Time-Oriented Medical Data

Yuval Shahar, M.D., Ph.D.

Medical Informatics CenterInformation Systems Engineering

Ben Gurion UniversityBeer Sheva, Israel

AndDepartments of Medicine and Computer Science

Stanford Medical InformaticsStanford UniversityStanford, CA, USA

Time in Natural Language

From—

“Mr. Jones was alive after Dr. Smith operated on him”

Does it follow that—

“Dr. Smith operated on Mr. Jones before Mr. Jones was alive?”

Is Before the inverse of After?

Timing is Everything :Applications of Temporal Reasoning

• Natural-language processing (e.g., medical record understanding)

• Planning (e.g., robot planning, therapy planning)

• Causal reasoning (e.g., diagnosis)

• Archeology (e.g., seriation)

• Psychology (e.g., developmental beahvioral psychology)

• Scheduling (e.g., optimal ordering)

• Circuit design (e.g., sequential circuits)

• Software design (e.g., parallel processing, communication, verification)

• Other, not necessarily time-oriented, domains where interval algebra is useful, such as molecular biology (e.g., arrangement of DNA segments along a linear DNA chain) and evaluation of spatiotemporal traffic-control patterns

Allen's Temporal Logic (1981–1984)• Only temporal intervals - no instantaneous events

• 13 basic (binary) interval relations (b,a,eq,o,oi,s,si,f,fi,d,di,m,mi)

and transitivity relations between them

• Properties hold over every subinterval of an interval

—> Holds(p, T) e.g., "Patient1's skin was blue throughout sunday"

• Events hold only over an interval and not over any subinterval of it

—> Occurs(e, T) e.g., "patient2 broke a leg at 5pm"

• Processes hold over some subintervals of the interval they occur in

—> Occuring(p, T) e.g., "patient3 is chasing the nurse"

Allen’s 13 Temporal RelationsA

B

A

B

A

B

A

B

A

B

A

B

A

B

A is EQUAL TO B

A is BEFORE B

B is EQUAL TO A

B is AFTER A

A MEETS B

B is MET BY A

A OVERLAPS B

B is OVERLAPPED BY A

A STARTS B

B is STARTED BY A

A FINISHES B

B is FINISHED BY A

A is DURING B

B CONTAINS A

The Temporal-Abstraction Task

• Input: time-stamped clinical data and relevant events (interventions)

• Output: interval-based abstractions

• Identifies past and present trends and states

• Supports decisions based on temporal patterns, such as: “modify therapy if the patient has a second episode of Grade II bone-marrow toxicity lasting more than 3 weeks”

• Focuses on interpretation, rather than on forecasting

Temporal Abstraction:The Bone-Marrow Transplantation

Domain

.

•

0 40020010050

•

1000

2000( )

100K

150K

( )

•••

• • • ••

• •

•••

Granu-locytecounts

• • •

•

Time (days)

Plateletcounts

PAZ protocol

M[0] M[1] M[2] M[3] M[1] M[0]

BMT

Expected CGVHD

Temporal Abstractions: A Graphical View

Uses of Temporal Abstractions

• Therapy planning and patient monitoring. E.g., the EON project (a modular architecture to support guideline-based care)

• Creating high-level summaries of time-oriented medical records

• Supporting an explanation module for a medical DSS

• Representing goals and policies of therapy plans and guidelines for quality assessment purposes (at runtime and retrospectively). E.g., the Asgaard project: Intentions of guideline designers with respect to both process and outcomes are captured as temporal patterns to be achieved or avoided.

• Visualization of time-oriented clinical data: the KNAVE project

The Temporal-Abstraction Ontology

• Events (insulin therapy) - part-of, is-a relations

• Parameters (hemoglobin values and abstractions) - abstracted-into, is-a relations

• Abstraction goals (therapy of diabetes patients) - is-a relations

• Interpretation contexts (effect of regular insulin) - subcontext, is-a relations

• Interpretation contexts are induced by other entities and can have any temporal relationship to the inducing entity

Temporal-Abstraction Output Types

• State abstractions (LOW, HIGH)• Gradient abstractions (INC, DEC)• Rate Abstractions (SLOW, FAST) • Pattern Abstractions (CRESCENDO)

- Linear patterns- Periodic patterns

Temperature

Hemoglobin Level

Linear Component

Week 2 Week 3Week 1

Anemia

Fever Fever

Anemia Anemia

FeverFever

Anemia

Fever

Linear ComponentLinear Component Linear Component

Periodic Pattern

Abstraction of Periodic Temporal Patterns

Temporal-Abstraction Knowledge Types

• Structural (e.g., part-of, is-a relations) - mainly declarative/relational

• Classification (e.g., value ranges; patterns) - mainly functional

• Temporal-semantic (e.g., “concatenable” property) - mainly logical • Temporal-dynamic (e.g., interpolation functions) - mainly probabilistic

The RÉSUMÉ System Architecture

Temporal-abstraction mechanisms

Temporal fact baseE v e n t s

C o n t e x t s

A b s t r a c t e d i n t e r v a l s

P r i m i t i v e d a t a •

Domain knowledge base

Event ontology

Parameter ontology

Primitive data

Events

••

Context ontology

External patient database

+ +

+

•+

Test Domains for the RÉSUMÉ System

• Medical domains:– Guideline-based care

• AIDS therapy

• Oncology

– Monitoring of children’s growth

– Therapy of insulin-dependent diabetes patients

• Non-medical domains:– Evaluation of traffic-controllers actions

– summarization of meteorological data

Acquisition of Temporal-Abstraction Knowledge

Temporal Reasoning and Temporal Maintenance

• Temporal reasoning supports inference tasks involving time-oriented data; often connected with artificial-intelligence methods

• Temporal data maintenance deals with storage and retrieval of data that has multiple temporal dimensions; often connected with database systems

• Both require temporal data modelling

Clinicaldecision-supportapplication

TM TR DB

Tzolkin: A Temporal-Mediation Architecture

or:Combining Temporal Reasoning and Temporal

Maintenance

DB

Temporal-mediation controller

RÉSUMÉ (TR)

Chronus (TM)

Clinicaldecision-supportapplication

Knowledge-Based Visualization andExploration of Time-Oriented Medical Data:

Desiderata

• Interactive composition of (temporal-abstraction) queries

• Visualization of query results

• Exploration of multiple levels of temporal abstractions

• The semantics of the query, visualization and exploration operators should be domain independent, but should use the terms and relations specific to each (e.g., medical) domain

The KNAVE Project(Knowledge-Based Navigation of Abstractions for Visualization and

Explanation)

• A conceptual and computational framework for temporal abstraction, visualization, and exploration

• Capitalizes on existing components (RÉSUMÉ, temporal database mediator, KA tool, domain-knowledge server)

• The exploration operators reuse (and are defined by) the domain’s temporal-abstraction ontology

• Introduces new graphical and computational modules

The KNAVE ArchitectureTemporal Mediator

DB

Visualization andexploration module Computational

Module

GraphicalInterface

Domain-knowledge Server

End user

Expertphysician

KATool

Résumé

Chronus

Controller

KB

Ontology server

Beginning a Visualization Session: A Temporal-Abstraction Query

The Browsing and Exploration Interface

Semantic Exploration Operators• Motion across semantic links in the domain’s knowledge base;

in particular, relations (and their inverse) such as: - part-of - is-a - abstracted-from - subcontext • Motion across abstraction types: state, gradient, rate, pattern

• Application of aggregation operators such as mean and distribution

• Dynamic change of temporal-granularity (e.g., from days to months) changes the display, using domain-specific aggregation knowledge

• Explanation by display of relevant knowledge, or through “What-if” queries, which allow hypothetical assertion or retraction of data or knowledge and examination of resultant patterns

An Abstracted-From Exploration Result

A Statistical-Query Example

Responding to an Explanation Query (“How”):

A Bone-Marrow–toxicity Classification Table

The Preliminary Evaluation Study

• Developmental assessment of the prototype

• Seven users with varying medical/computer use backgrounds

• Each user given a 10 minute introduction to the KNAVE system

• A single electronic patient file constructed from several cases in the domains of AIDS and bone-marrow transplantation

• Each user asked to perform three tasks (a complex temporal query, a context-sensitive abstraction, and a statistical query)

• Qualitative impression and quantitative (time) measures noted

The Preliminary-Evaluation Results

• All users answered all queries within 3 minutes; 6 of 7 users completed all three tasks within 90 seconds

• All users expressed enthusiasm and found the interface useful

• Striking redundancy noted in use of interface: At least four different paths were found to the same answers, and five different patterns of use of the exploration operators

• Difficult to compare to manual tools, since these do not support any automated abstraction or explanation of such

KNAVE: Current State and Future Directions

• Basic prototype in Visual Basic; Java implementation under way

• Collaboration with an industrial company to create a web-based version

• Current research issues:

– Implementation of temporal-granularity semantic zoom

– Runtime linear and periodic pattern queries

– Semantics and implementation of distributed What-If queries, which modify either the knowledge or the data at runtime and examine the effect of the result on the displayed patterns

– Enhancement of RÉSUMÉ and the KA tool as needed, including integration with statistical tools

– Future link to a text summarization module

Temporal-Abstraction and Visualization:Conclusions

• Temporal abstraction of time-oriented data can employ reusable domain-independent computational mechanisms that rely on access to a domain-specific temporal-abstraction ontology

• Temporal abstraction is useful for planning, monitoring, data summarization and visualization, explanation and critiquing

• Interactive query, visualization of, and exploration requires runtime access to the domain’s temporal-abstraction ontology

• The visualization and exploration semantics can be specific to the temporal-abstraction task, but not to the domain