September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health1

Implementing Survey Instruments in LOINC

Thomas White, MD, MSThomas White, MD, MS

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health2

Outline

History and VisionHistory and Vision Why are Survey Instruments Challenging?Why are Survey Instruments Challenging? Proposed Migration to an Implementation Proposed Migration to an Implementation

Model (“the question is the concept”)Model (“the question is the concept”) ValidationValidation

Nursing LOINC => DialogixNursing LOINC => Dialogix Dialogix Instruments => LOINCDialogix Instruments => LOINC

Lessons LearnedLessons Learned Next StepsNext Steps

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health3

History

MotivationMotivation Web-based instruments so can track symptoms Web-based instruments so can track symptoms

(diagnoses), clinical outcomes over time(diagnoses), clinical outcomes over time Aggregate across instruments / variants (e.g. severity Aggregate across instruments / variants (e.g. severity

of depression)of depression) ChallengesChallenges

100s of instruments, >10K variables; no standards100s of instruments, >10K variables; no standards Variants may not be comparableVariants may not be comparable

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health4

Vision

National infrastructure for assessment National infrastructure for assessment instrumentsinstruments Support all common instrumentsSupport all common instruments Use a standardized coding systemUse a standardized coding system Support multiple languagesSupport multiple languages Psychometric re-validationPsychometric re-validation Foster comparison of different ways of Foster comparison of different ways of

measuring constructmeasuring construct

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health5

Outline

History and VisionHistory and Vision Why are Survey Instruments Challenging?Why are Survey Instruments Challenging? Proposed Migration to an Implementation Proposed Migration to an Implementation

Model (“the question is the concept”)Model (“the question is the concept”) ValidationValidation

Nursing LOINC => DialogixNursing LOINC => Dialogix Dialogix Instruments => LOINCDialogix Instruments => LOINC

Lessons LearnedLessons Learned Next StepsNext Steps

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health6

Variants May Not Be Comparable

OriginalOriginal

CNYPCCNYPC

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health7

How Should Variants Be Coded? Current approachCurrent approach

One code for all variants One code for all variants Risks type I and II errors if scales differRisks type I and II errors if scales differ

Safer approachSafer approach Unique code per minor variantUnique code per minor variant Reduces data sharingReduces data sharing

Best approachBest approach Fewest codes needed to disambiguate versions with Fewest codes needed to disambiguate versions with

truly different meanings or reference rangestruly different meanings or reference ranges Catch-22 – research to compare versions requires Catch-22 – research to compare versions requires

separate codesseparate codes

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health8

How Are Survey Variables Defined?

Conceptual definitionConceptual definition Construct being assessedConstruct being assessed

Operational definitionOperational definition Wording of the questionWording of the question

Variable definitionVariable definition Available answer optionsAvailable answer options Validation criteriaValidation criteria Associated internal coding valuesAssociated internal coding values

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health9

What Can Modify a Variable’s Meaning or Reference Range? How questions are posedHow questions are posed

Wording and languageWording and language Context (order in which asked)Context (order in which asked) Presentation attributesPresentation attributes

The method used to ask themThe method used to ask them Likert scale, standard gamble, time trade-off, …Likert scale, standard gamble, time trade-off, …

Who does the askingWho does the asking Interviewer, self, computer, …Interviewer, self, computer, …

Who does the answeringWho does the answering Capability and willingness to answer truthfullyCapability and willingness to answer truthfully

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health10

Wording

SectionSection Headers, instructions, labelsHeaders, instructions, labels

QuestionQuestion Labels, instructionsLabels, instructions

Answer optionAnswer option Labels, anchors, codesLabels, anchors, codes

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health11

Context / Ordering

LengthLength Waning attention or motivationWaning attention or motivation

Proximity / GroupingProximity / Grouping Neighboring questions affect expectationsNeighboring questions affect expectations

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health12

Presentation

Look and feel of instructions / questionsLook and feel of instructions / questions Mechanism for asking / respondingMechanism for asking / responding

Likert scalesLikert scales Lengthy listsLengthy lists Standard gamble, …Standard gamble, …

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health13

More Complex Assessment Instruments

DISC: DiagnosticInterview Schedulefor Children

ClinicalClinicalGuidelineGuideline

StructuredStructuredDiagnosticDiagnosticInterviewInterview

Scored QuestionnaireScored Questionnaire

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health14

Sources of Instrument Complexity

Branching (order / context)Branching (order / context) Tailoring (wording)Tailoring (wording)

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health15

Hidden-Branching NodesHidden-Branching Nodes(e.g. NovaResearch’s QDS, Sawtooth Ci3)(e.g. NovaResearch’s QDS, Sawtooth Ci3)

Branching: Traditional Solutions Rely on Post-conditioned Gotos

Response-Level Branching Response-Level Branching (e.g. Perseus SurveySolutions)(e.g. Perseus SurveySolutions)

Q1Q1

Q2Q2

Hidden branching nodeHidden branching node if (exp) goto Q3if (exp) goto Q3Q1Q1

A1A1

A2A2

Q2Q2

Q3Q3

A3A3

goto Q2goto Q2

goto Q3goto Q3

(implicit goto Q2)(implicit goto Q2)

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health16

Relevance-based Branching Logic

Example:Example:

name:name: 11 What is the subject’s name?What is the subject’s name?

gender:gender: 11 What is [name]’s gender?What is [name]’s gender?

age:age: 11 How old is [name]? How old is [name]?

menarche:menarche: age > 9 && gender == ‘female’age > 9 && gender == ‘female’ Has [name] had her first period?Has [name] had her first period?

LMP:LMP: menarche == ‘true’menarche == ‘true’ When was [name]’s last period? When was [name]’s last period?

sexual:sexual: 11 Is [name] sexually active?Is [name] sexually active?

Relevance–Based Branching (e.g. Dialogix, ?XForms?)Relevance–Based Branching (e.g. Dialogix, ?XForms?)

Q1:Q1: Q1.relevant?Q1.relevant? Ask Q1Ask Q1

Q2:Q2: Q2.relevant?Q2.relevant? Ask Q2Ask Q2

yesyes Store N/A Store N/A

nono

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health17

Tailoring:Dynamically Composed Questions

Some [children/young people] get very upset or nervous when they are not with their parents or with the grown-ups who usually look after them. I am going to ask you some questions about that.

1. In the last year – that is, since [[NAME EVENT]/[NAME CURRENT MONTH] of last year] – was there a time when [CHILD’S NAME] often wanted to stay at home and not go (to [school/work] or other) places without [you/[ATTACHMENT FIGURE]]?

A. IF YES, Where there several weeks in a row when [he/she] seemed nervous or afraid about being away from [you/[ATTACHMENT FIGURE]] or away from home?

A. SI DICE SI, ¿Hubo varias semanas seguidas en las que se sintió nervioso(a) o asustado(a) por estar separado(a) de [useted/su [FIGURA DE APEGO]] o fuera de su casa?

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health18

Static vs. Dynamic Features

Instrument Definition (Static)Instrument Definition (Static) Wording & Presentation – untailored textWording & Presentation – untailored text Order – branching logicOrder – branching logic

Instrument Use (Dynamic)Instrument Use (Dynamic) Tailored contentTailored content Actual order of questions seenActual order of questions seen Time spent reading and answeringTime spent reading and answering Who does the asking (face-to-face, self, …)Who does the asking (face-to-face, self, …) Modality used to ask it (Likert, standard gamble, …)Modality used to ask it (Likert, standard gamble, …)

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health19

Implications for Coding Assessment Instruments Meaning and reference range are:Meaning and reference range are:

Determined by operational and variable definitionsDetermined by operational and variable definitions Altered by wording, order, presentation, assessment Altered by wording, order, presentation, assessment

methodmethod Further altered by dynamic tailoring, language, and Further altered by dynamic tailoring, language, and

actual order askedactual order asked Without separate codes, risk of type I and II errorsWithout separate codes, risk of type I and II errors Need explicit modeling to compare variantsNeed explicit modeling to compare variants

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health20

Outline

History and VisionHistory and Vision Why are Survey Instruments Challenging?Why are Survey Instruments Challenging? Proposed Migration to an Implementation Proposed Migration to an Implementation

Model (“the question is the concept”)Model (“the question is the concept”) ValidationValidation

Nursing LOINC => DialogixNursing LOINC => Dialogix Dialogix Instruments => LOINCDialogix Instruments => LOINC

Lessons LearnedLessons Learned Next StepsNext Steps

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health21

Current LOINC Schema for Nursing Instruments (1)

ElementElement Size Size (chars)(chars)

DefinitionDefinition

ComponentComponent 150150 Name of construct assessedName of construct assessed

PropertyProperty 3030 [Impression | Finding][Impression | Finding]

TimingTiming 1515 [Point | Interval][Point | Interval]

SystemSystem 100100 (e.g. eye, patient, family)(e.g. eye, patient, family)

ScaleScale 3030 (nominal, ordinal, …, narrative)(nominal, ordinal, …, narrative)

MethodMethod 5050 [Observed|Reported].instrument[Observed|Reported].instrument

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health22

Current LOINC Schema for Nursing Instruments (2)

ElementElement SizeSize DefinitionDefinition

ClassClass 2020 Name of instrumentName of instrument

Survey Question Survey Question TextText

255255 Exact text of questionExact text of question

Survey Question Survey Question SourceSource

5050 [InstrumentName].[Question#] [InstrumentName].[Question#] (e.g. OMAHA III.26)(e.g. OMAHA III.26)

CommentsComments 255255 Description of constructDescription of construct

FormulaFormula 255255 Equation for calculated valueEquation for calculated value

Answer ListAnswer List 64K64K List of allowable answersList of allowable answers

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health23

Current LOINC Schema – Limitations and Recommendations Assumes one version of each instrumentAssumes one version of each instrument

Alter Method to create hierarchy of variantsAlter Method to create hierarchy of variants Inadequate modeling of operational and variable Inadequate modeling of operational and variable

definitionsdefinitions Create formal XML model for theseCreate formal XML model for these

No modeling of instrument branching logicNo modeling of instrument branching logic Extend LOINC Panel to codify instrumentsExtend LOINC Panel to codify instruments

No support for dynamic featuresNo support for dynamic features Can annotate answers/values with tailoring, Can annotate answers/values with tailoring,

branching, and timing informationbranching, and timing information

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health24

Dialogix As Possible Solution

Syntax and schema for defining instrumentsSyntax and schema for defining instruments Each node represents a single measurable Each node represents a single measurable

entity or communicates an important messageentity or communicates an important message Engine for deploying themEngine for deploying them Data Schema for storing resultsData Schema for storing results

Records order, logs backtracking and changed Records order, logs backtracking and changed answers, stores fully tailored text and time answers, stores fully tailored text and time spent / method used to answer questions.spent / method used to answer questions.

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health25

Sample Dialogix Instrument

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health26

Dialogix Instrument SyntaxFieldField Syntax summarySyntax summary Example NodeExample Node

WWHHYY

ConceptConcept Free textFree text index event; trip or vacation; six index event; trip or vacation; six months agomonths ago

LocalNameLocalName Free text, must be uniqueFree text, must be unique intro_Q14Dintro_Q14D

SourceNameSourceName Free text, refers to source Free text, refers to source instrumentinstrument

Q14DQ14D

RelevanceRelevance Boolean expressions Boolean expressions intro_Q14 == 0 && intro_Q14B == 0 intro_Q14 == 0 && intro_Q14B == 0 && intro_Q14C == 0&& intro_Q14C == 0

HHOOWW

ActionTypeActionType qq – display expression – display expressionee – evaluate expression – evaluate expression[ ][ ] – delimit groups of questions – delimit groups of questions

displaydisplay

DataTypeDataType Number, text, date, month, dayNumber, text, date, month, day numbernumber

ValidationCriteriaValidationCriteria <min:max> inclusion criteria, <min:max> inclusion criteria, input mask, regex validationinput mask, regex validation

  

ActionPhraseActionPhrase Type Type qq – free text with – free text with embedded HTML or equations.embedded HTML or equations.Type Type ee -- equation -- equation

Did you take any trips or vacations Did you take any trips or vacations during `toMonth(getNow()) - 6`?during `toMonth(getNow()) - 6`?

AnswerOptionsAnswerOptions <format><format>

<|value|phrase> <|value|phrase>

listlist 0|No0|No

1|Yes1|Yes

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health27

Dialogix Data Logs

Data (per answer given)Data (per answer given) localName, answer, timestamp, sequence, localName, answer, timestamp, sequence,

tailored content, language usedtailored content, language used

Events (for each screen-full of questions)Events (for each screen-full of questions) Server, network, and browser processing timeServer, network, and browser processing time Per-question keystroke and mouse events and Per-question keystroke and mouse events and

timingtiming

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health28

Four Recommendations for LOINC

1.1. Alter Alter MethodMethod naming schema to disambiguate naming schema to disambiguate variantsvariants

2.2. Add Assessment_Instruments tableAdd Assessment_Instruments table Migrate existing nursing instrumentsMigrate existing nursing instruments

3.3. Store instrument branching logic in LOINC Store instrument branching logic in LOINC PanelPanel

4.4. Post-coordinate dynamic features with value Post-coordinate dynamic features with value portion of name-value pairsportion of name-value pairs

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health29

(1) Alter Method Schema

[Observed|Reported].[Inst name].[Variant identifier][Observed|Reported].[Inst name].[Variant identifier] Variant Identifier refers to entry in Variant Identifier refers to entry in

Assessment_Instruments tableAssessment_Instruments table HierarchiesHierarchies

Root level has unspecified (or unknown method)Root level has unspecified (or unknown method) Levels below root specify details – initially one level Levels below root specify details – initially one level

deep (no known relationship between versions)deep (no known relationship between versions) Re-arrange hierarchy as psychometric studies show Re-arrange hierarchy as psychometric studies show

similarity / differences among variantssimilarity / differences among variants

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health30

(2) Assessment_Instruments Table

One row per measurable entityOne row per measurable entity Foreign Keys pointing to LOINC entry to text Foreign Keys pointing to LOINC entry to text

and LOINC Panel which codifies instrument and LOINC Panel which codifies instrument New Dialogix fields to fully codify New Dialogix fields to fully codify

instrumentinstrument Re-purpose Existing LOINC ColumnsRe-purpose Existing LOINC Columns

SurveyQuestionText => ActionPhaseSurveyQuestionText => ActionPhase AnswerList => AnswerOptionsAnswerList => AnswerOptions

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health31

Sample Concept Hierarchy

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health32

Sample Question – BPRS

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health33

Specifying Validation Criteria

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health34

Simple Calculations – Anxiety and Depression Subscale of BPRS

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health35

Complex Calculations: List of DSM-IV Diagnoses

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health36

Multilingual Support – XML Model

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health37

Multilingual – Translator’s View

3939

(14 questions left)(14 questions left)

How alert do you feel during How alert do you feel during the first half hour after you the first half hour after you wake up in the morning?wake up in the morning?

[1] Not at all alert[1] Not at all alert

[2] Slightly alert[2] Slightly alert

[3] Fairly alert[3] Fairly alert

[4] Very alert [4] Very alert

3939

(Осталось 14 вопросов)(Осталось 14 вопросов)

Насколько легко Вам Насколько легко Вам соображать в первые соображать в первые полчаса после того, как полчаса после того, как проснулись утромпроснулись утром? ?

[1] [1] Очень трудноОчень трудно

[2] Довольно трудн[2] Довольно трудно о

[3] [3] Вполне легкоВполне легко

[4] [4] Очень легкоОчень легко

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health38

(3) Store Instrument Sequence Information in LOINC Panel Give each instrument variant its own PanelGive each instrument variant its own Panel Could use existing Dialogix XML schemaCould use existing Dialogix XML schema IssuesIssues

Consider using GLIF and Arden for metadataConsider using GLIF and Arden for metadata W3C XForms, VoiceXML, and P3P schemasW3C XForms, VoiceXML, and P3P schemas Possible many-to-many relationship between Possible many-to-many relationship between

questions and instrumentsquestions and instruments What about dummy variables?What about dummy variables? What about module-level presentation features?What about module-level presentation features?

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health39

Dialogix Schema – Prototype for LOINC Panel

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health40

(4) Post-Coordinate Dynamic Features Use ANO field within OBX of HL7Use ANO field within OBX of HL7 Might use Dialogix data modelMight use Dialogix data model Should consider using a standard schema, like Should consider using a standard schema, like

those in click-stream analysis.those in click-stream analysis. Consider normalizing to XML-Event modelConsider normalizing to XML-Event model

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health41

Outline

History and VisionHistory and Vision Why are Survey Instruments Challenging?Why are Survey Instruments Challenging? Proposed Migration to an Implementation Proposed Migration to an Implementation

Model (“the question is the concept”)Model (“the question is the concept”) ValidationValidation

Nursing LOINC => DialogixNursing LOINC => Dialogix Dialogix Instruments => LOINCDialogix Instruments => LOINC

Lessons LearnedLessons Learned Next StepsNext Steps

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health42

Columns in Assessment Instruments Table One row per measurable entityOne row per measurable entity

InstrumentInstrument SequenceSequence ConceptConcept LocalNameLocalName SourceNameSourceName RelevanceRelevance ActionTypeActionType ActionPhraseActionPhrase AnswerOptionsAnswerOptions

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health43

Validation: Existing Instruments LOINC => Dialogix Concept = Component; Relevance = 1; ActionType = qConcept = Component; Relevance = 1; ActionType = q SourceName = Survey_Quest_SrcSourceName = Survey_Quest_Src LocalName ~= Source Name or “q” + SequenceLocalName ~= Source Name or “q” + Sequence ActionPhrase = Best of:ActionPhrase = Best of:

Survey_Quest_Text (e.g. LIV-HIV)Survey_Quest_Text (e.g. LIV-HIV) Comments (HHCC)Comments (HHCC) Component (Omaha, QAM, HIV-SSC)Component (Omaha, QAM, HIV-SSC) Component: System (Omaha, reflexes in HHCC)Component: System (Omaha, reflexes in HHCC)

AnswerOptions = Best of AnswerList and AnswerOptions = Best of AnswerList and Exmpl_Answers, using Dialogix’s syntaxExmpl_Answers, using Dialogix’s syntax

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health44

Review Nursing Instruments in LOINC + Dialogix

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health45

Validation: New InstrumentsDialogix => LOINC (1) Diagnostic Instruments, with ScoringDiagnostic Instruments, with Scoring

AUDIT, BPRS, BASIS-32, SF-36AUDIT, BPRS, BASIS-32, SF-36 Structured InterviewsStructured Interviews

DISC, SCID-IIDISC, SCID-II Decision TreesDecision Trees

GAF-TreeGAF-Tree Clinical GuidelinesClinical Guidelines

NYC DOH TB Testing & Treatment GuidelineNYC DOH TB Testing & Treatment Guideline Healthcare EducationHealthcare Education

AutoMEQAutoMEQ

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health46

Validation: New InstrumentsDialogix => LOINC (2) Most Axes Can Be Derived From Dialogix SchemaMost Axes Can Be Derived From Dialogix Schema

Property = FIND for questions; IMP for eqnsProperty = FIND for questions; IMP for eqns System = ^Patient (some exceptions)System = ^Patient (some exceptions) Scale_Type = AnswerOptions and DataTypeScale_Type = AnswerOptions and DataType Method_Typ = [Observed|Reported].InstrumentMethod_Typ = [Observed|Reported].Instrument Class = Survey.[Domain].InstrumentClass = Survey.[Domain].Instrument

Some Axes Require Manual ReviewSome Axes Require Manual Review ConceptConcept Time_AspectTime_Aspect Sometimes System, Method_TypSometimes System, Method_Typ

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health47

Review Dialogix Instruments Newly Added to LOINC

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health48

Some Issues From Dialogix => LOINC Modeling global and modular instruction?Modeling global and modular instruction?

Should any them be named?Should any them be named? Modeling equations – which should be named?Modeling equations – which should be named? Add Add Date, Month, Date, Month, andand Day Day to Scale? to Scale? Time_Aspect – add Time_Aspect – add Lifetime Lifetime when asking when asking

whether a diagnosis / symptom ever occurred?whether a diagnosis / symptom ever occurred? Mapping local values to standard ones for dates, Mapping local values to standard ones for dates,

demographics, diseases, etc.demographics, diseases, etc. Supporting multiple languagesSupporting multiple languages

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health49

Conclusions

Instrument variants may need different codes.Instrument variants may need different codes. Proposed LOINC extensions canProposed LOINC extensions can

Model complex instrumentsModel complex instruments Disambiguate versionsDisambiguate versions Facilitate meta- and psychometric analysisFacilitate meta- and psychometric analysis Operationalize instrumentsOperationalize instruments Facilitate translation and ongoing QAFacilitate translation and ongoing QA

Additional modeling work needed for LOINC Additional modeling work needed for LOINC Panel and HL7 ANO attachments.Panel and HL7 ANO attachments.

September 30, 2003 Thomas White, MD, MS – NYS Office of Mental

Health50

Thank You