Using the Critical Incident Using the Critical Incident Technique to Better Understand Technique to Better Understand Patient Experiences of Patient Experiences of Ambulatory CareAmbulatory Care

Presented by:Kristin L. Carman, Ph.D.American Institutes for Research

Presented at:Academy for Health Services ResearchAnnual Meeting, San Diego, CA June, 2004

Project Team Roger Levine, PhD

Managing Research Scientist, AIR

Karen K. Shore, PhDSenior Social Scientist

Margarita Hurtado, PhDPrincipal Research Scientist

Kristin L. Carman, PhDPrincipal Research Scientist

Judy Mitchell, MSSenior Research Scientist

Steven A. Garfinkel, PhDManaging Research Scientist

San Keller, PhD• Principal Research Scientist

Funding Agency for Healthcare Quality

and Research• Part of the CAHPS II grant

Partners HMSA, Hawaii Humana, Chicago

Purpose of our projectPurpose of our project

To develop an A-CAHPS Survey

To use Critical Incident data in novel ways to address issues related to: Instrumentation

Quality improvement

Reporting, and

Cultural comparability

Critical Incident (CI) Critical Incident (CI) TechniqueTechniqueCritical Incident

“Incident”=an observable, specific behavior

“Critical”=means incident was crucial to the outcome of interest

Organized structure for data collection; focuses on observable behavior

Used to collect and analyze reports of behaviors associated with specific outcomes

Qualitative method; in-depth interviews

Methods and dataMethods and data

200 interviews; 40 providers and 260 patients

Patient respondents divided equally among four different racial/ethnic groups

Open-ended responses are transcribedEach interview usually generates 10

incidents; we’ll have 2000+ incidents

CI data management and CI data management and processingprocessing

Gathering extensive data

Using qualitative software to create and manage a very complex data base

Developed a very careful data processing protocol Raw data is open-ended responses, transcribed

Data is transformed into “incident write ups”

These two types of data are the sources for analyses

Specific GoalsSpecific Goals

Instrumentation goalsInstrumentation goals Develop a complete taxonomy of the

components of quality ambulatory health care, based on both patient and clinician perspectives Confirm that the domains measured by the draft

CAHPS® instrument are salient to patients and providers and can be assessed by patients

Determine whether the domains are salient and measurable for both men and women, for individuals with different levels of education, and across a range of racial and ethnic groups

Identify additional domains that should be measured

Instrumentation goals (cont’d)Instrumentation goals (cont’d)

Identify CAHPS® item content that can result in a spread of scores at the high end of the score distribution to minimize ceiling effects Generate objective patient reports of

health care experience

Create items for Ambulatory CAHPS® for which a positive rating would be rare

Instrumentation analysisInstrumentation analysis

Developing the taxonomy Randomly select at least 200 incidents

Two teams classify incidents into major categories, then subcategories

Iteratively validate and refine the taxonomy with additional set of incidents

Instrument analysis (cont’d)Instrument analysis (cont’d)

Using incidents as the unit of analysis, then respondents, we investigate statistical associations between personal characteristics and the way people conceive of quality of care Tabulation of respondent characteristics by

taxonomy categories which are quality of care themes (e.g., coordination of care)

Regress taxonomy categories on respondent characteristics

Instrument analysis (cont’d)Instrument analysis (cont’d)

Logistic Regression Simple and multiple

Dependent Variable=taxonomy category

Independent Variables= Individual demographics

Respondent: provider or patient

Quality improvement goalsQuality improvement goalsIdentify physician behaviors associated

with excellent and poor quality of care based on experiences of patients and clinicians Identify combinations or co-occurrences of

behavior

Identify key facilitators and barriers to quality of care, specifically related to CAHPS domains

Create tools for QI interventions to improve CAHPS scores

Quality improvement Quality improvement analysisanalysis Analyze interview and CI files Conduct additional coding of data Focus on which behaviors or actions

by clinicians co-occur to create positive (or negative) experiences for patients

Compare findings by respondent characteristics

Cultural Comparability goalsCultural Comparability goals

Identify variations in taxonomic structure for different racial/ethnic groups

If differences exist, identify the implications for: Supplemental domains, concepts and items

CAHPS domain labels and explanatory vignettes

Culturally appropriate interventions to improve care

Reports goals and analysisReports goals and analysis

Identify narratives (phraseology) that clearly and effectively explains CAHPS measures in reports

Next stepsNext steps

Complete interviews

Complete analyses

Disseminate findings

For more information, contact:For more information, contact:

Kristin L. Carman, PhDPrincipal Research ScientistAmerican Institutes for

Research1000 Thomas JeffersonWashington, DC 20007(202) 342-5090Kcarman@air.orgwww.air.org

Karen K. ShoreSenior Research ScientistAmerican Institutes for

ResearchADDRESSPalo Alto, CA (ZIP)PHONE NUMBERkshore@air.orgwww.air.org

[CI 1-2]

[CI 1-3]

[CI 2-1]

[CI 2-2]

[CI 3-1]

[CI 3-2]

[CI 1-1] Episode 1

Episode 3

Episode 2

Critical Incident behaviors

CI 1-1

Critical Incident Forms

CI 1-2

CI 1-3

CI 2-1

Verbatim TranscriptMarked by interviewer for Episode # andfor CI behaviors (not numbered—only numbered here for illustration) in Atlas/ti

CI Forms filled out by cut and paste fromTranscript, numbered to reflect source episode.This takes place in Word

CI 1-1

CI 1-2

CI 1-3

CI 2-1

CI forms cut into separatetext files for sorting into Taxonomy

Links are logical only: episode # in CI # allows analyst to trace back to

transcript

Transcript CIs for Interview Separate CIs

Database Creation

[CI 1-2]

[CI 1-3]

[CI 2-1]

[CI 2-2]

[CI 3-1]

[CI 3-2]

[CI 1-1] Episode 1

Episode 3

Episode 2

CI 1-1

CI 1-2

CI 1-3

CI 2-1

At this stage, the finalized concatenated CI file, with taxonomy codes embedded, is imported into Atlas/ti. Taxonomy codes are “autocoded” by searching for Taxonomy codewords. At this point, hypertext links can be created between CI forms and episodes in the transcript. If desired, Taxonomy codes may be applied manually to the transcript.

P-01-M-JM-1-1-ATAX-Communicates

P-01-M-JM-1-2-BTAX-Clarifies

The ID code includes #’s for episode and CI.Since ID #’s include identifiers for referents, they are assigned to demographic code families for referents

(See next page fordetail)

Transcript Concatenated CIs

Final Atlas/ti Database

Both transcript and concatenated CI filesare assigned to docfamilies for respondentdemographics

(See next page fordetail)

Taxonomy codes are

assigned by autocoding

Respondent demographics are represented by placing“primary documents” in “PD Families” or sets:

Example:

Gender::Male = {PD1; PD4; PD5; PD8…}Gender::Female = {PD2; PD3; PD6; PD7…}Age::20s = {PD1; PD2…}Age::30s = {PD4; PD8…}Age::40s = {PD3…}Age::50s = {PD6…}Age::60s = {PD5; PD7…}

Imported in a table:

Gender AgePD1 Male 20sPD2 Female 20sPD3 Female 40sPD4 Male 30sPD5 Male 60sPD6 Female 50sPD7 Female 60sPD8 Male 30s…

Dataset can be parsed according to Boolean combinations of set-memberships, for example, to restrict a query to documents that belong in both the Male and 20s sets.

Referent demographics are represented by placing CI ID codes into “Code Families” or sets. The logic is the same as for PD families, but at present there is no table import feature and assignment is made with the “code family manager” tool in Atlas/ti.

Again, set memberships can be used to focus queries ondifferent classes of referents.

RESPONDENT DEMOGRAPHICS REFERENT DEMOGRAPHICS

Strategies for associating data at episode or interview levels

Layered IDs:

Since ID’s are layered, a hierarchy of ID codes can be created in Atlas/ti. For example, a hierarchy could be structured as follows:

Interview ID +--Episode ID +--CI ID +--Referent ID

Code Families:

A code family that included all the CI ID codes for a given episode would enable searching by episode in the concatenated CI files.

Hypertext:

Hypertext links can be created from the CI forms to the episodes in the transcript, or even to the specific descriptions of behaviors from which the CIs are derived.