consumers’ appraisal of anecdotal accounts of patient experience ahrq annual conference steven c....

Consumers’ Appraisal of AnecdotalAccounts of Patient Experience

AHRQ Annual Conference

Steven C. Martino, PhD

September 10, 2012

Martino-2 Sep-12

Customer Reviews of Products and Servicesare Omnipresent Online

• Amazon.com first began allowing customers to post reviews of products in 1995

• Now almost all retail websites include them

• 78% of internet users (58% of adults) have researched a product or service online, and 32% have posted a comment or review (Pew Internet & American Life Survey, 2011)

• Reviews are a trusted source of information on product or service quality

• Current retail industry focus is on increasing the relevance of online customer reviews

Martino-3 Sep-12

Martino-4 Sep-12

Martino-5 Sep-12

Martino-6 Sep-12

Narrative Accounts of Patient Experience are Increasingly Available Online

• Dozens of commercial websites provide narrative accounts of patients’ experiences with their doctors (sometimes accompanied by ratings)

– e.g., Vitals.com, HealthGrades, Yelp, Angie’s List, and Consumer Reports

• RateMDs currently hosts narrative reviews of at least 1 in 6 practicing doctors in the U.S. (Gao et al., 2012)

• Currently, the only equivalent in the public sector is NHS Choices (British National Health Service)

Martino-7 Sep-12

Currently, the Audience for these Sitesis Small

• 80% of internet users (59% of adults) have looked online for health information

• 16% of internet users have consulted online rankings or reviews of doctors or other providers (4% have posted such reviews)

• 15% have consulted rankings or reviews of hospitals or other medical facilities (3% have posted such reviews)

Data source: Pew Internet & American Life Survey conducted Aug-Sep 2010

Martino-8 Sep-12

Martino-9 Sep-12

Martino-10 Sep-12

Promises and Limitations of Patient Narratives

• Patient narratives often cover the same domains as standardized surveys of patient experience but in ways that can be more vivid, concrete, and engaging

• Yet, the sample of respondents is often not representative (biased) and small (poor signal, low reliability)

– On RateMDs, the average number of reviews per physician is three (Lagu et al., 2010)

• If presented with standardized data, patient narratives could make unfamiliar, difficult-to-understand quality measures more salient and evaluable

• Yet, anecdotal data may overwhelm more formal statistical information

Martino-11 Sep-12

Extracting Meaning from Narratives is aComplex Cognitive Task

Enduring Characteristics of the Doctor

• Technical ability• Interpersonal skill

Enduring Characteristics of the Environment

• Health system guidelines• Case load and mix

Unique Aspects of the Situation

• Individual patient• Circumstantial context

Doctor’s Behavior

Martino-12 Sep-12

Extracting Meaning from Narratives is aComplex Cognitive Task

Enduring Characteristics of the Doctor

• Technical ability• Interpersonal skill

Enduring Characteristics of the Environment

• Health system guidelines• Case load and mix

Unique Aspects of the Situation

• Individual patient• Circumstantial context

Doctor’s Behavior

Consumers try to infer this from this

Martino-13 Sep-12

Cognitive Strategies that People Use to Discern Among Multiple Possible Causes of Behavior . . .

• Typically, people look for– Consistency in behavior across time and

situations: Has the doctor behaved a certain way repeatedly across time and situations?

– Consensus in people’s experiences with the person: Do other patients report getting similar treatment from the doctor?

– Distinctiveness of behavior: Is this doctor the only doctor to have behaved this way toward the patient?

Martino-14 Sep-12

. . . Are Not Necessarily Applicable in the Case ofPatient Narratives

• Consumers often do not have the information needed to answer questions about consistency, consensus, and distinctiveness.

– Usually limited to a small number of narratives– Do not get to “observe” the doctor’s behavior

over multiple occasions with the same patient or get the patient’s view of other doctors

– Narratives rarely mention situational factors that may have facilitated or constrained behavior; instead, they strongly imply dispositional causes

Martino-15 Sep-12

Consumers’ Values, Disposition, and Past Experiences May Affect Their Appraisal

• Consumers differ in how much they value data on patient experience and the importance they ascribe to various facets of patient experience and doctor quality

• Consumers with different decision-making styles may use narrative accounts differently

• Consumers have different levels of experience with healthcare quality data

Martino-16 Sep-12

Overview of Research Questions

• What cues do people use to judge the value and authenticity of patient narratives?

• How salient (memorable) are patient narratives relative to standardized measures of healthcare quality?

• How trustworthy and useful do consumers perceive narratives to be? What are bases for skepticism?

• Do people’s preferred decision-making styles, health status, or past exposure to healthcare quality data affect their attention to and appraisal of patient narratives?

Martino-17 Sep-12

• A fictitious public reporting website custom built for this and related studies

• Designed to be consistent with current public reports in terms of content, format, and functionality

• Presents three types of data– Standardized patient experience measures (CAHPS)– Clinical process and outcome measures (HEDIS)– Patient narratives (reviews)

• Hidden tracking system to monitor:– Click patterns– Time spent on each page

The SelectMD Website: Basic Design

17

Martino-18 Sep-12

• Modeled on actual narratives collected from RateMDs

• Matched to actual narratives based on length, cognitive complexity, and clarity of writing

• Pilot testing to compare actual (RateMD) narratives to a large set of fictitious ones constructed for this study

• Matched to reported CAHPS scores:– Focused on areas of experience covered by CAHPS– Higher scores associated with more positive

comments (but imperfectly)– Comments varied in emotional tone for all doctors

The SelectMD Website: Creating Realistic Patient Narratives

18

Martino-19 Sep-12

Martino-20 Sep-12

Martino-21 Sep-12

Martino-22 Sep-12

Martino-23 Sep-12

23

Martino-24 Sep-12

Pretesting of Patient Narratives

• Three rounds of testing, n = 8-9 per round

• Convenience sample of internet users with recent healthcare experience, mix of demographic characteristics

Online Experiment

• Random sample drawn from Knowledge Networks online research panel– Representative national sample (>60,000 households)– Limited to households with computer-based internet

access (89% of total panel), aged 25-64 years

• Of 1,757 invitations, over 48% accepted (n = 849)

Study Participants

Martino-25 Sep-12

Experimental Design:Six Study Arms

25

ARM MEASURES DOCTORS

1 CAHPS 12

2 CAHPS + HEDIS 12

3 CAHPS + COMMENTS 12

4 CAHPS + HEDIS + COMMENTS 12

5 CAHPS + COMMENTS 24

6 CAHPS + HEDIS + COMMENTS 24

Martino-26 Sep-12

Experimental Design:Six Study Arms

26

ARM MEASURES DOCTORS

1 CAHPS 12

2 CAHPS + HEDIS 12

3 CAHPS + COMMENTS 12

4 CAHPS + HEDIS + COMMENTS 12

5 CAHPS + COMMENTS 24

6 CAHPS + HEDIS + COMMENTS 24

Martino-27 Sep-12

• Questions asked of pretest participants– Emotional valence of narratives– Perceived informativeness & authenticity of narratives

• Questions asked prior to MD Choice– Prior exposure to public reports on plans, hospitals,

and doctors– Health status

• Questions asked after MD Choice– Recall of measures– Perceived usefulness and trustworthiness of measures– Decision making styles

Study Measures

27

Martino-28 Sep-12

Qualities of Narratives Associated with Their Perceived Usefulness and Authenticity

• Strongly worded narratives were perceived as less authentic (p = .001) but more informative (p < .001) than mildly worded narratives

• Narratives about negative experiences were perceived as less authentic (p < .001) than narratives about positive experiences

• Length of narratives was positively related to perceived authenticity, r (129) = 0.25, p = .005, and positively related to perceived informativeness, r (129) = 0.30, p = .001

Martino-29 Sep-12

Recall of Patient Narratives

• Of Ss who were shown patient narratives on the SelectMD site, 86% recalled seeing them

• Of Ss shown narratives plus CAHPS measures– 58% recalled both– 32% recalled narratives only– 4% recalled CAHPS measures only– 7% recalled neither

• Of Ss shown narratives plus HEDIS measures– 29% recalled both– 53% recalled narratives only– 3% recalled HEDIS measures only– 15% recalled neither

Martino-30 Sep-12

Recall of Patient Narratives (cont.)

• Ss low in avoidant decision-making were more likely to recall patient narratives than Ss high in avoidant decision-making (89% vs. 84%; p = .07)

• Ss high in intuitive decision-making were more likely to recall patient narratives than Ss low in intuitive decision-making (88% vs. 83%; p = .10)

• Ss high in intuitive decision-making and low on rational decision-making were especially likely (94%) to recall patient narratives

Martino-31 Sep-12

Recall of Patient Narratives (cont.)

• 81% of Ss who recalled the narrative data provided open-ended data on what they remembered specifically

Content recalled % of subjects who recalled this

content

% of narratives that contained this

content

Favorable comment 27 48

Unfavorable comment 28 52

Doctor communication/time spent 28 47

Access to care 14 33

Office staff 16 36

Doctor showed concern 10 36

Martino-32 Sep-12

Perceived Usefulness of Patient Narratives

• Among Ss who recalled seeing the narratives, 44% said that they were very useful for decision-making, 44% said somewhat useful, 9% said only a little useful, and 3% said not at all useful

• Unexpectedly, Ss high in intuitive decision-making judged the narratives as less useful than did Ss low in intuitive decision-making (p = .06)

• Ss with recent serious or chronic health conditions saw the narratives as less useful than Ss without those conditions (p = .01)

• Past exposure to healthcare quality data was not related to perceived usefulness of the narratives (p = .34)

Martino-33 Sep-12

Trustworthiness of Patient Narratives

• Ss perceptions of the usefulness and trustworthiness of narratives were strongly related, r = 0.59, N = 588, p <.001

• Among Ss who recalled seeing the narratives, 18% said that they were very trustworthy, 65% said somewhat trustworthy, 15% said only a little trustworthy, and 2% said not at all trustworthy

• Decision-making style and prior exposure to healthcare quality data were unrelated to how trustworthy narratives were perceived to be

• Ss with recent serious or chronic health conditions trusted narratives less than Ss without those conditions (p = .04)

Martino-34 Sep-12

Reasons Given for Mistrust of Patient Narratives

Reasons* Percentage of responses

Prefer to base my opinions on firsthand experience or information from people I know

35%

Everyone is looking for something different in a doctor; comments are not relevant to me or my situation

11%

Comments are subjective/biased 20%

Contextual information is missing 10%

Comments are too varied or contradict one another 7%

Too few comments to draw a reliable inference 5%

Comments could be faked or edited 12%

Distrust any information on the web 8%

* Given by 101 subjects who judged narratives to be “only a little” or “not at all” trustworthy

Martino-35 Sep-12

Attention Given to Patient Narratives

• Overall, Ss (N = 593) read reviews of 17.1% (SD = 24.4) doctors (includes those who did not drill down to reviews)

• Ss who drilled down to the reviews (N = 157) read the reviews of 30.2% (SD = 25.6) of doctors

• Ss high in avoidant decision-making read reviews of a smaller percentage (11.2%) of doctors than Ss low in avoidant decision-making (14.9%; p = 0.10)

Martino-36 Sep-12

Attention Given to Patient Narratives (cont.)

• Ss low in analytic decision-making and low in intuitive decision-making read reviews of a smaller percentage (11.7%) of doctors than Ss with other combinations of these two styles (who read the reviews of 17-20% of doctors; F = 3.2, p = .02)

• Ss with prior exposure to healthcare quality information read reviews of a larger percentage of (20.3%) of doctors than Ss without prior exposure (15.1%; p = .01)

• Ss with recent serious or chronic health conditions read reviews of a larger percentage of doctors (19%) than did Ss without such conditions (16%; p =.09)

Martino-37 Sep-12

Conclusions• People use cues in narratives to determine their value and relevance

• When presented together, patient narratives appear to be more memorable (salient) than standardized data on quality

• Most people trust patient narratives and find them useful for decision-making

• A minority recognizes the limitation of patient narratives as a source of data on data on doctor quality

• Positive and negative accounts of patient experience are recalled equally; no domain of patient experience is more salient than others

• Attention to and evaluation of patient narratives depends on people’s decision-making tendencies, health status, and prior experience with healthcare quality data

Martino-38 Sep-12

Implications

• It may be possible to increase healthcare consumers’ engagement in and use of reports of standardized quality data by incorporating patient narratives

• Need to elicit patient narratives in a way that makes them most useful to consumers (requires attention to issues of reliability and validity)

• Need to present narratives in a way that clarifies their value (as illustrative rather than representative), relevance (by building trust in the methods of elicitation and reporting) and limitations