hborSeadcttc

Usability and Accessibilityin Consumer Health Informatics

Current Trends and Future Challenges

Larry Goldberg, BA, Bettijoyce Lide, MS, Svetlana Lowry, PhD, Holly A. Massett, PhD,Trisha O’Connell, MBA, Jennifer Preece, PhD, Whitney Quesenbery, BA,

Ben Shneiderman, PhD

Abstract: It is a truism that, for innovative eHealth systems to have true value and impact, theymustfırst and foremost be usable and accessible by clinicians, consumers, and other stakeholders. In thispaper, current trends and future challenges in the usability and accessibility of consumer healthinformatics will be described. Consumer expectations of their healthcare providers and healthcarerecords in this new era of consumer-directed care will be explored, and innovative visualizations,assistive technologies, and other ways that healthcare information is currently being provided and/orshared will be described. Challenges for ensuring the usability of current and future systems will alsobe discussed. An innovative model for conducting systematic, timely, user-centered research onconsumer-facing websites at the National Cancer Institute (NCI) and the ongoing efforts at theNational Institute of Standards and Technology (NIST) to promote health information technology(HIT) usability standards and evaluation criteria will also be presented.(Am J Prev Med 2011;40(5S2):S187–S197) © 2011 American Journal of Preventive Medicine

Introduction

Health information technology (HIT) is a multi-disciplinary fıeld thatmust often coordinate andwork across many different silos. However, to

ave true value and impact, HITmust, fırst and foremost,e usable and accessible by clinicians, consumers, andther stakeholders. This paper addresses the major issueselated to consumer health informatics (i.e., eHealth).ection 1 addresses the consumer expectations forHealth. Section 2 discusses innovative visualizations andssistive technologies in consumer informatics. Section 3iscusses accessible eHealth information. Section 4 dis-usses usable eHealth information. Section 5 discusseshe necessity of usability and accessibility in the certifıca-ion process of HIT applications. Section 6 extracts con-lusions from the research-based discussion in the prior

From the Media Access Group, WGBH National Center for AccessibleMedia (Goldberg, O’Connell), Boston, Massachusetts; the InformationTechnology Laboratory, National Institute of Standards and Technology(Lide, Lowry), Gaithersburg; Offıce of Communications and Education,National Cancer Institute (Massett), Bethesda; College of InformationStudies, Maryland’s iSchool (Preece), Human–Computer Interaction Lab-oratory and Department of Computer Science (Shneiderman), Universityof Maryland, College Park, Maryland; and Whitney Interactive Design(Quesenbery), High Bridge, New Jersey

Address correspondence to: Bettijoyce Lide,MS, 100 BureauDrive,MailStop 8900, GaithersburgMD 20899-8900. E-mail: bettijoyce.lide@nist.gov.

0749-3797/$17.00doi: 10.1016/j.amepre.2011.01.009

©2011American Journal of PreventiveMedicine. All rights reserved.

sections essential for the success of consumer healthinformatics.

Consumer Expectations for eHealthThe Pew Internet & American Life Project (http://www.pewinternet.org) has tracked online activities sinceearly in 2000, including a focus on consumer use of onlinehealth information. Their 10-year data chart rising use ofthe Internet by all age groups (Figure 1). The use of theInternet for health information is also on the rise, althoughadults with chronic diseases are less likely to go online thanhealthy adults.1 These statistics are mirrored in the experi-ence of researchers at theNCI’sUser-Centered InformaticsLab in qualitative research. Early on, for example, fewpeople claimed to know about or use social media tools.More recently, answers at these interviews have shifted:the web has gone from something that is unusual or usedonly at work to something that is normal and used every-day, on both computers andmobile devices. The questionnow is not whether the public is ready for eHealth infor-mation, but whether eHealth information is ready tomeet the public’s expectations. eHealth is joining com-merce, banking, travel, and other industries, alreadyonline.2

The challenge for HIT is to design systems that arepowerful enough not only to handle the volume of infor-

mation and the complexity of medical data, but also to

Am J PrevMed 2011;40(5S2):S187–S197 S187

oano

aasgfiodwd

S188 Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197

support both patients andprofessionals in doingtheir work. This meanspaying attention to usabil-ity from the beginning of aproject.Usability is a qual-ity of the product, but it isdefıned in the terms of thepeople. The widely citedInternational Organiza-tion for Standardization(ISO) standard for usabil-ity says that a product isusable if the people whoare intended to use it cando so in a way that is effec-tive for them, effıcient intheir terms, and satisfyingfor them in terms of theirown goals.3 Another ISOstandard simply defınes ac-cessibility as usability for abroader group of people—those with the widestrange of capabilities.4 Inther words, all users haven equivalent experience no matter what assistive tech-ology they may use, or whether they are on a computerr mobile device.This result is not achieved by attempting to “add us-

bility” to a system after it is designed. Both usability andccessibility must be con-idered from the very be-inning of a project. In theıeld of human–computernteraction, this process isften called user-centeredesign (UCD) and comesith its own research tra-ition5–7 and interna-

tional standards.8 Thisprocess begins with anunderstanding of thecontext of use and incor-porates evaluation of de-sign solutions as a way tomodify the design until itmeets the needs of users(Figure 2).As important as stan-

dards are, they often donot address the usability

Figure 1. The Pew Interneof the Internet by all ageAs of May 2010, 79% of all aduaged �65 years, at only 42Infographics/2010/Internet-acpermission

Figure 2. The UCD procerequirements from an undproduce design solutions,used to update the design

and accessibility needs of UCD, user-centered design

the real people who have to use them because they are notwritten around broad goals for usability and accessibility.One of the key tenets of user-centered design is that thepeople who use the system for information must be seenas individuals in a context of use, not just as a collection ofdemographics. To better understand consumer needs, a

merican Life Project shows the dramatic rise in the useps.ed �18 years were online. The lowest use is by the cohort of adultsing to 95% for those aged 18–29 years (www.pewinternet.org/by-age-group-over-time-Update.aspx, 1/20/11). Printed with

in ISO 9241-210 is an iterative cycle that generatesanding of the context of use, uses those requirements tothen evaluates them. The results of that evaluation are

til the product meets both usability and business goals.

t & Agroults ag

%, risess-

sserstandun

www.ajpm-online.net

st(

porjnmiaa

wwtdtsp

fhe

tliNpthstctgr

aces

Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197 S189

case study is explored, involving the development of de-tailed “patient personas” to guide the design of aNationalCancer Institute consumer-facing website. Personas helppicture the audience for any system in a way that candrive the design process from the user perspective.9 Per-onas package demographic data, qualitative data, atti-udes, and typical scenarios into a portrait of a type of userFigure 3).Personas help address a challenge for most technologyrojects: health information specialists, designers, andther health professionals, although present, there isarely a patient actively working on a project. This is notust a challenge in health care; in many information tech-ology projects, developers work in environments thatake it diffıcult to have any real connection to the context

n which their work will be used. Designing without usernd task analysis often produces products that ignoreccessibility requirements,10 fail to address the variationsin roles, and do not provide the right balance of usabilityrequirements.11 For example, a health provider whoorks with the same program all the time, may want it toork quickly and effıciently and to give a lot of informa-ion at once. But a patient might want the informationelivered in a more gentle way and to be sure that it helpshem understand the implications. Both might want aystem that is error tolerant, supporting them by bothreventing errors and helping them undo mistakes.As work proceeds toward usability standards for certi-

ıcation, those standards must address the full range ofuman experience and health contexts. It is amistake, for

Figure 3. For the National Cancer Institute’s www.cancer.gand health literacy, but also differing needs at various pl

xample, to think that “consumers engaging with health

May 2011

information” happens in only a single type of interaction.In addition to the wide variety of systems, technologies,and devices in use, health information from people—healthcare providers, friends, and family—is still impor-tant.1 This means thinking about HIT in the context ofhe larger social ecosystem, providing links between on-ine systems and human support. Participants in a usabil-ty study were recently asked what hours they wanted theCI’s Cancer Information Service to be available forhone, e-mail, and online chat. Many answered that evenhough it is not necessarily practical, they would prefer toave 24-hour support. They understood the realities, buttill wanted to have access to information any time thathey need it. If usability and accessibility in the consumerontext are not taken into account, then there is no wayhat the fınal products willmeet usability and accessibilityoals for the broadest range of people with the widestange of capabilities.

Framework for DesignThe Reader-to-Leader Framework (Figure 4) is designedto help researchers, designers, and managers understandhuman motivation to participate in social media. Thisframework enables them to improve user-interface de-sign and social support for projects run by their compa-nies, government agencies, and nongovernmental orga-nizations. These improvements could reduce the numberof failed projects, while accelerating the applications ofsocialmedia in health care, disaster response, community

he map of consumer personas considers both technologyalong the cancer journey.

ov, t

safety, and more.12,13

lpcso

i

os

oemocfsh

asr

aiiesr

/5/)

S190 Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197

The culture of theInternet is about muchmore than informationtransfers—it has becomeincreasingly social andcommunal. While indi-viduals’ needs and char-acteristics initially shapeinterface designs, socialrequirements determinethe nature of technology-mediated communica-tion.14 Researchers, de-signers, and managerswant to learn from suc-cessful applications andto apply this knowledgein the design and man-agement of applicationsthat they care about.15

The successive levels of social participation canroughly be categorized as reading, contributing, collabo-rating, and leading. These are not complete descriptions,andusersdon’t alwaysprogress fromonetoanother,but thissimple framework is a useful basis to describe what manyusers do. Raising awareness of social media can be doneby creating interesting, attractive, and relevant content.16

Rogers17 points out, for example, that innovation is moreikely to be accepted if the value of the material is clear tootential users. Good user-interface design produces ac-essible and universally usable applications that enableolitary reading or social interactions that meet the needsf diverse user populations.15,18

Reader, Contributor, Collaborator, and LeaderThe most understandable motivation for people to readuser-generated content is that they personally benefıtfrom doing so. A critical mass of new content18 and usernteraction that engages but does not overwhelm19,20

helps to entice people to return regularly. A contributionis an individual act that adds to a larger communal effort.Individual contributions can bring substantial benefıts toall participants, even though there is no direct communi-cation between individuals.Collaboration involves two or more contributors

working together to create something or share informa-tion.21 An essential element in this process is the devel-pment of a common ground—that is, mutual under-tanding, shared beliefs, and assumptions.22 Trust andempathy play a large role in encouraging people to workand play together online just as they do offline.23,24 Peo-ple who trust each other often do so because they see

Figure 4. The Reader-to-Lmedia participation has foand then more frequen(3) collaborators who palonger-term efforts. A smadeal with problems, menaisnet.org/thci/vol1/iss1

similarities between themselves and the other people,25 p

so they encourage each other to participate.26 Designersf the patient support community PatientsLikeMe havexplicitly used this knowledge in their site design. Theyake it easy for patients to fınd others like them in termsf gender, age, medical problems, and so forth. Everyonean fınd a picture on the homepage that helps them toınd similar others, which can lead to collaborations ontories and exchanges of helpful tips for dealing with aealth problem.15,27

While individual contributions and group collabora-tions are the most visible aspects of social media partici-pation, every social systemmust have some way of estab-lishing community norms and explicit policies if it is tosurvive.28,29 Leaders tend to synthesize discussions andrguments that they then articulate for others.30 Leader-hip is a higher calling to which only a small fraction ofeaders, contributors, and collaborators aspire.15

A Research AgendaThe Reader-to-Leader Framework, characterizing theevolution from reader, to contributor, to collaborator,and fınally, to leader, is a simplifıed but helpful model ofreality. Synthesizing and analyzing this large body of re-search helps justify the claim for the Reader-to-LeaderFramework, but it still needs empirical testing in a varietyof contexts. Other frameworks, such as Li and Bernoff’s31

social technographic profıling and Porter’s32 funnel offerlternative models that deserve attention. They strive todentify rudimentary metrics that are essential for study-ng the success of design and social interventions forncouraging participation. An important component ofcientifıc research is the development of appropriatemet-ics in order to gauge progress, identify problems, and

r Framework suggests that the typical path for socialtages: (1) reading, followed by (2) modest contributionsontributions. Eventually some contributors becomepate in discussions of future directions and take onmber of participants become (4) leaders who set policy,ewcomers, and inspire all forms of participation (aisel..

eadeur st crticill nu

tor n

redict future performance. While some social scientists

www.ajpm-online.net

uwe

Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197 S191

engage in small-scale controlled experimentation withdozens of users or groups, the capacity to perform large-scale interventions with thousands of users opens up newopportunities for research. These intervention methodswere developed by web companies to study the impact ofpage layouts, product presentation graphics, or specialoffers on customer purchasing patterns, but they can beeasily applied to technology-mediated social participa-tion projects.33

Planned and natural interventions are forms of socialscience case studies that, with replication, can accumulateevidence in support of predictive hypotheses.34,35 Prog-ress in providing personal health information to patientscan be accelerated by innovative social-networking strat-egies, more effective search capabilities, and improveduser-interface design.15

Innovative VisualizationsComputerized PHRs pose tremendous problems to sys-tem developers. As infrastructure and privacy issues aredealt more realistically, patients will become increasinglycomfortable in usingweb-based services. Poorly designeduser interfaces that require horizontal scrolling, clumsysearches, endless menus, and chaotic layouts will eventu-ally be replaced by compact designs with clear layouts.Guidelines documents and usability testing can promoteconsistent and comprehensive systems that provide goodservice to consumers, their professional healthcare pro-viders, and clinical researchers whomay discover impor-tant patterns with these large data sets. Techniques arebeing developed to summarize, fılter, and present volu-minous information, enabling consumers and profes-sionals to make important discoveries.36

Personal health records (PHRs) contain a wealth ofinformation, but can be challenging to analyze becausecategoric event data, such as complaints, diagnoses, andtreatments, are outside the scope of standard statisticaltools. However, current research efforts are moving be-yond numeric data and single-record visualization tech-niques, thus enabling users to discover patterns of cate-goric events across multiple records.37

In a past project for theMarylandDepartment of Juve-nile Services, the University of Maryland‘s Human–Computer Interaction Lab developed a new techniquecalled LifeLines (www.cs.umd.edu/hcil/lifelines) to visu-alize personal history records that will also enable ad-vanced searching and fıltering capabilities. A one-screenoverview of the record using timelines provides directaccess to the data. LifeLines can reduce the chances ofmissing information, facilitate the spotting of anomalies

and trends, streamline the access to details, and still

May 2011

remain simple and able to be tailored to variousapplications.36

Lifelines238,39 is an interactive tool (www.cs.umd.edu/hcil/lifelines2) for visualizing temporal categoric dataacrossmultiple records (Figure 5). The goal of the projectis to enable discovery and exploration of patterns acrossthese records to support hypothesis generation, and fınd-ing cause-and-effect relationships in a population—tasksfırst motivated by electronic health records. Lifelines2provides a simple set of operators to allow users to ma-nipulate multiple records simultaneously to understandchronologic relationships across records. The alignmentoperator forces every record to be aligned by a certainfeature (e.g., 3rd Heart Attack) so the events that occurprior to and after the feature can be compared easily. Therank operator sorts patient records according to a featuresuch as the number of heart attacks, and the fılter removesirrelevant patient records, such as those who never expe-rienced angina. In addition, analysts can use temporalsummaries to view distribution of multiple event typesover time. Temporal summaries enable users to see theaverage time between pairs of events and allow multiplegroups of records to be compared.37

Accessible eHealth InformationHealth care is being restructured to take advantage oftechnology’s promise to improve quality of care, delivercost effıciencies, speed information exchange, and buildconsumer health literacy. However, missing from thenational efforts toward pervasive availability of HIT forpatients, families, doctors, and healthcare facilities is aprogrammatic and policy-based effort to ensure that peo-ple with disabilities are able to participate equally in allthe opportunities that newHealth 2.0 networks and toolshave to offer and be able to independently access andmanage their personal healthcare information, or use theemerging tools as healthcare workers with disabilities.The needs of people with disabilities—as patients,

caregivers, and professionals—must be addressed in thedevelopment of HIT requirements for information de-sign, exchange, and display within both public and pri-vate products and services, and the standards governingHIT. Although existing accessibility requirements re-quire federal agencies to purchase or deploy electronicand information technologies that can be used by peoplewith disabilities, there are numerous documented casesof lack of compliance, oversight, and enforcement of thesection 508 rules.40 Efforts to address the existing gaps arenderway at the White House and at the CIO Council,hose members are the chief information offıcers of fed-ral executive agencies and departments.41 In the mean-

time, numerous HIT standards and certifıcations are be-

l5

d.ed

S192 Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197

ing approved and deployed and new HIT products andservices are being used bymajor healthcare providers andemployers—but do not include reference to usability andaccessibility requirements of people with disabilities.Organizations42,43 that research and develop technol-

ogies, tools, and policies to advance technology equity forpeople with disabilities are actively seeking to promoteattention to these challenges, secure resources to addressthem, and influence implementation of accessible prod-ucts and services by public and private technology ven-dors. The goal is to encourage development and adoptionof interoperable standards, specifıcations, systems, andservices that can identify, fılter, and present content inways that are meaningful to people with a wide variety ofspecifıc needs.In order to achieve these goals, public- and private-sector

partners and collaborators need to apply their resources to

Figure 5. LifeLines2 shows each person’s history as a seblood tests and radiology contrast injections. A researchdays after the injection of radiology contrast (www.cs.um

helpmakeadvancements in accessible andassistive technol-

ogies in the HIT sector. Specifıc needs and requirementsinclude design and development of public and privatehealth-related websites according to W3C Web ContentAccessibility Guidelines,44 design and development of pub-ic and private health-related software according to section08 guidelines,45 and, for online media, addition of equiva-lents to audio for deaf and hard-of-hearing people (e.g.,closed captioning or synchronized text) and equivalents tovisuals for blind and visually impaired people (e.g., narrateddescriptions).46,47

The Food and Drug Administration48 and the CDC49

are two of many healthcare-related federal agencies thathave already made great strides in ensuring that theirelectronic media are fully inclusive. The initial stepsamong a few key federal agencies toward full HIT inclu-sion for people with disabilities need to be matched bysimilar efforts across the board, starting with an explicit

time-stamped events, in this case, results of creatininestudying the pattern of creatinine-high results within 14

u/hcil/lifelines2).

t ofer is

set of policies, standards, and tools that can be guided by

www.ajpm-online.net

t(maai

mIeimnm

uic

Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197 S193

theOffıce of theNational Coordinator forHITwithin theDHHS.50 Accessibility requirements must be included inhe certifıcation process for electronic health recordsEHRs) and PHRs. In this way, as the use of HIT becomesore widely adopted and as government funding affectsdoption, accessibility will become a prerequisite as wells an assumed aspect of the design, development, andmplementation of this- and next-generation HIT.

Usable eHealth InformationThe NCI is the U.S. federal government’s principalagency for cancer research and training and is responsiblefor the incorporation of state-of-the-art cancer treat-ments into clinical practice.51 The NCI has invested sub-stantially in new applications and technologies to supportand achieve its goals. However, as cautioned by the IOM,creating health technology tools and support systems willbe valuable only to the extent that they are engineered tobe useful and usable to the researchers, providers, andconsumers expected to use them.52,53 The costs ofmakingistakes and not “getting it right” can be prohibitive.nadequate software development projects cost the U.S.conomy about $30 billion per year, and once a system isn development, correcting a problem costs ten times asuch as fıxing it in design. The cost in missed opportu-ity for scientifıc discovery, or to individuals’ lives fromalfunctioning HIT, is incalculable.54

Evidence-based decision making is integral to mini-mizing mistakes and pushing out poorly executed tech-nologies that fail to be adopted. During his White HouseForum on Health Reform on March 5, 2009, PresidentObama said, “I think it is so important that all of us makedecisions . . . based on evidence and data and what works. . . . But that requires us to actually look at the evidence.”Dr. Harold Varmus, the newly appointed Director ofNCI, reiterated this point at his fırst NCI town hall meet-ing on July 12, 2010, by saying, “There is a unifyingprinciple, one I will adhere to extremely stringently: Ev-erything that we do, and everything that we say, will bebased on evidence.”The NCI has given great credence to this principle for

evidence-based decisionmaking by creating and fundingan internal offıce that is dedicated to providing in-housecommunication and informatics research expertise to itsoffıces and divisions in order to best understand its audi-ences’ needs and avoid costly developmentmistakes. Thisoffıce, the Offıce of Market Research and Evaluation(OMRE), provides a unique model within the federalgovernment that relies on methods and theories derivedfrom psychology, human factors, communication, andinformatics engineering. OMRE aims to eliminate the

problems and ineffıciencies described above by providing

May 2011

ongoing evidence on audiences’ needs, their reactions todissemination efforts, and their ability to use the technol-ogies developed to assist them in treating cancer.Specifıcally, OMRE works with others at NCI to con-

duct the following types of research: (1) formative orconsumer research to ensure an understanding of theinformation and other resource needs of people and or-ganizations that can benefıt from the NCI informationresources, and how best to deliver those resources;(2) marketplace research, also commonly referred to ascompetitive analysis, to identify gaps in information andother resources that NCI should fıll; (3) information de-sign research to ensure that the NCI’s information prod-ucts and services have maximum value to their intendedtarget audience; (4) user-centered informatics researchwith regard to interactive and online information prod-ucts, to ensure that the information products being devel-oped are effective in conveying their intended content;(5) process-tracking research to document the success, orlack therefore, of efforts to disseminate information re-sources, including web-analytics; and (6) evaluation re-search to assess the impact of NCI’s disseminationactivities.The OMRE serves a model within the U.S. federal

government system. One example that successfully ap-plies themultiple methodologies within OMRE to ensurethe development of useful and usable technologies is itscontribution to the development of caMATCH, orBreastCancerTrials.org, an online mechanism thatmatches the patients’ actual medical history record toinclusion and exclusion criteria in clinical trials. Theproject started with a needs assessment of women withbreast cancer in how to access clinical trial informationonline, leading to the development of a testable web-based tool to assist women with this task,55 and thenltimately an evaluation of two versions of the tool todentify the impact of the tool on women with breastancer while using it.56

Ensuring Usability and AccessibilityAt the end of the day, no one can be satisfıedwith a systemin which the system itself is a contributing factor to anerror in patient treatment. Therefore the only two thingsthat should matter are how these EHR systems promotethe ability of medical practitioners to perform theirroutine tasks and deliver high-quality medical services totheir patients and ensuring that the user-interface designsof these systems do not lead to critical errors. To achievethese objectives, systems must fırst be usable and accessi-ble by practitioners, patients, and other stakeholders. Themost critical component of an evaluation criterion is a

test of actual user performance that measures successful

ctiasDnwfmb

cp

wtdtsemtm

S194 Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197

task completion, accuracy, and an acceptably low fre-quency of critical errors. The way to ensure a truly mean-ingful adoption and use of HIT is to defıne objectiveusability standards and adhere to them through estab-lished test methods.To promote adoption andusage ofHIT, theDHHSwill

soon be offering incentives for doctors and hospitals touse information technology and integrate it in a mean-ingful way into their healthcare delivery processes. Us-ability is a key factor in the meaningful use of thesesystems, because functionality is for naught if not usedeffectively, effıciently, and safely by its human users. Theultimate goal of this work is the detailed specifıcation ofan objective, valid, and repeatable procedure for measur-ing and evaluating the usability of HIT systems.

Designing a Health Information TechnologyUsability Certification ProcessMethods of evaluation based principally on how much amedical practitioner likes using a system are irrelevant toperformance-based certifıcation. In the world where hightechnology systems meet the time-honored medical pro-fession, there is no room for user-interface designs andevaluation methods that operate on the margin of theproblems we face today. Therefore the hope is to achievesystems that will be certifıed for a medical purpose thatallow medical practitioners to deliver the right treatmentto the right patient at the right time and support thehealth or recovery of that patient; usability supporting themedical practitioner in meeting the goal listed above;accessibility of user interfaces supporting those whomight be “star” medical practitioners, were it not for thelack of a technology supporting a disability forwhich theyare challenged; and fınally, resolution of HIT-related dis-parities for the lower sociodemographics, people withdisabilities, those with English as their second language,and those who are simply in stress and cannot performadequately. Against these goals, the only thing that mat-ters is that there are standards for certifıcation and estab-lished test methods against the standards.There is a well-established user-centered design57 pro-

ess defıned that can make HIT systems more usableoday. For the usability test during this process, the focuss exclusively on the validity and methodology of the testnd not on the test results. Using design guidelines, re-earchers generally focus on what is happening, not how.esign guidelines tend to be component-specifıc and doot deal with how the components combine to supportorkflow. There are many ways to measure human per-ormance based on the different usability tests. Perfor-ance benchmarks are created as a goal for the users

eing tested. These tests are done by specifying tasks to be m

ompleted and testing protocols. The systems will eitherass or fail based on the benchmark criteria.

Stages for Certification Developmentand DesignExtensive human factors research needs to be conductedto establish pass/fail criteria and standards for usabilitycertifıcation. This research will result in the developmentof test methods that include specifıc protocols. The nextstep is to test the methods and protocols for validity andrepeatability. Once it has been established that the proto-cols are valid and repeatable after the fırst experimentwithin a single test environment, the protocols need to beapplied in other test environments (using different tes-ters, different labs, and different test participants repre-senting end users) to ensure that they as well producerepeatable results across all test environments. After thiswork has been completed, it is established that the strictpass/fail criteria for certifıcation using these test proto-cols are valid for use by accredited test entities.Timing (or task completion time) can be an important

criterion for effıciency; however, timing might not be anappropriate criterion for a pass/fail performance bench-mark (unless failing to complete a task within a specifıctime period is considered an error). Usability testingtakes place in a lab settingwhere userswho test the systemmay behave artifıcially because they know that they arenot performing tasks in the real world. Consequently,they may take extra time for exploratory activities inperforming their tasks. Essentially, themost critical com-ponent of a pass/fail criterion is one that tests actual userperformance and measures accuracy, successful taskcompletion, and freedom from critical errors.As fındings accumulate from research efforts, a usabil-

ity component will gradually be incorporated into themeaningful-use criteria in stages. In the short-term viewof the testing, the focus is on the process. This stagerequiresmanufacturers to submit a report of a summativeusability test using a standard industry format, custom-ized for the healthcare context, to show evidence of prod-uct usability and the presence of usability engineeringprocesses. Manufacturers must also submit a list of userrequirements as defıned by the Theofanos report58 inhich usability requirements consist of three parts: con-ext of use, measures and metrics, and the testing proce-ure. In themedium term, focus is on design features andhe process. This stage provides voluntary usability de-ign guidelines that manufacturers can use for self-valuation of their systems, in addition to requiring sub-ission of the usability report and user requirements. In

he long term, design features, the process, and perfor-ance are the main focus. In this stage, specifıc and

easurable usability objectives are adopted for HIT. A

www.ajpm-online.net

etastccdct

il

Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197 S195

formal testing procedure is instituted that involves theapplication of performance-based pass/fail usabilitycriteria.One should start out by specifying the type of technol-

ogy being studied, the users that are testing, and the tasksthat will be completed in the study. Human performancemeasures and their related benchmarks must be clearlydefıned in relation to the tasks being studied. Usabilitymeasurement is conditioned by the tasks carried out, theusers involved, and the settings in which the product willbe used. Since these factors will be different for eachproduct, customization is of utmost importance in theusability testing.

ConclusionConsumer health informatics discussed, as above, givesus various ideas for approach to its success. These view-points are considered in this section to summarize thewhole review. The personas are considered to help ad-dress the challenge for most technology projects. Perso-nas are one tool to help picture the audience for anysystem in away that can drive the design process from theuser perspective.9 Inference can be drawn from the Read-r-to-Leader Framework to understand human motiva-ion to participate in social media. This framework en-bles them to improve user-interface design and socialupport for projects. These improvements could reducehe number of failed projects while accelerating the appli-ations of social media in health care, disaster response,ommunity safety, and more users. Good user-interfaceesign produces accessible and universally usable appli-ations that enable solitary reading or social interactionshat meet the needs of diverse user populations.15,18

Progress in providing personal health information topatients can be accelerated by innovative social network-ing strategies, more effective search capabilities, and im-proved user-interface design.15 LifeLines, the Universityof Maryland’s Human–Computer Interaction Lab tech-nique developed to visualize personal history records isan example to be studied in depth. The most criticalcomponent of an evaluation criterion is a test of actualuser performance that measures successful task comple-tion, accuracy, and an absence of critical errors. The wayto ensure the trulymeaningful adoption and use ofHIT isto defıne objective usability standards and adhere to themthrough established test methods. Designing withoutuser and task analysis often produces products that ig-nore accessibility requirements,10 fail to address the vari-ations in roles, and do not provide the right balance ofusability requirements.11 Thismeans thinking aboutHITn the context of the larger social ecosystem, providing

inks between online systems and human support. If us-

May 2011

ability and accessibility in the consumer context are nottaken into account, then there is no way that the fınalproducts will meet usability and accessibility goals forthe broadest range of people with the widest range ofcapabilities.Accessibility requirements must be included in the

certifıcation process for EHRs andPHRs. Evidence-baseddecision making is integral to minimizing mistakes andpushing out poorly executed technologies that fail to beadopted. A unique model that relies on methods andtheories derived from psychology, human factors, com-munication, and informatics engineering aims to elimi-nate the problems and ineffıciencies described above byproviding ongoing evidence on audiences’ needs. TheOMRE research approach is noteworthy and a pattern tobe followed for effıcacy of the results. The way to ensurethe truly meaningful adoption and the use of HITis to defıne objective usability standards and adhere tothem through established test methods. Design guide-lines tend to be component-specifıc and do not deal withhow the components combine to support workflow. Us-ability requirements consist of three parts: context of use,measures and metrics, and the testing procedure. A for-mal testing procedure needs to be instituted that involvesthe application of performance-based pass/fail usabilitycriteria. Human performance measures and their relatedbenchmarks must be clearly defıned in relation to thetasks being studied. All these would lead to a successfulconsumer health informatics scenario.

BS acknowledges National Science Foundation IIS-0956571,ONC SHARP grant on Cognitive Information Design and Vi-sualization for the National Center for Cognitive Informaticsand Decision Making in Healthcare, and the Two Workshops:Technology-Mediated Social Participation and NIH–NationalCancer Institute (CA147489), Interactive Exploration of Tem-poral Patterns in Electronic Health Records.This document has been prepared by the National Institute

of Standards and Technology (NIST) and describes researchregarding the usability of electronic health records. It does notrepresent a consensus view or recommendation from NIST,nor does it represent any policy positions of NIST.Certain commercial entities, equipment, or material may

be identifıed in the document in order to describe an exper-imental procedure or concept adequately. Such identifıca-tion is not intended to imply recommendation or endorse-ment by NIST; nor is it intended to imply that these entities,materials, or equipment are necessarily the best available forthe purpose.WQ has consulted for the National Cancer Institute and the

National Institute of Standards and Technology.Publication of this article was supported by the National

Institutes of Health.

1

1

1

1

1

1

1

11

1

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

4

4

S196 Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197

No fınancial disclosures were reported by the authors of thispaper.

References1. Pew Internet. Cancer 2.0: a summary of the recent research. pewrsr.ch/

Cancer20.2. WQusability.What dowemeanby user friendly?Meeting expectations

for e-health applications. www.wqusability.com/articles/consumer-health-usability.html.

3. ISO/IEC. International standard 9241-151: ergonomic requirementsfor offıce work with visual display terminals (VDTs), Part 151: Guid-ance on usability. ISO/IEC. 2008.

4. ISO/IEC. International standard 9241-20: ergonomics of human–sys-tem interaction, Part 20: Accessibility guidelines for information/com-munication technology (ICT) equipment and services. ISO/IEC. 2008.

5. Mayhew DJ. The usability engineering life cycle: a practitioner’s hand-book for user interface design. Morgan-Kaufmann, 1999.

6. Stone D, Jarrett C, Woodroffe, M, Minocha S. User interface designand evaluation. Morgan-Kaufmann, 2005.

7. Rosson MB, Carroll JM. Usability engineering: scenario-based devel-opment of human–computer interaction. CA: Morgan-Kaufmann,2002.

8. ISO/IEC. International standard 9241-210: Ergonomics of human–system interaction, Part 210: Human-centered design for interactivesystems. ISOIEC. 2010.

9. Adlin T, Pruitt J. The persona lifecycle. Morgan-Kaufmann, 2006.0. Swierenga SJ. Incorporating web accessibility into the design process.

In: Righi C, James J, eds. The user-centered design casebook. Morgan-Kaufmann, 2007.

1. Quesenbery W. Dimensions of usability. In: Albers M, Mazur B, eds.Content and complexity: information design in technical communica-tion. Erlbaum, 2003.

2. Shneiderman B, Plaisant C. Designing the user interface: strategies foreffective human–computer interaction 5th ed. Boston MA: Addison-Wesley, 2009.

3. YeY, FischerG.Designing for participation in socio-technical softwaresystems. Proceedings of 4th International Conference on UniversalAccess in Human–Computer Interaction 2007;pt I C:312–21.

4. Moore TD, ServaMA.Understandingmembermotivation for contrib-uting to different types of virtual communities: a proposed framework.Proceedings of ACM SIGMIS-Conference on Personnel Research.2007:153–8.

5. Preece J, Shneiderman B. The Reader-to-Leader Framework: motivat-ing technology-mediated social participation. AIS Transactions onHuman–Computer Interaction. 2009;1(1):13–32. aisel.aisnet.org/thci/vol1/iss1/5/.

6. Kim AJ. Community building on the web: secret strategies for success-ful online communities. Berkeley CA: Peachpit Press, 2000.

7. Rogers E. Diffusion of innovations. 5th ed. NewYork: Free Press, 2003.8. Rainie L, Tancer B. Wikipedia: when in doubt, multitudes seek it out.

Pew Internet & American Life Project. Pewresearch.org/pubs/460/wikipedia.

9. Cheng R, Vassileva J. User and community adaptive rewards mecha-nisms for sustainable online communities. In: Ardissonho L, Brna P,Mitrovic A, eds. LNAI 3538. Berlin, Heidelberg: Springer-Verlag, 2005.

0. Cheng R, Vassileva J. Design and evaluation of an adaptive incentivemechanism for sustained educational online communities. UserModelUser-adapt Interact. 2006; 16(3–4):321–48.

1. Denning PJ, Yaholkovsky P. Getting to ‘we’: solidarity, not software,generates collaboration. Commun ACM 2008;51(4):19–24.

2. Convertino G, Mentis HM, Rosson MB, Carroll JM, Slavkovic A,

GanoeCH.Articulating common ground in cooperative work: content

and process. Proceedings of 26th Annual ACMConference onHumanFactors in Computing Systems. 2008:1637–46.

3. Preece J. Empathic communities: balancing emotional and factualcommunication. Interact Comput 1999;12(1):63–77.

4. Maloney-Krichmar D, Preece J. A multilevel analysis of sociability,usability and community dynamics in an online health community.ACM Transactions on Computer Human Interaction 2005;12(2):1–32.

5. TorreyC, BurkM, LeeM,DeyA, Fussell S, Kiesler S. Connected giving:ordinary people coordinating disaster relief on the Internet. Proceed-ings of 40th Annual Hawaii International Conference on System Sci-ences (HICSS). 2007.

6. Wu JJ, Tsang ASL. Factors affecting members’ trust belief and behav-iour intention in virtual communities. Behav Inf Technol 2008;27(2):115–25.

7. CobbNK, GrahamAL, AbramsDB, Social network structure of a largeonline community for smoking cessation. Am J Public Health2010;100(7):1282–9.

8. Butler B, Joyce E, Pike J. Don’t look now, but we’ve created a bureau-cracy: the nature and roles of policies and rules inWikipedia. Proceed-ings of 26th Annual ACM Conference on Human Factors in Comput-ing Systems 2008:1101–10.

9. Burke M, Kraut R. Taking up the mop: identifying future Wikipediaadministrators. Proceedings of 26th Annual ACM Conference on Hu-man Factors in Computing Systems 2008:3441–6.

0. Cassell J, Huffaker D, Tversky D, Ferriman K. The language of onlineleadership: gender and young engagement on the Internet.DevPsychol2006;42(3):436–49.

1. Li C, Bernoff J. Groundswell: winning in a world transformed by socialtechnologies. Boston MA: Harvard Business Review, 2008.

2. Porter J. Designing for the social web. Berkeley CA: New Riders,2008.

3. Kohavi R, Henne RM, Sommerfıeld D. Practical guide to controlledexperiments on the web: listen to your customers not to the HiPPO.Proceedings of ACM International Conference on Knowledge Dis-covery and Data Mining (KDD2007) 2007:959–67.

4. Yin RK. Case study research: design and methods. 3rd ed. ThousandOaks CA: Sage, 2003.

5. Konstan J, Chen Y. Online fıeld experiments: lessons from Communi-tyLab. Proceedings of Third International Conference on e-Social Sci-ence Ann Arbor MI: 2007.

6. Shneiderman B. LifeLines for visualizing patient records. www.cs.umd.edu/hcil/lifelines/.

7. ShneidermanB. Lifelines2.University ofMarylandHuman–ComputerInteraction Lab. 2006. www.cs.umd.edu/hcil/lifelines2/.

8. Wang T, Plaisant C, Quinn A, Stanchak R, Shneiderman B, Murphy S.Aligning temporal data by sentinel events: discovering patterns inelectronic health records. Proceedings of ACM CHI2008 Human Fac-tors in Computing Systems Conference New York: 2008.

9. Wang TD, Plaisant C, Shneiderman B, et al. Temporal summaries:supporting temporal categorical searching, aggregation and compari-son. IEEE Trans Vis Comput Graph 2009;15(6):1049–56.

0. Section 508. 1998 amendment to section 508 of the rehabilitation act.www.section508.gov/index.cfm?FuseAction�Content&ID�14.

1. Chief Information Offıcers Council. IT accessibility committee. www.cio.gov/pages.cfm/page/IT-Accessibility-Committee.

42. The Carl and Ruth Shapiro Family National Center for AccessibleMedia. ncam.wgbh.org/.

43. University of Wisconsin-Madison. Trace Research and DevelopmentCenter. trace.wisc.edu/.

44. W3C.Web Content Accessibility GuidelinesWorking Group (WCAGWG). www.w3.org/WAI/GL/.

45. U.S. Access Board. Section 508: Electronic and Information Technol-

ogy. www.access-board.gov/508.htm.

www.ajpm-online.net

Goldberg et al / Am J Prev Med 2011;40(5S2):S187–S197 S197

46. The Carl and Ruth Shapiro Family National Center for AccessibleMedia.Web�multimedia. ncam.wgbh.org/invent_build/web_multimedia.

47. W3C. Timed Text Markup Language (TTML) 1.0. www.w3.org/TR/ttaf1-dfxp/.

48. FDA. Internet Accessibility: a Commitment to Accessibility. www.fda.gov/AboutFDA/AboutThisWebsite/Accessibility/default.htm.

49. CDC. CDC-TV. www.cdc.gov/cdctv/.50. DHHS. User-Centered Design and Usability Testing. www.usability.

gov/basics/uc_design_testing/index.html.51. Bogner M. Human error in medicine. New Jersey: Lawrence Erlbaum

Associates, 1994.52. IOM. Crossing the quality chasm: a new health system for the 21st

century. Washington: National Academies Press, 2001.53. Wears RL, Berg M. Computer technology and clinical work: still wait-

ing for God

May 2011

54. Landauer TK. The trouble with computers: usefulness, usability, andproductivity. Cambridge MA: MIT Press, 1996.

55. Atkinson NL, Massett HA,Mylks C, Hanna B, DeeringMJ, Hesse BW.User-centered research on breast cancer patient needs and preferenceson an Internet-based clinical trials matching system. J Med InternetRes 2007;9(2):e13.

56. Atkinson NL, Massett HA, Mylks C, et al. Assessing the impact ofuser-centered research on a clinical trial eHealthTool via a randomizedcontrol study. J AmMed Inform Assoc Forthcoming.

57. NCI. NCI Mission Statement. www.cancer.gov/aboutnci/overview/mission.

58. Theofanos M. NISTIR 7432: Common industry specifıcation for us-ability requirements. Industry Usability Reporting Project Working

ot. JAMA 2005;293(10):1261–3. Group; 2007.

Did you know?The latest AJPM news is available online.

Visit www.ajpm-online.net to see the new Announcement section on the homepage.