The NIH Toolbox: An The NIH Toolbox: An Instrument for Instrument for Assessment Assessment

of Neurological and of Neurological and Behavioral FunctionBehavioral Function

DavidDavid TulskyTulsky - Kessler (KMRREC) & UMDNJ-NJMS- Kessler (KMRREC) & UMDNJ-NJMSSandraSandra Weintraub Weintraub - Northwestern University Medical School- Northwestern University Medical SchoolSureyyaSureyya DikmenDikmen - University of Washington- University of WashingtonRichard HavlikRichard Havlik - Westat- WestatDavid BlitzDavid Blitz - Evanston Northwestern Healthcare- Evanston Northwestern HealthcareRobertRobert HeatonHeaton - University of California, San Diego- University of California, San DiegoBeth BoroshBeth Borosh - Northwestern University Medical School- Northwestern University Medical SchoolNancy ChiaravallotiNancy Chiaravalloti - Kessler (KMRREC) & UMDNJ-NJMS - Kessler (KMRREC) & UMDNJ-NJMS Amanda O’BrienAmanda O’Brien - Kessler (KMRREC) & UMDNJ-NJMS - Kessler (KMRREC) & UMDNJ-NJMS Joni MachamerJoni Machamer – University of Washington– University of WashingtonRichard GershonRichard Gershon – Evanston Northwestern Healthcare– Evanston Northwestern HealthcareMolly WagsterMolly Wagster – NIH (NIA)– NIH (NIA)

ContributorsContributors

NIHNIH Toolbox for Neurological & Toolbox for Neurological & Behavioral FunctionBehavioral Function

There is little uniformity among the measures used to assess There is little uniformity among the measures used to assess neurological function and behavioral health. neurological function and behavioral health.

The NIH Toolbox initiative seeks to assemble brief, The NIH Toolbox initiative seeks to assemble brief, comprehensive assessment tools that will be useful to comprehensive assessment tools that will be useful to clinicians and researchers in a variety of settingsclinicians and researchers in a variety of settings

NIH Toolbox places particular emphasis on measuring NIH Toolbox places particular emphasis on measuring outcomes in longitudinal epidemiologic studies and outcomes in longitudinal epidemiologic studies and prevention or intervention trials across the lifespan. prevention or intervention trials across the lifespan.

Currently, standardized measures of cognitive, sensory, Currently, standardized measures of cognitive, sensory, motor, and emotional functioning are not included.motor, and emotional functioning are not included.

NIH Toolbox for Assessment of NIH Toolbox for Assessment of Neurological and Behavioral FunctionNeurological and Behavioral Function

If there were unified/integrated measures of multiple If there were unified/integrated measures of multiple indicators (cognitive, emotional, motor, sensory) of indicators (cognitive, emotional, motor, sensory) of neural and behavioral health functioning neural and behavioral health functioning …… . .

Could be used as a form of Could be used as a form of ““common currencycommon currency”” across diverse study designs and populations in across diverse study designs and populations in large cohort studieslarge cohort studies

Would maximize yield from large, expensive studies Would maximize yield from large, expensive studies with minimal increment in subject burden and costwith minimal increment in subject burden and cost

The NIH Blueprint for The NIH Blueprint for Neuroscience ResearchNeuroscience Research

Aims to develop new tools, Aims to develop new tools, resources, and training resources, and training opportunit ies to accelerate opportunit ies to accelerate the pace of discovery in the pace of discovery in neuroscience research.neuroscience research.

•Initiated in 2004Initiated in 2004•Designed to enhance collaboration among the NIH Office of the Designed to enhance collaboration among the NIH Office of the Director and 15 NIH institutes and centers (ICs) that support Director and 15 NIH institutes and centers (ICs) that support research on the nervous system. research on the nervous system. •Pooling resources and expertise, the Blueprint confronts Pooling resources and expertise, the Blueprint confronts challenges that transcend any single institute or center and serves challenges that transcend any single institute or center and serves the entire neuroscience community.the entire neuroscience community.

http://www.neuroscienceblueprint.nih.gov/http://www.neuroscienceblueprint.nih.gov/

SponsorshipSponsorship Funding is provided by 15 NIH agenciesFunding is provided by 15 NIH agencies The Toolbox effort is overseen by the National Institute of Aging The Toolbox effort is overseen by the National Institute of Aging

(Molly Wagster)(Molly Wagster) Funding is through a cost reimbursement contractFunding is through a cost reimbursement contract   This project has been funded in whole or in part with Federal This project has been funded in whole or in part with Federal

funds from the National Institute on Aging, National Institutes of funds from the National Institute on Aging, National Institutes of Health, under Contract No.:  HHS-N-260-2006-00007-CHealth, under Contract No.:  HHS-N-260-2006-00007-C

NIH Project Team MembersNIH Project Team Members

National Institute on Aging (NIA)National Institute on Aging (NIA)National Institute on Mental Health (NIMH)National Institute on Mental Health (NIMH)National Institute on Drug Abuse (NIDA)National Institute on Drug Abuse (NIDA)National Institute on Neurological Disorders & Stroke (NINDS)National Institute on Neurological Disorders & Stroke (NINDS)National Institute on Environmental Health Sciences (NIEHS)National Institute on Environmental Health Sciences (NIEHS)National Institute on Deafness and Other Communicative Disorders (NIDCD)National Institute on Deafness and Other Communicative Disorders (NIDCD)National Eye Institute (NEI)National Eye Institute (NEI)National Institute on Child Health and Human Development (NICHD)National Institute on Child Health and Human Development (NICHD)National Institute of Nursing Research (NINR)National Institute of Nursing Research (NINR)National Institute on General Medical Sciences (NIGMS)National Institute on General Medical Sciences (NIGMS)Office of Behavioral and Social Sciences Research (OBSSR)Office of Behavioral and Social Sciences Research (OBSSR)National Center on Complementary and Alternative Medicine (NCCAM)National Center on Complementary and Alternative Medicine (NCCAM)National Institute on Alcohol Abuse and Alcoholism (NIAAA)National Institute on Alcohol Abuse and Alcoholism (NIAAA)National Center on Research Resources (NCRR)National Center on Research Resources (NCRR)

Eight Domain-SitesEight Domain-Sites

Northwestern University - ENHRINorthwestern University - ENHRI1.1. Center for Outcomes, Research and Education and Center for Outcomes, Research and Education and

Northwestern UniversityNorthwestern University2.2. Rehabilitation Institute of ChicagoRehabilitation Institute of Chicago3.3. Feinberg School of MedicineFeinberg School of Medicine4.4. Northwestern School of CommunicationsNorthwestern School of Communications

Primary CollaboratorsPrimary Collaborators1.1. University of WashingtonUniversity of Washington2.2. University of PittsburghUniversity of Pittsburgh3.3. UCLAUCLA4.4. Kessler Medical Rehabilitation Research and Education Kessler Medical Rehabilitation Research and Education

CenterCenter

PI: Richard C. Gershon, PhD

Four DomainsFour Domains

CognitionCognitionEmotionEmotionMotor FunctioningMotor FunctioningSensationSensation

Phase I: 12 MonthsPhase I: 12 Months

Expert Survey of criteria to select sub-domainsExpert Survey of criteria to select sub-domains Scientific Advisory Board (SAB) meetings to select Scientific Advisory Board (SAB) meetings to select

final criteria final criteria Identification of existing psychometric tests and Identification of existing psychometric tests and

measurement toolsmeasurement tools Interviews with a variety of scientists Interviews with a variety of scientists Identification of domains of functioningIdentification of domains of functioning Review of all potential testsReview of all potential tests Nomination of tests to use in toolbox.Nomination of tests to use in toolbox.

Phase II: 18 MonthsPhase II: 18 Months

Official approval of tests (from NIH).Official approval of tests (from NIH). Development of the new tests, items for Development of the new tests, items for

adults and children.adults and children. OMB approval of tests and itemsOMB approval of tests and items Pilot Testing of the tests and comparison Pilot Testing of the tests and comparison

against against ““gold standardsgold standards””

Phase III (30 months)Phase III (30 months)

Spanish translation Spanish translation Program of CAT administrationProgram of CAT administration Calibration & analysis of item bank based MeasuresCalibration & analysis of item bank based Measures Field testing/Normative study of toolbox Field testing/Normative study of toolbox Psychometric testingPsychometric testing

Cognition Team ProgressOctober 19, 2007

NIH Toolbox Meeting 10.18.07

Cognition TeamSENIOR SCIENTISTSSureyya Dikmen, University of Washington Nathan Fox, Duke University Robert Heaton, UCSD David Tulsky, KMRREC Sandy Weintraub, Northwestern University

JUNIOR SCIENTISTSBeth Borosh, Northwestern University Nancy Chiaravalloti, KMRRECJoni Machamer, University of WashingtonAmanda O’Brien, KMRREC

PROCESS/ORGANIZATIONDavid Blitz, ENH COREDick Havlik, WestatJerry Slotkin, ENH CORE

NIH Toolbox Meeting 10.18.07

MANDATES FOR COGNITION TEAMMANDATES FOR COGNITION TEAM

1. Cover 4-6 Subdomains2. Take no more than 30 minutes 3. Cover the full range of normal function4. Cover ages 3-855. Instrument should be free/minimal cost6. Use state of the art methods

May Meeting 2007 and Beyond

• Submitted initial literature review

• Subdomains proposed:

• Executive Function• Episodic Memory• Language• Processing Speed

FEEDBACK:

Executive Function is very important.Working Memory should be included as part of EF or separate, additional domain. Questions about Attention.Get verification from experts outside of the team.

May Meeting 2007 and Beyond

2. Recruited Expert Consultants

4. Split into focused subdomain working groups, weekly conference

6. July 2007 Meeting

RESPONSE

• Initial lists assembled by CORE

• NIH recommendations

• N. Fox recommendations

• Researchers and Clinicians

• Pediatric emphasis

Recruitment of Expert Consultants

AttentionSandy Weintraub, Northwestern UniversityNathan Fox, University of Maryland Koraly Perez-Edgar, George Mason UniversityFrank Zelko, Northwestern UniversityDick Havlik, Westat

Processing SpeedDavid Tulsky, KMRREC Noelle Carlozzi, KMRRECNancy Chiaravalloti, KMRRECTimothy Salthouse, University of VirginiaKeith Yeates, Ohio State UniversityAmanda O’Brien, KMRREC

Working MemoryDavid Tulsky, KMRRECNoelle Carlozzi, KMRRECNancy Chiaravalloti, KMRRECTimothy Salthouse, University of VirginiaKeith Yeates, Ohio State University

SUBDOMAIN WORKING GROUPS

SUBDOMAIN WORK GROUPS

Episodic MemorySureyya Dikmen, University of WashingtonPatricia Bauer, Duke UniversityGordon Chelune, University of UtahDean Delis, University of California, San DiegoJoni Machamer, University of Washington

Executive FunctionPhil Zelazo, University of Minnesota Adele Diamond, University of VancouverJoel Kramer, University of California, San FranciscoBeth Borosh, Northwestern University

LanguageSandy Weintraub, Northwestern UniversityJean Berko Gleason, Boston UniversityKathy Hirsch-Pasek, Temple UniversityRoberta Golinkof, University of DelawareBeth Borosh, Northwestern University

GENERAL COGNITIVE

Bob Heaton, University of California, San Diego

Ida Sue Baron, University of VirginiaDan Mungas, University of California, DavidEsther Strauss, University of VancouverJerry Sweet, Northwestern University

1. CORE Librarian Search, supplemented with recommendationsfrom initial RFI and Expert Interviews ~ 400

4. Grid of all instruments containing information about time toadminister, psychometric properties, Spanish translation, acceptable age range, or if no information

8. Grid divided into subdomains for working groups to cull

10. Initial instrument cut

Instrument Review Methods

INITIAL CRITERIA FOR CUTTING INITIAL CRITERIA FOR CUTTING INSTRUMENTSINSTRUMENTS

Self-Report, Proxy (for adults), Clinician-Rated

Commercial instruments (IP Rights/purchase required); but if an exemplar or gold standard, or available in the public domain, consider as a “Model”

Screening/Global measures (e.g., anything that only screens for impairment)

Disease- or Population-Specific (e.g., Alzheimer’s, Aged/Elderly, etc.)

Administration Time: over 20 minutes for Executive Function and Working Memory; 10 minutes for other subdomains, in most instances unless instrument had very high standards value

July 2007 Meeting

ATTENDEES: Beth BoroshNancy Chiaravalloti (P)Kevin Conway (NIH)Gordon CheluneDean DelisSureyya DikmenNathan FoxDick Havlik

NIH Phone: Ellen Witt, Emmeline Edwards, Claudia Moy

CORE STAFF: Blitz, Gershon, Nowinski, Lai, Swantek, Wang

Bob HeatonJoel KramerJennifer Manly (P)Amanda O’Brien (P)David TulskyTimothy SalthousePhil ZelazoFrank Zelko

• Clinical and experimental measures each have their positives and negatives

• Experimental: technologically advanced, measure more discrete behavioral components, validated against biological markers• Clinical: sensitive to a variety of disorders, more ecological validity

• Computer VS Paper-and-Pencil

FINAL PRODUCT SHOULD BE ACCEPTABLE TO RESEARCHERS AND CLINICIANS

July 2007 Meeting - Considerations

RECOMMENDED TESTS/PARADIGMSRECOMMENDED TESTS/PARADIGMS

ExperimentalExperimental ValidationValidation

1. NINDS Executive Function 1. NINDS Executive Function Toolbox (EXAMINER)Toolbox (EXAMINER)

2. Dimensional Change Card 2. Dimensional Change Card SortSort

3. Flanker3. Flanker

4. Trail Making Tests, Parts A 4. Trail Making Tests, Parts A & B& B

Set-shifting task Set-shifting task

Objective is for participant to match figures by either color or Objective is for participant to match figures by either color or shape depending on instruction that appears on screen. shape depending on instruction that appears on screen.

3 blocks of trials3 blocks of trials All colorAll color All shapeAll shape Random switching Random switching

Dimensional Change Card SortP. Zelazo, Nature Protocols, 2006

Shape Trial Color Trial

Flanker Test (P. Zelazo)

Standard Incongruent Standard Congruent

Reverse Congruent Reverse Incongruent

•Target fish appear flanked by other fish, user indicates direction target fish is going•Congruent, they are all going in the same direction; Incongruent, target is different direction than the flanker fish•Reverse condition, user must indicate the direction of the flanker fish (rather than the target fish).

PROCESSING SPEED

SPEED OF PERFORMING SIMPLE MENTALACTIVITIES

NUMBER OF ITEMS THAT CAN BE PROCESSEDIN A SET AMOUNT OF TIME

* Opportunity to combine across subdomains, using RT, time to complete measures from other tests

RECOMMENDED TESTS/PARADIGMSRECOMMENDED TESTS/PARADIGMS

RECOMMENDED TESTS/PARADIGMSRECOMMENDED TESTS/PARADIGMS

PROCESSING SPEED

ExperimentalExperimental ValidationValidation

1. Pattern Comparison1. Pattern Comparison

2. Symbol-Digit Modalities2. Symbol-Digit Modalities

4. WAIS-III Processing Speed 4. WAIS-III Processing Speed Index (Digit Symbol, Digit Index (Digit Symbol, Digit Search)Search)

Person is instructed to classify the pairs as Same (S) or Different (D) as quickly as possible (Salthouse, 1991)

WORKING MEMORYWORKING MEMORY

1) to process information across a series of tasks and modalities,

2) to hold the information in a short-term buffer,

3) to manipulate the information, and

4) to hold the products in the same short-term buffer

Components

RECOMMENDED TESTS/PARADIGMSRECOMMENDED TESTS/PARADIGMS

WORKING MEMORY

ExperimentalExperimental ValidationValidation

1. Complex Span (paradigm)1. Complex Span (paradigm)

2. Continuous Updating 2. Continuous Updating (paradigm)(paradigm)

3. PASAT3. PASAT

4. Letter Number Sequencing4. Letter Number Sequencing

Instrument Selection and Development PlanInstrument Selection and Development Plan

THREE-PRONGED APPROACHTHREE-PRONGED APPROACH

1. IDENTIFY CRITERION MEASURES FOR EACH SUBDOMAIN

2. IDENTIFY KEY EXPERIMENTAL PARADIGMS FOR EACH SUBDOMAIN

3. IDENTIFY EXISTING COMPUTER MODELS THAT COULD BE ADAPTED FOR TOOLBOX (E.G. ANAM)

DEVELOPMENT PLANDEVELOPMENT PLAN

TEST AGAINST VALIDATION MEASURES

Work with Epi-Biostats group to finalize sampling plan

Administer experimental tests we developed and ANAM, clinical measures, demographic information (age, education, employment status, SES, ethnicity, reading level, acculturation, etc.)

Do our experimental measures work?

Acceptable psychometric properties

Cover age span

Correlate with criterion measures

DEVELOPMENT PLAN - NO SUPRISESDEVELOPMENT PLAN - NO SUPRISES

RETAIN PERFORMERS

MAKE NECESSARY REVISIONS

MOVE TO STANDARDIZATION WITH FINAL BATTERY

CognitionCognition EmotionEmotion

Processing SpeedProcessing Speed MemoryMemory AttentionAttention Working MemoryWorking Memory Language/VerbalLanguage/Verbal Executive FunctioningExecutive Functioning

Rev. 4-07Rev. 4-07

Negative AffectNegative Affect Positive AffectPositive Affect Stress & CopingStress & Coping Social RelationshipsSocial Relationships

Motor FunctioningMotor Functioning SensorySensory

Locomotion (20 Ft Walk Test)Locomotion (20 Ft Walk Test) Endurance (6 min walk test)Endurance (6 min walk test) Balance (Single Leg Stance)Balance (Single Leg Stance) Dexterity (9-hole Peg test)Dexterity (9-hole Peg test) Upper Strength (grip strength Upper Strength (grip strength

dynamometry)dynamometry) Lower/Trunk Strength Lower/Trunk Strength

(Electronic Muscle Strength)(Electronic Muscle Strength)

VisionVisionHearingHearingSomatosensationSomatosensationOlfactionOlfactionTasteTaste

Toolbox Open MeetingToolbox Open Meeting