NIH ToolboxAssessment of Neurological and Behavioral Function

www.NIHToolbox.org

Richard Gershon, PI Cindy Nowinski, Scientific Director Julie Hook, NIH Toolbox Product Manager Maria Varela, Director of Technology And a cast of thousands!

The NIH Toolbox was developed under contract to the NIH Neuroscience Blueprint starting in 2006. The goal was to develop assessments of neural and behavioral health functioning for use in large cohort studies and clinical trials. The assessments could be used as a form of “common currency” across diverse study designs and populations and would maximize yield from large, expensive studies with minimal increment in subject burden and cost.

The NIH Toolbox was initially made available to grants funded by the American Recovery and Reinvestment Act of 2009. The measures were released to the general research community in 2012.

The iPad version of the NIH Toolbox was released in 2015. In 2018 alone, the iPad version was used to administer 600,000 NIH Toolbox assessments in English, and over 30,000 in Spanish.

Since 2012, more than 100 validation studies have evaluated the NIH Toolbox for use in numerous additional sub-populations including aging, cancer, TBI, and SCI.

ClinicalTrials.Gov lists the NIH Toolbox as being used in 165 clinical trials.

Results of research conducted using the NIH Toolbox have been published in 253 peer reviewed journal articles.

89%

7% 4%

909* TotalSubscriptions

893 NIH Toolbox

34 PROMIS68 NIH TB Spanish

*NIHTB Cebuano has 2 subscriptions; All data from AppFigures

TOTAL

997

s listed in order of total sales; Data from AppFigures

*Purchased in 21 countries

Downloaded in 58 countries

Categories#

Inquiries App Support cases (NIH Toolbox, En Espanol and PROMIS) 1127 Cognition access application/Cognition unlocks 1136

PROMIS, NIH Toolbox and NIH Toolbox en español: 14 releases in 2018

Implementation of participant age as alternative to DOB New versions of NIH Toolbox language instruments:

Picture Vocabulary and Oral Reading Implementation of Emotion Composite/Summary Scores Improvements to Assessment and Longitudinal Reports Integration with Customer Service Desk.com system Improvements for handling of large Assessment

Databases

Audit log of Participant and Assessment actions Deletion of Participants and Assessment data Rescoring when participant demographics change Database of App diagnostic data to support troubleshooting Additional administration data added to exports Adaptation of Apps to new iPad models’ screens Bug fixes and upgrades to adapt to IOS changes

Spanish – complete Mandarin – active Portugese - active Hebrew – pending Arabic – pending Swahili and Dholuo - active Interest in several other languages

Instrument Group Name # AssessmentsNIHTB Pattern Comparison Age 7+ Instrument 50,327 NIHTB List Sorting Working Memory Test Age 7+ Instrument 49,479 NIHTB Flanker Inhibitory Control and Attn Age 12+ Instrument 42,926 NIHTB Picture Vocabulary Test Age 3+ Instrument 42,078 NIHTB Dimensional Change Card Sort Test Age 12+ Instrument 35,801 NIHTB Picture Sequence Memory Age 8+ Form A Instrument 34,950 NIHTB Oral Reading Recognition Test Age 3+ Instrument 29,873 NIHTB Flanker Inhibitory Control and Attn Age 8-11 Instrument 19,079 NIHTB Dimensional Change Card Sort Test Age 8-11 Instrument 16,882 NIHTB Flanker Inhibitory Control and Attn Ages 3-7 Instrument 12,757 NIHTB Dimensional Change Card Sort Test Ages 3-7 Instrument 11,167

Instrument Group Name # Assessments

NIHTB Emotional Support FF Age 18+ Instrument 6,843 NIHTB Instrumental Support FF Age 18+ Instrument 6,650 NIHTB Oral Symbol Digit Test 8+ Instrument 6,557 NIHTB Auditory Verbal Learning Test (Rey) 8+ Instrument 6,439 NIHTB Perceived Stress FF Age 18+ Instrument 6,382 NIHTB Loneliness FF Age 18+ Instrument 6,161 NIHTB Self-Efficacy CAT Age 18+ Instrument 6,048 NIHTB 9-Hole Pegboard Dexterity Age 3+ Instrument 5,794 NIHTB Sadness CAT Age 18+ Instrument 5,784 NIHTB Fear-Affect CAT Age 18+ Instrument 5,708 NIHTB Grip Strength Age 3+ Identifier 5,658 NIHTB Positive Affect CAT Age 18+ Instrument 5,600

Instrument Group Name # AssessmentsNIHTB Perceived Hostility FF Age 18+ Instrument 5,588 NIHTB Friendship FF Age 18+ Instrument 5,516 NIHTB Meaning and Purpose CAT Age 18+ Instrument 5,475 NIHTB Anger-Affect CAT Age 18+ v1.1 Instrument 5,465 NIHTB Perceived Rejection FF Age 18+ Instrument 5,402 NIHTB General Life Satisfaction CAT Age 18+ Instrument 5,395 NIHTB Picture Sequence Memory Age 8+ Form B Instrument 5,332 NIHTB Anger-Physical Aggression FF Age 18+ Instrument 5,198 NIHTB Anger-Hostility FF Age 18+ Instrument 5,128 NIHTB Words-In-Noise Test Age 6+ Instrument 5,095 NIHTB Fear-Somatic Arousal FF Age 18+ Instrument 5,029

Instrument Group Name # AssessmentsNIHTB Fear-Somatic Arousal FF Age 18+ Instrument 5,029 NIH TB 4-Meter Walk Gait Speed Age 7+ Identifier 4,605 NIHTB Visual Acuity Test Age 8+ Instrument 4,494 Picture Sequence Memory Age 8+ Form C Instrument 4,033 NIHTB 2-Minute Walk Endurance Test Age 3+ Instrument 3,484 PROMIS CAT v1.0 - Depression Instrument 3,455 NIHTB Pain Intensity FF Age 18+ Instrument 2,680 NIHTB Odor Identification Age 10+ 2,635 NIH TB Standing Balance Test Ages 7+ Instrument 2,558

Instrument Group Name # Assessments

NIHTB Flanker Inhibitory Control and Attn Age 12+ Formal Spanish 2,566 NIHTB Visual Acuity Test Age 8+ Formal Spanish Instrument 2,366 NIHTB Pattern Comparison Age 7+ Formal Spanish Instrument 2,078 NIHTB Fear-Affect CAT Age 18+ Spanish Instrument 2,032 NIHTB Anger-Affect CAT Age 18+ v1.1 Spanish Instrument 2,027 NIHTB Sadness CAT Age 18+ Spanish Instrument 2,023 NIHTB Positive Affect CAT Age 18+ Spanish Instrument 2,001 NIHTB Dimensional Change Card Sort Test Age 8-11 Informal Spanish Instrument 1,816 NIHTB Words-In-Noise Test Age 6+ Formal Spanish Instrument 1,553 NIHTB List Sorting Working Memory Test Age 7+ Formal Guatemalan 1,198

Instrument Group Name # Assessments PROMIS CAT v1.2 - Physical Function Instrument 1,214 PROMIS CAT v1.1 - Pain Interference Instrument 1,065 PROMIS CAT v1.0 - Depression Instrument 758 PROMIS CAT v2.0 - Satisfaction Roles Activities Instrument 646 NIH Toolbox General Life Satisfaction CAT Age 18+ Instrument 441 Neuro-QoL CAT v1.0 – L Ex Function (Mobility) Instrument 411 PROMIS CAT v1.0 - Anxiety Instrument 329 PROMIS Bank v1.2-Upper Extremity Instrument 317 PROMIS Ped Bank v2.0-Pain Interference Instrument 275 PROMIS Bank v1.0-Self-Efficacy Manage Symptoms Instrument 268 PROMIS Bank v1.2-Mobility Instrument 221 PROMIS CAT v1.0 - Sleep Disturbance Instrument 208 Neuro-QoL CAT v1.0 - Stigma Instrument 208 PROMIS Ped Bank v2.0-UpperExtremity Instrument 208

International Neuropsychological Society (February, 2019) Hispanic Neuropsychological Society (February, 2019) American Congress of Rehabilitation Institute (April, 2019) Gatlinburg Conference for Intellectual Disability (April, 2019) HealthMeasures Users Conference (June, 2019) Alzheimer’s Association International Conference (July, 2019)

Presentations and Poster sessions

American Psychological Association (August, 2019) Exhibitor Booth, Poster sessions, and Div 40 Presentation

American Congress of Rehabilitation (November, 2019) Exhibitor Booth and Workshop

Social Media AccountsFaceBook = 147 FollowersLinkedIn = 57 FollowersTwitter = 390 Followers

NIH Toolbox publicly released in September 2012 on Web National norms (enhanced in 2015), well validated

Large, NIH-sponsored research projects desired more portable solution

iPad solution initially released for NIH use in 2014, public use through App Store in 2015

Some studies continued to use Web version, some initiated with iPad; others transitioned and have both types of data

Web Version iPad Version

Stand-alone laptop computer with separate 19” monitor for test presentation

Uses single screen for both test presentation and participant responses – 9.7”

Participant responses recorded via keyboard (reaction time tests) and/or mouse

Uses touchscreen interface for all participant responses

Internet connection required, data loaded from web

No internet connection required for testing; only for uploading data

After review, we concluded that no mode effects would be expected for Motor, Sensation or Emotion measures

Vision test distance shortened to 3 M, optotypes mathematically adjusted for distance, screen size and resolution

Others do not require interaction with iPad or involve response selection only (e.g., PROs)

Performance differences on some Cognition tests deemed possible (hypothesized not likely for language measures)

Mode of administration did not influence test scores on three NIHTB tests: Picture Vocabulary Reading List Sorting

Four tests showed differences between Web and iPad, requiring adjustments: Picture Sequence Memory DCCS Flanker Pattern Comparison

NIH ToolboxAssessment of Neurological and Behavioral Function

~11,000 children between the ages of 9 and 10 Scanned every 2 years longitudinally

NIH Toolbox Cognition A subset of emotion measures

Phenotypic measures Diagnostic interview Measures of mental health and substance use related behaviors

Imaging High resolution T1 and T2 diffusion scan minutes of resting state tasks in the scanner

1,200 young adults between the ages of 22 and 35 Many sets of DZ and MZ twins

Full NIH Toolbox other than balance and vision Full diagnostic interview Imaging High resolution T1 and T2 60 minute diffusion scan 60 minutes of resting state 7 different tasks in the scanner

1,340 individuals between the ages of 5 and 21 A subset scanned multiple times

Full NIH Toolbox other than balance and vision Full diagnostic interview Imaging High resolution T1 and T2 22 minute diffusion scan 40 minutes of resting state 3-4 different tasks in the scanner

1,340 individuals between the ages of 35 and 100 A subset scanned two times

Full NIH Toolbox other than balance and vision Full diagnostic interview Imaging High resolution T1 and T2 22 minute diffusion scan 40 minutes of resting state 3-4 different tasks in the scanner

ECHO is a 7-year (2016-23) NIH-funded large research consortia of 84 existing longitudinal cohort studies representing 40 states and Puerto Rico, n~50,000 children

ECHO aims to leverage diverse cohorts to address a broad range of early environmental exposure effects (physical & chemical, societal, psychosocial, behavioral, biological) on children’s health outcomes by standardizing new data collection and harmonizing existing measures The goal is to advance scientific understanding to inform

high-impact programs, policies, and practices that improve the health of children and adolescents

NIH Toolbox

Physical Motor Battery

Neuro-developmental

Perceived Stress, Early Childhood Cognition Battery, Cognition Battery, Pattern Comparison

SocialPositive Peer Interactions, Friendship, Loneliness, Emotional Support

NIH Toolbox

Neuro-developmental Perceived Stress, Cognition Battery

Matrix Reasoning Test, Ages 5-85 Nonverbal/visual reasoning task Undergoing IRB review

Concept Formation, Ages 2-5 Basic cognitive task that measures perceptual and semantic reasoning Test item layouts being specified

Speeded Matching, Age 3+ Processing speed test Pilot testing March 2019

iPad based eye-tracking task Social attention/cognition task for infants Currently exploring methods for adapting adult-oriented face recognition and

gaze detection algorithms for use with infants

Developed a bi-directional interface between REDCap instances and the NIH Toolbox App Participants and assessments from REDCap projects are downloaded

to the NIH Toolbox App. Scores and item data uploaded to the corresponding REDCap project from the NIH Toolbox App

Beta release to small group of ECHO cohorts to follow Collaborated with the REDCap consortium group at Vanderbilt for an

alpha version of their personal research app, MyCap, which can administer IRT-based NIH Toolbox instruments Now available for both iPhone and Android

Validate the NIHTB (English & Spanish) in ethnically/racially diverse adults ages 65-85 along the trajectory of cognitive aging and in cognitively normal individuals aged 86+.

Expand the NIHTB by adding innovative instruments to assess neurological functions associated with cognitive aging.

Facilitate use of the NIHTB + new measures in aging research by ensuring it is a readily available resource with robust user support.

Multi-center: 9 participating sites from across the United States Northwestern, Massachusetts General, University of Wisconsin, Mayo Clinic

Jacksonville, University of California San Diego, University of Pittsburgh, Emory University, Columbia University (NY)

Recruit and follow 1,260 patients over 2 years 29% (n=370) Spanish-speaking 24% (n=300) African American

Biomarkers and clinical data for validation of NIHTB sourced through the Uniform Data Set (UDS) and managed by the National Alzheimer's Coordinating Center

New NIHTB measures incorporated into HealthMeasures Obtain FDA approval for use of the NIHTB+ in clinical trials

The Face-Name (FNAME) Associative Memory Test to increase sensitivity to early cognitive changes

An iPad-based test of Near Visual Acuity, because it is critical for interpretation of NIHTB performance but not currently measured by the NIHTB

A Pure Tone Audiometry Test to identify hearing threshold levels of an individual, enabling determination of the degree, type and configuration of a hearing loss.

1. Create the MobileToolbox Library of iOS and Android smartphone based assessments.

2. Develop and maintain the Mobile Toolbox Research Platform, a software platform to support clinical researchers and software developers in the dynamic, customizable integration of Mobile Toolbox Library into research studies

3. Conduct psychometric activities to validate, norm and assess reliability of the MobileToolbox in both general population and various research samples to create general population reference norms

The Case for the use of the NIH Toolbox for Clinical Assessment

NIH Toolbox Cognition Battery (NIHTB-CB) has been “validated as a research test battery and not for clinical use, nor would it substitute for a comprehensive clinical neuropsychological examination of patients with neurobehavioral symptoms and disorders.

It has several potential applications in clinical research and in longitudinal, large-scale epidemiologic studies where there is a need for brief instruments that tap different cognitive constructs within a very large age range …”

Weintraub, Dikmen, Heaton, Tulsky, Zelazo, et al. 2014. P. 568

NIH Toolbox was developed using traditional test development techniques. The battery was developed with extensive item development stages, tryout,

validation, and standardization. The NIHTB has state-of-the-art normative scores. The NIHTB has demographically corrected normative data. All point to the potential to use the NIHTB-CB in clinical practice.

However, all of the original development work was with “healthy” individuals..

41 studies published that used the NIH Toolbox in a clinical population 10 studies were conducted among cancer patients, of which 9 were

breast cancer patients/survivors 12 studies were conducted in patients with Traumatic brain injury

(TBI), Spinal cord injury (SCI), and stroke 39 studies used cognition battery or subset of cognition battery, 3

studies used emotion battery, 2 studies used motor tests, and 1 study used sensation tests

Pop/Conditions

Group NIH-TB Measure Study design

Dependent and Outcome Variables

Journal

Breast cancerPatients

Northwestern(2,18)

Cognitive Battery Casecontrol

Hippocampal functional connectivity, subtle hippocampal deformities,episodic memory, and cognition

NeuroImage: Clinical

City of Hope National Medical Center (3,4,6)

Cognition Battery Cohort Brain gray matter density, subcortical gray matter structures, brain volume changes, and cognition.

Breast cancer research and treatment, Magnetic resonance imaging, Breast Cancer Research.

Harvard & U of Wisconsin (13)

Cognition Battery Cross-sectional

Sedentary behavior and objective cognitive functioning

Supportive Care in Cancer

Texas State U(17)

Cognition Battery Cross-sectional

Low Serum Carotenoids andcognitive dysfunction

Nutrients

U of Alabama at Birmingham(34)

Executive function

Cohort Processing speed training andexecutive function

Breast Cancer Research and Treatment

UC-San Diego(26)

Cognition Battery RCT Physical activity intervention and cognition

Cancer

Oropharyngeal Cancer Patients

Washington U in STL(7)

Cognition Battery Cohort Baseline subjective vs objective cognition

JAMA

Conditions Group NIH-TB

MeasureStudy design

Dependent and Outcome variables Journal

Traumatic brain injury (TBI)

Harvard (5) Cognition Battery

Cross-sectional

Depression and cognition The Clinical Neuropsychologist

Ohio State U(32) Cognition Battery

Cross-sectional

TBI and cognition American Journal of Speech-Language Pathology

TBI, Spinal cord injury (SCI), and stroke

Northwestern, U of Delaware, U of Michigan, Washington U,Rehabilitation Institute of Chicago, University of Illinois at Chicago, UC- San Diego, etc(19,20,25, 28,29, 30,33,36,39)

Cognition Battery

Cross-sectional

Validation and test accommodation of NIHTB-CB in patients with TBI,SCI, and stroke; Use NIHTB-CB to Identify cognitive impairment in TBI patients; cognition of SCI patients; racial differences in cognition among post-stroke patients; health literacy and cognition in TBI,SCI, and stroke patients .

Rehabilitation Psychology, Archives of Clinical Neuropsychology, Journal of the International Neuropsychological Society, HLRP: HealthLiteracy Research and Practice

Acquired brain injury

Northwestern, U of Delaware, U of Michigan, Washington U,UC- San Diego, etc (40)

Cognition Battery

Cross-sectional

Evaluated the NIHTB-CB factor structure in individuals with acquired brain injury vs general pop

Rehabilitation Psychology

Conditions Group NIH-TB MeasureStudy design

Dependent and Outcome Variables Journal

Epilepsies U of Alabama at Birmingham(8)

Flanker Cross-sectional

Cortical excitability, seizure control and cognitive performance

Epilepsy and Behavior

Cincinnati Children's Hospital(14)

DCCS, Flanker, LSWM Cross-sectional

Executive Function deficits pattern, epilepsy type, and measurement tools

Epilepsyand Behavior

Alzheimer's and Dementia

Temple, Cornell, & U of Cincinnati(11,41)

Cognition battery Cross-sectional

Validate NIH-TB for clinical assessment, propose a novel framework in evaluate and treating AD patients in clinic

Alzheimer’s & Dementia

Patients at Brain Cognitive Screening Clinic

Emory (12) PSM Cross-sectional

Clinical use of PSM and its Diagnostic Relationship to the Rey Auditory Verbal Learning Test

Archives of ClinicalNeuropsychology

Patients with diffuse glioma

U of Calgary in Canada(31)

Cognition Battery,Emotion Battery,motor tests (Grip Strength & Pegboard)

Cohort Use NIHTB to assess emotion, cognition, and motor at baseline and throughout treatment.

World Neurosurgery

PTSD & Chronic Musculoskeletal Pain

U of Arkansas(38) Flanker, LSWM, PSM, Emotion Battery

Quasi-experimental

Taichi intervention, pain, emotion, and cognition

Rehabilitation Psychology

Conditions Group NIH-TB

MeasureStudy design

Dependent and Outcome Variables Journal

Veterans with major depression

Oregon Health & Science U(16)

Adult Social Relationship Scales

Cross-sectional

Social connectednessand depression

Arthritis Care and Research

Schizophrenia U of Texas Health Science(9,21,35)

Cognition battery

Cross-sectional

C-reactive protein,S100B calcium binding protein, selectins, and cognitive function

Frontiers in Public Health

Binge-Eating Disorder Temple U(23) DCCS, Flanker, PCPS, LSWM, PSM

Cross-sectional

Executive functioning and Binge-Eating

International Journal of Eating Disorders

Cocaine Use Disorder Columbia U(24) Cognition battery

Case-control

Cocaine use and cognitive functioning

Journal of Clinical and Experimental Neuropsychology

Chronic Marijuana Users U of Hawaii (37) Cognition battery

Case-control

Microbiota, mitochondrial function, and cognition

Journal of NeuroimmunePharmacology

Conditions Group NIH-TB

MeasureStudy design

Dependent and Outcome Variables Journal

Pediatrics Preterm infants U of Minnesota(1)

PCPS Cohort Postnatal body composition, speed of processing, and blood pressure

Pediatric research

Boys with Dystrophinopathies(aged 5-17)

U of New York(10)

Flanker, DCCS, LSWM,PCPS

Cross-sectional

Executive skills and academic achievement

Journal of the International Neuropsychological Society

Children with Recurrent Acute Tonsillitis

Dartmouth Medical Center(27)

Odor Identification Test

Cohort Recurrent acute tonsillitisand glossopharyngeal taste

Otolaryngology -Head and Neck Surgery

Preschool children with sickle cell disease

UCollegeDublin(22)

Flanker Case-control

Sickle cell disease and selective attention

Journal of Pediatric Psychology

Adults Systemic lupuserythematosuspatients

Emory (15) PSM, LSWM,PCPS,Flanker, DCCS

Case-Control

Physical and cognitivefunction in patients with SLE

Arthritis Care and Research

Update the NIH Toolbox to fully maximize iPad Delivery (2019) Renorm all of the measures in the NIH Toolbox (2020) Release NIH Toolbox Version 3.0 with updated measures and new

norms (2021)