description: tags: rel 2007034 body

8/14/2019 description: tags: REL 2007034 body

1/22

I S S U E S& AN SWER S

U . S . D e p a r t m e n t o f E d u c a t i o n

Measuringresilience

and youthdevelopment:the psychometricproperties of

the HealthyKids Survey

R E L 2 0 0 7 N o . 0 3 4

At WestEd


2/22

Measuring resilience and youthdevelopment: the psychometric

properties of the Healthy Kids SurveySeptember 2007

Prepared by

Thomas L. HansonRegional Educational Laboratory West

Jin-Ok KimRegional Educational Laboratory West

I S S U E S&ANSWERS R E L 2 0 0 7 N o . 0 3 4

U . S . D e p a r t m e n t o f E d u c a t i o n

At WestEd


3/22

Issues & Answers is an ongoing series o reports rom short-term Fast Response Projects conducted by the regional educa-

tional laboratories on current education issues o importance at local, state, and regional levels. Fast Response Project topicschange to reect new issues, as identied through lab outreach and requests or assistance rom policymakers and educa-

tors at state and local levels and rom communities, businesses, parents, amilies, and youth. All Issues & Answers reports

meet Institute o Education Sciences standards or scientically valid research.

September 2007

Tis report was prepared or the Institute o Education Sciences (IES) under Contract ED-06-CO-0014 by Regional Edu-

cational Laboratory West administered by WestEd. Te content o the publication does not necessarily reect the views or

policies o IES or the U.S. Department o Education nor does mention o trade names, commercial products, or organiza-

tions imply endorsement by the U.S. Government.

Tis report is in the public domain. While permission to reprint this publication is not necessary, it should be cited as:

Hanson, . L., & Kim, J. O. (2007).Measuring resilience and youth development: the psychometric properties o the Healthy

Kids Survey. (Issues & Answers Report, REL 2007No. 034). Washington, DC: U.S. Department o Education, Institute o

Education Sciences, National Center or Education Evaluation and Regional Assistance, Regional Educational Laboratory

West. Retrieved rom http://ies.ed.gov/ncee/edlabs

Tis report is available on the regional educational laboratory web site at http://ies.ed.gov/ncee/edlabs.

WA

OR

ID

MT

NV

CAUT

AZ

WY

ND

SD

NE

KS

CO

NM

TX

OK

CO

AR

LA

MSAL GA

SC

NC

VAWV

KY

TN

PA

NY

FL

AK

MN

WI

IA

IL IN

MI

OH

VT

NH

ME

MO

At WestEd


4/22

Summary

This report summarizes fndings rom a

study o the psychometric properties o

the resilience and youth development

module, a key component o the Healthy

Kids Survey. The study aims to improve

resilience assessment and research so

that educators can shape the school envi-

ronment to promote academic resilience.

Te Healthy Kids Survey (HKS) is a compre-

hensive student sel-report tool or monitoring

the school environment and student health

risks. Tis report ocuses on one module o the

survey, the resilience and youth development

module (RYDM), which assesses environmen-tal and internal assets associated with posi-

tive youth development and school success.

Environmental assets reer to meaningul and

pro-social bonding to community, school,

amily, and peers. Internal assets are personal

resilience traits, such as sel-ecacy and

problem-solving skills

A part o the resilience and youth development

module is administered to 600,000 studentsin Caliornia every year. School districts

and schools, which receive both single-year

prevalence data and trend data gathered by

the module, use the data to evaluate their local

programs and guide decisionmaking. Te

Healthy Kids Survey and the resilience and

youth development module were designed as an

epidemiological surveillance tool to track ag-

gregate levels o health risk and resilience. Te

module increasingly is being used in evaluation

work to assess student-level changes over time.

However, widespread use o the module,

particularly or evaluation, may be premature.

Te psychometric properties o specic scales

assessed by the elementary school module

have yet to be established. Te secondary

school module has not been validated since

2000, when the instrument was rst tested in

the eld. Te instrument has since undergone

several modications, however, and must be re-

validated. Moreover, measurement equivalenceacross dierent grades, males and emales, and

racial and ethnic groups has never been exam-

ined. Given Caliornias diversity, demonstrat-

ing the cultural appropriateness o the module

or dierent racial and ethnic groups is critical.

Using HKS data processed or school districts

by WestEds Health and Human Development

Program, Regional Educational Laboratory

West analyzed the modules psychometricproperties. Tis report describes the results o

this analysis, provides recommendations on

the proper use o the instrument, and suggests

modications to the module.

For the secondary school module, the results

are consistent with the instruments current

Ms s th

mt: th shmtts th Hth Ks S


5/22

Summary

use as an epidemiological tool and with its

conceptual oundation. It provides compre-

hensive and balanced coverage o eight envi-

ronmental resilience assets and our internal

resilience assets; its subscales exhibit goodinternal consistency and are associated with

student risk actors in expected ways. And i

certain items are dropped, the module also

demonstrates measurement equivalence across

racial/ethnic groups, males and emales, and

grades. Te secondary school RYDM scales ex-

hibit low test-retest reliability, however, which

suggests that the module is not well suited or

examining student-level changes over time.

Te instrument was not designed to examineindividual dierences across students and

should not be used this way. Moreover, two

o the six internal assets that the secondary

school module was designed to measure

cooperation and goals/aspirationscould not

be assessed validly. Several measures would

benet i additional items were included in

derived scales to increase domain coverage.

Te elementary school module was designed

to assess seven environmental resilience assets

and three internal resilience assets, but it can

reliably assess only two environmental as-

sets and one internal asset. Most o the scales

measured by the elementary school instru-

ment have poor psychometric properties. Te

elementary school instrument should thus be

modied considerably to make it suitable orresearch.

September 2007


6/22

Table o conTenTS

Why this study? 1

Developing a risk and resilience assessment tool 2Te Healthy Kids Surveyassessing risk and protective actors 2

Te resilience and youth development moduleassessing the other side o risk 4

Evaluating the psychometric properties o the resilience and youth development module 8Results o the analysis o the secondary school module 10Results o the analysis o the elementary school module 11

Recommendations 12Secondary school environmental resilience assets 12Secondary school internal resilience assets 12Elementary school environmental and internal assets 13

Appendix A Analytic strategy 15

Appendix B Results 21

Appendix C Results and model selection details 44

Appendix D Other assessments o resilience and related actors 53

Appendix E Detailed tables 55

Notes 164

Reerences 165

Boxes

1 Specications o the Healthy Kids Survey 32 Data and analytic strategies 9

Figures

1 Conceptual model or the resilience and youth development module 5

A1 Hypothetical example o MIMIC approach or testing or measurement equivalence 18

C1 Secondary environmental resilience asset scree plot, total analytic samples 44

C2 Elementary school environmental resilience asset scree plot, total analytic samples 46

C3 Secondary school internal resilience asset scree plot, total analytic samples 50

C4 Elementary school internal resilience asset scree plot, total analytic samples 52

Tables

1 Items on the secondary school resilience and youth development module by construct, 2006/07 6

2 Elementary school resilience and youth development module items by construct, 2006/07 8

3 Recommended measures o environmental resilience assets among secondary school students 13

4 Recommended measures o internal resilience assets among secondary school students 14


7/22

5 Recommended measures o environmental and internal resilience assets among elementary schoolstudents 14

A1 Missing data patterns or secondary and elementary samples rom the resilience and youth developmentmodule 16

B1 Secondary school environmental resilience asset exploratory actor analysis results, main sample, 8-actorsolution 22

B2 Secondary school environmental resilience asset exploratory actor analysis results, validation sample,8-actor solution 23

B3 Final secondary school environmental assets model, main sample 24

B4 Correlations among secondary school environmental resilience assets, nal conrmatory actor analysismodel 25

B5 Elementary school environmental resilience asset exploratory actor analysis results, main sample, 4-actorsolution 25

B6 Elementary school environmental resilience asset exploratory actor analysis results, validation sample,4-actor solution 26

B7 Final elementary school environmental resilience assets model, main sample 27

B8 Secondary school internal resilience asset exploratory actor analysis results, main sample, 4-actormodel 28

B9 Secondary school internal resilience asset exploratory actor analysis results, validation sample, 4-actormodel 29

B10 Final secondary school internal resilience assets model, main sample 30

B11 Elementary school internal resilience asset exploratory actor analysis results, main sample, 2-actormodel 30

B12 Elementary school internal resilience asset exploratory actor analysis results, validation sample, 2-actormodel 31

B13 Final elementary school internal resilience asset model, main sample 31

B14 Secondary school internal consistency reliability coecients by demographic subgroup 32

B15 Elementary school internal consistency reliability coecients by gender 32

B16 est-retest reliability o secondary school environmental resilience asset constructs and items 33

B17 est-retest reliability o secondary school internal resilience asset constructs and items 34

B18 est-retest reliability o elementary school resilience asset constructs and items 35

B19 Secondary school subscale means by demographic subgroup 36

B20 Elementary school subscale means by gender 37

B21 Correlations between secondary school environmental resilience assets and criterion variables 38

B22 Correlations between secondary school internal resilience assets and criterion variables 39


8/22

B23 Correlations between elementary school resilience assets and criterion variables 40

B24 Current and recommended measures o environmental resilience assets among secondary schoolstudents 41

B25 Current and recommended measures o internal resilience assets among secondary school students 42

B26 Current and recommended measures o environmental resilience assets among elementary schoolstudents 43

B27 Current and recommended measurement o internal resilience assets among elementary schoolstudents 43

C1 Secondary school environmental resilience assets, total analytic sample, goodness-o-t inormation orexploratory actor analysis models 44

C2 Secondary school environmental resilience asset, total analytic sample, goodness-o-t inormation orconrmatory actor analysis models 46

C3 Measurement intercept dierences or environmental resilience assets, secondary school sample 47

C4 Elementary school environmental resilience assets, total analytic sample, goodness-o-t inormation orexploratory actor analysis models 48

C5 Elementary school environmental resilience asset, total analytic sample, goodness-o-t inormation orconrmatory actor analysis models 49

C6 Gender measurement intercept dierences or environmental resilience assets, elementary school sample 49

C7 Secondary school internal resilience assets, total analytic sample, goodness-o-t inormation or exploratoryactor analysis models 50

C8 Secondary school internal assets, total analytic sample, goodness-o-t inormation or conrmatory actor

analysis models 51C9 Measurement intercept dierences or internal resilience assets, secondary school sample 51

C10 Elementary school internal resilience assets, total analytic sample, goodness-o-t inormation orconrmatory actor analysis models 52


9/22

Why thiS Study? 1

Ths tsmmzs

s m st thshmtts th s thmtm, kmt th HthKs S. Th

st ms tm sssssmt sh s thtts sh th shmtt mtms.

WHy THiS STudy?

As improvements to curriculum and instruction

raise academic standards, researchers are look-

ing more and more at what actors account or

the varied inuence o these improvements. Mosthave ocused on risk actors or academic ailure,

such as poverty or racial and cultural minority

status. But researchers are beginning to look at the

other side o r iskresilienceand have identi-

ed several traits common to resilient youth that

enable the youth to overcome barriers to academic

success. Tere is little research, however, on how

to measure these traits within the general student

population and how to determine the role o the

school environment in promoting these traits.

Te Healthy Kids Survey (HKS) is one o the ew

large-scale surveys to assess both risk and resil-

ience. Te surveys resilience and youth develop-

ment module (RYDM) is based on the premise

that youth who experience high levels o environ-

mental assets in three areashigh expectations

rom adults, caring relationships with adults, and

opportunities or meaningul participation

will develop the resilience traits, the connection

to school, and motivation to learn that lead to

positive academic, social, and health outcomes(Constantine, Benard, & Diaz, 1999).

Te resilience and youth development module

which has both elementary and secondary school

versionswas designed as an epidemiological

surveillance tool to track aggregate levels o pro-

tective actors. In Caliornia an average o about

600,000 students take the Healthy Kids Survey

and a part o the resilience and youth development

module every year. School districts and schools

use the resulting prevalence and trend data toguide programmatic decisionmaking. With such

widespread administration, school districts and

independent evaluators are increasingly using the

survey data to evaluate local programs by examin-

ing student-level changes over time. Capitalizing

on the mandated administration o a standard

instrument or local evaluation has the benet

o reducing the survey burden or students and


10/22

2 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey

provides comparable outcome data across dierent

program evaluations.

Widespread use o the module or research and

local evaluation may be premature, however.

Te psychometric properties o specic scalesassessed by the elementary school module have

yet to be established. And the secondary school

module has not been validated since 2000, when

the instrument was rst tested in the eld. Te

instrument has since been modied several times,

making validation o the current secondary school

resilience and youth development module neces-

sary. In addition, measurement equivalence across

racial and ethnic groups, males and emales, and

dierent grades has never been systematically

examined. Te stakes are thus high to ensure thatall parts o the module are valid and reliable.

o guide urther improvements o this important

assessment tool, Regional Educational Labora-

tory West conducted psychometric analyses o the

properties o the resilience and youth development

module, using a large set o recent survey data.1

Tis report describes the results o these analyses,

makes recommendations on the proper use o the

module, and suggests modications to improve

the instrument.

For the secondary school module, the results

are consistent with the instruments current use

as an epidemiological tool and with its concep-

tual oundation. It provides comprehensive and

balanced coverage o eight environmental resil-

ience assets and our internal resilience assets;2

its subscales exhibit good internal consistency

and are associated with student risk actors in

expected ways. And i certain items are dropped,

the module also demonstratesmeasurement equivalence across

racial/ethnic groups, males and

emales, and grades. Te second-

ary school RYDM scales exhibit

low test-retest reliability, however,

which suggests that the module

is not well suited or examining

student-level changes over time.

Te instrument was not designed to examine

individual dierences across students and should

not be used this way. Moreover, two o the six

internal assets that the secondary school module

was designed to measurecooperation and goals/

aspirationscould not be assessed validly. Severalmeasures would benet i additional items were

included in derived scales to increase domain

coverage.

Te elementary school module was designed to

assess seven environmental resilience assets and

three internal resilience assets, but it can reli-

ably assess only two environmental assets and

one internal asset. Most o the scales measured

by the elementary school instrument have poor

psychometric properties. Te elementary schoolinstrument should thus be modied considerably

to make it suitable or research.

developing a riSK and reSilience

aSSeSSMenT Tool

Te Healthy Kids Survey is a comprehensive health

risk and resilience data collection system that

relies on student sel-reporting. Te surveys core

module tracks health risks and problem behaviorsthat are signicant barriers to learning among

students. Te resilience and youth development

module assesses individual and environmental

assets associated with positive youth development

and school success. Tis section provides a brie

background on how the survey and the resilience

and youth development module were developed

and are now used in Caliornia.

The Healthy Kids Surveyassessing

risk and protective factors

Te Healthy Kids Survey is the largest eort in the

nation to require school districts to assess student

resilience and risk behaviors (box 1). Te Calior-

nia Department o Education requires all school

districts with ederal itle IV unding or with state

obacco Use Prevention and Education grants to

administer the survey every two yearsthe case

h s th

mt m

sssss

mt ssts

sst wth st

th mt

sh sss


11/22

developing a riSK and reSilience aSSeSSment tool 3

or 85 percent o Caliornia school districts. Inmandating the survey, the Caliornia Department

o Education aims to promote accountability and

data-driven decisionmaking and to improve health

and prevention programs in schools.

Te survey was developed in 1997 by WestEds

Health and Human Development Program in col-

laboration with Duerr Evaluation Resources and

an advisory committee o researchers, teachers,

prevention and health program practitioners,

and public agency representatives. Te CaliorniaDepartment o Education unded the develop-

ment o the survey in response to ederal require-

ments that schools implement the Principles o

Eectivenessto collect and use data to assess

student needs, justiy program unding, guide

program development, and monitor progress in

achieving program goals. Te immediate impetus

or mandating the biennial administration o the

survey, however, was meeting the requirements othe No Child Lef Behind Act (itle IVSae and

Drug-Free Schools and Communities Act).

Te Healthy Kids Survey consists o a general core

module, the resilience and youth development

module, and our optional modules on specic

risk behaviors. It can be customized to meet local

needs:

Te required core module assesses demo-

graphic inormation and health risks relat-ing to school violence, harassment, physical

health, mental health, school-related behavior

(such as truancy), and alcohol, tobacco, and

other drug use.

Te resilience and youth development module

assesses environmental actors (environmental

assets) and individual traits (internal assets)

Box 1

Specifcations o the HealthyKids Survey

Mandate

Mandated (since all 2003) by the

Caliornia Department o Educa-

tion or compliance with No Child

Lef Behind and state obacco Use

Prevention and Education (UPE)

grants

Survey type

Comprehensive health r isk and

resilience survey

Student sel-report

Anonymous, voluntary,

condential

Modular secondary school

instrument; single elementary

school version

Grade levels

Grades 5, 7, 9, 11, and students in

continuation schools

Sampling

Representative district sample;

school-level surveys optional

Required modules

(secondary school)

Core (required)A.

Resilience and youth develop-B.

ment (school and community

asset scales required)

Optional modules

(secondary school)

Resilience and youth develop-B.

ment (home, peer, and internal

asset scales)

Saety (violence and suicide)C.

and alcohol and other drug

use

obaccoD.

Physical healthE.

Sexual behavior (pregnancy andF.

HIV/AIDS risk)

Custom module (or addingG.

questions)

Sources

Items based on the Caliornia Student

Survey, Youth Risk Behavior Survey,

and Caliornia Student obacco Use

and Evaluation Survey

Requirements

Biennial administration

Module A and school & commu-

nity asset scales in module B

Module D by state UPE grantees

Written parental consent; passive

consent optional since all 2004

Representative district samples

Administration

By school, ollowing detailed

instructions

Processing and reporting by

WestEds Health & Human De-

velopment Program

Product

Local reports and aggregated state

database


12/22


associated with academic perormance, posi-

tive youth development, and protection rom

risky behaviors. Te Caliornia Department o

Education mandates that the sections on school

and community assets be administered to all

students who take the Healthy Kids Survey.

Four optional, topical modules (and one

customizable module) collect urther detail

on subjects covered by the core module, such

as violence and alcohol and other drug use

(module C); tobacco use and tobacco educa-

tion (module D); physical activity and diet

(module E); and sexual behavior, pregnancy,

and HIV risk (module F).

A custom module that allows schools to incor-

porate their own items.

Te survey was designed as a district surveillance

tool to provide prevalence estimates representative

o students in the school districts that administer the

survey rather than o students in the state as a whole.

It was not designed to evaluate student-level changes

over time or individual dierences across students.

Te Caliornia Department o Education requires

that districts administer the survey to 900 randomly

selected students rom each targeted grade (5, 7, 9,and 11). In districts with ewer than 900 students per

grade (the case or 85 percent o Caliornia districts),

all students in the targeted grades are surveyed. I

a district has more than 10 schools per grade, at

least 50 percent o schools are randomly sampled.

(Los Angeles Unied School District has dierent

requirements because o its size.)

WestEds Health and Human

Development Program provides

school districts administeringthe survey with technical as-

sistance and with a report on the

district-level data collected in each

module.

Although several adolescent

behavior surveys, such as the

Youth Risk Behavior Surveillance

System, assess student risk actors and problem

behavior, the Healthy Kids Surveys assessment o

student supports, strengths, and competencies sets

it apart. While some surveys incorporate protec-

tive actors, the resilience and youth development

module is one o the ew assessments that speci-cally addresses this dimension and does so with a

strong theoretical oundation.

The resilience and youth development module

assessing the other side of risk

Secondary school module.In early 1998 the HKS

Advisory Committee asked WestEd to develop a

survey module to assess middle and high school

student strengths, competencies, and positive so-

cial and health attitudes, eeling that the HKS coremodule did not give practitioners enough inorma-

tion about the actors behind positive development

and school success (Constantine et al., 1999).

WestEd ormed a Resilience Assessment Expert

Panel to develop and validate a new survey module

on youth resilience. Te assessment needed to be

brie enough to be widely administered along with

the HKS core module; have a strong theoretical

oundation; demonstrate reliability, validity, and

cultural and developmental appropriateness whenadministered in Caliornia school settings; and

provide a comprehensive, research-based assess-

ment o environmental actors (environmental

assets) and resilience traits (internal assets).

Environmental assets reer to meaningul and pro-

social bonding to community, school, amily, and

peers. Internal assets are personal resilience traits,

such as sel-ecacy and problem-solving skills

(Benard, 1991, 1995, 2004).

Failing to nd a survey that met its theoreti-cal and psychometric criteria, the panel built

on research on resilience and healthy human

development systemsparticularly the work o

Benard (1991, 1995, 2004)to develop a theoreti-

cal ramework that describes resilience actors

and their interrelationships (gure 1). Te result-

ing module or secondary school students was

designed to measure 11 environmental assets,

t s

ht mt ts tht

shmt

t, th t

sh t

tht mwk

ht ss s

ts th

ttshs


13/22


asking students their perception o adult high

expectations, their perceptions o caring rela-

tionships with adults, and their opportunities

or meaningul participation in school, home,

and community environments. Te module alsoassesses caring relationships and high expecta-

tions in the peer domain. Tese external sup-

ports promote positive outcomes, discouraging

risky behavior and stimulating academic success

(Benard, 2004; Constantine et a l., 1999; Hawkins,

Catalano, & Miller, 1992; Masten & Coatsworth,

1998; Resnick et al., 2000; Rutter, 1987; Werner &

Smith, 1982, 1992).

Internal resilience assetsthe personal strengths

o a resilient childinclude social competence,problem solving, autonomy, and sense o pur-

pose, which can each be broken down urther

(Benard, 1991, 2004). Socia l competence, or ex-

ample, entails social communication skills, em-

pathy and caring, and the ability to elicit positive

responses rom others (responsiveness) (Benard,

2004; Masten, 2001). Problem solving involves

planning, exibility, and resourceulness;

autonomy entai ls sel-ecacy, sel-awareness,

and mindulness; and sense o purpose in-

cludes goal direction, achievement motivation,

optimism, and hope (Benard, 2004). Internal

resilience assets develop both naturally and inresponse to environmental resilience assets. Te

resilience and youth development module was

designed to measure six internal assets: empathy,

problem solving, sel-ecacy, sel-awareness,

cooperation and communication, and goals and

aspirations.

A pool o 128 potential items was piloted in one

middle and one high school in all 1998. Re-

searchers, classroom teachers, and other school

practitioners helped select and modiy itemsrom the pool and revise the ormat and instruc-

tions. Te rst eld test o the resilience and

youth development module, with 92 resilience

items, was administered to 1,000 high school

students in three school districts in winter 1999.

Cognitive processing interviews with students

were also conducted to nd out students inter-

pretation o the items. Based on analysis o the

Improved

health,

social, and

academic

outcomes

Internal resilience assets

Environmental resilience assets Youth needs

School

Home

Community

Peers

igure 1

ct m th s th mt m


14/22


taBle 1

itms th s sh s th mt m stt, 2006/07

cs i ds

Environmental resilience assets

Sh ssts

c ss s

SchlCare r6

r8

r10

a s, s s w. . .

s b .

s w i .

ss w i s s.

h s s

SchlHigh r7

r9

r11

a s, s s w. . .

s w i jb.

ws ws bs.

bs i w b sss.

m s

SchlPart r12

r13

r14

a s. . .

i s s.

i s k ss s s.

i s k .

Hm sstsc ss

HomeCare r49

r51

r53

i , s s . . .

w s s swk.

w ks w b bs.

w ss w i s s.

h s

HomeHigh r48

r50

r52

i , s s . . .

w s w s.

w bs i w b sss.

w ws ws bs.

m

HomePart r54

r55

r56

a . . .

i s s w s s.

i s k .

i k ss w .

cmmt ssts

c ss

ComCare r15

r17

r20

os s, s . . .

w s b .

w s w i s b s.

w i s.

h s

ComHigh r16

r18

r19

os s, s . . .

w s w i jb.

w bs i w b sss.

w ws ws bs.

m

ComPart r21

r22

r23

os s, i s s. . .

i bs, ss s, /, s.

i s, , , ss bb.

i .

p ssts

c ss w s

PeerCare r42

r43

r44

i b w . . .

w s b .

w ks w b bs.

w s w i .

p-s s

PeerHigh r45

r46

r47

m s. . .

b.

w s .

w s.


15/22


cognitive interview data, requency distributions,and estimated Cronbachs alpha coecients, the

number o resilience items was reduced rom

92 to 51 (table 1). In 2001 the resilience instru-

ment was modied again, based on the results o

grade-, gender-, and race/ethnic-specic explor-

atory actor analyses o data collected during

the 1999/2000 academic year. Te constructed

resilience scales based on the 1999/2000 eld test

data orm the basis o the current RYDM reports

provided to school districts, even though the

module has since been modied urther.

Since 2003 all districts administering the Healthy

Kids Survey must also administer the school and

community asset parts o the module.3 Tirty-ve

percent o districts choose to administer the ull

resilience and youth development module, reect-

ing widespread interest in assessing resilience.

WestEd provides districts with the data or each

scale and a report on the meaning and use o thedataand on how schools can create supportive

learning environments that promote school con-

nectedness and achievement. WestEd also pro-

vides state-level data to researchers and evaluators

who apply or it.4

Elementary school module. Pools o resilience

items were not independently developed or the

elementary school module. Tey were selected

rom the secondary school module afer ocus

groups with elementary school students. Initially,the elementary school module used the same

constructs as the secondary school module, but

with two items per construct instead o three.

Analysis o the 1999 eld test data and cognitive

processing interviews with students suggested

item deletions and changes in item wordings and

response options. Te nal version has 21 items

(table 2).

cs i ds


c

Coop r31

r36

r37

hw s ss b s?

i wk w s w s s .

i j wk w ss .

i s s w s w.

S-

SelEf r29

r30

r32

hw s ss b s?

i wk bs.

i s s i .

t s i w.

e

Empathy r33

r34

r38

hw s ss b s?

i b w s s s .

i s w .

i s w k.

pb-s

ProbSolv r35

r27

r28

hw s ss b s?

W i i f s k w.

i kw w w b.

i wk bs b k w b .S-wss

SelAware r39

r40

r41

hw s ss b s?

t s s .

i s s s.

i s w i w i .

gs ss

Goals r24

r25

r26

hw s ss b s?

i s s .

i s.

i s s s.

Note: Possible responses include (1) not at all true, (2) a little true, (3) pretty much true, (4) very much true.


16/22


evaluaTing THe pSycHoMeTric

properTieS o THe reSilience and

youTH developMenT Module

o better understand and improve the psychomet-

ric properties o the resilience and youth develop-

ment module, this report analyzes local HKS data

processed between 1998 and spring 2005, askingthe ollowing questions:

How should school districts and local evalu-

ators best use the module? Should the instru-

ment be used exclusively to assess prevalence

o environmental and internal assets or

should it also be used to assess student-level

changes across time?

What are the psychometric properties o

specic scales assessed by the secondary and

elementary school resilience and youth devel-

opment modules (including the dimensional-

ity o scales, scale reliability, and construct

validity)?

Does the module exhibit measurement

equivalence across racial and ethnic groups?

In other words, is it culturally appropriate

or dierent racial and ethnic groups? Does it

exhibit measurement equivalence or males

and emales? Across dierent grades?

What modications should be made to im-

prove the module?

taBle 2

emt sh s th mt m tms stt, 2006/07

cs i ds


Sh ssts

c ss s

SchlCare

10

13

d s w-s s b ?

d s w-s s s w s s?

h s s

SchlHigh

11

14

d s w-s s w jb?

d s w-s s b jb?

m s

SchlPart

9

15

d k ss s s s s?

d s b s?

Hm ssts

c ss

HomeCare

52

55

ds s w- b swk?

ds s w- s w

s s?

h s

HomeHigh

53

54

ds s w- b jb?

ds s w- w bs?

m

HomePart

56

57

d ?

d k s s s ?

p ssts

h s w s

PeerHigh

50

51

d bs s b?

d bs s ?


e

Empathy

37

38

d s w ?

d b w s s s ?

pb-s

ProbSolv

39

40

d kw w w b?

d wk bs b k w b ?

gs ss

Goals

41

4216

d bs?

d s s ?d s s s?

Note: Possible responses include (1) no, never, (2) yes, some of the time, (3) yes, most of the time, (4) yes, all of the time.


17/22

evaluating the pSychometric propertieS o the reSilience and youth development module 9

Box 2

Data and analytic strategies

Te authors used the ollowing data

and analytic strategies to analyze

the psychometric properties o the

secondary and elementary school

resilience and youth development

modules.

Data

wo mutually exclusive analytic

samplesa main sample and a

validation samplewere drawn

rom an aggregate data le that

included all HKS data processed

between the spring 2003 and the

spring 2005 administrations o the

Healthy Kids Survey. For the second-

ary school analysis, separate samples

were drawn or each grade (7, 9, and

11), gender, and ethnicity (Chinese

American, Arican American, Mexi-

can American, and white European

American)with 500 respondents

randomly sampled per cell (12,000

total). Equal numbers were used or

each gender and ethnic group so that

models that do not adjust or gender

and/or ethnicity would not be a-

ected by gender/ethnic dierences in

the sample.

For the elementary school analysis,

random samples o 1,000 males and

1,000 emales (2,000 total) were

drawn rom the aggregated HKS data

le. Tus, or the elementary school

resilience and youth development

module, only gender dierences in

measurement structure were exam-

ined. Respondents with missing data

on more than hal the resilience items

were excluded rom the analysis. For

estimating models with missing data,

maximum likelihood estimation with

missing at random (MAR) assump-

tions were used, which assumes that

values are missing at random con-

ditional on the other observed items

in the data (Little & Rubin, 2002;

Muthn & Muthn, 2006).

Statewide data was supplemented

with two sets o HKS data originally

collected or local evaluation. Data

collected in 2006 rom a large urban

school district in Southern Caliornia

were used to describe the temporal

stability o the derived scales (test-

retest reliability). Te elementary

school Healthy Kids Survey and the

secondary school core module and re-

silience and youth development mod-

ule were administered two times in

two weeks to 132 fh-grade students

and 90 ninth-grade students. Data

collected in 2004/05 rom students in

a large county in Southern Caliornia

were used to examine the relation-

ship between the RYDM constructs

and standardized test scores.

Exploratory and confrmatory

actor analyses

Analyses were conducted to test em-

pirically whether the actor structure

o the resilience instrument is con-

sistent with current usage and with

its underlying conceptual model. For

each sample and subsample (grade,

gender, ethnicity), the measurement

structure o the resilience instrument

was established by tting a series o

exploratory and conrmatory actor

analysis models. Exploratory actor

analysis (EFA) models were estimated

to determine roughly the number o

actors underlying the data and the

measurement structure o the latent

actors. A combination o criteria was

used to determine the number o ac-

tors to retain in the EFAs, including

t indices, scree plots, the number o

eigenvalues greater than 1, concep-

tual clarity, and simplicity. Models

with the ewest possible actors and

models with no cross-loadings were

avored over more complex models.

Te results o the exploratory actor

analysis models were then used as a

starting point or a series o nested

conrmatory actor analysis (CFA)

models. Measures o model t, cor-

relations among the latent constructs

(actors), and actor-loading patterns

were used to make decisions about

models. Tis process was replicated

or each grade, gender, and ethnic

group, and or the main sample and

the validation sample.

o derive estimates or the EFA and

CFA models, Muthn and Muthns

(2006)Mplus statistical modeling

program was used. Because all the

items used to measure resilience

assets are ordinal, Muthns (1984)

approach to exploratory and conr-

matory actor analysis with ordinal

indicators was used.

Confrmatory actor analysis

models with covariates

Measurement equivalence across de-

mographic subgroups was examined

by estimating conrmatory actor

analysis models with covariates.

MIMIC modelingmultiple indica-

tor, multiple cause structural equa-

tion modelswas used to test or

dierential item unctioning across

school grade, gender, and ethnic-

ity. An applied strategy was used to

(continued)


18/22


Tis report nds that both the secondary school

and elementary school modules are used pri-

marily to report aggregate data on prevalence

and district-level changes across time. Although

several modications should be made, the RYDM

scales are generally consistent with current use o

the instruments and with the conceptual ounda-

tion o the module. (See box 2 and appendixes Aand B or a discussion o the analytic strategy and

the results o the analysis.)

Results of the analysis of the secondary school module

Te secondary school module is a short instru-

ment (51 items) suitable or widespread adminis-

tration. It provides comprehensive and balanced

coverage o both environmental (eight dimen-

sions) and internal (our dimensions) resilience

assets.5 Its subscales exhibit good internal consis-

tency and are associated with student risk actors

in expected ways. I certain items are dropped,

the module also demonstrates measurement

equivalence across racial/ethnic groups, males and

emales, and grades.

Te secondary school instrument is appropriate

as an epidemiological tool, but is not well suited

or evaluating student-level changes over time or

individual dierences across students. Te instru-

ment exhibits low test-retest reliability, suggesting

that the RYDM constructs are temporally specic.

Estimates o student-level changes across time are

ascertain whether group dierences

in measurement intercepts have

implications or evaluation research.Recommendations or item changes

are made only when the measure-

ment intercepts are substantively di-

erent across groups ( 0.20 standard

deviations) in both the main sample

and the validation sample.

Additional reliability and

validity analyses

Internal consistency estimates o

reliability o the derived scales werecalculated using Cronbachs alpha or

each grade, gender, and ethnic group

in both the main sample and the

validation sample. Nunnalys (1978)

criterion o 0.70 was used as the cuto

or determining acceptable internal

consistency reliability or the second-

ary school survey. Because o the no-

toriously low internal consistency evi-

dent in surveys o elementary school

students, this criterion was relaxedslightly to 0.60 or the elementary

school module. o examine test-retest

reliability, RYDM survey data col-

lected rom a small sample o fh andninth graders who took the resilience

and youth development module twice

in two weeks was used.

Dierences in resilience scale scores

across the demographic subgroups

were also examined. o make demo-

graphic dierences in the resilience

scales more interpretable, eect sizes

were calculated to represent the mag-

nitude o such dierences (Cohen,1988). With two groups (male/emale),

the dierence in scale means between

each group was divided by the pooled

standard deviation (Cohens d). Tus

the standardized dierence represents

the dierence between each group in

standard deviation units. With more

than two groups (race/ethnicity),

the standardized dierences were

represented by multiplying Cohens

by 2which is roughly equivalent tothe standardized dierence calculated

or two groups when the number

o observations in each cell is equal

(Cohen, 1988).

Construct validity was assessed by

examining the relationship o the

derived resilience scales to other theo-

retically related constructsinclud-

ing substance use, school violence,

school-related behavior, and stan-

dardized test scores. o examine these

relationships using a common metric,

correlations between resilience con-

structs and criterion variables romconrmatory actor analysis models

were estimated using the main and

validation samples. Latent constructs

represent continuous variables, while

the criterion variables are either

dichotomous or ordinal. Tus, polyse-

rial correlations are presented, which

represent the correlation between a

continuous variable and a dichoto-

mous or ordinal variable that reects

an underlying continuous variable(Bedrick & Breslin, 1996).

Box 2 (continued)

Data and analytic strategies


19/22

evaluating the pSychometric propertieS o the reSilience and youth development module 11

likely to be imprecise because o the instability o

the resilience measures. Even with low student-

level stability, however, the module is valuable or

tracking school and district prevalence estimates

o resilience assets. Student-level errors in mea-

surement likely cancel each other out when thedata are aggregated at the school, district, and

state levels.

Te secondary school module contains eight in-

ternally consistent and valid measures o environ-

mental resilience assets:

Tree measures representing supportive rela-

tionships in the school, community, and home

environments. Tese supportive relationships

include both caring relationships with andhigh expectations messages rom adults. Only

the measure or supportive relationships in

the home environment, however, demon-

strates sucient test-retest reliability or use

in research.

Tree measures o meaningul participation

or involvement in relevant, engaging, and

interesting activities with opportunities or

responsibility and contribution in school, in

the community, and at home.

wo measures o environmental assets in the

context o peerscaring relationships and

high expectations (aliation with pro-social

peers).

Tat the scales or caring relationships and high

expectations in the school environment turn out

to measure the same actor is consistent with

knowledge that has emerged since the resilience

and youth development module was developed inthe late 1990s. In ocus groups conducted by HKS

sta, when students were asked what they consider

to be actions that reect that a teacher cares about

you, they most ofen mentioned that the adult

is a good listener, sets high standards, expects

responsibility rom the student, praises successes,

and encourages the student through setbacks.

Akey (2006) ound that supportive teachers and

clear, high expectations

or behavior are key to

developing both stu-

dent engagement and

perceived competence.

eachers whom studentssee as supportive and

who set clear expecta-

tions or behavior create

an atmosphere where

students eel in control

and condent about their ability to succeed in

school. Akeys ndings suggest that supportive

teacher relationships, high academic expectations,

and high-quality pedagogy combine to enhance

student engagement and academic competence,

which lead to higher achievement, consistent withthe RYDM conceptual ramework. Te school and

home supportive relationships measures, however,

exhibit better psychometric qualities than many

other instruments designed to measure similar

constructs.

Scores on our o the internal asset scalessel-

ecacy, empathy, problem solving, and sel-

awarenessare internally consistent and adequate

or general research purposes. But the RYDM

items designed to measure cooperation and goals/aspirations do not, however, provide valid assess-

ments o these constructs.

Although the consistency o the associations o

environmental and internal resilience assets to

other related constructssuch as substance use,

school violence, school-related behavior, and stan-

dardized test scoressuggests that the measures

demonstrate construct validity, the associations

are weak. Tus the constructs exhibit only moder-

ate construct validity.

Results of the analysis of the elementary school module

Te elementary school resilience and youth devel-

opment module uses 21 items to assess seven en-

vironmental assets and three internal assets. Reli-

ably assessing so many actors with so ew items

is dicult, however, especially with a student

Th s

sh stmt

s t s

m t

t s t w st

t stt-

hs tm

f

ss stts


20/22


sel-report instrument. Not

surprisingly, the module reliably

assesses only two environmental

asset measures and one internal

asset measure, leaving consider-

able room or improvement.

Te elementary school module

measures meaningul participa-

tion, pro-social peers, and supportive relationships

in the school and home environments, but only the

school supportive relationships and home support-

ive relationships scales exhibit suciently high in-

ternal consistency or urther use. Only one reliable

internal resilience asset measure was detected or

elementary school studentsempathy. Te second

actor detected, goals/aspirations, was not reliableenough to be recommended or urther use. Te

third actor, problem solving, was not identied.

recoMMendaTionS

Tis report recommends that neither the second-

ary school nor the elementary school resilience

and youth development module be used to evaluate

student-level changes over time or individual di-

erences across students. Estimates o student-levelchanges across time are likely to be imprecise be-

cause o the instability o the resilience measures.

Other, longer, companion instruments should be

developed to assess student-level changes. Te

resilience and youth development module is still

useul as an epidemiological surveillance tool or

reporting aggregate district-level data, however.

Te ollowing sections provide recommendations

to drop or revise specic items in the module.

ables 3, 4, and 5 present the recommended mea-sures. (See appendix tables B24, B25, B26, and B27

or a side-by-side comparison o the current and

recommended measures.)

Secondary school environmental resilience assets

Recommendation 1Combine the caring relation-

ships and high expectations items. o maximize

construct validity and reduce redundancy across

scales, the caring relationships and high

expectations items should be combined to orm

one scale representing supportive relationships.

Caring relationships and high expectations are

indistinguishable as currently measured by themodule. Te new supportive relationships scale

should continue to be assessed separately or

school, community, and home environments.

Recommendation 2Drop Item R23 (I help other

people).Tis item should not be used to indicate

community meaningul participation because the

item unctions dierently, and thus has a dier-

ent meaning, or emales and Mexican American

youth. A new item that taps involvement in activi-

ties in the community should be developed.

Recommendation 3Drop Item R54 (I do un

things or go un places with my parents or other

adults). Te item is not developmentally appro-

priate or older adolescents because 11th graders

report substantially lower participation in such

activities or a given level o home meaningul

participation. Tis item distorts developmental

trends on the home meaningul participation scale

and should be dropped. A dierent item should be

developed to replace it.

Recommendation 4Drop item R45 (My riends

get into a lot o trouble). Because it is a biased in-

dicator o pro-social peers or emales and Chinese

American students, an alternative item should be

developed to measure this construct.

Secondary school internal resilience assets

Recommendation 5Drop the cooperation/com-

munication construct. wo o the items used tomeasure cooperation/communication measure

more than one construct: Items R36 (I enjoy

working together with other students my age)

and R37 (I stand up or mysel without put-

ting others down). Item R31 (I can work with

someone who has dierent opinions than mine)

should be moved to the sel-ecacy scale. Te

measurement models suggest that this item

h mt sh

m sssss

tw mt

sst mss

t sst ms,

s

m mmt


21/22

recommendationS 13

measures sel-ecacy better than it does coopera-

tion and communication.

Recommendation 6Drop the goals and aspira-tions construct. wo o the three items used to

measure this constructR24 (Goals and plans

or the uture) and R26 (I plan to go to college or

some other school afer high school)unction

dierently across racial/ethnic groups.

Recommendation 7Drop item R27 (I know

where to go or help with a problem). As an

indicator o problem solving, this item should be

dropped because it unctions dierently or males

and emales. An alternative item should be devel-

oped to assess problem solving.

Elementary school environmental and internal assets

Recommendation 8Develop more elementary

resilience items. Te elementary school resilience

and youth development module tries to assess too

many actors with too ew items. Because having

an elementary school resilience assessment that

taBle 3

rmm mss mt s ssts m s sh stts

cs i

S s

a w s b .

a w s w i .

a w ss w i s . . .

a w s w i jb.

a w ws ws bs.

a w bs i w b sss.

S

i s s.

i s k ss s s.

i s k .

c s

a w s b .

a w s w i s b . . .

a w i s.

a w s w i jb.

a w bs i w b sss.a w ws ws bs.

c i bs, ss s, /, . . .

i k sss s, , . . .

h s

a w s s s wk.

a w ks w b bs.

a w ss w i s . . .

a w s w s.

a w bs i w b sss.

a w ws ws bs.

h i s k .

i k ss w .

p ss

a w s b .

a w ks w b bs.

a w s w i .

p-s sm s w s .

m s w s.


22/22


is aligned with the secondary school module is

important, additional resilience items should be

developed or the elementary school survey. Each

o the elementary school RYDM scales demon-

strates inadequate domain coverage and marginalinternal consistency, at least one additional item

should be developed or each o the school sup-

portive relationships, home supportive relation-

ships, and empathy subscales. wo additional

items should be developed or the meaningul

participation at school and at home scales i it is

retained in the survey.

Recommendation 9Combine the caring rela-

tionships and high expectations items.As with

the secondary school module, the caring relation-ships and high expectations items should be

combined to orm one scale representing support-

ive relationships in both the school environment

and the home environment.

Recommendation 10Drop meaningul participa-

tion. Te meaningul participation scale should

either be dropped or redeveloped because o low

internal consistency. Moreover, item R15 (Do

you do things to be helpul at school?) shouldnot be used to indicate meaningul participation

because the item unctions dierently or males

and emales.

Recommendation 11Drop pro-social peers.Te

pro-social peers scale should be dropped because

one o the two items used to measure it unctions

dierently or males and emales. Perhaps items

rom other instruments that assess this construct

should be used instead.

Recommendation 12Drop goals and aspirations.

Te goals and aspirations scale should be dropped

or modied because o its low internal consistency.

Recommendation 13Develop a sel-efcacy

measure. Items should be developed to assess

sel-ecacy because this important construct is

currently not assessed.

taBle 4

rmm mss t s

ssts m s sh stts

cs i

S-

i wk w s w s

s .i wk bs.

i s s i .

t s i w.

e

i b w s s

s .

i s w

.

i s w

k.

pb s

W i i f s

k w.

i wk bs b k

w b .

S-wss

t s s .

i s s s.

i s w i w i .

taBle 5

rmm mss mt

t s ssts m

mt sh stts

cs i


S s

d s . . . s

b ?

ts . . . s w . . .

s s?

ts . . . w

jb?

ts . . . b

jb?

h s

p . . . b s wk?

p . . . s w

s s?

p . . . b

jb?

p . . . w

bs?


e

d s w

?

d b w s s

s ?

description: tags: rel 2007034 body

Documents